Dynamic Resolution Network
Year: 2021

Abstract:
Deep convolutional neural networks (CNNs) are often of sophisticated design with numerous learnable parameters for the accuracy reason. To alleviate the expensive costs of deploying them on mobile devices, recent works have made huge efforts for excavating redundancy in pre-defined architectures. Nevertheless, the redundancy on the input resolution of modern CNNs has not been fully investigated, i.e., the resolution of input image is fixed. In this paper, we observe that the smallest resolution for accurately predicting the given image is different using the same neural network. To this end, we propose a novel dynamic-resolution network (DRNet) in which the input resolution is determined dynamically based on each input sample. Wherein, a resolution predictor with negligible computational costs is explored and optimized jointly with the desired network. Specifically, the predictor learns the smallest resolution that can retain and even exceed the original recognition accuracy for each image. During the inference, each input image will be resized to its predicted resolution for minimizing the overall computation burden. We then conduct extensive experiments on several benchmark networks and datasets. The results show that our DRNet can be embedded in any off-the-shelf network architecture to obtain a considerable reduction in computational complexity. For instance, DR-ResNet-50 achieves similar performance with an about 34% computation reduction, while gaining 1.4% accuracy increase with 10% computation reduction compared to the original ResNet-50 on ImageNet. Code will be available at https://gitee.com/mindspore/models/tree/master/research/cv/DRNet.

heading: 1 Introduction
text: Deep convolutional neural networks (CNNs) have achieved remarkable success in various computer vision tasks, under the development of algorithms [6, 22, 37], computation power, and large-scale datasets [2, 17]. However, the outstanding performance is accompanied by large computational costs, which makes CNNs difficult to deploy on mobile devices. With the increasing demand for CNNs on real-world applications, it is imperative to reduce the computational cost and meanwhile maintain the performance of neural networks.
Recently, researchers have devoted much effort to model compression and acceleration methods, including network pruning, low-bit quantization, knowledge distillation, and efficient model design. ∗Equal contribution. † Corresponding author. This research has been supported by the Key R&D program of Zhejiang Province (Grant No. 2021C03139). This work was supported by NSFC (62072449, 61632003), Guangdong-Hongkong-Macao Joint Research Grant (2020B1515130004), and Macao FDCT (0018/2019/ AKP).
35th Conference on Neural Information Processing Systems (NeurIPS 2021).
Network pruning aims to prune the unimportant filters or blocks that are insensitive to model performance through a certain criterion [31, 15, 18, 25]. Low-bit quantization methods represent weights and activation in neural networks with low-bit values [11, 13]. Knowledge distillation transfers the knowledge of the teacher models to the student models to improve the performance [7, 38, 40]. The efficient model design utilizes lightweight operations like depth-wise convolution to construct some novel architectures successfully[8, 42, 5]. Orthogonal to those methods that usually focus on the network weights or architectures, Guo et.al. [4] and Wang et.al. [33] study the redundacy that exists in the input images. However, the resolutions of the input images in most of existing compressed networks are still fixed. Although deep networks are often trained using an uniform resolution (e.g., 224× 224 on the ImageNet), sizes and locations of objects in images are radically different. Figure 1 shows some samples that the required resolution for achieving the highest performance are different. For the given network architecture, the FLOPs (floating-number operations) of the network for processing image will be significantly reduced for images with lower resolution.
panda bee damselfly
R eso lu tio n lo w
h igh
understood easily just in blurry mode while the others need to be seen in clear mode.
In this paper, we propose a novel dynamic-resolution network (DRNet) which dynamically adjusts the input resolution of each sample for efficient inference. To accurately find the required minimum resolution of each image, we introduce a resolution predictor which is embedded in front of the entire network. In practice, we set several different resolutions as candidates and feed the image into the resolution predictor to produce a probability distribution over candidate resolutions as the output. The network architecture of the resolution predictor is carefully designed with negligible computational complexity and trained jointly with classifier for recognition in an end-to-end fashion. By exploiting the proposed dynamic resolution network inference approach, we can excavate the reduncancy of each image from its input resolution. Thus, computational costs of easy samples with lower resolutions can be saved, and the accuracy for hard samples can also be preserved by maintaining higher resolutions. Extensive experiments on the large-scale visual benchmarks and the conventional ResNet architectures demonstrate the effectiveness of our proposed method for reducing the overall computational costs with comparable network performance.

heading: 2 Related Works
text: Although deep CNN models have shown excellent accuracy, they often contain millions of parameters and FLOPs. Thus the model compression techniques are becoming a research hostpot for reducing the computational costs of CNNs. Here, we revisit existing works in two parts, i.e., static model compression and dynamic model compression.
We classify model compression methods that are not instance-aware as the static. Group-wise Convolution (GWC), Depth-wise Convolution and Point-wise Convolution are widely used for efficient model design, such as MobileNet [8], ResNeXt [36], and ShuffleNet [42]. Revealing pattern redundancy among feature maps, Han [5] proposes to generate more feature maps from intrinsic ones through some cheap operations and Zhang et.al. [41] adopts relatively heavy computation to extract intrinsic information while tiny hidden details are processed with some light-weight operations. The methods above achieve model acceleration to some extent. However, they treat all input samples equally, whereas the difficulty for CNN models or humans to recognize each sample is unequal. So instance-aware model compression can be further explored.
Dynamic model compression takes the unequal difficulty of each sample into consideration. Huang [9] proposes multi-scale dense networks with multiple classifiers to allocate uneven computation across "easier" and "harder" inputs. Wu et.al. [35] introduces BlockDrop which learns to dynamically execute the necessary layers so as to best reduce total computation. Veit et.al. [28] proposes ConvNetAIG to adaptively define their network topology conditioned on the input image. Except for dynamic adjustment on model architectures, recent works pay more attention to input images. Verelst et.al. [29] proposes a small gating network to predict pixel-wise masks determining the locations where dynamic convolutions are evaluated. Uzkent et.al. [27] proposes PatchDrop to dynamically identify when and where to use high-resolution data conditioned on the paired low-resolution images. Yang et.al. [39] dynamically utilize sub-networks of the base network to process images with different resolutions. Wang et.al. [33] proposes GFNet with patch proposal networks that strategically crop the minimal image regions to obtain reliable predictions. There exists many works on multiple resolutions and dynamic mechanisms. ELASTIC [30] uses different scaling policies for different instances and it learns from the data how to select the best policy. Hydranets [20] chooses different branches for different inputs by a gate and aggregates their outputs with a combiner. RS-Net [32] utilizes private BNs, shared convolutions, and fully-connected layer and to train input images with different resolutions.
Different from dynamic adjustment on model architectures and dynamic modification on input images with reinforcement learning, we consider the whole image and propose a resolution predictor to dynamically choose the performance-sufficient and cost-efficient resolution for a single model to obtain reliable prediction with end-to-end training.

heading: 3 Approach
text: In this section, we first introduce the overall framework of the proposed Dynamic Resolution Network (DRNet), then describe the resolution predictor, resolution-aware BN, and optimization algorithm in detail, respectively.

heading: 3.1 Dynamic Resolution Network
text: Inspired by the fact that different sample requires different resolution to achieve the least accurate prediction, we propose to develop an instance-aware resolution selection approach for a single large classifier network. As shown in Figure 2, the proposed method mainly consists of two components. The first is the large classifier network with both high performance and expensive computational costs, such as the classical ResNet [6] and EfficientNet [24]. The other is a resolution predictor for finding the minimal resolution so that we can adjust the input resolution of each image to have a better trade-off on the accuracy and efficiency. For an arbitrary input image, we first forecast its suitable resolution r using the resolution predictor. Then, the large classifier will take the resized image as inputs and the required FLOPs will be reduced significantly when r is lower than that of the origional resolution. To achieve better performance, the resolution predictor and the base network are optimized end-to-end during training.
Resolution Predictor. The resolution predictor is designed as a CNN-based preprocessing operation before input samples are fed to the large base network. It’s from the inspiration that our well-trained large models can also predict a relative amount of samples correctly though they are in relatively small resolutions while large amounts of computation cost can be saved. On the one hand, the goal of the resolution predictor is to find an appropriate instance-aware resolution by inferring a probability distribution over candidate resolutions. Note that there are a vast number of candidates from 1×1 to 224×224, which makes it difficult, also meaningless, for the resolution predictor to explore such a long-range of resolutions. As a simplification strategy and practical requirement, we choose m resolution candidates r1, r2, · · · , rm to shrink the exploration range. On the other hand, we have to keep the model size of the proposed resolution predictor as small as possible since it will bring extra FLOPs, otherwise, it becomes impractical to implement such a module if its extra introduced computation exceeds the saved one from the low resolution. In this spirit, we design the resolution predictor with a few convolutional layers and fully-connected layers to complete a resolution classification task. Then the preprocessing of the proposed resolution predictor R(·) can be given as follows:
pr = [pr1 , pr2 , ..., prm ] = R(X), (1)
where X is the input samples fed to the resolution predictor, m is the total number of candidate resolutions, and pr ∈ Rm is the outputs of the resolution predictor which represents the probability of each candidate. Then the resolution corresponding to the highest probability entry is selected as the resolution fed to the large classifier. Since the process from the soft outputs of resolution predictor to the discrete resolution resizing operation does not support end-to-end training, here we adopt Gumbel-Softmax [14] module G to turn soft decisions pr into hard decisions h ∈ {0, 1}m by applying Gumbel-Softmax trick to solve the non-differentiable problem:
h = G(pr) = G(R(X)), (2)
where the Gumbel-Softmax trick will be described in the next subsection. For validation, the resolution predictor makes decisions first then the input with the selected resolution only is fed to the large classifier in the normal way as shown in the left part of Figure 2.
Resolution-aware BN. Our framework is proposed to use only a single large classifier for the sake of storage pressure and loading latency, which results in that the single classifier has to process multi-resolution inputs and raises two problems. One obvious problem is that the first fully-connected layer will fail to work with a different input resolution and can be solved with global average pooling. Thus we can process multiple resolutions in one single network. The other hidden problem exists in Batch Normalization (BN) [12] layers. BN is used to make deep models converge faster and more stable through channel-wise normalization of the input layer by re-centering and re-scaling. However, activation statistics including means and variances under different resolutions are incompatible [26]. Using shared BNs under multiple resolutions leads to lower accuracy in our experiments as shown
in section 4.3. Since the batch normalization layer contains a negligible amount of parameters, we propose resolution-aware BNs as shown in Figure 2. We decouple the BN for each resolution and choose the corresponding BN layer to normalize the features:
xj = γj xj − µj√ σj2 + + βj , j ∈ {1, 2, ...,m}, (3)
where is a small number for numerical stability, µi and σi are private averaged mean and variance from the activation statistics under separate resolutions; βi and γi are private learnable scale weights and bias. Since shared convolutions are insensitive to performance, the overall adjustment for the original large classifier is shown in the right part of Figure 2.

heading: 3.2 Optimization
text: The proposed framework is optimized to perform instance-aware resolution selection for inputs of a single large classifier with end-to-end training. The loss function and Gumbel softmax trick are described in the following.
Loss Function. The base classifier and the resolution predictor are optimized jointly. The loss function includes two parts: the cross-entropy loss for image classification and a FLOPs constraint regularization to restrict the computation budget.
Given a pretrained base image classifier F which takes image X as input and outputs the probability predictions y = F(X) for image classification, we optimize the resolution predictor and finetune the pretrained base classifier together so as to make them compatible with each other. For the input image X , we first resize it into m candidate resolutions as Xr1 , Xr2 , · · · , Xrm . We use the proposed resolution predictor to produce the resolution probability vector pr ∈ Rm for each image. The soft resolution probability pr is transformed into hard one-hot selection h ∈ {0, 1}m using Gumbel-Softmax trick as equation 2 where the hot entry of h represents the resolution choice for each sample. We first obtain the final prediction of each resolution yrj = F(Xrj), and then sum them up with h to obtain the recognition prediction for the selected resolution:
ŷ = m∑ j=1 hjyrj . (4)
The Cross-Entropy lossH is performed between ŷ and target label y as follows: Lce = H(ŷ, y). (5)
The gradients from the loss Lce are back-propagated to both the base classifier and the resolution predictor for optimization.
If we use the Cross-Entropy loss only, the resolution predictor will converge to a sub-optimal point and tend to select the largest resolution because samples with the largest resolution correspond to relatively lower classification loss generally. Although the classification confidence of the lowresolution image is relatively lower, the prediction can be correct and requires fewer FLOPs. In order to reduce the computational cost and balance the different resolution selection, we propose a FLOPs constraint regularization to guide the learning of resolution predictor:
F = m∑ j=1 (Cj · hj), (6)
Lreg = max
( 0,
E(F )− α Cmax − Cmin
) , (7)
where F is the actual inference FLOPs, Cj is the pre-computed FLOPs value for the j-th resolution, E(·) is the expectation value over samples, and α is the target FLOPs. Through this regularization, there will be a penalty if averaged FLOPs value is too large, enforcing the proposed resolution predictor to be instance-aware and predict the resolution that is both performance-sufficient (with correct prediction) and cost-efficient (with low resolution).
Finally, the overall loss is the weighted summation over the classification loss and the FLOPs constraint regularization term: L = Lce + ηLreg, (8) where η is a hyper-parameter to match the magnitude of Lce and Lreg .
Gumbel Softmax Trick. Since there exists an non-differentiable problem in the process from the resolution predictor’s continues outputs to discrete resolution selection, we adopt Gumbel Softmax trick [19, 14] to make discrete decision differentiable during the back-propagation. In Eq. 1, the resolution predictor gives the probabilities for the resolution candidates pr = [pr1 , pr2 , ..., prm ]. Then the discrete candidate resolution selections can be drawn using:
h = one_hot[(argmax j (log prj + gj)], (9)
where gj is Gumbel noise obtained through two log operation applied on i.i.d samples u drawn from a uniform distribution as follows:
gj = − log(− log u), u ∼ U(0, 1). (10)
During training, the derivative of the one-hot operation is approximated by Gumbel softmax function which is both continuous and differentiable:
hj = exp (log(πj) + gj)/τ∑m
j=1exp((log (πj) + gj/τ)) , (11)
where τ is the temperature parameter. The introduction of Gumbel noise has two positive effects. On the one hand, it will not influence the highest entry of the original categorical probability distribution. On the other hand, it makes the gradient approximation from discrete hardmax to continuous softmax more fluent. By this straight-through Gumbel softmax trick, we can optimize the overall framework end-to-end.

heading: 4 Experiments
text: To show the effectiveness of our proposed method, in this section, we conduct experiments on the small-scale ImageNet-100 and large-scale ImageNet-1K [2] with classic large classifier networks, including ResNet [6] and MobileNetV2 [23], where we replace their single batch normalization layer(BN) with resolution-aware BNs and add the proposed resolution predictor to guide the resolution selection.

heading: 4.1 Implementation Details
text: Datasets. ImageNet-1K dataset (ImageNet ILSVRC2012) [2] is a widely-used benchmark to evaluate the classification performance of neural networks, which consists of 1.28M training images and 50K validation images in 1K categories. ImageNet-100 is a subset of the ImageNet ILSVRC2012, whose training set is random selected from the original training set and consists of 500 instances of 100 categories. The validation set is the corresponding 100 categories of the original validation set. The categories of ImageNet-100 is provided in supplementary materials. For the license of ImageNet dataset, please refer to http://www.image-net.org/download.
Experimental Settings. For data augmentation during training for both ImageNet-100 and ImageNet-1K, we follow the scheme as in [6] including randomly cropping a patch from the input image and resizing to candidate resolutions with the bilinear interpolation followed by random horizontal flipping with probability 0.5. For data processing during validation, we first resize the input image into 256× 256 and then crop the center 224× 224 part. The details of the resolution predictor are provided in supplementary materials. For both datasets, we firstly employ the images of different resolutions to pre-train a model without the resolution predictor. The losses of each resolution are summed up for optimization. Then we add a designed predictor to the model and conduct finetuning. Optimization is performed using SGD (mini-batch stochastic gradient descent) and learning rate warmup is applied for the first 3 epochs. In the pretraining stage, the model is trained with total epochs 70, batch-size 256, weight decay 0.0001, momentum 0.9, initial learning rate 0.1 which decays a factor of 10 every 20 epochs. We adopt a similar training scheme in finetuning stage. The total epochs are 100 with the learning rate decaying a factor of 10 every 30 epochs. We adopt 1× learning rate to finetune the large classifier and 0.1× learning rate to train the resolution predictor from scratch. The framework is implemented in Pytorch [21] on NVIDIA Tesla V100 GPUs.

heading: 4.2 ImageNet-100 Experiments
text: We conduct small-scale experiments on ImageNet-100 to guide the resolution selection for large classifiers ResNet-50. The resolution predictor is designed as a 4-stage residual network with input resolution 128 × 128 where each stage contains one residual basic block, which consumes about 300 million FLOPs. For candidate resolutions of the large classifier, we choose resolutions of [224× 224, 168× 168, 112× 112] and we denote them as [224, 168,112] for simplicity. Thus the last fully-connected layers of the resolution predictor contain three neurons. We only replace each batch normalization layer in ResNet-50 with three optional resolution-aware batch normalization layers, change the last average pooling layer to an adaptive average pooling layer, and then integrate the resolution predictor to form the overall framework. Experiment results are shown in Table 1.
For the calculation of the average FLOPs in Table 1, we sum up the FLOPs of each sample under the predicted resolution, take the extra FLOPs introduced by the resolution predictor into account and finally take the average over the whole validation set. From the results in Table 1, we can see our dynamic-resolution ResNet-50 obtains about 17% reduction of average FLOPs while gains 4.0% accuracy increase with the candidate resolutions [224, 168, 112]. When we tune the hyperparameters (i.e., η and α) in FLOPs constraint regularization, the dynamic-resolution ResNet-50 obtains about 32% FLOPs reduction and achieve 1.8% increase in accuracy. We also extend the range of resolutions (i.e., [224, 192, 168, 112, 96]) for fully exploration, especially the lower resolution. We can see that our DRNet still performs better than the baseline model. Setting a larger α can even obtain 44% FLOPs reduction with performance increase, which is shown in Table 2.

heading: 4.3 Ablation Study
text: To form the dynamic resolution network, we propose two adjustments, 1) replacing each BN layer with resolution-aware BNs; 2) proposing a FLOPs balance regularizer. Here we conduct ablation studies on ImageNet-100 to investigate the influence of each part.
Resolution-Aware BN. Here we compare the results where the large classifier ResNet-50 is equipped with resolution-aware BN or not. From Table 1, we can see that resolution-aware BNs obtain extra one more points for ResNet-50 with similar computational cost, which demonstrates that we need to normalize feature maps with different resolutions separately, thus their activation statistics can be more accurate.
Influence of FLOPs Constraint Regularization. Here we explore the influence of the penalty factor η and target FLOPs value α in the FLOPs constraint regularization as shown in Table 2. We first fix η as 0.2 and tune α from 2.0 to 3.5. We can see that the average FLOPs increase from 2.3G to 3.5G gradually, and the accuracy also increase consequently. As for η, we fix α = 3.0 and tune η in the range of [0.1, 1]. A larger penalty factor leads to lower FLOPs and accuracy. That is to say when selecting images dynamically, the resolution predictor with lower penalty η tends to choose the larger resolution, where the effect of the balance regularizer is relatively weaker.

heading: 4.4 ImageNet-1K Experiments
text: ResNet Results. We conduct large-scale experiments with DR-ResNet-50 on ImageNet-1K as shown in Table 3. When we set the candidate resolutions as [224, 168, 112], our DR-ResNet-50 also outperforms the baseline by 1.4 percentage with 10% FLOPs reduced. Similar to the results in Table 2, the effectiveness of FLOPs constraint regularization is also verified in Table 3, where the FLOPs drop with larger α. Our DRNet focuses on input resolution and keeps the structure of the large classifier almost unchanged, thus the parameters of our DRNet-equipped models are more than the original large classifier due to the introduction of the resolution predictor. In other words, since our DRNet is orthogonal to those architecture compression methods, careful combinations of the two methods would make a more compact result.
We also compare DR-ResNet-50 with other representative model compression methods to verify the superiority of the proposed method. The compared methods include Sparse Structure Selection (SSS) [10], Versatile Filters [34], PFP [16], and C-SGD [3]. As shown in Table 4, our DR-ResNet-50 achieves better performance than other methods with similar FLOPs.
Effect of Dynamic Resolution. To evaluate the effect of the proposed dynamic resolution mechanism, we compare DR-ResNet-50 with randomly selected resolution. We repeat the random selection 3 times and report their accuracies and FLOPs in Imagenet-1K dataset. From the results in Table 5, DRNet shows much better performance than random baseline, indicating the effectiveness of dynamic resolution.
On-device Acceleration of Dynamic Resolution. In Figure 3, we demonstrate the practical accelerations of our DR-ResNet-50, which are obtained by measuring the forward time on a Intel(R) Xeon(R) Gold 6151 CPU. We directly set the batch size as 1 and the input resolution for the resolution predictor is 128. The candidate resolutions are 224, 168, and 112. We average the test time in ImageNet-1K val set. Our model substantially outperforms ResNet-50 by a significant margin.
MobileNetV2 Results. We also test our method on a representative lightweight neural network, i.e.MobileNetV2 [23]. We set the candidate resolution as [224, 168, 112]. The training setting of MobileNetV2 follows that in the original paper [23] for a fair comparison. To reduce the FLOPs
Model FLOPs Acc (%) Random-1 2.6 G 74.70 Random-2 2.6 G 74.65 Random-3 2.6 G 74.60
Random (mean) 2.6 G 74.65±0.04 DRNet 2.7 G 76.2
Table 5: Dynamic resolution vs. Random resolution.
80 90 100 110 120 130 140 Latency (ms)
73
74
75
76
77
78
Ac cu ra cy ( %)
DRNet ResNet-50
Figure 3: Acc vs. Latency.
of the resolution predictor, we replace the residual block with an inverted residual block and set the input size as 64×64. From the results in Table 6, we can see that DRNet achieves 72.7% top-1 accuracy with fewer computational costs.

heading: 4.5 Visualization
text: The prediction results of the resolution predictor are visualized in Figure 4. The first four samples with obvious foreground which occupy most of the whole image are predicted to use 112 × 112 resolution in high confidence. The middle three whose foreground is a little blurred are predicted to select 168 × 168 resolution. The last three samples’ hidden foregrounds nearly blend with the background, thus the largest resolution are selected. Although the “easy” and “hard” examples may be different for humans and machines, these results are compatible with the human perception system.

heading: 5 Conclusion
text: In this paper, we reveal that different sample acquires different resolution threshold to achieve the least accurate prediction. Thus large amounts of computation cost can be saved for some easier samples under lower resolutions. To make CNNs predict efficiently, we propose a novel dynamic resolution network to dynamically choose the performance-sufficient and cost-efficient resolution for each input sample. Then the input is resized to the predicted resolution and fed to the original large classifier, in which we replace each BN layer with resolution-aware BNs to accommodate the multi-resolution input. The proposed method is decoupled with the network architecture and can be generalized to any network. Extensive experiments on various networks demonstrate the effectiveness of DRNet.

Review summaries:
A dynamic resolution network is proposed in this submission. There is a computationally efficient predictor which selects the suitable resolution for the large classification CNN model. The predictor generates hard decision of resolutions. The usage of Gumbel softmax trick enables the gradient to propagate to the predictor. A FLOPs loss is also proposed to control the computation budget. Experiments on ImageNet show that this method could largely reduce the FLOPs with similar accuracy or improve the accuracy with fewer FLOPs reduction.

The paper presents a method to reduce FLOP requirements by resizing images dynamically using a resolution prediction network, prepended to an image classifier network.  The resolution is selected from one of a handful of candidates, and resized before being passed to the classifier.  A training-time loss encourages smaller resolutions to be selected, effecting an accuracy vs resolution tradeoff controlled by two parameters.  Since selected resolution for inference is a one-hot categorical, the method uses Gumbel softmax to enable backpropagation during training.  A resolution-aware batchnorm layer is also described, which tracks batchnorm stats separately for each resolution.  The system performs very well, with over 40% reduction in average operations resulting in just one or two percent accuracy loss on imagenet, and small accuracy gains with 10% reduction in computation. 

This submission proposes a dynamic resolution network that adaptively selects the suitable resolution for image recognition. A tiny resolution predictor is proposed to predict a smaller resolution for the input image of the large classifier. Thus the computational costs can be saved. The experimental results are good and the novelty of this paper is sufficient.

This manuscript proposes a dynamic resolution network that can be used for the base CNN model to dynamically change the resolution of the input images. The intuition of the proposed method lies in the fact that the difficulty of the input may differ. For the easy sample, one does not need to use the full resolution for the classification. The experiments show that the proposed DRN can reduce the computational cost a lot while keeping a similar performance and even better one.