Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This submission proposes a dynamic resolution network that adaptively selects the suitable resolution for image recognition. A tiny resolution predictor is proposed to predict a smaller resolution for the input image of the large classifier. Thus the computational costs can be saved. The experimental results are good and the novelty of this paper is sufficient.
main_review: Generally, this paper proposes an intuitive and novel method, which reduces the FLOPs and improves the accuracy. The idea of reducing the resolution based on the input image is clever and convincing. The experimental results are impressive. The DRNet could yield a 1.4% accuracy increase while reducing 10% computational costs on ImageNet benchmark. Meanwhile, the author proposes a FLOPs constraint regularization, which makes the DRNet variant reduce 34% computational costs while maintaining similar accuracy. The paper is well written. Basically, I think this submission is above the standard of acceptance in NeurIPS2021.
limitations_and_societal_impact: 1. In Figure 3, the author shows the real speed improvement. The details of evaluating inference speed should be described, especially the batch size. I think that the dynamic mechanism may not reduce latency in batched inference.
2. There are many existing works on multiple resolutions and dynamic mechanisms. For example:
[1] ELASTIC: Improving CNNs with Dynamic Scaling Policies. In CVPR, 2019.
[2] Hydranets: Specialized dynamic architectures for efficient inference. In CVPR, 2018.
[3] Resolution Switchable Networks for Runtime Efficient Image Recognition. In ECCV, 2020.
These works have not been cited in this submission.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This manuscript proposes a dynamic resolution network that can be used for the base CNN model to dynamically change the resolution of the input images. The intuition of the proposed method lies in the fact that the difficulty of the input may differ. For the easy sample, one does not need to use the full resolution for the classification. The experiments show that the proposed DRN can reduce the computational cost a lot while keeping a similar performance and even better one.
main_review: The intuition of the proposed method is promising and reasonable since we do not always need to use the full resolution to make the decision. The techniques, including Resolution-aware BN and the Gumbel-softmax, to achieve the dynamic resolution are interesting and make the two networks can be trained in an end-to-end manner.  The experiments indeed show the effectiveness of the proposed method.
limitations_and_societal_impact: 1, All the experiments are conducted using images under 224*224 resolution, it would be interesting to see how the performance will be if we use a larger resolution.
2. The accuracy with lower resolutions for some examples is even better than the model with full resolution. Is there any underlying reason for this phenomenon?
3, It seems the improvement over the flops does align well with that over the real latency as shown in fig.3 and tab.3. It would be good to provide the performance and speed trade-off for real acceleration.
4, For the training process, the base models will be first trained and then combined with the resolution selector network for fine-tuning. I’m wondering if it is possible to train the whole model from scratch?

Some minor issues:
Line121: “The first is the large classifier network with both high performance and expensive computational costs is first trained”, is the “is first trained” redundant?
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: A dynamic resolution network is proposed in this submission. There is a computationally efficient predictor which selects the suitable resolution for the large classification CNN model. The predictor generates hard decision of resolutions. The usage of Gumbel softmax trick enables the gradient to propagate to the predictor. A FLOPs loss is also proposed to control the computation budget. Experiments on ImageNet show that this method could largely reduce the FLOPs with similar accuracy or improve the accuracy with fewer FLOPs reduction.
main_review: 1. the idea using a cheap and effective CNN network to determine the input resolution of a large model is novel and convincing. It is intuitive that the difficulties of recognizing the images are diverse. 
2. Besides, predicting the difficulties of the images is much easier than recognizing them, which means a tiny network is definitely sufficient to provide a good decision. 
3. DRNet achieves 1.4% accuracy improvement with 10% FLOPs reduction and 34% FLOPs reduction without degeneration of accuracy, which is a fairly good result. 
4. This paper clearly demonstrates this idea with good writing.
However, there are some concerns and limitations that need to be noted, e.g., some important related works are not cited.

-------------------post rebuttal---------------------

I think my concerns are well addressed in the authors' responses, So I keep my original score of 7.
limitations_and_societal_impact: 1. I think the dynamic mechanism is quite unfriendly for real-world applications. For example, if all the examples are “hard”, the computation cannot be saved. To make matters worse, the proposed predictor leads to more computation. 
2. There are other detailed issues in this submission. 
[1] Writing errors. For example, the resolution of the input image in most of the existing compressed networks are still fixed...(line 38). Han [5] propose to generate more feature maps...(line 91). There may be more writing errors that I do not list.
[2] The statements of ‘’FLOPs constraint regularization’’ and ‘’FLOPs loss’’ are better be unified since they represent the same thing.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper presents a method to reduce FLOP requirements by resizing images dynamically using a resolution prediction network, prepended to an image classifier network.  The resolution is selected from one of a handful of candidates, and resized before being passed to the classifier.  A training-time loss encourages smaller resolutions to be selected, effecting an accuracy vs resolution tradeoff controlled by two parameters.  Since selected resolution for inference is a one-hot categorical, the method uses Gumbel softmax to enable backpropagation during training.  A resolution-aware batchnorm layer is also described, which tracks batchnorm stats separately for each resolution.  The system performs very well, with over 40% reduction in average operations resulting in just one or two percent accuracy loss on imagenet, and small accuracy gains with 10% reduction in computation.

main_review: This method shows good results, and the approach of varying resolution to smallest required using a small convnet predictor is a sound approach.  The resolution-aware batchnorm is also a nice addition.  I had trouble understanding how Gumbel softmax is used: whether one-hot or mixtures are used in the training-time forward pass, and if they are one-hot, how gradients are estimated.  The description seems to indicate one-hot is used even at training time, but the referenced paper [15] describes a continuous mixture which is annealed down to one-hot over the course of training.  Currently, I don't understand how one resolution is compared to another, either within a training sample e.g. using a mixture or multiple distribution choices, or between samples e.g. using a reward or score estimator.  I think a clearer description of the mechanism that compares resolutions at training is needed.

The rest of the implementation and experiments are explained better, including the pretraining and finetuning phases, method comparisons, evaluation of parameters eta and alpha, separate batchnorms, and candidate resolutions.  It's interesting that the method performs slightly better than baseline with slightly lower FLOP count; it appears the multi-resolution training creates an effect similar to scaling data augmentation, though slightly different because images are not rescaled back to fixed size.

Overall this is a nice idea with good results.  But I think the explanation of how the forward pass sampling and backprop gradients are estimated should be clearer.




Detailed comments and questions:


- better describe Gumbel-Softmax use.  In [15], z is approximated with a continuous sample y, which is the Gumbel-Softmax, e.g. fig 2 in [15] shows y is continuous.  In this case, that would make h no longer one-hot --- how is this handled?

- [15] also use a temperature annealing schedule for tau to move from continuous to near-one-hot.  Is there an annealing schedule for tau here?


- if h is one-hot during training, how are the gradients estimated?

- X^\hat = sum_j X_j :  Since the X_j are different shapes for different resolutions, what does the sum mean here?



- what is the architecture of the resolution predictor R?  in particular, can the few conv layers used have relatively large stride? (or does increasing stride allow a resolution predictor with fewer flops, it seems fewer patch samples of the image may be needed to guess a good resolution compared to the classification net)?  l.255 mentions "4 stage residual net"


- largest resolution is 224 in these experiments --- what happens if this extended to be larger, up to 320 or even 512?  Even if this means upsampling the image, this will still change the network RF relative to object sizes.  Are the larger resolutions used when useful?  At the more extreme end, are candidate resolutions that are too large relative the network structure not selected?


- "fueled to":  is written several times, seems to be a typo.  not sure what was intended, probably "fed to" or possibly "funneled to"

- l.73: entired -> entire

- l.263: Tabel -> Table

limitations_and_societal_impact: yes
review_title: null

Meta-review: The manuscript has been reviewed by four experienced reviewers, all of whom, after reading the rebuttal provided by the authors, agree that the manuscript meets of the bar of NeurIPS and thus should be presented to a large audience. The AC also agrees that the proposed approach is novel, supported by sufficient emperical evaluations, and hence recommends acceptance. 
Decision: Accept (Poster)