Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper proposes a new method that involves multi-scale inputs for each layer that could be used as network architecture search or data agumentation or 

Pros)
(+) The idea looks interesting.
(+) The experimental results look promising.

Cons)
(-) Many typos when denoting figures and tables. See the minor comments below.
(-) I believe the authors could organize the paper better. Tables and figures that are referred in a page are hard to find quickly. I recommend the authors refine the paper again for better readability.
(-) Some notations (such as RS-, RS-NAS, and so on) are so vague that hard to follow.  
(-) I recommend the authors redraw all the figures for clarity. For example, each legend in Figure 2 is hard to take a look at.
(-) + the comments below.

Comments)
- When doing feature map resize in terms of the resolution, why Bilinear sampling was chosen? Could the authors provide a comparison with other sampling methods?
- In the related work section for dynamic neural networks, the authors claimed that "Most dynamic networks methods sacrifice accuracy in exchange of adaption in inference", but it seems to be quite overclaimed. As shown in the paper [1],  one can find that the author presented they could improve both accuracy and efficiency.
- How did you find the architecture shown In Figure 3 in the Appendix? What is Xception? Please specify the details.
-  Designing the pre-defined spatial list of L looks critical, so the authors should describe L in the implementation details. 
- One of the main problem I think is the training budget issue. According to algorithm 1, the inner loop of "for j=0,..,len(L)", the overall training time will clearly take L times longer than that of the training setting w/o resizable training. Thus, it does not seem to be fair comparison in terms of the training budget. Namely, it seems that the authors compared with the other data augmentation methods which spend much less training budgets.
- Hard to grasp Section 3.5 of Adaptive resizable neural network. ResizeLearner looks being attached at the last stage of the original network after the original network is trained, but there is no further information about what ResizeLearner learns and how ResizeLearner selects the optimal sub-network. 

[1] Universally Slimmable Networks and Improved Training Techniques, https://arxiv.org/pdf/1903.05134.pdf.

Minor comments)
- Wrong section and figure references:
  - 'It also mitigates the co-adaptation issue which we will discuss in Section 3.3'(indeed it is Section 3.4), 'The network architecture along with feature map resolution and channels number are shown in Figure 4' (it should be Figure 3).
  - - Figure 3(d) referred to in section 4.4  would be Figure 2(d) indeed.

About rating)
The authors provided a novel technique about the resizable approach and the experimental results look promising. However, the paper needs to be revised and looks like it does not ready to be published now. If the authors could revise the paper and concern my comments well, I would increase my rating. 
review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: This paper proposes Resizable Neural Networks, which trains networks with different resolution scalings at the same time with shared weights. It serves as data-augmentation and improves accuracy over base networks. Additionally, the same technique can perform an architecture search. Experimental results show significant accuracy gains.

The reported accuracy gains are substantial. The proposed method is potentially useful in many applications. However, several details are missing or hard to understand. Without additional descriptions, it is not straightforward to implement the method. Thus, I suggest for rejection. The score might be raised depending on updates and code release.

Major comments:
1) Algorithm 1 has "predefined spatial list L." How to choose it in practice?
2) Algorithm 1 indicates that the training time is len(L) times longer. Additionally, according to the implementation details, Resizable Networks are trained two times many epochs. It seems hard to justify such a longer training time.
3) This is similar to (2), but why Resizable-NAS is better than the all len(L) models separately to find better architectures?
4) In Sec. 3.4, how the "target model" is selected after the training of Resizable Networks?
5) Why is Resizable-Adapt omitted from Table 2?
6) References are out-dated. For example, there are no "Figure 5."

Minor comments:
1) Are any ablation studies available for Fair Sampling?

===== Update

Thank you for the response and update. The revision made the paper easier to understand. However, I still have concerns about the presentation of the paper, and I keep my score. I think the novelty and experimental results are significant and match the bar of ICLR. If the other two reviewers think that the revised paper is fairly well-written and recommend acceptance, I will not challenge the decision.

Major comments:
1) I do not understand how the training times of random-sampling in Appendix C were estimated.
2) Concerning Table 11, how was the result when we randomly select each scale and train network with the same number of epochs with fair-sampling? If there were no much difference with the result in Table 11, it seems fair sampling does not have clear advantages over random sampling. If so, I suggest to alter fiar-sampling by random-sampling and reduce the complexity of the proposed method.

Minor comments:
1) Table 10 has two captions.
2) I did not understand that L is a list of possible scaling factors (scalar) in the initial review. I guess it is partially because I did not understand why the number of sampling should be len(L) for each mini-batch (it is actually due to fair-sampling). I think adding some remarks on it will help to make the pseudo-code easier to understand. (nit: for j= 0, ..., len(L) should be for j=0, ..., len(L) -1 or for j=1,  ..., len(L))
review_title: Official Blind Review #3

final_score: rating: 6: Weak Accept
review_content: In a standard multi-stage neural network such as a ResNet, the resize operation between stages typically reduces the spatial resolution by 0.5.  (input ---> stage 1---> 0.5 reduction ---> stage 2 --> 0.5 reduction ---> ....). In this paper the authors apply a variety of reduction factors in between these stages for different training examples (input ---> stage 1---> variable reduction ---> stage 2 --> variable reduction ---> ....). They demonstrate that simply training in this way is a powerful form of augmentation. It also produces a network which may be scaled depending on a FLOP budget.

I think this is a really neat idea, and as far as I'm aware it is novel.  It is similar in spirit to EfficientNet, although more flexible. The experimental results are good. However, the paper is let down by poor writing and a lack of detail.

The paper needs a rewrite as there are many grammatical errors, which cause a bad impression:

- "performs arbitrary resize operation" ->  "performs arbitrary resize operations"
- "Scale is the fundamental component of the physical world" ---> *a* fundamental component!
- "i.e the acuracy of NISP" --> e.g.
- "compare with baseline" -> "compared to the baseline"
- There are several instances of a space missed out between a letter and an open bracket 
- "The Figure 2(a)" --> "Figure 2(a)"

The comparison to weight-sharing NAS methods is unnecessary. Those entail searching for architectures and sharing weights through the search process, whereas in this work it is having a network that can take different sized inputs at different stages. On that note, it doesn't feel right to me to refer to there being many "subnetworks". What there really is, is just a single network that is robust to multi-scaled inputs (which is a good thing!)

Figure 1 is nice!

EfficientNet-B0 is a base-network that can be scaled up with a compound scaling approach found through grid search. What happens if you scale up your Resizeable net to the same FLOPs as e.g. EfficientNet-B7?

I like the old-school vision citations, although referring to object detection makes me wonder why there are no experiments on it. For distillation, I recommend you cite https://arxiv.org/abs/1312.6184, as the Hinton paper is really just an extension of this.

The fair sampling seems important to the method. Could a detailed explanation be included in the appendix? Do you given the performance as a result of naive sampling? What do you mean by "Some convolutional layers can go through more training issues than others"? 

A big problem in this paper is that (as far as I can tell) the scaling factors considered aren't given (but we are told that they lie between 0 and 1). It isn't possible to sample these arbitrarily as you indicate that a batch-norm layer is needed for each one, so these must be discrete (because of this, it isn't correct to say that you have infinite networks contained within).  It therefore isn't clear e.g. in Table 1 which permutation of scaling factors were used in upsampling the networks. Are the results given representative of the best-case selection of these scaling? I hope this isn't the case. I am assuming it is uniform scaling in the case of the "individually trained counterparts". It would be interesting to know more generally which combinations worked well.

In Section 3.5 I'm not sure what is meant by "It is inefficient to process all images to one-subnetwork, as the algorithm spends equal energy at each sample".  I assume energy use is proportional to the number of FLOPS, which in turn depends on spatial resolution.

The legend in Figure 2 is close-to unreadable and needs changing.

The results are impressive, but error bars would be appreciated if possible. As ImageNet is the only dataset considered, this would give some needed clout. 

Pros
-------

- Nice, novel method
- Good experimental results

Cons
--------

- Paper is poorly written
- Very few details of the scaling factor variations
- Only one dataset considered

Although the paper is written badly and the narrative is muddled, the underlying idea is a nice one, which is executed well experimentally. Because of this I would like to recommend a Weak Accept, subject to the authors (i) doing a rewrite and (ii) including more information regarding the scaling permutations. 
review_title: Official Blind Review #2

Meta-review: This paper offers likely novel schemes for image resizing.  The performance improvement is clear.  Unfortunately two reviewers find substantial clarity issues in the manuscript after revision, and the AC concurs that this is still an issue.  The paper is borderline but given the number of higher ranked papers in the pool is unable to be accepted unfortunately. 
Decision: Reject