Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors propose a new training regime for multi-resolution slimmable networks. Their approach is based on US-Net training technique but in addition to sampling from different network widths they also sample from different input resolutions and show that using multi-scale inputs improves the top-1 accuracy on ImageNet comparing to US-Net or MobileNet v1/v2 within the same resource constraints.

Pros:
+ The authors correctly identify input resolution as one of the aspects of lightweight network design that is often overlooked
+ They propose a practically viable training scheme that can be used to train & select networks given resource constraints
+ The paper is well written and includes many insightful experimental findings

Con:

The authors specify the mutual learning from width and resolution as their main contribution. They insist that treating input resolution independently from network structure is what distinguishes previous work from the newly suggested technique. But the paper doesn't include extensive experimental comparisons with the approaches that treat input resolution independently. Thus its claim that joint width/resolution sampling is beneficial comparing to independent approaches is somewhat unfounded. 

For example, the authors show that MobileNet with 1.0-224 config (no sampling from widths nor from input resolutions during training) is outperformed by their network with 1.0-224 config (which effectively samples only from input resolutions during training). This is not surprising as one can view sampling from input resolutions as an equivalent to data augmentation. The importance of data augmentation is well known, so to prove the proposed mutual learning is beneficial the authors would need to compare against the networks that were trained using this multi-scale data augmentation. Figure 5 has a similar comparison but the only multi-resolution baseline there is US-Net+ which isn't using multi-resolution images in training. The paper would greatly benefit from adding such comparisons and proving they are not marginal.

On rating:

I'd summarize the idea of this paper as A) US-Net + B) multi-scale data augmentation + C) selecting the best network based on both input resolution and width to achieve optimal performance within resource constraints. Although C is practical and novel contribution, it is also quite straightforward. I would like to see authors response on how their approach differs from US-Net + multi-scale data augmentation for training and how/why this works better.
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper proposes a multi-resolution training scheme for a slimmable network. The proposed method provides a new regime leveraging diverse image resolutions for training sub-networks, which enables efficient network scaling. Throughout the experiments, the authors claim the proposed method shows better performance compared to US-net.

Pros)
(+) The idea of multi-resolution training combining to a slimmable network looks good
(+) Applying a slimmable network's technique to other tasks including detection and segmentation looks good
(+) The combination of multi-resolution and the slimmable network seems to be reasonable.
(+) The paper is well written and looks justified well.
(+) The authors provided extensive experiments.

Cons)
(-) There is no backups why the proposed method could outperform over US-net. 
(-) The proposed method is incremental and improvements are marginal.
(-) Looks like there exists missing in details of the experiments.
(-) The performance report of the compared methods is quite strange.

Comments)
- The proposed method is too straightforward, so the authors should clarify why it works over US-net. Additionally, can the authors provide advantages using a different image-scale need for training a different sub-network? 
- The authors should clarify the training details of US-Net used in this paper. The performance of US-Net in Figure 4 (a) looks the same as the performance of US-Net trained with [0.05, 1]x scaling in the original paper. However, in the original paper, the authors of US-net reported [0.05, 1]x scaling as the worst performance setting in the original paper. Therefore, the authors should compare their method with the best performance setting of US-Net, which is [0.25, 1]x (because the proposed method looks being used [0.25, 1]x training setting, so the comparison should be done in fair).
- The scaling parameters of US-Net used in the experiments should be specified. All the results of US-Net do not contain where they come from (i.e., the training width bound in US-net).
- Can the authors report the results for 0.5-224 and 0.15-224 in Figure.4(a)? Why 0.7-160 and 0.25-160 were picked?
- In Table 1, the performance of EfficientNet is weird. EfficientNet-B2 has 79.8% accuracy with 1.0B FLOPs, but the reported performance in this paper of EfficientNet has 75.6% accuracy with 2.3B FLOPs. Please clarify this.
- How much does KLdiv contribute to the overall performance?
- All the tables are not clearly shown. Please reattach all the tables for better readability.

About rating)
I think the idea looks novel, but the method is quite straightforward, and the paper does not incorporate any analysis as a backup for the proposed method. The initial rating is towards reject, but I would like to see the authors' response and the other reviewers' comments. After that, the final rating might be changed.
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: The paper explore how varying input image resolution can have a powerful effect in reducing computational complexity of Convolutional NN's to the point of outperforming state of the art efficient architectures.
More in detail, the paper proposes a method of joint training of multiple resolutions networks, leveraging student/teacher/distillation from scratch. This is based on training a high resolution teacher network and a low resolution student network, as well as a number of intermediate resolutions networks sampled randomly and jointly during training. Thanks to distillation well known regularization effects, the proposed method is achieving competitive results compared to existing state of the art efficient network architectures. The authors claim, and to some extent show, that this is due to the ability of the proposed method to take into account in a optimal way multi-resolution features available in the image. The paper is well written and presented with extensive results, comparing computational complexity/accuracy curves to existing state of the art architectures, as well as results on transfer learning to show that the feature learned do indeed generalize and don't necessarily overfit to imagenet. The idea is rather simple, but the results and the execution is inspiring.
review_title: Official Blind Review #2

Meta-review: Main content:new training regime for multi-resolution slimmable networks. 

Discussion:
reviewer 4: believes the main contribution of mutual learning from width and resolution is a bit weak
reviewer 1: incremental work, details/baselines missing in experimental section
reviewer 2: (least detailed): well-written with good results
Recommendation: I agree with reviewer 1, 4 that the experimental section could be improved. Leaning to reject. 


Decision: Reject