Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a mixed-size CNN training scheme, using several different input image sizes for one single model training. The authors assume the training budget, represented as S_i^2*B_i*D_i (i.e., spatial sample size, the number of batched distinct samples and the duplicates for each distinct sample), to be a fixed constant during training step i.  Under such an assumption, two mixed-size training scenarios are considered, one for training acceleration and the other for improved model generalization ability. The authors additionally use step-wise image size sampling, gradient smoothing, and per-size BN calibration to enhance the model performance under the above two mixed-size training scenarios. Experimental validation is performed on CIFAR and ImageNet datasets using diverse CNN structures.

Mixed-size training is a critical problem and the methods proposed in this paper are interesting. My main concerns to this paper are as follows.

--- Critical related works and comparison are missing.

Mixed-size training of CNNs for image classification are not new. Here are some recent works, however they are missed by the authors. 

“Resolution Adaptive Networks for Efficient Inference”, in CVPR 2020

 “Resolution Switchable Networks for Runtime Efficient Image Recognition”, in ECCV 2020.

“MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution”, in ECCV 2020. 

Besides, as described by the authors, NeurIPS 2019 paper “Fixing the train-test resolution discrepancy” also considers how to enhance the performance of CNN models when applying them with different input image sizes. But a performance comparison is missing.

To show the advantages of this paper, thorough discussion and performance comparison with the above works are necessary. Taking ResNet-50 trained on ImageNet as an instance, I notice that the proposed method shows obviously worse accuracy compared to some of these works.

---Another critical baseline is also missing. 

In page 8, “We note that for original fixed-size regimes this calibration procedure resulted, with degraded results and so we report accuracy without calibration for these models”. To my understanding, this is somewhat weird. Why BN calibration does not work on the other image sizes when the model is trained with a fixed image size? It is not clear. Furthermore, in NeurIPS 2019 paper “Fixing the train-test resolution discrepancy”, this line of methods work pretty well. Such a BN calibration should server as another baseline for more fair comparison.

---Regarding B+ design

How about the wall-clock training cost (in hours) instead of the number of iterations/epochs?
How about the performance of applying “scale the learning rate linearly” to train the baseline model? 

---How about model transfer ability? 

Only image classification tasks are considered in the experiments. How about the performance of the trained backbone models, when transferring them in the downstream tasks, such as object detection and semantic segmentation? Can performance gain be transferred?

---Others

Is there any principled way regarding the size sampling strategy? The current strategy is based on manual setting, which limits its use in real applications. 

I suggest the authors to also provide precise accuracy numbers, etc. regarding some figures (e.g., figure 1, figure 4) shown in the paper. 

Generally, I am on the fence, to this paper. I encourage the authors to address the questions I raised above.

review_title: An interesting paper, but needs improvements

final_score: rating: 5: Marginally below acceptance threshold
review_content: == Summary ==

The paper proposes to use different image resolutions during the training of a deep neural network image classifier, and varying the batch size or number of data augmented versions of the images, keeping the computational cost per step roughly constant. The authors apply this approach to several architectures and three datasets, and show that they can achieve or improve the same accuracy as the baselines but much faster; or achieve better results with the same computational budget.

== Pros ==

+ The authors conducted their experiments using three different datasets (Cifar10, Cifar100 and ImageNet), and six different architectures (ResNet44, ResNet50, WideResNet, DenseNet and EfficientNet).

+ The proposed approach outperforms the baselines consistently across the 8 pairs of (dataset, architecture) that they have studied. In addition, MixSize can be easily implemented (the authors also provide a PyTorch implementation).

+ The authors investigate different "tricks" to apply during training when using MixSize to stabilize training or achieving better results. For instance, they compared randomly sampling the size from a distribution (as they propose) versus increasing the image resolution as training progresses, and showed that randomly sampling yields slightly better results.

+ Figure 3a seems to suggest that MixSize yields more robust classifiers under a wide range of image sizes. The area under the "Mixed S=144" curve seems to be larger than the area under the "Fixed S=224". However, further experimentation is needed to confirm this, since the area of the "Mixed S=208" seems closer to "Fixed S=224", and in any case the maximum image size was capped around 415.

== Cons ==

- On of the claimed contributions is: "Faster training vs. high accuracy. We show that reducing the average image size at train- ing leads to a trade-off between the time required to train the model and its final accuracy.". However, I would not consider this a novel contribution, since the trade-off between speed and accuracy is well-known. In fact, the authors cite Huang et al. (2018) and Szegedy et al. (2016) which already showed this. EfficientNet is another well-known architecture that takes advantage of this fact.

- In the intro, the authors claim "Touvron et al. (2019) demonstrated that networks trained on specific image resolutions perform poorly on other image sizes at evaluation time, as confirmed in Figure 1". This is inaccurate, since Touvron et al. (2019) actually show that slightly increasing the test resolution improves accuracy, due to the discrepancy in object sizes introduced by data augmentation (cropping). In fact, Figure 1 shows the same effect (model trained with 224 res, achieves best results with 284 eval image size). The statement in the introduction is again contradicted at the end of the first paragraph in Section 2.1.

- The authors do not report any statistical significance metric. Some datasets have very close results, so it's hard to tell whether the improvements are (statistically) significant or not.

- Poor captions in figures and tables. For instance, the difference between solid lines and dashed lines is only explained at the very last figure in the appending (Figure 7). Also, the caption of Figure 3 reads "Test accuracy on validation set" which is ambiguous: Is it a typo, or is it that the authors report the results on the 50k validation set of ImageNet (and use some smaller subset of the training set as validation)? 

== Reasons for score ==

Although the proposed approach is simple and consistently improves the baseline results, I'm not convinced that the originality and significance of the work is enough for it to be accepted. 

Regarding originality, there is a plethora of works exploring the trade-offs between image size and accuracy. The most similar works are Howard (2018) and Karras et al. (2017), which increase the image size through training. It's not clear that random sampling offers a much better result, judging from Figure 8 in Appending E, if one compares the accuracy of "Small->Large" strategy at 125k steps (possibly before the last increase in size).

Regarding significance, if one restricts the analysis to the best architectures in each dataset, the increase in accuracy does not seem to be very large. Cifar10: 98.16% -> 98.32% (AmoebaNet), ImageNet: 76.32% -> 76.53% (EfficientNet-B0). Cifar100 shows a larger improvement, but the authors did not use AmoebaNet (which worked best in Cifar10) for some unknown reason. The fact that no statistical significance metrics are reported, does not help to discern whether the improvements are meaningful or not.
review_title: A nice trick, but not enough novelty

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes to increase training costs to compensate for the reduced costs from multi-scale CNN training by either increasing batch size (and therefore lowering the number of iterations per epoch) or increasing the number of augmented versions (duplicates) of the same samples within a batch. The former allows for smaller total training costs than conventional single-scale training, while the latter maintains the total training costs but improves the final performance. Several training improvement methods are introduced to improve the multi-scale training.

Paper's strengths
- The paper is quite well-written.
- Code and models are provided for reproducibility.
- Gradient smoothing is a nice way to mitigate the variability of gradient statistics resulted from different input sizes. As far as I know, this is quite novel, particularly in the context of multi-scale training.

Paper's weaknesses
- Multi-scale training is a common practice in many computer vision tasks especially in object detection** (less common in image classification). This paper also does multi-scale training but only introduces some minor improvements that are neither breakthroughs nor that they provide any interesting insight.
> - Bag of Freebies for Training Object Detection Neural Networks. arXiv.
> - YOLO9000: Better, Faster, Stronger. CVPR 2017.
> - MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv.

- For "step-wise size sampling", it seems like that conclusion to use this variant of sampling is heuristically chosen and totally ignores the existing practice in other computer vision tasks. One of the straightforward ways to do multi-scale training in object detection is to select different input sizes even for the images within the same batch (by padding zeros for the smaller images). Alternatively, one could sample different input sizes for different GPU batches (all images within a GPU share the same size) while doing multiple-GPU training.

- The three training improvements (step-wise size sampling, gradient smoothing and batch-norm calibration) are what separate this paper from prior work but they are not extensively evaluated. Some of them are briefly evaluated in the appendix and some of their effects are just briefly mentioned in the method section. They ought to appear in the experimental section of the main paper. Gradient smoothing seems like a nice idea but it is unclear how important it is given that there is only one figure (Fig.7) showing its impact on the performance.

- This paper strives to increase the number of batch samples given a fixed budget of computational and time resources for per iteration step. I wonder why this should be limited to the cost of an iteration step but not the entire training cost from all epochs/iterations. It makes more sense to measure the cost for the entire training process which accurately tells how much is spent to train the model until convergence.

- In Sec. 5.1, the size ratios for different datasets are carefully chosen based on cross-validation. This makes it hard to directly apply MixSize to other datasets or settings without going through this step. It also adds additional computations which defeat the purpose of increasing batch size to maintain the same training cost as conventional single-scale training.

- Using separate BatchNorm statistics for multi-scale inputs/features has been explored in the following papers (published at least few months before ICLR deadline). They should be cited and compared against MixSize:
> - Learning to Learn Parameterized Classification Networks for Scalable Input Images. ECCV 2020.
> - Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks. CVPR 2018.

Overall, this paper shows good performance and has some good ideas (e.g., gradient smoothing) for improving multi-scale training. But it fails to give more emphasis to or do a deeper dive into the potentially novel aspects of the work. The current performance improvements may come from doing just trivial multi-scale training which was already widely-explored in prior work.
review_title: Not a bad paper but lacking of significant contributions

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes the use of mixed image sizes during training. They argue empirically that such an approach improves generalization for both fixed image size (e.g. 224 in ImageNet) as well as for variable image size. The proposed training algorithm maintains the same computational budget at each step by either changing the batch size or by using more/less augmentation. They show that adjusting the batch size leads to a faster training but using augmentation leads to a better test accuracy (hence a tradeoff).

However, in order for their proposed method to work, the authors also propose modifications to standard training procedures (i.e. smoothing the gradient, adjusting the batchnorm layers) but without carrying out an ablation study that shows the impact with/without each of these steps. My particular concern is in the use of "gradient smoothing". If I understand it correctly, this is not very different from using momentum, which is known to reduce the variance and improve generalization. However, the authors use gradient smoothing only in their proposed method and do not use it in the baseline method (why not?). It is possible that the reported improvements (e.g. for fixed image size) come solely from this step. 

The other concern is when sampling the image size per step. The authors  propose distributions that seem odd in their experiments (e.g. why is p=0.6 for size 128 in ImageNet which is much larger than others, and why not uniform in CIFAR10). It is important to know if the results are sensitive to the choice of the distribution, to make sure that the benefit is not due to random chance. Also, if this distribution needs to be fine-tuned, then the discussion about improving the training time would be meaningless. 

The last issue is the robustness to different image sizes. Figure 3(a) shows  that if the average image size during training is small, the network will perform better for small images but not for large images. Conversely, if the average image size during training is large, it will perform better for large images, but not for small images. If the concern here is around using mixed-image sizes at inference time, then the red curve in Figure 3(a) shows that a fixed image size is reasonably robust. If one knows in advance that the average image size would be smaller than 224, one can train with a fixed image size that is smaller.

One minor last remark (feel free to ignore) regarding the motivation: the authors study the correlation between the full gradients for the same image with different sizes, on the one hand, and for different images with the same size, on the other hand. They conclude that the first case (different sizes) shows a stronger correlation, which is true according to Table 2, but this statement omits the fact that most correlations were low anyway. For example, for partially trained network, it is 0.08 vs. 0.02. I do not think that one can use such figures to conclude that "smaller image gradients can be used as an estimate to the full image gradients". 

The improvement in test accuracy is very promising but I believe, some ablation is needed to identify exactly where this improvement comes from and whether it can be obtained using simpler approach (e.g. smoothing the gradient alone or using augmentation alone, etc). 
review_title: A mixture of multiple techniques without ablation

Meta-review: This work proposes to train networks with mixed image sizes to allow for faster inference and also for robustness. The reviewers found the paper was well-written and appreciated that the code was available for reproducibility. However, the paper does not sufficiently compare to related methods. The authors should resubmit once the comparisons suggested by the reviewers have been added to the paper.
Decision: Reject