MixSize: Training Convnets With Mixed Image Sizes for Improved Accuracy, Speed and Scale Resiliency
Authors: Elad Hoffer
Berry Weinstein
Itay Hubara
Tal Ben-Nun
Torsten Hoefler
Daniel Soudry
Venue:  (2021)

Abstract: Convolutional neural networks (CNNs) are commonly trained using a fixed spatial image size predetermined for a given model. Although trained on images of a specific size, it is well established that CNNs can be used to evaluate a wide range of image sizes at test time, by adjusting the size of intermediate feature maps. \\
In this work, we describe and evaluate a novel mixed-size training regime that mixes several image sizes at training time. We demonstrate that models trained using our method are more resilient to image size changes and generalize well even on small images. This allows faster inference by using smaller images at test time. For instance, we receive a $76.43\%$ top-1 accuracy using ResNet50 with an image size of $160$, which matches the accuracy of the baseline model with $2 \times$ fewer computations.
Furthermore, for a given image size used at test time, we show this method can be exploited either to accelerate training or the final test accuracy. For example, we are able to reach a $79.27\%$ accuracy with a model evaluated at a $288$ spatial size for a relative improvement of $14\%$ over the baseline.
Our PyTorch implementation and pre-trained models are publicly available\footnote{\url{https://github.com/paper-submissions/mix-match}

Under review as a conference paper at ICLR 2021
MIXSIZE: TRAINING CONVNETS WITH MIXED IMAGE SIZES FOR IMPROVED ACCURACY, SPEED AND SCALE RESILIENCY
Anonymous authors Paper under double-blind review
ABSTRACT
Convolutional neural networks (CNNs) are commonly trained using a fixed spatial image size predetermined for a given model. Although trained on images of a specific size, it is well established that CNNs can be used to evaluate wide range of image sizes at test time, by adjusting the size of intermediate feature maps. In this work, we describe and evaluate novel mixed-size training regime that uses several image sizes at training time. We demonstrate that models trained using our method are more resilient to image size changes and generalize well even small images. This allows faster inference by using smaller images at test time. For instance, we receive a 76.43% top-1 accuracy (ResNet50 on ImageNet) with an image size of 160, which matches the accuracy of the baseline model with 2x fewer computations. Furthermore, for a target image size used at test time, we show this method can be exploited either accelerate training improve the final test accuracy. For example, we are able to reach a 79.27% accuracy with the same model evaluated at a 288 spatial size for a relative improvement of 14% over the baseline. MixSize regimes pave the way for faster and more accurate training and inference using convolutional networks. Our PyTorch implementation and pre-trained models are publicly available
INTRODUCTION
Convolutional neural networks are successfully used
solve various tasks across multiple domains such
75-
as visual (Krizhevsky et al. 2012: Ren et al. 2015).
70
audio (van den Oord al. 2016), language Gehring
66
let al. (017) and speech (Abdel- et al. 2014).
60
$=64
While ecale-invariance considered important for visual representations (Lowe, 1999), convolutional networks are not scale invariant with respect to the spatial resolution of the image input, change in image dimension may lead to non-linear change of their output. Even though CNNs able to achieve state-of-the-ae results in many tasks and domains, their sensitivity to the image size is inherent de- ficiency that limits practical use cases, and requires that images at evaluation time match training image size. For example, [Touvron et al. (2019) demonstrated that networks trained on specific image size, perform poorly on other image sizes at evaluation time, as confirmed in Figure [
56
Training Bize 8-128
Training Size S-160
Training Size S=224
Training Size
100
250
300
350
400
image (evablation)
Figure 1: Top-1 test accuracy per image size, models trained on specific sizes (ResNet50, ImageNet).
The most common method to improve scale invariance in CNNs artificially enlarge the dataset using a set of label label-preserving transformations also known as "data augmentation" (Howard) 2013) Krizhevsky et al. 2012).. Several of these transformations scale and crop objects appearing within the data, thus increasing the network's 's robustness to inputs of different scale. Several works at- tempted to achieve scale invariance by modifying the network structure to learn over multiple pos- sible target input scales (Takahashi et al. 2017 Xu et al. 2014 Zhang et al. 2019). These methods
httpp::/ thubihhubbooo
httpgggtttt........ageeeeeee www..////ppppe.eeee summ ww...o...... x8ize

---

Under review as a conference paper at ICLR 2021
explicitly change the model for specific imput size, thus not benefiting from possible lower computa-- tional requirements of using smaller image sizes, nor with ability of inferring on sizes not observed during training. Another approach suggested by CCa et al. (2020) modifies network structure and training regime to account for variety of inference modes without additional specialized training.
In this work, introduce novel training regime, "MixSize" for convolutional networks that uses stochastic image and batch sizes. The main contributions of the MixSize regime are:
Reducing image size sensitivity. We show that the MixSize training regime can improve model performance on a wide range of sizes used at evaluation.
Faster inference. As our mixed-size models can be evaluated smaller image sizes, show up to 2x reduction in computations required at inference to reach the same accuracy as the baseline model.
Faster training vs. high accuracy. We show that reducing the average image size at train- ing leads to trade-off between the time required train the model and its final accuracy.
BACKGROUND AND RELATED WORK
2.1 USING MULTIPLE IMAGE SIZES
Deep convolutional networks are traditionally trained using fixed-size inputs, with spatial dimen- sions H W and batch size B. The network architecture is configured such that the spatial dimensions are reduced through strided pooling or convolutions, with the last classification layer applied on a 1 spatial dimension. Modern convolutional networks usually conclude with a fi- nal "global" average pooling (Lin et al. 2013 Szegedy et al. 2015), which reduces any remaining spatial dimensions with a simple averaging operation. Moditying the spatial size of an input convolutional layer by a factor 7, will yield an output with size scaled by the same factor 7. This modification does not require any change to the number of parameters of the given convolutional layer, nor its underlying operation. It was observed by practitioners and previous works that net- work trained on a specific input dimension can still be used at inference using a modified image size to some extent (Simonyan & Zisserman| 2014). Moreover, evaluating with an image size that is larger than used for training can improve accuracy up to a threshold, after which it quickly deterio- rates (Touvron et al. 2019).
Although not explicitly trained to handle varying image sizes, CNNs are commonly evaluated on multiple scales post-training, such as in the case of detection (Lin et al. 2017: Redmon & Farbadi 2018 Liu et al. (020)) and segmentation (He et al.] 2017) tasks. In these tasks, a network that was pretrained with fixed image size for classification used as the backbone of larger model that is expected to adapt to a wide variety of image sizes.
Recently, Tan & Le (219)) showed a computation- computatiion-ssacccraaayy trade-off in applying scaling to the image size used for convolutional networks training and evaluation. This finding is consistent with past findings, which demonstrated that training with a larger image size can result in better clas- sification accuracy (Huang et al., 2018 Szegedy et al. 2016).. In addition, previous works explored the notion of "progressive resizing" (Howard) [2188] Karras al.) 2017) increasing image size as training progresses to improve model performance and time to convergence. A similar idea by Wu al. (2020) was used to improve performance of training on video data, by balancing resolution with batch size. Another related work by Touvron et al. (2019) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to larger size, with which eval- uation will be performed. This procedure reduced the train-test discrepancy caused by the change in image size and allowed faster training time and improved accuracy at the cost of additional fine-tuning procedure and additional computations inference time.
In this work we will further explore the notion of using multiple image sizes at training, so the CNN performance will be resilient test time changes of the image size.
2.2 LARGE BATCH TRAINING OF DEEP NETWORKS
Deep neural network training can be distributed across many computational units and devices. The most common distribution method is by parallpprallllism" computing average estimate of the
2

---

Under review as a conference paper at ICLR 2021
gradients using multiple, separately computed data samples. As training NN models is done using batch-SGD method and its variants, scaling this process across multiple computational devices while maintaining similar utilization for each device inflates the global batch size.
Large batch training is known to affect the gencralization capabilities of NNs and as such, it requires to modify its optimization regime. While several works claimed that large-batch training leads to an inherent "generalization gap" (Keskar et al.) (016)) more recent works demonstrated that this gap is largely caused by insufficient number of optimization steps performed and can be partly mitigated by hyper-parameter tuning (Hoffer et al. 2017; Shallue et al. (018)). In order to cope with the variations in the network training dynamics, several modifications of the optimization procedure have been proposed. For instance, a linear (Goyal et al. (017) or square-root (Hoffer et al. (017) scaling of the learning rate with respect to the batch size growth. Other modifications include per- layer gradient scaling schemes (You et al. (017) and optimizer modifications (Ginsburg et al., 2019). Several works also explored using incremented batch-sizes (Smith et al., 2018)) in order to decrease the number of training iterations required to reach the desired accuracy.
Recent work (Hoffer et al. 2020) introduced the notion of "Batch Augmentation" (BA) increas- ing the batch size by augmenting several instances of each sample within the same batch. BA aids generalization across a wide variety of models and tasks, with the expense of an increased compu- tational effort per step. A similar method called "Repeated Augmentation" (RA) was proposed by Berman et al. 2019)). It was also demonstrated that BA may allow to decrease the number of training steps needed to achieve a similar accuracy and also mitigate I/O throughput bottlenecks (Choi al., 2019). As previous works investigated mostly homogeneous training settings (e.g., using a fixed batch size), an open question still exists on the utility of rapidly varying batch-sizes. We will ex- plore this notion by modifying the optimization method, enabling training with varying batch-sizes and limited apper-parameter tuning.
3 MIXSIZE: TRAINING WITH MULTIPLE IMAGE SCALES
The traditional practice of training convolutional networks using fixed-size images holds several shortcomings. As is a common practice to use different image size for evaluation than that used for training (He et al. [0117: Lin et al. 2017 Redmon & Farhadi 20188), it was observed by Touvron al. (2019) and empirically verified (Figure I that classification accuracy may degrade above or below certain size threshold. Similarly, we observed this phenomenon in a wide variety of common networks (Appendix Figure 4 This hints the issue here is related more to the training procedure, rather than to a specific model. To remedy these issues, we suggest a stochastic training regime, where image sizes may vary in each optimization step.
Motivation. We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time. As a first step to substantiate this claim, we evaluated the impact of various image sizes on the CNN training progress by exam- ining gradient statistics during training. Specifically, we measured the correlation between weight gradients with respect to varying image sizes (Appendix Table 2). We see that gradients computed across different sizes of the same images have stronger correlation than those obtained across dif- ferent images. Moreover, the correlation is most significant during the first stages of training and decreases as the model converges. This observation suggests that smaller image gradients can be used as an estimate to the full image gradients, with a smaller computational footprint. Therefore, using large images along the entire training process may be sub-optimal in terms of computational resource utilization. We continue to test our hypothesis using a novel mixed-size training regime.
MixSize training regime. We propose "MixSize", a stochastic training regime where input sizes can vary. In this regime, we modified the spatial dimensions H, W (height and width) of the input image size?] as well as the batch size. The batch size is modified either by the number of used samples, denoted B, or by the number of auuch-augmenttiions for each sample (Hoffer al. 2020), denoted D ("duplicates"). To simplify our notation and use-cases, we will follow the common practice of training on square images and use = = W. Formally, in the MixSize regime, these
Thhe spatial dimensions of all intermediate maps in the CNN are changed accordingly, at the same scale as the input.
3

---

Under review as a conference paper at ICLR 2021
sizes can be described as random variables sharing single discrete distribution
(S, B. = ( wpp p.).
(1)
where Vi: p. 2 and P
As the computational cost of each training step approximately proportional S B D, we choose these quantities to reflect nearly fixed budget for any choice such that S B, D; 2 Const. Thus, in this regime, the computational and memory requirements for each step in remain fixed.
Benefits and Trade-off'. We will show that using our MixSize regime can have positive impact on the resiliency trained networks to various images sizes at evaluation time by demonstrating better accuracy across wide range of sizes. This entails a considerable saving in computational burden required for inference, especially when using smaller models. Furthermore, given a fixed budget of computational and time resources (per step), can modify regime along spatial and batch axes. We will explore two trade-offs:
Decrease number of iterations per epoch by enlarging B at the expense of S.
Improve generalization per epoch by enlarging D at the expense S.
We denote our modified mixed-size regimes as B+ for an increased effective batch-size and D+ for an increased number of BA duplicates of the same ratio.
4 IMPROVED TRAINING PRACTICES FOR MIXSIZE
MixSize regimes continuously change the statistics of the model's inputs, by modifying the image size as well as batch-size. This behavior may require hyper-parameter tuning and may also affect size-dependent layers such as batch normalization (loffe Szegedy 2015). To easily adapt training regimes to the use of MixSize as well as improve their final performance, we continue to describe three methods we found useful: step-wise size sampling, gradient smoothing and batch-norm calibration.
4.1 SAMPLING IMAGE SIZE PER-STEP
Considering our findings on correlation of gradients across different image size in early training, we were initially motivated consider training variation where size progressively enlarged (Howard) 2018 Karras et al. (017).
In this method we consider increasing image size from small to large, where each image size used for predefined number of epochs out of total number of training epochs required.
We observed progressive resizing to cause noticeable accuracy deterioration when switching one size for another (Appendix E). which may be attributed to the network tendency adapt to specific size. Unfortunately, this hampered use cases where the final image size was larger than the one used evaluation.
We therefore considered two alternatives for image size variations, where size randomly sampled and changed throughout training:
Sampling image size per epoch
Sampling image size per training step
These sampling methods can be seen as a stochastic alternative to long and short cycle multigrid approach suggested by WW et al. 22000). We found that random sampling regimes perform better than scaling image size from small to large, keeping accuracy high across all used sizes at evaluation (Appendix E.. While sampling both at epoch and step time frames performed similarly, replacing sizes each step converged faster and exhibited less noise in measured test accuracy. Considering these findings, we continue to perform our experiments using the third regime sampling image size per training step.

---

Under review as a conference paper at ICLR 2021
4.2 GRADIENT SMOOTHING
Training with varying batch and spatial sizes inadvertently leads to a change in the variance of the accumulated gradients. For example, in T the gradient variance larger when computed over a small image size (unsurprisingly). This further suggests that the optimization regime should be adapted to smaller spatial sizes, manner similar to learning-rate adaptations that are used for large-hatch training. This property was explored in previous works concerning large-batch regimes, in which learning rate modification was suggested to compensate for the variance reduction for larger batch-sizes. Unfortunately, the nature of this modification can vary from task to task across models (Shallue et al. (2018), with solutions such as a square-root scaling (Hoffer et al., 2017). linear fixed norm ratio (You et al. 2017). Here suggest changing both the spatial size as well as the batch size, which is also expected to modify the variance gradients within each step and further complicates the choice of optimal scaling.
scaling (Goyal et al. (017)
Previous works suggested methods to control the gradient norm by gradient normalization (Hazan et al. 2013) and gradient clipping (Pascanu et al., 2013). These methods explicitly disable or limit the gradient's s norm used for each optimization step, but also limit naturally occurring variations in gradient statistics. We suggest an alternative solution previous approaches, which we refer to as " "Gradient smoothing". Gradient smoothing mitigates the variability of gradient statistics when image sizes are constantly changing across training.
Denoting L for the objective loss function, 9r and wi as weights and gradients of optimization step we introduce an exponentially moving weighted average of the gradients' norm &e (scalar) which is updated according to
9t = =og--1 + (1 -a)g
where
aL
9t
and Go 90
Ow,
We normalize the gradients used for each step by the smoothing coefficient, such that each consec- utive step is performed with gradients of similar norm. For example, for the vanilla SGD step, we weight update rule of the form
OL 9t. Owe
W+++1 1)
This running estimate of gradient norm is similar to the optimizer suggested by Ginsburg et al. (201)), which keeps a per-layer estimate of gradient moments. Gradient smoothing, however, is designed to adapt globally (across all layers) to the batch and spatial size modification and can be used regardless of the optimization method used.
We found gradient smoothing to be beneficial in B+ regimes, where multiple varying batch sizes are used, as the effective learning rate is also modified according common practices. We observed that gradient smoothing reduces the gap between gradient norms of different sizes (see Appendix Figure ()) and allow the use of an average learning rate scaling (weighted across batch sizes). Measuring test error on the same model shows significant advantage for gradient-smotthing (Appendix Figure]
4.3 BATCH-NORM CALIBRATION FOR VARYING IMAGE SIZES
As demonstrated by Touvron et al. (2019), using different image size at evaluation may incur a dis- crepancy between training and evaluation protocols, caused by using different data pre-processing. (Touvron et al.) 2019) suggested a post-training procedure, where network trained on a specific fixed-size is fine-tuned on another size, later used for evaluation. Their solution required dozens of training epochs, amounting to thousands of full forward and pack-propagation computations, along with parameter updates for batch-norm and classifier layers. In contrast, we surmise that for net- works trained with mixed-regimes, discrepancy issues mainly arise from the use of the batch-norm layers (loffe & Szegedy 2015) and can be solved by targeting them specifically.
Batch-nome layers introduce discrepancy between training and test evaluations Ioffe 2017), as at inference a running estimate of the mean and variance (of training data) are used instead of the actual mean and variance values. This difference is emphasized further in the of varying image
5

---

Under review as a conference paper at ICLR 2021
size, changing the spatial size of input map significantly modify the measured variance of that map. While a fine-tuning process per image size can eliminate this discrepancy (Touvron et al. 2019), it required significant amount of computation and labeled data. We offer simpler alternative. For each evaluated size, we calibrate the mean and variance estimates used for that size average value over small number of training examples. This calibration requires only a few (hundreds) of feed-forward operations with no pack-propagation or parameter update, and takes only few seconds on single GPU. As calibration requires very little resources, well as small amount of unlabelled data, it can be potentially performed on-demand at test-time.
by computing
Interestingly, we highlight the fact that although this process has little no effect on models trained using fixed-size input, it does improve our mixed-size models considerably on wide range of image sizes.
EXPERIMENTS
5.1 MIXSIZE WITH FIXED IMAGE SIZE AT TEST-TIME: THE SPEED-ACCURACY TRADE-OFF
We used our sampling strategy to train and compare our regime to the baseline results of various models over the CIFARI0/100 (Krizhevsky) 2009) and ImageNet (Deng et al. 2009) visual classi- fication tasks. In each experiment the B regime, use our gradient smoothing method, as described in Section 4.2
the original hyper-pane sperr-parameter without modification. For
As CIFAR datasets are limited in size, consider the following balanced stochastic regime chosen:
40, w.p. p == 0.2
32, w.p. p = 0.3
S 5=
24, w.p. == 0.3
16, w.p. p 0.2
The regime was designed to be centered around the mean value of 28, As the original image size now able to increase either the batch size or number of BA 3r such that 52 B- approximately constant.
used for training 32 x 32, duplicates for each training step by a factor of 322
For the ImageNet dataset, use the following stochastic regime found by cross-validation on several alternatives over the ResNet-50 model (He et al. 2016):
256,
w.p p = 0.1
S(L11); : S
224,
w.p p = 0.1
128,
w.p p 0.6
96, w.p p 0.2
While the original training regime consisted of images of size 224 224, proposed regime makes for an average image size of S 5 = 144 x 144. This regime was again designed so that the reduced spatial size be used increase the corresponding batch size the number of BA duplicates, as described in Section 3
For each result, we measure our final accuracy on the original image size (32 32 for CIFAR, 224 x 224 for ImageNet). We also perform batch-norm calibration as described in Section 4.3
We present the final test accuracy, as well as the number of training steps for each model in Table We see that our MixSize regimes yield two possible improvements:
Reduced number of training steps to achieve a similar test accuracy using B+ regime.
Better test accuracy using D regime (enabling BA with same computational budget).
We note that for both B+ and D+ regimes, the final accuracy matches surpasses the baseline results. B+ enables larger batches on average, hence the reduction in training steps, while D+ leverages benefits of aateh-augmentttion achieve better accuracy within the same compute. The ResNet-50 model benefits the most from our MixSize regime on the ImageNet model and so we chose for further exploration.

---

Under review as a conference paper at ICLR 2021
Table Test accuracy (Top-1) results for CIFAR and ImageNet. Each row represents models trained using the same computational and memory budget per step. Steps and accuracy are reported at the completion of fixed epoch budget (i.e., 90 epochs for ResNet on ImageNet, 200 for ResNet on CIFAR), Accuracy is reported for model's original size (32 for CIFAR, 224 for ImageNet).
Network
Dataset
Steps
Accuracy
Baseline
B+
D
Baseline
B+
1
ResNet-44
CIFARIO
156K
109K
156K
92.84%
94.30%
94.46 %
WRN-28-10
CIFARIO
156K
109K
156K
96.60%
97.28%
97.68 %
AmoebaNet
CIFARIO
469K
328K
469K
98.16%
98.16%
98.32% %
ResNct-44
CIFARION
156K
109K
156K
70.36%
72.19%
73.10% %
WRN-28-10
CIFARI00
156K
109K
156K
79.85%
83.08%
83.52%
ResNct-50
ImageNet
450K
169K
450K
76.40%
76.87%
78.04 %
DenseNet169
ImageNet
450K
169K
450K
76.20%
76.66%
77.26 %
EfficientNet-Bi
ImageNet
1000K
376K
1000K
76.32%
76.37%
76.53%
5.2
CASE STUDY: RESNET-50 IN MIXED SIZE REGIMES
We showcase our finding on the common ResNet-50 model (He et al. 2016) using the training regime suggested by Goyal et al. (2017) (see Appendix C.2, We employed the standard data aug-
mentation and did not incorporate any additional regularization or augmentation techniques.
We are first interested in accelerating the time needed for convergence of the tested models using our 3+ scheme. We enlarge the batch size used for each spatial size by a factor of 224 S B is kept approximately fixed. As the average batch size is larger than Bo, which was used with the original optimization hyper-parameters,
5 such that
scale the learning rate linearly suggested by _ We note that for the proposed regimes did not require any
Goyal al. 2017 by factor of T. learning rate warm-up, due the use of gradient smoothing.
As can be seen in Figure 2 regime B+ enables train- ing with approximately 2.7x fewer training steps, while reaching an improved accuracy of 76.61%. %. As sizes were chosen to reflect in approximately equal computational cost per iteration, B regime offers a similar improvement in total wall-clock time.
Beoelinn Regime Ragina
50
40
30
Next, we perform a similar experiment with a D+ regime, where the number of BA duplicates is sim- ilarly increased with respect to D, instead of the batch size. This scaling results with an average du- plicates of D 3.
100
200
300
400
shpps (163)
Figure 2: Training (dotted) and test accuracy on ImageNet using suggested regimes (224 evaluation size). All regimes required similar resources per step. B+ regime required se 2.7x less steps per epoch.
As the computational cost for each step remains ap- proximately constant, as well as the number of re- quired steps per epochs, training a model under this regime requires an equal wall-clock time. However, the increased batch-augmentatiin improves the final test accuracy to 78.04%, approximately 7% relative improvement the 76.4% baseline.
5.3 INCREASING MODEL RESILIENCY TEST-TIME CHANGES IN IMAGE SIZE
Next, we examine how MixSize affects the resulting model resiliency to changes in the image size during test-time. We evaluated the models by varying the test-time image sizes around the original 224 spatial size: S 224 + 32 m, for m (-6,.., 6). The common evaluation procedure for ImageNet models first scales the image to a 256 smallest dimension and crops center 224 224 image. We adapt this regime for other image sizes by scaling the smallest dimension to [ (since 224 = 256) and then cropping the center S S patch. Models trained with a mixed regime were calibrated specific evaluation size by measuring batch-norm statistics for 200 batches of
7

---

Under review as a conference paper at ICLR 2021
training samples. We note that for original fixed-size regimes this calibration procedure resulted with degraded results and so we report accuracy without calibration for these models. We did not use any fine-tuning procedure post training for any of the models.
As can be seen in Figure 3a the baseline model trained using a fixed size, reaches 76.4% top-1 accuracy at the same 224 spatial size it was trained on. As observed previously, the model continues to slightly improve beyond that size, to a maximum of 76.8% accuracy. However, it is apparent that the model's performance quickly degrades when evaluating with sizes smaller than 224.
We compare these results with D regimes, trained with an average size of $ = 144 and S = 208 (see Appendix FF. As described earlier, this model requires the same time and computational resources as the baseline model. However, due to the decreased average size, we were able to leverage more than I duplicates per batch on average, which improved the model's top-1 accuracy to 77.14% at size 224. Furthermore, we find that the model performs much more favorably at sizes smaller than 224, scoring an improved (over baseline) accuracy of 76.43% at only 160 spatial size.
The model trained with the ((208) regime offers a similar improvement in accuracy, but across larger spatial size, as it observed an average size of 208 during training. Figure demoonsssrtsss that while all three models (Fixed with S = 224, 51144) and 5(208)) were trained with the same compute and memory budget, mixed-size regimes offer superior accuracy sizes. Specifically, mixed-regime at S = 208 dominates the baseline fixed-size regime at all sizes, while our mixed regime at S 144 achieves best results sizes smaller than 224.
a wide range of evaluation
80
80
75
75
70
70
Fixed S=128
0000000
65
65
Fixed B=144
Fixed 8-224
Fixed 8-224
Fixed S+384
Mixed S-144
Mixed S-144
00
Mbxdd S=208
56
Mixed S=208
100
150
200
250
300
350
400
2
8
10
12
14
FLORS (BNonn))
(a)
(b)
Figure 3: (a): Test accuracy on validation per image size, all models trained using the same computational and memory resources (regime D+). (b): Test accuracy per Gflop (at evaluation).
We also compared the classification performance across evaluated image sizes, using networks trained on a variety of fixed sizes and our mixed regimes. As baseline, we use results obtained by Touvron et al. (2019) (trained with RA, without fine-tuning) and compare them with mixed-regima models trained with an equal computational budget, by setting the base number of BA duplicates to D D==2 2. As can be seen in Figure 3b, mixed-regime trained models offer a wider range of resolu- tions with cose-to-bbseline accuracy (within a 2% change) and perform better than their fixed-size counterparts at all sizes. As the number of floating-poin operations (flops) grows linearly with the number of pixels, using mixed regime significantly improves accuracy per compute evaluation. We further note that our 5(224) model reaches a top accuracy of 79.27% at a 288 evaluation size.
6 SUMMARY
In this work, we introduced and examined a performance trade-off between computational load and classification accuracy governed by the input's spatial size. We suggested stochastic image size regimes, which randomly change the spatial dimension as well as the batch size and the number of augmentation duplicates in the batch. Stochastic regime benefits are threefold: (1) reduced number of training iterations; or (2) improved model accuracy (generalization) and (3) improved model robustness to changing the image size. We believe this approach may have a profound impact on the practice of training convolutional networks. Given computational and time budget, stochastic size regimes may enable to train networks faster, with better results, as well as to target specific image sizes that will be used at test time. As the average size chosen train is reflected in the optimal operating point for evaluation resolution, mixed regimes can be used to create networks with better performance across multiple designated use cases.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang. Li Deng, Gerald Penn, and Dong Yu. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on audio, speech, and language processing, 22(10):1533-155 1545, 2014.
Maxim Berman, Hervé Jégou, Andrea Vedaldi, Iasonas Kokkinos, and Matthijs Douze. Multigrain: unified image embedding for classes and instances. arXiv preprint arXiv: 1xXiv:992.00500 2019.
Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. Once-for-all: Train one network and specialize it for efficient deployment. In International Conference on Learning Representations, 2020. URL httpt:::/tpperreeeee
net forummiccc. = Hy1xE1HKwS
Dami Choi, Alexandre Passos, Christopher Shallue, and George E. Dahl. Faster neural network training with data echoing. arXiv preprint arXivv 2019.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 005550180,.00501,, 2018.
Deng, Dong, R. Socher, I.-J. Li, Li, and I.. Fei-Fei. ImageNet: Large-Scale Hierarchical
Image Database. In CVPR09, 2009.
Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: (70.::078..5522 2017.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann Dauphin. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pp. 1243-1252, JMLR. org, 2017.
Boris Ginsburg, Patrice Castonguay, Oleksii Hrinchuk, Oleksii Kuchaiev, Vitaly Lavrukhin, Ryan Leary, Jason Li, Huyen Nguyen, and Jonathan M Cohen. Stochastic gradient methods with layer- wise adaptive moments for training of deep networks. arXiv preprint arXiv:1905.11 1286, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv:1706.02677 2017.
Elad Hazan, Kfir Levy, and Shai Shalev-Shwartz. Beyond convexity: Stochastic quasi-convex opti- mization. In Advances in Neural Information Processing Systems, pp. 1594-1602, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Train longer, generalize better: closing the generaliza- tion gap in large batch training of neural networks. In Advances Neural Information Processing Systems, pp. 1731-1741 2017.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Augment your batch: Improving generalization through instance repetition. In CVPR20, 2020.
Andrew G Howard. Some improvements on deep convolutional neural network based image classi- 2013.
fication. arXiv preprint
Jeremy Howard. Fastai progressive resizing.
fast.
dawnbench-fastsil 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.

---

Under review as a conference paper at ICLR 2021
Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoe V Le, and Zhifeng Chen. Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv preprint arXiv:1811.06965. 2018.
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch normalized models. In Advances in neural information processing systems, pp. 1945-1953, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp- 458-456, 2015.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint /710v 10196, 2017.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. 2016.
Alex Krizhevsky. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, pp. 1097-1105, 2012.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint wrXiv:1312.44000 2013.
Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. Proceedings of the IEEE international conference on computer vision, pp. 2980-2983 2017.
Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey, International journal of computer vision, 128(2):261-318, 2020.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:160..0390 2016.
David G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the Seventh IEEE International Conference Computer Vision, volume 2, pp. 1150-1157 vol.2, 1999.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens van der Maaten. Exploring the limits weakly supervised pretraining. In Proceedings of the European Conference Computer Vision (ECCV), pp. 181- 196, 2018.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013.
Esteban Real, Alok Aggarwal, Yanping Huang, and Quoe V Le. Regularized evolution for image classifier architecture search. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4780 2019.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv, 2018.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99999990000 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In 2018 JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4510-4520. IEEE, 2018
Christopher J Shallue, Jaehoon Lee, Joe Antognini, Jascha Sohl-Dickstein, Roy Frostig, and George E Dahl. Measuring the effects of data parallelism on neural network training. arXiv preprint arXiv: /...:////000000 .03600, 2018
10

---

Under review as a conference paper at ICLR 2021
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rxiv:1409.1533 2014.
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V Le. Don't decay the learning rate, increase the batch size. 2018.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 18882822, 2016.
Ryo Takahashi, Takashi Matsubara, and Kumiaki Uehara. Scale-invarian recognition by weight- shared cnns in parallel. In Asian Conference on Machine Learming, pp. 295-310, 2017.
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114, 2019.
Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Hervé Jégou. Fixing the train-test resolution discrepancy. CoRR, 0tss/1906.06423. 2019. URL
www.ooggg////o/oo/../....t....at aaaaab///si 190.69966...0 06423
Aäron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alexander Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: generative model for raw audio. 2016.
Chao-Yuan Wu, Ross Girshick, Kaiming He, Christoph Feichtenhofer, and Philipp Krahenbuhl. A multigrid method for efficiently training video models. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 153-162, 2020.
Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, and Zheng Zhang. Scale-invariant con- volutional neural networks. arXiv preprint 1xXX:::411.66000 2014.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv:1708. 700.000888888 2017.
Komodakis Zagoruyko. Wide residual networks. In BMVC, 2016.
Jinpeng Zhang, Jinming Zhang, Guyue Hu, Yang Chen, and Shan Yu. Scalenet: A convolutional network to extract multi-scale and fine-grained visual features. IEEE Access, 2019.
Appendix
A SCALE RESILIENCY IN PRETRAINED CONVOLUTIONAL NETWORKS
We used 6 common networks (Simonyan Zisserman 2014 Szegedy al. 2015 He al. 2016
Huang et al., 2017 Mahajan et al., 2018 Sandler et al. 2018)) pretrained on the ImageNet dataset resolution of 224 224 and measured their accuracy on varying sizes. As can be seen, these networks show similar patters of sensitivity to the evaluation size, suggesting this is caused by the training protocol (which similar for all), rather than the used model.
B GRADIENT CORRELATION OVER IMAGE SIZES
We evaluated the impact of various image sizes on the CNN training progress by examining gradient
statistics during training (Table 2}. We used a ResNet-44 model (He et al. 2016),, trained on the CIFAR10 dataset (Krizhevsky) 2009), whose standard image size is 32 32. Measurements performed on the whole network's gradient vector. Images were sampled in uniform. The smaller 24 24 images down-sampled with bilinear interpolation.
11

---

Under review as a conference paper at ICLR 2021
80
70
accuracy
densenet169
60
googlenet mobilenet v2
50
resnet50
resnext101_32w1 32x8d
40
vgg16 bn
100
150
200
250
300
350
image size
Figure Top-1 accuracy over ImageNet validation set per evaluation image size.
Table 2: ResNet-44 gradient correlation on CIFARIO. We measure the Spearman correlation coeffi-
cient between gradients computed over different spatial size of random images P (96"1) (51) (#2) as
well as non-identical random images of the same size
giz (a1)
(8:)).
We also compute the variance
(gx) for the gradients of each spatial size.
Mensuee
Network State
Initial
Partially Trained
Fully Trained
Epech
100
Test Accuracy
55, 12%
87.56%
92.62%
(32)
(24)
0.2
0.08
0.03
(32)
0.086
0.02
00.044
V 9
(32)
1.03e 6
1.44e
6
6.24e 7
1.95e
6.34e
2.26e
C EXPERIMENTAL SETTINGS
C.I CIFAR
We examine our method using the common visual datasets CIFAR10/100 (Krizhevsky 2009) that
consist of 32 x 32 color images. We use the ResNet--44 model suggested by (He al. (0166). Wide Resnet WRN-28- 10 (Zagoruyko) 2016) and AmoebaNet (Real et al. 2019) with their original regime and batch size of 64. While for 44 we use the original augmentation protocol as described by He et al. (016). In this method, the input image is padded with 4 zero-valus vero-valued pixels at each side, top and bottom. A random 32 x 32 part of the padded image then cropped and with 0.5 probability flipped horizontally. In order to adapt to varying input scales, we add additional augmentation step, that resizes for each step. We note that this keeps the exact original augmentation procedure for S F2r WRN-28-1E 10 and AmoebaNet we apply cutout (DeVrics & Taylor 2017) and auto-augment policies Cubuu et al. 2018)
images using bilinear interpolation to x S, depending on sampled size
Training progress on the CIFARIO using ResNet44 is depicted in Figure Interestingly, although designed only to reduce training time, some cases. This improvement can be attributed to a regularization effect induced by changing image sizes during training, also manifested by increase in training error throughout its progress. C.2 IMAGENET
see that our B regime also improves accuracy in
We used the Re model (He et al. 2016) with the training regime suggested by (Goya al. 2017)) that consists base learning rate of 0.1, decreased by factor of 10 on epochs 30, 60, 80, stopping at epoch 90. We used the base batch size of 256 over 4 devices and La regularization
12

---

Under review as a conference paper at ICLR 2021
80
Baseline
Baselin
50
Rugi no D Rogi mo B+
7.5
Regime
Rogine B+
20
10
5.5
20
80
80
100 140
80
90
100
110
120
130
140
150
(a) Test error
(b) Final test error
Figure 5: Training (dotted) and test accuracy vs optimization step (ResNet44, CIFARIO). We com- pare vanilla training with two computationally equivalent stochastic regimes: increased duplicates (D+) and increased batch (B+). B+ regime achieves better test accuracy at a reduced number of iterations, while D improves accuracy further at a similar computational cost.
over weights of convolutional layers. We employed the standard data augmentation and did not incorporate any additional regularization or augmentation techniques.
Additionally, we also used the EfficientNet-BOO model suggested by (Tan & Le 2019). We used the same data augmentation and regularization the original paper, but opted for shorter train- ing regime with a momentum--S [Loshhhlovv & Hutter 2016) over 200 epochs starting from initial base 0.1 value.
optimizer that consisted of fosine-annealed learning rate
D IMPACT OF GRADIENT SMOOTHING
We show the impact of gradient smoothing on B+ MixSize regimes on CIFAR106and ImageNet 7 We that gradient smoothing helps to reduce gap between gradient norms difference batch sizes and improves training and validation accuracy.
Gradient Smoolhing
No Gradient Smoothing
0.9
Grad Norm
0.8
0.7
0.6
0.5
0
50
100
150
200
epoch
Figure 6: Impact of gradient smoothing on CIFARIO, ResNet-44. The training regime includes
two image sizes: 32 x 32 and 16 x 16 (average size is S = 24). Using B regime creates two batch sizes: 256 and 2, 048 respectively. Gradient norm values are shown with and without Grad-smoothing. Solid lines are gradients for $ = 32 while dotted lines are for S = 16. Gradient smoothing helps to reduce gap between gradient norms difference batch sizes.
COMPARING MIXSIZE REGIMES
We consider training regimes with varying image sizes, such that the average image size is smaller than the desired evaluation size. For example, for the height dimension H, wish to obtain an average size of = Σ, H, such that H < H,, We consider three alternatives for image size variations:
13

---

Under review as a conference paper at ICLR 2021
50
Gradient- -moothing
No Gradient-Smoolhing
45
error %
40
35
30
25
0
20
40
60
80
epoch
Figure 7: Impact of gradient smoothing on ImageNet, ResNet50. Shown are the validation and training (dashed) top-1 accuracy using gradient smoothing and without it. The training regime used is B using an average size of 144 x 144 (s(144)
Increase image size from small to large (progressive resizing), where each image size is used for number of epochs Fg p; Protal: where Frotal is the total number training epochs required.
Using a random image size for each epoch, keeping the epoch number for each size at E,
Sampling image size per training step at probability p:
We used B regime and adapt learning rate for each batch-size according to best practices (Hoffer et al. 20177; Goyal et al. 2017). In both CIFARIO (Figure [8) and ImageNet (Figure ()) sampling methods exhibit better accuracy than progressive resizing regime. This is most apparent when the final size used at training is larger than target size for inference (on which test accuracy is measured).
40
Small->Large
Epoch
Step
30
error %
20
10
0
20
40
60
80
100
120
140
steps (1e3)
Figure 8: Test accuracy vs step (ResNet44 on CIFARIO, + with s28)) for size sampling regimes: (1) From small to large (2) Sample each Epoch (3) Sample each step. All methods reached a similar accuracy, but sampling each epoch was less noisy and did not require hyper-parameter tuning.
F ALTERNATIVE SIZE DISTRIBUTION REGIMES
We used alternative size regimes balanced around 224, named S(208) and S(224) They can be described by the following distributions:
14

---

Under review as a conference paper at ICLR 2021
70:
Small->Large
Slep
60
error %
50
40
30
20
0
20
40
60
80
epoch
Figure 9: Test accuracy vs epoch (ResNet50 on ImageNet, B+ with s(14)) for progressive resizing (small large) compared to per-step sampling.
320,
w.p P = 0.1
288,
w.p = 0.1
256,
w.p P 0.1
224,
w.p P 0.2
S(208) S
192,
w.p P = 0.2
160,
w.p P = 0.1
128,
w.p P 0.1
96,
w.P P 0.1
15


Reviews:
[{"review_id": "Z4y7Y839OxC", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a mixed-size CNN training scheme, using several different input image sizes for one single model training. The authors assume the training budget, represented as S_i^2*B_i*D_i (i.e., spatial sample size, the number of batched distinct samples and the duplicates for each distinct sample), to be a fixed constant during training step i.  Under such an assumption, two mixed-size training scenarios are considered, one for training acceleration and the other for improved model generalization ability. The authors additionally use step-wise image size sampling, gradient smoothing, and per-size BN calibration to enhance the model performance under the above two mixed-size training scenarios. Experimental validation is performed on CIFAR and ImageNet datasets using diverse CNN structures.\n\nMixed-size training is a critical problem and the methods proposed in this paper are interesting. My main concerns to this paper are as follows.\n\n--- Critical related works and comparison are missing.\n\nMixed-size training of CNNs for image classification are not new. Here are some recent works, however they are missed by the authors. \n\n“Resolution Adaptive Networks for Efficient Inference”, in CVPR 2020\n\n “Resolution Switchable Networks for Runtime Efficient Image Recognition”, in ECCV 2020.\n\n“MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution”, in ECCV 2020. \n\nBesides, as described by the authors, NeurIPS 2019 paper “Fixing the train-test resolution discrepancy” also considers how to enhance the performance of CNN models when applying them with different input image sizes. But a performance comparison is missing.\n\nTo show the advantages of this paper, thorough discussion and performance comparison with the above works are necessary. Taking ResNet-50 trained on ImageNet as an instance, I notice that the proposed method shows obviously worse accuracy compared to some of these works.\n\n---Another critical baseline is also missing. \n\nIn page 8, “We note that for original fixed-size regimes this calibration procedure resulted, with degraded results and so we report accuracy without calibration for these models”. To my understanding, this is somewhat weird. Why BN calibration does not work on the other image sizes when the model is trained with a fixed image size? It is not clear. Furthermore, in NeurIPS 2019 paper “Fixing the train-test resolution discrepancy”, this line of methods work pretty well. Such a BN calibration should server as another baseline for more fair comparison.\n\n---Regarding B+ design\n\nHow about the wall-clock training cost (in hours) instead of the number of iterations/epochs?\nHow about the performance of applying “scale the learning rate linearly” to train the baseline model? \n\n---How about model transfer ability? \n\nOnly image classification tasks are considered in the experiments. How about the performance of the trained backbone models, when transferring them in the downstream tasks, such as object detection and semantic segmentation? Can performance gain be transferred?\n\n---Others\n\nIs there any principled way regarding the size sampling strategy? The current strategy is based on manual setting, which limits its use in real applications. \n\nI suggest the authors to also provide precise accuracy numbers, etc. regarding some figures (e.g., figure 1, figure 4) shown in the paper. \n\nGenerally, I am on the fence, to this paper. I encourage the authors to address the questions I raised above.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "UMffaNsMSru", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes the use of mixed image sizes during training. They argue empirically that such an approach improves generalization for both fixed image size (e.g. 224 in ImageNet) as well as for variable image size. The proposed training algorithm maintains the same computational budget at each step by either changing the batch size or by using more/less augmentation. They show that adjusting the batch size leads to a faster training but using augmentation leads to a better test accuracy (hence a tradeoff).\n\nHowever, in order for their proposed method to work, the authors also propose modifications to standard training procedures (i.e. smoothing the gradient, adjusting the batchnorm layers) but without carrying out an ablation study that shows the impact with/without each of these steps. My particular concern is in the use of \"gradient smoothing\". If I understand it correctly, this is not very different from using momentum, which is known to reduce the variance and improve generalization. However, the authors use gradient smoothing only in their proposed method and do not use it in the baseline method (why not?). It is possible that the reported improvements (e.g. for fixed image size) come solely from this step. \n\nThe other concern is when sampling the image size per step. The authors  propose distributions that seem odd in their experiments (e.g. why is p=0.6 for size 128 in ImageNet which is much larger than others, and why not uniform in CIFAR10). It is important to know if the results are sensitive to the choice of the distribution, to make sure that the benefit is not due to random chance. Also, if this distribution needs to be fine-tuned, then the discussion about improving the training time would be meaningless. \n\nThe last issue is the robustness to different image sizes. Figure 3(a) shows  that if the average image size during training is small, the network will perform better for small images but not for large images. Conversely, if the average image size during training is large, it will perform better for large images, but not for small images. If the concern here is around using mixed-image sizes at inference time, then the red curve in Figure 3(a) shows that a fixed image size is reasonably robust. If one knows in advance that the average image size would be smaller than 224, one can train with a fixed image size that is smaller.\n\nOne minor last remark (feel free to ignore) regarding the motivation: the authors study the correlation between the full gradients for the same image with different sizes, on the one hand, and for different images with the same size, on the other hand. They conclude that the first case (different sizes) shows a stronger correlation, which is true according to Table 2, but this statement omits the fact that most correlations were low anyway. For example, for partially trained network, it is 0.08 vs. 0.02. I do not think that one can use such figures to conclude that \"smaller image gradients can be used as an estimate to the full image gradients\". \n\nThe improvement in test accuracy is very promising but I believe, some ablation is needed to identify exactly where this improvement comes from and whether it can be obtained using simpler approach (e.g. smoothing the gradient alone or using augmentation alone, etc). ", "extra_scores": {}, "extra_text": {}}, {"review_id": "IJ6U4CE3Lbz", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to increase training costs to compensate for the reduced costs from multi-scale CNN training by either increasing batch size (and therefore lowering the number of iterations per epoch) or increasing the number of augmented versions (duplicates) of the same samples within a batch. The former allows for smaller total training costs than conventional single-scale training, while the latter maintains the total training costs but improves the final performance. Several training improvement methods are introduced to improve the multi-scale training.\n\nPaper's strengths\n- The paper is quite well-written.\n- Code and models are provided for reproducibility.\n- Gradient smoothing is a nice way to mitigate the variability of gradient statistics resulted from different input sizes. As far as I know, this is quite novel, particularly in the context of multi-scale training.\n\nPaper's weaknesses\n- Multi-scale training is a common practice in many computer vision tasks especially in object detection** (less common in image classification). This paper also does multi-scale training but only introduces some minor improvements that are neither breakthroughs nor that they provide any interesting insight.\n> - Bag of Freebies for Training Object Detection Neural Networks. arXiv.\n> - YOLO9000: Better, Faster, Stronger. CVPR 2017.\n> - MMDetection: Open MMLab Detection Toolbox and Benchmark. arXiv.\n\n- For \"step-wise size sampling\", it seems like that conclusion to use this variant of sampling is heuristically chosen and totally ignores the existing practice in other computer vision tasks. One of the straightforward ways to do multi-scale training in object detection is to select different input sizes even for the images within the same batch (by padding zeros for the smaller images). Alternatively, one could sample different input sizes for different GPU batches (all images within a GPU share the same size) while doing multiple-GPU training.\n\n- The three training improvements (step-wise size sampling, gradient smoothing and batch-norm calibration) are what separate this paper from prior work but they are not extensively evaluated. Some of them are briefly evaluated in the appendix and some of their effects are just briefly mentioned in the method section. They ought to appear in the experimental section of the main paper. Gradient smoothing seems like a nice idea but it is unclear how important it is given that there is only one figure (Fig.7) showing its impact on the performance.\n\n- This paper strives to increase the number of batch samples given a fixed budget of computational and time resources for per iteration step. I wonder why this should be limited to the cost of an iteration step but not the entire training cost from all epochs/iterations. It makes more sense to measure the cost for the entire training process which accurately tells how much is spent to train the model until convergence.\n\n- In Sec. 5.1, the size ratios for different datasets are carefully chosen based on cross-validation. This makes it hard to directly apply MixSize to other datasets or settings without going through this step. It also adds additional computations which defeat the purpose of increasing batch size to maintain the same training cost as conventional single-scale training.\n\n- Using separate BatchNorm statistics for multi-scale inputs/features has been explored in the following papers (published at least few months before ICLR deadline). They should be cited and compared against MixSize:\n> - Learning to Learn Parameterized Classification Networks for Scalable Input Images. ECCV 2020.\n> - Stochastic Downsampling for Cost-Adjustable Inference and Improved Regularization in Convolutional Networks. CVPR 2018.\n\nOverall, this paper shows good performance and has some good ideas (e.g., gradient smoothing) for improving multi-scale training. But it fails to give more emphasis to or do a deeper dive into the potentially novel aspects of the work. The current performance improvements may come from doing just trivial multi-scale training which was already widely-explored in prior work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IAIDXwcGzpR", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "== Summary ==\n\nThe paper proposes to use different image resolutions during the training of a deep neural network image classifier, and varying the batch size or number of data augmented versions of the images, keeping the computational cost per step roughly constant. The authors apply this approach to several architectures and three datasets, and show that they can achieve or improve the same accuracy as the baselines but much faster; or achieve better results with the same computational budget.\n\n== Pros ==\n\n+ The authors conducted their experiments using three different datasets (Cifar10, Cifar100 and ImageNet), and six different architectures (ResNet44, ResNet50, WideResNet, DenseNet and EfficientNet).\n\n+ The proposed approach outperforms the baselines consistently across the 8 pairs of (dataset, architecture) that they have studied. In addition, MixSize can be easily implemented (the authors also provide a PyTorch implementation).\n\n+ The authors investigate different \"tricks\" to apply during training when using MixSize to stabilize training or achieving better results. For instance, they compared randomly sampling the size from a distribution (as they propose) versus increasing the image resolution as training progresses, and showed that randomly sampling yields slightly better results.\n\n+ Figure 3a seems to suggest that MixSize yields more robust classifiers under a wide range of image sizes. The area under the \"Mixed S=144\" curve seems to be larger than the area under the \"Fixed S=224\". However, further experimentation is needed to confirm this, since the area of the \"Mixed S=208\" seems closer to \"Fixed S=224\", and in any case the maximum image size was capped around 415.\n\n== Cons ==\n\n- On of the claimed contributions is: \"Faster training vs. high accuracy. We show that reducing the average image size at train- ing leads to a trade-off between the time required to train the model and its final accuracy.\". However, I would not consider this a novel contribution, since the trade-off between speed and accuracy is well-known. In fact, the authors cite Huang et al. (2018) and Szegedy et al. (2016) which already showed this. EfficientNet is another well-known architecture that takes advantage of this fact.\n\n- In the intro, the authors claim \"Touvron et al. (2019) demonstrated that networks trained on specific image resolutions perform poorly on other image sizes at evaluation time, as confirmed in Figure 1\". This is inaccurate, since Touvron et al. (2019) actually show that slightly increasing the test resolution improves accuracy, due to the discrepancy in object sizes introduced by data augmentation (cropping). In fact, Figure 1 shows the same effect (model trained with 224 res, achieves best results with 284 eval image size). The statement in the introduction is again contradicted at the end of the first paragraph in Section 2.1.\n\n- The authors do not report any statistical significance metric. Some datasets have very close results, so it's hard to tell whether the improvements are (statistically) significant or not.\n\n- Poor captions in figures and tables. For instance, the difference between solid lines and dashed lines is only explained at the very last figure in the appending (Figure 7). Also, the caption of Figure 3 reads \"Test accuracy on validation set\" which is ambiguous: Is it a typo, or is it that the authors report the results on the 50k validation set of ImageNet (and use some smaller subset of the training set as validation)? \n\n== Reasons for score ==\n\nAlthough the proposed approach is simple and consistently improves the baseline results, I'm not convinced that the originality and significance of the work is enough for it to be accepted. \n\nRegarding originality, there is a plethora of works exploring the trade-offs between image size and accuracy. The most similar works are Howard (2018) and Karras et al. (2017), which increase the image size through training. It's not clear that random sampling offers a much better result, judging from Figure 8 in Appending E, if one compares the accuracy of \"Small->Large\" strategy at 125k steps (possibly before the last increase in size).\n\nRegarding significance, if one restricts the analysis to the best architectures in each dataset, the increase in accuracy does not seem to be very large. Cifar10: 98.16% -> 98.32% (AmoebaNet), ImageNet: 76.32% -> 76.53% (EfficientNet-B0). Cifar100 shows a larger improvement, but the authors did not use AmoebaNet (which worked best in Cifar10) for some unknown reason. The fact that no statistical significance metrics are reported, does not help to discern whether the improvements are meaningful or not.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes to train networks with mixed image sizes to allow for faster inference and also for robustness. The reviewers found the paper was well-written and appreciated that the code was available for reproducibility. However, the paper does not sufficiently compare to related methods. The authors should resubmit once the comparisons suggested by the reviewers have been added to the paper.
Decision: Reject