MixSize: Training Convnets With Mixed Image Sizes for Improved Accuracy, Speed and Scale Resiliency

1 INTRODUCTION Convolutional neural networks are successfully used to solve various tasks across multiple domains such as visual ( Krizhevsky et al. , 2012 ; Ren et al. , 2015 ) , audio ( van den Oord et al. , 2016 ) , language ( Gehring et al. , 2017 ) and speech ( Abdel-Hamid et al. , 2014 ) . While scale-invariance is considered important for visual representations ( Lowe , 1999 ) , convolutional networks are not scale invariant with respect to the spatial resolution of the image input , as a change in image dimension may lead to a non-linear change of their output . Even though CNNs are able to achieve state-of-the-art results in many tasks and domains , their sensitivity to the image size is an inherent de- ficiency that limits practical use cases , and requires that images at evaluation time match training image size . For example , Touvron et al . ( 2019 ) demonstrated that networks trained on specific image size , perform poorly on other image sizes at evaluation time , as confirmed in Figure 1 . The most common method to improve scale invariance in CNNs to artificially enlarge the dataset using a set of label-preserving transformations also known as “ data augmentation ” ( Howard , 2013 ; Krizhevsky et al. , 2012 ) . Several of these transformations scale and crop objects appearing within the data , thus increasing the network ’ s robustness to inputs of different scale . Several works attempted to achieve scale invariance by modifying the network structure to learn over multiple possible target input scales ( Takahashi et al. , 2017 ; Xu et al. , 2014 ; Zhang et al. , 2019 ) . These methods 1https : //github.com/paper-submissions/mixsize explicitly change the model for specific input size , thus not benefiting from possible lower computational requirements of using smaller image sizes , nor with ability of inferring on sizes not observed during training . Another approach suggested by Cai et al . ( 2020 ) modifies network structure and training regime to account for variety of inference modes without additional specialized training . In this work , we introduce a novel training regime , “ MixSize ” for convolutional networks that uses stochastic image and batch sizes . The main contributions of the MixSize regime are : • Reducing image size sensitivity . We show that the MixSize training regime can improve model performance on a wide range of sizes used at evaluation . • Faster inference . As our mixed-size models can be evaluated at smaller image sizes , we show up to 2× reduction in computations required at inference to reach the same accuracy as the baseline model . • Faster training vs. high accuracy . We show that reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy . 2 BACKGROUND AND RELATED WORK . 2.1 USING MULTIPLE IMAGE SIZES . Deep convolutional networks are traditionally trained using fixed-size inputs , with spatial dimensions H × W and a batch size B . The network architecture is configured such that the spatial dimensions are reduced through strided pooling or convolutions , with the last classification layer applied on a 1 × 1 spatial dimension . Modern convolutional networks usually conclude with a final ” global ” average pooling ( Lin et al. , 2013 ; Szegedy et al. , 2015 ) , which reduces any remaining spatial dimensions with a simple averaging operation . Modifying the spatial size of an input to a convolutional layer by a factor γ , will yield an output with size scaled by the same factor γ . This modification does not require any change to the number of parameters of the given convolutional layer , nor its underlying operation . It was observed by practitioners and previous works that a network trained on a specific input dimension can still be used at inference using a modified image size to some extent ( Simonyan & Zisserman , 2014 ) . Moreover , evaluating with an image size that is larger than used for training can improve accuracy up to a threshold , after which it quickly deteriorates ( Touvron et al. , 2019 ) . Although not explicitly trained to handle varying image sizes , CNNs are commonly evaluated on multiple scales post-training , such as in the case of detection ( Lin et al. , 2017 ; Redmon & Farhadi , 2018 ; Liu et al. , 2020 ) and segmentation ( He et al. , 2017 ) tasks . In these tasks , a network that was pretrained with fixed image size for classification is used as the backbone of a larger model that is expected to adapt to a wide variety of image sizes . Recently , Tan & Le ( 2019 ) showed a computation-vs-accuracy trade-off in applying scaling to the image size used for convolutional networks training and evaluation . This finding is consistent with past findings , which demonstrated that training with a larger image size can result in a better classification accuracy ( Huang et al. , 2018 ; Szegedy et al. , 2016 ) . In addition , previous works explored the notion of “ progressive resizing ” ( Howard , 2018 ; Karras et al. , 2017 ) — increasing image size as training progresses to improve model performance and time to convergence . A similar idea by Wu et al . ( 2020 ) was used to improve performance of training on video data , by balancing resolution with batch size . Another related work by Touvron et al . ( 2019 ) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size , with which evaluation will be performed . This procedure reduced the train-test discrepancy caused by the change in image size and allowed faster training time and improved accuracy — at the cost of additional fine-tuning procedure and additional computations at inference time . In this work we will further explore the notion of using multiple image sizes at training , so the CNN performance will be resilient to test time changes of the image size . 2.2 LARGE BATCH TRAINING OF DEEP NETWORKS . Deep neural network training can be distributed across many computational units and devices . The most common distribution method is by “ data-parallelism ” — computing an average estimate of the gradients using multiple , separately computed data samples . As training NN models is done using batch-SGD method and its variants , scaling this process across multiple computational devices while maintaining similar utilization for each device inflates the global batch size . Large batch training is known to affect the generalization capabilities of NNs and as such , it requires to modify its optimization regime . While several works claimed that large-batch training leads to an inherent “ generalization gap ” ( Keskar et al. , 2016 ) , more recent works demonstrated that this gap is largely caused by insufficient number of optimization steps performed and can be partly mitigated by hyper-parameter tuning ( Hoffer et al. , 2017 ; Shallue et al. , 2018 ) . In order to cope with the variations in the network training dynamics , several modifications of the optimization procedure have been proposed . For instance , a linear ( Goyal et al. , 2017 ) or square-root ( Hoffer et al. , 2017 ) scaling of the learning rate with respect to the batch size growth . Other modifications include perlayer gradient scaling schemes ( You et al. , 2017 ) and optimizer modifications ( Ginsburg et al. , 2019 ) . Several works also explored using incremented batch-sizes ( Smith et al. , 2018 ) in order to decrease the number of training iterations required to reach the desired accuracy . Recent work ( Hoffer et al. , 2020 ) introduced the notion of “ Batch Augmentation ” ( BA ) — increasing the batch size by augmenting several instances of each sample within the same batch . BA aids generalization across a wide variety of models and tasks , with the expense of an increased computational effort per step . A similar method called “ Repeated Augmentation ” ( RA ) was proposed by Berman et al . ( 2019 ) . It was also demonstrated that BA may allow to decrease the number of training steps needed to achieve a similar accuracy and also mitigate I/O throughput bottlenecks ( Choi et al. , 2019 ) . As previous works investigated mostly homogeneous training settings ( e.g. , using a fixed batch size ) , an open question still exists on the utility of rapidly varying batch-sizes . We will explore this notion by modifying the optimization method , enabling training with varying batch-sizes and limited hyper-parameter tuning . 3 MIXSIZE : TRAINING WITH MULTIPLE IMAGE SCALES . The traditional practice of training convolutional networks using fixed-size images holds several shortcomings . As it is a common practice to use different image size for evaluation than that used for training ( He et al. , 2017 ; Lin et al. , 2017 ; Redmon & Farhadi , 2018 ) , it was observed by Touvron et al . ( 2019 ) and empirically verified ( Figure 1 ) that classification accuracy may degrade above or below a certain size threshold . Similarly , we observed this phenomenon in a wide variety of common networks ( Appendix Figure 4 ) . This hints the issue here is related more to the training procedure , rather than to a specific model . To remedy these issues , we suggest a stochastic training regime , where image sizes may vary in each optimization step . Motivation . We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time . As a first step to substantiate this claim , we evaluated the impact of various image sizes on the CNN training progress by examining gradient statistics during training . Specifically , we measured the correlation between weight gradients with respect to varying image sizes ( Appendix Table 2 ) . We see that gradients computed across different sizes of the same images have a stronger correlation than those obtained across different images . Moreover , the correlation is most significant during the first stages of training and decreases as the model converges . This observation suggests that smaller image gradients can be used as an estimate to the full image gradients , with a smaller computational footprint . Therefore , using large images along the entire training process may be sub-optimal in terms of computational resource utilization . We continue to test our hypothesis using a novel mixed-size training regime . MixSize training regime . We propose ” MixSize ” , a stochastic training regime where input sizes can vary . In this regime , we modified the spatial dimensions H , W ( height and width ) of the input image size2 , as well as the batch size . The batch size is modified either by the number of used samples , denoted B , or by the number of batch-augmentations for each sample ( Hoffer et al. , 2020 ) , denoted D ( “ duplicates ” ) . To simplify our notation and use-cases , we will follow the common practice of training on square images and use S = H = W . Formally , in the MixSize regime , these 2The spatial dimensions of all intermediate maps in the CNN are changed accordingly , at the same scale as the input . sizes can be described as random variables sharing a single discrete distribution ( Ŝ , B̂ , D̂ ) = { ( S , B , D ) i w.p . pi } , ( 1 ) where ∀i : pi ≥ 0 and ∑ i pi = 1 . As the computational cost of each training step is approximately proportional to S2 · B · D , we choose these quantities to reflect a nearly fixed budget for any choice i such that S2iBiDi ≈ Const . Thus , in this regime , the computational and memory requirements for each step in remain fixed . Benefits and Trade-offs . We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes . This entails a considerable saving in computational burden required for inference , especially when using smaller models . Furthermore , given a fixed budget of computational and time resources ( per step ) , we can now modify our regime along spatial and batch axes . We will explore two trade-offs : • Decrease number of iterations per epoch – by enlarging B at the expense of S. • Improve generalization per epoch – by enlarging D at the expense of S. We denote our modified mixed-size regimes as B+ for an increased effective batch-size and D+ for an increased number of BA duplicates of the same ratio .

Summary: This paper presents a mixed-size CNN training scheme, using several different input image sizes for one single model training. The authors assume the training budget, represented as S_i^2*B_i*D_i (i.e., spatial sample size, the number of batched distinct samples and the duplicates for each distinct sample), to be a fixed constant during training step i.  Under such an assumption, two mixed-size training scenarios are considered, one for training acceleration and the other for improved model generalization ability. The authors additionally use step-wise image size sampling, gradient smoothing, and per-size BN calibration to enhance the model performance under the above two mixed-size training scenarios. Experimental validation is performed on CIFAR and ImageNet datasets using diverse CNN structures.