MixSize: Training Convnets With Mixed Image Sizes for Improved Accuracy, Speed and Scale Resiliency

1 INTRODUCTION Convolutional neural networks are successfully used to solve various tasks across multiple domains such as visual ( Krizhevsky et al. , 2012 ; Ren et al. , 2015 ) , audio ( van den Oord et al. , 2016 ) , language ( Gehring et al. , 2017 ) and speech ( Abdel-Hamid et al. , 2014 ) . While scale-invariance is considered important for visual representations ( Lowe , 1999 ) , convolutional networks are not scale invariant with respect to the spatial resolution of the image input , as a change in image dimension may lead to a non-linear change of their output . Even though CNNs are able to achieve state-of-the-art results in many tasks and domains , their sensitivity to the image size is an inherent de- ficiency that limits practical use cases , and requires that images at evaluation time match training image size . For example , Touvron et al . ( 2019 ) demonstrated that networks trained on specific image size , perform poorly on other image sizes at evaluation time , as confirmed in Figure 1 . The most common method to improve scale invariance in CNNs to artificially enlarge the dataset using a set of label-preserving transformations also known as “ data augmentation ” ( Howard , 2013 ; Krizhevsky et al. , 2012 ) . Several of these transformations scale and crop objects appearing within the data , thus increasing the network ’ s robustness to inputs of different scale . Several works attempted to achieve scale invariance by modifying the network structure to learn over multiple possible target input scales ( Takahashi et al. , 2017 ; Xu et al. , 2014 ; Zhang et al. , 2019 ) . These methods 1https : //github.com/paper-submissions/mixsize explicitly change the model for specific input size , thus not benefiting from possible lower computational requirements of using smaller image sizes , nor with ability of inferring on sizes not observed during training . Another approach suggested by Cai et al . ( 2020 ) modifies network structure and training regime to account for variety of inference modes without additional specialized training . In this work , we introduce a novel training regime , “ MixSize ” for convolutional networks that uses stochastic image and batch sizes . The main contributions of the MixSize regime are : • Reducing image size sensitivity . We show that the MixSize training regime can improve model performance on a wide range of sizes used at evaluation . • Faster inference . As our mixed-size models can be evaluated at smaller image sizes , we show up to 2× reduction in computations required at inference to reach the same accuracy as the baseline model . • Faster training vs. high accuracy . We show that reducing the average image size at training leads to a trade-off between the time required to train the model and its final accuracy . 2 BACKGROUND AND RELATED WORK . 2.1 USING MULTIPLE IMAGE SIZES . Deep convolutional networks are traditionally trained using fixed-size inputs , with spatial dimensions H × W and a batch size B . The network architecture is configured such that the spatial dimensions are reduced through strided pooling or convolutions , with the last classification layer applied on a 1 × 1 spatial dimension . Modern convolutional networks usually conclude with a final ” global ” average pooling ( Lin et al. , 2013 ; Szegedy et al. , 2015 ) , which reduces any remaining spatial dimensions with a simple averaging operation . Modifying the spatial size of an input to a convolutional layer by a factor γ , will yield an output with size scaled by the same factor γ . This modification does not require any change to the number of parameters of the given convolutional layer , nor its underlying operation . It was observed by practitioners and previous works that a network trained on a specific input dimension can still be used at inference using a modified image size to some extent ( Simonyan & Zisserman , 2014 ) . Moreover , evaluating with an image size that is larger than used for training can improve accuracy up to a threshold , after which it quickly deteriorates ( Touvron et al. , 2019 ) . Although not explicitly trained to handle varying image sizes , CNNs are commonly evaluated on multiple scales post-training , such as in the case of detection ( Lin et al. , 2017 ; Redmon & Farhadi , 2018 ; Liu et al. , 2020 ) and segmentation ( He et al. , 2017 ) tasks . In these tasks , a network that was pretrained with fixed image size for classification is used as the backbone of a larger model that is expected to adapt to a wide variety of image sizes . Recently , Tan & Le ( 2019 ) showed a computation-vs-accuracy trade-off in applying scaling to the image size used for convolutional networks training and evaluation . This finding is consistent with past findings , which demonstrated that training with a larger image size can result in a better classification accuracy ( Huang et al. , 2018 ; Szegedy et al. , 2016 ) . In addition , previous works explored the notion of “ progressive resizing ” ( Howard , 2018 ; Karras et al. , 2017 ) — increasing image size as training progresses to improve model performance and time to convergence . A similar idea by Wu et al . ( 2020 ) was used to improve performance of training on video data , by balancing resolution with batch size . Another related work by Touvron et al . ( 2019 ) demonstrated that CNNs can be trained using a fixed small image size and fine-tuned post-training to a larger size , with which evaluation will be performed . This procedure reduced the train-test discrepancy caused by the change in image size and allowed faster training time and improved accuracy — at the cost of additional fine-tuning procedure and additional computations at inference time . In this work we will further explore the notion of using multiple image sizes at training , so the CNN performance will be resilient to test time changes of the image size . 2.2 LARGE BATCH TRAINING OF DEEP NETWORKS . Deep neural network training can be distributed across many computational units and devices . The most common distribution method is by “ data-parallelism ” — computing an average estimate of the gradients using multiple , separately computed data samples . As training NN models is done using batch-SGD method and its variants , scaling this process across multiple computational devices while maintaining similar utilization for each device inflates the global batch size . Large batch training is known to affect the generalization capabilities of NNs and as such , it requires to modify its optimization regime . While several works claimed that large-batch training leads to an inherent “ generalization gap ” ( Keskar et al. , 2016 ) , more recent works demonstrated that this gap is largely caused by insufficient number of optimization steps performed and can be partly mitigated by hyper-parameter tuning ( Hoffer et al. , 2017 ; Shallue et al. , 2018 ) . In order to cope with the variations in the network training dynamics , several modifications of the optimization procedure have been proposed . For instance , a linear ( Goyal et al. , 2017 ) or square-root ( Hoffer et al. , 2017 ) scaling of the learning rate with respect to the batch size growth . Other modifications include perlayer gradient scaling schemes ( You et al. , 2017 ) and optimizer modifications ( Ginsburg et al. , 2019 ) . Several works also explored using incremented batch-sizes ( Smith et al. , 2018 ) in order to decrease the number of training iterations required to reach the desired accuracy . Recent work ( Hoffer et al. , 2020 ) introduced the notion of “ Batch Augmentation ” ( BA ) — increasing the batch size by augmenting several instances of each sample within the same batch . BA aids generalization across a wide variety of models and tasks , with the expense of an increased computational effort per step . A similar method called “ Repeated Augmentation ” ( RA ) was proposed by Berman et al . ( 2019 ) . It was also demonstrated that BA may allow to decrease the number of training steps needed to achieve a similar accuracy and also mitigate I/O throughput bottlenecks ( Choi et al. , 2019 ) . As previous works investigated mostly homogeneous training settings ( e.g. , using a fixed batch size ) , an open question still exists on the utility of rapidly varying batch-sizes . We will explore this notion by modifying the optimization method , enabling training with varying batch-sizes and limited hyper-parameter tuning . 3 MIXSIZE : TRAINING WITH MULTIPLE IMAGE SCALES . The traditional practice of training convolutional networks using fixed-size images holds several shortcomings . As it is a common practice to use different image size for evaluation than that used for training ( He et al. , 2017 ; Lin et al. , 2017 ; Redmon & Farhadi , 2018 ) , it was observed by Touvron et al . ( 2019 ) and empirically verified ( Figure 1 ) that classification accuracy may degrade above or below a certain size threshold . Similarly , we observed this phenomenon in a wide variety of common networks ( Appendix Figure 4 ) . This hints the issue here is related more to the training procedure , rather than to a specific model . To remedy these issues , we suggest a stochastic training regime , where image sizes may vary in each optimization step . Motivation . We hypothesize that a considerable part of the training of convolutional networks can be performed at a lower image size than the target size used at test time . As a first step to substantiate this claim , we evaluated the impact of various image sizes on the CNN training progress by examining gradient statistics during training . Specifically , we measured the correlation between weight gradients with respect to varying image sizes ( Appendix Table 2 ) . We see that gradients computed across different sizes of the same images have a stronger correlation than those obtained across different images . Moreover , the correlation is most significant during the first stages of training and decreases as the model converges . This observation suggests that smaller image gradients can be used as an estimate to the full image gradients , with a smaller computational footprint . Therefore , using large images along the entire training process may be sub-optimal in terms of computational resource utilization . We continue to test our hypothesis using a novel mixed-size training regime . MixSize training regime . We propose ” MixSize ” , a stochastic training regime where input sizes can vary . In this regime , we modified the spatial dimensions H , W ( height and width ) of the input image size2 , as well as the batch size . The batch size is modified either by the number of used samples , denoted B , or by the number of batch-augmentations for each sample ( Hoffer et al. , 2020 ) , denoted D ( “ duplicates ” ) . To simplify our notation and use-cases , we will follow the common practice of training on square images and use S = H = W . Formally , in the MixSize regime , these 2The spatial dimensions of all intermediate maps in the CNN are changed accordingly , at the same scale as the input . sizes can be described as random variables sharing a single discrete distribution ( Ŝ , B̂ , D̂ ) = { ( S , B , D ) i w.p . pi } , ( 1 ) where ∀i : pi ≥ 0 and ∑ i pi = 1 . As the computational cost of each training step is approximately proportional to S2 · B · D , we choose these quantities to reflect a nearly fixed budget for any choice i such that S2iBiDi ≈ Const . Thus , in this regime , the computational and memory requirements for each step in remain fixed . Benefits and Trade-offs . We will show that using our MixSize regime can have a positive impact on the resiliency of trained networks to various images sizes at evaluation time by demonstrating better accuracy across a wide range of sizes . This entails a considerable saving in computational burden required for inference , especially when using smaller models . Furthermore , given a fixed budget of computational and time resources ( per step ) , we can now modify our regime along spatial and batch axes . We will explore two trade-offs : • Decrease number of iterations per epoch – by enlarging B at the expense of S. • Improve generalization per epoch – by enlarging D at the expense of S. We denote our modified mixed-size regimes as B+ for an increased effective batch-size and D+ for an increased number of BA duplicates of the same ratio .

Summary: The paper proposes the use of mixed image sizes during training. They argue empirically that such an approach improves generalization for both fixed image size (e.g. 224 in ImageNet) as well as for variable image size. The proposed training algorithm maintains the same computational budget at each step by either changing the batch size or by using more/less augmentation. They show that adjusting the batch size leads to a faster training but using augmentation leads to a better test accuracy (hence a tradeoff).