Revisiting ResNets: Improved Training and Scaling Strategies

1 Introduction . The performance of a vision model is a product of the architecture , training methods and scaling strategy . Novel architectures underlie many advances , but are often simultaneously introduced with other critical – and less publicized – changes in the details of the training methodology and hyperparameters . Additionally , new architectures enhanced by modern training methods are sometimes compared to older architectures with dated training methods ( e.g . ResNet-50 with ImageNet Top-1 accuracy of 76.5 % [ 13 ] ) . Our work addresses these issues and empirically studies the impact of training methods and scaling strategies on the popular ResNet architecture [ 13 ] . We survey the modern training and regularization techniques widely in use today and apply them to ResNets ( Figure 1 ) . In the process , we encounter interactions between training methods and show a benefit of reducing weight decay values when used in tandem with other regularization techniques . An additive study of training methods in Table 1 reveals the significant impact of these decisions : a Correspondence to Irwan Bello and Barret Zoph { ibello , barretzoph } @ google.com . Code and checkpoints available in TensorFlow : https : //github.com/tensorflow/models/tree/master/ official/vision/beta and https : //github.com/tensorflow/tpu/tree/master/ models/official/resnet/resnet_rs 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . canonical ResNet-200 with 79.0 % top-1 ImageNet accuracy is improved to 82.2 % ( +3.2 % ) through improved training methods alone . This is increased further to 83.4 % by two small and commonly used architectural improvements : ResNet-D [ 15 ] and Squeeze-and-Excitation [ 21 ] . Figure 1 traces this refinement over the starting ResNet in a speed-accuracy Pareto curve . We offer new perspectives and practical advice on scaling vision architectures . While prior works extrapolate scaling rules from small models [ 55 ] or from short training duration [ 39 ] , we design scaling strategies by exhaustively training models across a variety of scales for the full training duration ( e.g . 350 epochs instead of 10 epochs ) . In doing so , we uncover strong dependencies between the best performing scaling strategy and the training regime ( e.g . number of epochs , model size , dataset size ) . These dependencies are missed in any of these smaller regimes , leading to suboptimal scaling decisions . Our analysis leads to new scaling strategies summarized as ( 1 ) scale the model depth when overfitting can occur ( scaling the width is preferable otherwise ) and ( 2 ) scale the image resolution more slowly than prior works [ 55 ] . Using the improved training and scaling strategies , we design a family of re-scaled ResNets , ResNetRS , across model various scales ( Figure 1 ) . ResNet-RS models use less memory during training and are 1.7x - 2.7x faster on TPUs ( 2.1x - 3.3x faster on GPUs ) than the popular EfficientNets on the speed-accuracy Pareto curve . In a large-scale semi-supervised learning setup , ResNet-RS obtains a 4.7x training speed-up on TPUs ( 5.5x on GPUs ) over EfficientNet-B5 when co-trained on ImageNet [ 30 ] and an additional 130M pseudo-labeled images . Finally , we conclude with a suite of experiments testing the generality of the improved training and scaling strategies . We first demonstrate that our scaling strategy improves the speed-accuracy Pareto curve of EfficientNet . Next , we show that the improved training strategies yield representations that rival or outperform those from self-supervised algorithms ( SimCLR and SimCLRv2 [ 4 , 5 ] ) on a suite of downstream tasks . The improved training strategies also extend to video classification , yielding an improvement from 73.4 % to 77.4 % ( +4.0 % ) on the Kinetics-400 dataset . Through combining lightweight architectural changes ( used since 2018 ) and improved training and scaling strategies , we discover the ResNet architecture sets a state-of-the-art baseline for vision research . This finding highlights the importance of teasing apart each of these factors in order to understand what architectures perform better than others . We summarize our contributions : • An empirical study of regularization techniques and their interplay , which leads to a training strategy that achieves strong performance ( e.g . +3.2 % top-1 ImageNet accuracy , +4.0 % top-1 Kinetics-400 accuracy ) without having to change the model architecture . • An empirical study of scaling which uncovers strong dependencies between training and the best performing scaling strategy . We propose a simple scaling strategy : ( 1 ) scale depth when overfitting can occur ( scaling width can be preferable otherwise ) and ( 2 ) scale the image resolution more slowly than prior works [ 55 ] . This scaling strategy improves the speed-accuracy Pareto curve of both ResNets and EfficientNets . • ResNet-RS : a Pareto curve of ResNet architectures that are 1.7x - 2.7x faster than EfficientNets on TPUs ( 2.1x - 3.3x on GPUs ) by applying the training and scaling strategies . Semi-supervised training of ResNet-RS with an additional 130M pseudo-labeled images achieves 86.2 % top-1 ImageNet accuracy , while being 4.7x faster on TPUs ( 5.5x on GPUs ) than the corresponding EfficientNet-NoisyStudent [ 57 ] . • Empirically show that representations obtained from supervised learning using modern training techniques rival or outperform state-of-the-art self-supervised representations ( SimCLR [ 4 ] , SimCLRv2 [ 5 ] ) on suite of downstream computer vision tasks . 2 Characterizing Improvements on ImageNet . Since the breakthrough of AlexNet [ 30 ] on ImageNet [ 45 ] , a wide variety of improvements have been proposed to further advance image recognition performance . These improvements broadly arise along four orthogonal axes : ( a ) architecture , ( b ) training/regularization methodology , ( c ) scaling strategy and ( d ) using additional training data . ( a ) Architecture . The works that perhaps receive the most attention are novel architectures . Notable proposals since AlexNet include VGG [ 49 ] , ResNet [ 13 ] , Inception [ 52 , 53 ] , and ResNeXt [ 58 ] . Automated search strategies for designing architectures have further pushed the state-of-the-art [ 67 , 41 , 55 ] . There have also been efforts in going beyond standard ConvNets for image classification , by adapting self-attention [ 56 ] to the visual domain [ 2 , 40 , 20 , 47 , 8 , 1 ] . ( b ) Training and Regularization Methods . ImageNet progress has simultaneously been boosted by innovations in training ( e.g . improved learning rate schedules [ 34 , 12 ] ) and regularization methods , such as dropout [ 50 ] , label smoothing [ 53 ] , stochastic depth [ 22 ] , dropblock [ 11 ] and data augmentation [ 61 , 59 , 6 , 7 ] . Regularization methods have become especially useful to prevent overfitting when training ever-increasingly larger models [ 23 ] on limited data ( e.g . 1.2M ImageNet images ) . ( c ) Scaling Strategies . Increasing the model dimensions ( width , depth and resolution ) has been another successful axis to improve quality [ 44 , 17 ] . ResNet architectures are typically scaled up by adding layers ( depth ) : ResNets-18 to ResNet-200 and beyond [ 14 , 62 ] . Wide ResNets [ 60 ] and MobileNets [ 19 ] instead scale the width . Increasing image resolutions consistently improves performance : EfficientNet uses 600 image resolutions [ 55 ] while both ResNeSt [ 62 ] and TResNet [ 43 ] use 400+ image resolutions for their largest model . In an attempt to systematize these heuristics , EfficientNet proposed the compound scaling rule , which jointly scales network depth , width and image resolution using a constant scaling factor . However , Section 7.1 shows this scaling strategy is sub-optimal for not only ResNets , but EfficientNets as well . ( d ) Additional Training Data . Finally , ImageNet accuracy is commonly improved by training on additional sources of data ( either labeled , weakly labeled , or unlabeled ) . Pre-training on large-scale datasets [ 51 , 35 , 27 ] has significantly pushed the state-of-the-art , with ViT [ 8 ] and NFNets [ 3 ] recently achieving 88.6 % and 89.2 % ImageNet accuracy respectively . Using pseudo-labels on additional unlabeled images [ 57 , 37 ] in a semi-supervised learning fashion has also been a fruitful avenue for improving accuracy . We present semi-supervised learning results in Section 7.2 . 3 Related Work . Improved training methods combined with architectural changes to ResNets have routinely yielded competitive ImageNet performance [ 15 , 31 , 43 , 62 , 1 , 3 ] . [ 15 ] achieved 79.2 % top-1 ImageNet accuracy ( a +3 % improvement over their ResNet-50 baseline ) by modifying the stem and downsampling block and using label smoothing and mixup . [ 31 ] further improved the ResNet-50 model with additional architectural modifications such as Squeeze-and-Excitation [ 21 ] , selective kernel [ 32 ] , and anti-alias downsampling [ 63 ] , while also using label smoothing , mixup , and dropblock to achieve 81.4 % accuracy . [ 43 , 62 ] incorporate several architectural modifications to the ResNet architectures along with improved training methodologies to outperform EfficientNet models on the speed-accuracy Pareto curve on GPUs . Many prior works do remark the importance of improved training and regularization methods . However experiments are still largely concerned with architectural changes and the simultaneous introduction of improved training techniques can make it hard to identify where the gains come from1 . Additionally , due to the ever-increasing performance of machine learning accelerators , newer architectures are routinely pushed to much larger scales than the original ResNets [ 13 ] . As a result , works that propose novel architectures do not ( can not ) compare against properly trained and scaled ResNets ( since such a baseline did not exist ) , making it challenging to evaluate the significance of the proposed architectural changes compared to simple ResNets . Lastly , prior work often puts little emphasis on studying scaling strategies or advocates for scaling strategies which we find to be sub-optimal . For example , the largest models in EfficientNet [ 55 ] , TResNet [ 43 ] and ResNeSt [ 62 ] use 600 , 448 and 416 image sizes respectively , which our scaling analysis reveals to be excessively large . RegNet [ 39 ] advocates for width scaling , which we find only works well when overfitting does not occur ( e.g . 10 epochs ) . In contrast to other works , we only consider lightweight architectural changes ( that are widely used since 2018 ) and keep the architecture fixed . Instead , we focus exclusively on training and scaling strategies to build a Pareto curve of models . Perhaps surprisingly , we find that doing so suffices to outperform models that were introduced after ResNets : our improved training and scaling methods lead to ResNets that are significantly faster than EfficientNets on TPUs on GPUs ( see Section 7.1 ) . We note that our scaling improvements are sometimes orthogonal to the architectural innovations introduced in prior works in which case we expect them to be additive . 4 Methodology . Architecture . Our work studies the ResNet architecture , with two widely used architecture changes , the ResNet-D [ 15 ] modification and Squeeze-and-Excitation ( SE ) in all bottleneck blocks [ 21 ] . These architectural changes are used in used many architectures , including TResNet , ResNeSt and EfficientNets . The exact details of our architecture can be found in Appendix E. In our experiments , we sometimes use the original ResNet implementation without SE ( referred to as ResNet ) to compare different training methods . Clear denotations are made in table captions when this is the case . Regularization and Data Augmentation . We apply weight decay , label smoothing , dropout and stochastic depth for regularization . Dropout [ 50 ] is a common technique used in computer vision and we apply it to the output after the global average pooling occurs in the final layer . Stochastic depth [ 22 ] drops out each layer in the network ( that has residual connections around it ) with a specified probability that is a function of the layer depth . We use RandAugment [ 7 ] data augmentation as an additional regularizer . RandAugment applies a sequence of random image transformations ( e.g . translate , shear , color distortions ) to each image independently during training . Our training method closely matches that of EfficientNet , where we train for 350 epochs , but with a few small differences ( e.g . we use Momentum with cosine learning rate schedule as opposed to RMSProp with exponential decay ) . See Appendix D for details . Hyperparameter Tuning . To select the hyperparameters for the various regularization and training methods , we use a held-out validation set comprising 2 % of the ImageNet training set ( 20 shards out of 1024 ) . This is referred to as the minival-set and the original ImageNet validation set ( the one reported in most prior works ) is referred to as validation-set . Unless specified otherwise , results are reported on the validation-set . The hyperparameters of all ResNet-RS models are in Table 8 in the Appendix C. 1A notable exception is RegNet [ 39 ] which purposely makes no use of improved training techniques and shows improvements over worsened EfficientNet baselines , but does not demonstrate ImageNet accuracies above ( a rather low ) 81 % . While this approach facilitates fair comparisons with prior work , it is unclear whether improvements are sustained at larger scales with improved training setups . For example , our scaling analysis shows that the scaling strategy advocated by RegNet does not generalize to training regimes where overfitting can occur .

Summary: This paper revisits the performance of ResNet from three aspects, architecture, training scheme, and scaling strategy. The author shows that a better training and scaling strategy can give even more improvements than better architecture design. The author conducted many experiments to empirically show what is a good training and scaling strategy and how good it can be.