Revisiting ResNets: Improved Training and Scaling Strategies
Authors: Irwan Bello
William Fedus
Xianzhi Du
Ekin Dogus Cubuk
Aravind Srinivas
Tsung-Yi Lin
Jonathon Shlens
Barret Zoph
Venue: NeurIPS 2021 Spotlight (2021)
Keywords: Scaling
Architectures
ResNet
EfficientNet
ViT
Image Classification
Vision
ImageNet

Abstract: Novel computer vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes to training methodology and scaling strategies.
Our work revisits the canonical ResNet and studies these three aspects in an effort to disentangle them. Perhaps surprisingly, we find that training and scaling strategies may matter more than architectural changes, and further, that the resulting ResNets match recent state-of-the-art models. We show that the best performing scaling strategy depends on the training regime and offer two new scaling strategies: (1) scale model depth in regimes where overfitting can occur (width scaling is preferable otherwise); (2) increase image resolution more slowly than previously recommended.
Using improved training and scaling strategies, we design a family of ResNet architectures, ResNet-RS, which are 1.7x - 2.7x faster than EfficientNets on TPUs, while achieving similar accuracies on ImageNet. In a large-scale semi-supervised learning setup, ResNet-RS achieves 86.2% top-1 ImageNet accuracy, while being 4.7x faster than EfficientNet-NoisyStudent. The training techniques improve transfer performance on a suite of downstream tasks (rivaling state-of-the-art self-supervised algorithms) and extend to video classification on Kinetics-400. We recommend practitioners use these simple revised ResNets as baselines for future research.

Revisiting ResNets: Improved Training and Scaling Strategies
Irwan Bello
William Fedus Google Brain
Xianzhi Du Google Brain
Ekin D. Cubuk Google Brain
Aravind Srinivas UC Berkeley
Google Brain
Tsung-Yi Lin
Jonathon Shlens
Barret Zoph Google Brain
Google Brain
Google Brain
Abstract
Novel computer vision architectures monopolize the spotlight, but the impact of the model architecture is often conflated with simultaneous changes training method- ology and scaling strategies. Our work revisits the canonical ResNet [13] and studies these three aspects in an effort disentangle them. Perhaps surprisingly, we find that training and scaling strategies may matter more than architectural changes, and further, that the resulting ResNets match recent state-of-the-art mod- els. We show that the best performing scaling strategy depends on the training regime and offer two new scaling strategies: (1) scale model depth in regimes where overfitting can occur (width scaling is preferable otherwise); (2) increase image resolution more slowly than previously recommended [55]. Using improved training and scaling strategies, we design family of ResNet architectures, ResNet- RS, which are 1.7x 2.7x faster than EfficientNets on TPUs, while achieving similar accuracies on IeageNet. In parge-scale semi-supervised learning setup, ResNet-RS achieves 86.2% top-1 ImageNet accuracy, while being 4.7x faster than The training techniques improve transfer performance on a suite of downstream tasks (rivaling state-of-the-as self-supervised algorithmss) and extend to video classification on Kinetics-400. We recommend practitioners use these simple revised ResNets as baselines for future research.
SfficietNee-NNoisyStudeee
Introduction
The performance of vision model is product of the architecture, training methods and scaling strategy. Novel architectures underlie many advances, but are often simultaneously introduced with other critical and less publicized changes in the details of the training methodology and hyper- parameters. Additionally, new architectures enhanced by modern training methods are sometimes compared older architectures with dated training methods (e.g. ResNet-50 with ImageNet Top-1 accuracy of 76.5% [13]). Our work addresses these issues and empirically studies the impact of training methods and scaling strategies on the popular ResNet architecture [13].
We survey the modern training and regularization techniques widely in use today and apply them to ResNets (Figure 1). In the process, encounter interactions between training methods and show benefit reducing weight decay values when used in tandem with other regularization techniques. An additive study of training methods in Table 1 reveals the significant impact of these decisions:
Correspondence to Irwan Bello and Barret Zoph (ibello.barrtzoohh) ggogle..com. Code and check- points available in TensorFlow: https: official/visicn/beett
httts///eeeeeeeeeeeeeeeeeee/e / and https: htt.a///ee////.tooooooooooooooon models/officiaa//renett
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Speed-Acouracy Pareto Curve
x Speedup
84
Top-1 ImageNet Accuracy
83-
+ Minor Architectural Changes
82
81
Improved Training Strategies
80
ResNet- RS
EticientNet
79
ResNet
0.0
0.5
1.0
1.5
2.0
2.5
3.0
Time Per Training Step (Sec)
Figure Improving ResNets to state-of-the-art performance. improve on the canonical ResNet [13] with modern training methods (as also used in EfficientNets [55]), minor architectural changes and improved scaling strategies. The resulting models, ResNet-RS, outperform EfficientNets the speed-accuracy Pareto curve with speed-ups ranging from 1.7x 2.7x TPUs and 2.1x 3.3x GPUs. ResNet (+) is ResNet-220 trained at 256 resolution. Training times reported on TPUs.
canonical ResNet-200 with 79.0% top-1 ImageNet accuracy is improved to 82.2% (+3.2%) through improved training methods alone. This is increased further to 83.4% by two small and commonly used architectural improvements: ResNet-D [15] and Squeeze-and-Exeitatinn this refinement over the starting ResNet in speed-accuracy Pareto curve.
[21]. Figure traces
We offer new perspectives and practical advice on scaling vision architectures. While prior works extrapolate scaling rules from small models [55] from short training duration [39]. we design scaling strategies by exhaustively training models across a variety of scales for the full training duration (e.g. 350 epochs instead of 10 epochs). In doing so, we uncover strong dependencies between the best performing scaling strategy and the training regime (e.g. number of epochs, model size, dataset size). These dependencies optimal scaling decisions. Our analysis leads to new scaling strategies summarized as (1) scale the model depth when overfitting can occur (scaling the width is preferable otherwise) and (2) scale the image resolution more slowly than prior works [55].
missed in any of these smaller regimes, leading to sub-
Using the improved training and scaling strategies, design family of re-scaled ResNets, ResNer- RS, across model various scales (Figure 1). ResNet-RS models use less memory during training and are 1.7x 2.7x faster on TPUs (2.1x 3.3x faster GPUs) than the popular EfficientNets on the speed-accuracy Pareto curve. In a large-scale semi-supervised learning setup, ResNet-RS obtains 4.7x training speed-up TPUs (5.5x GPUs) over EfficientNet-B5. when co-trained on ImageNet [30] and an additional 1 130M pseudo-labeled images.
Finally, we conclude with suite of experiments testing the generality of the improved training and scaling strategies. We first demonstrate that our scaling strategy improves the speed-accuracy Pareto curve of EfficientNet. Next, we show that the improved training strategies yield representations that rival or outperform those from self-supervised algorithms (SimCLR and SimCLRv2 [4, 5)) on a suite of downstream tasks. The improved training strategies also extend to video classification, yielding an improvement from 73.4% to 77.4% (+4.0%) on the Kinetics-400 dataset.
Through combining lightweight architectural changes (used since 2018) 1) and improved training and scaling strategies, we discover the ResNet architecture sets a -fate-of-the-ar baseline for vision research. This finding highlights the importance of teasing apart each of these factors in order to understand what architectures perform better than others. We summarize our contributions:
An empirical study of regularization techniques and their interplay, which leads to training strategy that achieves strong performance (e.g. +3.2% top-1 ImageNet accuracy, +4.0% top-1 Kinetics-400 accuracy) without having change the model architecture.

---

An empirical study of scaling which uncovers strong dependencies between training and the best performing scaling strategy. We propose a simple scaling strategy: (1) scale depth when overftting can occur (scaling width can be preferable otherwise) and (2) scale the image resolution more slowly than prior works [55]. This scaling strategy improves the speed-accuracy Pareto curve of both ResNets and EfficientNets.
ResNet-RS: a Pareto curve of ResNet architectures that are 1.7x 2.7x faster than Efficient on TPUs (2.1x- 3.3x on GPUs) by applying the training and scaling strategies. Semi-supervised training of ResNet-RS with an additional 130M pseudo-labeled images achieves 86.2% top-1 ImageNet accuracy, while being 4.7x faster on TPUs (5.5x on GPUs) than the corresponding ElficientNet-NNoiyySuueenn [57].
Empirically show that representations obtained from supervised learning using modern train- ing techniques rival outperform state-of-the art self--uperrvsed representations (SimCLR [4], SimCLRv2 [5]) on suite of downstream computer vision tasks.
2 Characterizing Improvements on ImageNet
Since the breakthrough of AlexNet [30] on ImageNet [45], wide variety of improvements have been proposed to further advance image recognition performance. These improvements broadly arise along four orthogonal axes: (a) architecture, (b) training/regularizationn methodology, (c) scaling strategy and (d) using additional training data.
(a) Architecture. The works that perhaps receive the most attention are novel architectures. Notable proposals since AlexNet include VGG [49], ResNet [13], Inception [52, 53), and ResNeXt [58]. Automated search strategies for designing architectures have further pushed the state-of-the-ard [67, 41, 55J. There have also been efforts in going beyond standard ConvNets for image classification, by adapting self-attention [56] to the visual domain [2, 40, 20, 47, 8, 1J.
(b) Training and Regularization Methods. ImageNet progress has simultancously been boosted by innovations in training (e.g. improved learning rate schedules [34, 12]) and regularization methods, such as dropout [50], label smoothing [53], stochastic depth [22], dropblock [11] and data augmenta- tion [61, 59, 6, 7]. Regularization methods have become especially useful to prevent overfitting when training verreecceyyy larger models [23] on limited data (e.g. 1.2M ImageNet images).
(c) Scaling Strategies. Increasing the model dimensions (width, depth and resolution) has been another successful axis to improve quality [44, 17]. ResNet architectures typically scaled up by adding layers (depth): ResNets-18 to ResNet-200 and beyond [14, 62]. Wide ResNets [60] and MobileNets [19] instead scale the width. Increasing image resolutions consistently improves performance: EfficientNet uses 600 image resolutions [55] while both ResNeSt [62] and TResNet [43] use 400+ image resolutions for their largest model. In an attempt to systematize these heuristics. EfficientNet proposed the compound scaling rule, which jointly scales network depth, width and image resolution using a constant scaling factor. However, Section 7.1 shows this scaling strategy is sub-optimal for not only ResNets, but EfficientNets as well.
(d) Additional Training Data. Finally, ImageNet accuracy is commonly improved by training on additional sources of data (either labeled, weakly labeled, or unlaheled). Pre-training on large-scale datasets [51, 35, 27] has significantly pushed the state-of-the-arr, with VIT [8] and NFNets [3] recently achieving 88.6% and 89.2% ImageNet accuracy respectively. Using pseudo-labels on additional unlabeled images [57, 37] in a semi-supervised learning fashion has also been a fruitful avenue for improving accuracy. We present semi-supervised learning results in Section 7.2.
3 Related Work
Improved training methods combined with architectural changes to ResNets have routinely yielded competitive ImageNet performance [15, 31, 43, 62, 1, 3]. [15] achieved 79.2% top-1 ImageNet accuracy (a +3% improvement over their ResNet-50 baseline) by modifying the stem and downsam- pling block and using label smoothing and mixup. [31] further improved the ResNet-50 model with additional architectural modifications such as Squeeze-and- -nd--axtttioiionn [21], selective kernel [32], and anti-alias downsampling [63], while also using label smoothing, mixup, and dropblock to achieve 81.4% accuracy. [43, 62] incorporate several architectural modifications to the ResNet architectures
3

---

along with improved training methodologies to outperform EfficientNet models on the speed- Pareto curve on GPUs. Many prior works do remark the importance of improved training and regular- ization methods. However experiments are still largely concerned with architectural changes and the simultaneous introduction of improved training techniques can make hard to identify where the gains come from!.
Additionally, due the ever-increasing performance of machine learning accelerators, newer ar- chitectures are routinely pushed to much larger scales than the original ResNets [13]. As a result, works that propose novel architectures do not (cannot) compare against properly trained and scaled ResNets (since such a baseline did not exist), making it challenging to evaluate the significance of the proposed architectural changes compared to simple ResNets.
Lastly, prior work often puts little emphasis on studying scaling strategies or advocates for scaling strategies which we find to be sub-optimal. For example, the largest models in EfficientNet [55], TResNet [43] and ResNeSt [62] use 600, 448 and 416 image sizes respectively, which our scaling analysis reveals to be excessively large. RegNet [39] advocates for width scaling, which we find only works well when overfitting does not occur (e.g. 10 epochs).
In contrast to other works, we only consider lightweight architectural changes (that are widely used since 2018) and keep the architecture fixed. Instead, we focus exclusively on training and scaling strategies to build a Pareto curve of models. Perhaps surprisingly, we find that doing so suffices to outperform models that were introduced after ResNets: our improved training and scaling methods lead to ResNets that significantly faster than EfficientNets on TPUs GPUs (see Section 7.1). We note that our scaling improvements are sometimes orthogonal to the architectural innovations introduced in prior works in which case expect them be additive.
Methodology
Architecture. Our work studies the ResNet architecture, with two widely used architecture changes, the ResNet-D [15] modification and gqueeze-and-Excitatie (SE) in all bottleneck blocks [21]. These architectural changes are used in used many architectures, including TResNet, ResNeSt and EfficientNets. The exact details of our architecture can be found in Appendix we sometimes use the original ResNet implementation without SE (referred to as ResNet) to compare different training methods. Clear denotations are made in table captions when this is the case.
In our experiments,
Regularization and Data Augmentation. We apply weight decay, label smoothing, dropout and stochastic depth for regularization. Dropout [50] is common technique used in computer vision and we apply it to the output after the global average pooling occurs in the final layer. Stochastic depth [22] drops out each layer in the network (that has residual connections around it) with a specified probability that is a function of the layer depth. We use Rand RandAugment [7] data augmentation as an additional regularizer. RandAugment applies sequence of random image transformations (e.g. translate, shear, color distortions) to each image independently during training. Our training method closely matches that of EfficientNet. where we train for 350 epochs, but with few small differences (e.g. we use Momentum with cosine learning rate schedule as opposed to RMSProp with exponential decay). See Appendix D for details.
To select the hyperparameters for the various regularization and training methods, we use a held-out validation set comprising 2% of the ImageNet training set (20 shards out of 1024). This is referred to as the minival -set and the original ImageNet validation set (the one reported in most prior works) is referred as va 1 didation set. Unless specified otherwise, results are reported on the va validation ceatiiionsett set. The hyperparameters of all ResNet-RS models are in Table in the Appendix C.
Hyperparameter Tuning.
'A notable exception is RegNet [39] which purposely makes use of improved training techniques and shows improvements over worsened EfficientNet baselines, but does demonstrate ImageNet accuracies above (a rather low) 81%. While this approach facilitates fair comparisons with prior work, it unclear whether improvements are sustained at larger scales with improved training setups. For example, our scaling analysis shows that the scaling strategy advocated by RegNet does not generalize to training regimes where overfitting occur.

---

Improved Training Methods
5.1 Additive Study Improvements
We present an additive study of training, regularization methods and architectural changes in Table 1 (left). The baseline ResNet-200 gets 79.0% top-1 accuracy. We improve its performance to 82.2% (+3.2%) through improved training methods alone without any architectural changes. Adding two common and simple architectural changes (Squeeze-and und-Excitation and ResNet-D) further boosts the performance 83,4%. Training methods alone cause 3/4 of the total improvement, which demonstrates their critical impact on ImageNet performance.
Improvements
Top-1
ResNet-200 256x256
79.0
Model
Regularization
Weight Decay
Cosine LR Decay
79.3
+0.3
le-4
4e-5
+ Increase training epochs
78.8
-0.5
+ EMA of weights
79.1
+0.3
ResNet-50
None
79.7
78.7 (-1.0)
Label Smoothing
80.4
+1.3
ResNet-500
RA-LS
82.4
82.3 (-0.1)
+ Stochastic Depth
80.6
+0.2
ResNet-50
RA-LS-DO
82.2
82.7 (+0.5)
+ RandAugment
81.0
+0.4
ResNet-200
None
82.5
81.7 (-0.8)
Dropout FC
80.7
-0.3
ResNet-220
RA-LS
85.2
84.9 (-0.3)
+ Decrease weight decay
82.2
+1.5
ResNet-200
RA-LS-SD-DO
85.3
85.5 (+0.2)
+ Sencere-and-Fxxiinee
82,9
+0.7
+ ResNet-D
83.4
+0.5
Table 1: (Left) Additive study of training regularization and architecture improvements. The
baseline ResNet-200 is trained at resolution 256 for the standard 90 epochs using a stepwise learning rate decay schedule. All numbers are reported on the ImageNet val salidatio on-set and averaged over 2 runs. Increasing training duration 350 epochs only becomes useful once the regularization methods used, otherwise the accuracy drops due over-fitting. weight decay. (Right) Decreasing weight decay improves performance when combining regularization methods such as dropout (DO), stochastic depth (SD), label smoothing (LSS and RandAugment (RA). Image resolution is x 224 for ResNet-50 and 256x 256 for ResNet-200. All numbers reported on the ImageNet minival- vaee- from average of two
dropout hurts as we have not yet decreased
5.2 Importance of decreasing weight decay when combining regularization methods
Table (right) highlights the importance of changing weight decay when combining regularization methods together. When applying RandAugment and label smoothing, there is no need to change the default weight decay of lc-4. But when we further add dropout and/or stochastic depth, the performance can decrease unless we further decrease the weight decay. The intuition is that since weight decay acts as regularizer, its value must be decreased order not overly regularize the model when combining many techniques. Furthermore, [65] presents evidence that the addition of data augmentation shrinks the L.2 norm of the weights, which renders some of the effects of weight decay redundant. Other works use smaller weight decay values, but do not point out the significance of the effect when using more regularization [54, 55].
Improved Scaling Strategies
The prior section demonstrates the significant impact of training methodology and now show the scaling strategy is similarly important. In order to establish scaling trends, we perform an
extensive search on ImageNet over width multipliers in [0. 25, 5, 1.500.000 0, 0, 2. 0], depths of [26, 50, 1C1, 200, 300, 350, 400] and resolutions of (128, 160, 224, 320, 448], We train these architectures for 350 epochs, mimicking the training setup of state-of-the-art ImageNet models, and increase regularization with model size in an effort to limit overfitting. See Appendix F for regularization and model hyperparameters.
5

---

FLOPs do not accurately predict perfor- mance in the bounded data regime. Prior works on scaling laws observe a power law between error and FLOPss in unbounded data regimes [25, 16]. In order to test whether this also holds in our scenario, we plot ImageNet er- for against FLOPs for all scaling configurations in Figure 2.
Scaling Properties of ResNets
160
ImageNet Error
224
310
440
For the smaller models, we observe an overall power law trend between error and FLOPs, with minor dependency on the scaling configuration (i.e. depth, width and image resolution). How- ever, the trend breaks for larger model sizes and we observe a large variation ImageNet perfor- mance for a fixed amount of FLOPs, especially in the higher FLOP regime. Therefore the ex- act scaling configuration (i.e. depth, width and image resolution) can have a big impact on per- formance even when controlling for the same amount of FLOPs.
10F
1011
FLOPs
Figure 2: Scaling properties ResNets across vary- ing model scales. Error approximately scales as a power law with FLOPs (lincar fit on the log-log curve) in the lower FLOPs regime but the trend breaks for larger FLOPs. We observe diminishing returns of scaling the image resolutions beyond x20x320, which motivates the slow image resolution scaling (Strategy #2). All results the ImageNet minival-set.
The best performing scaling strategy depends on the training regime. We next look directly at latencies the hardware of interest identify scaling strategies that improve the speed-accuracy Pareto curve. Figure 3 presents accuracies and latencies of models scaled with either width or depth across four image resolutions and three different training regimes (10, 100 and 350 epochs). We observe that the best performing scaling strategy, especially whether to scale depth and/or width, highly depends on the training regime.
6.1 Strategy #1 Depth Scaling in Regimes Where Overfitting Can Occur
Depth scaling outperforms width scaling for longer epoch regimes. In the 350 epochs setup (Figure 3 right), we observe depth scaling to significantly outperform width scaling across all image resolutions. Scaling the width is subject overfitting and sometimes hurts performance even with increased regularization. We hypothesize that this due to the larger increase in parameters when scaling the width. The ResNet architecture maintains constant FLOPs across all block groups and multiplies the number of parameters by 4x every block group. Scaling the depth, especially in the earlier layers, therefore introduces fewer parameters compared to scaling the width.
Width scaling outperforms depth scaling for shorter epoch regimes. In contrast, width scaling is better when only training for 10 epochs (Figure 3 3- left). For 100 epochs (Figure 3 middle), the best performing scaling strategy varies between depth scaling and width scaling, depending on the image resolution. The dependency of the scaling strategy on the training regime reveals a pitfall of extrapolating scaling rules. We point out that prior works also choose scale the width when training for a small number of epochs on large-scale datasets (e.g. <00 epochs on 300M images), consistent with our experimental findings that scaling the width is preferable shorter epoch regimes. In particular, [27] train a ResNet- 1.2 with 4x filter multiplier while [3] scales the width with ~ 1.5xx filter multiplier.
6.2 Strategy #2 Slow Image Resolution Scaling
In Figure we also observe that larger image resolutions yield diminishing returns. We therefore propose to increase the image resolution more gradually than previous works. This contrasts with the compound scaling rule proposed by EfficientNet which leads very large images (e.g. 600 for EfficientNet-BB B7, 800 for EfficientNet-22 [57]). Other works such as ResNeSt [62] and TResNet [43]) scale the image resolution up to 400+ Our experiments indicate that slower image scaling improves not only ResNet architectures, but also EfficientNets on speed-accuracy basis (Section 7.1).
FLOPPs is not good indicator of latency on modemn hardware. See Section 7.1 for more detailed discussion.

---

Epoche:
Epochi: 100
Epochs 360
Imgeller Focurloy
Top Iruga Requning
Tup- Imsqe Accurncy
Poee 201
Neo
Rize 224
Rea: 00
Rex
Rax
Rec: 23
Widh Scaling
Dooth Sealing
Deptrs Sosling Wilth Scoling
Nox: 122
Whtth Scaling
Depih Seal
225
0.3
030
Time Per Trikiing Step (Sec))
Time Per Triniig Step (feer)
Time Por Traning Stap (Sw0)
Figure Scaling of ResNets across depth, width, image resolution and training epochs. We com- pare depth scaling and width scaling across four different image resolutions [128, 1 50, 224, 320] when training models for 10, 100 or 350 epochs. We find that the best performing seating strategy depends on the training regime, which reveals the pitfall of extrapolating scaling rules from small scale regimes. (Left) 10 Epoch Regime: width scaling the best strategy for the speed-accuracy Pareto curve. (Middle) 100 Epoch Regime: depth scaling is sometimes outperformed by width scaling. (Right) 350 Epoch Regime: depth scaling consistently outperforms width scaling by a large margin. Overfitting remains an issue even when using regularization methods. Model Details: All models start from depth of 101 and are increased through [101, 200, 300, 400 All model widths start with multiplier of 1 0x and are increased through 11.0,1.5, 2. For all models, we tune regularization in an effort to limit overfitting (see Appendix F). Accuracies are reported on the ImageNet mi nivalll--- -set and training times are measured on TPUs.
6.3 Designing Scaling Strategies
Our scaling analysis surfaces two common pitfalls in prior research on scaling strategies.
Pitfall #1: Extrapolating scaling strategies from small-scale regimes. Scaling strategies found in small scale regimes (e.g. on small models or with few training epochs) can fail to generalize to larger models or longer training iterations. The dependencies between the best performing scaling strategy and the training regime are missed by prior works which extrapolate scaling rules from either small models [55] or shorter training epochs [39]. We therefore do not recommend generating scaling rules exclusively in a small scale regime because these rules can break down.
Pitfall #2: Extrapolating scaling strategies from single and potentially sub-optimal initial architecture. Beginning from a sub-optimal initial architecture can skew the scaling results. For example, the compound scaling rule derived from a small grid search around EfficientNet-BO which was obtained by architecture search using a fixed FLOPs budget and a specific image resolution. However, since this image resolution can be sub-optimal for that FLOPs budget, the resulting scaling strategy can be sub-optimal. In contrast, our work designs scaling strategies by training models across variety of widths, depths and image resolutions.
Summary of Improved Scaling Strategies. For image classification, the scaling strategies are summarized as (1) scale the depth in regimes where overfitting can occur (scaling the width is preferable otherwise) and (2) slow image resolution scaling. Experiments indicate that applying these scaling strategies to ResNets (ResNet-RS) and EfficientNets (EfficientNet-RS) leads to significant speed-ups over EfficientNets. We note that similar scaling strategies are also employed in recent works that obtain large speed-ups over EfficientNets such as Nests [1] and NFNets [3]. For a new task, we recommend running a small subset of models across different scales, for the full training epochs, gain intuition which dimensions the most useful across model scales. While this approach may appear more costly, we point out that the cost is offset by not searching for the architecture.
7 Experiments with Improved Training and Scaling Strategies
7.1 ResNet-RS on Speed-Accuracy Basis
Using the improved training and scaling strategies, we design ResNer-RS, a family of re-scaled ResNets across a wide range of model scales (see Appendix C and E. for experimental and architectural details). Figure and Table 2 compare EfficientNets against Re on a speed-accuracy Pareto curve. We find that ResNet-RS match EfficcientNes' performance while being 1.7x 2.7x faster on TPUs (2.1x 3.3x faster on GPUs). We point that these speed-ups are superior to those obtained by TResNest and ResNeSt', suggesting that Re also outperform TResNet and ResNeSt.
TeReNeet and ResNeSt report ~133 2.0x speed-ups ups over EfficientNet GPU V100.
7

---

Speed-Accurane Accuracy Pareto Curve
420-320
ResNat-FRS
8E0
320
Efficientett 270-256
330
206
2.7x Spendup
Top-1 ImageNet Accuracy
200-256
Speedup
B5
152-236
83
Model
RS-350
ENet-B6
RS-420
ENet-B7
B3
Resolution
256
528
320
600
81-
Top-1
Params (M)
164
43 (3800
192
66 (2.9)
FLOPs (B)
69
(1.80)
128
74 (1.7g)
80
B2
Memory
(2.3g)
28.3
TPU-x3 Lat (s)
1.1
3.0 (2.7g)
2.1
6.0 (2.9k
B1
(s)
(3.3g)
(2.80)
0.0
0.5
1.0
1.5 Time Per Training Step (Sec)
2.0
2,5
3.0
Figure 4: Speed-Accuracy Pareto curve comparing ResNets-RS EfficientNet. ResNet-RS (annotated with depth- image resolution) are 1.7x 2.7x faster than the popular EfficientNets when closely matching their training setup. Although ResNet-RS has more parameters and FLOPs, the model employs less memory and runs faster TPUs and GPUs. See Appendix C and I for results and profiling details.
Model
Image Resolution
Params (M)
FLOPs (B)
V100 Latency (s)
TPUv3 Latency (ms)
Top-1
EfficientNet- BO
224
5.3
0.8
0.47
77.1
EfficientNet-BB B1
240
7.8
1.4
0.82
150
79.1
ResNet-RSS50
160
36
4.6
0.31
70
78.8
EfficienttettB3
260
9.2
2.0
1,03
210
80.1
ResNet-RS-101
160
64
8.4
0.48 (2.1x)
120 (1.8x)
80.3
EfficientNet-B3
300
12
3.6
1.76
340
81.6
ResNec-RS-101
192
64
12
0.70
170
81.2
ResNet-RS-152
192
87
18
0.99
240
82.0
EfficientNet-B4
380
19
8.4
4.0
710
82.9
ResNet-RS-152
224
87
24
1.48 (2.7x)
320 (2.2x)
82.8
ResNet-RS-152
256
87
31
1.76 (2.3x)
410 (1.7x)
83.0
EfficientNet-55
456
30
20
8.16
1510
83.7
ResNet-RS-200
256
93
40
2.86
83.4
ResNet-RS-270
256
130
54
3.76 (2.2x)
780 (1.9x)
83.8
EfficientNet-B6
528
43
38
15.7
3010
84.0
ResNet-RS-350
256
164
69
4.72 (3.3x)
1100 (2.7 x)
84.0
EfficientNet-B7
600
66
74
29.9
6020
84.7
ResNet-RS-350
320
164
107
8.48
1630
84.2
ResNet-RS-420
320
192
128
10.16
2090
84.4
Table 2: Details of ResNet-RS models in Pareto curve. See Table for hyperparameters and Section for profiling details.
This large speed-up over EfficientNet may be non-intuitive since EfficientNets have significantly reduced parameters and FLOPs compared to ResNets. We next discuss why model with fewer parameters and fewer FLOPs (EfficientNet) slower and more memory-intensive during training.
FLOPs Lateney. While FLOPs provide a hardware-agnostic metric for assessing computational demand, they may not be indicative of actual latency times for training and inference [19, 18, 39]. In custom hardware architectures (e.g. TPUs and GPUs), FLOPs are an especially poor proxy because operations are often bounded by memory access costs and have different levels of optimization on modern matrix multiplication units [24]. The inverted bottlenecks [46] used in EfficientNets employ depthwise convolutions with large activations and have small compute to memory ratio (operational intensity) compared to the ResNet's bottleneck blocks which employ dense convolutions on smaller activations. This makes EfficientNets less efficient modern accelerators compared to ResNets.

---

Figure 4 (table on the right) illustrates this point: a ResNet-RS model with 1.8x more FLOPs than EfficientNet-B6 is 2.7x faster on a TPUv3 hardware accelerator.
Parameters Memory. Parameter count does not necessarily dictate memory consumption during training because memory is often dominated by the size of the activations". The large activations used in EfficientNets also cause larger memory consumption, which is exacerbated by the use of large image resolutions, compared to our re-scaled ResNets. A ResNet-RS model with 3.8x more parameters than EfficientNet-BB B6 consumes 2.3x less memory for a similar ImageNet accuracy (Table in Figure 4). We emphasize that both memory consumption and latency are tightly coupled to the software and hardware stack (TensorFlow on TPUv3) due to compiler optimizations such operation layout assignments and memory padding.
The scaling analysis from Section 6 reveals that scaling the image resolution results in diminishing returns. This suggests that the compound scaling rule advocated in EfficientNet which jointly increases model depth, width and resolution at a constant rate is sub-optimal. To test this hypothesis, we apply the slow image resolution scaling strategy (Strategy #2) to EfficientNets and train several versions with reduced image resolutions, without changing the width or depth. Figure 5 (Appendix) demonstrates a marked improvement of the re-scaled EfficientNets (EfficientNet-RSS) on the speed-accuracy Pareto curve over the original EfficientNets.
Improving scaling of EfficientNets
7.2 Semi-Supervised Learning with ResNet-RS
We next measure how ResNet-RS performs as we scale to larger datasets large scale semi- supervised learning setup. We train ResNets-RS on the combination of 1.3M labeled Ima- geNet images and 130M pseudo-labeled images, in a similar fashion to Noisy Student [57]. We use the same dataset of 130M images pseudo-labeled as Noisy Student, where the pseudo labels are generated from an EfficientNet-L.2 model with 88.4% ImageNet accuracy. Models jointly trained both the la- beled and pseudo-labeled data and training hyperaarameters are kept the same. Ta- ble 3 reveals that ResNet-RS models are very strong in the semi-supervised learning setup as well, achieving a strong 86.2% top-1 Im- ageNet accuracy while being 4.7x faster on TPU (5.5x on GPU) than the corresponding EfficientNet model.
Model
V100 (s)
TPUv3 (ms) Top-1
EfficientNet-55 ResNet-RS-152
8.16
1510
86.1
1.48 (5.5x)
320 (4.7x)
86.2
Table 3: ResNet-RS are efficient semi-supervised learn- ers. ResNet-RS- 152 with image resolution 224 is 4.7x faster on TPU (5.5x GPU) than EfficientNet-B5 Noisy Student for similar ImageNet accuracy.
7.3 Transfer Learning to Downstream Tasks with ResNet-RS
We now investigate whether the improved supervised training strategies yield better representations for transfer learning and compare them with self-supervised learning algorithms. Recent self- supervised learning algorithms claim to surpass the transfer learning performance of supervised learning and create more universal representations [4, 5]. Self-supervised algorithms, however, make several changes to the training methods (e.g training for more epochs, data augmentation) making comparisons to supervised learning difficult.
Fairly comparing supervised learning and self-supervised learning, In an effort to closely match SimCLR's 's training setup and provide fair comparisons, we restrict the RS training strategies to a subset of its original methods. Specifically, we train for for 400 epochs with cosine learning rate decay, data augmentation (RandAugment), label smoothing, dropout and decreased weight decay but do not use stochastic depth or exponential moving average (EMA) of the weights. We choose this subset to closely match the training setup of SimCLR: longer training (800 epochs) with cosine learning rate decay, a tailored data augmentation strategy, a tuned temperature parameter in the contrastive loss and tuned weight decay.
Table 4 compares the transfer performance of supervised learning with or without improved train- ing strategies (respectively denoted RS and Supervised) against SimCLR/SimCLRv2 [4, 5] on five downstream tasks; CIFAR- 100 Classification [29], Pascal Detection & Segmentation [9], ADE Segmentation [64] and NYU Depth [48]. Our experiments demonstrate that the improved training
'Activations are typically stored during training as they are used in backpropagation. At inference, activations be discarded and parameter count is better proxy for actual memory consumption.

---

Model
Training Method
Epochs
CIFAR-I00 Accuracy
Pascal Detection
Pascal Segmentation
ADE Segmentation
NYU
Depth
ResNet-152
Supervised
90
85.5
80.0
70.0
40.2
81.2
ResNet-152
SimCLR
800
87.1
83.3
72.2
41.0
83.5
ResNet-152
SimCLRv2
800
84.7
79.1
73.1
41.1
84.7
ResNet-155 152
RS
400
88.1
82.2
78.2
42.2
83.4
Table 4: Representations from supervised learning with improved training strategies rival outper- form representations from state-of-the-art self-supervised learning algorithms. Comparison of supervised training methods (supervised, RS) and self supervised methods (SimCLR, SimCLRv2) on a variety of down- stream tasks. The improved training strategies (RS) greatly outperforms the baseline supervised training, which highlights the importance of using improved supervised training techniques when comparing self-supervised learning algorithms. models employ the vanilla ResNet architecture and are pre-trained on ImageNet.
strategies significantly improve transfer performance, in line with works that observe that higher Ima- geNet accuracy strongly correlates with improved transfer learning performance [28]. Furthermore, find that the improved supervised representations (RS) rival outperform SimCLR/SimCLRy2 even when restricted to a smaller subset. These results challenge the notion that self-supervised universal representations than supervised learning when labels are available.
algorithms lead
7.4 Revised 3D ResNet for Video Classification
Improvements
Top-1
We conclude by applying the training strategies to the Kinetics-400 video classification task [26], using
3D ResNet-50
73.4
Dropout on FC
74.4
+1.0
Label smoothing
74.9
+0.5
3D ResNet as the baseline architecture [38]. Table presents additive study of the RS training recipe and architectural improvements. The training strategies extend to video classification, yielding combined im- provement from 73.4% to 77.4% (+4.0%). The ResNet- and Squeeze-ande andezzeendd--ae and-Exxitation architectural changes further improve the performance to 78.2% (+0.8%). Similarly to our study on image classification (Table 1), we find that most of the improvement can be obtained without architectural changes.
Stochastic depth
76.1
+1.2
EMA weights
76.1
Decrease weight decay
76.3
+0.2
Increase training epochs
76.4
+0.1
Scale jittering
77.4
+1.0
Squuezz-and-Exxiiaatin Excitation
77.9
+0.5
ResNettD
78.2
+0.3
Table 5: Additive study of regularization
training and architecture improvements with 3D-ResNet on video classification.
Conclusion
By updating the de facto vision baseline with modern training methods and an improved scaling strategy, we have revealed the remarkable durability of the ResNet architecture. Simple architectures set strong baselines for state-of-the-art methods: the accuracy gains that motivate complicated architectural changes may be surpassed with thoughtful scaling and training strategies. Our work suggests that the field has myopically overemphasized architectural innovations at the expense of experimental diligence, and we hope it encourages further scrutiny in maintaining consistent methodology for both proposed innovations and baselines alike. We do not foresee any negative societal impact of our work. We include further discussion in the Appendix B.
Acknowledgements. We would like thank Ashish Vaswani, Prajit Ramachandran, Ting Chen, Thang Luong, Hanxiao Liu, Gabriel Bender, Quoe Le, Neil Houlsby, Mingxing Tan, Andrew Howard, Raphael Gontijo Lopes, Andy Brock and David Berthelot for helpful feedback on this work; Jing Li, Pengchong Jin, Yeqing Li and Yin Cui for the support on open-sourcing and infrastructure.
10

---

References
[1] I. Bello. Lambdanetworks: Modeling long-range interactions without attention. International Conference in Learning Representations, 2021.
[2] I. Bello, B. Zoph, A. Vaswani, J. Shlens, and Q. L. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3286-3295, 2019.
[3] A. Brock, S. De, S. L. Smith, and K. Simonyan. High- per--performnce largo-scale image recognition without normalization, arXiv preprint arXiv: 2102.06171, 2021.
[4] T. Chen, S. Korbllihh, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual
representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
[5] T. Chen, S. Kornblith, K. Swersky, M. Norouzi, and G. Hinton. Big self-supervised models are strong sumi-supervised learners. arXiv preprint arXiv:2006.1000 2020.
[6] E. D. Cubuk, B. Zoph, D. Mane. V. Vasudevan, and Q. V. Le. Autoaugment: Learning augmentation policies from data, arXiv preprint arXiv: 1805.0950, 2018.
[7] E. D. Cubuk, B. Zoph. J. Shiens, and Q V. Le. Randaugment: Practical automated data augmentation with reduced search space. arXiv preprint arXiv: 13719, 2019.
[8] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 19299, 2020.
[9] M. Everingham, L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88(2):303-338, 2010.
[10] C. Feichtenbofer, H. Fan, J. Malik, and K. He. Slowfast networks for video recognition. 2019 JEEE/CVF International Conference Computer Vision (ICCV), pages
2019.
[11] G. Ghiasi, T.-Y. Y. Lin, and Q V. Le. Dropblock: A regularization method for convolutional networks. In Advances in Neural Information Processing Systems, pages 10727-10773 2018.
[12] P. Goyal, P. Dollár, R. Girshick, P. Noordhuis, L. Wesolowski, A. Kyrola, A. Tulloch, Y. Jia, and K. He. Accurate, large minibatch sgd: Training imagenet in 1 hour. arXiv preprint arXiv: 130fv:7006.022 026777 2017.
[13] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. arXiv preprint arXiv: 1512.0338, 2015.
[14] K. He, X. Zhang, S. Ren, and J. Sun. Identity mappings in deep residual networks. arXiv preprint arXiv: 1603.05027, 2016.
[15] T. He, Z. Zhang, Н. Zhang, Z. Zhang. J. Xie, and M. Li. Bag of tricks for image classification with 2018.
convolutional neural networks. arXiv preprint
[16] T. Henighan, J. Kaplan, M. Katz, M. Chen, C. Hesse, J. Jackson, H. Jun, T. B. Brown, P. Dhariwal, S. Gray, et al. Scaling laws for autoregressive generative modeling. arXiv preprint 2Xi::2010.14701, 2020.
[17] J. Hestness, S. Narang, N. Ardalani, G. Diamos, H. Jun, H. Kianinejad, M. M. A. Patwary, Y. Yang, and Y. Zhou. Deep learning scaling is predictable, empirically. arXiv preprint arXiv: 71499 2017.
[18] A. Howard, M. Sandler, G. Chu, L.-C. Chen, B. Chen, M. Tan, W. Wang, Y. Zhu, R. Pang, Vasudevan, et al. Scarching for mobilenctv3. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1314 13244, 2019.
[19] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mohilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint 170vv:170.004611, 2017.
[20] H. Hu, Z. Zhang. Z. Xie, and S. Lin. Local relation networks for image recognition. arXiv preprint arXiv: 1 190..1904.1111 2019.
[21] J. Hu, L. Shen, and G. Sun. Squeeze-and-excitaeee networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[22] G. Huang, Y. Sun, Z. Liu, D. Sedra, and K. Q. Weinberger. Deep networks with stochastic depth. In European conference computer vision, pages 646 6611 Springer, 2016.
11

---

[23] Y. Huang, Y. Cheng, A. Bapna, O. Firat, M. X. Chen, D. Chen, H. Lee, J. Ngiam, Q. V. Le, Y. Wu. et al. Gpipe: Efficient training of giant neural networks using pipeline parallelism. arXiv preprint P.Xiv:881.0666.. 2018.
[24] N. Jouppi. Young. N. Patil, Patterson, G. Agrawal, Bajwa, Bates. S. Bhatia, N. Boden, A. Borchers, et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th annual infernational symposium on computer architecture, pages 1-12, 2017.
[25] J. Kaplan, S. McCandlish, T. Henighan, T. B. Brown, B. Chess, R. Child, S. Gray, A. Radford, J. Wu, and
D. Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.088 003361, 2020.
[26] W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green,
T. Back, A. Natsev, M. Suleyman, and A. Zisserman, The kinetics human action video dataset. ArXiv, 0tss/7705.069555 2017.
[27] A. Kolesnikov, L. Beyer, X. Zhai, J. Puigcerver. J. Yung. S. Gelly, and Houlsby. Big transfer (bit): General visual representation learning. arXiv preprint 191iv:1912.11370, 6(2),8, 2019.
[28] S. Kornblith, J. Shlens, and Q V. Le. Do better imagenet models transfer better? In Proceedings the the JEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 2661-2671, 2019.
[29] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[30] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in neural information processing systems, pages 1097-1105, 2012.
[31] J. Lee, T. Won, and K. Hong, Compounding the performance improvements of assembled techniques in convolutional neural network. arXiv preprint arXiv:2001.0660 066268, 2020.
[32] X. Li, W. Wang, X. Hu, and J. Yang. Selective kernel networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 310-519, 2019.
[33] T.-Y. Lin, P. Dollár, Girshick, K. He, Hariharan, and S. Belongie. Feature pyramid networks for object detection. In CVPR, 2017.
[34] I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint vrXiv:1608.03945 2016.
[35] D. Mahajan, R. Girshick, V. Ramanathan, K. Hc, M. Paluri, Li, A. Bharambe, and L. Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings the European Conference Computer Vision (ECCV), pages 181-196, 2018
[36] T. Norrie, N. Patil, D. H. Yoon, G. Kurian, S. Li, J. Laudon, C. Young. N. Jouppi, and D. Patterson. The design process for google's training chips: Tpuv2 and tpuv3. IEEE Micro, 2021.
[37] Pham. Q. Xie, Z. Dai, Q. V. Le. Meta pseudo labels. arXiv preprint arXiv:2003.
2020.
[38] R. Qian, Meng, Gong, M.-H. Yang, H. Wang. J. Belongie, and Y. Cui. Spatiotemporal contrastive video representation learning. arXiv: Computer Vision and Pattern Recognition, 2020.
[39] Radosavovic, R. P. Kosaraju, R. Girshick, K. He, and P. Dollár. Designing network design spaces. arXiv preprint arviv: 2003.13678, 2020.
[40] P. Ramachandran, N. Parmar, A. Vaswani. I. Bello, A. Levskaya, and J. Shlens. Stand-alnee aelf-attention in vision models. arXiv preprint arXiv: h 1906005999 2019.
[41] E. Real, A. Aggarwal, Y. Huang, and Q. Le. Regularized evolution for image classifier architecture search. In Proceedings the aaai conference artificial intelligence, volume 33, pages 4780-4789, 2019.
[42] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv:1506.01493 2015.
[43] T. Ridnik, H. Lawen, A. Noy, and I. Friedman. Tresnet: High performance gpu-dedicated architecture. arXiv preprint arXiv: 100003/3660 /6300, 2020.
[44] J. S. Rosenfeld, A. Rosenfeld, Y. Belinkov, and N. Shavit. A constructive prediction of the generalization error across scales. arXiv preprint arXiv: 1909.12673, 2019.
12

---

[45] O. Russakovsky, J. Deng. H. Su, J. Krause, S. Satheesh, S. Ma. Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, al. Imagenet large scale visual recognition challenge. IJCV, 2015.
[46] M. Sandler, A. Howard, M. Zhu, A. Zhmoginov, and L.-C. Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[47] Z. Shen, I. Bello, R. Vemulapalli, X. Jia, and C.-H. Chen. Global self-attention networks for image recognition. arXiv preprint arXiv: 2010.03019, 2020.
[48] N. Silberman, D. Hoiem, P. Kohli, and R. Forgus. Indoor segmentation and support inference from rghd images. In European conference on computer vision, pages 746-760. Springer, 2012.
[49] K. Simonyan and A. 7isserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
[50] N. Srivastava, G. Hinton, A. Krizhevsky, Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research, 2014.
[51] C. Sun, A. Shrivastava, S. Singh, and A. Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. arXiv preprint arXiv: 1707.02968, 2017.
[52] C. Szegedy, Liu, Y. Jia, Sermanet. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[53] C. Szegedy, V. Vanhoucke, S. loffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEFE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[54] M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q V. Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820 2828, 2019.
[55] M. Tan and Q. V. Lc. Efficientmet: Rethinking model scaling for convolutional neural networks. arXiv preprint arviv: /905 / 11946, 2019.
[56] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. arXiv preprint arXiv: 1706.03762, 2017.
[57] Q. Xie, M.-T. Luong, E. Hovy, and Q. V. Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pottern Recognition, pages 10687-10698, 2020.
[58] S. Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[59] S. Yun, D. Han, S. J. Oh, S. Chun, J. Choc, and Y. Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE International Conference Computer Vision, pages 6023-6032, 2019.
[60] S. Zagoruyko and N. Komodakis. Wide residual networks. arXiv preprint arXiv:6005.07146, 2016.
[61] H. Zhang, M. Cisse, Y. N. Dauphin, and D. Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09411 2017.
[62] H. Zhang, C. Wu, Z. Zhang, Y. Zhu, Z. Zhang, H. Lin, Sun, He, J. Muller, R. Manmatha, M. Li, and A. Smola. Resnest: Split-attention networks. arXiv preprint 20Xiv:2004.0895.. 2020.
[63] R. Zhang. Making convolutional networks shift- invariant again. arXiv preprint arXiv:1904. 11486 2019.
[64] B. Zhou, H. Zhao, X. Puig. S. Fidler, A. Barriuso, and A. Torralba. Scene parsing through ade20k datasct. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 633-641, 2017.
[65] B. Zoph, E. D. Cubuk. G. Ghiasi, T.-Y. Lin, J. Shlens, and Q V. Le. Learning data augmentation strategies for object detection. In European Conference on Computer Vision, pages 566-583. Springer, 2020.
[66] B. Zoph, G. Ghiasi, T.YY Lin. Y. Cui, H. Liu, E. D. Cubuk, and Q V. Le. Rethinking pre- pre-training and self telf--trainng. arXiv preprint arXiv:2006.0666 06882, 2020.
13

---

[67] B. Zoph, V. Vasudevan. J. Shlens, and Q. V. Le. Learning transferable architectures for scalable image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 8697-8710, 2018.
14


Reviews:
[{"review_id": "pQ6f7A9Sge", "reviewer": "Reviewer_trei", "summary": "This paper presents an in-depth analysis of how improvements in training, regularization, and model architecture affect the performance and efficiency of ResNet models. Combining all the improvements, the proposed ResNet-RS models achieve significant speedup over EfficientNet (a strong baseline).", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. While the methods of training neural networks used in this paper are not novel, the authors provide a thorough and original analysis of existing methods. \n2. This paper proposes a novel model scaling strategy: 1. scaling the depth when overfitting occurs 2. scaling the image resolution slower.\n3. The writing is clear and well-written.\n4. This paper provides several interesting observations and insights.\n5. This paper shows that the choice of regularization can be affected by the number of training epochs which suggests that some models searched with small training budgets (such as RegNet) may not generalize to longer training procedures.\n6. The authors show that with improvements in training, regularization, and model architecture, simple ResNets can outperform strong EfficientNets. Further, in Figure 5, the authors show that ResNet-RS outperforms EfficientNet-RS, which suggests that the EfficientNet architecture is also worse under a fairer comparison.\n7. The authors also provide a comparison with the unsupervised method and show that the proposed model also achieves better fine-tuning results.\n8. Not limited to ImageNet, the authors also show that the proposed method can be applied to 3D ResNet for Video Classification.\n\nWeaknesses:\n1. The text in Appendix B1 is missing. I can't find any text describing Figure 5 and the comparison with EfficientNet-RS.\n2. Scaling the image resolution slower has been proposed in the fast compound scaling method [1]. While other parts are different, I wonder how the proposed scaling strategy performs when compared with the fast compound scaling in terms of performance-efficiency trade-off.\n\nQuestions:\n1. In table 4, the EMA of weights does not lead to any performance improvement. Does it also happen to the final model? If it is unnecessary, why don't you just remove it?\n2. I wonder how general the scaling strategy is. Is it only limited to vision models or it generalizes to NLP and speech? Or is it only limited to ConvNets?\nCan it also be applied to vision transformers [2] or their variants?\n\n\nOverall, this is a well-written paper and provides several new observations and insights. I view the lack of novelty in the training methods and model design as a strength of this paper instead of a weakness. Undoubtedly, I recommend accepting it. While I give it an 8 currently, I am willing to raise my rating to 9 if the authors can address the limitations mentioned below or even to 10 if my questions are addressed as well.\n\nReferences:\n[1] Dollár, Piotr, Mannat Singh, and Ross Girshick. \"Fast and accurate model scaling.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[2] Dosovitskiy, Alexey, et al. \"An image is worth 16x16 words: Transformers for image recognition at scale.\" arXiv preprint arXiv:2010.11929 (2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. The scaling rule in Appendix D Table 9 seems to be arbitrary to me which makes the scaling method hard to be applied to other architecture. In the text, the authors say that they try to keep the ratio roughly constant which contradicts what Table 9 presents: 24:36 = 1:1.5 while 29:53 ~= 1:1.8, 36:72 = 1:2, and 44:87 ~= 1:1.97. Only the last two models have a similar ratio. Moreover, I understand that the authors try to keep the small models (ResNet-50, ResNet-101, ResNet-152) the same as the prior work, but I wonder whether these choices are optimal. Would it be possible to find some small ResNets which push the Pareto curve further?\n2. The authors do not study how the improvements in three directions generalize to other model architectures and datasets. I understand that the authors focus on ResNet architecture, but the paper would be more impactful if the authors can provide a study on the impact of these methods on different model architectures. Given that EfficientNet-RS is worse than ResNet-RS, I believe the ranking of model architectures can be shuffled.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "nQX6cGI0_eb", "reviewer": "Reviewer_DbhL", "summary": "This paper revisits the performance of ResNet from three aspects, architecture, training scheme, and scaling strategy. The author shows that a better training and scaling strategy can give even more improvements than better architecture design. The author conducted many experiments to empirically show what is a good training and scaling strategy and how good it can be.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. There are very thorough experiments to validate the effectiveness of better training and scaling strategy.\n2. The concluded training and scaling strategy outperforms state-of-the-art methods on different tasks. The empirical results are very impressive.\n3. The paper is easy to follow.\n\nWeakness:\nI would say this is a good empirical study about how to better train a network. However, I don’t know if it would be interesting to the NeurIPS audience. The followings are some of my concerns:\n1. The effectiveness and interplay of different training/regularization methods have been studied before [1]. Although the author may have conducted more thorough empirical evaluations, the conclusions about this are pretty similar. There are no surprising findings here.\n2. The author pointed out the importance of scaling strategy, but the importance of scaling strategy has also been studied before [2][3]. The author further proposed two scaling strategies, but they are all based on empirical results on ResNet. There are no (or very few) analyses on these two strategies. I don’t know if they could also work well on different architectures and tasks.\nThe author claimed depth scaling is better than width scaling for longer epoch regimes because width scaling is subject to overfitting. However, in Fig.3, width scaling seems to work well in Reso-128 and Reso-160, but fails in Reso-224 and 320. But I think a smaller input should lead to more overfitting, why width scaling works well in small resolutions.\n3. The author claimed width scaling could largely increase the number of parameters and lead to overfitting, but we can control the width scaling factor (or even non-uniform width scaling) to make them have the same number of parameters as depth scaling, will the width scaling work better or as good as depth scaling if they have the same number of parameters?\n4. In practice, how can we determine the overfitting occurs, or how long can be termed as the long epoch regime to switch from width scaling to depth scaling in different architectures and tasks. If we need to do this empirical evaluation for every architecture and task, I don’t think it is practical.\nOverall, I think the results are impressive based on the evaluated architectures, but it lacks further analyses about the results. I don’t know if the proposed scaling strategies can really be useful for practitioners.\n\nReference\n\n[1] Bag of Tricks for Image Classification with Convolutional Neural Networks. CVPR 2019.\n[2] EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019.\n[3] MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution. ECCV 2020.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The author has addressed the limitations and potential negative societal impact of the work.", "review_text": "", "extra_scores": {"ethical_concerns": "No ethical concerns.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "mATkrdOrU0q", "reviewer": "Reviewer_jadX", "summary": "The paper proposes a methodology to improve ResNets like architecture by changing a little bit the architecture and using better training procedure. The rules proposed in the paper allow to have a better trade off between speed and accuracy with ResNet-like architecture and be competitive with efficientNet. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n\n- The point \"Novel architectures underlie many advances, but are often simultaneously introduced with other critical – and less publicized – changes in the details of the training methodology and hyperparameters.\" L22 is very interesting it is true that it would be beneficial for the field of image classification if it was more taken into account.\n- The paper offers a broad and instructive analysis of different elements of training and how to scale models with resolution. \n- The paper is well written and the ideas are easy to understand.\n\nWeaknesses:\n- Significance of the results: It would be interesting to also have the results on ImageNet v2 or on another dataset to see how the different results of the different analyses evolve.\n\n- Related work: The EfficientNetv2 paper[1] also proposes to make changes to the training procedure and some architectural changes to have more efficient models. It would be interesting to discuss this paper.\n\nComment:\n- \"a canonical ResNet with 79.0%\" L31 It would be interesting to specify which type of ResNet, since it is not a ResNet-50, that can be a bit confusing.\n\n[1] Le et al., EfficientNetV2: Smaller Models and Faster Training", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers unanimously recommend acceptance of the paper. The paper is well written and instructive about the importance of training and scaling strategies. It may deserve some elevated attention as a spotlight contribution.
Decision: Accept (Spotlight)