Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: This paper presents an in-depth analysis of how improvements in training, regularization, and model architecture affect the performance and efficiency of ResNet models. Combining all the improvements, the proposed ResNet-RS models achieve significant speedup over EfficientNet (a strong baseline).
main_review: Strengths:
1. While the methods of training neural networks used in this paper are not novel, the authors provide a thorough and original analysis of existing methods. 
2. This paper proposes a novel model scaling strategy: 1. scaling the depth when overfitting occurs 2. scaling the image resolution slower.
3. The writing is clear and well-written.
4. This paper provides several interesting observations and insights.
5. This paper shows that the choice of regularization can be affected by the number of training epochs which suggests that some models searched with small training budgets (such as RegNet) may not generalize to longer training procedures.
6. The authors show that with improvements in training, regularization, and model architecture, simple ResNets can outperform strong EfficientNets. Further, in Figure 5, the authors show that ResNet-RS outperforms EfficientNet-RS, which suggests that the EfficientNet architecture is also worse under a fairer comparison.
7. The authors also provide a comparison with the unsupervised method and show that the proposed model also achieves better fine-tuning results.
8. Not limited to ImageNet, the authors also show that the proposed method can be applied to 3D ResNet for Video Classification.

Weaknesses:
1. The text in Appendix B1 is missing. I can't find any text describing Figure 5 and the comparison with EfficientNet-RS.
2. Scaling the image resolution slower has been proposed in the fast compound scaling method [1]. While other parts are different, I wonder how the proposed scaling strategy performs when compared with the fast compound scaling in terms of performance-efficiency trade-off.

Questions:
1. In table 4, the EMA of weights does not lead to any performance improvement. Does it also happen to the final model? If it is unnecessary, why don't you just remove it?
2. I wonder how general the scaling strategy is. Is it only limited to vision models or it generalizes to NLP and speech? Or is it only limited to ConvNets?
Can it also be applied to vision transformers [2] or their variants?


Overall, this is a well-written paper and provides several new observations and insights. I view the lack of novelty in the training methods and model design as a strength of this paper instead of a weakness. Undoubtedly, I recommend accepting it. While I give it an 8 currently, I am willing to raise my rating to 9 if the authors can address the limitations mentioned below or even to 10 if my questions are addressed as well.

References:
[1] Dollár, Piotr, Mannat Singh, and Ross Girshick. "Fast and accurate model scaling." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.
[2] Dosovitskiy, Alexey, et al. "An image is worth 16x16 words: Transformers for image recognition at scale." arXiv preprint arXiv:2010.11929 (2020).
limitations_and_societal_impact: 1. The scaling rule in Appendix D Table 9 seems to be arbitrary to me which makes the scaling method hard to be applied to other architecture. In the text, the authors say that they try to keep the ratio roughly constant which contradicts what Table 9 presents: 24:36 = 1:1.5 while 29:53 ~= 1:1.8, 36:72 = 1:2, and 44:87 ~= 1:1.97. Only the last two models have a similar ratio. Moreover, I understand that the authors try to keep the small models (ResNet-50, ResNet-101, ResNet-152) the same as the prior work, but I wonder whether these choices are optimal. Would it be possible to find some small ResNets which push the Pareto curve further?
2. The authors do not study how the improvements in three directions generalize to other model architectures and datasets. I understand that the authors focus on ResNet architecture, but the paper would be more impactful if the authors can provide a study on the impact of these methods on different model architectures. Given that EfficientNet-RS is worse than ResNet-RS, I believe the ranking of model architectures can be shuffled.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper proposes a methodology to improve ResNets like architecture by changing a little bit the architecture and using better training procedure. The rules proposed in the paper allow to have a better trade off between speed and accuracy with ResNet-like architecture and be competitive with efficientNet. 
main_review: Strength:

- The point "Novel architectures underlie many advances, but are often simultaneously introduced with other critical – and less publicized – changes in the details of the training methodology and hyperparameters." L22 is very interesting it is true that it would be beneficial for the field of image classification if it was more taken into account.
- The paper offers a broad and instructive analysis of different elements of training and how to scale models with resolution. 
- The paper is well written and the ideas are easy to understand.

Weaknesses:
- Significance of the results: It would be interesting to also have the results on ImageNet v2 or on another dataset to see how the different results of the different analyses evolve.

- Related work: The EfficientNetv2 paper[1] also proposes to make changes to the training procedure and some architectural changes to have more efficient models. It would be interesting to discuss this paper.

Comment:
- "a canonical ResNet with 79.0%" L31 It would be interesting to specify which type of ResNet, since it is not a ResNet-50, that can be a bit confusing.

[1] Le et al., EfficientNetV2: Smaller Models and Faster Training
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper revisits the performance of ResNet from three aspects, architecture, training scheme, and scaling strategy. The author shows that a better training and scaling strategy can give even more improvements than better architecture design. The author conducted many experiments to empirically show what is a good training and scaling strategy and how good it can be.
main_review: Strength:
1. There are very thorough experiments to validate the effectiveness of better training and scaling strategy.
2. The concluded training and scaling strategy outperforms state-of-the-art methods on different tasks. The empirical results are very impressive.
3. The paper is easy to follow.

Weakness:
I would say this is a good empirical study about how to better train a network. However, I don’t know if it would be interesting to the NeurIPS audience. The followings are some of my concerns:
1. The effectiveness and interplay of different training/regularization methods have been studied before [1]. Although the author may have conducted more thorough empirical evaluations, the conclusions about this are pretty similar. There are no surprising findings here.
2. The author pointed out the importance of scaling strategy, but the importance of scaling strategy has also been studied before [2][3]. The author further proposed two scaling strategies, but they are all based on empirical results on ResNet. There are no (or very few) analyses on these two strategies. I don’t know if they could also work well on different architectures and tasks.
The author claimed depth scaling is better than width scaling for longer epoch regimes because width scaling is subject to overfitting. However, in Fig.3, width scaling seems to work well in Reso-128 and Reso-160, but fails in Reso-224 and 320. But I think a smaller input should lead to more overfitting, why width scaling works well in small resolutions.
3. The author claimed width scaling could largely increase the number of parameters and lead to overfitting, but we can control the width scaling factor (or even non-uniform width scaling) to make them have the same number of parameters as depth scaling, will the width scaling work better or as good as depth scaling if they have the same number of parameters?
4. In practice, how can we determine the overfitting occurs, or how long can be termed as the long epoch regime to switch from width scaling to depth scaling in different architectures and tasks. If we need to do this empirical evaluation for every architecture and task, I don’t think it is practical.
Overall, I think the results are impressive based on the evaluated architectures, but it lacks further analyses about the results. I don’t know if the proposed scaling strategies can really be useful for practitioners.

Reference

[1] Bag of Tricks for Image Classification with Convolutional Neural Networks. CVPR 2019.
[2] EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. ICML 2019.
[3] MutualNet: Adaptive ConvNet via Mutual Learning from Network Width and Resolution. ECCV 2020.
limitations_and_societal_impact: The author has addressed the limitations and potential negative societal impact of the work.
review_title: null

Meta-review: All reviewers unanimously recommend acceptance of the paper. The paper is well written and instructive about the importance of training and scaling strategies. It may deserve some elevated attention as a spotlight contribution.
Decision: Accept (Spotlight)