Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper overall tackles a very important question and pin-points the important dissociation that when the computational complexity of two models is equalized: one with a first stage convolutional operator and another one without one -- both pre-seeded to Vision Transformers; it is the one with a convolutional stem/operator *that is only sufficiently necessary in the first layer!* that exploits the locality structure in image, has increase in accuracy, faster convergence and flexible optimizability. 

main_review: 
Strengths:

This paper presents all the correct ingredients for acceptance at any computer vision venue -- despite this being NeurIPS: The paper presents a strong set of experiments, baselines, and several simulations ran for different conditions (example GFlops, and hidden units in the ViT’s) that have been carefully tuned to show the stability of the results despite a lack of errorbars. Overall this paper should be accepted because it presents a simple idea that has been rigorously tested through various experimental manipulations and the novelty is presented quite clearly in Figure 1. The take-home (See above) is easy: if you want *better* results (accuracy/faster convergence/flexible optimizability) when training a Vision Transformer, add a convolutional stem/layer in the early block of visual processing (somewhat surprisingly similar -- to how the human visual system operates in early stages of processing).

I have no follow-up questions to the authors, but I would like to add some limitations that prevent me from increasing my score (that is already very positive), as this paper could have had potential to have a more interdisciplinary scope beyond the Transformer movement.

Limitations:

There is more to computer vision (and vision in general) than object recognition and ImageNet-based experiments. Several follow-up questions arise, such as: How would a convolutional-stem aid a perceptual system if the task was *not* object recognition (the title of ‘seeing’, suggests Object Recognition is the ultimate standard). But more critically, experiments with Robustness to either common corruptions (Hendrycks & Dietterich, ICLR 2019) or Adversarial attacks like any of the PGD-flavors should be explored in future work. Not to be nosey, but these are indeed important questions as several authors have shown that there is a trade-off between accuracy and robustness (Zhang et al., ICML 2019).

I do wish that the paper adds more discussion on the ML theoretical aspects of the advantages of convolution when a locality constraint is induced by the image structure as explored in Poggio (PNAS 2020) and recently, Deza, Banburski, Liao & Poggio (ArXiv 2021). Similarly, it would be nice if the paper connects the importance of the convolutional operator as a biologically plausible inductive bias (eg. V1 gabor-like filters; Hubel & Wiesel in the 60’s) and how recent trends in computer vision have started to revisit the importance of inductive biases in 2-stage *neuroscience-inspired* perceptual systems as the work of Dapello, Marques et al. for the case of Adversarial Robustness (NeurIPS, 2020); Parthasarathy & Simoncelli (ArXiv, 2020) for texture recognition w/ a self-supervised objective and Deza & Konkle (ArXiv, 2020) for scene recognition w/ a foveated module. Perhaps adding a small section in the Relevant work about these and other works that have explored the importance of locality, convolutional operators in CNNs and 2-stage models should be addressed.

Other works worth adding that are particularly relevant to the theory of locality and convolutional operators as a critical computation in machines:

* Elsayed et al. “Revisiting Spatial Invariance with Low-Rank Local Connectivity”. ICML 2020.

* Pogodin et al.: “Towards Biologically Plausible Convolutional Networks”. ArXiv 2021.

* d’Ascoli et al: “Finding the Needle in the Haystack with Convolutions: on the benefits of architectural bias” NeurIPS 2019.

limitations_and_societal_impact: Not Applicable
review_title: null

final_score: rating: 9: Top 15% of accepted NeurIPS papers, strong accept
review_content: summary: The authors perform an extensive study investigating the stem of ViT models and replacing it with one that consists of a stack of downsampling 3x3 convolutions. The study reveals that this simple change makes ViT training more robust, faster to converge, but also scales better (cf. i21k pre-training).
main_review: I have to admit, that I was initially set to reject the paper, because I have counter-evidence of the main claim that "ViT cannot be optimized by sgdm": I have been training original patch-stem ViT models with SGDM for a while, to same accuracy as AdamW (but unpublished), and am certain that the negative results reported in DeiT are due to confounders, such as not properly tuning sgdm.

However, carefully reading the paper made me change my mind by 180°. This paper is really good and needs to be accepted. Many statements reflect my thoughts:
- It is crucial to tune lr/wd, or one gets misleading results such as "random erasing is necessary", and actually tuning those reveals it (and other tricks) to be *unnecessary*.
- Especially for ViT and EfficientNet models, flops and #params are not a good measurement to put on the X axis.
- We need to compare architectures in ideal settings for each.

I find the very last presented finding, Fig6(right), to be the most interesting: the ViT_C seems to scale better than both ViT_P and Re{s/g}Nets. This is very promising.

Now some criticism on the paper:

1) line 112: I would argue that the conv-stem is not "strictly beneficial" since:
1) A) the patchify stem has led to some interesting recent ideas which are simply not possible with a conv-stem, such as using different random augmentations per patch.
1) B) It re-introduces BatchNorm into the model, with all of its problems. I wish authors had tried using GroupNorm here too.
2) line 94: the cited papers do not really support "optimization difficulty".
3) Why do we need the final 1x1 conv in the stem? Couldn't we simply have the last 3x3 conv output the required number of channels? That would furhter simplify the design.
4) line 139: "we do not optimize the stem to maximize model accuracy". This is not really true to spirit, given AppendixA. Even if the best stem ends up being the one that authors first thought of. What if a different one from AppendixA was found to perform substantially better? Surely, that one would have been proposed instead. Simply drop this sentence.
5) Reusing optimal lr/wd from 50ep for up to 400ep. This is known to be suboptimal (cf AdamW paper itself) and generally for much longer training, wd needs to be decreased. It likely does not invalidate the results though.
6) It is not explicitly stated what lr/wd are used in 5.1 and 5.2. Is it the best ones found in 5.3? If not, this might be an issue.
7) Does the SGDM sweep also use "decoupled" WD as per AdamW? Do the authors actually use decoupled WD from AdamW? (That is not commonly used in practice, but better for such sweeps.) By "decoupled", I mean doing w_t+1 = w_t - lr * grad w_t - wd * w_t,  as opposed to w_t+1 = w_t - lr * grad w_t - wd * lr * w_t, see also the AdamW paper.
limitations_and_societal_impact: None
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper studies the impact of stem in ViT. Specifically, the authors propose to replace the patch-based stem in ViT with a standard, lightweight convolutional stem, and show that this simple revision leads to substantially improved stability wrt the choices of optimizers and several other hyper-parameters. The resulting model achieves comparable performance with strong convnets baselines when trained on ImageNet-1K and has better accuracy-speed tradeoff than both ViTs and convents when trained over larger dataset (ImageNet-21K).
main_review: Strengths:
+ Overall, the paper is very well written and easy to follow.
+ Extensive amount of ablation studies have been provided to back up the main claim that the conv stem can improve ViT's optimization stability/robustness.
+ IMO the most interesting part of the manuscript is not the resulting model itself, but rather the methodology. For example, the notion of "optimizability" examined in the paper can be potentially used as an additional metric to evaluate new architectures and provide complementary insights other than accuracy or speed.

Weaknesses:
- The overall message is interesting but not strong enough to be exciting:
  * In most ablation studies, ViT_c behaves like something in the middle of ViT and ConvNets (e.g., see Figure 2, Figure 3 and Figure 5). Those results are interesting, but on the other hand are more or less expected given the hybrid nature of the model.
  * More importantly, although the authors provided extensive empirical evidence to back up their claim that a conv stem is helpful, the critical question that why it is effective is left unanswered. An in-depth analysis of the underlying reason for those empirical observations would make the paper a lot more informative.
- The paper claims their model outperforms the state-of-the-art convnets, but I'm not sure whether it is true: while RegNet is a competitive baseline, there are more recent networks with stronger performance (e.g., NFNets).
- There are missing details about the ImageNet-21K training setup that I cannot find in paper or appendix. E.g., what is the input resolution during fine-tuning (224 or 384)?

Overall, I'm leaning positive about the paper because the methodology is interesting and I believe the data points would be useful for future studies. On the negative side, not much insight was provided to answer why conv stems are helpful, and I believe answering that technical question would substantially strengthen the work.

========= post rebuttal ==========

Raising my rating to 7 since some of my technical questions have been addressed in the author's response.
limitations_and_societal_impact: The authors mentioned "understanding the theoretical foundation of why such a minimal architectural modification can have such large (positive) impact on optimizability" as a future work. This is actually a major missing piece IMO, and I believe answering this question in future revisions could substantially strengthen the technical contribution of this work.
review_title: null

Meta-review: Reviewers agree after author's response that this paper is a (strong) accept (7,9,7). The paper doesn't have any major weak points. The message is simple and clear and the contribution thorough. I don't consider the implications of the result worthy of a spotlight though.
Decision: Accept (Poster)