Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors proposed three modifications for convolutional network to make it more robust: 1) split the input image into non-overlapping image patches; 2) use large kernel size; 3) remove some BN and ReLU layer. Numerical experiments on several ImageNet variant datasets validate the robustness improvement with slight degradation of the accuracy on clean datasets.

strength_and_weaknesses: [Strength]

* The proposed method is able to improve the robustness of CNN models and make them comparable to DeiT-S in terms of robustness.
* The proposed method is validated on four popular ImageNet variants for robustness evaluation
* Experiments are reported in great detail, with sufficient ablation studies.

[Weakness]

* The paper is mostly empirical and lacks theoretical motivation or guarantee. The authors simply report the accuracy increment or decrement without further investigation. So it is a good-to-know fact but cannot inspire in-depth understanding of the robustness of CNN.

* The numerical experiments suggest that, the proposed three modifications, are not universal. Sometimes they can improve accuracies on ImageNet variants but sometimes dramatic performance drop can happen. What is worse, the patterns of good / bad cases are random. So the proposed method cannot transfer well. For a new given dataset, it is hard to tell whether we should apply the proposed method, or how we should choose the pattern in BN/ReLU removal.

* The proposed method will hurt the performance on clean datasets most of the time. This is a hard trade-off in practice.

* The robustness improvement over DeiT is not very significant for some networks and some datasets. Sometimes it is worse than DeiT (Table 1).

* Only DeiT is selected for comparison. Why DeiT? What about Swin-Transformer or other robust VIT models?

clarity,_quality,_novelty_and_reproducibility: * The paper is well written and easy to follow
* Lack novelty in theory. 
* Training details are provided in Appendix A. Should be easy to reproduce  most of the results in this work.
summary_of_the_review: This work lacks novelty in theory. The proposed method is not universal and sometimes even degrade the performance dramatically.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper seeks to understand architecture changes to traditional CNNs that render them as, or more, robust as Transformers when applied to out of distribution data. The main conclusions of the paper are that employing (a) an initial partition of the image into non-overlapping patches; (b) larger kernel sizes in a resnet block; and (c) having fewer non-linear layers (i.e., with normalizations and activations), can improve robustness.
strength_and_weaknesses: **Strengths**

- The paper conducts a thorough empirical evaluation in terms of different combination of the main design choices.
- It is interesting to look at the effect of architecture choices not just on accuracy but also robustness.

**Weaknesses**

- Most of the underlying design choices themselves have appeared in previous papers. As authors note, their work is very close to the ConvNeXT work, with the main difference being that their focus is on robustness.
- I missed an evaluation of ConvNeXT itself in the various tables. While the ConvNeXT paper did not focus on robustness, is it the case that the ConvNeXT models are also just as robust as the best models proposed in the paper? If so, then this paper's analysis would still be useful, but the narrative would need to be re-cast as understanding the robustness of ConvNeXT vs ViT.
- The paper lists accuracy numbers but without error-bars. It is difficult to gauge if the differences above are statistically significant. It would be worth repeating each experiment 3-5 times (at least for the -S models) and report both mean and standard deviation.
- The tables compare models in terms of FLOPs, but not in terms of number of parameters. Do the proposed models have a larger or smaller number of parameters? How does the accuracy compare when we compare to models with equivalent numbers of parameters? Again, it is reasonable to want models with equivalent FLOPs, but the #parameters analysis could reveal an alternate hypothesis for what makes the architectures robust (viz, number of parameters).
- The paper also misses an evaluation of adversarial robustness --- in terms of adversarial perturbations and to Imagenet-A (the so called natural adversarial examples). Given that the paper's contribution is largely empirical, these comparisons are needed for completeness. 
clarity,_quality,_novelty_and_reproducibility: Reasonably clear. The experiments should be reproducible based on the text. Novelty is not in techniques but more in the empirical analysis.
summary_of_the_review: Overall, this is an interesting empirical analysis of how architecture design choices can affect robustness, and specifically, of the elements of the Transformer architecture that can be 'back-ported' to CNNs to increase the latter's robustness.

However, since the main contribution is this empirical analysis, there is more of a burden for it to be complete, and as mentioned above, it is lacking in several respects.

### Post Rebuttal

The authors have provided a detailed response with several additional experiments that address all my concerns. I believe the paper should be accepted (I'd give it a 7 instead of 8, but since that's no longer an option, I believe it's closer to 8 than 6).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: * **Motivation**: This paper takes a deep dive into why Vision Transformer (ViT) networks have empirically been found to be more robust than Convolutional Neural Networks (as evaluated by performance on out-of-distribution samples).

* **Approach**: They isolate some of the key components in DeiT that is missing in CNNs (besides the obvious attention mechanism), and do A/B experiments to figure out which components contribute the most to the differences in robustness. The components they evaluate are:
1. Patchifying Stem: DeiT partitions images into PxP non-overlapping patches. The authors replicate this in CNNs by applying non-overlapping convolutions.
2. Large Kernel Size: DeiT has better global receptive field. The authors replicate this in CNNs by increasing kernel size.
3. Reducing Activation and Normalization Layers: DeiT has fewer activation and normalization layers than a typical ResNet block. The authors replicate this by reducing #activation/normalization layers in the ResNet block.
The authors call the model with above 3 modifications a **Robust-ResNet**.

* **Findings**:
1. Patchifying has the most noticeable impact in improving robustness
2. Robust-ResNet is very similar in terms of #FLOPs and clean accuracy to both a DeiT and the (unmodified) ResNet. However, whereas the (unmodified) ResNet had noticeably worse robustness performance compared to DeiT, Robust-ResNet is able to achieve robustness results that are even slightly better than DeiT.
3. Whereas ResNet students (in model distillation) struggle in terms of robustness when the teacher is a ViT, Robust-ResNets seem to outperform the ViT teacher in terms of robustness.
4. Improved robustness seems to typically come at a cost to (clean) accuracy - E.g. Table 2
strength_and_weaknesses: **Strengths**
1. Well written with an easy to follow flow
2. I like the overall thought behind the paper of trying to figure out which components in ViTs have what amount of importance in getting better robustness. This type of paper shines some light on making better architectural decisions and understanding the effects of each component. Well done!
3. Experiments are well organized and easy to extract information from.

**Weaknesses**
1. Table 7 seems incomplete without results from unmodified ResNet.
2. The paper shows what matters through empirical evidence (and the experiments are interesting on their own), but it doesn't seem to provide any explanation/potential hypothesis about why patchifying/large kernel size/reducing #activation/normalization layers could improve robustness.

**Questions**
1. Table 4 description: What is meant by "It is observed that the optimal choice always lead to best robustness." I don't see that pattern.
2. All tables: How did you decide which row to highlight?
3. Is patchifying stem (i.e. non-overlapping convolution) only done in the first convolution operation of Robust-ResNet?
4. page 6 last paragraph: why such a drastic speed up from removing activation/normaliaztion layers?

**Typos**
1. page 2, paragraph 1: due to less normalization layers are used -> due to less normalization layers being used
2. page 3, paragraph 1: Transformers are inherently much more robustness than CNNs -> Transformers are inherently much more robust than CNNs 
3. page 3, paragraph 1: last sentence was hard to read ("Our work is a direct follow-up...")
4. page 3, "Settings" section,  paragraph 2: in whichthe hidden -> in which the hidden 
5. page 3, last paragraph: FLOating Point operations per second -> FLoating-point Operations Per Second
6. page 6, last paragraph: speep up training -> speed up training
7. page 7, table 3 description: block is keepped -> block is kept
8. page 7, table 4: always lead to -> always leads to
9. page 8, "Components Combination" section:  16×16 patchify stem with patch size ->  16×16 patchify stem with patch size 16
10. page 9, section 7: still performs favorably -> still perform favorably
clarity,_quality,_novelty_and_reproducibility: * To the best of my knowledge the experiments and the breaking down of ViT components is original.
* The writing, figures, tables and experiments were all of high quality and very clear.

summary_of_the_review: I recommend for this paper to be accepted because:
1. The strengths outlined above outweigh the weaknesses
2. I think the findings & experiments would be interesting to the community
3. This type of paper (A/B comparison of different components) helps in shining some light on the underlying workings of neural networks so that we are not just random searching for better architectures.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Paper Summary:
This paper does a deep dive on architectural design considerations of CNNs, inspired by choices in transformers, and demonstrates that by changing CNN designs to be more similar with transformers, CNNs can outperform transformers without using self-attention mechanisms. The design choices involved include patchifying inputs, expanding receptive fields with depth-wise convolutions, and reducing the number of normalization and activation layers.

Review Summary:

Pros:
	- Well written (DpNa)
	- CNN performance improved relative to transformers such as DeIT (rxfk, DpNa)
	- Thorough evaluation (rxfk, ntqh,DpNa)
	- Robustness analysis is interesting (ntqh,DpNa)

Cons:
	- Mostly empirical work without sufficient understanding (rxfk,DpNa) -- authors confirm this but argue their empirical observations still have value to community.
	- Unclear if results are generalizable across datasets (rxfk) 
	- Only DEiT selected for comparison. What about othr architectures? (rxfk) -- authors added Swin.
	- Many of these design choices have appeared in previous works (ntqh)
	- How does ConvNeXT compare with the robustness measures? (ntqh) -- Authors added experiments.
	- Statistical significance not reported (ntqh) -- variance now reported
	- What about number of parameters? (ntqh) -- added
	- What about adversarial robustness? (ntqh) -- authors commented that this is addressed in prior work
	


AC Recommendation: Accept. Reviewers strongly lean accept. Interesting experiments that are helpful generally when considering architecture design.
justification_for_why_not_higher_score: Mostly empirical work. Novelty low and theoretical understanding low.
justification_for_why_not_lower_score: Reviewers still felt this paper carried enough useful information for publication.
Decision: Accept: poster