Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Paper presents empirical analysis Multi-headed Self-Attention (MSA) as part of Vision Transformer (ViT) and its variants. It identifies the following properties of MSA: (1) ViT with MSA improves both accuracy and generalization by flattening the loss landscape; (2) ViT with MSA behaves differently from Convolutional Neural Nets (CNNs). Specifically, ViT with MSA behaves as a low-pass and CNNs as high-pass filters. And (3) multi-stage neural nets behave like small independent models connected in series. Based on these observations, paper proposes a new architecture -- AlterNet, where CNN block is combined with MSA block and this "base" structure is replicated to produce deeper models. AlterNet is shown to outperform pure CNNs in both large and small data regimes. 
main_review: Pros:
- Paper focuses on addressing important problem and draws interesting insights
- Paper provides extensive empirical results and evidence
- Paper proposed a new Alter-ResNet architecture that is both more accurate and robust

Cons: 
- Paper is generally hard to read and understand. Imperial results are reported in a variety of different forms and entirety of observations is hard to grasp. I read the paper carefully twice and I am still not 100% sure I understand ALL the intricacies of every experiment and how those unequivocally lead to insights drawn. The paper is just very dense. I believe the observations and insights are correct, the issue is manly expositions which could benefit from substantial improvements. 
- The form of analysis conducted isn't well motivated or explained. This is perhaps the most negative aspect of the paper. Many different aspects of models are analyzed, e.g., NLL, Hassian max eigenvalue spectra, Frequency / Log Amplitude, etc. It is very hard to keep track of these different measures and little intuition is given for why these are ultimately "the right" measures to look at for a given empirical evaluation. Most experiments take the form (and I am simplifying of course) of "We did X, which means Y." However, no explanation is given for why X is an appropriate measure for Y; or explanation of potential alternative measures for Y and why those were not assess. Finally, no discussion of what optimal value or behavior of X would be appropriate for a good model and so on. I would suggest re-structured each of the empirical experiments to have clearer flow and argumentation. 

summary_of_the_review: Overall, the paper is empirically addressing a series of important questions about the (relatively) new class of architectures (ViT). The paper serves as both: (1) effectively a survey of recent papers and observations made in prior works and (2) empirical evaluation of these models. Empirical evaluations point to some interesting insights that are ultimately leveraged to build a new form of the model which is a (relatively simple) combination of the MSA and CNN. 

The main concern with the paper is density of the text and the general lack of motivation for the various experimental designs, as well as sometimes somewhat tenuous (and not well explained) connections between empirical results and high-level insights. Despite these issues with exposition, which I think would benefit from fixing, I think the paper does address an important topic and drawn insights can be broadly useful. For these reasons I am in favor of acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper analyzes how vision transformers (and in contrast convolutional architectures) work from an optimization standpoints. The paper shows experimentally that contrary to popular belief ViTs do not directly benefit from their more expressive underlying representation, but instead smooth the loss landscape which aids training. The paper then presents a hybrid CNN-ViT architecture that combines some of the benefits of ViTs with advantages of CNNs.
main_review: + The paper is very well reasoned and supports all arguments with solid experimental evidence. The presented story and explanations are both interesting, refreshingly different, and solidly argued.
+ The presented architecture is quite interesting, as it mixes many conv and MSA blocks, rather than a simple transition (i.e. in ViT-Hybrid or LeVIT, ...). Though the "Building-up rule" seems a bit arbitrary, and a more thorough reasoning would help the paper.
- The organization of the paper could be improved a bit. There is currently a bit of repetition within section 1 when talking about points (1)-(3) multiple times.
- Section 3 is a bit hand-wavy and relies on prior work for much of the argumentation. A similar line of though to section 2 would make the paper stronger.

Minor: Stylistically the numbered circles don't work all that well. Simple text 1., or (1) might work much better.
summary_of_the_review: I like this paper, and the insights it provides in terms of optimization landscapes of ViT vs CNN architectures. It also provides a pretty clear route towards better ViT or CNN architectures by combining the strengths of both. I wish though the experimental evaluation would focus more in ImageNet (currently mainly in appendix), and not CIFAR.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors show that the robustness and better performance of ViTs is attributed to their property of flattening the loss surface. Insights are taken from a parallel ICLR submission, where spatial smoothing has been shown to smooth loss landscape, and help improve robustness. The paper shows that the properties of spatial smoothing are exhibited by MSA as well.


main_review: The paper shows that MSA is low-pass while convs are high-pass, thus they complement each other. The paper interprets deep models as a series of independent blocks, and then proposes a new architecture, in which the last conversation block is replaced with multi-head self-attention. The architecture called AlterNet, shows better performance and robustness on ImageNet and CIFAR-100 datasets.

Paper says “local MSAs with a 3 × 3 receptive field outperforms global MSA”. Can the authors please provide a relevant reference to a ViT work, which does not encapsulate patch-level interactions (with MSAs) altogether, and only relies on local within-patch attention? I would be amused to see a deep architecture with a receptive field as small as 3*3.

While the paper makes a good effort in explaining how ViTs work. Some of the findings and explanations might not be in line with existing literature e.g., Fig.1 of the paper https://arxiv.org/pdf/2106.01548.pdf shows that ViT landscape is not flattened, and is sharper. 

The authors mention that "Multi-stage architecture in PiT and local MSA in Swin also flatten the loss landscapes” I think it is the hierarchial features, captured in PiT and Swin, that help them compared with vanilla ViTs where feature hierarchies are missing. CNNs also have feature hierarchies with subsequent pooling. I guess vanilla ViTs, PiT, Swin or CNNs, are all multi-stage. It is multi-scale feature hierarchies that is absent in vanilla ViTs, and is present in PiT, Swin, ResNets.  Also, Vanilla ViTs (based upon the standard attention is all you need architecture) do not capture local relationships within small image patches. Recent variants of ViTs which capture such local cues with dilated convolutions, or local self-attention, basically, introduce better inductive biases, since local cues are important for images. 
summary_of_the_review: - The visualized loss landscape in the paper is different from existing literature https://arxiv.org/pdf/2106.01548.pdf; where a sharpness aware optimizer is developed to flatten the loss landscape of ViTs.

- I am not entirely convinced that the findings in the paper fully explain how ViTs work. I guess, the paper draws analogy between MSA and spatial smoothing, and then builds most of its argument around that. An appropriate title reflecting that will be better, in my opinion.

- Some of the terminologies used in the paper confused me, e.g., multi-stage vs multi-scale. All deep architectures are multi-stage in my understanding. Most have multi-scale feature hierarchies. Vanilla ViTs have single-scale features throughout all stages.

- Small patches improve performance (counter to what is shown in the paper). See https://arxiv.org/pdf/2106.09681.pdf
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper attempts to explain the behaviour of multi-headed self-attention in vision transformers using a series of empirical observation. Some main observation are that the poor performance of ViT in the small data regime is not due to overfitting but to issues induced into the optimization procedure and that the behaviour of MSA is somewhat complementary to convolutions. Based on the observations a new model is proposed that combines convolution and MSAs. The experiments show that the new architecture improves results on smaller dataset.
main_review: Strenghts:

- Studies to better understand the behaviour of these models are valuable. The presented study is comprehensive in the sense that it includes many recent variants of ViTs and the explanations seem plausible.


Weaknesses:

- It is not clear to me how the construction rule for AlterNet is implemented in practice. The paper states "... does not improve performance sufficiently ...": on what dataset is this measured? Is this on a separate validation set or was this applied to the reported test set? 

- It would have been great to see if the AlterNet modification can be universally applied, by showing thi on some additional base CNN models beyond ResNet-50.

- There are a couple of interesting findings and explanations, however, it seems hard to get to actionable design rules from them (beyond the proposed AlterNet). For example, the insight that loss smoothing methods aid in training seems to be generally true, not only for ViT, so it is not clear how much this insight really helps.

- It would be great to see a discussion on how the number of parameters influence the findings. Are the tested ViT variants and ResNets comparable in this respect? The main results are reported on "Tiny" variants of transformer architectures. Would some of the insights change with larger models?

Minor:

- Figure 10 (b): it seems that the ResNet figure is cut off from below.
summary_of_the_review: I overall think that this paper shows some interesting phenomena, even though many of the observations are post-hoc and don't seem to lead to immediate actionable design rules. However, follow-up work may still benefit from these insights.
review_title: null

Meta-review: The paper presents an empirical analysis of Vision Transformers - and in particular multi-headed self-attention - and ConvNets, with a focus on optimization-related properties (loss landscape, Hessian eigenvalues). The paper shows that both classes of models have their strengths and weaknesses and proposes a hybrid model that takes the best of both worlds and demonstrates good empirical performance.

Reviewers are mostly very positive about the paper. Main pro is that analysis is important and this paper does a thorough job at it and draws some useful insights. There are several smaller issues with the presentation and the details of the content, but the authors did a good job addressing these in their responses.

Overall, it's a good paper on an important topic and I recommend acceptance.
Decision: Accept (Spotlight)