Patches Are All You Need?

Abstract: Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https://github.com/tmp-iclr/convmixer.

Reviews and author responses:

This work proposed a new design for the image classification task named ConvMixer, which brings the idea from CNN to Visual Transformer. Unlike previous ConvNets, Transformer-based models, and MLP-based models, ConvMixer simply applies depth-wise (with skip connection) and point-wise convolutions on the patches. The key point authors claimed is that using patch embeddings is a powerful and important takeaway besides the architecture design. Experiments on ImageNet demonstrate the effectiveness of the proposed model. Strengths: 
1.	The model is interesting and simple, which would inspire the vision community a lot.
2.	The results are promising, especially on the accuracy. Even a simple depth-wise / point-wise convnet without resolution reduction can achieve a promising result.
3.	An interesting but not discussed finding is that, different from VIT  and MLP-mixer (and variants) that emphasize the global context, this work did not introduce global context (although the receptive field is large enough). A close look at ResNet and BagNets [1] would be interesting.

Weaknesses: 
1.	Typos, e.g. , “simiar” in page 2; “do ConvMixers” in page 8, etc.
2.	At the end of Section 1, the authors claim that “This suggests that, at least to some extent, the patch representation itself may be the most critical component to the superior performance of newer architectures like Vision Transformers.” I don’t think this is true, and the authors have not fully verified such a claim. There are many other factors, like learnable parameters, effective receptive fields, etc. This statement seems to be overclaimed.
3.	Most importantly, I don’t think this method achieves “better” results than others.  i), Comparing with ResNet152, the throughout is 89 vs. 872. The slow speed of ConvMixer is not acceptable in real-world applications; ii) Besides, the performance of ResNet would be higher than reported if trained with appropriate strategies [2].  iii) Small patch properly can achieve better performance.  I would suggest authors try some other Transformer models, like Swin Transformer [3].
4.	In future work of page 4, authors claim that “We are optimistic that a deeper ConvMixer with larger patches ... and hyperparameter tuning”. However, this may not be true. Please see Appendix A. I have not seen the strong superiority of ConvMixer over the ResNet in the accuracy with similar parameters. And the ConvMixer’s speed is far below the ResNet.
5.	In VITs and MLPs (including their variants), we have two skip connections (it is not called residual connection as shown in Fig 2) in a block. However, authors designed with only one (tuned on cifar10). I just want to know if we add two skip connections in ConvMixer, what is the result for ImageNet.
6.	I’m not a fan of cifar10 dataset due to the dataset size and inevitable randomness in results. Considering the limited computational resources, some other datasets may be helpful (like cifar100, Tiny-ImageNet, etc.) or authors can report the mean/std results of 5 runs on cifar10.
7.	I could not get the idea of “A note on paper length” paragraph in a conference paper, considering some of the important experiments are missing. It is nothing to do with the main idea of ConvMixer.
8.	Some important ablation studies are missing, like the influence of depth, kernel size, patch sizes, etc (I mean detailed ablation studies). I would also be interested in the performance of ConvMixer on some downstream tasks like detection. 

[1] Brendel, Wieland, and Matthias Bethge. "Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet." International Conference on Learning Representations. 2018.

[2] Wightman, Ross, Hugo Touvron, and Hervé Jégou. "ResNet strikes back: An improved training procedure in timm." arXiv preprint arXiv:2110.00476 (2021).

[3] Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." arXiv preprint arXiv:2103.14030 (2021).
 I recommend reject due to the weaknesses above, and I will consider updating my score if seeing a strong rebuttal.

 Dear Authors,

Thanks for your update on the manuscript. However, I am still not fully convinced by the layout of the paper contents that some important details are put into the appendix. Considering the novelty and overall contributions, I decided to upgrade the score to 5 since this current manuscript is not ready for a decent publication yet.

Reviewer

 Hello, thanks again for your review. As we have updated the paper to address the reviewers' concerns, and the discussion period is nearly over, we wanted to see if you have any additional comments based on the revision. We have also summarized our revisions in a recent post.

 Thank you again to the reviewers and commenters for their feedback and suggestions! In our revision, we tried our best to clarify the main concerns, and we outline those changes below (also highlighted in the pdf):

- It seemed to be a common misunderstanding that we were searching for the "best" performance, when this was not the case at all. We emphasized the fact that we did not design our experiments to compete with SOTA models, and that neither our model nor our experiments were in search of real-world applicability (i.e., speed, performance). 

- On that note, we emphasized the important differences between our training procedure and that of competitors. (Namely, that we made no attempts at hyperparameter tuning _and_ trained for fewer epochs.)

- Our comparison to ResNet was called "unfair", so we tried to clear up potential causes of misunderstanding here. We emphasized that we trained ResNets in the same setting as ConvMixers for an "apples-to-apples" comparison, and also discussed how a more optimal ResNet training procedure could have changed the results, as seen in "ResNet Strikes Back" [1]. In particular, despite _much_ tuning and much longer training, a comparably-sized ResNet only does 0.2\% better than a ConvMixer.

    **Aside:** "ResNet Strikes Back" [1] was published 1 day before the ICLR deadline, and our comparison with ResNet was said to be "unfair" partially in reference to this. According to the [reviewer guidelines](https://iclr.cc/Conferences/2022/ReviewerGuide), we could not have been expected to include this in our original draft. However, we have added ample discussion of this paper in the revision.

- We integrated the references suggested by the reviewers and commenters; in particular, we added [1] and [2] along with additional discussion, and we added additional discussion for results using convolutional layers from [3].

- We added one new ConvMixer experiment; we tried our best-performing model, ConvMixer-1536/20, with a larger patch size (14 instead of 7). We added this to the larger results table and also discussed it in the main text.

- A common misunderstanding was that we "overclaimed" or tried to "prove" that patch representations were key. Consequently, we emphasized the **uncertainty** of our hypothesis that patches are an important aspect of the performance of new ViT-style architectures. **In particular**, we changed the confusing wording _"may be the most critical component"_ to _"may be **a** critical component"_. We also mentioned _in the introduction_ (to increase visibility) that more experiments would be necessary to _exactly_ separate out the effect of patch embeddings.

- We de-emphasized the CIFAR-10 results.

- We additionally highlighted the important detail that ConvMixers use _large-kernel_ depthwise convolution.

- We better-justified our optimism that hyperparameter tuning/customization of the training procedure would enhance ConvMixer performance, pointing out the successful application of this technique to enhance ResNets in [1].


- We added the Swin Transformer ImageNet-1k results to the larger results table along with some discussion in the appendix (as we do not view this as a particularly useful comparison, but rather an interesting side note).

- We updated future work with the reviewers' suggestions. The future work section now suggests: experiments on semantic segmentation, object detection, and other tasks; experiments to more clearly separate the effect of patch embeddings from other factors; and investigations of ConvMixers in the regime of large-scale pre-training.


To summarize, our main contribution is to demonstrate that a _fully-convolutional_ model in the style of recent MLP-Mixer and ViT variants has surprisingly good initial performance. While the performance of such models has typically been attributed to the choice of novel (non-convolutional) operations like MLPs or self-attention, our results using convolution in place of such operations suggests that other aspects, such as the use of patch embeddings, may be at least partly responsible. The key takeaway is that we think that it is important to consider the effect of the input representation and the factors besides self-attention in vision transformers.

From another perspective, we have proposed a _surprisingly simple_, yet performant, new convolutional network. 

We hope that ConvMixer can serve as a convolutional-but-patch-based baseline for future experiments on more advanced architectures, and we believe it serves as an effective starting point for future investigations disentangling the most important components of vision models.


[1] Ross Wightman, Hugo Touvron, Hervé Jégou. "ResNet strikes back: An improved training procedure in timm".

[2] Luke Melas-Kyriazi. "Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet".

[3] Hugo Touvron et al. "ResMLP: Feedforward networks for image classification with data-efficient training".

 To be clear, we just posted these comments as a way of initiating discussion on these points. We are uploading a revision tonight.

 Dear authors,

Thanks for your hard work on rebuttal. I am convinced by the rebuttal that the main contribution is a pioneering idea for ViT structure. While my major concern of this paper is about writing, which seems a rush paper. It is good to see that there are some planning revisions. Unfortunately, I have not seen the update of the manuscript. I will only consider upgrading the score after seeing a new paper version qualified for a decent publication.

ICLR Reviewer

 Thanks for bringing this up. We will definitely mention this paper in Related Work, as the hypothesis is indeed similar.

However, the point of ConvMixer was to use convolution instead of “novel” operations like self-attention or MLPs. In contrast, the model in [1] is actually almost exactly the same as MLP-Mixer (indeed, the author said they were scooped by MLP-Mixer [2]). So while they also propose that the performance of ViTs may be due to other factors such as patch embeddings, we don’t think that their work extricates novel operations from other architecture changes like ours. This is why we contrasted our model with ViT- _and_ MLP-Mixer-style models: they both use “novel” operations, whether it be self-attention or feedforward networks.

Essentially, we don’t think we can learn much more from [1] than we can from the MLP-Mixer paper itself; but since it was concurrent with and independent from MLP-Mixer, we will definitely mention it and make sure that the difference with our work is as clear as possible.

[2] https://www.reddit.com/r/MachineLearning/comments/n62qhn/r_do_you_even_need_attention_a_stack_of/

 [Abstract]
> This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation?

[Introduction, 2nd paragraph]
> In this work, we explore the question of whether, fundamentally, the strong performance of vision transformers may result more from this patch-based representation than from the Transformer architecture itself.

[Introduction, final paragraph]
> This suggests that, at least to some extent, the patch representation itself may be the most critical component to the “superior” performance of newer architectures like Vision Transformers.

This is the hypothesis or the research question that the paper aims to tackle. The proposed ConvMixer model is support or evidence for this hypothesis, and it should be noted that the paper isn't strongly claiming that this evidence is conclusive. The results seem quite promising as well, and can definitely help fuel new applications and research.

However, one detracting impression I got from reading the paper is that: *In the pursuit of conciseness, the authors missed important discussion details.*

I think existing reviews already touch upon this point, so I will not elaborate further. I would like to highlight a specific instance: I think the paper should have talked about some of the findings and insights from *"Do You Even Need Attention? A Stack of Feed-Forward Layers Does Surprisingly Well on ImageNet"*, Melas-Kyriazi 2021 (https://arxiv.org/abs/2105.02723) [1].

Why? To quote from the abstract of [1]:
> These results indicate that aspects of vision transformers other than attention, such as the patch embedding, may be more responsible for their strong performance than previously thought.

And introduction, 4th paragraph:
> These results suggest that the strong performance of vision transformers may be attributable less to the attention mechanism and more to other factors, such as the inductive bias produced by the patch embedding and the carefully-curated set of training augmentations.

Melas-Kyriazi's hypothesis is quite similar and relevant to this ICLR submitted paper. To prove their point, they remove all attention layers in ViT and replace them with a very simple pair of feedforward networks which: a) project the patch dimension to a higher dimensional space and b) project back again to the original dimension. (I consider this to be a much simpler version of the more performant g-MLP [2], which introduces a form of spatial multiplicative gating in between the linear projections).

They also performed an unusual but useful ablation experiment: they replaced all the feedforward layers with attention, thus training a model that has nothing but attention layers (the results were terrible).

Patches Are All You Need / PAAYN argues that the novel operations and elements in MLP-Mixer, Res-MLP, Cycle-MLP, and g-MLP are conflated with the idea of patch embeddings and isotropic blocks, and thus the actual effect of the patch representation cannot be sufficiently observed. In this regard, I believe Melas-Kyriazi's work could be more useful to the related work discussion, as they use a very basic architecture to provide support for their hypothesis.

Furthermore, note that [1] also says that it isn't just the patch embedding, it could also very well be the carefully curated set of hyperparameters and augmentations that contribute to the effectiveness of ViT-like models.

Much like PAAYN, Melas-Kyriazi's paper also doesn't provide conclusive evidence of their hypothesis. As it is a preceding work however, I believe the ideas provided by [1] should have been highlighted in PAAYN, and the authors should have clearly stated how their work extends existing evidence in this area. If a future paper is able to conclusively prove this hypothesis, it would be a shame if they fail to note the ideas and inferred conclusions of [1] and PAAYN clearly.

- [1] Melas-Kyriazi, Luke. "Do you even need attention? a stack of feed-forward layers does surprisingly well on imagenet." arXiv preprint arXiv:2105.02723 (2021).
- [2] Liu, Hanxiao, Zihang Dai, David R. So, and Quoc V. Le. "Pay Attention to MLPs." arXiv preprint arXiv:2105.08050 (2021).




 Many thanks for your quick and detailed answer. 

Thanks also for the reference, which is also relevant to our own work.


 Thanks for the pointer. We are definitely aware that we are not the first to try using convolution within an isotropic architecture [1], and we highlighted this in Related Work. And we also did directly compare to ResMLP in the paper.  But we're certainly happy to make it more clear that you also experimented with convolution in the ResMLP paper and got good results if you felt this wasn't clear enough -- indeed, we think it would strengthen our argument.


Just to clarify: the main contribution of the ConvMixer architecture is that we specifically highlight _large-kernel, depthwise_ convolution in isotropic architectures, with the intuition that this gets closer to the “full” receptive field allowed by self-attention and MLP-based architectures (e.g., as you can see in your ResMLP weight visualizations).  This seems largely orthogonal to the ResMLP contribution, though please let us know if you feel differently (unfortunately, public comments look like they are closed again, but whenever they open back up, we're happy to continue discussing).


Further, we’ll also edit the paper to point out on p.13 in the appendix that you also saw the effect of early layers looking like small convolutional filters while later layers are more diffused -- i.e., it’s interesting ConvMixer learns weights that, at a very high level, looks similar to those in ResMLP.


[1]  Mark Sandler, Jonathan Baccash, Andrey Zhmoginov, and Andrew Howard. Non-discriminative data or weak model? on the relative importance of data and model resolution. ICCVW 2019.

 In our paper ResMLP (https://arxiv.org/abs/2105.03404), we propose in Table 3 "patch communication" as an alternative to the pure MLP model by replacing the linear communication layer between the tokens by a patch based depthwise convolution. 

We kindly bring your attention that this proposal is discussed in the section "Communication across patches " (Page 6) in our paper. The idea of the paper "Patches are all you need?" is very similar to this subpart of ResMLP. We feel like it would be nice if this could be discussed.



 6. We actually think that the story behind the poor throughput is not quite settled, which is why we didn’t want to dwell _too much_ on it in the main text. While it’s most likely the case that vanilla ConvMixers are simply not the most performant of vision models, we think a large part of this comes from the fact that large-kernel depthwise convolution is poorly-optimized in current deep learning frameworks, and we anticipate a 2x-2.5x speedup [see note below] if this could be improved. We do mention this in the main text, e.g., in the “Future work” part of Sec. 5.   

   Another aspect of this problem is that ConvMixers are much more efficient for larger patches (e.g., patch size 14 instead of 7), at the cost of some accuracy. For instance, as mentioned in another response here, with everything else held equal, ConvMixer-1536/20 gets 78.92% top-1 accuracy while being 3-4x faster.   
  &nbsp;
  
   Consequently, if we spent time tuning the training setup to maximize accuracy on ConvMixers with larger patches and also worked on optimizing large-kernel depthwise convolution, we might find that ConvMixers can be much more reasonable in terms of throughput. We mentioned this in the main text, e.g., in the last paragraph of “Comparisons” in Sec. 3 and in Future work; it’s not clear if we should say more given that this remains hypothetical. And we believe that we made the throughput of current ConvMixers visible in the main text -- especially keeping in mind the (important) point that it was _not_ our goal to create an efficient “real-world-applicable” model, but rather investigate the factors leading to the success of ViT- and MLP-Mixer-style models.  
  &nbsp;

   Hopefully this discussion doesn’t come across as exceedingly defensive -- we just wanted to make our thoughts on ConvMixer’s throughput visible.

    - _**Proposed changes:**_ We would be glad to hear your thoughts, with the above comments in mind, on the discussion of throughput in the main text. While it’s currently unclear exactly what changes we should make to improve this, we will continue thinking about it and post an update here.

Thanks again for your thoughtful review and suggestions! 


**[Note on ConvMixer Throughput]** Some additional details on throughput: ConvMixers use a blend of depthwise (kernel size = large, groups = # channels), and pointwise (kernel size = 1, groups = 1) convolution to do across-patch and per-patch mixing respectively. The complexity of a pointwise convolution is O(patches * channels^2) and for a depthwise convolution it is O(patches * channels * kernel_size^2). For the ConvMixer-1536/20, patches=1024, channels=1536 and kernel_size=9, so depthwise convolution is _19X fewer_ FLOPS than pointwise. But the depthwise convolution is 1.5x-2x _slower_ than the pointwise one, mainly (we believe) because the CUDA operators for depthwise convolution have not been optimized.  If we could get the same FLOP efficiency for depthwise convolutions, ConvMixers could be 2-2.5X faster. Now, of course this is not the whole story.  It may be that, e.g., cache locality makes it fundamentally harder to optimize large-kernel depthwise convolutions well.  We honestly don't know yet. (So reporting FLOPs instead of throughput also seems disingenuous.) But there does seem to be a _ton_ of room for improvement, _and_ then in just doing normal architecture tuning (patch size, depth, channels, etc.).



 Thank you for your review! We appreciate you bringing up some nice discussion points and suggestions for improving the paper. We’ll address those point-by-point below along with proposed changes, but we’ll present an actual updated version of the paper with any proposed changes by the deadline on the 22nd.

In particular, we appreciate that you pointed out our reproducibility efforts -- we have even merged our model into timm itself (anonymously) to ensure that the community can pick up where we left off and run additional experiments as easily as possible, and we’re continuing to take suggestions on GitHub to make our project as reproducible and transparent as possible.

**A note on paper length:** Thanks for taking the time to read the appendix! Were there any points in particular that you think the main text is missing? While we intended to put all of our important points in the main text, and believe that a short paper is sufficient to present our work, we may not have achieved the optimal presentation -- we hope to improve this through the comments, suggestions, and hopefully continued discussion here -- and thus hopefully demonstrate that some conference-level ML research can be communicated effectively via short papers. 


1. We agree that it is indeed hard to disentangle the exact effect of patches, and just to be clear, our work certainly does not prove (or claim to prove) that patches are behind the performance of ViT and MLP-Mixer-style models. We tried to convey this uncertainty with our language in the paper, e.g., including a question mark in the title; we more so just wanted to point out that the success of these models is possibly due to a factor other than the novel operations that they introduce. After replacing such novel operations (e.g., self-attention or MLPs) with convolution, the main “novel” change in these architectures that sticks out is the use of patch embeddings (which leads to, e.g., isotropic architectures).   
   &nbsp;

   Designing an experiment that more explicitly disentangles the effect of patches from other things seems nontrivial to us, but it would definitely be an interesting (and important) future direction that we plan to pursue.
   - _**Proposed changes:**_ We will mention more clearly in the main text that it is hard to exactly disentangle the effect of patches, and that more detailed ablations are suggested as future work.

2. We absolutely agree that this is interesting future work which we unfortunately do not currently have the resources to pursue. In the case of vision-transformer-scale pretraining, we simply don’t have access to the JFT datasets -- and if we had pretrained on ImageNet-21k (at prohibitive cost), we would probably still have the same “threat to validity” due to not having JFT300M. It’s not obvious to us if the stronger inductive bias of ConvMixer would be better or worse than more “flexible” operations in the very-large-data regime; we think that convolution comes with intuitively reasonable inductive biases, but we obviously cannot make any conclusions without actually running the experiments.

    - _**Proposed changes:**_ We will mention large-scale pretraining of ConvMixers as proposed, especially important, future work.

3. While we agree that the CIFAR-10 results are secondary to the point of our paper, we think they are unusually good and expected that readers would be interested in them. Consequently, we wanted to at least make them visible in the main text but not say very much otherwise. But we generally agree with your point here and will look into moving this paragraph to the end of the experiments section rather than the beginning for better flow.

    - _**Proposed changes:**_ We will move this paragraph to the end of the section to improve flow (as well as make any other additional language changes to supplement this goal), and would be glad to hear any additional thoughts on this.

4. Definitely. While we probably are not able to explore all of these tasks, like you said, the community may fill in some of the gaps, which we think is great.

5. Agreed, semantic segmentation seems like a particularly good use of ConvMixers -- though of course we do not yet have any experiments on this.
    - _**Proposed changes:**_ Again, we will point out these additional experiments as future work, but we definitely do not currently have the resources to do these kinds of experiments in a useful timeframe for the discussion period.

 Thanks very much for your comments.  At some level, we think that we actually don’t disagree all that much about several aspects of the paper and its contributions. Specifically, we are fully aligned in that the benefits of our architecture lie in its simplicity and understanding of how depthwise convolution with large kernels (plus pointwise convolution) applied to patches can capture some of the same aspects as ViTs/MLP-Mixers.  We believe that the disparity between our assessments comes largely from different interpretations about the ultimate conclusion of our method, and (we believe) possibly some misunderstandings about the comparisons. We’ll try to address all of these below, because we absolutely **do not** want people to take away some of the conclusions you mentioned.  We will update the paper with these changes by the 22nd, but we wanted to first discuss them here, to determine whether they are sufficient in your opinion, or if you have other things in mind. 

**Strengths:**

3. If we understand correctly, you’re absolutely right that a key portion of the ConvMixer is precisely that its large receptive field side moves _more_ toward that global context used by ViTs/MLP-Mixers.  We definitely do want to emphasize this point (and thought we did at least to some extent), but you’re right that it likely wasn’t emphasized enough. It’s indeed a key aspect of the similarity we want to focus on, so we’ll update it.  Or are we misunderstanding your comment here?
    - _**Proposed fix:**_  Update text to emphasize the similarities in large-kernel depthwise convs and the global mixing provided by ViTs, MLP-Mixers.

**Weaknesses:**

1. Thanks for pointing these out, we’ll fix them.

2. We agree with you that there are potentially other confounding factors that prevent us from proving this. However, we genuinely attempted to be careful to avoid “over claiming” as you said: we merely tried to point out that our results _suggest_ that the patch representation _could_ be a critical component of the performance of ViT and its derivatives: once we replaced the “more expressive” operations with convolutions, the patch embedding layer was the main difference that stood out in contrast to traditional vision models. We put a question mark at the end of our title to highlight this uncertainty, and obviously also as a reference to the attention paper (and in the conclusion stated “Our title, while an exaggeration, points out that attention isn’t the only export from language processing into computer vision: tokenizing inputs, i.e., using patch embeddings, is also a powerful and important takeaway.”).  Alternatively, one could claim that it’s the fact that the architecture is isotropic rather than the fact that it uses patches -- but in our opinion, replacing the patch stem with a different (e.g., multi-layered convolutional) one results in a fundamentally differently “shaped” network that could no longer be called “isotropic”.  
    &nbsp;
    
    We’re not sure which parts of the paper made you believe that we overclaimed here; even in the sentence you quoted, we tried to express our uncertainty by using language such as “suggests”, “to some extent”, and “may”.  We genuinely tried, including, e.g., in the conclusion text mentioned above, to emphasize as much as possible that the current paper provides an interesting insight, not that it “proves” patches are in any way truly “all you need.”  
    &nbsp;
    
    Despite these attempts, upon re-reading it might be that the term “the most critical” in the sentence you mentioned is the problematic one here, so see the fix below.

    - _**Proposed fix:**_ Perhaps it would be more clear if we changed “the most critical component” to “_a_ critical component”? We really do not want readers to think that we claimed that we “proved” the patch representation is key, so it would be great to discuss how we can clear up this confusion in the text.


 
3. We think it’s important to highlight that whether or not ConvMixer is “better” than other models is actually secondary to the main point of the paper, which is (as we have mentioned in other responses here) that we think it is important to consider the effect of the input representation and the factors besides self-attention in vision transformers, and believe that this paper will generate important discussion and serve as a stepping-stone for these points. We merely think that ConvMixer is _good enough_ to provide strong evidence for this point; we were never aiming for real-world applicability, but rather just investigating why ViTs and MLP-Mixers work so well.  
  &nbsp;

   (i). ConvMixer is indeed quite slow, though we think this could be improved somewhat by optimizing large-kernel depthwise convolution (which we are still looking into). The goal of our work was not to create a highly efficient model that can be used in real-world scenarios; that is, we strictly wanted to investigate the factors behind the success of ViT, MLP-Mixer, and their derivatives. There are _numerous_ ways one could make ConvMixer less simple in exchange for enhanced accuracy and/or speed, but we feel that these actually get in the way of the point we tried to make. 
      - _**Proposed fix:**_ We will try to make our intentions more clear, i.e., that we were not trying to make the “best” or most performant model, but rather study the factors leading to the success of recent ViT- and MLP-Mixer-styled models. It would be great to continue to discuss how we could make it more clear, e.g., that we did not intend to make a model that is optimized for real-world use.  
 
   (ii). We agree that our ResNets were not trained in the most optimal way possible, and we have read and appreciated “ResNet strikes back”. However, we _did_ train the ResNets in our paper in _exactly the same way as the ConvMixers_. We have no reason to think that the training setup choices were optimal for ResNet _nor_ for ConvMixer, as we only picked common sense default settings for a modern vision model training setup and used them across all experiments, finding that we achieved “good enough” accuracy on ConvMixer to elicit interesting discussions. Again, we emphasize that we did _absolutely no tuning_ for either ConvMixer or ResNet in our ImageNet experiments. In contrast, “ResNet strikes back” [2]  is entirely focused on perfecting the training regimen for ResNets -- it could be that if similar efforts were invested into ConvMixer, it would still outperform ResNets. Also, it’s worth noting that despite the _significant_ effort put forth to optimize ResNet training in [2], their best ResNet only does around 0.2% better than a comparably-sized ConvMixer which we _did not tune at all_. Also, note that [2] was released **one day** before the ICLR deadline, so we were not aware of it to discuss it in our paper.  
   - _**Proposed fix:**_ We will make the precise details of our training setup (i.e., no parameter tuning; exactly the same training setup for the ResNets) more clear in the main text, and will also mention [2].   

   (iii).  It’s not entirely clear to us that decreasing the patch size would increase the accuracy -- it would definitely make the network less efficient. For example, ResMLP tried decreasing the patch size and did not get any performance improvements. We have not tried a patch size smaller than 7 due to compute constraints. We do, however, think that larger patches could achieve higher accuracy than we reported if effort were put into parameter tuning for that setting -- and larger patches result in substantially more compute-efficient ConvMixers.  
   &nbsp;

     We didn’t think it was salient to compare the Swin transformer since its “Swin Block” is substantially more complicated/engineered than ours; we only intended to compare to the most “basic” instantiations of isotropic, patch-based architectures. However, note that Swin-S achieves 83% top-1 accuracy for 50M parameters when trained on ImageNet-1k only (of course, Swin-S also has better throughput than ConvMixer). While this is around 1.6% better than ConvMixer, they also trained for twice as long (300 vs. 150 epochs), and we do not know how much parameter tuning went into this result. However, simply taking the reported results at face value, it’s not surprising that there are more performant blocks than the extremely simple block used by ConvMixer; it is _still surprising_ to us that _plain convolution_, within a patch-based architecture, does not lag behind Swin by very much. We never sought “SOTA” accuracy with ConvMixer and it still does pretty well.

      - _**Proposed fix:**_ Given the interest in comparing to Swin, we will add it to the larger results table.


 4.  We admit that this may not be true -- we are merely optimistic based on the results we have seen training a handful of ConvMixers. Indeed, we have seen from [2] that putting more effort into perfecting the training setup and hyperparameters of a particular model can lead to large gains in accuracy. We mentioned increasing the patch size since this is one simple way to narrow the gap between ConvMixer and ResNet throughput, and we mentioned additional parameter tuning since this would likely result in better accuracy.
    - _**Proposed fix:**_ We will make the fact that we did no tuning, and the consequences of this (like being optimistic that tuning would help), more clear in the main text. 
5. We agree that it would be interesting to try to add the second residual connection back in for the ImageNet setting, since our findings on CIFAR-10 do not necessarily generalize to ImageNet-1k. 
    - _**Proposed fix:**_ We will look into running this experiment and including it in the paper; while it may not be possible given the timeframe and our compute constraints, we will attempt to do this before the 22nd.
6. We agree that results on CIFAR-10 are no longer a very relevant benchmark for “unconstrained” image classification -- but we also acknowledge that, historically, there is some degree of correlation between CIFAR-10 performance and performance on more complicated tasks. But we avoided saying very much about CIFAR-10 in the main paper because we are aware of its shortcomings. We agree that it may have been helpful to include an intermediate step where we tuned ConvMixers on these smaller datasets, though again, we were not searching for the best accuracy possible, and achieved interesting results on ImageNet itself without such searching.
    - _**Proposed fix:**_ Keep the content the same (CIFAR-10 results aren’t really part of our main argument, and just an interesting detail which we put in the appendix), but move the mention to slightly later in the experimental results section, as we indeed to do not want to make them the focus of the paper.
7. We understand your point here, but we did not have the compute to try ConvMixer on a wide variety of tasks and parameter settings, and believe that even just the ImageNet-1k classification results are sufficiently surprising to generate interesting discussions at ICLR.  
  &nbsp;

   To be clear, we primarily intend the “note on paper length” to clarify that our unconventional presentation is deliberate, and in our opinion, optimal for this particular study. We did not, e.g., want the reviewers and AC to think that this was a half-finished or hastily-resubmitted workshop paper; to the contrary, we intentionally invested effort into writing a clear and concise paper that says everything we found particularly important in the main text. (And for that matter, we hope our discussion here leads to an even more clear and concise short paper.)
8. We agree that more detailed ablations would be nice; we tried to do our best with our limited compute budget. In particular, we trained a ConvMixer-1536/20 with both large and small kernel sizes and reported the results (which indicates large kernel sizes are important), and we trained some ConvMixers with larger patches. We agree that we do not have a great comparison for shallow vs. deep ConvMixers without simultaneously changing other parameters, and we will try to run a relevant experiment. However, since submitting this paper, we did run ConvMixer-1536/20 with a larger patch size (14 instead of 7), all other settings held equal, achieving 78.92% top-1 accuracy while being significantly faster (by around 3-4x, though we do not currently have access to the same type of GPU that we used for throughput measurements in the paper). We will include this in the final version.
    - _**Proposed fix:**_ We will include our new result for the large-patch ConvMixer-1536/20 in the revision, and attempt to run experiments that help investigate depth (e.g., adding a ConvMixer-768/24 and a ConvMixer-768/40). We likely will not have sufficient compute to do both of these before the 22nd, but we will try.

At a high level, we think your review reflects some misunderstandings about our training setup and motivation behind the comparisons -- and we want to improve the paper to make these aspects more clear, so we will absolutely change portions which cause confusion. We are optimistic that we can make these improvements through our continued discussion here, and we appreciate your time in making these comments.

 Finally, from a slightly higher level, we also want to point out that we did not aim to _prove_ that the patch representation is the key. Indeed, we tried to be very careful to not make any strong claims in this regard, and even have a question mark in the title to make this uncertainty clear. As stated in the conclusion: “Our title, while an exaggeration, points out that attention isn’t the only export from language processing into computer vision: tokenizing inputs, i.e., using patch embeddings, is also a powerful and important takeaway.”  The key takeaway is that we think that it is important to consider the effect of the input representation and the factors besides self-attention in vision transformers, and believe that this paper will generate important discussion and serve as a stepping-stone for these points.

- _**Proposed fix:**_ We’re a bit unsure of which portions of the paper led you to believe that we were really trying to make the claim that we are “proving” that the patch representation is the key. We definitely do _not_ want to come across as making this claim, so will absolutely change portions that suggested it to you.

 Thanks very much for your comments. We'll try to address the majority of these, along with proposing changes, in the hopes that you'll increase your evaluation of the paper.  Please let us know if you think these comments and proposed changes address your concerns, or if there are other aspects you’d like to discuss.  For now we're just going to state the changes we make here for the purposes of discussion, but we'll present an updated version of the paper with these changes by the 22nd deadline.

- First and foremost, we need to emphasize a potential point of misunderstanding in your review. You stated that:  
 

     _"Compared to other previous convolution-based network architectures, such as ResNet, the ConvMixer has too many factors beyond the patch representation itself that may lead to performance differences, such as data augmentation, appropriate hyper-parameters, ..."_  
 

    While it would be extremely difficult to train every model in our comparison from scratch, we need to emphasize that (as stated in the "Comparisons" paragraph on p.3), **we trained ResNets using the exact same settings (data augmentation, optimizers, etc.) as ConvMixer**. And the overall training setup is very similar to that used for DeiT, ResMLP, and other modern vision models, with the largest difference (in our opinion) being that we trained for _fewer_ epochs. Specifically, we trained ResNets using exactly the timm code base, using a recipe that was indeed specialized to a large degree for ResNets. It’s important to also emphasize that (as also mentioned in Training Setup paragraph on page 3), we did **no parameter tuning** for ConvMixer and only chose common sense defaults.  The paper [1] that you mention _also_ uses the timm framework, but trains for many more epochs (600 vs. the 150-300 we use), adds many additional components, and still only improves performance over our equivalently-sized ConvMixer by 0.2% accuracy.  We should also emphasize that this paper was published on Arxiv **one day prior** to the ICLR deadline, and thus really can't be considered "prior" work as far as this submission goes, though we agree it's worth mentioning in the paper.  And while it’s true that comparing our convolutional block to that of a ResNet is naturally comparing across different models to some degree, we think it is standard in the literature to compare ResNet with models that are very different (e.g., in the case of ViT, MLP-Mixer, and its derivatives).  

  - _**Proposed fix:**_ We propose to clarify more the precise training process, especially with regards to the ResNet recipe we used.  We will also absolutely mention the results in the paper [1], in the context as discussed above.


- We indeed limited the number of comparison points in the paper itself.  Because our overall effort is not to argue that “ConvMixer is the best-performing current model”, but rather that “ConvMixer is a simple architecture that uses the patch-based representation adopted by ViTs/MLP-Mixers within a convolutional architecture leading to promising initial performance”, we tried to compare to models that we viewed as conceptually similar, such as DeiT and ResMLP, which are also similar in their simplicity and design.  To be fair, we could also add e.g., MLP-Mixer or ViTs directly: ConvMixer works much better than these when trained on ImageNet-1K alone, but to be fair, these two papers both didn’t tune for this setting and instead relied on large-scale pretraining, so this seemed to be an unfair comparison as well.  Instead, we compared to representative MLP-Mixer- and ViT-style models that _are_ specialized to this size of data, which struck us as the best possible comparison.  We do of course compare to other results in the appendix (but not versions that are intended largely for pre-training), and can, e.g., add Swin Transformer to that table as well.  

   - _**Proposed fix:**_ Add more discussion about the rationalization for comparison points in the main text, and add Swin Transformer as an additional comparison point in the larger results table.


- While it would have been great to run more experiments on a wider array of tasks, e.g., object detection and semantic segmentation, the reality is that we feel this is honestly outside the scope of the current paper.   Other reviewers have even pointed out that e.g., ConvMixer may be especially well-suited to semantic segmentation due to its large internal resolution and isotropy.  However, we simply did not have the compute to run more extensive experiments at the time, and feel that it’s okay for this to be left to future work.  

    - _**Proposed fix:**_ No additional experiments, but we’ll add a mention of these settings for future work and possible directions under which to proceed in the final paragraphs of the paper.

This work proposes a new CNN architecture: ConvMixer, which integrates some techniques/techniques used in the recently proposed visual transformer model/mlp model. ConvMixer is simple and effective, and it also prompts us to think deeper about where is key to improving performance in the vision transformer and mlp models. Pros:
Well written. Clear motivation and the problem raised in this work is very interesting.

Cons:

a. Although the proposed ConvMixer illustrates very promising results compared with its counterparts, such as DeiT, MLPMixer, and ResNets, the main claim of this work:  "the patch representation may be the most critical component" has not been sufficiently well validated: 
- Compared to other previous convolution-based network architectures, such as ResNet, the ConvMixer has too many factors beyond the patch representation itself that may lead to performance differences, such as data augmentation, appropriate hyper-parameters, optimizer and other macro/micro network architectures, For example, the ConvMixer adopts the spatial-conv + pointwise-conv as its basic building block, which is also an unfair comparison with ResNets. In addition, through carefully tuning, [1] reports a much higher ResNet performance than this work.

- Compared to vision transformer/mlp-based networks, this work only considers some computation/parameter unfriendly architectures, such as DeiT, while ignoring some recent progress in this field, like Swin. 

- Lack of comparison for downstream tasks, such as object detection/semantic segmentation. This is necessary to fully understand the importance of patch representation in a broader range of tasks.

These under-explored factors challenge the conclusion/connection of this work that the 'patch representation' is the key.
 
[1] ResNet strikes back: An improved training procedure in timm
 This work is well written and presents a very interesting conjecture/conclusion that the patch representation may be the key to the success of visual transformers and visual Milp. However, its research is not strong enough to support its claims.

The paper presents a very simple architecture which consist of patching the input image and then applying a combination of depth-wise and point-wise convolutions. In the paper, authors evaluate the performance of this model when used for image classification. In their main experiment, they train the architecture using only the ImageNet-1k dataset. On that experiment, they show how their simple architecture is competitive with state-of-the-art architectures.  **Summary**: Overall, I believe this is a good paper. It consist on a simple model which is very well described and one single experiment which shows well the advantages of the paper. However, I believe the authors are missing a few points which are worth addressing before the paper is published.

**A note of the paper length**: I understand the message from the authors about the paper length, but as a reviewer, I needed to check the Appendix many times to produce a complete review of the paper. Although I can agree this is a good discussion to have as a community, I don't think the note from the authors is fair as if reviewers only looked at the main paper (which is supposed to be sufficient for a full understanding of the paper), many relevant points would be missed. For this reason, I would recommend the authors to move to the main paper some sections which provide relevant insights (for instance, Appendix A gives very relevant details). 

**Strengths**:
1. The model is very simple and effective. I agree with authors that this is a relevant piece of research for the community, as it disentangles a bit the performance improvement from the transformer architecture.

2. In terms of reproducibility, I believe this can be a great baseline for future research. Code is simple and authors seem to imply that it does not need a lot of hyper-parameter tuning to make it work.

3. The model description is clear and well written. I think the figures and the text are very clear and the community will really appreciate that.

4. The experimental results on ImageNet-1k are very good. Specially, when taking into account the parameter count, the model seems to show very good performance overall.


**Weaknesses and proposals for improvement**:

1. Patches and architecture: authors introduce the model as validation of the idea of the patches being important for final performance on SOTA architecture such as transformers (see title and abstract). However, in the paper they introduce the idea of using patches in convolutional networks along with a novel architecture, but they never decouple the gains of this two contributions in the experimental section. If authors want to claim that patches are relevant for final performance, in my opinion they should ablate only that change in an already known architecture. 

2. Although the experimental results in the paper are encouraging, in my opinion authors are missing a very relevant point about pre-training. It is clear that transformers do need large scale pre-training for them to get high performance. I would be very curious to see this model operating on that data regime. Otherwise, I am not sure if we can conclude much, as it was already known that these large models were not very good with regimes without much data. 

3. The first paragraph of the experimental section is a bit confusing for me. If authors want to include the CIFAR experiments in the paper, they should just include that table in the main paper. If they think those experiments are not relevant, then they should not start the experimental section by discussing those experiments. But providing the table in the Appendix but some discussion in the main paper is very confusing for the reader. 

4. Other than large-scale pre-training, I would be curious to see the performance of this model on other classification task. Authors include CIFAR, which is a small dataset. But I think it would be useful for the community to benchmark the proposed model to other classification tasks such as scene classification, fine-grained recognition, etc. Essentially, evaluate the ability of the model to perform on other domains and make sure the results are not overfitting to object classification.

5. Following up with the previous points, isotropic models should be good to perform segmentation-like tasks, as they keep most of the input resolution (except for the patching). I am wondering if that could be an additional experiment to include. 

6. Looking at the results, it is clear that the model has lower throughput than its competitors. Figure 1 revolves around the parameter count of the model, but I believe it would be fair for the authors to include also some discussion about the throughput. I realised they had included some in the Appendix, but I believe authors should include a proper discussion on the effects of this low throughput in the main paper, as I believe it's one of the main weaknesses of the model. 
 I believe the paper is good and it should be accepted. It will be a relevant contribution to the community as a baseline as well as a good datapoint to improve our understanding of SOTA models. However, I believe the paper would be more complete if authors evaluated in additional task. Furthermore, I think authors should address some relevant points when comparing to other models such as data scale and throughput. 

Meta-review: This paper observes that a fully-convolutional model in the style of recent MLP-Mixer and ViT variants can have surprisingly good initial performance. As this paper attracts certain amount of attentions, three expert reviewers have provided very detailed and serious comments, and two actively engaged with author discussions. AC also carefully read the paper as well as all discussion threads.

AC agrees the authors should not be penalized by not achieving the best performance, nor not comparing with very recent work. The main legitimate critiques, however, focus on three aspects: (1) over-claimed contribution; (2) experiment solidness/competitiveness; and (3) writing completeness/clarity.

First, this paper established an interesting ablation experiment that a very simple model, that uses only standard convolutions to achieve the mixing steps, can roughly "do the work". However, AC disagrees this is a very "surprisingly new" result, on top of MLP-mixer: given convolutions are increasingly re-injected into ViTs to gain the vision inductive bias, their similar role in MLP-mixer should be expected too. Moreover, as in general agreement by reviewers, the paper title might have over-claimed - the authors cannot directly prove this concept "patch is the most critical component" yet. The authors later also agreed and changed some confusing wording, which is a good move (but also, making their contribution now even less obvious). 

Second, this method does not achieve noteworthy competitive results compared to others, in order to justify its merit (simplicity alone is good to have, but insufficient to justify a strong work). Importantly, it has been pointed out by two reviewers that the model throughput is much worse than the competitors. AC also noticed that the comparison was not very rigorous, e.g., comparing ConvMixer patch size 7 with DeiT-B 16 patch size 16 doesn't help draw much fair informative conclusion. The cifar-10 results alone did not provide strong support and were later de-emphaszied by authors too. 

Third, while NOT being the main reason of rejection, AC personally suggests the authors to responsibly enrich their main text, and to remove the  “A note on paper length” paragraph. The authors intentionally kept the paper length unusually short. Reviewers generally dislike this idea. Being an innovative writer is good, but very relevant details and discussions were left in the supplemental as a result. Especially, AC agrees the whole section A and part of section B of the supplemental should have been in the main paper at very least.

In summary, the authors strive to tell an interesting story, but it is not yet a well settled story. The experiments are not solid enough to support their bold claims. The authors are suggested to improve their work further by taking into account reviewer comments.

Decision: Reject