Patches Are All You Need?
Authors: Asher Trockman
J Zico Kolter
Venue: Accepted by TMLR ()

Abstract: Although convolutional neural networks have been the dominant architecture for computer vision for many years, Vision Transformers (ViTs) have recently shown promise as an alternative. Subsequently, many new models have been proposed which replace the self-attention layer within the ViT architecture with novel operations (such as MLPs), all of which have also been relatively performant. We note that these architectures all share a common component--the patch embedding layer--which enables the use of a simple isotropic template with alternating steps of channel- and spatial-dimension mixing. This raises a question: is the success of ViT-style models due to novel, highly-expressive operations like self-attention, or is it at least in part due to using patches? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple and parameter-efficient fully-convolutional model in which we replace the self-attention and MLP layers within the ViT with less-expressive depthwise and pointwise convolutional layers, respectively. Despite its unusual simplicity, ConvMixer outperforms the ViT, MLP-Mixer, and their variants for similar data set sizes and parameter counts, in addition to outperforming classical vision models like ResNet. We argue that this contributes to the evidence that patches are sufficient for designing simple and effective vision models. Our code is available at https://github.com/locuslab/convmixer.

Published in Transactions on Machine Learning Research (03/2023)
Convolutions Attention MLPs Patches Are All You Need?
Asher Trockman, J. Zico K
Carnegic Mellon University and Bosch Center for AI
Reviewed on OpenReview: https: // openreview. net/ forun? id= Adn.77J5K
Abstract
Although convolutional neural networks have been the dominant architecture for computer vision for many years, Vision Transformers (ViTs) have recently shown promise as alternative. Subsequently, many new models have been proposed which replace the self-attention layer within the ViT architecture with novel operations (such as MLPs), all of which have also been relatively performant. We nole that these architectures all share a common component the patch embedding layer- which enables the use of a simple isotropic template with alternating steps of channel- and spatial-dimension mixing. This raises a question: the success of ViT-style models due novel, highly- operations like sell-attention, is it at least in part due to using patches? Tn this paper, present
evidence for the latter: specifically, extremely simple and parfeeter-efffiient fully-convolutional model in which we replace the self-attention and MLP layers within the ViT with less-expressive depthwise and pointwise convolutional layers, respectively. Despite its unusual simplicity, ConvMixer outperforms the VIT, MLP-Mixer, and their variants for similar data set sizes and parameter counts, in addition outperforming classical vision models like ResNet. We argue that this contributes to the evidence that patches sufficient for designing simple and effective vision models. Our code is available at attps://e //gi http://///.ccclaacccooaaiiaaaaaa
propose the ConvMixer,
1 Introduction
For many years, convolutional neural networks have been the dominant architecture for deep learning systems applied to computer vision tasks. But recently, architectures based upon Transformer models, e.g., the so-called Vision Transformer architecture (Dosovitskiy et al., 2020), have demonstrated compelling performance in many of these tasks, often outperforming classical convolutional architec- tures, especially for large data sets. An understandable assumption, then, that it only a matter of time before Transformers become the dominant architecture for vision
ImageNet Top-1 Accuracy (%)
82
76
74
domains, just as they have for language processing. In order to apply Transformers to images, however, the representa- tion had to be changed: bocause the computational cost of the self-allention layers used in Transformers would scale quadratically with the number of pixels per image if applied naively the per-pixel level, the compromise was to first split the image into multiple "patches", linearly embed them, and then apply the transformer directly to this collection of patches.
72
20
40
60
80
100
120
Parameters (Millions)
ConvMixer ResMLP DeiT ResNet
Figure 1: Acc. us. params., trained & tested on LnNet-lk: ResNets newly-trained (same procedure as ConvMixers).

---

Published in Transactions on Machine Learning Research (03/2023)
ConvMixer Layer
Residual connection
Global Average Pooling
Embedding
Fully Noy-Connectedd
Batch
Deptioriss
Convolltion
BatchNorm
Polntwie
Canvollution
Batch
*
Cless
GELU
72
n/p n/p
X depih
Figure ConvMixer uses "tensor layout" patch embeddings to preserve locality, and then applies d copies of simple collouoonvlltional block consisting of large-kernel kernel depthwise convolution followed by pointwise convolution, before finishing with global pooling and simple linear classifier.
import torch.nn nn
class Residual (an Module): def (self, fn): super _i__iiit___)))) self fn in
def (srl,, x): return self fx(x) +
def ConvMixer (dim, depth, kernel_size=9, patch_size=7, c_classep=1000): return nn. Sequential(
(3, dim, harnel_size CELU(), (dtch)
prrch_spatcc_size)
15
16
[[nn. Senuuntnntt
17
Residual (nn. Sequential(
18
(dim, dim, hernel_size, groupa-din, padding""sane") GELU(),
19
20
an Batchlormmdd(imn)
21
))
22
nn (din, dim, kernel_size=1),
28
nn. GELUO,
24
nn. Batchllorm22dd1in)
for in range (depth)). nn sdaptiieerrpPoo222(((11))) nn Flatten(), nn Lin.Linea (din, r_classes)
Figure 3: Implementation of ConvMixer in Pyloorch; see Appendix E for more implementations.
Many subscquent works have modified the architccture of the VIT, replacing self-attention with novel oper- ations and making other small changes, all of which have been relatively performant. These architectures follow common and very simple "template": they are isotropic, maintaining equal size and resolution throughout the network, and apply alternating steps of spatial and channel mixing. They also all use patch embeddings, which moves all downsampling to the beginning the network and enables the simple, isotropic mixing design.
In this work, we explore the question of whether, fundamentally, the strong performance of vision transform- may result more from this patch-based representation and its simplifying consequences for architecture design, than from the use of novel and develop very simple convolutional architecture which we dub the "ConyMixer" due to its similarity to the recently-proe pyreoooypropose MLP-Mixer (Tolstikhin et al., 2021). This architecture is similar to the Vision Transformer (and MLP-Mixer) in many respects: it directly operates on patches, maintains an equal-resolution and- size representation throughout all layers, does no downsampling of the representation at successive layers, and separates "channel-vii mixing" from the "spatial mixing" of information. But unlike the Vision Transformer and MT.P-Mixer, our architecture does all these operations via only standard convolutions. As
operations such self-attention and MLPs. We
2

---

Published in Transactions on Machine Learning Research (03/2023)
depthwise and pointwise convolution are less expressive than self-attention and MLPs respectively, we believe this suggests that the patch-based isotropic mixing architecture is a powerful primitive that works well with almost any choice of well-behaved mixing operations.
The chief result we show in this paper is that this ConvMixer architecture, despite its extreme simplicity (it can be implemented in 280 characters of dense PyTorch code; see Appendix E), outperforms both "standard" computer vision models such as ResNets of similar parameter counts and some corresponding Vision Transformer and MLP-Mixer variants, even with slate of additions intended to make those architectures more performant on smaller data sets. Importantly, this is despite the fact that we did not design our experiments to mazimize accuracy nor speed, in contrast to the models we compared against. Our results suggest that, at least lo some extent, the patch represenlation ilself may be a critical component to the "superior" performance naturally just a snapshot, and more experiments are required to exactly disentangle the effect of patch embeddings from other factors, we believe that this provides strong "convolutional-bui but.plthhtttt------aaaa-aaa but-patchh-bsed" based" baseline to compare against for more advanced architectures in the future.
newer architectures like Vision Transformers. While these results are
2 A simple model: ConvMixer
Our model, dubbed ConuMizer, consists of patch embedding layer followed by repeated applications of simple fully=-convouttionall block. We maintain the spatial structure of the patch embeddings, as illustrated in Fig. 2. Patch embeddings with patch size and embedding dimension h can be implemented as convolution with Gin input channels, h output channels, kernel size p, and stride p:
2n BN (a (C -
(1)
The ConvMixer block itself consists of depthwise convolution (i.e., grouped convolution with groups equal to the number of channels, h) followed by pointwise (i.e., kernel size 1 x 1) convolution. As we will explain in Sec. 3, ConvMixers work best with unusually large kernel sizes for the depthwise convolution. Each of the convolutions is followed by activation and post-activation BatchNorm:
2/ = BN (a ((anwDDepttuisse(==1))))) + 21-1
(2)
11+1 BN (a [ConPPointwisee
After many applications of this block, we perform global pooling to get a feature vector of size h, which we pass to softmax classifier. See Fig. 3 for an implementation ConvMixer in PyTorch.
Design parameters. An instantiation of ConvMixer depends on four parameters: (1) the "width" hidden dimension h (i.e, the dimension of the patch embeddings), (2) the depth d, or the number of repetitions of the ConvMixer layer, (3) the patch size p which controls the internal resolution of the model, and (4) the kernel size k of the depthwise convolutional layer. We name ConvMixers after their hidden dimension and depth, like ConvMixer-h/dd. We refer to the original input size n divided by the patch size p as the internal resolution; note, however, that ConvMixers support variable-sized inputs.
Motivation. Our architocture is based on the idea of mixing, as Tolstikhin et al. (2021). In particular, we chose depthwise convolution mix spatial locations and pointwise convolution mix channel locations. A key idea from previous work is that MLPs and self-attention mix distant spatial locations, i.e., they can have an arbitrarily large receptive field. Consequently, we used convolutions with an unusually large kernel size to mix distant spatial locations.
While self-attention and MLPs are theoretically more flexible, allowing for large receptive fields and content- aware bohavior, the inductive bias of convolution is well-suited to vision tasks and Ieads to high data efficiency. By using such a standard operation, we also get glimpse into the effect of the patch representation itself in contrast to the conventional pyramid-shaped, progressively-downsann
design of convolutional networks.
3

---

Published in Transactions on Machine Learning Research (03/2023)
Table 1: Models trained and evaluated on 224 224 ImageNet- Ik only, See more in Appendix A.
Current " "Most Interesting" ConvMixer Configurations es. Other Simple Models
Patch Size
Kernel Size
# Params (x10%)
Throughput (img/ sec)
Act. Fn.
ImNet
Network
# Epochs
top-1 (%)
ConvMixer-15366
7
51.6
134
G
150
81.37
ConvMixer-7668/33
7
7
21.1
206
R
300
80.16
Conv-Mixer-1536/20
14
52.3
538
G
150
78.92
ResNet-152
60.2
828
R.
150
79.64
DeiT--
16
86
792
G
300
81.8
ResMLP-B21/8
8
129
181
G
400
81.0
3 Experiments
Training setup. We primarily evaluate ConvMixers on ImageNet-1k classification without any pretraining additional data. We added ConvMixer to the timm framework (Wightman, 2019) and trained it with nearly-standard settings for the common training procodure from this library: we used RandAngment (Cubuk et al., 2020), mixup (Zhang et al., 2017), CutMix (Yun et al., 2019), random erasing (Zhong et al., 2020), and gradient norm elipping in addition to default timm augmentation. We used the AdamW (Loshchilov & Hutter, 2018) optimizer and a simple triangular learning rate schedule. Due to limited compute, we did virtually no hyperparameter tuning on ImageNet and trained for fewer opochs than competitors. Consequently, our models could be over- or rndel--egularried, and the accuracies we report. likely underestimate the capabilities of our model.
Results. A ConvMixer-153 1 and a ConwMixer-768/32 with 21M parameters 80.2% (see Table 1). Wider ConvMixers seem to converge in fewer epochs, but are more memory- and compute-hungry. They also work best with large kernel sizes: NonvMixer-1536/ lost a 1% accuracy when reducing the kernel size from = to k 3 (we discuss kernel sizes more in Appendix A, B, C). ConvMixers with smaller patches are substantially better in our experiments, similarly to Sandler et al. (2019); believe larger patches require deeper ConvMixers. With everything held equal except incrcasing the patch size from 7 to 14, ConvMixer-15366 78.9% top-1 accuracy but is around 4x faster. We trained one model with ReLU to demonstrale that GELU (Hendrycks & Gimpel, 2016), which is popular in recent isotropic models, isn't necessary.
1536/20 with 52M parameters can achieve 81.4% top-1 accuracy on ImageNet,
achieves
Comparisons. Our model and ImageNetlk-only training setup closcly resemble that of recent patch- models like DeiT (Touvron el. al., 2020). Due to ConvMixer's simplicity, we focus on comparing to only the most basic isotropic patch-based architectures adapted the ImageNet-Iis 1k setting, namely DeiT and ResMLP. Attempting fair comparison with standard bascline, trained ResNets using exactly the same parameters ConvMixers; while this choice of parameters is suboptimal (Wightman et al., 2021), it is likely also suboptimal for Cixer, since we did no hyperparameter tuming aside from our recent adoption of hy- perparameters from Wightman et al. (2021) for some models (presented separately in Appendix A). Looking at Table 1 and Fig. 1, ConvMixers achieve competitive accuracies for a given parameter budget: ConvMixer- 1536/20 outperforms both R55, ResMLP--R2, and DeiT-B despite having substantially fewer param- eters. ConvMixar-768/22 uses just a third of the parameters ResNet-152,, but is similarly accurate. Notc that unlike ConvMixer, the Dei'T and ResMLP results involved hyperparameter tuning, and when substantial resources dedicated to tuning ResNets, including training for twice many epochs, they only outperform equivalently-sizer ConvMixer by 0.2% (Wightman et al., 2021). However, ConvMixers are substantially slower at inference than the competitors, likely due to their smaller patch size; hyperparameter tuning and optimizations could narrow this gap. For more discussion and comparisons, see Table 2 and Appendix A.
Hyperparameters. For almost all experiments presented the main text, we used only one set of "common sense" hyperparameters for the regularization methods. Recently, we adapted hyperparaneters from the A1 procedure in Wightman et al. (2021); published after our work, which better than
4

---

Published in Transactions on Machine Learning Research (03/2023)
our initial guess, e.g., giving -0.8% for Suw-Mixer-1536/22 or 82.2% top-1 accuracy (see Appendix A). However, we note that such optimal ResNet hyperparameters are likely not optimal for ConvMixers.
Additional experiments. We present additional ImageNet experiments in Appendix B; notably, we provide more evidence for the advantage of large-kernel convolutions. We also performed smaller-scale experiments on CTFAR-10, where ConvMixers achieve over 96% accuracy with as few as 0.7M parameters, demonstrating the data efficiency of the convolutional inductive bias. Details of these experiments are presented in Appendix C.
4 Related work
Isotropic architectures. Vision transformers have inspired a new paradigm of "isotropic" architectures, i.e,, those with equal size and shape throughout the network, which use patch embeddings for the first layer. These models look similar to repeated transformer-enoode blocks (Vaswani et al., 2017) with different operations replacing the self-attention and ML. operations. For example, MLP-Mixer (Tolstikhin et al., 2021) replaces them both with MLPs applied across different dimensions (i.e., spatial and channel location mixing); ResMLP (Touvron et al., 2021a) is data-efficient variation on this theme. CycleMLP (Chen et al., 2021), gMLP (Liu et al., 2021a), and vision permutator (Hou et al., 2021), replace with various novel operations. These are all quite performant, which is typically attributed to the novel choice of operations. In contrast, Helas-Kyriad, (2021) proposed an MLP-based isotropic vision model, and also hypothesized patch embeddings could be behind its performance. ResMLP tried replacing its linear interaction layer with (small-kernel) convolution and achieved good performance, but kept its MLP-basod cross-channel layer and did not explore convolutions further. As our investigation of ConvMixers suggests, these works may conflate the effect of the new operations (like self-attention and MLPs) with the effect of the use of patch embeddings and the resulting isotropic architecture.
both blocks
After our investigation, Liu et al. (2022) proposed an architecture similar to ConvMixer, the isotropic Nex Similarly to our work, they provide evidence that the success of ViTs comes from design choices other than the use of self-attention, such as patches; however, ConvMixer goes a step further and eliminates even the MLPs, which suggests that neither of the original ViT operations are crucial to the success of the more general architecture design. Further, Yu et al. (2022) replaced self-attention with a simple pooling operation and demonstrated its effectiveness; they also argued this supports the effectiveness of the VIT template. In contrast, our work suggests the template is even more general, not even requiring MLPs.
A study predating vision transformers investigates isotropic (or "isometric") MobileNets (Sandler et al., 2019), and even implements patch embeddings under another name. Their architecture simply repeats an isotropic MobileNetv3 block. They identify a tradeoff between patch size and accuracy that matches our experience, and train similarly performant models (see Appendix A, Table 2). However, their block is substantially more complex than ours; simplicity and motivation sets our work apart.
Patches aren't all you need? Several papers have increased vision transformer performance by replacing standard patch embeddings with different stem: Xiao et al. (2021) and Yuan et al. (2021a) use a standard convolutional stem, while Yuan et al. (2021b) repeatedly combines nearby patch embeddings. However, this conflates the effect using patch embeddings with the effect of adding convolution or similar inductive biases e.g., locality. We attempt to focus on the use of patches.
CNNs meet VITs. Many efforts have been made to incorporate features of convolutional networks into vision transformers and vice versa. Self-attention can emulate convolution (Cordonnier et al., 2019) and can be initialized or regularized to be like it (d' Ascoli et al., 2021); other works simply add convolution operations to transformers (Dai et al., 2021; Guo et al., 2021), or include downsampling to be more like traditional pyramid-shaped convolutional networks (Wang et al., 2021). Conversely, self-allention or allention-like operations can supplement or replace convolution in ResNet-style models (Bello et al., 2019; Ramachandran et al., 2019; Bello, 2021). While all of these attempts have boen successful in one way or another, they are orthogonal to this work, which aims to emphasize the effect of the architecture common to most ViTs by showcasing it with less-expressivee operation.
5

---

Published in Transactions on Machine Learning Research (03/2023)
5 Conclusion
We presented ConvMi an extremely simple class of models that independently mixes the spatial and channel locations of patch embeddings using only standard convolutions. We also highlighted that using large kernel sizes, inspired by the large receptive fields of ViTs and MLP-Mixers, provides a substantial performance boost. While neither our model nor our experiments were designed maximize accuracy or speed, i.e., we did not search for good hyperparameters, ConvMixers outperform the Vision Transformer and MLP-Mixer, and are competitive with ResNets, DeiTTs and ResMLPs.
We provided evidence that the increasingly common "isotropic" architecture with simple patch embedding stem is itself a powerful template for deep learning. Patch embeddings allow all the downsampling to happen at once, immediately decreasing the internal resolution and thus increasing the effective receptive field size, making easier to mix distant spatial information. Our title, while an exaggeration, points out that attention isn't the only export from language processing into computer vision: tokenizing inputs, i.e., using patch embeddings, is also a powerful and important takeaway.
While our model is not state-of-the-art we find its simple patch-mixing design to be compelling, We hope that ConvMixers can serve as bascline for future patch-bascr architectures with novel operations, or that they can provide a basic template for new conceptually simple and performant models.
Given that such simple architectures as ConvMixer can be successful, we question the role of continued architecture searches; in particular, are more complicated architectures fundamentally better modeling phenomena, or are they ultimately just more computationally efficient? Much of the variance in accuracies may be explained by more advanced training pipelines and augmentation techniques, demonstrated by Wightman et al. (2021) and our work.
Future work. We are optimistic that a deeper ConvMixer with larger patches could reach desirable tradeoff between accuracy, parameters, and throughput after longer training and more regularization and hyperparameter luning, similarly to how Wightman et al. (2021) enhanced ResNet performance through carefully-designeed training regimens. Low-level optimization of barge-kerned depthwise convolution could substantially increase throughput, and small enhancements to our architecture like the addition of bottlenecks more expressive classifier could trade simplicity for performance.
Due to its large internal resolution and isotropic design, ConvMixer may be especially well-suited for sernantic segmentation, and it would be useful to run experiments on this task with a ConvMixer-like model and on other tasks such as object detection. More experiments could be designed to more clearly extricate the effect of patch embeddings from other architectural choices. In particular, for a more in-depth comparison ViTs and MIP--Mxxrs, which excel when trained on very large data sets, it important investigate the performance of Mixers in the regime of large-scale pre-training.
More work is necessary to extricate the effect of the patch embeddings from the rest of the architecture. In particular, we have preliminary evidence that it is not necessary to separate the spatial and channel mixing steps; patches followed by any stack of nonlinear operalions (say, plain convolution) may be sufficient for simple, performant models.
Note on paper length. We acknowledge that this paper is shorter than most, and this intentional. In the main texl, we have presented our main thesis, proposed an extremely simple architecture used to validate the thesis, included complete implementation, highlighted the results that we believe to be most relevant, and finished with concluding thoughts. The work here is very simple, and thus we believe that a short paper is ultimately more effective at conveying the main messages. While additional experiments and results are included in the appendix, we fully argue that the results in the main text are sufficient to establish our point, and that the supplementary material is genuinely of secondary importance. Hence, we felt the shorter length was more than sufficient.
References
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.06600 2021.
6

---

Published in Transactions on Machine Learning Research (03/2023)
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoe V Le. Attention augmented con- volutional works..s. In Proceedings of the TEEE/CVI international conference on compuler vision, pp- 3286 3295, 2019.
Shoufa Chen, Enze Xie, Chongjian Ge, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prodiction. arXiv preprint 0rXic:2107.00224 2021.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self-attention and convolutional layers. arXiv preprint 1rXi::1911.0$55 2019.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition Workshops, pp. 702 703, 2020.
Zihang Dai, Hanxiao Liu, Quoc V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. arXiv preprint arXiv:2106.0488 2021.
Stéphane d' d'Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXie preprint orXiv:2103.1066 2021.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition at scale. arXiv preprint arXiv:2010. 11929, 2020.
Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolu- tional neural networks meet vision transformers. arXic preprint orXiv:207..00 200.0100,,663, 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint arXio:1606.08// 2016.
Qibin Hou, Zihang Jiang, Li Yuan, Ming-Minn Cheng, Shuicheng Yan, and Jiashi Feng. A permutable mlp-like architecture for visual recognition, 2021.
Vision permutator:
Hanxiao Liu, Zihang Dai, David R So, and Quoe V Le. Pay attention to mlps. arXiv preprint arXiv:2105.08050 2021a.
Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021b.
Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11976-11984 2022.
llya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018
Luke KyriszK, Do you even need attention? stack of feed-forward layers does surprisingly well imagenet, 2021.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv:1906. 05909, 2019.
Mark Sandler, Jonathan Baccash, Andrey Zhmoginov, and Andrew Howard. Non-discriminative data weak model? on the relative importance of data and model resolution. Tn Proceedings of the IEEE/CVP International Conference on Computer Vision Workshops, pp. 0 0, 2019.
Tya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Boyer, Xiaohua Zhai, Thomas Unterthiner, Jes- sica Yung, Daniel Keysers, Jakob Uszkorcit, Mario Lucic, al. Mlp-mixer: An all-mlp architecture for vision. XrXiv preprint arXiv:2105.0160) 2021.

---

Published in Transactions on Machine Learning Research (03/2023)
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. ai preprint arXiv:2012.190 2020.
Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with dala-efficient. training. arxiv preprint or orXie:2105.0022 03400044, 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXie preprint arXi::2103.17239 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Usckoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. arXiv preprint arXit:2102.122 12122, 2021.
Ross Wighlman. Pytorch image models. petps://githhe
2019.
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm, 2021.
Tele Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. arXiv preprint 14t:2206611881, 2021.
Weihao Yu, Mi Luo, Pan Zhou, Chenyang Si, Yichen Zhou, Xinchao Wang, Jiashi Feng, and Shuicheng Yan. Metaformer is actually what you need for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10819 10829, 2022.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv:2108. 11816, 2021a.
Li Yuan, Yumpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. anXiv preprint ar xrx::2210111 11966, 2021b.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pp. 6023-6032. 2019.
Hongyi Zhang, Moustapha Cissc, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. XrXiv preprint arXiv: 1rk0:1700 1ec::1110.0944 09412, 2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random crasing data augmentation. In Proceedings of the ATAT Conference Arlificial Intellignce,, volume 34, pp. 13001-13008, 2020.

---

Published in Transactions on Machine Learning Research (03/2023)
A Comparison to other models
Throughputs mcasured on an RTX8000 GPU using batch size 64 and fp16. ConvMixers and ResNets trained ourselves. Other statistics: DeïT (Touvron et al., 2020), ResMLP (Touvron et al., 2021a), Swin (Liu et al., 2021b), VIT (Dosovitskly et al., 2020), MLP-Mixer (Tolstikkin et al., 2021), Isotropic MobileNets (Sandler ct al., 2019). We think models with matching colored dots (+) are informative compare with each other. "Throughput tested, but not trained. Activations: ReLU, GELU.
Table 2:
* Using new, better regularization hyperparameters based on Wightman et al. (2021)'s A1 procedure.
Comparison with other simple models trained on ImageNet- 1k only with input size 224.
Patch
Kernel Size
# Params (x10%)
Act.
Im.Nt top-1 1 (%)
Network
Epochs
Size
Fn.
Throughput (img/sec)
ConvMixer-1536 1536/200
51.6
134
150
82.20
ConvMixor-1536//2
51.6
134
G
150
81.37
ConvMixer-1536/20
3
49.4
246
G
150
81.60
ConvMixer-1536/22
3
49.4
246
G
150
80.43
Conw:Mixer-1536/00
14
9
52.3
538
G
150
78.92
ConvMixer-
14
9
62.3
447
G
150
80.21
Conv>Mixex-768//222
7
21.1
206
G
150
80.74
Conv ConwMixex-768/822
7
21.1
206
R
300
80.16
1onx-iixer-024/116
9
19.4
244
G
100
79.45
MoovMixxr-1024/11
8
14.6
358
G
90
77.75
ConwMixer-512/16
8
5.4
599
G
90
73.76
Conv
8
4.2
798
G
90
72.59
ConwMixer--668/32
14
3
20.2
1235
R
300
74.93
ConvMixxer-024//0
14
9
24.4
750
G
150
76.94
ResNet-152**
3
60.2
828
R
150
81.15
ResNet-152
3
60.2
828
R
150
79.64
ResNet-101
3
44.6
1187
R
150
78.33
ResNet-50
3
25.6
1739
R
150
76.32
DeiT-Bt
86.7
83
G
DefT-St
22.1
174
G
DeiT-Tif
5.7
336
G
DeîT-B
16
86
792
G
300
81.8
DeïT-S
16
22
1610
G
300
79.8
DeñT-Ti
16
5.7
2603
G
300
72.2
ResMLP-S12/8
22.1
872
G
400
79.1
ResMLP-B24/8
129
181
G
400
81.0
ResMT.P-B24
16
116
1597
G
400
81.0
Swin-S
50
576
G
300
83.0
Swin-T
29
878
G
300
81.3
ViT-B/16
16
86
789
G
300
77.9
MixeerBB/15
16
59
1025
G
300
76.44
Isotropic MobileNetv3
3
20
355
R
80.6
Isotropic MobileNetv3
3
20
1296
R
77.6
9

---

Published in Transactions on Machine Learning Research (03/2023)
Experiment overview. We did not design experiments to maximize accuracy: We chose "common sense" parameters for timm and its augmentation settings, found that it worked well for Conv>Mixer-1044111 and stuck with them for the proceeding experiments. We admit this is not an optimal strategy, however, we were aware from our early experiments on CIFAR-10 that results seemed robust to various small changes. We did not have access to sufficient. compute to attempl to lune hyperparameters for each model: e.g-, larger ConvMixers could probably benefit from more regularization than we chose, and smaller ones from less regularization. Keeping the parameters the same across ConvMixer instances seemed more reasonable than guessing for each.
However, to some extent, we changed the number of epochs per model: for carlier experiments, we merely wanted a "proof of concept", and used only 90 100 epochs. Once we saw potential, we increased this to 150 epochs and trained some larger models, namely ConvMixer-1536/21 with p = 7 patches. Then, believing that we should explore deeper-but-les--we ConvMixers, and knowing from CIFAR-10 that the deeper models convergod more slowly, we trained ConvMixer-768 with p = 14 and p- 7 for 300 epochs. Of course, training time was consideration: ConvMixer- took about 9 days to train (on 10x RTX8000s) 150 epochs, and ConvMixer-768/32 is over twice as fast, making 300 epochs more feasible.
www.Mixer-1224220 with = 14 patches and
If anything, we believe that in the worst case, the lack of parameter tuning in our experiments resulted in underestimating the accuracies of ConvMixers. Further, due to our limited compute and the fact that large models (particularly Mixess are expensive to train on large data sets, we generally trained our models for fewer epochs than competition like DeiT and ResMLP (see Table 2).
In this revision, we have added some additional results (denoted with * in Table 2) using hyperparamcters loosely based on the precisely-cratted "A1 training procedure" from Wightman et al. (2021). In particular, we adjusted parameters for RandAug, Mixup, CutMix, Random Erasing, and weight decay to match those in the procedure. Importantly, we still only trained for 150 epochs, rather than the 600 epochs used in Wightman et al. (2021), and we did not use binary cross-entropy loss nor repeated augmentation. While we do not think optimal hyperparameters for ResNet would also be optimal for ConvMixer, these settings are significantly better than the ones we initially chose. This further highlights the capabilities of ConvMixers, and we are optimistic that further tuning could lead to still-better performance. Throughout the paper, we still refer to Conv Mixers trained using our initial "one shot" " selection of hyperparamelers.
A note on throughput. We measured throughput using batches of 64 images in half precision on a single RTX8000 GPU, averaged over 20 such batches. In particular, we measured CUDA exccution time rather than "wall-clock" time. We noticed discrepancies in the relative throughputs of models, e.g., Touvron et al. (2020) reports that ResNet-152 is 2x faster than DeiT-B, but our measurements show that the two models have nearly the same throughput. We therefore speculate that our throughputs may underestimate the performance of ResNets and ConvMixers relative to the transformers. The difference may be due to using RTX8000 rather than V100 GPUs, or other low-level differences. Our throughputs were similar for batch sizes 32 and 128.
ResNets. As a simple baseline to which to compare ConvMixers, we trained three standard ResNets using exactly the same training setup and parameters as Conv 1536//0. We also trained ResNet-152* using the new Al-basod procedure for comparison against ConvMiixer-1336/200 Despite having fewer paramoters and being architecturally much simpler, ConvMixers substantially outperform these ResNets in terms of accuracy. A possible confounding factor is that ConvMixers use GELU, which may boost performance, while ResNets use ReLU. In an attempt to rule out this confound, we used ReLU in a later ConvMixer- 768/32 experiment and found that it still achieved competitive accuracy. We also note that the choice of ReLU vs. GELU was not important on CIFAR-10 experiments (see Table 7). However, Cony Mixers do have substantially less throughput.
DeiTs. We believe that DeiT is the most reasonable comparison in terms of vision transformers: It only adds additional regularization, as opposed to architectural additions in the case of CaiT (Touvron et al., 2021b), and is then essentially "vanilla" ViT modulo the distillation token (we don't consider distilled architectures). In terms of a fixed parameter budget, ConvMixers generally outperform DeiTIs. For example, ConvMixer- 1536/20 /20 is only 0.43% less accurate than D despite having over 30M fewer parameters; ConvMixer-
10

---

Published in Transactions on Machine Learning Research (03/2023)
768/32 is 0.36% more accurate than DeiT-S despite having 0.9M fewer parameters; and ConvMixer-512/16 is 0.39% more accurate than DeiT-Ti for nearly the same number of parameters. Admittedly, none of the ConvMixers are very competitive in terms of throughput, with the closest being the ConvMixer-512/16 which is 4x slower than DeiT-Ti.
A confounding factor is the difference in patch size between DeiT and ConvMixer; DeiT uses p = 16 while ConeMixer uses p = 7. This means DeiT is substantially faster. However, ConvMixers using larger patches are not as competitive. While we were not able to train DeiTs with larger patch sizes, it is possible that they would outperform ConvMixers on the parameter count us. accuracy curve; however, we tested their throughput for 7, and they are even slower than ConvMixers. Given the difference between convolution and self-attention, we are not sure it is salient to control for patch size differences.
DeiTss were subject to more hyperparameter tuning than ConvMixers, as well as longer training times. They also used stochastic depth while we did not, which can in some cases contribute percent differences in model accuracy (Touvron el al., 2021a). It. is therefore possible that further hyperparameter tuning and more epochs for ConvMixers could close the gap between the two architectures for large patches, e.g., p= 16.
RcsMLPs. Similarly to DeiT for ViT, we believe that RcsMLP is the most relevant MLP-Mixer variant to compare against. Unlike DeiT, we can compare against instances of ResMLP with similar patch size: ResMLP-B24/8 has p = 8 patches, and underperforms ConvMiixr-11538 1536/20 20 by 0.37% despite having over twice the number of parameters; it also has similarly low throughput. ConvMixer-768/33 also outperforms ResMLP-S12/8 for millions fewer parameters, but 4x less throughput.
ResMLP did not significantly improve in terms of accuracy for halving the patch size from 16 to 8, which shows that smaller patches do not always lead to better accuracy for a fixed architecture and regularization strategy (e.g., training p = 8 DeiT may be challenging).
Swin Transformers. While we intend to focus on the most basic isotropic, patch-based architectures for fair comparisons with ConvMixer, it is also interesting to compare to a more complicated model that is closer to slate-of-the-art. For a similar parameter budget, ConvMixer is around 1.2-1.6% less accurate than the Swin Transformer, while also being 4-6x slower. However, considering we did not attempt to tune or optimize our model in any way, we find it surprising that an exccodingly simple patch-based model that uses only plain convolution does not lag too far behind Swin Transformer.
These models are closest in design to ours, despite using a repeating block that is substantially more complex than the ConvMixer one. Despite this, for a similar number of parameters, we can get similar performance. Notably, isotropic MobileNets seem to suffer less from larger patch sizes than ConvMixers, which makes us optimistic that sufficient parameter tuning could lead to more performant large-patch ConvMixers. As Sandler et al. (2019) did not provide an implementation, we cannot be sure if ours is exactly the same; e.g., we were unsure if 5x5 stride-5 convolutions were replaced with 3x3 5x5 stride-1 convolutions, so we chose 3x3. The throughputs in Table 2 are based on our implementation. We also trained patch-sizo-16 Isotropic MobileNet using exactly the same pipeline used for our ConvMixers, which achieved only 70.76% accuracy.
Isotropic MobileNets.
Other models. We included VIT and MLX-Mixer instances in our table, though they are not competitive with ConvMixer, DoiT, or ResMLP, even though ALP-MMixe. has comparable regularization to ConvMixer. That is, Mixe seems to outperform Mix-Mixer and ViT, while being closer to complexity to them in terms of design and training regime than the other competitors, DeiT and ResMLP.
Kernel size. While we found some evidence that larger kernels are better on CIFAR-10, we wanted to see if this finding transferred to ImageNet. Consequently, we trained our best-performing model, ConvMixer- 1536/20, with kernel size k = 3 rather than k = 9. This resulted in a decrease of 0.94% top-1 accuracy, which we believe is quite significant relative to the mere 2.2M additional parameters. However, k = 3 is substantially faster than & 9 for spatial-domain convolution; we speculate that low-level optimizations could close the performance gap to some extent, e.g., by using implicit instead of explicit padding. Since large-kernel convolutions throughout a model are unconventional, there has likely been low demand for such optimizations.
11

---

Published in Transactions on Machine Learning Research (03/2023)
B Additional Experiments on ImageNet
In this section, we present additional experiments on ImageNet-1k. We primarily used ConvMixer-512212 trained using the new A1-like (+) technique. Note that the throughputs in this section were recorded using Tesla V100 GPUs, while those in Table 2 used RTX8000s (hence, the two measurements should not be compared across tables).
Table 3: We investigate the effect of different patch sizes throughput and accuracy. Smaller patches result in higher accuracy at the expense of throughput.
Effect of Patch Size
Patch Size
Kernel Size
Throughput (img/secc
ImNNet top-1 (%)
Network
ConvMixer-512/12
5
9
388
75.60
ConvMixer-512212
9
644
74.60
Con/Mixee-512/12
9
9
1120
73.55
1on2Mixer-512212
12
9
1908
71.79
Co1-Mixer-512/12
16
9
2892
69.65
Patch sizes. Larger patch sizes result in lower accuracy, while smaller patches increase accuracy. However, ConvMixers using smaller patches are substantially slower. For most of our experiments, we used 7 x 7 patches; however, in some cases, it may be desirable to use slightly larger 9 9 patches in exchange for a bit less accuracy (see Table 3).
Table 4: We tested ConvMixers with ResNet-style stems and ResNets with patch embedding stems; in both cases, patch embeddings worked better.
Patch Embeddings vs. ResNet-Style Stems
ImNet top-1 (%)
Network
Stem
ResNet50
ResNet Stem Patches (4 x 4)
78.32
ResNet50
78.74
Conv
ResNet Stem Patches (12 x 12)
71.24
1onvMMixer512212
71.79
Disentangling the effect of patches. We found that using patch embedding stem with ResNet improves accuracy relative to the default stem, while using ResNet stem with ConvMixer hurts accuracy (see Table 4). This provides some evidence that patches are good choice of imput representation, and may even improve the performance of existing models compared to their default input representation. For the ConvMixer, we used a ResNet stem with 12 x 12-kernel convolutions with stride 6 followed by max pooling; this ensured that ResNet-stem ConvMixer had the same internal resolution as the version using patches.
Kernel sizes. Here, we investigate whether larger kernel sizes are really beneficial to ConvMixers. In Table 5, we that x kernels strongly outperform 3 3 kernels. This may be unsurprising, the model with 9 x 9 kernels has significantly more parameters; to control for this, trained ConvMixer-512/ with 3 kernels which has comparable number of parameters. However, this still does not achieve the performance of the 9 x 9-kernel model. Further, conventional wisdom states that three stacked 3 x 3 convolutional layers (with GELUs between the layers) has the same receptive field as 9 x 9 convolution while being more expressive. Consequently, we replaced plain 3 x 3 convolutions with three stacked 3x convolutions; however, this still did not surpass the accuracy of 9 x convolutions. Finally, using the same intuition, we stack three ConvMixer-512) -51211- and tried a ConvMixer-512,/ large-kernel convolutions. This perhaps unsurprising, given the 24 additional pointwise layers.
only then do we outperform
12

---

Published in Transactions on Machine Learning Research (03/2023)
Table 5: Here, we investigate whether larger kernels are really more effective than smaller ones. Our results suggest that larger kernels are advantagoous compared to variety of "control" experiments.
Effect of Kernel Size
Patch Size
Kernel Size
Params (x10%)
Throughput (img/scc)
ImNet top-1 (%)
Network
ConvMMixer-122122 ConvMMixrr512212
7
4.07
724
74.54
15
5.15
401
75.25
ConvMixer-512/12
7
9
4.27
644
74.60
ConvMixer-511211
3
3.83
72.96
MonvMixer-5111
7
3
4.37
856
74.03
1on1/Miee-5122/1
3
3.95
732
74.53
(3 stacked 3x3-GELU convs)
ConvMiixe-512236
3
10.3
338
77.67
Table 6: We investigated choices of activation functions and normalization layers, as well training with reduced data augmentation. While reducing augmentation improves performance this small model, we did not adopt this change elsewhere.
Ablation of ConvMixer-512/12 on ImageNet
ImNet Acc. (%)
Ablation
Baseline
74.60
BatchNorm + LayerNorm
74.51
GELU + ReLU
74.44
Mixup and CutMix
75.65
RandAug
75.26
Architectural choices. In Table 6, we demonstrate that the choice of activation function (ReLU vs. GELU) and norm layer (BatchNorm LayerNorm) does not have large impact on performance.
Data augmentation. We also investigate removing some of the data augmentations from the A1 recipe (see Table 6). We saw substantial performance boost from removing Mixup and CutMix, and to lesser extent, Augment as well. This is likely due to the relatively small model used for the comparison (ConvMixer-512112) for which this level of augmentation may be excessive. We did not adopt these changes for other experiments. For comparison, a DeiT trained exactly the same way as the baseline ConyMixe achieves 70.28% accuracy, while a DeiT without RandAug, CutMix, and MixUp gets 69.65% accuracy. That is, it seems augmentations are more important to DeTT than to ConvMixer.
Input size. Unlike ViTs, MLP-Mixers, MesMLPPs and other recent models, Conerixers can handle variable input sizes with no modifications whatsoever. In Fig. 4, we show the effect of input size on the inference time ConyMixor-668/ using a batch size of 32, averaged over 16 trials on an RTX 3080Ti GPU in half precision. Note the rapid growth of inference time for kernel sizes and compared to 3 and 5; we believe this shows that the underlying implementation of depthwise convolution is suboptimal for large kernel sizes.
C Additional Experiments on CIFAR-10
Residual connections. We experimented with lcaving out one, the other, or both residual connections before settling on the current configuration, and consequently chose to leave the second residual connec- tion. Our baseline model without the connection achieves 95.88% accuracy, while including the connection
13

---

Published in Transactions on Machine Learning Research (03/2023)
ConvMixer 768/332
50
Kernel Size
Inference Time (ms)
40
5
30
20
10
16
32
48
64
80
96
112
128
Input Size
Figure Inference time vs. input size for ConvMixer-668/32 with variety of kernel sizes.
reduces it to 94.78%. Surprisingly, we see only a 0.31% decrease accuracy for removing all residual connec- tions. We acknowledge that these findings for residual connections may not generalize to deeper ConyMixers trained on larger data sets.
Table 7: Small ablation study training a 2 on CIFAR-10.
Ablation of 2onvMixer-256/8 on CIFAR-10
CIFAR-10 Acc. (%)
Ablation
Baseline
95.88
Residual in Eq. 2 + Residual in Eq.
95.57
94.78
BatchNorm LayerNorm
94.44
GELU ReLU
95.51
Mixup and CutMix
95.92
Random Erasing
95.24
RandAug
92.86
Random Scaling
86.24
Gradient Norm Clipping
86.33
Normalization. Our model is conceptually similar to the vision transformer and ML.P-Mixer, both of which use LayerNorm instead of BatchNorm. We attempted to use LayerNorm instead, and saw a decrease in performance around 1% as well as slower convergence (see Table 7). However, this was for a relatively shallow model, and we cannot guarantee that LayerNorm would not. hinder ImageNeetscalee models to an even larger degree. We note that the authors of ResMLP also saw relatively small increase in accuracy for replacing LayerNorm with NarchNorm, but for larger-scale experiment (Touvron et al., 2021a). We conclude that BatchNorm is more crucial to architecture than other regularizations parameter settings (c.g., kernel sizc).
Having settled on an architecture, we proceeded to adjust its parameters h, d, k as well as weight decay on CIFAR-10 10 experiments. (Initially, we took the unconventional approach of excluding weight decay since we were already using strong regularization the form of RandAug and mixup.) We acknowledge that tuning our architecture on CIFAR-10 does not necessarily generalize to performance on larger data sets, and that this is limitation of our study.
14

---

Published in Transactions on Machine Learning Research (03/2023)
C.1 Results
ConvMixers are quite performant on CIFAR-10, easily achieving > 91% accuracy for as little as 100,000 parameters, or > 96% accuracy for only 887, 000 parameters (see Table 8). With additional refinements e.g.. a more expressive classifier or bottlenecks, we think that ConvMixer could be even more competitive. For all experiments, we trained for 200 epochs on CIFAR-10 with RandAug, mixup, cutmix, random erasing, gradient norm clipping, and the standard augmentations timm. We remove some of these augmentations Table 7, finding that RandAug and random scaling ("default" in timm) are very important, each accounting for over 3% of the accuracy.
Scaling ConvMixer. We adjusted the hidden dimension h and the depth d, finding that deeper networks take longer to converge while wider networks converge faster. That said, increasing the width or the depth is an effective way to increase accuracy; a doubling of depth incurs less compute than a doubling of width. The number of parameters in a ConvMixer is given exactly by:
#params = hid(k? + h + 6) + CinP" + Aclasses + 3] + Relasses
(4)
including affinc scaling paramoters in BatchNorm layers, convolutional kernels, and the classifier.
Kernel size. We initially hypothesized that large kernels would be important for ConvMixers, as they would allow the mixing of distant spatial information similarly unconstrained MLPs or self-attention layers. We tried to investigate the effect of kernel size on CIFAR-10: we fixed the model to be ConvMixer-256/88 and increased the kernel size by 2s from 3 to 15.
Using a kernel size of 3, the ConvMixer only achieves 93.61% accuracy. Simply increasing it to 5 gives an additional 1.50% accuracy, and further to 7 an additional 0.61%. The gains afterwards are relatively marginal, with kernel size 15 giving an additional 0.28% accuracy, It could be that with more training iterations or more regularization, the effect of larger kernels would be more pronouneed. Nonetheless, we concluded that ConvMixers benefit. from larger-than-usual kernels, and thus used kernel sizes 7 in most of our later experiments.
It is conventional wisdom that large-kernel convolutions can be "decomposed" into stacked small-kernel convolutions with activations between them, and it is therefore standard practice to use k = convolutions, stacking more of them to increase the receptive field size with additional benefits from nonlinearities. This raises a question: is the benefit of larger kernels in ConvMixer actually better than simply increasing the depth with small kernels? First, we note that deeper networks are generally harder to train, so by increasing the kernel size independently of the depth, we may recover some of the benefits of depth without making it harder for signals to "propagate back" through the network. To test this, we trained ConvMixer-256/ with k = (698K parameters) in the same setting we increased depth in small-kernel model to roughly match the parameters of a large-kernel model. The Conv Mixe-Mixe2256/11 /10 achieved 94.29% accuracy (1.5% less), which provides more evidence for the importance of larger kernels in ConvMixers. Next, instcad of fixing the paramcter budget, we tripled the depth (using the intuition that 3 stacked = 3 convolutions have the receptive field of a k = 9 convolution), giving a ConvMixer-255 256/24 22 with 1670K parameters, and got 95.16% accuracy, i.e., still less.
ConvMixer-256; with k = 9 (707K parameters), i.e.,
Patch size. CIFAR-10 imputs so small that initially only used p = i.e., the patch embedding layer does little more than compute h linear combinations of the input image. Using p = 2, we see reduction in accuracy of about 0.80%; this is a worthy tradeoff in terms of training and inference time. Further increasing the patch size leads to rapid decreases in accuracy, with only 92.61% for p p-
Since the Sinternal resolution" decreased by a factor of p when increasing the patch size, we assumed that larger kernels would be less important for larger p. We investigated this by again increasing the kernel size from to 11 for - with p = 2: however, this time, the improvement going from 3 to 5 is only 1.13%, and larger kernels than 5 provide only marginal benefit.
Weight decay. We did many of our initial experiments with minimal weight decay. However, this was not optimal: by tuning weight decay, we can get an additional 0.15% of accuracy for no cost. Consequently, we used weight decay (without tuning) for our larger-scale experiments on ImageNet.
15

---

Published in Transactions on Machine Learning Research (03/2023)
Table 8: An investigation of ConvMixer design parameters h, d, k and weight decay CIFAR-10
Tiny ConvMixers trained on CIFAR-10.
Width
Depth
Patch Size
Kernel
# Params (x10*)
Weight Deray
CIFAR-10
Size
Anc. (%)
128
4
103
91.26
128
8
205
93.83
128
12
306
94.83
256
4
338
93.37
256
8
672
95.60
256
12
1
1006
96.39
256
16
1339
96.74
256
20
8
1673
96.67
Kernel adjustments
256
8
3
559
93.61
256
8
592
95.19
256
641
95.80
256
8
707
95.88
256
8
1
11
788
95.70
256
8
1
13
887
96.04
256
8
15
1001
96.08
Patch adjustments
256
8
2
709
95.00
256
8
718
92.61
256
8
755
85.57
Weight decay adjustments
256
8
707
1 10
95.88
256
8
707
1 10
96.03
256
8
707
1X 10-3
95.76
256
8
1
707
1X 10
95.63
256
8
707
1 10
95.88
Kernel size adjustments when p =
2
256
8
2
3
561
94.08
256
8
2
594
0
95.21
256
8
2
643
95.35
256
8
709
95.00
256
8
2
11
791
95.14
Adding woight decay to the above
256
8
2
561
1 10
94.69
256
8
2
594
1 10-2
95.26
256
8
2
643
1x 10
95.25
256
8
2
9
709
1 10
95.06
256
8
2
11
791
1 10-2
95.17
16

---

Published in Transactions on Machine Learning Research (03/2023)
D
Weight Visualizations
Figure 5: Patch embedding weights for ConvMixer-1024// with patch size 14 (see Table 2).
Figure 6: Patch embedding weights for a Conv with patch size (see Table 2).
17

---

Published in Transactions on Machine Learning Research (03/2023)
Layer
Layer
(c) Layer
(d) Layer
(e) Layer 11
Layer 13
(g) Layer 15
(h) Layer 17
(i) Layer 19
Figure 7: Random subsets of 64 depthwise convolutional kernels from progressively deeper layers of ConvMixer-15300 (Table 1).
In Figure and 6, we visualize the (complete) weights of the patch embedding layers of ConvMixer-1536, with p 14 and Con /2 with 7, respectively. Much like Sandler et al. (2019), the layer consists of Gabor-like filters well as "colorful globs" or rough edge detectors. The filters seem to be more structured than those learned by MLP-Mixer (Tolstikhin et al., 2021); also unlike MLP-Mixer, the weights look much the same going from p = 14 to = 7: the latter simply looks like a downsampled version of the former. It is unclear, then, why we see such a drop accuracy for larger patches. However, some of the filters essentially look like noise, maybe suggesting a need for more regularization or longer training, or even more data. Ultimately, we cannol read too much into the learned representations here.
In Figure plot the hidden convolutional kernels for successive layers of ConvMixer. Initially, the kernels seem to be relatively small, but make use of their allowed full size in later layers; there is clear hicrarchy of features as one would expect from a standard convolutional architecture. Interestingly, Touvron et al. (2021a) saw a similar effect for ResMLP, where earlier layers look like small-kernel convolution, while later layers were more diffuse, despite these layers being representated by an unconstrained matrix multiplication rather than convolution.
18

---

Published in Transactions on Machine Learning Research (03/2023)
E Implementation
def ConvMixer dp,)
S,cAA-Sequential,com Conv2d, lambda x:S(x,GELUO) Batchloorm2dhh))
( forrar'd :lambda s,x:s[0] (x)+x})
return S(A(C(3,h.p.))), AdaptiveevvpPoo22((1)) Flattte)) ,i,nea
htwp:hhhhh....uup......dd.iiiii Altpg////////)),c.c.hh..... ,((C(,,,,1))
in range(d)),
Figure 8: An implementation of our model in less than 280 characters, in case you happen to know of any means of disseminating information that could benefit from such length.
All you need to do to run this is from torch.. nn import
We present an even more terse implementation of ConvMixer in Figure 8, which to the best of our knowledge is the first model that achieves the elusive dual goals of 82%+ ImageNet lop-1 accuracy while also fitting into a tweet.
19


Reviews:
[{"review_id": "7IvMWQTt7M", "reviewer": "Reviewer_9PYK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n\nThis paper has a number of notable strengths; below I list a few salient strengths with supporting evidence:\n\n1. **Presents food-for-thought to the vision community:**\n   This paper presents poses a simple question that sounds very contrarian to the design philosophy of recent models:\n   is the success of recent architectures like ViT and MLP-Mixer largely due to the early processing of images as patches?\n   Vision Transformers were introduced with a motivation to bring the benefits of transformer architectures\n   that have revolutionized the field of natural language processing (dubbed the \"BERT moment\").\n   As the community indulges more in the healthy competition of _transformers vs ConvNets_ and\n   ponders whether self-attention is even necessary, the question posed in this paper is well-timed.\n   The paper motivates readers and practitioners to rethink the reason behind recent successful vision models.\n\n2. **Simple model architecture:**\n   ConvMixer follows an extremely simple fully-convolutional design.\n   The authors start with a design philosophy of enabling _mixing_ of information across spatial and channel dimensions,\n   and achieve it solely by convolution layers (depthwise and pointwise, respectively).\n   Depthwise convolution is much simpler to understand and implement than self-attention.\n   Moreover, the entire model architecture can be expressed and specified using only four scalar parameters.\n\n3. **Efficient model architecture:**\n   ConvMixer has convolutional layers that operate in patches.\n   Depthwise convolutions do not incur quadratic computation costs like self-attention since they have \"sliding window\" computation.\n   This also carries a few other subtle benefits: first, this model does not require position embeddings (like MLP-Mixer).\n   Second, it also enables easy inference and transfer learning for tasks that require a different resolution\n   (e.g. object detectors use much higher resolution) without needing additional tricks such as bilinear interpolation of patch embeddings.\n\n4. **Strong empirical performance:**\n   The authors state that the reported results are obtained by using sensible defaults of the ImageNet-1k training recipe.\n   It is encouraging to see the strong empirical performance of ConvMixer despite limited hyperparameter search.\n   Comparisons with recent model architectures in the main paper are sufficient; more are included in Appendix A.\n   I side with the authors' proposition that the goal here is to not show state-of-the-art performance.\n   With this in mind, I believe the performance trends matter more than absolute numbers, and they are sufficient.\n\n5. **Neat writing clarity and presentation:**\n   I believe that the overall writing quality and presentation in this paper are excellent.\n   The authors have done due diligence in extensively discussing relevant literature.\n   The model architecture is visually illustrated neatly with a self-contained caption\n   -- it aids the reader in quickly understanding the structure of ConvMixer.\n   Each table that presents quantitative results, has a single takeaway message.\n   The authors provide a clean source code, and an even more concise _tweet-worthy_ implementation in Appendix E.\n   I tried it and it successfully instantiates a PyTorch module.\n\n**Additional remark on contributions and impact:**\nOverall, I am comfortable recommending this paper for acceptance, without further changes.\nI agree with the authors' proposition — the goal here is to not introduce state-of-the-art architecture.\nAs a member of the research community, I am well aware that an earlier version of this paper has been openly accessible for more than one year.\nThe paper has gained 100+ citations already — the community has already realized its positive impact.\nThis paper was also in the limelight for its short length and its bold and unconventional writing style;\nI would not gate-keep simply because the writing style does not ~confer~ (EDIT typo:) conform to my preference,\nso long as it does not violate any official policy (to the best of my knowledge, it does not).\nGiven its technical strengths and already observed impact, I think this paper is ready to be accepted at TMLR.\n\n### Weaknesses:\n\nI found a few minor weaknesses that I list here for completeness.\nI don't feel any of them to be critical enough for recommending rejection.\n\n1. There is one main weakness that underpins even some of my listed strengths —\n   the authors could further strengthen their contributions by presenting ConvMixer as a powerful architecture even at a larger scale.\n   This would include more thorough tuning of the model architecture, optimization, and presenting results with even larger models (compared to the size of ViT-L/H).\n   The authors may have limited computing resources, so this weakness is not critical to address.\n\n2. The authors claim that ConvMixer can work at any resolution (as listed in strengths).\n   While valid by design, it is worthwhile to show inference speed and accuracy trends when a trained ConvMixer is employed for inference at varying image resolutions.\n\n3. ConvMixer uses BatchNorm by default, instead of LayerNorm as used in ViT.\n   The authors include an ablation showing that LayerNorm works well but incurs a very small performance drop.\n   I suggest recommending LayerNorm as a default as it allows seamless batch size scaling across GPUs (without complicated BatchNorm synchronization across devices).\n   The authors may consider ConvNeXt paper for gaining some intuition on architectural changes that can facilitate the use of LayerNorm instead of BatchNorm effortlessly.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors propose a new model for computer vision tasks, named ConvMixer.\nThis model replaces modules within layers of a Vision Transformer (ViT) with fully convolutional operations:\nself-attention is replaced with depthwise convolutions and MLPs with pointwise (1$\\times$1) convolutions.\nThe design motivation stems from a simple question posed by the authors:\nwhether the success of ViT-style models could be due to the use of patches,\nwhich are common components in such models.\nThe proposed ConvMixer is simple and parameter-efficient, and experiments show that it outperforms prevalent computer vision models\nlike ViT, MLP-Mixer, and classical ResNets, under controlled comparisons with similar datasets and model sizes.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would suggest changes according to Weaknesses (1, 2, 3) as per the authors' capacity and available computational resources.\nIn my opinion, none of them are critical to the core scientific merits of the paper,\nbut they would be valuable additions nonetheless for maximizing the potential impact of this paper.\n", "broader_impact_concerns": "The impact of this work is no different than hundreds of existing computer vision models that are benchmarked on the ImageNet dataset.\nIt presents a simple scientific question and provides experimental evidence to support it.\nFurther discussion is not necessary for this context.\n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Q0A5PMlFOA", "reviewer": "Reviewer_p1Lh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\nThe paper investigates a single, simple and meaningful goal of understanding the effect of using patches and finding most (more?) efficient ways of processing them. I'm very curious to see if indeed ConvMixer will prove to be useful in other tasks, perhaps involving image-to-image transformations. I find the experiments done, again, in a clear and convincing manner. \n\n### Weaknesses \n\nThis should not necessarily reflect negatively on the paper, but unlike in ViT and MLPMixer papers we do not necessarily understand better which are better universal building blocks for composing efficient architectures. Not all data modalities have analogs to patches and not every one that do will benefit from ConvMixer or at least that's what I expect. I still think that the paper findings are important and should not be underestimated even if my argument here is true.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new architecture for image classification (and possible useful in other CV tasks) called ConvMixer which can be seen as a convolutional version of MLPMixer. Authors find that it is more parameter-efficient than ResNet, ResMLP and DeiT on ImageNet-1k and attribute the improvement to the use of patch representation as opposed to the more expressive attention / MLP transformations. The authors also analyze the effect of model hyperparameters (such as the patch size) to on the throughput and classification accuracy. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would be interested in hearing (reading) authors' opinion on what does it mean for us if indeed patches are all we need? Fundamentally,  I see no reason why this kind of representations should be globally optimal even withing the current knowledge frontier of deep learning. What do we learn beyond just this fact that would matter for finding more efficient architectures in the future?", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "SDFWyiYZph", "reviewer": "Reviewer_7eQ2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- Simplicity. Simple approaches are more practical and applicable than complex approaches. This will be an important baseline moving forward.\n- An ablation for the works that use transformers/MLPs for vision, showing that patches may be the critical component.\n- The paper is well-written, clear, and honest.\n\nDue to these strengths, I believe this paper will have high impact.\n\n## Weaknesses\n\n1. **Simplicity of MLPs vs convolutions.** The insight, “patches are all we need”, already has partial evidence from two prior works: MLP-Mixer and Melas-Kyriazi (2021). These works replaced the attention layers in the original Vision Transformer with simple components, namely MLPs. In fact, MLPs are conceptually simpler than convolutions (although, the authors argue that convolutions are less expressive than MLPs). \nSimilarly, the authors use the phrase “new operations (like self-attention and MLPs)”  but MLPs are not newer than convolutions.\nThat said, ConvMixers do still fill an important niche: they are significantly simpler than MLP-Mixer (which was optimized much more for performance), and significantly better-performing than Melas-Kyriazi -- 74.9% vs. 82.2% on ImageNet. This niche is important because it makes ConvMixer a strong baseline and easy to build off of.\n\n2. **\"Accuracies we report likely underestimate capabilities.\"**\nI understand that the authors explicitly did not optimize their architecture for accuracy or inference speed, mentioning this several times throughout the text, because it was not necessary for their message, and because it costs a lot to train ImageNet models. I agree that this should not be necessary. The authors also compared to the most basic isotropic patch-based architectures (DeiT and ResMLP).\nIt is still somewhat of a downside, because (a) there is no way to tell how much the model will improve with HPO, and (b) there is no way to tell how much hyperparameter tuning has already been done by the authors (although they claim only light HPO). One option would be to compare ConvMixer with other approaches, each with HPO, on one of the popular versions of ImageNet that is a small subset and heavily downsampled. \n\n3. **Table 2.** It is hard to interpret Table 2. Each training run has several decisions: width, depth, patch size, kernel size, activation function, and #epochs, and it is hard to tell why these 14 runs were chosen, and how they are ordered. The colored dots make it easier to see which results to compare, but that also begs the question, what is the point of the rows without colored dots?\nPerhaps, whenever possible, specific experiments/insights can be shown in their own table (for example, Table 3 is a clean insight). The authors could also consider making Table 3 and similar experiments where just one variable is varied, into a plot.\n\nI believe that the strengths clearly outweigh the weaknesses.\n\n## Minor comment\n\n**Paper length.**\nThe fact that the authors only use 6 pages is in stark contrast to the unspoken rules of the ML community (especially the NeurIPS, ICML, communities) where a paper must exactly hit the page limit and generally minimize whitespace, or else face a likely rejection. (I tried to find a citation for this beyond my own experience, and I could only find [this StackExchange thread](https://academia.stackexchange.com/questions/25650/is-it-a-good-idea-to-submit-paper-which-is-shorter-than-conferences-page-limit)).\nWhile TMLR has a 12-page limit, generally papers below 9 pages (the NeurIPS limit) are seen as short.\n\nHowever, papers should be reviewed based on their scientific merit, not their page count.\n\nI agree with the authors that 6 pages is a good length for their paper. It is healthy for the community to remember to focus on scientific merit rather than reaching the page limit. Furthermore, too many “unspoken rules” make it harder for newcomers to join the ML community, especially rules that have to do with maximizing page numbers (e.g., there is a [new ICLR initiative](https://iclr.cc/Conferences/2023/CallForTinyPapers) to address this).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors consider the success of Vision Transformers and other recent computer vision models such as MLP-Mixer. The authors devise an architecture with similar attributes: the patch embedding and an isotropic architecture with alternating spatial and channel dimension mixing, but they use convolutions instead of attention or MLPs from prior work. The resulting architecture, ConvMixer, is simple compared to prior work, both conceptually and in terms of the number of parameters, yet the authors show it achieves fairly strong performance on ImageNet even without many optimizations.\n\nThe main contributions are as follows\n - They release a new baseline that is conceptually simple and lightweight, and achieves good performance, allowing for ease of follow-up work.\n - They give evidence that patches+convolutions are sufficient for designing high-performing vision models. This acts as an ablation study for prior works that used attention or MLPs for vision.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "None, but I encourage the authors to address the points in `weaknesses' above, which would make this paper even stronger.", "broader_impact_concerns": "There are no broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Meta Review for “Patches Are All You Need?”

As the three reviewers (9PYK, 7eQ2, p1Lh) wrote, the authors of "Patches Are All You Need?" pose an interesting question for the computer vision community: whether the success of recent Vision Transformers and MLP-Mixers can be largely attributed to processing images as patches. Instead of following the trend in deep learning papers that propose ever more complex architectures, this work investigates a single, simple and meaningful goal of understanding the effect of using patches and finding most efficient ways of processing them. To do this, the authors consider the recent success of Vision Transformers and other recent computer vision models such as MLP-Mixer, and pose a highly simplified architecture.

The resulting architecture, ConvMixer, is simple compared to prior work, both conceptually and in terms of the number of parameters, yet the authors show it achieves fairly strong performance on ImageNet even without many optimizations. To synthesize the expression of simplicity, the authors have made great efforts to summarize the message of their work in 6 pages, which hopes to send a message to the community to emphasize the value of being concise in scientific communication.

Given the technical strengths and presentation quality, all reviewers, and myself recommend acceptance. This work is relevant to all deep learning practitioners and researchers working with CNNs, and we expect this work to have a good impact even beyond the computer vision sub-field. Lastly, the paper sets the standard for the community to follow for presenting ideas elegantly in a succinct manner.
Decision: Accept (poster)