Patches Are All You Need?
Authors: Asher Trockman
J Zico Kolter
Venue: ICLR 2022 Submitted (2022)
Keywords: computer vision
vision transformer
mixer
patch embeddings
convolution
convolutional neural network

Abstract: Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based models, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of the self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation? In this paper, we present some evidence for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit to the ViT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. In contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite its simplicity, we show that the ConvMixer outperforms the ViT, MLP-Mixer, and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https://github.com/tmp-iclr/convmixer.

Under review as conference paper at ICLR 2022
CONVOLUTIONS ATTENTION MLPs PATCHES ARE ALL YOU NEED?
Anonymous authors Paper under double-blind review
ABSTRACT
Although convolutional networks have been the dominant architecture for vision tasks for many years, recent experiments have shown that Transformer-based mod- els, most notably the Vision Transformer (ViT), may exceed their performance in some settings. However, due to the quadratic runtime of self-attention layers in Transformers, ViTs require the use of patch embeddings, which group together small regions of the image into single input features, in order to be applied to larger image sizes. This raises question: Is the performance of ViTs due to the seterennly-more-pwweee
Transformer architecture, or is it at least partly due to using patches the input representation? In this paper, present for the latter: specifically, we propose the ConvMixer, an extremely simple model that is similar in spirit the VIT and the even-more-basic MLP-Mixer in that it operates directly on patches as input, separates the mixing of spatial and channel dimensions, and maintains equal size and resolution throughout the network. contrast, however, the ConvMixer uses only standard convolutions to achieve the mixing steps. Despite simplicity, we show that the Mixer outperforms the and some of their variants for similar parameter counts and data set sizes, in addition to outperforming classical vision models such as the ResNet. Our code is available at https: / //g thub. com/tmp-iclt cott///oro.ii......
evidence
VIT,
INTRODUCTION
For many years, convolutional neural networks have been the dominant architecture for deep learning systems ap- plied to computer vision tasks. But recently, architectures based upon Transformer models, e.g., the so-called Vision Transformer architecture (Dosovitskiy et al., 2020), have demonstrated compelling performance in many of these tasks, often outperforming classical convolutional archi- tectures, especially for large data sets. An understandable assumption, then, is that it is only matter of time be- fore Transformers become the dominant architecture for vision domains, just as they have for language processing. In order to apply Transformers to images, however, the representation had be changed: because the computa- tional cost of the self-attention layers used in Transformers would scale quadratically with the number of pixels per image if applied naively at the per-pixel level, the compromise was to first split the image into multiple "patches", linearly embed them, and then apply the transformer directly to this collection of patches.
ImageNet Top-1 Accuracy [%)
78
74
20
100
120
# Parameters (Millions)
ConvMixer ResMLP DeiT ResNet Figure 1:
Accuracy vs. parameters,
trained and evaluated on ImageNet- lk.
In this work, we explore the question of whether, fundamentally, the strong performance of vision transformers may result more from this patch-based representation than from the Transformer archi- tecture itself. We develop very simple convolutional architecture which we dub the "ConvMixer" due to its similarity to the recently- proposed MLP-Mixer (Tolstikhin et al., 2021). This architecture is similar to the Vision Transformer (and MLP-Mixer) in many respects: it directly operates patches, it maintains an equal-resolution and--and--aa and-size representation throughout all layers, it does no downsampling of the representation at successive layers, and separates "channel-wise mixing"

---

Under review as a conference paper at ICLR 2022
ConvMixer Layer
Residual connection
Global Average Pooling
Patch Embooding
FullyCorraeee
Cless
BetchNom
Depthwise
Convolution
BatchNorm
Paintiss
Convolltion
BetchNom
GELU
x
19
h xxn/p x n/p
X diopth
ConvMixer uses "tensor layout" patch embeddings to preserve locality, and then applies copies of simple fully-convolutional block consisting of large-kerne depthwise convolution followed by pointwise convolution, before finishing with global pooling and simple linear classifier.
Figure 2:
def ConvMixer(h, depth, kernel_size=9, patch size=7, n_classes=1000)
Seq, ActBn = mn n.. Sequential, lambda x: Seq(x, nn. GELU(), nn. BatchNormm2((h))
Residual ()pee((iudua))
(Seq.), forward' lambda self, self[0] x})
return Seq (nn Conv2d(3,
patch_size stride-patch_ssizee
[Seq (Residual (ActBn Conv2d h, kernel_size, groups-h, padding= ")),
ActBn (nn. (hnv2d h, 1))) for in range (depttt)) nn. Aew.ptiveevgooolll(((C..
mn. Fm.atttee)) nn Linearrrr,, n_classes))
Figure 3: Implementation of ConvMixer in PyTorch; see Appendix D for more implementations.
from the "spatial mixing" of information. But unlike the Vision Transformer and MIP-Mixer, our architecture does all these operations via only standard convolutions.
The chief result we show in this paper is that this ConvMixer architecture, despite its extreme sim- plicity (it can be implemented in As lines dense PyTorch code), outperforms both "standard" computer vision models such ResNets of similar parameter counts and some corresponding Vision Transformer and MLP-Mixer variants, even with slate of additions intended to make those architec- tures more performant smaller data sets. Importantly, this is despite the fact that we did not design our experiments to maximize accuracy nor speed, in contrast to the models we compared against. Our results suggest that, at least some extent, the patch representation itself may be critical com- ponent to the "superior" performance of newer architectures like Vision Transformers. While these results are naturally just snapshot, and more experiments are required exactly disentangle the effect of patch embeddings from other factors, we believe that this provides a strong "convolutional- but-patch- patch-based" based" baseline to compare against for more advanced architectures in the future.
2
A SIMPLE MODEL: CONVMIXER
Our model, dubbed ConyMixer, consists of patch embedding layer followed by repeated applications of a simple fully-convolutional block. We maintain the spatial structure of the patch embeddings, illustrated in Fig. 2. Patch embeddings with patch size p and embedding dimension h can be implemented as convolution with input channels, h output channels, kernel size and stride
20 BN (o{ConVe,,>/) (X, (tridee-pkeemel_siiee
The Minv block itself consists of depthwise convolution (i.e., grouped convolution with groups equal the number of channels, h) followed by pointwise (i.e., kernel size 1 x 1) convolution. As we will explain in Sec. 3, ConvMixen work best with unusually large kernel sizes for the depthwise convolution. Each of the convolutions is followed by an activation and post-activation BatchNorm:
2/ = BN ( [onvDeptthwise((a_11) + 21-1
(2)
21+1 = BN (a [ConvPointwise({))
(3)
After many applications of this block, perform global pooling to get feature vector of size h, which we pass to a softmax classifier. See Fig. 3 for an implementation of ConyMixer in PyToc..
Design parameters. An instantiation of Mixerixer depends four parameters: (1) the "width" or hidden dimension h (i.e,, the dimension of the patch embeddings), (2) the depth d, or the number of repetitions of the ConvMixer layer, (3) the patch size p which controls the internal resolution of the
2

---

Under review as a conference paper at ICLR 2022
Current "Most Interesting" ConvMixer Configurations vs. Other Simple Models
Patch Size
Kernel Size
Params (x10%)
Throughput (img/sec)
Act. Fn.
ImNet
Network
Epochs
top-1 (%)
ConvMixer- 1536/22
7
9
51.6
89
G
150
81.37
ConvMixer-768/32
21.1
203
300
80.
ResNet-152
60.2
872
150
79.64
DeiT-B
16
86
703
G
300
81.8
ResMLP-B24/8
129
140
400
81.0
Table 1: Models trained and evaluated on 224 x 224 ImageNet-Il 1k only. See more in Appendix A.
model, (4) the kernel size l of the depthwise convolutional layer. We name ConvMixers after their hidden dimension and depth, like ConyMixer-h./4 d. We refer to the original input size n divided by the patch size the internal resolution; note, however, that ConvMixers support variable-sized inputs.
Motivation. Our architecture based on the idea of mixing, as in Tolstikhin et al. (2021). In particular, chose depthwise convolution mix spatial locations and pointwise convolution to mix channel locations. A key idea from previous work is that MLPs and self-attention can mix distant spatial locations, i.e,, they have arbitrarily large receptive field. Consequently, we used convolutions with an unusually large kernel size to mix distant spatial locations.
While self-attention and MLPs are theoretically more flexible, allowing for large receptive fields and content-aware behavior, the inductive bias of convolution is well-suited to vision tasks and leads high data efficiency. By using such standard operation, we also get glimpse into the effect of the patch representation itself in contrast to the conventional pyramid- -haped,, progressively- downsampling design of convolutional networks.
EXPERIMENTS
Training setup. We primarily evaluate ConvMixers on ImageNet-Il Ik classification without any pretraining or additional data. We added ConvMixer to the timm framework (Wightman, 2019) and trained it with nearly-standard settings: we used RandAugment (Cubuk et al., 2020), mixup (Zhang et al., 2017), CutMix (Yun et al., 2019), random erasing (Zhong et al., 2020), and gradient norm clipping in addition to default timm augmentation. We used the AdamW (Loshchilov & Hutter, 2018) optimizer and a simple triangular learning rate schedule. Due to limited compute, did absolutely no hyperparaneter tuning on ImageNet and trained for fewer epochs than competitors. Consequently, our models could be underestimate the capabilities of our model.
or under-regularized and the accuracies report likely
Results. A ConyMixer-153622 with 52M parameters achieve 81.4% top-1 accuracy Ima- geNet, and ConvMixer--668/22 with 21M parameters 80.2% (see Table 1). Wider ConvMixers seem converge in fewer epochs. but are memory- and compute-hungry. They also work best with large kernel sizes: ConvMixer- 1536/00 lost z 1% accuracy when reducing the kernel size from k = to k 3 (we discuss kernel sizes more in Appendix A & B). ConvMixers with smaller patches are substantially better in our experiments, similarly to Sandler et al. (2019); we believe larger patches require deeper ConvMixers. With everything held equal except increasing the patch size from 7 to 14, Mixe 1irr-113366 1536/20 achieves 78.9% top-1 accuracy but is around 4x faster. We trained one model with ReLU to demonstrate that GELU (Hendrycks & Gimpel, 2016), which popular in recent isotropic models, isn't necessary.
Comparisons. Our model and ImageNetlk-onfl training setup closely resemble that of recent patch- based models like DeiT (Touvron et al., 2020). Due to ConvMixer's simplicity, we focus comparing to only the most basic isotropic patch-based architectures adapted to the ImageNet- 1k setting, namely DeiT and ResMLP. Attempting fair comparison with standard baseline, we trained ResNets using exactly the same parameters as ConvMixers; while this choice of parameters is suboptimal (Wightman al., 2021), it is likely also suboptimal for Cony Mixers, since did hyperparameter tuming.
3

---

Under review as a conference paper at ICLR 2022
Looking at Table 1 and Fig. 1, ConvMixers achieve competitive accuracies for given parameter budget: ConvMixer- 1536/20 outperforms both ResNet- 152 and ResMLP-B24 despite having sub- stantially fewer parameters and is competitive with DeiT-B. ConvMixer-768/22 uses just a third of the parameters of 152, but is similarly accurate. Note that unlike ConvMixer, the DeiT' and ResMLP involved hyperparameter tuning. and when substantial resources are dedicated tun- ing ResNets, including training for twice as many epochs, they only outperform an equivalently-sizer ConvMixer by 0.2% (Wightman et al., 2021). However, ConvMixers are substantially slower at inference than the competitors, likely due to their smaller patch size; hyperparameter tuning and opti- mizations could narrow this gap. For more discussion and comparisons, see Table 2 and Appendix A.
CIFAR-10 Experiments, We also performed smaller-scale experiments on CIFAR- 10, where Con- vMixers achieve over 96% accuracy with as few as 0.7M parameters, demonstrating the data efficiency of the convolutional inductive bias. Details of these experiments are presented in Appendix B.
4 RELATED WORK
Isotropic architectures. Vision transformers have inspired a new paradigm of "isotropic" archi- tectures, i.e., those with equal size and shape throughout the network, which use patch embeddings for the first layer. These models look similar to repeated transformer-encoder blocks (Vaswani al., 2017) with different operations replacing the self-attention and MLP operations. For example, MLP-Mixer (Tolstikhin et al., 2021) replaces them both with MLPs applied across different dimensions (i.e., spatial and channel location mixing); ResMLP (Touvron et al., 2021a) is a data-efficient variation on this theme. CycleMLP (Chen et al., 2021), gMLP (Liu et al., 2021a), and vision permutator (Hou et al., 2021), replace one or both blocks with various novel operations. These are all quite performant, which is typically attributed to the novel choice of operations. contrast, Melas Melas-Kyriazi (2021) proposed an MLP-based isotropic vision model, and also hypothesized patch embeddings could be behind its performance. ResMLP tried replacing its linear interaction layer with (small-kernel) convolution and achieved good performance, but kept its MLPP-ased cross-channel layer and did not explore convolutions further. As our investigation of ConvMixers suggests, these works may conflate the effect of the new operations (like self-attention and MLPs) with the effect of the use of patch embeddings and the resulting isotropic architecture.
A study predating vision transformers investigates isotropic (or "isometric") MobileNets (Sandler al., 2019), and even implements patch embeddings under another name. Their architecture simply repeats an isotropic MobileNetv3 block. They identify tradeoff between patch size and accuracy that matches our experience. and train similarly performant models (see Appendix A, Table 2). However, their block is substantially more complex than ours; simplicity and motivation sets our work apart.
Patches aren't all you need. Several papers have increased vision transformer performance by replacing standard patch embeddings with a different stem: Xiao et al. (2021) and Yuan et al. (2021a) use standard convolutional stem, while Yuan et al. (2021b) repeatedly combines nearby patch embeddings. However, this conflates the effect of using patch embeddings with the effect of adding convolution or similar inductive biases e.g., locality. We attempt to focus on the use of patches.
CNNs meet ViT'. Many efforts have been made to incorporate features of convolutional networks into vision transformers and vice versa. Self-attention can emulate convolution (Cordonnier et al., 2019) and can be initialized or regularized to be like it (d'Ascoli et al., 2021); other works simply add convolution operations to transformers (Dai et al., 2021; Guo et al., 2021), or include downsampling to be more like traditional pyramid-shaped convolutional networks (Wang et al., 2021). Conversely, self-attention or attention-like operations can supplement or replace convolution in ResNet-style models (Bello al., 2019; Ramachandran et al., 2019; Bello, 2021). While all of these attempts have been successful in one way or another, they are orthogonal to this work. which aims to emphasize the effect of the architecture common to most ViTs by showcasing it with a less-expressive operation.
5 CONCLUSION
We presented ConeMs extremely simple class of models that independently mixes the spatial and channel locations of patch embeddings using only standard convolutions. We also highlighted that using large kernel sizes, inspired by the large receptive fields of ViT's and MLP-Mixers, provides
4

---

Under review as a conference paper at ICLR 2022
substantial performance boost. While neither model nor our experiments were designed to max- imize accuracy or speed, f.e., we did not search for good hyperparameters. ConvMixers outperform the Vision Transformer and MLP-Mixer, and are competitive with ResNets, DeiTs, and ResMLPs.
We provided evidence that the increasingly common "isotropic" architecture with a simple patch embedding stem is itself powerful template for deep learning. Patch embeddings allow all the downsampling to happen at once, immediately decreasing the internal resolution and thus increasing the effective receptive field size, making easier mix distant spatial information. Our title, while an exaggeration, points out that attention isn't I the only export from language processing into computer vision: tokenizing inputs, i.e., using patch embeddings, also a powerful and important takeaway.
While our model is not state-of-the-ars we find its simple patch-mixing design to be compelling, We hope that ConvMixers can serve as a baseline for future patch-based architectures with novel opera- tions, or that they can provide a basic template for new conceptually simple and performant models.
Future work. We are optimistic that a deeper ConvMixer with larger patches could reach desirable tradeoff between accuracy, parameters, and throughput after longer training and more regularization and hyperparameter tuning, similarly to how Wightman et al. (2021) enhanced ResNet performance through carefully-designed training regimens. Low-level optimization of large-kernel depthwise convolution could substantially increase throughput, and small enhancements to our architecture like the addition of bottlenecks more expressive classifier could trade simplicity for performance.
Due to its large internal resolution and isotropic design, ConvMixer may be especially well-suited for semantic segmentation, and it would be useful to run experiments on this task with ConvMixer-like model and on other tasks such as object detection. More experiments could be designed to more clearly extricate the effect of patch embeddings from other architectural choices. In particular, for more in-depth comparison to ViTs and MI M-X-ers, which excel when trained on very large data sets, is important to investigate the performance of MoneMixers in the regime of large-scale pre-training.
note on paper length. Expecting more text in this paper? Wondering if it's a workshop paper we hastily submitted ICLR? No. This paper presents a simple idea, one where we genuinely believe that a short paper presentation is more effective. Do we really need exactly 8 (now 9? 10?) pages to describe every machine learning architecture and algorithm in existence? We proposed an incredibly simple architecture and made very simple point that we think is worth more discussion: patches work well convolutional architectures. We think that four and a half pages is more than enough space for this. The details of the experiments and architectures are in the appendix for those who want to read through it all.
REFERENCES
Irwan Bello. Lambdanetworks: Modeling long-range interactions without attention. arXiv preprint arXiv:2102.08602, 2021.
Irwan Bello, Barret Zoph, Ashish Vaswani, Jonathon Shlens, and Quoc V Le. Attention augmented convolutional networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 3286-3295, 2019.
Shoufa Chen, Enze Xie. Chongjian Ge, Ding Liang, and Ping Luo. Cyclemlp: A mlp-like architecture for dense prediction. arXiv preprint arXiv:2 10224, 2021.
Jean-Baptiste Cordonnier, Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. arXiv preprint arXiv: 191.::1911.........1 2019.
Ekin Cubuk, Barret Zoph, Jonathon Shlens, and Quoe Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 702-703, 2020.
Zihang Dai, Hanxiao Liu, Quoe Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. arXiv preprint 200.22106004033, 2021.
Stéphane d'Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv preprint arXiv:2103. 10697, 2021.
5

---

Under review as a conference paper at ICLR 2022
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 1929, 2020.
Jianyuan Guo, Kai Han, Han Wu, Chang Xu, Yehui Tang, Chunjing Xu, and Yunhe Wang. Cmt: Convolutional neural networks meet vision transformers. arXiv preprint arXiv:2107.06260 2021.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint
arXiv:1606.08415, 2016.
Qibin Hou, Zihang Jiang, Li Yuan, Ming-Ming Cheng, Shuicheng Yan, and Jiashi Feng. Vision permutator: A permutable mlp-like architecture for visual recognition, 2021.
Hanxiao Liu, Zihang Dai, David R So, and Quoe V Le. Pay attention to mlps. arXiv preprint arXiv:2105.08050 2021a.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows, 2021b.
Ilya Loshchilov and Frank Hutter. Fixing weight decay regularization in adam. 2018.
Luke Melas- KerrasKy Do you even need attention? stack of feed-forward layers does surprisingly well on imagenet, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Bus Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems 32, pp. 8035. Curran Associates, Inc., 2019.
Prajit Ramachandran, Niki Parmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint arXiv: 190099900099999 2019.
Mark Sandler, Jonathan Baccash, Andrey Zhmoginov, and Andrew Howard. Non-diseriminative data or weak model? on the relative importance of data and model resolution. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lueas Beyer, Xiaohua Zhai, Thomas Un- terthiner, Jessica Yung, Daniel Keysers, Jakob Uszkoreit, Mario Lucic, et al. Mlp-mixer: An all-mlp architecture for vision. arXiv preprint arXiv:2005.01611, 2021.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers distillation through attention. arXiv preprint a0Xiv:2012. /2////222777 /287,, 2020.
Hugo Touvron, Piotr Bojanowski. Mathilde Caron, Matthieu Cord, Alaacldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv preprint arXiv: 105.2000000000 105.00040 2021a.
Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. arXiv preprint arXiv:2103. 2021b.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. arXiv preprint arXiv:2102.12122 2021.

---

Under review as conference paper at ICLR 2022
Ross Wightman. Pytorch image models. pytorch- iattt-iaaeeeaa mooage, 2019.
https://ggttubb..mm///iigggatait
Ross Wightman, Hugo Touvron, and Hervé Jégou. procedure timm, 2021.
Resnet strikes back: An improved training
Tete Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piotr Dollár, and Ross Girshick. Early convolutions help transformers see better. arXiv preprint arXiv:2106. 14881, 2021.
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXiv preprint arXiv: 11816,,,.11,,, 2021a.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zihang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch imagenet. arXiv preprint arXi::2101111986, 2021b.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strateg to train strong classifiers with localizable features. Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 6023-6032, 2019.
Hongyi Zrang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:
2017.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data aug- mentation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 13001-13008, 2020.

---

Under review as conference paper at ICLR 2022
A COMPARISON TO OTHER MODELS
Comparison with other simple models trained on ImageNet-Ik only with input size 224.
Patch Size
Kernel Size
# Params
Throughput (img/sec)
Act. Fn.
ImNet
Network
Epochs
(x10%)
top-1 (%)
ConvMixer-1536/20
7
51.6
89
G
150
81.37
1onvMixer-1536/20
49.4
136
G
150
80.43
ConvMixer-1536/20
14
9
52.3
334
G
150
78.92
ConvMixer-768/32
21.1
203
R
300
80.16
ConvMixer-1024/16
7
9
19.4
173
G
100
79,45
Con-Mixer-1024/12
14.6
248
G
90
77.75
ConvMixer-512/16
7
5,4
403
G
90
73.76
Con-Mixer-5122112
7
4.2
532
G
90
72.59
ConvMixer-768/32
14
20.2
1258
R
300
74.93
10nMMixer-1024/20
14
24.4
520
G
150
76.94
ResNet-152
3
60.2
872
R
150
79.64
ResNet-101
44.6
1040
R
150
78.33
ResNet-50
3
25.6
1942
R
150
76.32
DeiT-Bt
86.7
77
G
DeiT-St
7
22.1
164
G
DeiT-Tit
7
5.7
327
G
DeiT-B
703
G
300
81.8
DeiT-S
16
22
1491
G
300
79.8
DeiT-Ti
5.7
2727
G
300
72.2
ResMLP-S12/8
22.1
638
G
400
79.1
ResMLLPBB4/8
8
129
140
G
400
81.0
ResMLP-B22
16
116
1191
G
400
81.0
Swin-S
4
50
566
G
300
83.0
Swin-T'
884
G
300
81.3
ViT-B/16
16
86
704
G
300
77.9
Mixer-B/16
16
59
816
G
300
76.44
Isotropic MobileNetv3
8
3
20
R R
80.6
Isotropic MobileNetv3
20
77.6
Table 2: Throughputs measured an RTX8000 GPU using batch size 64 and fp16. ConvMixers and ResNets trained ourselves. Other statistics: DeiT' (Touvron et al., 2020), ResMLP (Touvron et al., 2021a), Swin (Liu al., 2021b), VIT (Dosovitskiy al., 2020), MLP-Mixer (Tolstikhin et al., 2021), Isotropic MobileNets (Sandleret et al., 2019). We think models with matching colored dots (+) are infor- mative to compare with each other. Throughhut tested, but trained. Activations: ReLU, GELU.
Experiment overview. We did not design our experiments maximize accuracy: We chose "common sense" parameters for timm and its augmentation settings. found that it worked well for manvMixer-1024113 and stuck with them for the proceeding experiments. We admit this is not an optimal strategy, however, we were aware from our early experiments on CIFAR-10 that results seemed robust to various small changes. We did have access to sufficient compute attempt to tune hyperparameters for each model: e.g., larger ConvMixers could probably benefit from more regularization than we chose, and smaller ones from less regularization. Keeping the parameters the same across ConvMixer instances seemed more reasonable than guessing for each.
However, to some extent, we changed the number of epochs per model: for earlier experiments, merely wanted "proof of concept", and used only 90-100 epochs. Once we increased this to 150 epochs and trained some larger models, namely ConvMixer 1024/20 with p 14 patches and ConnMMixr-11366 with p 7 patches. Then, believing that we should explore
potential,
8

---

Under review as a conference paper at ICLR 2022
deeper-but- weddes--wde wide ConvMixers, and knowing from CIFAR-10 10 that the deeper models converged more slowly, we trained ConvMixer- 768/32s with p = 14 and p = 7 for 300 epochs. Of course, training time was consideration: Conv Seixer--136/200 1536/20 took about 9 days to train (on x0 RTX8000s) 150 epochs, and ConvMixer-666/32 is over twice as fast, making 300 epochs more feasible.
If anything. we believe that in the worst case, the lack of parameter tuning in our experiments resulted in underestimating the accuracies of ConvMixers. Further, due to our limited compute and the fact that large models (particularly ConvMixers) are expensive train on large data sets, we generally trained our models for fewer epochs than competition like DeiT and ResMLP (see Table 2).
A note on throughput. We measured throughput using batches of 64 images in half precision single RTX8000 GPU, averaged over 20 such batches. In particular, we measured CUDA execution time rather than "wall-clock" time. We noticed discrepancies in the relative throughputs of models, e.g., Touvron et al. (2020) reports that ResNet- 155 is 2x faster than DeïT-B, but our measurements show that is only 1.25x faster. We therefore speculate that our throughputs may underestimate the performance of ResNets and MonvMixers relative to the transformers. The difference may be due to using RTX8000 rather than V100 GPUs, or other low-level differences. Our throughputs were similar for batch sizes 32 and 128.
ResNets. As simple baseline to which to compare ConvMixers, we trained three standard ResNets using exactly the same training setup and parameters as ConvMixer-1536221 Despite having fewer parameters and being architecturally much simpler, ConvMixers substantially outperform these ResNets in terms of accuracy. A possible confounding factor is that ConvMixers may boost performance, while ResNets use ReLU. In an attempt to rule out this confound, we used ReLU in a later ConvMixer-768/32 experiment and found that still achieved competitive accuracy. We also note that the choice of ReLU vs. GELU was not important on CIFAR-10 experiments (see Table 3). However, ConeMixers do have substantially less throughput.
GELU, which
DeiTs. We believe that DeiT is the most reasonable comparison in terms of vision transformers: It only adds additional regularization, as opposed to architectural additions in the case of CaiT (Touvron et al., 20) and is then essentially a "vanilla" VIT modulo the distillation token (we don't consider distilled architectures). In terms of fixed parameter budget, ConyMixers generally outperform DeiTs. For example, ConvMixer- 1 1536/20 is only 0.43% less accurate than DeÎT-B despite having over 30M fewer parameters; MoneMiier--Pa 768/32 is 0.36% more accurate than DeiT-S despite having 0.9M fewer parameters; and same number of parameters. Admittedly, none of the ConvMixers are very competitive in terms of throughput, with the closest being the ConvMixer-5 522/16 which is 5x slower than DeiT-Ti.
12/16 is 0.39% more accurate than DeiT-Ti for nearly the
A confounding factor is the difference in patch size between DeiT and ConvMixer; DeiT uses p = 16 while ConvMixer uses p 7. This means DeiT' is substantially faster. However, ConvMixers using larger patches are not as competitive While we were not able to train DeiT' with larger patch sizes, is possible that they would outperform ConyMixers the parameter count vs. accuracy curve; however, we tested their throughput for p 7, and they are even slower than ConvMixers. Given the difference between convolution and self-attention we are not sure is salient to control for patch size differences.
DeiT's were subject to more hyperparameter tuning than ConyMixers, as well as longer training times. They also used stochastic depth while we did not, which can in some cases contribute percent differences in model accuracy (Touvron et al., 2021a). It is therefore possible that further hyperparameter tuning and more epochs for ConvMixers could close the gap between the two architectures for large patches, e.g., 16.
ResMLPs. Similarly to DeiT for ViT, we believe that ResMLPis the most relevant MLP-Mixer variant to compare against. Unlike DeîT, we can compare against instances of ResMLP with similar patch size: ResML.P-24/8 has p = patches, and underperforms ConvMixer- Sinieer-13366 1536/20 by 0.37%, despite having over twice the number of parameters; it also has similarly low throughput. 70nMM13en7768/32 also outperforms ResMLP-S122 for millions fewer parameters, but 3x less throughput.
ResMLP did not significantly improve in terms of accuracy for halving the patch size from 16 to 8, which shows that smaller patches do not always lead to better accuracy for a fixed architecture and regularization strategy (e.g., training a p 8 DeiT may be challenging).
Swin Transformers. While we intend to focus on the most basic isotropic, patch- architectures for fair comparisons with Mixer, it is also interesting to compare to a more complicated model

---

Under review as a conference paper at ICLR 2022
that is closer to state-of-the-an For similar parameter budget, ConvMixer is around 1.2- 1.2-1.6% less accurate than the Swin Transformer, while also being 4-6x slower. However, considering we did not attempt to tune optimize our model in any way, find it surprising that exceedingly simple patch-based model that uses only plain convolution does not lag too far behind Swin Transformer.
Isotropic MobileNets. These models closest in design to ours, despite using repeating block that is substantially more complex than the ConvMixer one. Despite this, for a similar number of parameters, we can get similar performance. Notably, isotropic MobileNets seem suffer less from larger patch sizes than Mixers which makes us optimistic that sufficient parameter tuning could lead to more performant large-patch Cixerss
Other models. We included VIT and MLP-Mixer instances in our table, though they are not compet- itive with ConvMixer, DeiT, or ResMLP, even though MLP-Mixer has comparable regularization to Mixer That is, ConvMixer seems to outperform MLP-Mixer and VIT. while being closer to com- plexity to them terms of design and training regime than the other competitors, DeřT and ResMLP.
Kernel size. While we found some evidence that larger kernels are better on CIFAR-10, 10, we wanted to see if this finding transferred to ImageNet. Consequently, we trained our best-performing model, Conv MineM 1536/20, with kernel size k = 3 rather than & = 9. This resulted in decrease of 0.94% top-1 accuracy, which we believe quite significant relative to the mere 2.2M additional parameters. However, & 3 is substantially faster than & 9 for spatial-domain convolution; that low-level optimizations could close the performance gap to some extent, e.g., by using implicit instead of explicit padding. Since large-kernel convolutions throughout model are unconventional, there has likely been low demand for such optimizations.
speculate
B EXPERIMENTS ON CIFAR-10 10
Residual connections. We experimented with leaving out one, the other, or both residual connections before settling the current configuration, and consequently chose leave out the second residual connection. Our baseline model without the connection achieves 95.88% accuracy, while including the connection reduces removing all residual connections. We acknowledge that these findings for residual connections may generalize deeper ConvMixers trained on larger data sets.
44.78%. Surprisingly, we see only 0.31% decrease in accuracy for
Ablation of ConvMixer:256/8 on CIFAR-10
CIFAR-10 Acc. (%)
Ablation
Baseline
95,88
Residual in Eq. 2
95.57
Residual in Eq. 3
94.78
BatchNorm - LayerNorm
94.44
GELU ReLU
95.51
Mixup and CutMix
95.92
Random Erasing
95.24
RandAug
92.86
Random Scaling
86.24
Gradient Norm Clipping
86.33
Table 3: Small ablation study of training ConvMixer-25688 on CIFAR-
Normalization. Our model is conceptually similar to the vision transformer and MLP-Mixer, both of which use LayerNorm instead of BatchNorm. We attempted to use LayerNorm instead, and saw a decrease performance of around 1% as well as slower convergence (see Table 3). However, this was for relatively shallow model, and cannot guarantee that LayerNorm would not hinder ImageNet-scale models to an even larger degree. We note that the authors of ResMLP also saw relatively small increase in accuracy for replacing LayenNorm with BatchNorm, but for larger-
10

---

Under review as a conference paper at ICLR 2022
scale experiment (Touvron al., 2021a). We conclude that BatchNorm is more crucial to architecture than other regularizations or parameter settings (e.g., kernel size).
Having settled decay on CIFAR- 10 experiments. (Initially, we took the unconventional approach of excluding weight decay since were already using strong regularization in the form of RandAug and mixup.) We acknowledge that tuning our architecture on CIFAR- 10 does not necessarily generalize to performance on larger data sets, and that this is limitation of our study.
architecture, proceeded to adjust its parameters h, d, p, A well weight
B.1 RESULTS
ConvMixers are quite performant on CIFAR-10, easily achieving > 91% accuracy for as little as 100, 000 parameters, or > 96% accuracy for only 887, 000 parameters (see Table 4). With additional refinements e.g., a more expressive classifier or bottlenecks, we think that ConvMixer could be even more competitive. For all experiments, we trained for 200 epochs on CIFAR-10 with RandAug, mixup, cutmix, random erasing, gradient norm clipping, and the standard augmentations timm. We remove some of these augmentations in Table 3, finding that RandAug and random scaling ("default" in t timm) are very important, each accounting for over 3% of the accuracy.
Scaling MixeM We adjusted the hidden dimension h and the depth d, finding that deeper networks take longer to converge while wider networks converge faster. That said, increasing the width or the depth is an effective way increase accuracy; doubling of depth incurs less compute than doubling of width. The number of parameters in a ConvMixer is given exactly by:
#params = h]d(k?2 + h h + CinP" nllaases + 3] + Relasses:
(4)
including affine scaling parameters BatchNorm layers, convolutional kernels, and the classifier.
Kernel size. We initially hypothesized that large kernels would be important for ConvMixers, as they would allow the mixing of distant spatial information similarly unconstrained MLPs self- attention layers. We tried to investigate the effect of kernel size on CIFAR-10: we fixed the model to be a 20n/ and increased the kernel size by 2s from to 15.
Using a kernel size of 3. the ConvMixer only achieves 93.61% accuracy. Simply increasing it to gives an additional 1.50% accuracy, and further an additional 0.61%. The gains afterwards are relatively marginal, with kernel size 15 giving additional 0.28% accuracy. It could be that with more training iterations or more regularization, the effect of larger kernels would be more pronounced. Nonetheless, we concluded that ConvMixers benefit from furger-than-use kernels, and thus used kernel sizes or in most of our later experiments.
It is conventional wisdom that large-kernel convolutions can be "decomposed" into stacked small- kernel convolutions with activations between them, and it is therefore standard practice to use = 3 convolutions, stacking more of them to increase the receptive field size with additional benefits from nonlinearities. This raises a question: is the benefit of larger kernels in ConyMixer actually better than simply increasing the depth with small kernels? First, note that deeper networks generally harder to train, so by increasing the kernel size independently of the depth, we may recover some of the benefits of depth without making harder for signals to "propagate back" through the network. To test this, we trained a ConvMixer-256// with k = 3 (698K parameters) in the same setting as 2on with k: (707K parameters), i.e., we increased depth small- kernel model to roughly match the parameters of a large-kerna model. The ConvMixer-2560 achieved 94.29% accuracy (1.5% less), which provides more evidence for the importance of larger kernels in Cixers Next, instead of fixing the parameter budget, we tripled the depth (using the intuition that stacked Is = 3 convolutions have the receptive field a k = convolution), giving a 256/2/2 with 1670K parameters, and got 95.16% accuracy, i.e.. still less.
Patch size. CIFAR- 10 inputs are so small that we initially only used p = 1, i.e., the patch embedding layer does little more than compute h linear combinations of the input image. Using reduction in accuracy of about 0.80%; this a worthy tradeoff in terms of training and inference time. Further increasing the patch size leads to rapid decreases in accuracy, with only 92.61% for 4.
2,
Since the "internal resolution" is decreased by a factor of p when increasing the patch size, we assumed that larger kernels would be less important for larger p. We investigated this by again
11

---

Under review as conference paper at ICLR 2022
increasing the kernel size from 3 to 11 for ConvMixer- 2566 with p 2: however, this time, the improvement going from to 5 is only 1.13%, and larger kernels than provide only marginal benefit.
Weight decay. We did many of our initial experiments with minimal weight decay. However, this was not optimal: by tuning weight decay, we can get an additional 0.15% of accuracy for no cost. Consequently, used weight decay (without tuning) for larger-scale experiments ImageNet.
Tiny ConvMixers trained on CIFAR-I0.
Width
Depth d
Patch Size p
Kernel Size
Params (x10")
Weight Decay
CIFAR-10 10 Acc. (%)
128
103
0
91.26
128
8
205
0
93.83
128
12
306
0
94.83
256
338
0
93.37
256
8
8
672
0
95.60
256
12
1006
0
96.39
256
16
8
1339
0
96.74
256
20
1673
96.67 0
Kernel adjustments
256
8
559
0
93.61
256
592
95.19 0
256
8
641
95.80 0
256
707
0
95.88
256
8
11
788
0
95.70
256
887
96.04
256
8
15
1001
96.08 0
4 Patch adjustments
256
8
2
709
0
95.00
256
8
4
718
0
92.61
256
8
8
755
85.57 0
Weight decay adjustments
256
8
707
1 10
95.88
256
8
707
1 10
96.03
256
8
707
1 10
95.76
256
8
707
1 1 10-4
95.63
256
8
707
1 1
10
95.88
1 Kernel size adjustments when p = 2
256
8
2
3
561
0
94.08
256
2
594
0
95.21
256
8
2
7
643
0
95.35
256
8
2
709
0
95.00
256
8
2
11
791
0
95.14
1 Adding weight decay to the above
256
2
561
1 10-2
94.69
256
8
2
594
1 10
95.26
256
8
2
7
643
1 1 10
95.25
256
8
2
709
1 10
95.06
256
8
2
791
1 10
95.17
Table 4: An investigation of ConvMixer design parameters h, d, d, k and weight decay CIFAR-10
12

---

Under review as a conference paper at ICLR 2022
C
WEIGHT VISUALIZATIONS
Figure 4: Patch embedding weights for MonMiieer-124/200 with patch size 14 (see Table 2).
Figure 5: Patch embedding weights for a Con/Mixer-768/32 with patch size (see Table 2).
13

---

Under review as conference paper at ICLR 2022
(a) Layer
Layer
(c) Layer
(d) Layer 9
(c) Layer 11
(f) Layer
(g) Layer 15
(h) Layer 17
ii) Layer 19
Figure 6: Random subsets of 64 depthwise convolutional kernels from progressively deeper layers of ConvMixer- (see Table l).
In Figure and 5, we visualize the (complete) weights of the patch embedding layers of ConvMixer- 1536/20 with = 14 and 7668/12xr-768832 with p 7, respectively, Much like Sandler et al. (2019), the layer consists of Gabor-like filters well "colorful globs" rough edge detectors. The filters seem to more structured than those learned by MLP-Mixer (Tolstikhin et al., 2021); also unlike MLP-Mixer, the weights look much the same going from p 14 to p 7: the latter simply looks like a downsampled version of the former. is unclear, then, why we see such a drop in accuracy for larger patches. However, some of the filters essentially look like noise, maybe suggesting a need for more regularization or longer training, or even more data. Ultimately, we cannot read too much into the learned representations here.
In Figure 6, we plot the hidden convolutional kernels for successive layers of ConvMixer. Ini- tially, the kernels seem to be relatively small, but make use of their allowed full size later layers; there is a clear hierarchy of features as one would expect from a standard convolutional architec- ture. Interestingly, Touvron look like small-kernel convolution, while later layers were more diffuse, despite these layers being representated by an unconstrained matrix multiplication rather than convolution.
(2021a) saw similar effect for ResMLP, where earlier layers
14

---

Under review as conference paper at ICLR 2022
D IMPLEMENTATION
import torch.nn
2
class Residual (nn. Module):
def _init_ (self, fn):
super O)
self.fn
def fsel,rd(self,, x):
return self. fn(x)
def ConvMixer depth, kernel_size=9, sizeh, t_classes=1000):
12
return nn. Sequential(
nn. Conv2d dim, kermel stride-patch.size)
GELUO,
nn BatchNormmdddim))
16
Semuenttnt
17
Residual Sequential
nn. Conv dim, kernel_size, groups-dim, madding="same").
nn. GELUO,
nn. (...Batchhorm2((iim)
21
)
nn. (dim, dim, kernel_size=1),
nn. GELUO)
) for (dimh))orm2dddi))
(depth))
26
nn. adapiiveavgPoo22(((11.)) nn. Fm.alteenn), Ln. n_classes)
Figure 7: more readable PyTorch (Paszke et al., 2019) implementation of ConvMixer, where dim, d = depth, p patch_size, k kernel_size.
def ConvMixr d, k..,.)
S.C.A-Sequential, Conpuuentia,,,nnnd,,,,. lambda lambaa GELUO, Batchllorm2d(h)) x::CxccEEU()BBBtcchomm2)())
R-type(" (S.). forward' :lambda s.x: (x) +x})
return S(ACC(3,h.p.p)). (SCCCACCCCCCC wwwpuppshh.ppddggg////oooo ACCh,,,, for i + in (atpgeddd....ppiivvevgolll....1 Fllatten() ,Linear(h,n))
(d)
Figure 8: An implementation of our model exactly 280 characters, in case you happen to know of any means of disseminating information that could benefit from such length. All you need to do to run this is from torch.m. nn import
This section presents expanded (but still quite compact) version of the terse ConvMixer imple- mentation that we presented paper. The code is given in Figure 7. We also present an even more terse implementation in Figure 8, which to the best of our knowledge is the first model that achieves the clusive dual goals of 80%+ ImageNet top-1 accuracy while also fitting into tweet.
15


Reviews:
[{"review_id": "wtX7HGDvFC5", "reviewer": "Reviewer_J6jX", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1.\tThe model is interesting and simple, which would inspire the vision community a lot.\n2.\tThe results are promising, especially on the accuracy. Even a simple depth-wise / point-wise convnet without resolution reduction can achieve a promising result.\n3.\tAn interesting but not discussed finding is that, different from VIT  and MLP-mixer (and variants) that emphasize the global context, this work did not introduce global context (although the receptive field is large enough). A close look at ResNet and BagNets [1] would be interesting.\n\nWeaknesses: \n1.\tTypos, e.g. , “simiar” in page 2; “do ConvMixers” in page 8, etc.\n2.\tAt the end of Section 1, the authors claim that “This suggests that, at least to some extent, the patch representation itself may be the most critical component to the superior performance of newer architectures like Vision Transformers.” I don’t think this is true, and the authors have not fully verified such a claim. There are many other factors, like learnable parameters, effective receptive fields, etc. This statement seems to be overclaimed.\n3.\tMost importantly, I don’t think this method achieves “better” results than others.  i), Comparing with ResNet152, the throughout is 89 vs. 872. The slow speed of ConvMixer is not acceptable in real-world applications; ii) Besides, the performance of ResNet would be higher than reported if trained with appropriate strategies [2].  iii) Small patch properly can achieve better performance.  I would suggest authors try some other Transformer models, like Swin Transformer [3].\n4.\tIn future work of page 4, authors claim that “We are optimistic that a deeper ConvMixer with larger patches ... and hyperparameter tuning”. However, this may not be true. Please see Appendix A. I have not seen the strong superiority of ConvMixer over the ResNet in the accuracy with similar parameters. And the ConvMixer’s speed is far below the ResNet.\n5.\tIn VITs and MLPs (including their variants), we have two skip connections (it is not called residual connection as shown in Fig 2) in a block. However, authors designed with only one (tuned on cifar10). I just want to know if we add two skip connections in ConvMixer, what is the result for ImageNet.\n6.\tI’m not a fan of cifar10 dataset due to the dataset size and inevitable randomness in results. Considering the limited computational resources, some other datasets may be helpful (like cifar100, Tiny-ImageNet, etc.) or authors can report the mean/std results of 5 runs on cifar10.\n7.\tI could not get the idea of “A note on paper length” paragraph in a conference paper, considering some of the important experiments are missing. It is nothing to do with the main idea of ConvMixer.\n8.\tSome important ablation studies are missing, like the influence of depth, kernel size, patch sizes, etc (I mean detailed ablation studies). I would also be interested in the performance of ConvMixer on some downstream tasks like detection. \n\n[1] Brendel, Wieland, and Matthias Bethge. \"Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNet.\" International Conference on Learning Representations. 2018.\n\n[2] Wightman, Ross, Hugo Touvron, and Hervé Jégou. \"ResNet strikes back: An improved training procedure in timm.\" arXiv preprint arXiv:2110.00476 (2021).\n\n[3] Liu, Ze, et al. \"Swin transformer: Hierarchical vision transformer using shifted windows.\" arXiv preprint arXiv:2103.14030 (2021).\n", "summary_of_the_paper": "This work proposed a new design for the image classification task named ConvMixer, which brings the idea from CNN to Visual Transformer. Unlike previous ConvNets, Transformer-based models, and MLP-based models, ConvMixer simply applies depth-wise (with skip connection) and point-wise convolutions on the patches. The key point authors claimed is that using patch embeddings is a powerful and important takeaway besides the architecture design. Experiments on ImageNet demonstrate the effectiveness of the proposed model.", "summary_of_the_review": "I recommend reject due to the weaknesses above, and I will consider updating my score if seeing a strong rebuttal.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ocgid221QSc", "reviewer": "Reviewer_RgZh", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Summary**: Overall, I believe this is a good paper. It consist on a simple model which is very well described and one single experiment which shows well the advantages of the paper. However, I believe the authors are missing a few points which are worth addressing before the paper is published.\n\n**A note of the paper length**: I understand the message from the authors about the paper length, but as a reviewer, I needed to check the Appendix many times to produce a complete review of the paper. Although I can agree this is a good discussion to have as a community, I don't think the note from the authors is fair as if reviewers only looked at the main paper (which is supposed to be sufficient for a full understanding of the paper), many relevant points would be missed. For this reason, I would recommend the authors to move to the main paper some sections which provide relevant insights (for instance, Appendix A gives very relevant details). \n\n**Strengths**:\n1. The model is very simple and effective. I agree with authors that this is a relevant piece of research for the community, as it disentangles a bit the performance improvement from the transformer architecture.\n\n2. In terms of reproducibility, I believe this can be a great baseline for future research. Code is simple and authors seem to imply that it does not need a lot of hyper-parameter tuning to make it work.\n\n3. The model description is clear and well written. I think the figures and the text are very clear and the community will really appreciate that.\n\n4. The experimental results on ImageNet-1k are very good. Specially, when taking into account the parameter count, the model seems to show very good performance overall.\n\n\n**Weaknesses and proposals for improvement**:\n\n1. Patches and architecture: authors introduce the model as validation of the idea of the patches being important for final performance on SOTA architecture such as transformers (see title and abstract). However, in the paper they introduce the idea of using patches in convolutional networks along with a novel architecture, but they never decouple the gains of this two contributions in the experimental section. If authors want to claim that patches are relevant for final performance, in my opinion they should ablate only that change in an already known architecture. \n\n2. Although the experimental results in the paper are encouraging, in my opinion authors are missing a very relevant point about pre-training. It is clear that transformers do need large scale pre-training for them to get high performance. I would be very curious to see this model operating on that data regime. Otherwise, I am not sure if we can conclude much, as it was already known that these large models were not very good with regimes without much data. \n\n3. The first paragraph of the experimental section is a bit confusing for me. If authors want to include the CIFAR experiments in the paper, they should just include that table in the main paper. If they think those experiments are not relevant, then they should not start the experimental section by discussing those experiments. But providing the table in the Appendix but some discussion in the main paper is very confusing for the reader. \n\n4. Other than large-scale pre-training, I would be curious to see the performance of this model on other classification task. Authors include CIFAR, which is a small dataset. But I think it would be useful for the community to benchmark the proposed model to other classification tasks such as scene classification, fine-grained recognition, etc. Essentially, evaluate the ability of the model to perform on other domains and make sure the results are not overfitting to object classification.\n\n5. Following up with the previous points, isotropic models should be good to perform segmentation-like tasks, as they keep most of the input resolution (except for the patching). I am wondering if that could be an additional experiment to include. \n\n6. Looking at the results, it is clear that the model has lower throughput than its competitors. Figure 1 revolves around the parameter count of the model, but I believe it would be fair for the authors to include also some discussion about the throughput. I realised they had included some in the Appendix, but I believe authors should include a proper discussion on the effects of this low throughput in the main paper, as I believe it's one of the main weaknesses of the model. \n", "summary_of_the_paper": "The paper presents a very simple architecture which consist of patching the input image and then applying a combination of depth-wise and point-wise convolutions. In the paper, authors evaluate the performance of this model when used for image classification. In their main experiment, they train the architecture using only the ImageNet-1k dataset. On that experiment, they show how their simple architecture is competitive with state-of-the-art architectures. ", "summary_of_the_review": "I believe the paper is good and it should be accepted. It will be a relevant contribution to the community as a baseline as well as a good datapoint to improve our understanding of SOTA models. However, I believe the paper would be more complete if authors evaluated in additional task. Furthermore, I think authors should address some relevant points when comparing to other models such as data scale and throughput. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DFwPFtdXPHY", "reviewer": "Reviewer_jSQ1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\nWell written. Clear motivation and the problem raised in this work is very interesting.\n\nCons:\n\na. Although the proposed ConvMixer illustrates very promising results compared with its counterparts, such as DeiT, MLPMixer, and ResNets, the main claim of this work:  \"the patch representation may be the most critical component\" has not been sufficiently well validated: \n- Compared to other previous convolution-based network architectures, such as ResNet, the ConvMixer has too many factors beyond the patch representation itself that may lead to performance differences, such as data augmentation, appropriate hyper-parameters, optimizer and other macro/micro network architectures, For example, the ConvMixer adopts the spatial-conv + pointwise-conv as its basic building block, which is also an unfair comparison with ResNets. In addition, through carefully tuning, [1] reports a much higher ResNet performance than this work.\n\n- Compared to vision transformer/mlp-based networks, this work only considers some computation/parameter unfriendly architectures, such as DeiT, while ignoring some recent progress in this field, like Swin. \n\n- Lack of comparison for downstream tasks, such as object detection/semantic segmentation. This is necessary to fully understand the importance of patch representation in a broader range of tasks.\n\nThese under-explored factors challenge the conclusion/connection of this work that the 'patch representation' is the key.\n \n[1] ResNet strikes back: An improved training procedure in timm\n", "summary_of_the_paper": "This work proposes a new CNN architecture: ConvMixer, which integrates some techniques/techniques used in the recently proposed visual transformer model/mlp model. ConvMixer is simple and effective, and it also prompts us to think deeper about where is key to improving performance in the vision transformer and mlp models.", "summary_of_the_review": "This work is well written and presents a very interesting conjecture/conclusion that the patch representation may be the key to the success of visual transformers and visual Milp. However, its research is not strong enough to support its claims.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper observes that a fully-convolutional model in the style of recent MLP-Mixer and ViT variants can have surprisingly good initial performance. As this paper attracts certain amount of attentions, three expert reviewers have provided very detailed and serious comments, and two actively engaged with author discussions. AC also carefully read the paper as well as all discussion threads.

AC agrees the authors should not be penalized by not achieving the best performance, nor not comparing with very recent work. The main legitimate critiques, however, focus on three aspects: (1) over-claimed contribution; (2) experiment solidness/competitiveness; and (3) writing completeness/clarity.

First, this paper established an interesting ablation experiment that a very simple model, that uses only standard convolutions to achieve the mixing steps, can roughly "do the work". However, AC disagrees this is a very "surprisingly new" result, on top of MLP-mixer: given convolutions are increasingly re-injected into ViTs to gain the vision inductive bias, their similar role in MLP-mixer should be expected too. Moreover, as in general agreement by reviewers, the paper title might have over-claimed - the authors cannot directly prove this concept "patch is the most critical component" yet. The authors later also agreed and changed some confusing wording, which is a good move (but also, making their contribution now even less obvious). 

Second, this method does not achieve noteworthy competitive results compared to others, in order to justify its merit (simplicity alone is good to have, but insufficient to justify a strong work). Importantly, it has been pointed out by two reviewers that the model throughput is much worse than the competitors. AC also noticed that the comparison was not very rigorous, e.g., comparing ConvMixer patch size 7 with DeiT-B 16 patch size 16 doesn't help draw much fair informative conclusion. The cifar-10 results alone did not provide strong support and were later de-emphaszied by authors too. 

Third, while NOT being the main reason of rejection, AC personally suggests the authors to responsibly enrich their main text, and to remove the  “A note on paper length” paragraph. The authors intentionally kept the paper length unusually short. Reviewers generally dislike this idea. Being an innovative writer is good, but very relevant details and discussions were left in the supplemental as a result. Especially, AC agrees the whole section A and part of section B of the supplemental should have been in the main paper at very least.

In summary, the authors strive to tell an interesting story, but it is not yet a well settled story. The experiments are not solid enough to support their bold claims. The authors are suggested to improve their work further by taking into account reviewer comments.
Decision: Reject