Patches Are All You Need?

1 Introduction For many years , convolutional neural networks have been the dominant architecture for deep learning systems applied to computer vision tasks . But recently , architectures based upon Transformermodels , e.g. , the so-called Vision Transformer architecture ( Dosovitskiy et al. , 2020 ) , have demonstrated compelling performance in many of these tasks , often outperforming classical convolutional architectures , especially for large data sets . An understandable assumption , then , is that it is only a matter of time before Transformers become the dominant architecture for vision domains , just as they have for language processing . In order to apply Transformers to images , however , the representation had to be changed : because the computational cost of the self-attention layers used in Transformers would scale quadratically with the number of pixels per image if applied naively at the per-pixel level , the compromise was to first split the image intomultiple “ patches ” , linearly embed them , and then apply the transformer directly to this collection of patches . In this work , we explore the question of whether , fundamentally , the strong performance of vision transformers may result more from this patch-based representation than from the Transformer architecture itself . We develop a very simple convolutional architecture which we dub the “ ConvMixer ” due to its similarity to the recently-proposed MLP-Mixer ( Tolstikhin et al. , 2021 ) . This architecture is similar to the Vision Transformer ( and MLP-Mixer ) in many respects : it directly operates on patches , it maintains an equal-resolution-and-size representation throughout all layers , it does no downsampling of the representation at successive layers , and it separates “ channel-wise mixing ” from the “ spatial mixing ” of information . But unlike the Vision Transformer and MLP-Mixer , our architecture does all these operations via only standard convolutions . The chief result we show in this paper is that this ConvMixer architecture , despite its extreme simplicity ( it can be implemented in ≈ 6 lines of dense PyTorch code ) , outperforms both “ standard ” computer vision models such as ResNets of similar parameter counts and some corresponding Vision Transformer andMLP-Mixer variants , even with a slate of additions intended to make those architectures more performant on smaller data sets . Importantly , this is despite the fact that we did not design our experiments to maximize accuracy nor speed , in contrast to the models we compared against . Our results suggest that , at least to some extent , the patch representation itself may be a critical component to the “ superior ” performance of newer architectures like Vision Transformers . While these results are naturally just a snapshot , and more experiments are required to exactly disentangle the effect of patch embeddings from other factors , we believe that this provides a strong “ convolutionalbut-patch-based ” baseline to compare against for more advanced architectures in the future . 2 A Simple Model : ConvMixer . Ourmodel , dubbedConvMixer , consists of a patch embedding layer followed by repeated applications of a simple fully-convolutional block . We maintain the spatial structure of the patch embeddings , as illustrated in Fig . 2 . Patch embeddings with patch size p and embedding dimension h can be implemented as convolution with cin input channels , h output channels , kernel size p , and stride p : z0 = BN ( σ { Convcin→h ( X , stride=p , kernel_size=p ) } ) ( 1 ) The ConvMixer block itself consists of depthwise convolution ( i.e. , grouped convolution with groups equal to the number of channels , h ) followed by pointwise ( i.e. , kernel size 1 × 1 ) convolution . As we will explain in Sec . 3 , ConvMixers work best with unusually large kernel sizes for the depthwise convolution . Each of the convolutions is followed by an activation and post-activation BatchNorm : z′l = BN ( σ { ConvDepthwise ( zl−1 ) } ) + zl−1 ( 2 ) zl+1 = BN ( σ { ConvPointwise ( z′l ) } ) ( 3 ) After many applications of this block , we perform global pooling to get a feature vector of size h , which we pass to a softmax classifier . See Fig . 3 for an implementation of ConvMixer in PyTorch . Design parameters . An instantiation of ConvMixer depends on four parameters : ( 1 ) the “ width ” or hidden dimension h ( i.e. , the dimension of the patch embeddings ) , ( 2 ) the depth d , or the number of repetitions of the ConvMixer layer , ( 3 ) the patch size p which controls the internal resolution of the model , ( 4 ) the kernel size k of the depthwise convolutional layer . We name ConvMixers after their hidden dimension and depth , like ConvMixer-h/d . We refer to the original input size n divided by the patch size p as the internal resolution ; note , however , that ConvMixers support variable-sized inputs . Motivation . Our architecture is based on the idea of mixing , as in Tolstikhin et al . ( 2021 ) . In particular , we chose depthwise convolution to mix spatial locations and pointwise convolution to mix channel locations . A key idea from previous work is that MLPs and self-attention can mix distant spatial locations , i.e. , they can have an arbitrarily large receptive field . Consequently , we used convolutions with an unusually large kernel size to mix distant spatial locations . While self-attention and MLPs are theoretically more flexible , allowing for large receptive fields and content-aware behavior , the inductive bias of convolution is well-suited to vision tasks and leads to high data efficiency . By using such a standard operation , we also get a glimpse into the effect of the patch representation itself in contrast to the conventional pyramid-shaped , progressivelydownsampling design of convolutional networks . 3 Experiments . Training setup . We primarily evaluate ConvMixers on ImageNet-1k classification without any pretraining or additional data . We added ConvMixer to the timm framework ( Wightman , 2019 ) and trained it with nearly-standard settings : we used RandAugment ( Cubuk et al. , 2020 ) , mixup ( Zhang et al. , 2017 ) , CutMix ( Yun et al. , 2019 ) , random erasing ( Zhong et al. , 2020 ) , and gradient norm clipping in addition to default timm augmentation . We used the AdamW ( Loshchilov & Hutter , 2018 ) optimizer and a simple triangular learning rate schedule . Due to limited compute , we did absolutely no hyperparameter tuning on ImageNet and trained for fewer epochs than competitors . Consequently , our models could be over- or under-regularized , and the accuracies we report likely underestimate the capabilities of our model . Results . A ConvMixer-1536/20 with 52M parameters can achieve 81.4 % top-1 accuracy on ImageNet , and a ConvMixer-768/32 with 21M parameters 80.2 % ( see Table 1 ) . Wider ConvMixers seem to converge in fewer epochs , but are memory- and compute-hungry . They also work best with large kernel sizes : ConvMixer-1536/20 lost ≈ 1 % accuracy when reducing the kernel size from k = 9 to k = 3 ( we discuss kernel sizes more in Appendix A & B ) . ConvMixers with smaller patches are substantially better in our experiments , similarly to Sandler et al . ( 2019 ) ; we believe larger patches require deeper ConvMixers . With everything held equal except increasing the patch size from 7 to 14 , ConvMixer-1536/20 achieves 78.9 % top-1 accuracy but is around 4× faster . We trained one model with ReLU to demonstrate that GELU ( Hendrycks & Gimpel , 2016 ) , which is popular in recent isotropic models , isn ’ t necessary . Comparisons . Our model and ImageNet1k-only training setup closely resemble that of recent patchbasedmodels likeDeiT ( Touvron et al. , 2020 ) . Due toConvMixer ’ s simplicity , we focus on comparing to only the most basic isotropic patch-based architectures adapted to the ImageNet-1k setting , namely DeiT and ResMLP . Attempting a fair comparison with a standard baseline , we trained ResNets using exactly the same parameters asConvMixers ; while this choice of parameters is suboptimal ( Wightman et al. , 2021 ) , it is likely also suboptimal for ConvMixers , since we did no hyperparameter tuning . Looking at Table 1 and Fig . 1 , ConvMixers achieve competitive accuracies for a given parameter budget : ConvMixer-1536/20 outperforms both ResNet-152 and ResMLP-B24 despite having substantially fewer parameters and is competitive with DeiT-B . ConvMixer-768/32 uses just a third of the parameters of ResNet-152 , but is similarly accurate . Note that unlike ConvMixer , the DeiT and ResMLP results involved hyperparameter tuning , andwhen substantial resources are dedicated to tuning ResNets , including training for twice as many epochs , they only outperform an equivalently-sized ConvMixer by ≈ 0.2 % ( Wightman et al. , 2021 ) . However , ConvMixers are substantially slower at inference than the competitors , likely due to their smaller patch size ; hyperparameter tuning and optimizations could narrow this gap . For more discussion and comparisons , see Table 2 and Appendix A. CIFAR-10 Experiments . We also performed smaller-scale experiments on CIFAR-10 , where ConvMixers achieve over 96 % accuracywith as few as 0.7Mparameters , demonstrating the data efficiency of the convolutional inductive bias . Details of these experiments are presented in Appendix B . 4 Related Work . Isotropic architectures . Vision transformers have inspired a new paradigm of “ isotropic ” architectures , i.e. , those with equal size and shape throughout the network , which use patch embeddings for the first layer . These models look similar to repeated transformer-encoder blocks ( Vaswani et al. , 2017 ) with different operations replacing the self-attention and MLP operations . For example , MLP-Mixer ( Tolstikhin et al. , 2021 ) replaces them both with MLPs applied across different dimensions ( i.e. , spatial and channel location mixing ) ; ResMLP ( Touvron et al. , 2021a ) is a data-efficient variation on this theme . CycleMLP ( Chen et al. , 2021 ) , gMLP ( Liu et al. , 2021a ) , and vision permutator ( Hou et al. , 2021 ) , replace one or both blocks with various novel operations . These are all quite performant , which is typically attributed to the novel choice of operations . In contrast , Melas-Kyriazi ( 2021 ) proposed an MLP-based isotropic vision model , and also hypothesized patch embeddings could be behind its performance . ResMLP tried replacing its linear interaction layer with ( small-kernel ) convolution and achieved good performance , but kept its MLP-based cross-channel layer and did not explore convolutions further . As our investigation of ConvMixers suggests , these works may conflate the effect of the new operations ( like self-attention and MLPs ) with the effect of the use of patch embeddings and the resulting isotropic architecture . A study predating vision transformers investigates isotropic ( or “ isometric ” ) MobileNets ( Sandler et al. , 2019 ) , and even implements patch embeddings under another name . Their architecture simply repeats an isotropicMobileNetv3 block . They identify a tradeoff between patch size and accuracy that matches our experience , and train similarly performant models ( see Appendix A , Table 2 ) . However , their block is substantially more complex than ours ; simplicity and motivation sets our work apart . Patches aren ’ t all you need . Several papers have increased vision transformer performance by replacing standard patch embeddings with a different stem : Xiao et al . ( 2021 ) and Yuan et al . ( 2021a ) use a standard convolutional stem , while Yuan et al . ( 2021b ) repeatedly combines nearby patch embeddings . However , this conflates the effect of using patch embeddings with the effect of adding convolution or similar inductive biases e.g. , locality . We attempt to focus on the use of patches . CNNs meet ViTs . Many efforts have been made to incorporate features of convolutional networks into vision transformers and vice versa . Self-attention can emulate convolution ( Cordonnier et al. , 2019 ) and can be initialized or regularized to be like it ( d ’ Ascoli et al. , 2021 ) ; other works simply add convolution operations to transformers ( Dai et al. , 2021 ; Guo et al. , 2021 ) , or include downsampling to be more like traditional pyramid-shaped convolutional networks ( Wang et al. , 2021 ) . Conversely , self-attention or attention-like operations can supplement or replace convolution in ResNet-style models ( Bello et al. , 2019 ; Ramachandran et al. , 2019 ; Bello , 2021 ) . While all of these attempts have been successful in one way or another , they are orthogonal to this work , which aims to emphasize the effect of the architecture common to most ViTs by showcasing it with a less-expressive operation .

Summary: This work proposed a new design for the image classification task named ConvMixer, which brings the idea from CNN to Visual Transformer. Unlike previous ConvNets, Transformer-based models, and MLP-based models, ConvMixer simply applies depth-wise (with skip connection) and point-wise convolutions on the patches. The key point authors claimed is that using patch embeddings is a powerful and important takeaway besides the architecture design. Experiments on ImageNet demonstrate the effectiveness of the proposed model.