Caterpillar: A Pure-MLP Architecture with Shifted-Pillars-Concatenation

Authors: jin  sun
xiaoshuang  shi
zhiyuan  wang
kaidi  xu
tao  heng
  shen
xiaofeng  zhu
tao  shen
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-14 06:58:51

Abstract: Modeling in Computer Vision has evolved to MLPs.
Vision MLPs naturally lack local modeling capability, to which the simplest treatment is combined with convolutional layers.
Convolution, famous for its sliding window scheme, also suffers from this scheme of redundancy and lower parallel computation.
In this paper, we seek to dispense with the windowing scheme and introduce a more elaborate and parallelizable method to exploit locality. 
To this end, we propose a new MLP module, namely Shifted-Pillars-Concatenation (SPC), that consists of two steps of processes: (1) Pillars-Shift, which generates four neighboring maps by shifting the input image along four directions, and (2) Pillars-Concatenation, which applies linear transformations and concatenation on the maps to aggregate local features. SPC module offers superior local modeling power and performance gains, making it a promising alternative to the convolutional layer.
Then, we build a pure-MLP architecture called Caterpillar by replacing the convolutional layer with the SPC module in a hybrid model of sMLPNet. 
Extensive experiments show Caterpillar's excellent performance on both small-scale and ImageNet-1k classification benchmarks, with remarkable scalability and transfer capability possessed as well.
The code is available at https://github.com/sunjin19126/Caterpillar.

Main paper: 
 introduction: 
 Deep architectures in computer vision have evolved from Convolutional Neural Networks (CNNs), through Vision Transformers (ViTs), and now to Multi-Layer Perceptrons (MLPs). CNNs [16, 22, 39] primarily utilize convolution to aggregate local features but struggle to capture global dependencies between long-range pillars (tokens) in an image. ViTs [9, 44] employ self-attention mechanism to consider all pillars from a global perspective. Unfortunately, the self-attention mechanism suers from high computational complexity. To overcome this weakness, MLP-based models [42] replace the self-attention layers with simple MLPs to perform token(spatial)-mixing across the input pillars, thereby signicantly reducing the computational costs. However, early MLP models [42, 43] encounter the challenges in suciently incorporating local dependencies. As a solution, researchers have proposed hybrid models [27, 40] that combine convolutional layers with MLPs to achieve a balance between capturing local and global information, bringing stable performance improvements.Convolutional layers slide a local window across an image to introduce locality and translation-invariance, which have brought great successes for CNNs [16, 22] and also inspired a number of inuential ViTs [33, 52] . Nevertheless, convolution has inherent drawbacks. First, it may introduce redundancy, especially to the edge features. The convolution aggregates pixels in a local window with a larger receptive scope, while the edge features, such as shape and contour, often consist of only a few pixels that cannot fully ll the scope. Therefore, the edges can get mixed information with the background, leading to redundant representation. Additionally, the sliding window needs to encode features sequentially and individually at each position. This sequential nature leads to convolution calculations with limited parallel computing capability.In this paper, we seek to break the sequential windowing scheme and present an alternative to convolution. To this end, we propose a new MLP-based module called Shifted-Pillars-Concatenation (SPC), which consists of two processes: Pillars-Shift and Pillars-Concatenation. In Pillars-Shift, we shift the input image along four directions (i.e., up, down, left, right) to create four neighboring maps, with the local information for all pillars decomposed into four respective groups according to the orientation of neighboring pillars. In Pillars-Concatenation, we apply four linear transformations to individually encode these maps of discrete groups and then concatenate them together, with each pillar achieving the simultaneous and elaborate aggregation of local features from its four neighbors. Based on the proposed SPC, we introduce a pure-MLP architecture namely Caterpillar, which is built by replacing the (depth-wise) convolutional layer with the proposed SPC module in a Conv-MLP hybrid model (i.e., sMLPNet [40] ). The Caterpillar inherits the advantages of sMLPNet, which clearly separates the localand global-mixing operations in its spatial-mixing blocks and utilizes the sparse-MLP (sMLP) layer to aggregate global features (Figure 3 , left), while leveraging the SPC module to exploit locality.For experiments, we uniformly validate the direct application of the Caterpillar with various vision architectures (i.e., CNNs, ViTs, MLPs, Hybrid models) on common-used small-scale images [21, 47, 53] , among which the Caterpillar achieves the best performance on all used datasets. On the popular ImageNet-1K benchmark, the Caterpillar series attains better or comparable performance to recent state-of-the-art methods (e.g., Caterpillar-B, 83.7%). Caterpillar also possesses excellent scalability and transfer capability through corresponding experiments. On the other hand, in all experiments, the Caterpillars obtain higher accuracy than the baseline sMLPNets, while changing the convolution to the SPC module in ResNet-18 brings 4.7% top-1 accuracy gains on ImageNet-1K dataset, demonstrating the potential of SPC to serve as an alternative to convolution in both plug-and-play and independent ways. In summary, the major contributions of this paper are listed as follows:• We propose a novel SPC module, which adopts a windowfree scheme and can exploit local information more elaborately and simultaneously than convolution. transfer capability, as well as classication performance of the Caterpillar on both small-and large-scale image recognition tasks, with better performance of SPC than convolution. 
related work .1 local modeling approaches: 
 The idea of local modeling can be traced back to research on the organization of the visual cortex [18, 19] , which inspired Fukushima to introduce the Cognitron [10] , a neural architecture that models nearby features in local regions. Departing from biological inspiration, Fukushima further proposed Neocognitron [11] , which introduces weight sharing across spatial locations through a sliding window strategy. LeCun combined weight sharing with backpropagation algorithm and introduced LeNet [23] [24] [25] , laying the foundation for the widespread adoption of CNNs in the Deep Learning era. Since 2012, when AlexNet [22] achieved remarkable performance in the ImageNet classication competition, convolutionbased methods have dominated the eld of computer vision for nearly a decade. With the popularity of CNNs, research eorts have been devoted to improving individual convolutional layers, such as depth-wise convolution [54] and deformable convolution [7] . On the other hand, alternative approaches to replace convolution have also been explored, such as the shift-based methods involving sparse-shift [3] and partial-shift [30] . The idea behind these approaches is to move each channel of the input image in dierent spatial directions, and mix spatial information through linear transformations across channels. The proposed SPC module also builds upon the shift idea but shifts the entire image into four neighboring maps in the process of Pillars-Shift, while making use of the linear projections and concatenation in Pillars-Concatenation.CNNs and Vision Transformers. CNNs have achieved remarkable success in computer vision, with well-known models including AlexNet [22] , VGG [39] and ResNet [16] . The attention-based Transformer, initially proposed for machine translation [46] , has been successfully applied to vision tasks with the introduction of Vision Transformer (ViT) [9] . Since then, various advancements have been proposed to improve training eciency and model performance for ViTs, such as data-ecient training strategy [44] and pyramid architecture [33, 49] , which have also beneted the entire vision eld. At the core of Transformer models lies the multi-head selfattention mechanism. The proposed SPC module shares the similar operation to the multi-head settings, as it encodes local neighboring information from dierent representation subspaces with multiple linear lters in the Pillars-Concatenation process.Vision MLPs. Vision MLPs [2, 12, 17, 29, 41-43, 50, 55, 56] have also made signicant progress since the invention of MLP-Mixer [42] , which alternatively conducts the token-mixing (cross-location) operations and channel-mixing (per-location) operations to aggregate spatial and channel information, respectively. Early MLP-based models, such as MLP-Mixer [42] and ResMLP [43] , perform tokenmixing across all pillars from a global perspective, lacking the ability to eectively model local features. As a result, a number of studies propose to enhance MLPs with local modeling capabilities. Hire-MLP [12] , for instance, performs Innerand Cross-region Rearrangement to encode the local and global information in parallel, while AS-MLP [29] adopts an axial-shift strategy that shifts each channel of the image along two directions. Our Caterpillar is built with the SPC module, which shifts the entire image into four neighboring maps, enabling the elaborate and simultaneous encoding of local information for all pillars.Hybrid Architectures. Apart from the pure-MLP methods [12] , which capture both local and global dependencies fully in MLPbased approaches, there have been developments in combining MLPs with convolutional layers to separately aggregate these two types of information [1, 27, 40] . Among them, sMLPNet [40] introduces a sparse-MLP module to aggregate global information while using the depth-wise convolutional (DWConv) layer to model local features. Concurrent with our work, Strip-MLP [1] chiey replaces the sparse-MLP (i.e., the global-mixing module) in sMLPNet with a Strip-MLP layer, achieving superior scores on both large-and small-scale image datasets. The proposed Caterpillar is also built upon the sMLPNet but replaces the DWConv (i.e., the local-mixing module) with the SPC module, resulting in a pure-MLP architecture, which also attains excellent performance on various-scale image recognition tasks. 
method .1 shifted-pillars-concatenation module: 
 In this section, we rst introduce the SPC module, of which the working procedure can be decomposed into Pillars-Shift and Pillars-Concatenation, as shown in Figure 2 . Then, we analyze its computational parameters with that of the standard and depth-wise convolutional layers.Pillars-Shift. This process is to shift and pad an input image into four neighboring maps, which can be formulated as:EQUATIONwhere X is an image, 38A , B and ? < denote the shifting direction, shifting steps and padding mode, respectively. D B is a set containing shifting directions.Specically, let x 8 9 2 R ⇠ denote a feature vector (referred to as "pillar" and also depicted as pillars in Figure 2 , so as to clearly and accurately express and visualize the workow in the SPC module), we can have an image:X 8= = 2 6 6 6 6 6 6 6 6 6 4 x 11 x 12 • • • x 1, x 21 x 22 • • • x 2, . . . . . . . . . . . . x ( 1)1 x ( 1)2 • • • x ( 1), x 1 x 2 • • • x , 3 7 7 7 7 7 7 7 7 7 5, which means X 8= 2 R ⇥, ⇥⇠ , where , , and ⇠ represent the width, height and channel number, respectively. Taking X D (i.e., the up-wise neighboring map) as an example, we rst perform a shift operation on X 8= by setting 38A ='D? 0 and B = 1, so that X 8= is transformed to:X 0 u = 2 6 6 6 6 6 6 6 6 6 4 x 21 x 22 • • • x 2, . . . . . . . . . . . . x ( 1)1 x ( 1)2 • • • x ( 1), x 1 x 2 • • • x , 3 7 7 7 7 7 7 7 7 7 5, where X 0 u 2 R ( 1) ⇥, ⇥⇠ . Then, we pad X 0 u according to the Zero Padding and attain the X D :EQUATION, where X u 2 R ⇥, ⇥⇠ . By default settings with D B of ['D? 0 , '3>F= 0 , ';4 5 C 0 , 'A86⌘C 0 ], the input X 8= will be transformed into four neighboring maps of X D , X 3 , X ; , X A , where X d , X l , X r 2 R ⇥, ⇥⇠ .Pillars-Concatenation. Obviously, Pillars-Shift has no parameter learning, which would weaken the representation capability of the module. To overcome this deciency, we introduce the Pillars-Concatenation process. Specically, the neighboring maps X D , X 3 , X ; , X A are projected through four independent fully-connected (FC) layers. The parameters are W D , W 3 , W ; , W A 2 R ⇠ ⇥⇠/4 , respectively, so as to reduce the number of neighboring maps' channels into ⇠/4. After that, all of the reduced maps are concatenated along the channel dimension and then projected again, by an FC layer with the parameters W 2 R ⇠ ⇥⇠ to fuse the local features. This process can be represented as:EQUATIONThrough the Pillars-Concatenation, the four neighboring maps are reduced, concatenated and nally fused into the X >DC 2 R ⇥, ⇥⇠ , with the local information for all pillars within the image aggregated in parallel.For an input image with the input dimension of 3 8= and output dimension of 3 >DC , the number of parameters in standard and depthwise convolution (DWConv) can be calculated as 3 8= ⇥ : 2 ⇥ 3 >DC and 3 8= ⇥ : 2 , respectively, with : representing the kernel size. In comparison, The parameters of SPC module are 3 8= ⇥3 8= +3 8= ⇥3 >DC (detailed as 3 8= ⇥3 8= /4⇥4+3 8= ⇥3 >DC ). In the typical scenario, where 2 , demonstrating that the SPC module has lower computational complexity than the standard convolutional layers. Additionally, the depth-wise settings reduce the parameters in DWConv into 9 ⇥ 3 8= , which is lower than SPC and might inspired future works that improve SPC through such lightweight techniques.Caterpillar block is built by replacing the depth-wise convolution with the SPC module in sMLPNet block [40] , in which a sparse-MLP (sMLP) module (illustrated in Appendix A .8) is introduced for aggregating global features. As illustrated in Figure 3 , a Caterpillar block contains three basic modules: an SPC module and an sMLP module, with a BatchNorm (BN) and a GELU nonlinearity applied before them, and a FFN module, which follows a LayerNorm (LN) layer. The SPC and sMLP form the token-mixing component and the FFN servers as channel-mixing module, with applied two residual connections.Given an image X 2 R ⇥, ⇥⇠ , the calculation in the Caterpillar block can be formulated as:EQUATIONEQUATIONEQUATIONwhere X 0 denotes the output features of the SPC layer, Y and Z represent the output of token-mixing and channel-mixing modules, respectively.We build the Caterpillar architectures in a pyramid structure of four stages, which rst represent the input images into patch-level features, and gradually shrink the spatial size of the feature maps as the network deepens. This enables Caterpillar to leverage the scaleinvariant property of images as well as make full use of multi-scale features.We introduce ve variants of the Caterpillar architectures, i.e., Caterpillar-Mi, -Tx, -T, -S and -B, with dierent numbers of Caterpillar blocks stacked in their four stages. The architecture hyperparameters of these variants are:• where ⇠ is the channel number of the hidden layers in the rst stage, and layer numbers denote the number of blocks in each of their four stages. Detailed congurations are in Appendix A .5. 
experiments: 
 We compare the classication performance of Caterpillar with various vision models on small-scale images as well as ImageNet-1k dataset in Sec. 4 .1, 4 .5 and Sec. 4 .2, respectively, with its scalability and transfer capability also veried in Sec. 4 .1 and Sec. 4 .5. Then, we conduct ablation studies in Sec. 4 .3 to nd the best settings for the proposed SPC module and Caterpillar architecture. Additionally, the comparison between sMLPNet and Caterpillar in Sec. 4 .1, 4 .2 as well as the experiments in Sec. 4 .6 is to verify SPC to be an alternative to convolution in both plug-and-play and independent manners, with visualization analysis for SPC provided in Sec. 4 .4.Datasets. We conduct small-scale image recognition experiments on four commonly-used benchmarks: Mini-ImageNet (MIN) [47] , CIFAR-10 (C10) [21] , CIFAR-100 (C100) [21] and Fashion-MNIST (Fashion) [53] . We utilize these images in their original sizes, dierent from the settings in [1] that resized images into 224 ⇥ 224. Experimental Settings. We evaluate the Caterpillar with fourteen representative vision models, including six MLP models [2, 12, 17, 41, 43, 50] , two CNN models [16, 45] , three Transformer models [33, 44, 60] , and three hybrid models [1, 14, 40] , as tagged in Table 1 . All models are directly trained on the small images without extra data. To enable the models adaptable to small-sized images (e.g., 32 ⇥ 32), we change their patch embedding layers into small patch sizes according to uniform rules, of which the detailed implementation is provided in Appendix A .6. For a fair comparison, we adopt the same training strategies that were presented in their original papers (for ImageNet-1K), which are presented in Appendix A .7. Results. Table 1 presents the classication results of dierent methods on the four small-scale image datasets. As we can see, the proposed Caterpillar outperforms the sMLPNet on all the four benchmarks (e.g., Caterpillar-T † , 23M, 77.56% vs sMLPNet-T, 24M, 77.07% on MIN), showcasing the better classication capability of the SPC than convolutional layers and its potential to be an alternative to convolution in plug-and-play ways. Additionally, the Caterpillar attain the best scores among all tested architectures, i.e., the Caterpillar-T reaches 78.16% accuracy on MIN, 97.10% on C10, 84.86% on C100, and 95.72% on Fashion, making it an eective tool for small-scale image recognition tasks. Scalability analysis. "Simple algorithms that scale well are the core of deep learning" [15] . Thus, we scale the Caterpillar from -Mi with FLOPs about 0.4G to -B about 5.5G, i.e., Caterpillar-Mi, -Tx, -T, -S and -B. It is credible that Caterpillar exhibits excellent scalability on small-scale datasets, as it obtains steady improvement from bigger models.Datasets. We test the Caterpillar on the ImageNet-1K benchmark [8] , which consists of 1.28M training and 50K validation images belonging to 1,000 categories. Experimental Settings. We train our models on 8 NVIDIA GeForce RTX 3090 GPUs with gradient accumulation techniques. For training strategies, we employ the AdamW [35] optimizer to train our models for 300 epochs, with a weight decay of 0.05 and a batch size of 1024. The learning rate is initially 1e-3 and gradually drops to 1e-5 according to the consine schedule. The data augmentation includes Random Augment [6] , Mixup [58] , Cutmix [57] , Random Erasing [61] . More details are shown in Appendix A .7.Results. In this section, we ablate essential design components in the proposed Caterpillar architecture. We use the same datasets and experimental settings as in Section 4 .1. The base architecture is Caterpillar-T.Number of shift directions. This hyper-parameter (# ⇡ ) controls the shifting directions of input images so as to determine the receptive eld of SPC on local features. We experiment with # ⇡ values ranging from 4 to 9. Among them, 4 represents the scope of four neighboring directions (up, down, left, and right), and 5 includes the center pillar itself. 8 covers a wider scope, including up, down, left, right, up-left, up-right, down-left, and down-right directions. When # ⇡ is set to 9, it adds the center pillar itself, which is similar to the scope of 3x3 convolution. From Table 3 , increasing the number of neighbors can bring redundancy, because more background information can be injected into the target pillar -the similar drawback existed in convolution. This result underscores that the local features can be suciently obtained from a 4-scoped receptive eld. Number of shift steps. The hyper-parameter B determines the range of local features for the Pillars-Shift operation. When B is set to 0, 1, or 2, the input image is shifted 0, 1, or 2 steps along the corresponding directions, which allows the local information for each pillar to be aggregated from itself (no shifting, lacking local modeling capability), neighboring pillars (with a distance of 1), or distant pillars (with a distance of 2), respectively. Table 4 displays the results of the proposed method with dierent numbers of shifting steps. The ndings indicate that the best performance can be achieved when B = 1. what noises (extra information) would be injected to the pillars on the margin of images. We test four popular padding modes in Table 5 . Among them, Replicated Padding can inject repeated features to the marginal pillars, which might bring redundancy. Both the Circular and Reect modes add long-distance information to those pillars, which is obviously detrimental to the locality bias. Zero Padding is clean and thus achieves the best accuracy. The Pillars-Concatenation process involves three key operations: (1) Reduce, which enables diverse representation by transforming neighboring maps in multiple representation spaces; (2) Concatenation (Concat), which integrates four neighboring maps to combine local features for all pillars in parallel; and (3) Fuse, which selectively learns and weights neighboring features to enhance the representation capabilities. We ablate ve combinations of these operations in Table 6 and observe that Reduce+Concat+Fuse and Concat+Fuse perform better than the other options. Among them, Reduce+Concat+Fuse achieves a better trade-o between computational costs and accuracy. In Section 4 .1 and 4 .2, the sMLPNet, Strip-MLP, and Caterpillar all attained excellent performance on both small-and large-scale image recognition tasks. This success could be attributed to the common strategy that they clearly separate the local-and globalmixing operations in their token-mixing modules. However, both the sMLPNet and Strip-MLP missed the experiments to further explore the eects of local-global combination ways. To ll this gap, we conduct this ablation and perform various strategies as depicted in Figure 4 . We evaluate six strategies, denoted as (a), (b), (c) for sequential regimes, and (e), (f), (g) for parallel methods, by rearranging the SPC and sMLP modules in Caterpillar blocks. Table 7 shows that the simply sequential methods generally outperform the complicated parallel strategies. We attribute this phenomenon to the idea that the 'High-Cohesion and Low-Coupling' principle leads to higher performance, since the internal SPC and sMLP modules are both working in sophisticated parallel ways. Furthermore, the L-G strategy achieves the best performance. To understand how the SPC module processes image data, we visualize the feature maps encoded by SPC coupled with two control ways. Specically, we build three Caterpillar-T models with the local modules of identity, convolution and SPC, and implement them on the CIFAR-100 dataset. Figure 5 (in Appendix A .1) illustrates the feature maps of six samples, each of which is presented with 3 rows and 4 columns, where rows represent dierent local-mixing ways and columns are feature maps of dierent phases in models. For these samples, with the (a) cattle as an example, the patterns in SPC features are closer to the convolution and dierent from the identity. Since convolutional layers capably capture local features, the SPC is also capable of aggregating local information. Furthermore, compared to convolution, the objects in SPC maps exhibit more prominent edge features and are closer to the original input image, indicating that the proposed SPC module can encode local information more elaborately and avoid redundancy issues.In this subsection, we compare the Transfer Learning capability of the proposed Caterpillar architecture with recent SOTA models. Following the recent MLP-based models [17, 50] , we pre-train the Caterpillar-T on the ImageNet-1k and then ne-tune it on CIFAR-10 and CIFAR-100. From Table 8 , the Caterpillar attains higher scores than other representative networks with similar computational costs, indicating that Caterpillar can work well on Transfer tasks.Despite the remarkable Transfer capability, we highlight that Caterpillar can achieve exceptional performance on small-scale images using the 'Direct Training' (Direct) strategy (Section 4 .1). To further illustrate Caterpillar's eectiveness in data-hungry domains without relying on pre-training data, which always faces challenges to domain-shift and task-compatibility, we conduct this study. We adopt the same datasets as in Section 4 .1, i.e., MIN, C10, C100 and Fashion, while including two more datasets in certain scientic elds of remote sensing, i.e., Resisc45 (R45) [5] with 27,000 training and 4,500 testing images in 45 categories, and disease diagnosis [20] i.e., Chest_Xray (Chest) with 5,216 training images and 624 testing images belonging to 2 classes. All images are resized to 224 ⇥ 224. Then, we utilize two Caterpillar-T models as the base architectures. The model of 'Transfer' is pre-trained on the ImageNet-1K and then ne-tune on the target datasets, while the other one for 'Direct' is trained from scratch. From Table 9 , the 'Transfer' strategy performs better on MIN, C10, and C100, while the 'Direct' strategy achieves higher scores on Fashion, R45, and Chest, which have dissimilar distributions to the pre-trained data (ImageNet-1K). Therefore, for the data-hungry scientic tasks (especially those without the same distribution to the pre-trained natural images), directly training the deep model can be a more cost-eective approach than the 'Transfer' strategy, with Caterpillar serving as the backbone architecture. Previous comparison between Caterpillars and sMLPNets demonstrates the potential of SPC as an alternative to convolution in plugand-play ways (Table 1 , 2). We further explore the SPC to serve as the main module for neural architectures. Datasets. We utilize the same large-scale benchmark of ImageNet-1K as well as the small-scale datasets of MIN, C10, C100 and Fashion, as those in Section 4 .1 and 4 .2. Experimental Settings. We adopt classic ResNet-18 (Res-18) [16] as the baseline CNN. Then, we replace all convolutional layers within Res-18's basic blocks with the SPC module and obtain three SPC-based variants referred to as 'Res-18(SPC)', with # ⇠ utilized as channel numbers to adjust model complexity. For training these models, we follow the 'Procedure A2' in [51] . Results. Table 10 displays the ImageNet-1K classication results for the original Res-18 and Res-18(SPC) variants. As we can see, the proposed SPC module can provide higher performance than Increasing # ⇠ to 128, the Res-18(SPC) reaches similar computational costs to the baseline Res-18 while achieving 4.7% higher accuracy. Similar trends can be observed on small-scale recognition tasks, as shown in Table 11 . This conrms that the SPC module can also be used as the main component to construct neural networks, potentially serving as an alternative to convolutional layers in independent manners. 
conclusion: 
 This paper proposes the SPC module that conducts the Pillars-Shift and Pillars-Concatenation to achieve an elaborate and parallelizable aggregation of local information, with superior classication performance than convolutional layers. Based on SPC, we introduce Caterpillar, a pure-MLP network that attains impressive scores on both small-and large-scale image recognition tasks.The philosophy of "simple and eective" and the principle of "control variable" have run through this work. Therefore, Caterpillar only replaces the DWConv with SPC module in sMLPNet and thus has more parameters. We anticipate that integrating the SPC module with lightweight techniques, like depth-wise, will further reduce computational costs. Additionally, the experiments are primarily conducted on the most fundamental classication tasks, since SPC and Caterpillar are introduced for the rst time. We hope the SPC and Caterpillar can be explored in broader tasks like detection and segmentation, particularly in data-hungry domains. 
 
 Appendix:  


Decision: This paper proposes Caterpillar, a pure-MLP architecture with a Shifted-Pillars-Concatenation (SPC) module aimed at enhancing local feature modeling in computer vision tasks. 

Initial concerns mainly lie on the novelty of the shift operation, the lack of comprehensive comparison with state-of-the-art methods, insufficient discussion on parameter efficiency and computational complexity, and the limited scope of experiments focusing primarily on image classification tasks.

The authors provided a thorough response addressing the concerns raised by the reviewers, leading to a final scores of two Weak Reject and two Weak Accept, with an average score of 3.5. 

The AC carefully reviewed the paper and all the reviewers' comments.
Considering the good performance of the proposed architecture on benchmark datasets and its potential impact for the community, the AC agrees to accept the paper and strongly recommends incorporating the content from the rebuttal into the final version.

Reviews: review: paper_summary: paper_summary: The paper introduces the Caterpillar architecture, a pure-MLSLP structure enhanced by the Shifted-Pillars-Concatenation (SPC) module, designed to improve local feature integration without reliance on traditional convolutional methods. This model leverages shifts and linear transformations across image pillars to efficiently capture and concatenate local spatial information, demonstrating robust performance on various image datasets.
strength_weakness: strength_weakness: 1. Caterpillar achieves impressive results on benchmarks like ImageNet-1k, showcasing its potential against established MLP, CNN, and Transformer models.
2. The architecture is designed to be computationally efficient, reducing the parameter count and computational overhead compared to conventional methods.
3. The paper is well-written and easy to follow.
limitations: limitations: 1.The shift operation is not novel in the vision community, such as [1] and [2]. More detailed comparison and discussion with these methods are needed. The novelty also needs further clarification.

[1]. A Simple Baseline for Video Restoration with Grouped Spatial-temporal Shift
[2]. X-volution: On the Unification of Convolution and Self-Attention

2.Comparison with SOTA methods is missing.
[3] A ConvNet for the 2020s
[4] MLP-Mixer: An All-MLP Architecture for Vision
[5] InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
[6] HorNet: Efficient High-Order Spatial Interactions with Recursive Gated Convolutions.
[7] CYCLEMLP: A MLP-like Architecture for Dense Prediction

3.As a foundation model, experiments on detection and segmentation are missing, which makes the validity of the method less credible.

4. The effective receptive field of the method needs to be analyzed. Also, the feature map visualization is insufficient.

5. Why the shift operation works needs to be further explained.
paper_summary: paper_summary: This paper propose a MLP module which exploit local modeling capability. The main contributions of the paper include proposing a new SPC module and constructing a new pure-MLP model Caterpillar, which is able to perform superior or better on image datasets without relying on Transfer Learning.
strength_weakness: strength_weakness: -This paper introduces local modeling capabilities for MLPs. The SPC module proposed in this paper adopts a windowless scheme and can exploit locality in a more refined and parallel way than convolution.

-The writing of this paper is well, and the organization of the paper is good.
limitations: limitations: -The paper does not discuss in depth enough the comparison of the parameter efficiency and computational complexity of the SPC module and the traditional convolutional.

-The improvement in model performance is not significant, and many additional parameters are introduced. MLPs have no advantage in terms of computational complexity.

-The authors could provide theoretical analysis for the proposed method and comparison algorithms.

-The empirical comparison can be done in a more thoroughly by comparing with other latest state-of-the-art algorithms.
paper_summary: paper_summary: This paper introduces a new MLP-based architecture for computer vision tasks that aims to replace traditional convolutional methods with a pure multi-layer perceptron (MLP) approach. The core innovation, the Shifted-Pillars-Concatenation (SPC) module, enhances the local modeling capability by shifting the input image in four directions and concatenating the results after applying linear transformations. This architecture is tested on a variety of datasets, including ImageNet-1k, showing competitive or superior performance compared to existing models.
strength_weakness: strength_weakness: -   Introducing a pure-MLP architecture that does away with convolutions and instead uses pillar shifting and concatenation is a significant deviation from traditional models, offering a fresh perspective in the field. 

-   The proposed module has the ability to capture local dependencies without convolution providing a promising alternative for reducing computational redundancy. 

-   Experiments show the effectiveness of the proposed method and the performance is promising.
limitations: limitations: The authors propose to use Pillars-Shift to achieve the local receptive fields, which is a good idea. Some other papers such as AS-MLP [25] also utilize a similar operation to shift features along two directions (horizontal and vertical) to cover the local receptive fields. The author could highlight the main differences between the proposed method and AS-MLP. In addition, as we know, although the parameters and flops are the metrics to measure the complexity of a model, the inference time (or throughput) is a more direct way. The authors could also list the throughput of different models in Table 1 or Table 2 to make the experimental results more complete. As pointed out in the conclusion, some experiments on detection or segmentation tasks are necessary because the classification task only needs a small receptive field to recognize an image but the detection or segmentation tasks need a larger receptive field dependency due to the property of the tasks and larger resolution images.

This work proposes the SPC module that conducts the Pillars-Shift and Pillars-Concatenation to achieve an elaborate and parallelizable aggregation of local information. Experimental results on the classification datasets demonstrate the effectiveness and efficiency of the proposed approach. My questions are listed above. I tend to vote weak accept.
paper_summary: paper_summary: This paper proposes a novel MLP-based module called Shifted-Pillars-Concatenation (SPC) for local modeling in computer vision. The SPC module consists of two key processes: 1) Pillars-Shift, which generates four neighboring feature maps by shifting the input image along four directions, and 2) Pillars-Concatenation, which applies linear transformations and concatenation on the maps to aggregate local features. Building upon the SPC module, the authors construct a pure-MLP architecture named Caterpillar by replacing the convolutional layers. Experiments demonstrate that Caterpillar achieves excellent performance on both small-scale datasets without extra data and the large-scale ImageNet-1k benchmark. The SPC module also exhibits strong local modeling capability and brings significant performance gains when replacing convolutional layers.
strength_weakness: strength_weakness: **(S1) The Research Question:** The paper addresses an important problem in vision MLPs: exploiting local information in a more elaborate and parallelizable way. The SPC module captures local features from neighboring pillars in parallel, preventing the limitations of convolution, such as redundancy and sequential computation. This is a key contribution that addresses limitations of existing convolution models and MLPs.

**(S2) The Experiments:** Extensive experiments on both small-scale datasets and the large-scale ImageNet-1k benchmark comprehensively validate the effectiveness of Caterpillar and the SPC module. The Caterpillar architecture, built upon the SPC module, consistently achieves superior or competitive performance compared to several existing MLP, ConvNets, and Transformer models. Moreover, the significant performance gains obtained by replacing convolution with SPC in existing architectures further demonstrate the plug-and-play potential of SPC module. The paper also conducts detailed ablation studies to justify the architectural design choices in SPC and Caterpillar, further evaluating the impact of key hyper-parameters such as shift directions, shift steps, padding modes, and local-global combination strategies. These ablations provide valuable insights into the working mechanism and the robustness of this work.

**(S3) The Presentation Clarity:** The paper is overall well-structured and follows a clear logical flow. Necessary background information on vision MLPs is provided, setting the context for their contributions. The methodology section explains the technical details of the SPC module and Caterpillar architecture in a coherent manner, accompanied by illustrative figures and algorithmic descriptions. The extensive experimental results are thoroughly reported and analyzed, with informative tables and plots to support the findings. Visualizations of feature maps learned by SPC and convolution are also presented, facilitating a deeper understanding of their differences in local feature learning. Overall, the clear writing style, detailed methodology, and comprehensive experiments enhance the readability and reproducibility of the paper.
limitations: limitations: ### **Limitations:**

**(L1) Efficiency Concerns:** Although the SPC module demonstrates competitive performance, it introduces additional parameters and computational costs. The authors acknowledge the importance of improving the efficiency of SPC for practical usage and discuss potential solutions, such as integrating depth-wise convolution techniques. However, the paper lacks concrete implementations or evaluations of these efficiency optimizations. Providing empirical results or theoretical analysis on the computational complexity, such as total training & inference time, latency, throughput, and memory footprint of SPC would strengthen the practical value of this work. Exploring and demonstrating effective strategies to reduce the overhead of SPC while maintaining its performance gains is crucial for its wider adoption in resource-constrained scenarios.

**(L2) Insufficient Discussion and Comparisons:** While the paper compares Caterpillar with several existing models, it misses comparisons with some recently published state-of-the-art convolutional models. For example, on the ImageNet-1K benchmark, including models like VAN [1], SLaK [2], and MogaNet [3] would provide a more comprehensive evaluation of Caterpillar's performance against modern ConvNets. These models have demonstrated strong results and introduce novel architectural designs or training strategies. Thus, comparing Caterpillar with these models and discussing their similarities and differences would help position the proposed approach in the broader context of vision architecture research. Additionally, the paper would benefit from an expanded discussion section that analyzes the recent developments and trends in ConvNets, highlighting the advantages and limitations of the proposed MLP-based approach.

**(L3) Limited Experiment Scope:** The experimental evaluation in the paper primarily focuses on image classification tasks. While classification is a fundamental problem in computer vision, the versatility and potential of Caterpillar and SPC could be further demonstrated by applying them to a broader range of vision tasks, such as object detection and semantic segmentation. The authors mention this as a future direction, but including initial experiments or a more detailed discussion on the potential of Caterpillar and SPC in these domains would enhance the impact and significance of the work.

### Reference

[1] Visual Attention Network, CVMJ 2023

[2] More ConvNets in the 2020s: Scaling up Kernels Beyond 51x51 using Sparsity, ICLR 2023

[3] MogaNet: Multi-order Gated Aggregation Network, ICLR 2024

---

### **Justification of Rating:**

After carefully reading the manuscript, I first recommend a Borderline Accept for this paper.

**(1) Research Questions** 

This paper tackles the important problem of local feature learning in vision MLPs. It identifies the limitations of convolution in terms of redundancy and sequential computation, and proposes the SPC module as a more effective and parallelizable alternative. The research questions are well-motivated and clearly stated.

**(2) Challenge Analysis** 

The authors provide a thorough analysis of the challenges in local modeling with vision MLPs. They discuss the drawbacks of existing solutions such as traditional convolution and previous MLP-based approaches. This analysis justifies the necessity of the proposed SPC module and Caterpillar architecture.

**(3) Philosophy** 

The key philosophy behind this work is to exploit locality in vision MLPs in a more elaborate and efficient way. The SPC module achieves this by encoding local features from neighboring pillars in parallel, avoiding the sliding window scheme. This philosophy aligns well with the goal of designing effective vision MLPs.

**(4) Technical Soundness and Originality** 

The proposed SPC module and Caterpillar architecture are technically sound and original. SPC introduces a novel way to aggregate local features through Pillars-Shift and Pillars-Concatenation operations, demonstrating superior performance over convolution. Caterpillar leverages SPC to build a pure-MLP model that separately captures local and global information, leading to excellent results. 

**(5) Evaluation** 

The experiments are extensive. The authors evaluate Caterpillar on both small-scale datasets and ImageNet-1k, comparing it with several MLP, CNN, and Transformer models. However, it is better to discuss and empirically compare the missing recent convolutional methods as I stated above. Ablation studies are conducted to justify the design choices. The consistent superior performance of Caterpillar and the significant gains brought by replacing convolution with SPC  support the effectiveness of the proposed methods.

**(6) Presentation Clarity** 

The paper is well-structured and clearly written. The motivation, methodology, and experiments are presented in a logical flow. Necessary details are provided for reproducibility. The illustrations and visualizations help understand the technical contributions.

Considering the above factors, I believe the paper in its current form meets the bar for a Borderline Accept at ACM MM 2024.

---

### **Additional Comments:**

I hope my review helps to further strengthen this paper and helps the authors, fellow reviewers, and Area Chairs understand the basis for my recommendation. I look forward to the rebuttal feedback and would be glad to raise my rating if thoughtful responses and improvements are presented.