Container: Context Aggregation Networks
Authors: Gao Peng
Jiasen Lu
Hongsheng Li
Roozbeh Mottaghi
Aniruddha Kembhavi
Venue: NeurIPS 2021 Poster (2021)
Keywords: Convolution
MLP-Mixer
Transformer

Abstract: Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations. Recently, Transformers -- originally introduced in natural language processing -- have been increasingly adopted in computer vision. While early adopters continued to employ CNN backbones, the latest networks are end-to-end CNN-free Transformer solutions. A recent surprising finding now shows that a simple MLP based solution without any traditional convolutional or Transformer components can produce effective visual representations. While CNNs, Transformers and MLP-Mixers may be considered as completely disparate architectures, we provide a unified view showing that they are in fact special cases of a more general method to aggregate spatial context in a neural network stack. We present the \model (CONText AggregatIon NEtwoRk), a general-purpose building block for multi-head context aggregation that can exploit long-range interactions \emph{a la} Transformers while still exploiting the inductive bias of the local convolution operation leading to faster convergence speeds, often seen in CNNs. Our \model architecture achieves 82.7 \% Top-1 accuracy on ImageNet using 22M parameters, +2.8 improvement compared with DeiT-Small, and can converge to 79.9 \% Top-1 accuracy in just 200 epochs. In contrast to Transformer-based methods that do not scale well to downstream tasks that rely on larger input image resolutions, our efficient network, named \modellight, can be employed in object detection and instance segmentation networks such as DETR, RetinaNet and Mask-RCNN to obtain an impressive detection mAP of 38.9, 43.8, 45.1 and mask mAP of 41.3, providing large improvements of 6.6, 7.3, 6.9 and 6.6 pts respectively, compared to a ResNet-50 backbone with a comparable compute and parameter size. Our method also achieves promising results on self-supervised learning compared to DeiT on the DINO framework. Code is released at https://github.com/allenai/container. 

Container: Context Aggregation Network
Peng Gao 1:2, Jiasen Lu", Hongsheng Li?, Roozbeh Mottaghi Aniruddha Kembhayi Joint Lab, CUHK
Shanghai AI Laboratory 2 3 University of Washington PRIOR Allen Institute for Al
Abstract
Convolutional neural networks (CNNs) are ubiquitous in computer vision, with a myriad of effective and efficient variations. Recently, Transformers originally introduced in natural language processing have been increasingly adopted in computer vision. While early adopters continue to employ CNN backbones, the lat- est networks are end-to-end CNN-free Transformer solutions. A recent surprising finding shows that a simple MLP based solution without any traditional convo- lutional or Transformer components can produce effective visual representations. While CNNs, Transformers and MLP-Mixers may be considered as completely disparate architectures,
provide unified view showing that they in fact special cases more general method to aggregate spatial context in a neural network stack. We present the CONTAINER (CONText AggregatIon NEtwoRk), general-purpose building block for multi-head context aggregation that can exploit long-range interactions la Transformers while still exploiting the inductive bias the local convolution operation leading to faster convergence speeds, often seen in CNNs. Our CONTAINER architecture achieves 82.7 % Top-1 accuracy on ImageNet using 22M parameters, +2.8 improvement compared with DeiT-Small, and can converge 79.9 % Top-1 accuracy in just 200 epochs. In contrast Transformer-basee methods that do not scale well to downstream tasks that rely on larger input image resolutions, our efficient network, named CONTAINER LIGHT, LIGHT, can be employed in object detection and instance segmentation networks such as DETR, RetinaNet and Mask-RCNV
obtain impressive detection mAP of 38.9, 43.8, 45.1 and mask mAP of 41.3, providing large improvements of 6.6, 7.3, 6.9 and 6.6 pts respectively, compared ResNet-50 backbone with a comparable compute and parameter size. Our method also achieves promising results on self- supervised learning compared to DeiT on the DINO framework. Code released at httts::/// //gi thub. com/ hth//aleen allenai/ contiiierottiier..
1 Introduction
Convolutional neural networks (CNNs) have become the de facto standard for extracting visual representations, and have proven remarkably effective at numerous downstream tasks such as object detection [37], instance segmentation [22] and image captioning
[1]. Similarly, in natural language processing, Transformers rule the roost [13, 43, 42, 4]. Their effectiveness at capturing short and long range information have led to state-of-the-an results across tasks such as question answering [45] and language understanding [58].
In computer vision. Transformers were initially employed as long range information aggregators across space (c.g., in object detection (5]) and time (c.g., in video understanding [61]), but these methods continued use CNNs [34] to obtain raw visual representations. More recently however, CNN-free visual backbones employing Transformer modules [54, 14] have shown impressive per- formance on image classification benchmarks such as ImageNet [33]. The race to dethrone CNNs has now begun to expand beyond Transformers recent unexpected result shows that multi-layer perceptron (MLP) exclusive network [52] can be just as effective at image classification.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

On the surface, CNNs [34, 8, 63, 23], Vision Transformers (ViTs) [14, 54] and MLP-mixers [52] are typically presented as disparate architectures. However, taking a step back and analyzing these methods reveals that their core designs are quite similar. Many of these methods adopt cascade of neural network blocks. Each block typically consists of aggregation modules and fusion modules. Aggregation modules share and accumulate information across predefined context window over the module inputs (e.g., the self attention operation in a Transformer encoder), while fusion modules combine position-wis features and produce module outputs (e.g., feed forward layers in ResNet).
In this paper, we show that the primary differences in many popular architectures result from variations in their aggregation modules. These differences can in fact be characterized as variants of an affinity matrix within the aggregator that used to determine information propagation between a query vector and its context. For instance, in ViTs [14, 54], this affinity matrix is dynamically generated using key and query computations; but the Xception architecture [8] (that employs depthwise convolutions), the affinity matrix is static the affinity weights are the same regardless of position, and they remain the same across all input images regardless of size. And finally the MLP-Mixer [52] also uses static affinity matrix which changes across the landscape of the input.
Along this unified view, we present CONTAINER (CONText Aggregation NEtwoRk), a general purpose building block for multi-head context aggregation. A CONTAINER block contains both static affinity as well as dynamic affinity based aggregation, which are combined using learnable mixing coefficients. This enables the CONTAINER block to process long range information while still exploiting the inductive bias of the local convolution operation. CONTAINER blocks are easy to implement, can easily he substituted into many present day neural architectures and lead to highly performant networks whilst also converging faster and being data efficient.
Our proposed CONTAINER architecture obtains 82.7 % Top-1 accuracy on ImageNet using 22M parameters, improving +2.8 points over DeiT-S [54] with a comparable number of parameters. also converges faster, hitting DeîT-S's accuracy of 79.9 % in just 200 epochs compared to 300.
We also propose more efficient model, named HONTAINER-LIGHT that employs only static affinity matrices early on but uses the learnable mixture of static and dynamic affinity matrices in the latter stages of computation. In contrast to VITs that are inefficient at processing large inputs, CONTAINER-LIGHH can scale to downstream tasks such as detection and instance segmentation that require high resolution input images. Using CONTAINER-LIGHT backbone and 12 epochs of training, RetinaNet [37] is able to achieve 43.8 mAP, while Mask-RCNN [22] is able to achieve 45.1 mAP on box and 41.3 mAP +6.6 respectively, compared to a ResNet-50 backbone. The more recent DETR and its variants SMCA-DETR and Deformable DETR [5, 19, 75] also benefit from CONTAINER-LIGIT and achieve 38.9, 43.0 and 44.2 mAP, improving significantly over their ResNet-50 backbone baselines.
instance mask prediction, improvements of +7.3, +6.9 and
CONTAINER-LIGHT is data efficient. Our experiments show that it can obtain an ImageNet Top-1 1 accuracy of 61.8 using just 10% of training data, significantly better than the 39.3 accuracy obtained by DeiT. CONTAINER- compared to DeiT (69.6) under DINO self-supervised training framework [6].
also convergences faster and achieves better kNN accuracy (71.5)
The CONTAINER unification and framework enable us to easily reproduce several past models and even extend them with just few code and parameter changes. We extend multiple past models and show improved performance - for instance, we produce Hierarchical DeiT model, multi-head MLP-Mixer and add a static affinity matrix to the DeiT architecture. Our code base and models will be released publicly. Finally, we analyse CONTAINER model containing both static and dynamic affinities and show the emergence of convolution like local affinities in the early layers of the network.
In summary, our contributions include: (1) A unified view of popular architectures for visual inputs CNN, Transformer and MIP-mixer, (2) A novel network block - CONTAINER, which uses mix of static and dynamic affinity matrices via learnable parameters and the corresponding architecture with strong results in image classification and (3) An efficient and effective extension with strong results in detection and segmentation. Importantly, we see that number of concurrent works are aiming to fuse the CNN and Transformer architectures [36, 64, 40, 24, 55, 69, 64, 47], validating our approach. We hope that our unified view helps place these different concurrent proposals in context and leads to better understanding of the landscape of these methods.
CONTAINER-
2

---

2 Related Work
Visual Backbones. Since AlexNet [33] revolutionized computer vision, host of CNN based archi- tectures have provided further improvements in terms of accuracy including VGG [46], ResNet [23], Inception Net [48], SENet [28], ResNeXt [63] and Xception [8] and efficiency including Mobile-net v1 [26], Mobile-net v2 [26] and Efficient-net v2 [50]. With the success of Transformers [56] in NLP such as BERT [13] and GPT [43], researchers have begun apply them towards solving the long range information aggregation problem in computer vision. VIT [14]/Dei' [54] are transformers that achieve better performance on ImageNet than CNN counterparts. Recently, several concurrent works explore integrating convolutions with transformers and achieve promising results. Con [11] explores soft convolutional inductive bias for enhancing DeiT. CeiT [66] directly incorporates CNNs into the Feedforward module of transformers to enhance the learned features. PVT [60] proposes a pyramid vision transformer for efficient transfer to downstream tasks. Pure Transformer models such as ViT/DeiT however, require huge GPU memory and computation for detection [60] and segmentation [73] tasks, which need high resolution input. MLP-Mixer [52] shows that simply performing transposed MLP followed by MLP can achieve near state-of-the-an performance. We pro- pose CONTAINER, a new visual backbone that provides a unified view of these different architectures and performs well across several vision tasks including ones that require high resolution input.
Transformer Variants. Vanilla Transformers are unable to scale to long sequences or high-resolution images due to the quadratic computation in aelf-attention. Several methods have been proposed to make Transformer computations more efficient for high resolution input. Reformer [32], Clus- terform [57], Adaptive Clustering Transformer [73] and Asymmetric Clustering [10] propose to use Locality Sensitivity Hashing to cluster keys or queries and reduce quadratic computation into
linear computation. Lightweight convolution [62] explore convolution architectures for replacing Transformers but only explore applications in NLP. RNN Transformer [31] builds a connection between RNN and Transformer and results in attention with linear computation. Linformer [59] changes the multiplication order of key.query, value into snerr,,alue,key by deleting the softmax normalization layer and achieve linear complexity. Performer [9] uses Orthogonal Random Feature to approximate full rank softmax attention. MLIN [18] performs interaction between latent encoded nodes, and its complexity is linear with respect to input length. Bigbird [3] breaks the full rank atten- tion into local, randomly selected and global attention. Thus the computation complexity becomes linear. Longformer [68] uses local Transformers to tackle the problem of massive GPU memory requirements for long sequences. MLP-Mixer [52] is pure MIP architecture for image recognition. In the unified formulation we provide. MLP-Mixer can be considered as a single-head Transformer with static affinity matrix weight. MLP-Mixer can provide more efficient computation than vanilla transformer due no need to calculate affinity matrix using key query multiplication. Efficient Trans- formers mostly use approximate message passing which results in performance deterioration across tasks. Lightweight Convolution [62], Involution [35]. Synthesizer [51], and MUSE [71] explored the relationship between Depthwise Convolution and Transformer. Our CONTAINER unification performs global and local information exchange simultaneously using a mixture affinity matrix, while CONTAINER-LIGHT switches off the dynamic affinity matrix for high resolution feature maps to reduce computation. Although switching off the dynamic affinity matrix slightly hinders classification performance, CONTAINER-L.Ioe
still provides effective and efficient generalization to downstream tasks compared with popular backbones such as ViT and ResNet.
Transformers for Vision. Transformers enable high degrees of parallelism and are able to capture long-range dependencies in the input. Thus Transformers have gradually surpassed other architectures such as CNN [34] and RNN [25] on image [14, 5, 70], audio [2], molti-modality [17, 21, 20], and language understanding [13]. In computer vision, Non-local Neural Network [61] has been proposed to capture long range interactions to compensate for the local information captured by CNNs and used for object detection [27] and semantic segmentation [16, 29, 76, 67]. However, these methods use Transformers as a refinement module instead of treating the transformer as first-class citizen. ViT [14] introduces the first pure Transformer model into computer vision and surpasses CNNs with large scale pretraining on ImageNet-Il Ik and achieve better performance than CNN counterparts. DETR [5] uses Transformer as encoder and decoder architecture for designing the first end-to-end object detection system. Taming Transformer [15] use Vector Quantization [41] GAN and GPT [43] for high quality high- resolution image generation. Motivated by the success of DETR object detection, Transformers have been applied widely on tasks such as semantic segmentation [74], pose estimation [65], trajectory
the non publicly available JFT dataset. DeiT [54] trains ViT from scratch
3

---

estimation [39], 3D representation learning and self-supervised learning with MOCO v3 [7] and DINO [6]. ProTo [72] verify the effective of transformer on reasoning tasks.
3 Methods
In this section we first provide generalized view of neighborhood//cneex aggregation modules commonly employed in present neural networks. Then we revisit three major architectures - Trans- former [56], Depthwise Convolution [8] and the recently proposed MLP-Mixer [52], and show that they are special cases of our generalized view. We then present our CONTAINER module in Sec 3.3 and its efficient version - CONTAINEER-LIGH in Sec 3.5.
3.1 Contextual Aggregation for Vision
Consider an input image X E RC*H where C' and H x W denote the channel and spatial dimensions of the input image, respectively. The input image first flattened to a sequence of tokens {X; E IR I = 1, 1.....N) N), where N HW and input to the network. Vision networks typically stack multiple building blocks with residual connections [23], defined as
= F(X, (W:)) +X.
Here, X and are the input and output vectors the layers considered, and W, represents the learnable parameters. F determines how information across X is aggregated to compute the feature at a specific location. We first define an affinity matrix A E NNxN X that represents the neighborhood for contextual aggregation. Equation can be re-written
Y = (AV) X,
(2)
where V E R is transformation of X obtained by linear projection V = XW2. W1 and W2 are the learnable parameters. At is the affinity value between X, and Xj. Multiplying the affinity matrix with V propagates information across features in accordance with the affinity values.
The modeling capacity of such context aggregation module can be increased by introducing multiple affinity matrices, allowing the network to have several pathways contextual information across X. Let (V' E Nx & j = 1, ..... M } be slices of V. where M is the number of affinity matrices, also referred to as the number of heads. The multi-head version of Equation is
Y = Concat(A1 11, AM V W₂ + X,
(3)
where A.. denotes the affinity matrix in each head. Different Am can potentially capture different relationships within the feature space and thus increase the representation power of contextual aggregation compared with single-head version. Note that only spatial information is propagated during contextual aggregation using the affinity matrices; cross-channel information exchange does not occur within the aftinity matrix multiplication, and that there is no non-linear activation function.
3.2 The Transformer, Depthwise Convolution and MLP-Mixer
Transformer [56], depthwise convolution [30] and the recently proposed MLP-Mixer [52] are three distinct building blocks used in computer vision. Here, we show that they can be represented within the above context aggregation framework, by defining different types of affinity matrices.
Transformer. In the self-attention mechanism Transformers, the affinity matrix modelled by the similarity between the projected query-key pairs. With M heads, the affinity matrix in head m, A can be written as
A Softmaxx (Qmimaxxxmmmmm KT m/ /C/M).
where K,, Qm the corresponding key, query in head m. respectively. The affinity matrix in self-attention is dynamically generated and can capture instance level information. However, this introduces quadratic computational, which requires heavy computation for high resolution feature.
Depthwise Convolution. The convolution operator fuses both spatial and channel information in parallel. This different from the contextual aggregation block defined above. However, depthwise convolution [30] which is an extreme case of group convolution performs disentangled convolution.

---

Considering the number of the heads from the contextual aggregation block to be equal to the channel size C, we can define the convolutional affinity matrix given the 1-d kemel Ker RXxIx;;
Ker(m, O, 1 ill
< k
come
(5)
mij
where Amiy is the affinity value between X; and X, on head In contrast with the affinity matrix obtained from self-attention whose value conditioned on the input feature, the affinity values for convolution are static they do not depend on the input features, sparse only involves local connections and shared across the affinity matrix.
The recently proposed MLP-Mixer [52] does not rely on any convolution or self- attention operator. The core of MI.-Mixer is the transposed MLP operation, which can be denoted as X X (VTWMLP) We can define the affinity matrix as
MLP-Mixer
Amip (W (WMLP)"
where W MLP represents the learnable parameters. This simple equation shows that the transposed- MLP operator is contextual aggregation operator a single feature group with a dense affinity matrix. Comparing with self-attention and depthwise convolution, the transpose-MLL MasspsseMMLPPP affinity matrix is static, dense and with no parameter sharing.
The above simple unification reveals the similarities and differences between Transformer, depthwise convolution and MLP-Mixed Each of these building blocks can be obtained by different formulating different affinity matrices. This finding leads create powerful and efficient building block for vision tasks - the CONTAINER.
3.3 The CONTAINER Block
As detailed in Sec 3.2, previous architectures have employed either static or dynamically generated affinity matrices each of which provides its unique set of advantages and features. Our proposed building block named CONTAINER, combines both types of affinity matrices via a learnable parameter. The single head CONTAINER is defined as:
Dynamic
Static
Y (=a A(X)
+B A VWW, +X
A(X) is dynamically generated from X while A is static affinity matrix. We now present a few special cases of the CONTAINER block. the following, L denotes a learnable parameter.
a -==1, 1, A(x) = = A vanilla Transformer block with self-attention (denoted sa).
0. 3 1 M = C.A- Acono: A depthwise convolution block. In depthwise convolution, each channel has a different static affinity matrix. When M * C. the resultant block can be considered Multi-head Depthwise Convolution block (MH-DW). MHHDD shares kernel weights.
0, 3 1, M A Amlp; An MLP-Mixer block. When M 1, we name the module Multi-head MLP (MH-MLP). MH-MLP splits channels into M groups and performs independent transposed MLP capture diverse static token relationships.
a L. 3 L, A(x) ATh, A = mipp This CONTAINER block fuses dynamic and static information, but the static affinity resembles the MI.P-Mxer matrix. We call this block CONTAINER- -PAM (Pay Attention to ML.P).
o o = L, A(x)= = A A = A This CONTAINER block fuses dynamic and static information, but the static affinity resembles the depthwise convolution matrix. This static affinity matrix contains a locality constraint which shift invariant, making it more suitable for vision tasks. This is the default configuration used in our experiments.
The CONTAINER block easy to implement and can readily swapped into an existing neural network. The above versions of CONTAINER provide variations on the resulting architecture and its performance and exhibit different advantages and limitations. The computation cost of CONTAINER block is the same as vanilla Transformer since the static and dynamic matrices are linearly combined.
5

---

3,4 The CONTAINER network architecture
We now present a base architecture used in our experiments. The unification of past works explained above allows us to easily compare self-attention, depthwise convolution, MLP and multiple variations of the CONTAINER block, and we perform these comparison using a consistent base architecture.
Motivated by networks in past works [23, 60], our base architecture contains 4 stages. In contrast to ViT/DeiT which down-sample the image to a low resolution and keep this resolution constant, each stage in our architecture down-samples the image resolution gradually. Gradually down-sampling can retain image details, which is important for downstream tasks such as segmentation and detection. Each of the 4 stages contains a cascade of blocks. Each block contains two sub-modue aob-modules, the first to aggregate spatial information (named spatial aggregation module) and the second to fuse channel information (named feed-forward module). In this paper, the channel fusion module is fixed to 2 2-1ayer MLP as proposed in [56]. Designing better spatial aggregation module is the main focus of this paper. The 4 stages contain 2, 3, 8 and blocks respectively. Each stage uses patch embeddings which fuse spatial patches of size x p into single vector. For the 4 stages, the values of are 4,4,2,2 respectively. The feature dimension within stage remains constant - and is set to 128, 256, 320, and 512 for the four stages. This base architecture augmented with the CONTAINER block results in a similar parameter size as DeiT-S [54].
3.5 The CONTAINER-LIGHT network
We also present efficient version known
SENTAINER-LIGHT which uses the same basic architecture as CONTAINER, but switches off the dynamic affinity matrix in the first 3 stages. The absence of the computation heavy dynamic attention at the early stages of computation help efficiently scale the model to process large image resolutions and achieve superior performance on downstream tasks such detection and instance segmentation.
CONNTINNER-LIGH
cone
Stage 1,
m
aAT sa + 3.ACORO
Stage =
(8)
a and 3 are learnable parameters. In network stage 1, 3, CONTAINER LIGHT will switch off A
4 Experiments
We now present experiments with CONTAINER for ImageNet and with CONTAINER- for the tasks of object detection, instance segmentation and supe-supe learning. We also present appropriate baselines. Please see the appendix for details of the models, training and setup.
4.1 ImageNet Classification
Top-1 Accuracy. Table compares several highly performant models within the CNN. Transformer, MAAAINERRLIGHTT out-
MLP, Hybrid and our proposed CONTAINER families. CONTAINER and perform the pure Transformer models ViT [14] and DeiT [54] despite far fewer parameters. They outperform PVT [60] which employ a hicrarchical representation similar to our base architecture. They also outperform the recently published sate-of-thhear SWIN [40] (they outperform Swin-T T which has more parameters). The best performing models continue to be from the EfficientNet [49] family, but we note that EfficientNet [49] and RegNet [44] apply an extensive neural architecture search, which we do not. Finally note that CONTAINER- not only achieves high accuracy but does so at lower FLOPs and much faster throughput than models with comparable capacities.
The CONTAINER framework allows us to easily reproduce past architectures but also to create effective extensions over past work (outlined in Sec 3.3), several of which are compared in Table 2. H-DeiT-S is a hierarchical version of DeiT-S obtained by simply using A within our hierarchical architecture and provides 1.2 gain. Conv-3 (naive convolution (conv) with 3 kernel) aggregates spatial and channel information, where as Group Conv-3 splits input features and performs convs using different kernels- it is cheaper and more effective. When group size channel dim., we get depth-wiss wiit conv. DW-3 is a depthwise convs with 3 by 3 kernel that only aggregates spatial information. Channel information is fused using x convs. MH-DW-3 a multi-head version of DW-3. MH-DW-3 shares kernel parameters within the same group. With fewer kernels, MH-DW-3 achieves comparable performance with DW-3. MIP is an implementation of transposed MI.P for spatial propagation. MLP-LR stands for MLP with fow-rank decompostion. MLP-LR provides better

---

Family
Network
Top-1 Acc Params
FLOPs
Throughput Input dim NAS
ResNet-50 [23]
78.5
25.6M
4.1G
1250.3
224
X
ResNet-101 [23]
79.8
44.7M
7.9G
753.7
224
X
Xception71 [8]
79.9
42.3M
N/A
423.5
299
X
RegNetY-4G [44]
80.0
21M
4.0G
1156.7
224
RegNetY-8G [44]
81.7
39M
8.0G
591.6
2242
CNN
RegNerY-16G [44]
82.9
84M
16.0G
334.7
224
EfficientNNe-B33 [49]
81.6
12M
1.8G
732.1
300
EfficienttN:tB44 [49]
82.9
19M
4.2G
349.4
3802
EfficientNet-B5 [49]
83.6
30M
9.9G
169.1
4562
EfficientNettB66 [49]
84.0
43M
19.0G
96.9
5282
EfficientNet-B7 [49]
84.3
66M
37.0G
55.1
6002
ViT-B/16 [14]
77.9
86M
55.4G
85.9
384
X
ViT-L/16 [14]
76.5
307M
190.7G
27.3
384²
X
DeiT-S [54]
79.9
22.1M
4.6G
940.4
224
X
DeiT-B [54]
81.8
86M
17.5G
292.3
224
X
PVT-T [60]
75.1
13.2M
1.9G
N/A
224
X
PVT-S [60]
79.8
24.5M
3.8G
N/A
2242
X
Transformer
PVT-Medium [60]
81.2
44.2M
6.7G
N/A
2242
X
P6T-
81.7
61.4M
9.8G
N/A
2242
X
ViL-T [69]
76.3
6.7M
1.3G
N/A
224
X
ViL-S [69]
82.0
24.6M
4.9G
N/A
224
X
Swin-T [40]
81.3
29M
4.5G
755.2
224
X
Swin-S [40]
83.0
50M
8.7G
436.9
224
X
Swin-B [40]
83.3
88M
15.4G
278.1
224
X
Mixer-B/16 [52]
76.4
79M
N/A
N/A
224
X
MLP
ResMLP-24 [53]
79.4
30M
6.0G
715.4
224
X
Hybrid
ConvViT [11]
81.3
27M
5.4G
N/A
224
X
BoT-S1-50 [47]
79.1
20.8M
4.3G
N/A
2242
X
BoT-S1-59 [47]
81.7
33.5M
7.3G
N/A
2242
X
Container
CONTAINER
82.7
22.1M
8.1G
347.8
2242
X
(Ours)
CONTAINER-LIGHT
82.0
20.0M
3.2G
1156.9
224
X
Table ImageNet [12] Top-1 accuracy comparison for CNN, Transformer, MLP, Hybrid and Container models. Throughput (images/s) is not reported in all papers (noted N/A). Models that have fewer parameters than CONTAINER upto 10% more parameters highlighted.
performance with fewer parameters. MH-MLP-LR adds a multi-head mechanism over MLP-LR and provides further improvements. In contrast the original MLP-Mixer [52], do add any non-linearity like GELU into CONTAINER as specified in the contextual aggregation equation.
EIITAINER-LIGHH has built-in shift-invariance and parameter sharing mecha- nism. As a result it is more data efficient in comparison to DeiT [54]. Table shows that at the low data regime of 10%, CONTAINER-LIGHH outperforms DeiT by massive 22.5 points.
Data Efficiency.
Convergence Speed. Figure (left) compares the con-
Data ratio CONTAINER-I LINHHERRRIIITTT LIGHT
DeiT
vergence speeds of the two CONTAINER variants with CNN and Transformer (DeiT) [54]. The inductive biases
100
82.0 (+2.1)
%
81.1 (+2.6)
in the CNN enable it to converge faster than DeiT [54], they eventually perform similarly 300 epochs, sug-
50 %
78.8 (+4.8)
74.0
10 %
61.8 (+22.5)
39.3
gesting that dynamic, long range context aggregation is
Table 3:
ImageNet Top-1 Acc for CONTAINER-LIGITE and DeiT-S with varying training sizes. slight accuracy drop.
powerful but slow to converge. CONTAINER combines the best of both and provides accuracy improvements with fast convergence. CONTAINER-
converges as fast with
Emergence of locality. Within our CONTAINER framework, we can easily add static affinity matrix the DeiT architecture. This simple change (1 line of code addition), can provide +0.5 Top-1

---

Method
Top-1 1 Acc Params
a
M
Aymamii atattc
H-DeiT-S
81.0
22.1M
32
A
N/A
Conv-3
79.6
33.8M
N/A
N/A
N/A
N/A
N/A
Group Conv-3
79.7
20.5M
N/A
N/A
N/A
N/A
N/A
DW-3
80.1
18.7M
N/A
cone
MH-DW-3
79.9
18.6M
0
32
N/A
MLP
77.5
50.9M
0
C
N/A
A Amip
MLP-LR
78.9
36.5M
0
C
N/A
A Anlp
MH-MLP-LR
79.6
41.6 M
32
N/A
A mllp
CONTAINER 82.7
22. 1M
L
32
A 90
conv
CONTAINER- -IGHT
82.0
20.0M
C
32
cone
Table 2: ImageNet accuracies for architecture variations (with convolutions, self eeff-attention and MLP) enabled within the CONTAINER framework. As per our notation, C: num channels, M:; num heads, C/M: head dimension. See Sec 3.3 and 4.1 for notation and model details.
Static affinity weights at different positions
30
A25
#50
#90
#100
Imagetes th Vel B0S0Ci
60-
Netwark Layer
80
Network Layer 12
270
280
240
300
Container
Container UGHT
UGHT
Depthwese
Source pisel et this position
Spettaly affinhes enhanced near very
Affinityy
50
100
150
200
250
300
Traningg Epochs
Figure I: (left) Convergence speed comparison between CONTAINER, CONTAINER-LIGIAE Depth- wise conv and DeiT. (right) Visualization of the static affinity weights at different positions and layers. Layer displays the emergence of local affinities (resembling convolutions).
improvement from 79.9% to 80.4%. This suggests that static and dynamic affinity matrices provide complementary information. As noted in See 3.3, we name this CONTAINER- PAM.
It is interesting to visualize the learnt static affinities at different network layers. Figure (right) displays these for layers. Each matrix represents the static affinities for single position, reshaped to 2-d grid to resemble the landscape of the neighboring regions. Within Layer 1, we interestingly observe the emergence of local operations via the enhancement of affinity values next to the source pixel (location). These are akin to convolution operations. Furthermore, the affinity value for the source pixel very small, i.e. each location, the context aggregator does not use current feature. We hypothesize that this is a result of the residual connection [23], thereby alleviating the need to include the source feature within the context. Note that in contrast dynamic affinity, the learnt static matrix is shared for all input images. Notice that Layer 12 displays more global affinity matrix without any specific interpretable local patterns.
4.2 Detection with RetinaNet
Since the attention complexity for CONTAINER-LIGHT is linear high image resolutions (initial layers) and then quadratic, it can be employed for downstream tasks such as object detection which usually require high resolution feature maps. Table 4 compares several backbones applied to the RetinaNet detector [37] on the COCO dataset [38]. Compared to the popular ResNet-50 [23], CONTAINER- achieves 43.8 mAP, an improvement of 7.0, 7.2 and 10.4 on APs, APM. and AP, with comparable parameters and cost. The significant increase for large objects shows the benefits global attention via the dynamic global affinity matrix in our model. CONTAINER LIGHT also surpasses the large convolution-banl with similar number of parameters such as PVT-S [60], ViL-S [69], and SWIN-T [40] by large
backbone X-10 1-101-64 [63] and pure Transformer models

---

Mask R-CNN
RetinaNet
Method
#P
FLOPs
AP
A1%0
A1%
APM
A0
APT3
FLOPs
mAP
APs
A
APr.
ResNet50 [23]
44.2
180G
38.2
58.8
41.4
34.7
55.7
37.2
37.7
239G
36.5
20.4
40.3
48.1
ResNet101 [23]
63.2
259G
40.0
60.5
44.0
36.1
57.5
38.6
56.7
319G
38.5
21.7
42.8
50.4
X-101-32 [63]
62.8
259G
41.9
62.5
45.9
37.5
59.4
40.2
56.4
319G
39.9
22.3
44.2
52.5
X-101-64 [63]
101.9
424G
42.8
63.8
47.3
38.4
60.6
41.3
95.5
483G
23.9 41.0
45.2
54.0
PVT-S [60]
44.1
245G
40.4
62.9
43.8
37.8
60.1
40.3
34.2
226G
40.4
25.0
42.9
55.7
ViL-S [60]
45.0
174G
41.8
64.1
45.1
38.5
61.1
41.4
35.6
252G
41.6
24.9
44.6
56.2
SWIN-T [40]
48.0
267G
43.7
66.6
47.4
39.8
63.6
42.7
385
244G
41.5
26.4
45.1
55.7
ViL-M [60]
60.1
261G
43.4
65.9
47.0
39.7
62.8
42.1
50.7
338G
42.9
27.0
46.1
57.2
ViL-B [60]
76.1
365G
45.1
67.2
49.3
41.0
64.3
44.2
66.7
443G
44.3
28.9
47.9
58.3
BoT50 [47]
39.5
N/A
39.4
60.3
43.0
35.3
57
37.5
NA
N/A
N/A
N/A
N/A
N/A
(6X) [47]
39.5
N/A
43.7
64.7
47.9
38.7
61.8
41.1
N/A
N/A
N/A
N/A
N/A
N/A
CONTAINER- LIGHT
39.6
237G
45.1
67.3
49.5
41.3
64.2
44.5
29.7
218G
43.8
27.4
47.5 58.5
Table 4:
Comparing the LONTAINER-LIGHT backbone with several previous methods at the tasks of object detection and instance segmentation using the Mask-RCNN and RetinaNet networks.
margins. Compared to large Transformer backbones such ViL-M [69] and ViL-B [69], we achieve comparable performance with significantly fewer parameters and FLOPs.
4.3 Detection and Segmentation with Mask-RCNN
Table 4 also compares several backbones for detection and instance segmentation using the Mask
R-CNN network [22]. As with the findings for RetinaNet [37]. CONTAINER- outperforms
convolution and Transformer based approaches such as ResNet [23], X-101 [63], PVT [60]. ViL [69]
and recent state-of-the-arr SWIN-T [40] and the recent hybrid approach BoT [47]. obtains comparable numbers to the much larger ViL-B [69].
4.4 Detection with DETR
Method
mAP
DETR-ResNeL50 [S]
32.3
Table 5 shows that our model can consistently improve object detection performance com- pared to ResNet-50 [23] backbone (compa- rable parameters and computation) on end-to- end object detection using DETR [5]. We demonstrate large improvements with DETR [5], DDETR [75] as well SMCA-DETR [19]. See appendix for APS, APM and APL numbers.
DETR- WEETR-COONAINER--IE LIGHT
38.9
DDETR w/o multi-scale mulli-scale-ResNeibhh [75]
39.3
DDETR w/o multi-scale- CONTAAINE--- LIGHT
43.0
SMCA w/o multi scali-sscaa--Bee Nulti-scall-ReeNeet500 [19
41.0
SMCA w/o multi serri-scall--Cee Sesle-CONTAINER LIGHT
44,2
Table 5: CONTAINER-
and ResNet-50 back-
bones with DETR and variants for object detection.
All models in table 5 are trained using 50 epochs schedule.
4.5 Self supervised learning
Epochs
20
40
60
80
100
We train DeiT [54] and CONTAINER-LIGHT for 100 epochs at the self supervised task of visual representa- tion learning using the DINO framework [6]. Table 6 compares top- 10 kNN accuracy for both backbones different epochs of training. CONTAINER-Lioe significantly outperforms DeiT with large improve- ments initially demonstrating more efficient learning.
DeiT [6]
52.0 63.3 66.5
68.9
69.6
CONTAINER LIGHT
58.0
67.0
70.0
71.1
71.5
Table 6: CONTAINER- and DeiT on DINO self-supervised learning.
Conclusion
In this paper, we have shown that disparate architectures such as Transformers, depth-wise CNNs and MLP-based methods are closely related via an affinity matrix used for context aggregation, Using this view, we have proposed CONTAINER, generalized context aggregation building block that combines static and dynamic affinity matrices using learnable parameters. Our proposed networks, CONTAINER and CONTAINER-L LIGHT show superior performance at image classification, object detection, instance segmentation and self-supervised representation learning. We hope that this unified view motivate future research in the design of effective and efficient visual backbones.

---

CONTAINER is very effective at image classification but cannot be directly applied to high resolution inputs. The efficient version CONTAINER-LIGHT However, its limitation is that it is partially hand-crafted - the dynamic affinity matrix is switched off in the first 3 stages. Future work will address how to learn this using the task at hand.
Limitations:
can be used for a variety of tasks.
This research does not have a direct negative societal impact. However, we should be aware that powerful neural networks, particularly image classification networks can be used for harmful applications like face and gender recognition.
Negative societal impact:
Disclosure of Funding This work was partially supported by the Shanghai Committee of Science and Technology, China (Grant No. 21DZ1100100 and 20DZI 100800).
References
[1] Peter Anderson, X.
He, Chris Buchler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018. 1
[2] Alexei Baevski, Henry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vee 2.0; A framework for self-supervised learning of speech representations. In NewrIPS, 2020. 3
[3] Iz Beltagy, Matthew Peters, and Arman Cohan. Longformer: The long-document transformer. arXiv,
2020. 3
[4] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berer, Sam McCandlish, Alec Radford, lyya Sutskever, and Dario Amodei. Language models are few-shot learners. In NewrlPS, 2020. 1
[5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers, In ECCV, 2020. 1, 3.9 [6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. arXiv, 2021. 2, 4,
[7] Xinlei Chen, Saining Xie, and Kaiming He., An empirical study of training self seff-supervised visual transformers. arXiv,, 2021.
[8]
François Chollet. Xception: Deep learning with depthwise separable convolutions. In CVPR, 2017. 2, 3, 4,
[9] Krzysztof Choromanki. Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamas Sarlos, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, et al. Rethinking attention with performers.
In ICLR, 2021. 3
[10] Giannis Daras, Nikita Kitaev, Augustus Odena, and Alexandros G Dimakis. Smyrf: Efficient attention
using asymmetric clustering. In NeurlPS, 2020. 3
[11] Stéphane d' Ascoli, Hugo Touvron, Matthew Leavitt, Ari Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. arXiv, 2021. 3,7
[12] Jia Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical
image database. In CVPR, 2009. 7
[13] Jacob Devlin, Ming-Wes -ei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In NAACL, 2019. 3,3
[14] Alexey Dosovitskiy. Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn Xiaohua Zhai. Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR, 2021. 1, 2. 6,
[15] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis.
In CVPR, 2021. 3
[16] Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In CVPR, 2019. 3
[17] Peng Gao, Zhengkai Jiang, Haoxuan You, Pan Lu, Steven CH Hoi, Xiaogang Wang, and Hongsheng Li. Dynamic fusion with intra-and moter-modality attention flow for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages (639-6643 2019. 3
[18] Peng Gao, Haoxuan You, Zhanpeng Zhang, Xiaogang Wang, and Hongsheng Li. Multi-modality latent interaction network for visual question answering. In ICCV, 2019. 3
[19] Peng Gao, Minghang Zheng, Xiaogang Wang. Jifeng Dai, and Hongsheng Li. Fast convergence of detr
with spatially modulated co-attention. arXiv, 2021. 2. 9
[20] Shijie Geng, Peng Gao, Moitreya Chatterjee, Chior Hori, Jonathan Le Roux, Yongfeng Zhang, Hongsheng
Li, and Anoop Cherian. Dynamic graph representation learning for vidco dialog via multi-modal shuffled transformers. arXiv preprint arXiv:2007.00 00000000088..11 2020.
[21] Shijie Geng. Ji Zhang, Zuohui Fu, Peng Gao, Hang Zhang, and Gerard de Melo. Character matters: Video story understanding with character- aware relations. arXiv preprint arXiv:2005. 2020. 3 Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017. 1, Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition.
[22]
[23]
In CVPR, 2016. 2, 3, 4, 6, 7,
10

---

[24] Byeongho Heo. Sangdoo Yun, Dongyoon Han. Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh.
Rethinking spatial dimensions of vision transformers. arXiv, 2021. 2
[25] Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory, Neural computation. 9(8), 1997. 3 Andrew G Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision
[26]
applications. arXiv. 2017. 3
[27]
Han Hu, Jiayuan Gu, Zheng Zhang, Jifeng Dai, and Yichen Wei. Relation networks for object detection.
In CVPR, 2018. 3
[28]
Jie Hu, Li Shen, and Gang Sun. Squeeze-and excitation networks. In CVPR, 2018.
[29]
Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, and Wenyu Liu. Cenet: Criss-cross attention for semantic segmentation. In ICCV, 2019. 3
[30] Lukasz Kaiser, Aidan N Gomez. and Francois Chollet. Depthwise separable convolutions for neural machine translation. arXivv 2017. 4
[3]] Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas, and François Fleuret. Transformers are rns:: Fast
autoregressive transformers with linear attention. In ICML, 2020. 3
[32] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. Reformer: The efficient transformer. In ICLR, 2020.
[33] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. NewrIPS, 2012. 1,3
[34] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient- learning applied to document recognition. Proceedings of the IEEE, 86(11), 1998. 1, 2.
[35] Duo Li, Jie Hu, Changhu Wang, Xiangtai Li, Qi She, Lei Zhu, Tong Zhang, and Qifeng Chen. Involution: Inverting the inherence of convolution for visual recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition. pages 12321-12330, 2021.
[36] Yawci Li, Kai Zhang, Jiezhang Cao, R. Timofte, and L. Gool. Localvit: Bringing locality to vision transformers. Xrv, 2021.
[37] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár, Focal loss for dense object detection. In ICCV, 2017. 2, 9
[38] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014, 8
[39] Yicheng Liu, Jinghuai Zhang, Liangi Fang, Qinhong Jiang, and Bolei Zhou. Multimodal motion prediction
with stacked transformers. In CVPR, 2021. 4
[40] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin
transformer: Hierarchical vision transformer using shifted windows. arXiv, 2021. 2, 6, 7, 8, 9
[41] Aaron van den Oord. Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabricl Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning
transferable visual models from natural language supervision. arXiv, 2021.
[43] Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv, 2015. 1,
[44] Iija Radosavovic, Raj Prateek Kosaraju. Ross Girshick. Kaiming He, and Piotr Dollár. Designing network design spaces. In CVPR, 2020. 6.7
[45] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv, 2016.
[46] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni-
tion. In ICLR, 2015. 3
[47] Aravind Srinivas, Tsung- Yi Lin, Niki Parmar, Jonathon Shlens, Picter Abbeel, and Ashish Vaswani.
Bottleneck transformers for visual recognition. arXiv, 2021. 2, 7
[48] Christian Szegedy. Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In CVPR, 2015.
[49] Mingxing Tan and Quoc Lee. Effiiienttnt: Rethinking model scaling for convolutional neural networks. In
ICMI., 2019. 6,
[50] Mingxing Tan and Quae V. Le. Efficienttetv2: Smaller models and faster training. arXiv, 2021. 3
[51] Yi Tay, Dara Bahri, Donald Metzler, Da-Cheng Juan, Zhe Zhao, and Che Zheng, Synthesizer: Rethinking self-attention for transformer models. In International Conference on Machine Learning, pages 10183-
10192. PMLR, 2021. 3
[52] Ilya Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner. Jessica Yung. Daniel Keysers. Jakob Uszkoreit, Mario Lucic, and Alexey Dosovitskiy. Mip-mixer: An all-mip architecture for vision. arXiv, 2021. 1, 2, 3, 4, 5,
[53] Hugo Touvron, Piotr Bojanowski, Mathilde Caron, Matthieu Cord, Alaaeldin El-Nouby, Edouard Grave, Armand Joulin, Gabriel Synnaeve, Jakob Verbeek, and Hervé Jégou. Resmlp: Feedforward networks for image classification with data-efficient training. arXiv, 2021. 7
[54] Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. arXiv, 2020. 1, 1, 6,
7.
[55] Ashish Vaswani. Prajit Ramachandran, A. Srinivas, Niki Parmar, Blake A. Hechtman, and Jonathon Shlens.
Scaling local self-attention for parameter efficient visual backbones. arXiv, 2021. 2
11

---

[56]
Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, 2017. 3, 4,
[57]
Apoorv Vyas, Angelos Katharopoulos, and François Fleuret. Fast transformers with clustered attention. In NeurIPS, 2020.
[58]
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A
multi-task benchmark and analysis platform for natural language understanding, arXivv 2018. 1
[59] Sinong Wang. Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear
complexity. Xrviv, 2020.
[60] Wenhai Wang. Enze Xie, Xiang Li, Deng-Ping Ping Fan, Kaitao Song. Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.
arXiv, 2021. 3, 6, 7, 8, 9
[61]
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks, In CVPR,
2018. 1,3
[62]
Felix Wu, Angela Fan, Alexci Baevski, Yann N Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In ICLR, 2019. 3
[63]
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transforma-
tions for deep neural networks. In CVPR, 2017. 2, 3, 8,
[64]
Weijian Xu, Yifan Xu, Tyler Chang, and Zhuowen Tu. Co-scale conv-attentional image transformers.
arXiv, 2021. 2
[65]
Sen Yang. Zhibin Quan, Mu Nie, and Wankou Yang. Transpose: Towards explainable human pose
estimation by transformer. arXiv, 2020.
[66]
Kun Yuan, Shaopeng Guo, Ziwei Liu, Aojun Zhou, Fengwei Yu, and Wei Wu. Incorporating convolution designs into visual transformers. arXivv 2021.
[67]
Yuhui Yuan, Xilin Chen, and Jingdong Wang. Object-contextual representations for semantic segmentation.
In ECCV, 2020. 3
[68]
Manzil Zaheer, Guru Guruganesh, Avinava Dubey, Joshua Ainslic, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, et al. Big bird: Transformers for longer sequences. In NeurlPS, 2020. 3
[69] Pengchuan Zhang, Xiyang Dai, Jianwei Yang. Bin Xiao, Lu Yuan, Lei Zhang, and Jianfeng Gao. Multi- scale vision longformer: A new vision transformer for high-resolution image encoding. arXiv, 2021. 2, 7, 8,
[70] Qinglong Zhang and Yubin Yang. Rest: An efficient transformer for visual recognition. arXiv preprint
arXiv:2105. 2021.
[71] Guangxiang Zhao, Xu Sun, Jingjing Xu, Zhiyuan Zhang, and Liangchen Luo. Muse: Parallel multi-scat attention for sequence to squuene learning. arXiv preprint arX:::1991.0090000 1.04433, 2019. 3
[72] Zelin Zhao, Karan Samel. Binghong Chen, and Le Song. Proto: Program-guided transformer for program- guided tasks. arXiv preprint 10 00804, 2021. 4
[73] Minghang Zheng, Peng Gao, Xiaogang Wang, Hongsheng Li, and Hao Dong. End-to-end object detection with adaptive clustering transformer. arXiv, 2020.
[74] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng
Feng. Tao Xiang. Philip H. S. Torr, and Li Zhang. Rethinking semantic segmentation from a sequence-to- sequence perspective with transformers. In CVPR, 2021. 3
[75] Xizhou Zhu, Weijie Su, Lewei La, Bin Li, Xiaogang Wang. and Jifeng Dai. Deformable detr: Deformable
transformers for end-to-end object detection. In ICLR, 2021. 2,9
[76] Zhen Zhu, Mengde Xu, Song Bai, Tengteng Huang, and Xiang Bai. Asymmetric non-local neural networks for semantic segmentation. /CCV, 2019.
12


Reviews:
[{"review_id": "o3PB0tZroR7", "reviewer": "Reviewer_ecxB", "summary": "The paper proposes an architecture that combines Self-Attention, Mixer and DWise Conv blocks for spatial aggregation/mixing; along with channel mixing MLPs. While there's a lot of mathematical notation and it was a bit hard for me to understand quickly, my understanding is that: basically, the paper says - you can combine self-attention (dynamic filters generated) with DWise Conv (static + local) OR self-attention with Mixer (static + global) - and calls the two models as Container and Container-PAM (PAM = Pay Attention to MLP). Paper also proposes a hybrid model called Container-Lite which is basically Container with first three stages just using DWise Conv and no self-attention (for speeds) - and they use this architecture for detection/segmentation (in a manner similar to BoTNet). There are plenty of experiments in the paper - ImageNet, COCO, Self-Supervised Learning, etc. and some unifying presentation around MLP-Mixer, ViT and DWise Convs. Main review in the review section.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives:\n1. Good empirical results.\n2. Many benchmarks evaluated.\n\nNegatives:\n\n0. Unnecessary mathematical and verbose presentation of the idea. The core idea is extremely simple - just combine Mixer & Attention (or DWise and Attention). A simple diagram would go a long way. But please do correct me if my understanding is incorrect.\n1. Citation to gMLP - which is also a hybrid model with mixer + attention makes sense to add.\n2. Mixer - does share params across channels - \"no parameter sharing\" doesn't typecheck.\n3. Improvements are compared wrt DeiT, a single-scale model, but since Container is doing multi-scale modeling, with engineered # of blocks like [2, 3, 8, 3] (similar to Swin), shouldn't you ideally be comparing to Swin? Container seems to be worse than Swin-S, but Container-Lite is better than Swin-T -- however Swin-T and Container-Lite don't seem directly comparable either - given Swin-T is pure attention and Container-Lite is like BotNet (3 stages conv and 1 attention) - it's expected that hybrids could also improve Swin-T right?\n4. Dwise vs GConv - why does the design use DWise - and also - whats the hardware for the throughput measurements?\n5. if Container relies heavily on Dwise, how does it scale to larger models? It's known DWise doesn't scale well to large models (ex EfficientNets-B6,7, L2, etc). \n6. How much can a pure Mixer trained with all the data augmentation achieve? (in order to know the contribution of the hybrid design) -- FLOPs/param matched.\n7. What about pure Dwise - with same design? - FLOPs/param matched.\n8. BoTNet throughputs are easy to report since implementation is just ResNet with final stage using SA blocks. Would recommend authors to do it for ImageNet table at least.\n9. Why no throughput numbers for Mask R-CNN results? Would be interested in knowing them since EfficientNets (and Dwise convs) are really slow for larger images.\n\nI think paper has more negatives right now, but I am open to hearing back from authors.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "N/A", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "cAps8We_9vn", "reviewer": "Reviewer_sSVS", "summary": "This paper first provides a unified view on three existing network architectures: depthwise  CNNs, Transformers and MLP-Mixers. It then proposes Container and a light-weight version Container-Light, which combine CNNs and self-attention to aggregate spatial information. Experiments on ImageNet classification and COCO object detection verify the high modeling ability of Container.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper first proposes a unified spatial context aggregation framework for vision models, and points out that depthwise convolutions, Transformers and MLP-Mixers are special cases of this framework. Then it introduces Container blocks which employ self-attention and convolutional modules at the same time for context aggregation. \n\nThe discussion about how to unify different architectures and create new extensions in Section 3.2 and Section 4.1 are interesting. This discussion may help researchers to understand relationships between different architectures and inspire the community to create new ones\n\nThe proposed Container model achieves reasonably high classification accuracy and it consistently boost object detection performance with different frameworks, by a large margin. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1. Container model adopts global self-attention modules. For image classification task, it requires much more computation (higher FLOPs) than other models with comparable parameters. For detection task, it is hard to directly applied and have to use the light-weight version.\n\n2. The proposed model simply combines self-attention and depthwise convolution in one block, which lacks novelty in a degree.  With the unified framework of context aggregation, more brand-new modules can be developed. However, this part is missed in this paper.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "KKPa1-MmQR", "reviewer": "Reviewer_sZdD", "summary": "This paper presents a unified view of CNNs, Transformers and MLP-mixer, and proposes the CONTAINER (CONText AggregatIon NEtwoRk), a general-purpose building block for multi-head context aggregation that combines convolution, attention, and depth-wise MLP. Various experiments on ImageNet classification, object detection with RetinaNet, Mask R-CNN, and DETR, as well as self-supervised learning. ", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The connection between depth-wise convolution, attention, and channel-wise MLP is interesting. I believe it is novel when this paper is submitted to NeurIPS 2021, and there is a concurrent work: \"Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight\" https://arxiv.org/abs/2106.04263, providing more studies about relation. \n\nThe study will motivate researchers to deeply think about what transformer attention really brings. This paper provides some insights about the relation between transformer attention, MLP and depth-wise convolution. This is a good start though the combination is a little straightforward. Thus I was positive to this paper. \n\nClarity and quality: This paper is clearly written. The reviewer can easily understand this paper. The results are also good for supporting the claim. \n\nSignificance: The reviewer believes that the relation between depth-wise convolution, attention, and channel-wise MLP is more significant than the combination. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Overall this paper is acceptable. The reviewer expects to see more discussion about the relation, like \"Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight\" https://arxiv.org/abs/2106.04263. This will help researchers deeply understand transformer attention. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1 hour"}, "extra_text": {}}, {"review_id": "0WukUmI1gSc", "reviewer": "Reviewer_wcp4", "summary": "The paper proposes a unified view between CNNs (but really just depthwise CNNs), ViTs, and MLP-Mixer models. In essence this unified view consists of having a*static + b*dynamic, where \"static\" is an input-independent learnable filter (for CNN and MLP-Mixer) and \"dynamic\" is an input-dependent filter as in self-attention, a/b are \"switches\".\n\nIn addition to this container model, the paper also suggests a \"light\" variant, where b=0 for all but the last stage, \"stage\" being the resnet-like definition. This variant is especially good for high-resolution tasks.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think this is an overall solid paper. The unifying view is not ground-breaking or even surprising, but it's good to investigate it, and the resulting model is thoroughly benchmarked across multiple tasks. I really like the related work, it seems pretty thorough to me.\n\nThe model performs well, but agian, it's not ground-breaking: compared to models with similar throughput in Table1, container reaches very similar accuracy.\n\nTwo things got me wondering:\n1) Container-light seems *very* similar to ResNet50. The throughput is also very similar. However, performance is much better. My guess is this is due to the training recipe and not the model. Hence, it would be good to also include a more modern ResNet for comparison, such as ResNet-RS or NFNet.\n\n2) I must be missing something, but the model consists of 4 stages with downsampling, similar to ResNet. However, the filters shown in Figure1/right have the same size for both the first and the 12th layer? How can this be, shouldn't the one from the 12th layer be lower resolution?\n\nBesides that, I have a few nitpicks:\n\n3) related work: DETR is mentioned as if it were a pure transformer, but it actually uses a ResNet encoder.\n4) Table3 is not very useful. There are many more papers on 10% ImageNet with much better results than DeiT. Even ignoring unlabeled data (as used by SimCLR, MoCo, ...), the S4L paper shows a ResNet50 trained on 10% gets 56%. The 50% and 80% ImageNet are almost pointless settings, but 1% is also widespread and missing.\n5) l54: what does \"the landscape of the input\" mean?\n6) l303: It's a stretch to call them akin to convolutions, since they don't actually apply across locations. The point is that they are mostly local.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "None", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper is somewhat borderline: three reviewers argue for acceptance, while one leans towards rejection. Based on the reviews, the rebuttal, the discussion, and the paper itself, below is the summary of key pros and cons.

Pros:
1) An interesting framework for unifying several classes of vision models (ConvNet, ViT, Mixer).
2) Good empirical results in terms of performance/compute tradeoff on classification and, especially, on detection.

Cons:
1) Not overwhelmingly novel
(some other points authors successfully addressed in their responses and I strongly recommend to include these responses/results in the paper)

To conclude, this is a solid paper, and the authors quite successfully responded to the concerns of the only negative reviewer, so overall I recommend acceptance.
Decision: Accept (Poster)