Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes an architecture that combines Self-Attention, Mixer and DWise Conv blocks for spatial aggregation/mixing; along with channel mixing MLPs. While there's a lot of mathematical notation and it was a bit hard for me to understand quickly, my understanding is that: basically, the paper says - you can combine self-attention (dynamic filters generated) with DWise Conv (static + local) OR self-attention with Mixer (static + global) - and calls the two models as Container and Container-PAM (PAM = Pay Attention to MLP). Paper also proposes a hybrid model called Container-Lite which is basically Container with first three stages just using DWise Conv and no self-attention (for speeds) - and they use this architecture for detection/segmentation (in a manner similar to BoTNet). There are plenty of experiments in the paper - ImageNet, COCO, Self-Supervised Learning, etc. and some unifying presentation around MLP-Mixer, ViT and DWise Convs. Main review in the review section.
main_review: Positives:
1. Good empirical results.
2. Many benchmarks evaluated.

Negatives:

0. Unnecessary mathematical and verbose presentation of the idea. The core idea is extremely simple - just combine Mixer & Attention (or DWise and Attention). A simple diagram would go a long way. But please do correct me if my understanding is incorrect.
1. Citation to gMLP - which is also a hybrid model with mixer + attention makes sense to add.
2. Mixer - does share params across channels - "no parameter sharing" doesn't typecheck.
3. Improvements are compared wrt DeiT, a single-scale model, but since Container is doing multi-scale modeling, with engineered # of blocks like [2, 3, 8, 3] (similar to Swin), shouldn't you ideally be comparing to Swin? Container seems to be worse than Swin-S, but Container-Lite is better than Swin-T -- however Swin-T and Container-Lite don't seem directly comparable either - given Swin-T is pure attention and Container-Lite is like BotNet (3 stages conv and 1 attention) - it's expected that hybrids could also improve Swin-T right?
4. Dwise vs GConv - why does the design use DWise - and also - whats the hardware for the throughput measurements?
5. if Container relies heavily on Dwise, how does it scale to larger models? It's known DWise doesn't scale well to large models (ex EfficientNets-B6,7, L2, etc). 
6. How much can a pure Mixer trained with all the data augmentation achieve? (in order to know the contribution of the hybrid design) -- FLOPs/param matched.
7. What about pure Dwise - with same design? - FLOPs/param matched.
8. BoTNet throughputs are easy to report since implementation is just ResNet with final stage using SA blocks. Would recommend authors to do it for ImageNet table at least.
9. Why no throughput numbers for Mask R-CNN results? Would be interested in knowing them since EfficientNets (and Dwise convs) are really slow for larger images.

I think paper has more negatives right now, but I am open to hearing back from authors.
limitations_and_societal_impact: N/A
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper first provides a unified view on three existing network architectures: depthwise  CNNs, Transformers and MLP-Mixers. It then proposes Container and a light-weight version Container-Light, which combine CNNs and self-attention to aggregate spatial information. Experiments on ImageNet classification and COCO object detection verify the high modeling ability of Container.
main_review: This paper first proposes a unified spatial context aggregation framework for vision models, and points out that depthwise convolutions, Transformers and MLP-Mixers are special cases of this framework. Then it introduces Container blocks which employ self-attention and convolutional modules at the same time for context aggregation. 

The discussion about how to unify different architectures and create new extensions in Section 3.2 and Section 4.1 are interesting. This discussion may help researchers to understand relationships between different architectures and inspire the community to create new ones

The proposed Container model achieves reasonably high classification accuracy and it consistently boost object detection performance with different frameworks, by a large margin. 

limitations_and_societal_impact: 1. Container model adopts global self-attention modules. For image classification task, it requires much more computation (higher FLOPs) than other models with comparable parameters. For detection task, it is hard to directly applied and have to use the light-weight version.

2. The proposed model simply combines self-attention and depthwise convolution in one block, which lacks novelty in a degree.  With the unified framework of context aggregation, more brand-new modules can be developed. However, this part is missed in this paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a unified view between CNNs (but really just depthwise CNNs), ViTs, and MLP-Mixer models. In essence this unified view consists of having a*static + b*dynamic, where "static" is an input-independent learnable filter (for CNN and MLP-Mixer) and "dynamic" is an input-dependent filter as in self-attention, a/b are "switches".

In addition to this container model, the paper also suggests a "light" variant, where b=0 for all but the last stage, "stage" being the resnet-like definition. This variant is especially good for high-resolution tasks.
main_review: I think this is an overall solid paper. The unifying view is not ground-breaking or even surprising, but it's good to investigate it, and the resulting model is thoroughly benchmarked across multiple tasks. I really like the related work, it seems pretty thorough to me.

The model performs well, but agian, it's not ground-breaking: compared to models with similar throughput in Table1, container reaches very similar accuracy.

Two things got me wondering:
1) Container-light seems *very* similar to ResNet50. The throughput is also very similar. However, performance is much better. My guess is this is due to the training recipe and not the model. Hence, it would be good to also include a more modern ResNet for comparison, such as ResNet-RS or NFNet.

2) I must be missing something, but the model consists of 4 stages with downsampling, similar to ResNet. However, the filters shown in Figure1/right have the same size for both the first and the 12th layer? How can this be, shouldn't the one from the 12th layer be lower resolution?

Besides that, I have a few nitpicks:

3) related work: DETR is mentioned as if it were a pure transformer, but it actually uses a ResNet encoder.
4) Table3 is not very useful. There are many more papers on 10% ImageNet with much better results than DeiT. Even ignoring unlabeled data (as used by SimCLR, MoCo, ...), the S4L paper shows a ResNet50 trained on 10% gets 56%. The 50% and 80% ImageNet are almost pointless settings, but 1% is also widespread and missing.
5) l54: what does "the landscape of the input" mean?
6) l303: It's a stretch to call them akin to convolutions, since they don't actually apply across locations. The point is that they are mostly local.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper presents a unified view of CNNs, Transformers and MLP-mixer, and proposes the CONTAINER (CONText AggregatIon NEtwoRk), a general-purpose building block for multi-head context aggregation that combines convolution, attention, and depth-wise MLP. Various experiments on ImageNet classification, object detection with RetinaNet, Mask R-CNN, and DETR, as well as self-supervised learning. 
main_review: Originality: The connection between depth-wise convolution, attention, and channel-wise MLP is interesting. I believe it is novel when this paper is submitted to NeurIPS 2021, and there is a concurrent work: "Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight" https://arxiv.org/abs/2106.04263, providing more studies about relation. 

The study will motivate researchers to deeply think about what transformer attention really brings. This paper provides some insights about the relation between transformer attention, MLP and depth-wise convolution. This is a good start though the combination is a little straightforward. Thus I was positive to this paper. 

Clarity and quality: This paper is clearly written. The reviewer can easily understand this paper. The results are also good for supporting the claim. 

Significance: The reviewer believes that the relation between depth-wise convolution, attention, and channel-wise MLP is more significant than the combination. 
limitations_and_societal_impact: Overall this paper is acceptable. The reviewer expects to see more discussion about the relation, like "Demystifying Local Vision Transformer: Sparse Connectivity, Weight Sharing, and Dynamic Weight" https://arxiv.org/abs/2106.04263. This will help researchers deeply understand transformer attention.
review_title: null

Meta-review: The paper is somewhat borderline: three reviewers argue for acceptance, while one leans towards rejection. Based on the reviews, the rebuttal, the discussion, and the paper itself, below is the summary of key pros and cons.

Pros:
1) An interesting framework for unifying several classes of vision models (ConvNet, ViT, Mixer).
2) Good empirical results in terms of performance/compute tradeoff on classification and, especially, on detection.

Cons:
1) Not overwhelmingly novel
(some other points authors successfully addressed in their responses and I strongly recommend to include these responses/results in the paper)

To conclude, this is a solid paper, and the authors quite successfully responded to the concerns of the only negative reviewer, so overall I recommend acceptance.
Decision: Accept (Poster)