Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper presented a Dual-stream Network for image classification. It combines the representation of local and global pattern features by using self-attention and convolution together. It also proposed an Intrascale Propagation module to process two different resolutions in each block.  In addition, it also introduced such block to FPN to demonstrate the benefit.
main_review: This paper is easy to read. The methodology and its novelty are clear. However, the setup of experiments are problematic and confusing.

Strength:
+ The proposed Intra-scale Propagation module is novel. Combing SA with Convolution makes sense.
+ The performance and speed is comparable to concurrent works.  

Weakness: There are some issues needed to be addressed before accepted.
- In Table 3, the model setup for tiny and small model is confusing. It is not aligned with competitors on parameters and flops, which make hard to do fairly comparison.  
- It lacks the performance comparison on large models. Will the performance hold? 
- It lacks necessary ablation study on the proposed Intra-scale Propagation module. Based on Table 3, the improvement of such module is small. It is also decreasing when model size increases. The author needs to justify the effectiveness of this module.
- The benefit of DS-FPN is not convincing. Although it shows improvement in Table 4, it also significantly increases the number of parameters. How about the latency?
- Writing issues: The precision used in reporting numbers is not consistent.  The presentation of tables (e.g. font, whitespace) needs improvement.

Post-rebuttal:
Thanks authors for providing more details. However,  the newly provided ablation study doesn’t justify the effectiveness of proposed components. Hence, I will hold my rating as is.

limitations_and_societal_impact: Please justify the weakness to improve the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes a hybrid convolution+transformer model for 2d images. To summarize the model, it is broken up into 4 stages that operate at successively higher resolutions much like a typical Resnet.  At each stage, there are two parallel paths, one that apples a depthwise convolution at high resolution (capturing local effects), and one that applies a standard a multi-headed attention at coarse (capturing global effects) resolution.  These paths are fused by a final attention module which allows for local to attend to global and vice versa.

Another contribution is the DS-FPN (which simply slots one of the above “dual stream” modules into the lateral paths of a typical FPN). Experiments show strong performance on both image classification for ImageNet and detection/segmentation for COCO (relative to competing Transformer-only models like Deit).

main_review: Overall, the authors have proposed an intuitive and conceptually simple model that enjoys the benefits of both convolution and transformer/attention.  They achieve strong performance without an overly expensive model, which to my knowledge is novel. I have some complaints on exposition/writing but no major criticisms otherwise.  Based on good execution and reasonably strong results, I’m recommending acceptance.

Some more detailed comments/questions:
* In several places throughout the paper, the authors claim that something is “obvious” when it is not really obvious (and certainly not provable in a mathematical sense)… I recommend tempering this language.
* In the intro, the authors say “​​the conflicting properties of such two operations, convolution for local patterns but attention for global patterns, might cause ambiguity during training,...”: it’s not clear what this means and how one might quantify this statement.
* There is a sentence in the related work section: “progressively aggregate neighboring tokens before transformers via overlapping convolution-alike way to enhance their representations” that makes no sense.
* Related paper: “MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers” by Wang et al also use convolutions and multi-headed attention in a manner where both types of modules are woven fairly tightly together through the network. It’s quite a different approach in some ways, but should be cited here.
* Figure 2: is missing a 1x1 convolution at the end of the Inter-scale alignment block
* Is there a network stem that first does a reduction to stride 4 (as in Resnet)?
* I’m very surprised that only depthwise convolutions are used in the convolution path — and would like to clarification as to whether these are literally depthwise convolutions as stated in Eqn 1 or whether there is also a 1x1 convolution following (thus making it a depthwise separable convolution, which is far more commonly used in convnets).  Using a depthwise-only convolution is pretty limited, allows for no interaction between channels and essentially would amount to a blur operation, which is why this is so surprising.
    * Along the same lines, some papers have found it important to make the first convolution
* Clarification: are positional encodings used anywhere?  Would they help if not?
* For Table 3, I will just note that there are many more recent convolution-only networks than ResNet — I know that this is not necessarily the point, but it would be good to have a comparison against the most recent convolution-only networks to see how the transformer based models stand.
* For Table 4, I recommend adding in some other baselines, especially DETR (a transformer based detection model).
* In Section 4.3, authors claim to evaluate DS-Net in the Cascade R-CNN model, but these experiments do not seem to be in the paper.
* Why is DS-Net-T used in Table 6 instead of DS-Net-S?
* What is the running time impact of using DS modules in the FPN?

limitations_and_societal_impact: Yes.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The authors propose a new architecture for visual recognition. Like standard convolutional architectures, the representation is structured in a multi-scale hierarchy, with deeper layers being coarser and more abstract. At every level of the hierarchy, a coarse, global representation is appended and simultaneously processed with the more fine-grained, local one. Local features are processed with depth-wise convolutions, and global ones with multi-head self-attention. In addition to this separate processing of local and global features, they are co-processed by cross-attending between local and global features. 

In addition to this recognition backbone, the authors adapt Feature Pyramid Networks in a similar spirit. Whereas FPN's use dense convolutions to fuse an intermediate representation with a coarser one, the authors instead use the proposed "Dual-Stream" block, combining local depth-wise convolutions, global self-attention, and local/global cross-attention. 

The authors evaluate the recognition backbone for classification on ImageNet and detection and instance segmentation on COCO, reporting large gains over standard convolutional architectures such as ResNet, and reasonable gains over more recent transformer-based ones.
main_review: The authors propose a new hybrid architecture which advances the state of the art in visual recognition. Rather than seeking to entirely replace the convolutional architecture with attentional ones, as prior work has, the authors introduce a careful combination of convolutional and attention-based operations. In particular, the main obstacle for introducing attention in visual architectures is the quadratic scaling of attention operations with respect to the number of tokens. The authors overcome this limitation by introducing low-resolution latents at every stage, and attending within or across these latents. Nevertheless, the architecture is simple and general enough to be used as a drop-in replacement to other visual backbones (such as ResNet), indicating the generality of the approach. The gains afforded by their FPN variant (which incorporates the same design principle) also support this generality. Together, these qualities make this a solid contribution. 
limitations_and_societal_impact: The authors propose a new architecture for visual recognition and evaluate it on standard benchmarks. As such the work does not raise any new ethical concerns.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper proposes Dual-stream Network (DS-Net) for visual recognition. The two streams correspond to high-resolution convolutions for local features and low-resolution vision transformers for global patterns. The paper proposes an intra-scale propagation module to process the two streams and an inter-scale alignment module to fuse them into one. For object detection, it designs a Dual-stream FPN (DS-FPN) that adds dual-stream blocks to every feature pyramid. The proposed DS-Net achieves good results for image classification, object detection, and instance segmentation.
main_review: The paper is overall nicely organized. I have some comments that might need the authors’ attention.

1. The writing quality needs improvements. There are many grammar errors here and there. I would recommend the authors pay more attention to proofreading. The frequency of the writing errors is high and it concerns me. This factor is not considered when giving the ratings, though, because I believe they can be fixed.

2. The paper likes to claim that a statement is obvious. For example, in Line 38, the paper states that the previous works that attempt to introduce convolutions to vision transformers are obviously not the most ideal design. I personally don’t think it’s appropriate to claim they are obvious especially for topics that do not have a consensus. The other example is Line 172, stating that the naive fusion is not good enough obviously. However, according to the experimental results in Table 3 that compare DS-Net-S (naive) and DS-Net-S* (inter-scale alignment module), the proposed method outperforms the naive solution by 0.4% (81.9% → 82.3%) but at the cost of 16.7% additional computations (3.0 GFLOPs → 3.5 GFLOPs). Therefore, I wouldn’t think the naive fusion is not good enough obviously. By the way, there is an error when introducing DS-Net and DS-Net* in Line 251 -- “with” and “without” should be swapped.

3. The novelty is not strong. The idea of the dual-stream network is just to have one convolutional stream and one transformer stream. This is similar to dual-path network [a], which is missing in the related work. The idea of using different resolutions has also been explored, e.g. [b]. I wouldn’t think the novelty of this paper is particularly strong. In addition, after looking at Table 3, the improvements over the previous methods are also marginal.

In summary, I find the novelty and the performance are not that strong. Unfortunately, I wouldn’t be able to recommend acceptance for this paper. The authors should pay attention to the writing quality as well when revising this submission.

[a] Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S. and Feng, J., 2017. Dual path networks. In NeurIPS 2017.

[b] Wang, H., Kembhavi, A., Farhadi, A., Yuille, A.L. and Rastegari, M., 2019. Elastic: Improving cnns with dynamic scaling policies. In CVPR 2019

**EDIT: Post-rebuttal**

Thanks for preparing a detailed response to the initial comments. After reading the response, the concerns about the novelty and performance improvements still remain. Moreover, as mentioned in the authors' response, when the model size is larger, the improvement over other baselines also becomes smaller -- 83.1% using 46M for the proposed method v.s. 83.0% with 50M of Swin-S. The proposed method also has less throughput in general. Unfortunately, I will keep the previous rating.
limitations_and_societal_impact: The authors have adequately addressed them in the submission.
review_title: null

Meta-review: The reviews are split between accept and borderline reject recommendations. Two reviewers with positive ratings (7 and 8) appreciated that the hybrid CNN+Transformer approach is simple and intuitive. They also liked that the paper reports good empirical performance. The other two reviewers with negative ratings (5 and 5) raised concerns about limited novelty and insufficient/unconvincing empirical results. They criticized that the novelty isn't particularly strong because prior have already looked into the dual-stream approach combining different architectures as well as multi-resolution processing. They also shared a common concern that there is insufficient empirical evidence showing that the proposed approach is truly beneficial compared to simpler alternatives, especially at a large-scale regime, such as Swin Transformer (Swin-S) that achieves similar performance at a comparable parameter size (Swin-S: 83% w/ 50M params vs. this work: 83.1% w/ 46M params). Separate from novelty/experiment concerns, all four reviewers raised several writing issues.

This meta-reviewer carefully read the reviews, rebuttal, post-rebuttal discussion, and the paper in detail to fully understand the concerns raised by the reviewers. I agree with reviewers cxxV and LMod that the proposed idea of combining convolution and self-attention operations is interesting and novel --  it is a simple and effective, yet non-trivial combination than simply stacking CNN and Transformer side-by-side. Plus, as the reviewer cxxV pointed out, the paper reports impressive transfer results on ImageNet and COCO, the latter of which is particularly interesting because it is far from being solved (compared to ImageNet). I do however share the concerns with tvGe and ybP5, and including additional results as requested by the two reviewers will make the paper much stronger. The authors are urged to incorporate all the feedback provided by the reviews in their final version.
Decision: Accept (Poster)