Dual-stream Network for Visual Recognition
Authors: Mingyuan Mao
Gao Peng
Renrui Zhang
Honghui Zheng
Teli Ma
Yan Peng
Errui Ding
Baochang Zhang
Shumin Han
Venue: NeurIPS 2021 Poster (2021)
Keywords: Computer vision
dual-stream
transformer
convolution

Abstract: Transformers with remarkable global representation capacities achieve competitive results for visual tasks, but fail to consider high-level local pattern information in input images. In this paper, we present a generic Dual-stream Network  (DS-Net) to fully explore the representation capacity of local and global pattern features for image classification.  Our DS-Net can simultaneously  calculate fine-grained and integrated features and efficiently fuse them. Specifically,  we propose an Intra-scale Propagation module to process two different resolutions in each block and an Inter-Scale Alignment module to perform information interaction across features at dual scales. Besides, we also design a Dual-stream FPN (DS-FPN) to further enhance contextual information for downstream dense predictions. Without bells and whistles, the proposed DS-Net outperforms DeiT-Small by 2.4\% in terms of top-1 accuracy on ImageNet-1k and achieves state-of-the-art performance over other Vision Transformers and ResNets. For object detection and instance segmentation, DS-Net-Small respectively outperforms ResNet-50 by 6.4\% and 5.5 \% in terms of mAP on MSCOCO 2017, and surpasses the previous state-of-the-art scheme, which significantly demonstrates its potential to be a general backbone in vision tasks. The code will be released soon.


Dual-stream Network for Visual Recognition
Mingyuan Ma Peng Gao Renrui Zhang*t, Honghui Zheng', Teli Ma²,
Yan Peng', Errui Ding', Baochang Zhang",, ,Sumin Han
Beihang University, Beijing, China
Shanghai AI Laboratory, China
DDeaarmeent of Computer Vision Technology (VIS), Baidu Inc "Corresponding author, email: bezhang @ buaa. enua.edu.cn, hanshumin @ bbiidu.com 'quual contributions
Abstract
Transformers with remarkable global representation capacities achieve competitive results for visual tasks, but fail to consider high-level local pattern information input images. In this paper, we present generic Dual-stream Network (DS-Net) fully explore the representation capacity of local and global pattern features for image classification. Our DS-Net can simultaneously calculate fine-grained and integrated features and efficiently fuse them. Specifically, we propose an Intra-scale Propagation module to process two different resolutions in each block and an -Sale- Alignment module to perform information interaction across features at dual scales. Besides, also design Dual-stream FPN (DS-FPN) further enhance contextual information for downstream dense predictions. Without bells and whistles, the proposed DS-Net outperforms DeiT- Small by 2.4% in terms of top-1 accuracy on ImageNet- k and achieves state-of-the-arr performance over other Vision Transformers and ResNets. For object detection and instance segmentation, DS-Net DS-NNt-Small respectively outperforms ResNet-50 by 6.4% and 5.5 % in terms of mAP on MSCOCO 2017, and surpasses the previous state-of-the-art scheme, which significantly demonstrates its potential to be general backbone in vision tasks. The code will be released soon.
Introduction
In recent years, convolutional neural networks (CNNs) have dominated various vision tasks including but not limited to image recognition [22
18
38
39
19
41 20 52], object detection [37 26 36
24 [1 30. 21 []]] and segmentation [17
28 thanks their unprecedented representation capacity. However, limited receptive fields of convolutions inevitably neglect the global patterns images, which might be crucial during inference. For example, is more likely chair than elephant nearby a table. Such object-level information cannot be fully explored by local convolutions, that hampers further improvement of CNNs. Motivated by the success of Transformer architecture [K] in Natural Language Processing (NLP) 9 [I] and Multi-modality Fusion (29 40 12). rescarchers are trying apply Transformer vision tasks and have obtained promising results. In [10] 55 48 4
7 45 46
53 many novel architectures and optimizing strategies are proposed and achieve comparable even better performance than CNNs, emphasizing the significance of extracting global features in vision tasks.
27
3 [44]
Inspired by the verified efficacy of CNN and Transformer, many concurrent works, such Cont- Net [47] a CvT [46), attempt to introduce convolutions to vision transformers in different manners, hoping combine both advantages. In CvT [46), linear projection of self-attention block in the Transformer module are replaced with convolutional projection. In contrast, ContNet 47 performs
35th Conference on Neural Information Processing Systems (NcurIPS 2021)

---

convolution on different token maps to build mutual connections, which is similar to Swin Trans-
former 27
A ttention maps
Feature maps
DelT
DS-Net
DeiT
DS-Net (cony)
DS-Net (coas +alteation)
Input Image
(a)
(b)
(6)
(d)
(n)
Figure I: Comparison of attention maps and feature maps of DeiT and our proposed DS-Net, both of which obtained from the last block. (a) and (b) are attention weights of the blue points in the images. To achieve clearer illustration, the figures are acquired by overlaying heatmaps to the input images.
However, some drawbacks of existing models still remain to be solved. Firstly, works mentioned above either perform convolution and attention mechanisms sequentially, or just replace linear projection with convolutional projection in attention mechanisms, which may not be the most ideal design. Additionally, the conflicting properties of such two operations, convolution for local patterns but attention for global patterns, might cause ambiguity during training, which prevents their merits from merging to the maximum extent. Furthermore, self-attention can capture long-range information via the built-in all-pair interaction theory, but it quite possible that attentions might be confused and disturbed by neighboring details in high resolution feature maps, fail to build up object-level global patterns(see Fig. a)). Finally, the computation cost of self-attention is unaffordable due to the quadratic computation complexity of sequence length. Although PVT
45 and APNB [56] downsample the key-query features to improve the efficiency of self-attentic operator, they both abandon fine-graind local details of the image, which greatly impairs their performance.
In this paper, address the these issues by introducing a Dual-stream Network (DS-Ner). Instead of single stream architecture as previous works, our DS-Net adopts Dual-stream Blocks (DS- (DS-Blocks), which generates two feature maps with different resolutions, and retains both local and global information of the image via two parallel branches. We propose a Intra-scale Propagation module here to process two feature maps. Specifically, high-resolution features are used to extract fine-graid ane-grained local patterns with depth-wise convolution, while low-resolution features are expected to summarize long-range global patterns. Considering low-resolution features themselves contain more integrated information, it would be much more easier for self-attention mechanism to capture object-level patterns rather than overwhelmed by trivial details (see Fig. [b)). Such dual-stream architecture disentangles local and global representations, which helps maximize both their merits, and thus generates better representations compared to DeiT baseline (see (c) and (e) Fig. [.. Besides, low-resolution feature maps for self-attention dramatically reduce the memory cost and computation complexity, After parallelly processing dual streams, we present Inter-scale Alignment module based co-attention mechanism at the end of DS-Blocks. This is because local details and global patterns capture different perspectives of the image, which are misaligned not only in pixel positions but also in semantics. Hence, this module is designed for modeling complex cross-scale relations and adaptively fuse local and global patterns.
Besides, we apply our DS-Blocks to Feature Pyramid Networks for further feature refinement, named DS-FPN. In this way, multi-level features are capable extracting contextual information from both local and global views, improving performance of downstream tasks. This demonstrates that our Dual Stream design could be utilized as a plug-in building block not only for image recognition, but for many other vision tasks.
The contributions of this work are concluded as follows:
1. We present novel Dual-Stream Network, named DS-Net, which retains both local and global features in DS-Block. The independent propagation maximize the advantages of convolution and self self-attention, thus eliminating the conflicts during training.
2. We propose Intra-scale Propagation and Inter-Scale Alignment mechanism to achieve effec- tive information flows within and between features of different resolutions, thus generating better representations of the image.
2

---

3. We introduce Dual-stream Feature Pyramid Network (DS-FPN) to enhance contextual information for downstream dense tasks and achieves better performance with little extra costs.
4. Without bells and whistles, the proposed DS-Net outperforms DeiT baseline by significant margins in terms of top-1 accuracy on ImageNet- Ik and achieves state-of-the-ar perfor- mance over other Vision Transformers and CNN-based networks on image classification and downstream tasks, including object detection and instance segmentation.
2 Related work
Vision Transformers. Motivated by the great success of Transformer in natural language processing, researchers are trying apply Transformer architecture to Computer Vision tasks. Unlike mainstream CNN-based models, Transformer is capable of capturing aftention module and provides the paradigm without image-specific inductive bias. ViT [10 views 16 x 16 image patches as token sequence and predicts classification via a unique class token, which shows promising results. Subsequently, many works, such as DeiT 42 and PVT [55 achieve further improvement on ViT, making it more efficient and applicable in downstream tasks. ResT [50] proposed a efficient transformer model. Besides, models based on Transformer also give leading performance in many other vision tasks such as object tracking and video analysis.
visual relations by its self-
Local and Global Features in Transformers. Despite the Transformer's superiority upon extracting global representations, image-leved self-attention is unable to capture fine- fine-graing -rained details. To tackle this issue, previous works assert obtain the local information at the same time and then fuse features of the two scales. TNT [66 proposes intra-patch self-attention block to model local structures and aggregates them with global patch embeddings. Swin Transformer (7 extracts local features within each partitioned window and fuses them by self-attention in successive shifted windows. Similarly, shifted windows are replaced by convolutions in ContNet [47]. Following such design of local windows, Twins [6] uses inter-window attention to aggregate global features. Swin Transformer and Twins both give advanced performance, but still rely local-window self-attention to capture fine-level features, which works intuitively worse than CNN. Besides, Swin's 's shifted windows are complex for device optimization and Twins 6 recurrently local and global attentions would constrain each other's 's representation learning.
Equip Transformers with Convolution. Compared self-attention mechanism, CNN has its unique advantages of local modeling and translation invariance. Therefore, some existing transformers explore the hybrid architecture incorporate both merits for better visual representation. T2T [48] progressively aggregate neighboring tokens to one token to capture local structure information, similar to convolution operation. Followingly, CvT (46) designs Convolutional Token Embedding and Convolutional Transformer Block for capturing more precise local spatial context. CPVT 77] utilizes one convolutional layer dynamically generate positional encodings, adaptive for varying image scales. Conformer 33 combines transformer with an independent CNN network for specializing local features and flows information via lateral connections. Nonetheless, none of the aforementioned models maximizes the potential of the hybrid architecture. They either use CNN for marginal operations to boost Transformer, e.g., tokens generation in CvT [46] and positional encodings in CPVT 77. or view CNN as a relatively separate branch, like Conformer.
Different from previous works, in proposed DS-Blocks, treat self-attention and convolution as dual resolution processing paths via Intra-scale Propagation module, where the former aims to extract local fine- ine-grained details, and the latter focus on exploring features from a global view. On top of that, their information is reasonably fused by Inter-scale Alignment module based on co-altention, which addresses the issue of feature misalignmennt By this design, the local and global patterns could be simultaneously extracted and the capability of CNN is fully exerted in Transformer architecture.
3 Method
3.1
Dual-scale Representations
The overall pipeline of our proposed DS-Net is shown in Fig. 2 Motivated by stage-wise design in previous networks like ResNet [18), we set 4 stages in the architecture, whose down-sampling factors
3

---

Stage
Stage
Stage 3
Stage BlockoK,
Blockok;
Blockok,
Blockok,
ResNet
7
DS Block
DS illcc
DS-Net
DS-Block
Intra-scale Propagation
Inter-scale A lignment
Co atention
Figure 2: Illustration of the proposed DS-Net, including Intra-scale Propagation module and Inter- scale Alignment module. Compared to ResNet, in which only single resolution is processed, our DS-Net, instead, generates dual-stream representations via DS-Blocks.
4, 8, 16, 32, respectively. Within each stage, Dual-stream Blocks (DS-Blocks) adopted to generate and combine dual-scale representations.
Our key idea is to keep local features in relatively high resolution to reserve local details, while represent global features with a lower resolution (1/32 of the image size) to retain global patterns. Specifically, in each DS-block, we split the input feature map into two parts at the channel dimension. One is for extracting local features, denoted as St, the other is for summarizing global features, denoted as g. Notably, we keep the size of unchanged in all stages across the network by down- sampling it with proper factor, W x H C. where W and H represents the width and height of the input image, and C, represents the channel number of input features in current stage. Aided by the high resolution of fi. the local patterns could be much reserved for subsequent extraction, but thanks to the low resolution of fg the exploration of non-local and object-level information is greatly benefited.
3.2 Intra-scale Propagation
Local and global features are representations of one image from two totally different views. The former focuses on fine-grained details, essential for tiny-object detection and pixel-level localization, while the latter aims at modeling object-level relations between long-range parts. Therefore, given dual-scale features fi and fg. perform Intra-scale Propagation module parallelly process them.
Local representation. For high- high-resolution with the size of W, H, Ct where C/ equals C. we perform 3x3 depth-wise convolution to extract local features as follows and obtain fz:
2.
M,N
fz(i.))
Σ W(m, n) @ fi(i m,j+m
where W(m, n), (m, n) E 0, represents the convolution filters, W(m, n) and fi( are both G. dimensional vectors, @ denotes element-wise product. By the power of depth-wise convolution, f. is able to contain fine-grained local details of the input image.
Global representation. For low-resolution representation fa with the fixed size of 32
x C as
illustrated in Section 3.1 we first flatten fy to sequence with the length of lg. the product of 32 and
W

---

H. in which each element is a C. dimensional vector. By doing this. each vector in the sequence 32 treated as visual token without spatial information. Here, the dependencies between different token pairs are unrelated with their spatial positions in the feature map. which is totally different with the convolution. Then, we summarize the global information and model the object-level coupling relation via self-attention mechanism:
2
fQ= 1e-1yWa Wg - =V Jgw,
(2)
where Wq. WK. Wy denote matrixes to generate queries, keys and values respectively. By calculating the similarity between fo and fv. obtain attention weights for aggregating information from different locations of fg. Finally, we calculate the weighted sum of attention weights and fv, thus obtaining integrated features:
folk
Sa = softmax
1v.
Vd
where d equals *, N denotes the number of attention head that we set to 1, 2, 5, 8 for stages respectively in DS-Net.
Our dual-stream architecture disentangles fine finennrained and integrated features in two pathways. which significantly eliminates the ambiguity during training. Additionally, the Intra-scale Propagation module processes feature maps with dual resolutions by two domain-specificeeeccc effective mechanisms separately, extracting local and global features to the maximum extent.
3.3 Inter-scale Alignment
delicate fusion of dual-scale representations is vital for the success of DS-Net, since they capture two different perspectives of one image. To address this, a naive idea is to upsample the low-resolution representation using bilinear interpolation, and then fuse the dual-scale representations via by I convolution after simply concatenating them position-wisely. Such naive fusion is not convincing enough. Furthermore, by visualizing the feature maps of dual-scale features, we observe that global features in low resolutions and local features in high-resolutions are actually misaligned (see (b) and (d) in Fig. [ Therefore, considering the relation of two representations are not explicitly explored, it is unpersuasive to pre-define fixed strategy to fuse them, such as concatenation, element-wise addition or production. Enlightened by 13 32 we propose a novel co-attention- based Inter- scale Alignment module, whose scheme of Inter-scale Alignment module is shown in Fig. 2 This module aims to capture the mutual correlations between each local-global token pair, and propagate information bidirectionally in Icarnable and dynamic manner. Such mechanism prompts local features to adaptively explore their relations with global information, enabling themselves to be more representative and informative, and vice versa.
Given extracted local features fr. with the size of W; X H; x C. and global features fo with the size Ci, of lg 7 we first flatten fr to sequence with the length of 4, the product of W; and H,, in which each element C. dimensional vector. fr and fo are now the same format but with different length. Then we perform co- co-attention on two sequences as follows:
2
QL fL Qa foWa.
K2 fLWk,
V. W/
(4)
KG faWg,
VG foWe,
where the sizes of W are all C. dim, and the dim is transformed features of local global representations. Then we calculate the similarities between every pair of fú. and fA to obtain the corresponding attention weights:
Thus we have the
2
OLKT ) Vd
WI.-G = softmax QGKT Vd
WG-1 softmax(
(S)
The size of Wa-,. and Wr.-GG are 4, x ly and I, x 4 respectively. The non-linear function softmax is performed at the last dimension. WG-1. reflects the importance of different tokens in global features to the local tokens. Likewise, global features can also extract useful information from local features via W W. Instead of fixed fusion strategy that might bring constraints, what and how the
5

---

information transfers are automatically determined by features themselves here. We can then obtain hybrid features as:
h = Wa-IVe ha WL-GVV
where the size of , and big are 1 x dim and l x dim respectively. Then we add x convolution layer after hybrid features to further fuse the channels and reshape them to W; x He x C. and W
x Ci
32
Such bidirectional information flow is able identify cross-scale relations between local and global tokens, by which dual-scale features are highly aligned and coupled with each other. After this, we could safely upsample low-resolution representation hgg concatenate it with high high-resolution h and perform by convolution for channel-wise dual-scale information fusion. At the end of the last block in stage 4, we add fully-connected layer the classifier conduct classification.
3.4 Dual-stream Feature Pyramid Networks
Introducing contextual information into Feature Pyramid Networks(FPN) 23 has been explored by [49, 31]. However, previous methods often cause large extra memory and computation costs, due to their complicated architectures and utilized high resolution feature maps. Besides, Non-local contexts would miss local details, which disastrous for tiny object detection and segmentation. Here, apply Dual-stream design into FPN, named Dual-stream Feature Pyramid Networks(DS-FPP) by simply adding DS-Blocks to every feature pyramid scale. In this way, DS-FPN able to better attain non-local patterns and local details with marginal increased costs all scales, which further enhance the performance of subsequent object detection and segmentation heads. This shows our DS-Net not only serve backbone but also general plug-in building block in many other vision architectures.
Similar to FPN 23 we take image features from various scales from the backbone as input, and output corresponding refined feature maps of fixed channels number by a top-down aggregation methods. Our structure is composed of bottom-up pathways, Dual-stream lateral connections, and top-down pathways. The bottom-up and top-down pathways follow the design of FPN, but lateral connection here adopts DS-BU DS-Block to process features in dual scales via Intra-scale Propagation and Inter-scale Alignment. See Fig.
B
Di-BBakk
1xt Dxd
Eledi
C3
1x0
44.
C2
DS Blodd
4Fg
412
1c
Bledi
412
(a) FPN
DS-FPN
Figure 3: The architecture of DS-FPN. C, denotes the feature maps in stages from backbone, and denotes the reconstructed features for detection and segmentation.
Experiments
In this section, we first provide three ablation studies to explore the optimal structure of DS-Net and interpret the necessity of dual-stream design. Then we give the experimental results of image classification and downstream tasks including object detection and instance segmentation. Specifically, We use ImageNet- 1K [8] for classification and MSCOCO 2017 25 for object detection and instance segmentation. All experiments are conducted on 8 V100 GPUs and the throughput is tested on V100 GPU. As the number of blocks we set different stages in the architecture (see Fig. 2 is flexible, we conduct experiments on 3 models with different parameter size, denoted as DS-Net-T (Tiny), DS-Net-S (Small) and DS-Net-B (Big), which have comparable parameters with ResNet- ResNet-50 and ResNet- 101 respectively. The detailed setup is shown in Table [

---

Table 1: Detailed settings of DS-Net. Deonv denotes 3 x 3 depth-wise convolution, and MHSA
denotes multi-head self-attention. C; denotes the number of channels in ith stage. The feature dimension expansion ratio of each block is set to 4.
Stage
Input size
DS-Net-T
DS-Net-S
DS-Net-B
Stage
224x224
4x4, 64, stride=4, padding=0
Dconv MHSA
Dconv
Dconv
Stage
56x56
x 2
MHSA
x x3
MHSA 8
x
C1= 64
C1 64
C1 = 64
Dconv MHSA 8
Dconv
Dconv
Stage 2
28x28
MHSA
x4
MHSA 8
x4
C2 = 128
C2 128
C2 128
Doonv MHSA 8
Doonv
Deonv
Stage 3
14x14
x4
MHSA 8
x
MHSA 8
x 28
C3 = 320
C3 = 320
C3 = 320
Dconv
Dcony
Dconv
Stage 4
7x7
MHSA 8
x
MHSA 8
x
MHSA 8
x
C4 512
C4 512
C4 512
7x7
global average pooling, 1000-d fc, softmax
Table 2: DS-Net-T performance on ImageNet- Ik validation set with different a.
0.25
0.5
0.75
Top-
77.1
78.0
78.1
77.9
77.6
Top-5(%)
93.3
94.1
94.1
94.0
93.9
Params (M)
8.6
8.7
9.1
9.8
10.7
FLOPs (G)
1.573
1.578
1.592
1,615
1.647
Throughput (Images/s)
3240
1733
1199
912
740
4.1 Ablation Study
4.1.1 Ratio of Local to Global Features
Settings. As described Section 3.1 we split the features into two parts in channel dimension as local and global representations, which are later processed by convolution and self-attention, respectively. We conduct an experiment explore a appropriate partition ratio. We the proportion of channel number of fg (global features) to the total channel number. When equals only depth-win wwse convolution is performed. and when a equals only self-attention is performed. Noting that in this experiment, we choose DS-Net-T as the testing model, which simply concatenates fr and fA at the end of each block, without Inter-scale Alignment module.
denote
The results in Table 2 indicates that when a equals 0.5, DS-Net-T achieves the highest accuracy ImageNet-1k validation set, which, some extent, implies that local and global informa- tion play equally important roles for visual representations. Thus, in the following experiments, we equally split features into two parts the channel dimension obtain dual-scale representations.
Results.
4.1.2 None is Dispensable DS-Block
Settings. Here, illustrate that every component within DS-Block plays vital role in the prediction, including either pathway of Intra-scale Propagation and bidirectional co-attention of Inter-scale Alignment. In Table we respectively remove one of the four modules in every block of DS-Net-T NDS-Net" represents the corresponding DS-Ne version with Inter-scale Alignment module), while remain the others the same. Therein, w/o Sr. denotes that fi split at the beginning is directly fed to Inter-scale Alignment Module without exploring local features, and w/o G I. denotes that co-attention from global features to local features not implemented, analogous w fo and w/o L + G. The version w/o L ++ G removes the entire co-attention, which equals DS-Net-T.

---

Table 3: Ablations of removing components of DS-Net-T" ImageNet-IK validation set.
w/o
w/o
w/o
Versions
DS-Net-T
w/o fL w/o fG
G-b
L L- G
G
Top-1(%)
79.0
76.7
76.6
76.5
76.4
78.1
Top-5(%)
94.8
93.6
93.7
93.5
93.4
94.1
Comparison with the accuracy of other state-of-art methods on ImageNet- validation set, The input images are reshape 224 x 224 resolution. DS-Net" represents the corresponding DS-Net version with Inter-scale Alignment module.
Table 4:
Method
Params (M)
FLOPs (G)
Throughput (Images/s)
Top-1 (%)
ConvNet
ResNet-18 18
11.8
2
69.9
ResNet-50 [8
25.6
4.1
74.2
ResNet- 101
[88
44.5
7.8
77.4
RegNetY-8GH
35
39.2
79.9
RegNetY-16GF 35
83.6
15.9
80.4
Transformer / Hybrid
DeiT-T 42
2536
72.2
CPVT-Ti
72.4
T2T-ViT-12 48
6.9
76.5
ConTNet-S [7
10.1
1.5
76.5
DS-Net-T (ours)
9.1
1.6
1199
78.1
DS-Net-T" (ours)
10.5
1.8
1034
79.0 (+6.8)
DeiT-S 42
22.1
4.6
940
79.9
CrossViT-15 4
27.4
5.8
640
81.5
T2T-ViT-14
5.2
81.5
ConTNet-MM 47
19.2
3.1
80.2
TNT-S [16
23.8
5.2
81.3
CvT--3 [46
20
4.5
81.6
PVT-Small 45
24.5
3,8
820
79.8
CPVT-Small-GAP
23
4.6
817
81.5
Swin-T 27
29
4.5
766
81.3
DS-Net-S (ours)
19.7
3
582
81.9
DS-Net-S* (ours)
3.5
510
82.3 (+2.4)
DeîT-B 42
86
17.5
292
81.8
Cross ViT- 18
43.3
430
82.5
ConTNet-B 47
39.6
6.4
81.8
PVT-L 45
61.4
9.8
81.7
Swin-S
27
50
8.7
437
83.0
DS-Net-B (ours)
48.8
7.6
387
82.8
DS-Net-B* (ours)
49.3
8.4
335
83.1 (+1.3)
Results. As shown in Table 3 any absence of the components in DS-Block deteriorates the perfor-
mance. Compared to extracting local and global features simultaneously, the model's representation capacity is constrained with either fL fG. Surprisingly, the unilateral co-ate co-attention of w/o G L and w/ o + L performs worse than removing any fusion strategy, because implementing G would confuse the original local features ft., and only by L, + G could the missing local features be complemented. This further demonstrates the importance of the designed Inter-scale Alignment module.

---

Table 5: Comparison with the performance for object detection and instance segmentation on the MSCOCO minival set of other state-of-art methods.
Backbone
Params(M)
Ap
AP3
AP
AP
AP
APS
APM
APi.
RetinaNet
24
ResNet-50 18
37.7
36.3
19.3
40.0
48.8
Swin-T 27
38.5
41.5
25.1
44.9
55.5
DS-Net-S* (ours)
33.2
42.7
26.8
46.3
56.7
Mask R-CNN [17]
ResNet-50 18
44.2
38.2
21.9
40.9
49.5
34.7
18.3
37.4
47.2
Swin-T 27
48
43.7
28.5
47.0
57.3
39.8
24.2
43.1
54.6
DS-Net-S* (ours)
43.2
44.3
28.3
47.7
58.8
40.2
24.0
43.4
54.7
4.2 Image Classification
Settings. Image classification experiments are performed on ImageNet-Ik (8, comprising 1.28M training images and 50K validation images of 1000 classes. For fair comparison with other works, we follow training settings DeiT. We train our model for 300 epochs by Adam W optimizer. The initial learning rate is set to le-3 and scheduled by the cosine strategy.
Results. We report the results Table 4 where DS-Net and DS-Net" denote our proposed networks with and without Inter-scale Alignment module respectively. For DS-Net, only concatenation operation is used fuse the local and global features. DS-Net" the version with Inter-scale Alignment module. With comparable parameters and complexity, our proposed DS-Net consistently outperforms the DeiT [42] baseline by significant margins, which is, specifically, 6.8% improvement for tiny model, 2.4% improvement for small model and 1.3% for big model with only half parameters.
The efficacy Inter-scale Alignment module is also clearly indicated Table 4 which gains further 0.9% 0.4% and 0.3% improvement compared to the simple version of DS-Net-T, T, DS-Net- and DS-Net-B. Notably, without bells and whistles, DS-Net-T" DS-Net-S* also outperform other state-of-the-art methods, including CNN, vision transformer and some hybrid architectures.
4.3 Object Detection and instance segmentation
Settings. Experiments in this part are conducted on the challenging MSCOCO 2017 25 containing 1 118K training images and 5K validation images. Specifically, the backbone is initialized by pretraind weights from ImageNet- IK, and other layers adopt Xavier. We evaluate our DS-Net on typical detectors:
RetinaNet and Mask R-CNN [17). As the standard x schedule(12 epochs), we adopt AdamW optimizer with initial learning rate of le-4, decayed by 0.1 at epoch 8 and 11. We set stochastic drop path regularization of 0.1 and weight decay of 0.05. Following previous methods, training and testing images are resized to 800x 1333.
Results. The results are shown in Table For object detection, DS-Net-S" achieves significant improvement compared ResNet-50 backbone [24] by 6.4% with RetinaNet and 6. % with Mask R-CNN in terms of A pbboz and outperforms the previous state-of-the-art Swin-T [27) by 1.2% and 0.6%. For instance segmentation, DS-Net-S* achieves 40.2% in terms of A Aseegm which surpasses the Re and Swin-T by 5.5% and 0.4%.
4.4 Dual-stream Feature Pyramid Networks
Settings. Experiments of DS-FPN are implemented DS-Net-T and Swin Transformer [27 for object detection and instance segmentation on MSCOCO 2017 (25. by replacing the original FPN with DS-FPN. The training setting and scheme the same those mentioned above, which is 1 schedule(12 epochs) and Adam optimizer with initial learning rate of le-4. Likewise, the backbone initialized with pretrained weights, and DS-FPN trained from scratch with newly added heads.
A Ablaioon Study. In Table 6 we report results of several different designs combining DS-Blocks with FPN on RetinaNet, with DS-Net-T backbone. Four inserted positions of DS- Blocks have been

---

Table 6: Performance of DS Si-BBlcks'' different inserted positions FPN of RetinaNet.
Inserted Position
None
Last
Lateral
Lateral
Lateralentra
A piboz
39.0
40.2
40.3
40.2
40.4
Params (M)
18.83
19.59
23.72
21.36
24.43
Flops (G)
184.02
194.27
197.22
195.27
199.92
Throughput (imgs/s)
19.4
16.2
15.1
15.9
14.6
Table 7: Comparison on the performance of DS-FPN and FPN for object detection and instance segmentation on MSCOCO 2017.
Backbone
Neck
APb
AP
AP
APL
AP
AP3
APM
APL
RetinaNet
(4
FPN DS-FPN
39.0
23.3
42.6
51.3
DS-Net-T
40.4
26.1
44.2
52.3
FPN
41.5
26.4
45.1
55.7
Swin-T 27
DS-FPN
42.3
27.7
45.9
55.0
Mask R-CNN
17
FPN DS-FPN
40.1
24.9
43.4
52.1
37.2
21.4
40.4
50.0
DS-Net-T
41.3
25.8
44.6
53.0
37.9
22.1
41.1
50.4
FPN
42.5
26.3
45.9
56.1
39.2
22.8
42.4
53.8
Swin-T 27
DS-FPN
44.2
27.6
47.6
58.0
40.6
23.9
43.9
55.2
experimented, where "Last" means replacing the last 3x3 convolution layer, and "Lateral" equips them lateral connections. More specifically, "LLateral moves the inner channel-alignment 1 convolution layer from the last place to the first, and "Lateral adds DS-Blocks also on extra feature outputs. The results show that "Lateraleætra" has the best performance. Therefore, "Lateralestra" as our default, which is 1.4% higher than FPN with marginal memory costs.
Comparison with FPN.As reported in Table For object detection, DS-Net-T and Swin Transformer with DS-FPN gain 1.4% and 0.8% mAP improvement, respectively. For instance segmentation, DS- FPN helps to gain 0.7% and 1.4% A.segm improvement for DS-Net-T and Swin Transformer, respectively. The results significantly prove the generality and effectiveness our Dual-stream design.
5 Conclusion
In this paper, we propose generic Dual-stream Network, called DS-Net, which disentangles local and global features by generating two representations with different resolutions. present Intra- scale Propagation module and Inter-scale Alignment module combine the merits of convolution and self-attention mechanisms, and identify cross-scale relations between local and global tokens. Besides, for downstream tasks, design Dual-stream Feature Pyramid Network (DS-FPN) to introduce contextual information to feature pyramid for further refinement with marginal costs. With the outstanding performance image classification and dense downstream tasks including object detection and instance segmentation, our proposed DS-Net has shown its promising potential in vision tasks.
Disclosure of Funding
This work was supported by the National Natural Science Foundation of China under Grant 62076016 and the Shanghai Committee of Science and Technology, China (Grant No. 21DZI 100100).
10

---

References
[1] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan,
Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. arXiv preprint
20tv:20005.1415,, 2020.
[2] Z. Cai and N. Vasconcelos. Cascade r-cmn: Delving into high quality object detection. In IEEE
CVPR, pages 1154-6162, 2018.
[3] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. WrXv:2205..12872 2020.
[4] C.-F. Chen, Fan, and R. Panda. Crossvit: Cross-attention multi-scale vision transformer for image classification. arXiv:2103. 14829, 2021.
[5] Chollet. Xception: Deep learning with depthwise separable convolutions. arXiv:1610.02357,
2017.
[6] X. Chu, Z. Tian, Y. Wang. B. Zhang, H. Ren, and X. W. et al. Twins: Revisiting the design of spatial attention in vision transformers. VrXiv:2103.0011 2021.
[7] X. Chu, 7. Tian, B. Zhang. X. Wang, X. Wei, and H. Xia. Conditional positional encodings for vision transformers. IEEE CVPR, 2021.
[8] J. Deng, W. Dong, R. Socher, L..J. Li, K. Li, and Fei-Fei. Imagenet: A large-scale hierarchical image database. In JEEE CVPR, pages 248-255, 2009.
[9] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional
transformers for language understanding. arXiv preprint arXiv: 180.04408, 2018.
[10] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, and T. U. et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICIRR, 2021.
[11] K. Duan, S. Bai, L. Xie, H. Qi, Q. Huang, and Q. Tian. Centernet:
Object detection with
keypoint triplets. IEEE CVPR, 2019.
[12] P. Gao, S. Geng, R. Zhang, T. Ma, R. Fang. Y. Zhang, H. Li, and Y. Qiao. Clip-adapter: Better vision-language models with feature adapters. arXiv preprint arXiv:21 10.04544, 2021.
[13] Gao, Jiang, H. You, Lu, S. C. H. Hoi, X. Wang, and H. Li. Dynamic fusion with intra- and inter-modality attention flow for visual question answering. In IEEE CVPR, pages 6632-6641, 2019.
[14] P. Gao, J. Lu, H. Li, R. Mottaghi, and A. Kembhavi. Container: Context aggregation network. arXiv preprini
(0ti::2006.01000)) 2021.
[15] Gao, M. Zheng, X. Wang, Dai, and H. Li. Fast convergence of detr with spatially modulated 2021.
co-attention. arXiv preprint
[16] K. Han, A. Xiao, E. Wu, Guo, C. Xu, and Y. Wang. Transformer in transformer. arXiv:210..01112. 2021.
[17] K. He, G. Gkioxari, P. Dollár, and R. B. Girshick. Mask R-CNN. In IEEE ICCV, pages 2980-2988, 2017.
[18] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In IEEE CVPR, pages 770-778, 2016.
[19] A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and
H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications.
arXiv:1704. 04861, 2017.
[20] J. Hu, L. Shen, and G. Sun. Squeeze-and-exciiation networks. In IEEE CVPR), 2018.
[21] H. Law and J. Deng. Cornemet: Detecting objects as paired keypoints. In ECCV, pages 765-781, 2018
[22] Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied document recognition. Proceedings of the IEEE, 86(11):2278-232) 1998.
[23] T. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie. Feature pyramid networks for object detection. In IEEE CVPR, pages 936-944, 2017.
[24] Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In 2017.
IEEE ICCV, pages

---

[25] T.-Y. Lin, M. Maire, S. J. Belongie, L. D. Bourdev, R. B. Girshick, J. Hays, P. Perona, D. Ra- manan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, pages 740-755, 2014.
[26] W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. E. Reed, C. Fu, and A. C. Berg. SSD: single shot multibox detector. In ECCV, pages 21 21-37, 2016.
[27] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, and Z. Z. et al. Swin transformer: Hierarchical vision transformer using shifted windows. arXiv:2 /403/ 2021.
[28] Long, E. Shelhamer, and T. Darrell. Image segmentation fully convolutional networks for semantic segmentation. In IEEE CVPR, 2015.
[29] Lu, D. Batra, D. Parikh, and S. Lee. Vilbert: Pretraining task agskstin visiolinguistic representations for vision-and aaan---nndlllaauagae tasks. arXiv preprint arXiv:1908.0223 2019.
[30] M. Mao, Y. Tian, B. Zhang. Q. Ye, W. Liu, G. Guo, and D. Doermann, iffDetector: Inference- aware feature filtering for object detection. In arXiv preprint arXiv:2006. 12708, 2020.
[31] J. Pang, K. Chen, J. Shi, H. Feng, W. Ouyang, and D. Lin. Libra r-cnn: Towards balanced learning for object detection. In IEEE CVPR, 2019.
[32] G. Peng, H. You, Z. Zhang, X. Wang, and H. Li. Multi-modality latent interaction network for visual question answering. In IEEE ICCV, pages 5824-5834, 2019.
[33] Z. Peng, W. Huang, S. Gu, L. Xie, Y. Wang. J. Jiao, and Q. Ye. Conformer: Local features coupling global representations for visual recognition. 1xXiv:2105.03800 2021.
[34] A. Radford, K. Narasimhan, T. Salimans, and Sutskever. Improving language understanding by generative pre- pre-training. 2018.
[35] Radosavovic, R. Kosaraju, R. Girshick, K. He, and P. Dollár. Designing network design spaces. In IEEE CVPR, 2020.
[36] Redmon, S. K. Divvala, R. B. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In JEEE CVPR, pages 779-788, 2016.
[37] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster R-CNN: towards real-time object detection with region proposal networks. In NeuriPS, pages 91-99, 2015.
[38] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv:1409, 1556, 2015.
[39] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In IEEE CVPR, pages 1-9, 2015.
[40] H. Tan and M. Bansal. Lxmert:
Learning cross-modality encoder representations from trans- formers. arXiv preprint arXiv: 1908. 2019.
[41] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[42] H. Touvron, M. Cord, M. Douze, Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers and distillation through attention. arXiv:2012. 12877, 2020.
[43] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, and A. N. G. et al. Attention is all you need. In NeurlPS, 2017.
[44] H. Wang, Y. Zhu, H. Adam, A. Yuille, and L.-C. Chen. MaX-DeepLab: End-to-end panoptic segmentation with mask transformers. In JEEE CVPR, 2021.
[45] W. Wang, Xie, X. Li, D. Fan, K. Song, and D. L. et al. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. orXiv:2102.12122 2021.
[46] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, and L. Y. ct al. Cyt: Introducing convolutions to
vision transformers. arXiv:2103: 15808, 2021.
[47] H. Yan, Z. Li, W. Li, C. Wang. M. Wu, and C. Zhang. Contnet: Why not use convolution and transformer at the same time? arXiv:2104 /3497, 2021.
[48] I.. Yuan, Y. Chen, T. Wang, W. Yu, Y. Shi, and 7. J. al. Tokens-to-token vit: Training vision transformers from scratch on imagenet. arXiv:2101. 1986, 2021.
[49] D. Zhang, H. Zhang, J. Tang, M. Wang, X. Hua, and Q. Sun. Feature pyramid transformer. In ECCV, 2020.
12

---

[50] Q. Zhang and Y. Yang. Rest: An efficient transformer for visual recognition. arXiv preprint MiXiv:2005.13677 2021.
[51] X. Zhang, F. Wan, C. Liu, R. Ji, and Q. Ye. FreeAnchor: Learning match anchors for visual object detection. In NeurlPS, 2019.
[52] X. Zhang, X. Zhou, M. Lin, and J. Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In JEEE CVPR, pages 6848 6856, 2018.
[53] Z. Zhao, K. Samel, B. Chen, and L. Song. Proto: Program -uidedd transformer for program-
guided tasks. arXiv preprinl urXiv:21 10.008044 2021.
[54] S. Zheng, Lu, H. Zhao, X. Zhu, Z. Luo, and Y. W. et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers. In JEEE CVPR, 2021.
[55] D. Zhou, B. Kang, X. Jin, L. Yang, X. Lian, and Z. et al. Deepvit: Towards deeper vision transformer. In IEEE CVPR, 2021.
[56] Z. Zhu, M. Xu, S. Bai, T. Huang, and X. Bai. Asymmetric non-local neural networks for semantic segmentation. In IEEE ICCV, pages 593-602, 2019.
13


Reviews:
[{"review_id": "eEPB1D74CgU", "reviewer": "Reviewer_cxxV", "summary": "The authors propose a new architecture for visual recognition. Like standard convolutional architectures, the representation is structured in a multi-scale hierarchy, with deeper layers being coarser and more abstract. At every level of the hierarchy, a coarse, global representation is appended and simultaneously processed with the more fine-grained, local one. Local features are processed with depth-wise convolutions, and global ones with multi-head self-attention. In addition to this separate processing of local and global features, they are co-processed by cross-attending between local and global features. \n\nIn addition to this recognition backbone, the authors adapt Feature Pyramid Networks in a similar spirit. Whereas FPN's use dense convolutions to fuse an intermediate representation with a coarser one, the authors instead use the proposed \"Dual-Stream\" block, combining local depth-wise convolutions, global self-attention, and local/global cross-attention. \n\nThe authors evaluate the recognition backbone for classification on ImageNet and detection and instance segmentation on COCO, reporting large gains over standard convolutional architectures such as ResNet, and reasonable gains over more recent transformer-based ones.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors propose a new hybrid architecture which advances the state of the art in visual recognition. Rather than seeking to entirely replace the convolutional architecture with attentional ones, as prior work has, the authors introduce a careful combination of convolutional and attention-based operations. In particular, the main obstacle for introducing attention in visual architectures is the quadratic scaling of attention operations with respect to the number of tokens. The authors overcome this limitation by introducing low-resolution latents at every stage, and attending within or across these latents. Nevertheless, the architecture is simple and general enough to be used as a drop-in replacement to other visual backbones (such as ResNet), indicating the generality of the approach. The gains afforded by their FPN variant (which incorporates the same design principle) also support this generality. Together, these qualities make this a solid contribution. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors propose a new architecture for visual recognition and evaluate it on standard benchmarks. As such the work does not raise any new ethical concerns.  ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "bnRVSK6kyE3", "reviewer": "Reviewer_ybP5", "summary": "This paper presented a Dual-stream Network for image classification. It combines the representation of local and global pattern features by using self-attention and convolution together. It also proposed an Intrascale Propagation module to process two different resolutions in each block.  In addition, it also introduced such block to FPN to demonstrate the benefit.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is easy to read. The methodology and its novelty are clear. However, the setup of experiments are problematic and confusing.\n\nStrength:\n+ The proposed Intra-scale Propagation module is novel. Combing SA with Convolution makes sense.\n+ The performance and speed is comparable to concurrent works.  \n\nWeakness: There are some issues needed to be addressed before accepted.\n- In Table 3, the model setup for tiny and small model is confusing. It is not aligned with competitors on parameters and flops, which make hard to do fairly comparison.  \n- It lacks the performance comparison on large models. Will the performance hold? \n- It lacks necessary ablation study on the proposed Intra-scale Propagation module. Based on Table 3, the improvement of such module is small. It is also decreasing when model size increases. The author needs to justify the effectiveness of this module.\n- The benefit of DS-FPN is not convincing. Although it shows improvement in Table 4, it also significantly increases the number of parameters. How about the latency?\n- Writing issues: The precision used in reporting numbers is not consistent.  The presentation of tables (e.g. font, whitespace) needs improvement.\n\nPost-rebuttal:\nThanks authors for providing more details. However,  the newly provided ablation study doesn’t justify the effectiveness of proposed components. Hence, I will hold my rating as is.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Please justify the weakness to improve the paper.\n\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "SuaEKqI-2I2", "reviewer": "Reviewer_tvGe", "summary": "This paper proposes Dual-stream Network (DS-Net) for visual recognition. The two streams correspond to high-resolution convolutions for local features and low-resolution vision transformers for global patterns. The paper proposes an intra-scale propagation module to process the two streams and an inter-scale alignment module to fuse them into one. For object detection, it designs a Dual-stream FPN (DS-FPN) that adds dual-stream blocks to every feature pyramid. The proposed DS-Net achieves good results for image classification, object detection, and instance segmentation.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is overall nicely organized. I have some comments that might need the authors’ attention.\n\n1. The writing quality needs improvements. There are many grammar errors here and there. I would recommend the authors pay more attention to proofreading. The frequency of the writing errors is high and it concerns me. This factor is not considered when giving the ratings, though, because I believe they can be fixed.\n\n2. The paper likes to claim that a statement is obvious. For example, in Line 38, the paper states that the previous works that attempt to introduce convolutions to vision transformers are obviously not the most ideal design. I personally don’t think it’s appropriate to claim they are obvious especially for topics that do not have a consensus. The other example is Line 172, stating that the naive fusion is not good enough obviously. However, according to the experimental results in Table 3 that compare DS-Net-S (naive) and DS-Net-S* (inter-scale alignment module), the proposed method outperforms the naive solution by 0.4% (81.9% → 82.3%) but at the cost of 16.7% additional computations (3.0 GFLOPs → 3.5 GFLOPs). Therefore, I wouldn’t think the naive fusion is not good enough obviously. By the way, there is an error when introducing DS-Net and DS-Net* in Line 251 -- “with” and “without” should be swapped.\n\n3. The novelty is not strong. The idea of the dual-stream network is just to have one convolutional stream and one transformer stream. This is similar to dual-path network [a], which is missing in the related work. The idea of using different resolutions has also been explored, e.g. [b]. I wouldn’t think the novelty of this paper is particularly strong. In addition, after looking at Table 3, the improvements over the previous methods are also marginal.\n\nIn summary, I find the novelty and the performance are not that strong. Unfortunately, I wouldn’t be able to recommend acceptance for this paper. The authors should pay attention to the writing quality as well when revising this submission.\n\n[a] Chen, Y., Li, J., Xiao, H., Jin, X., Yan, S. and Feng, J., 2017. Dual path networks. In NeurIPS 2017.\n\n[b] Wang, H., Kembhavi, A., Farhadi, A., Yuille, A.L. and Rastegari, M., 2019. Elastic: Improving cnns with dynamic scaling policies. In CVPR 2019\n\n**EDIT: Post-rebuttal**\n\nThanks for preparing a detailed response to the initial comments. After reading the response, the concerns about the novelty and performance improvements still remain. Moreover, as mentioned in the authors' response, when the model size is larger, the improvement over other baselines also becomes smaller -- 83.1% using 46M for the proposed method v.s. 83.0% with 50M of Swin-S. The proposed method also has less throughput in general. Unfortunately, I will keep the previous rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have adequately addressed them in the submission.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "Rb4f_ocCLl2", "reviewer": "Reviewer_LMod", "summary": "This paper proposes a hybrid convolution+transformer model for 2d images. To summarize the model, it is broken up into 4 stages that operate at successively higher resolutions much like a typical Resnet.  At each stage, there are two parallel paths, one that apples a depthwise convolution at high resolution (capturing local effects), and one that applies a standard a multi-headed attention at coarse (capturing global effects) resolution.  These paths are fused by a final attention module which allows for local to attend to global and vice versa.\n\nAnother contribution is the DS-FPN (which simply slots one of the above “dual stream” modules into the lateral paths of a typical FPN). Experiments show strong performance on both image classification for ImageNet and detection/segmentation for COCO (relative to competing Transformer-only models like Deit).\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Overall, the authors have proposed an intuitive and conceptually simple model that enjoys the benefits of both convolution and transformer/attention.  They achieve strong performance without an overly expensive model, which to my knowledge is novel. I have some complaints on exposition/writing but no major criticisms otherwise.  Based on good execution and reasonably strong results, I’m recommending acceptance.\n\nSome more detailed comments/questions:\n* In several places throughout the paper, the authors claim that something is “obvious” when it is not really obvious (and certainly not provable in a mathematical sense)… I recommend tempering this language.\n* In the intro, the authors say “​​the conflicting properties of such two operations, convolution for local patterns but attention for global patterns, might cause ambiguity during training,...”: it’s not clear what this means and how one might quantify this statement.\n* There is a sentence in the related work section: “progressively aggregate neighboring tokens before transformers via overlapping convolution-alike way to enhance their representations” that makes no sense.\n* Related paper: “MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers” by Wang et al also use convolutions and multi-headed attention in a manner where both types of modules are woven fairly tightly together through the network. It’s quite a different approach in some ways, but should be cited here.\n* Figure 2: is missing a 1x1 convolution at the end of the Inter-scale alignment block\n* Is there a network stem that first does a reduction to stride 4 (as in Resnet)?\n* I’m very surprised that only depthwise convolutions are used in the convolution path — and would like to clarification as to whether these are literally depthwise convolutions as stated in Eqn 1 or whether there is also a 1x1 convolution following (thus making it a depthwise separable convolution, which is far more commonly used in convnets).  Using a depthwise-only convolution is pretty limited, allows for no interaction between channels and essentially would amount to a blur operation, which is why this is so surprising.\n    * Along the same lines, some papers have found it important to make the first convolution\n* Clarification: are positional encodings used anywhere?  Would they help if not?\n* For Table 3, I will just note that there are many more recent convolution-only networks than ResNet — I know that this is not necessarily the point, but it would be good to have a comparison against the most recent convolution-only networks to see how the transformer based models stand.\n* For Table 4, I recommend adding in some other baselines, especially DETR (a transformer based detection model).\n* In Section 4.3, authors claim to evaluate DS-Net in the Cascade R-CNN model, but these experiments do not seem to be in the paper.\n* Why is DS-Net-T used in Table 6 instead of DS-Net-S?\n* What is the running time impact of using DS modules in the FPN?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviews are split between accept and borderline reject recommendations. Two reviewers with positive ratings (7 and 8) appreciated that the hybrid CNN+Transformer approach is simple and intuitive. They also liked that the paper reports good empirical performance. The other two reviewers with negative ratings (5 and 5) raised concerns about limited novelty and insufficient/unconvincing empirical results. They criticized that the novelty isn't particularly strong because prior have already looked into the dual-stream approach combining different architectures as well as multi-resolution processing. They also shared a common concern that there is insufficient empirical evidence showing that the proposed approach is truly beneficial compared to simpler alternatives, especially at a large-scale regime, such as Swin Transformer (Swin-S) that achieves similar performance at a comparable parameter size (Swin-S: 83% w/ 50M params vs. this work: 83.1% w/ 46M params). Separate from novelty/experiment concerns, all four reviewers raised several writing issues.

This meta-reviewer carefully read the reviews, rebuttal, post-rebuttal discussion, and the paper in detail to fully understand the concerns raised by the reviewers. I agree with reviewers cxxV and LMod that the proposed idea of combining convolution and self-attention operations is interesting and novel --  it is a simple and effective, yet non-trivial combination than simply stacking CNN and Transformer side-by-side. Plus, as the reviewer cxxV pointed out, the paper reports impressive transfer results on ImageNet and COCO, the latter of which is particularly interesting because it is far from being solved (compared to ImageNet). I do however share the concerns with tvGe and ybP5, and including additional results as requested by the two reviewers will make the paper much stronger. The authors are urged to incorporate all the feedback provided by the reviews in their final version.
Decision: Accept (Poster)