Dual-stream Network for Visual Recognition

1 Introduction . In recent years , convolutional neural networks ( CNNs ) have dominated various vision tasks including but not limited to image recognition [ 22 , 18 , 38 , 39 , 5 , 19 , 41 , 20 , 52 ] , object detection [ 37 , 2 , 26 , 36 , 24 , 51 , 30 , 21 , 11 ] and segmentation [ 17 , 28 ] , thanks to their unprecedented representation capacity . However , limited receptive fields of convolutions inevitably neglect the global patterns in images , which might be crucial during inference . For example , it is more likely a chair than a elephant nearby a table . Such object-level information can not be fully explored by local convolutions , that hampers further improvement of CNNs . Motivated by the success of Transformer architecture [ 43 ] in Natural Language Processing ( NLP ) [ 9 , 34 , 1 ] and Multi-modality Fusion [ 29 , 40 , 12 ] , researchers are trying to apply Transformer to vision tasks and have obtained promising results . In [ 10 , 55 , 48 , 4 , 7 , 45 , 46 , 27 , 3 , 54 , 14 , 15 , 44 , 53 ] , many novel architectures and optimizing strategies are proposed and achieve comparable or even better performance than CNNs , emphasizing the significance of extracting global features in vision tasks . Inspired by the verified efficacy of CNN and Transformer , many concurrent works , such as ContNet [ 47 ] a CvT [ 46 ] , attempt to introduce convolutions to vision transformers in different manners , hoping to combine both advantages . In CvT [ 46 ] , linear projection of self-attention block in the Transformer module are replaced with convolutional projection . In contrast , ContNet [ 47 ] performs 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) convolution on different token maps to build mutual connections , which is similar to Swin Transformer [ 27 ] . Attention maps ( a ) ( b ) Input image Feature maps ( c ) ( d ) Deit DS-Net Deit DS-Net However , some drawbacks of existing models still remain to be solved . Firstly , works mentioned above either perform convolution and attention mechanisms sequentially , or just replace linear projection with convolutional projection in attention mechanisms , which may not be the most ideal design . Additionally , the conflicting properties of such two operations , convolution for local patterns but attention for global patterns , might cause ambiguity during training , which prevents their merits from merging to the maximum extent . Furthermore , self-attention can capture long-range information via the built-in all-pair interaction in theory , but it is quite possible that attentions might be confused and disturbed by neighboring details in high resolution feature maps , fail to build up object-level global patterns ( see Fig . 1 ( a ) ) . Finally , the computation cost of self-attention is unaffordable due to the quadratic computation complexity of sequence length . Although PVT [ 45 ] and APNB [ 56 ] downsample the key-query features to improve the efficiency of self-attention operator , they both abandon fine-graind local details of the image , which greatly impairs their performance . In this paper , we address the these issues by introducing a Dual-stream Network ( DS-Net ) . Instead of single stream architecture as previous works , our DS-Net adopts Dual-stream Blocks ( DS-Blocks ) , which generates two feature maps with different resolutions , and retains both local and global information of the image via two parallel branches . We propose a Intra-scale Propagation module here to process two feature maps . Specifically , high-resolution features are used to extract fine-grained local patterns with depth-wise convolution , while low-resolution features are expected to summarize long-range global patterns . Considering low-resolution features themselves contain more integrated information , it would be much more easier for self-attention mechanism to capture object-level patterns rather than overwhelmed by trivial details ( see Fig . 1 ( b ) ) . Such dual-stream architecture disentangles local and global representations , which helps maximize both their merits , and thus generates better representations compared to DeiT baseline ( see ( c ) and ( e ) in Fig . 1 ) . Besides , low-resolution feature maps for self-attention dramatically reduce the memory cost and computation complexity . After parallelly processing dual streams , we present Inter-scale Alignment module based on co-attention mechanism at the end of DS-Blocks . This is because local details and global patterns capture different perspectives of the image , which are misaligned not only in pixel positions but also in semantics . Hence , this module is designed for modeling complex cross-scale relations and adaptively fuse local and global patterns . Besides , we apply our DS-Blocks to Feature Pyramid Networks for further feature refinement , named DS-FPN . In this way , multi-level features are capable of extracting contextual information from both local and global views , improving performance of downstream tasks . This demonstrates that our Dual Stream design could be utilized as a plug-in building block not only for image recognition , but for many other vision tasks . The contributions of this work are concluded as follows : 1 . We present a novel Dual-Stream Network , named DS-Net , which retains both local and global features in DS-Block . The independent propagation maximize the advantages of convolution and self-attention , thus eliminating the conflicts during training . 2 . We propose Intra-scale Propagation and Inter-Scale Alignment mechanism to achieve effective information flows within and between features of different resolutions , thus generating better representations of the image . 3 . We introduce Dual-stream Feature Pyramid Network ( DS-FPN ) to enhance contextual information for downstream dense tasks and achieves better performance with little extra costs . 4 . Without bells and whistles , the proposed DS-Net outperforms DeiT baseline by significant margins in terms of top-1 accuracy on ImageNet-1k and achieves state-of-the-art performance over other Vision Transformers and CNN-based networks on image classification and downstream tasks , including object detection and instance segmentation . 2 Related work . Vision Transformers . Motivated by the great success of Transformer in natural language processing , researchers are trying to apply Transformer architecture to Computer Vision tasks . Unlike mainstream CNN-based models , Transformer is capable of capturing long-distance visual relations by its selfattention module and provides the paradigm without image-specific inductive bias . ViT [ 10 ] views 16 × 16 image patches as token sequence and predicts classification via a unique class token , which shows promising results . Subsequently , many works , such as DeiT [ 42 ] and PVT [ 45 ] achieve further improvement on ViT , making it more efficient and applicable in downstream tasks . ResT [ 50 ] proposed a efficient transformer model . Besides , models based on Transformer also give leading performance in many other vision tasks such as object tracking and video analysis . Local and Global Features in Transformers . Despite the Transformer ’ s superiority upon extracting global representations , image-level self-attention is unable to capture fine-grained details . To tackle this issue , previous works assert to obtain the local information at the same time and then fuse features of the two scales . TNT [ 16 ] proposes intra-patch self-attention block to model local structures and aggregates them with global patch embeddings . Swin Transformer [ 27 ] extracts local features within each partitioned window and fuses them by self-attention in successive shifted windows . Similarly , shifted windows are replaced by convolutions in ContNet [ 47 ] . Following such design of local windows , Twins [ 6 ] uses inter-window attention to aggregate global features . Swin Transformer and Twins both give advanced performance , but still rely on local-window self-attention to capture fine-level features , which works intuitively worse than CNN . Besides , Swin ’ s shifted windows are complex for device optimization and Twins [ 6 ] recurrently local and global attentions would constrain each other ’ s representation learning . Equip Transformers with Convolution . Compared to self-attention mechanism , CNN has its unique advantages of local modeling and translation invariance . Therefore , some existing transformers explore the hybrid architecture to incorporate both merits for better visual representation . T2T [ 48 ] progressively aggregate neighboring tokens to one token to capture local structure information , similar to convolution operation . Followingly , CvT [ 46 ] designs Convolutional Token Embedding and Convolutional Transformer Block for capturing more precise local spatial context . CPVT [ 7 ] utilizes one convolutional layer to dynamically generate positional encodings , adaptive for varying image scales . Conformer [ 33 ] combines transformer with an independent CNN network for specializing local features and flows information via lateral connections . Nonetheless , none of the aforementioned models maximizes the potential of the hybrid architecture . They either use CNN for marginal operations to boost Transformer , e.g. , tokens generation in CvT [ 46 ] and positional encodings in CPVT [ 7 ] , or view CNN as a relatively separate branch , like Conformer . Different from previous works , in proposed DS-Blocks , we treat self-attention and convolution as dual resolution processing paths via Intra-scale Propagation module , where the former aims to extract local fine-grained details , and the latter focus on exploring features from a global view . On top of that , their information is reasonably fused by Inter-scale Alignment module based on co-attention , which addresses the issue of feature misalignment . By this design , the local and global patterns could be simultaneously extracted and the capability of CNN is fully exerted in Transformer architecture . 3 Method . 3.1 Dual-scale Representations . The overall pipeline of our proposed DS-Net is shown in Fig . 2 . Motivated by stage-wise design in previous networks like ResNet [ 18 ] , we set 4 stages in the architecture , whose down-sampling factors are 4 , 8 , 16 , 32 , respectively . Within each stage , Dual-stream Blocks ( DS-Blocks ) are adopted to generate and combine dual-scale representations . Our key idea is to keep local features in a relatively high resolution to reserve local details , while represent global features with a lower resolution ( 1/32 of the image size ) to retain global patterns . Specifically , in each DS-block , we split the input feature map into two parts at the channel dimension . One is for extracting local features , denoted as fl , the other is for summarizing global features , denoted as fg . Notably , we keep the size of fg unchanged in all stages across the network by downsampling it with a proper factor , W32 × H 32 × Ci 2 , where W and H represents the width and height of the input image , and Ci represents the channel number of input features in current stage . Aided by the high resolution of fl , the local patterns could be much reserved for subsequent extraction , but thanks to the low resolution of fg , the exploration of non-local and object-level information is greatly benefited . 3.2 Intra-scale Propagation . Local and global features are representations of one image from two totally different views . The former focuses on fine-grained details , essential for tiny-object detection and pixel-level localization , while the latter aims at modeling object-level relations between long-range parts . Therefore , given dual-scale features fl and fg , we perform Intra-scale Propagation module to parallelly process them . Local representation . For high-resolution fl with the size of Wi ×Hi × Cl , where Cl equals Ci2 , we perform 3×3 depth-wise convolution to extract local features as follows and obtain fL : fL ( i , j ) = M , N∑ m , n W ( m , n ) ⊙ fl ( i+m , j + n ) , ( 1 ) where W ( m , n ) , ( m , n ) ∈ ( −1 , 0 , 1 ) represents the convolution filters , W ( m , n ) and fl ( i , j ) are both Ci 2 dimensional vectors , ⊙ denotes element-wise product . By the power of depth-wise convolution , fL is able to contain fine-grained local details of the input image . Global representation . For low-resolution representation fg with the fixed size of W32 × H 32 × Ci 2 as illustrated in Section 3.1 , we first flatten fg to a sequence with the length of lg , the product of W32 and H 32 , in which each element is a Ci 2 dimensional vector . By doing this , each vector in the sequence is treated as a visual token without spatial information . Here , the dependencies between different token pairs are unrelated with their spatial positions in the feature map , which is totally different with the convolution . Then , we summarize the global information and model the object-level coupling relation via self-attention mechanism : fQ = fgWQ , fK = fgWK , fV = fgWV , ( 2 ) where WQ , WK , WV denote matrixes to generate queries , keys and values respectively . By calculating the similarity between fQ and fV , we obtain attention weights for aggregating information from different locations of fg . Finally , we calculate the weighted sum of attention weights and fV , thus obtaining integrated features : fG = softmax ( fQf T K√ d ) fV , ( 3 ) where d equals Ci 2 N , N denotes the number of attention head that we set to 1 , 2 , 5 , 8 for 4 stages respectively in DS-Net . Our dual-stream architecture disentangles fine-grained and integrated features in two pathways , which significantly eliminates the ambiguity during training . Additionally , the Intra-scale Propagation module processes feature maps with dual resolutions by two domain-specifically effective mechanisms separately , extracting local and global features to the maximum extent .

Summary: This paper proposes a hybrid convolution+transformer model for 2d images. To summarize the model, it is broken up into 4 stages that operate at successively higher resolutions much like a typical Resnet.  At each stage, there are two parallel paths, one that apples a depthwise convolution at high resolution (capturing local effects), and one that applies a standard a multi-headed attention at coarse (capturing global effects) resolution.  These paths are fused by a final attention module which allows for local to attend to global and vice versa.  Another contribution is the DS-FPN (which simply slots one of the above “dual stream” modules into the lateral paths of a typical FPN). Experiments show strong performance on both image classification for ImageNet and detection/segmentation for COCO (relative to competing Transformer-only models like Deit).