Dual-stream Network for Visual Recognition

1 Introduction . In recent years , convolutional neural networks ( CNNs ) have dominated various vision tasks including but not limited to image recognition [ 22 , 18 , 38 , 39 , 5 , 19 , 41 , 20 , 52 ] , object detection [ 37 , 2 , 26 , 36 , 24 , 51 , 30 , 21 , 11 ] and segmentation [ 17 , 28 ] , thanks to their unprecedented representation capacity . However , limited receptive fields of convolutions inevitably neglect the global patterns in images , which might be crucial during inference . For example , it is more likely a chair than a elephant nearby a table . Such object-level information can not be fully explored by local convolutions , that hampers further improvement of CNNs . Motivated by the success of Transformer architecture [ 43 ] in Natural Language Processing ( NLP ) [ 9 , 34 , 1 ] and Multi-modality Fusion [ 29 , 40 , 12 ] , researchers are trying to apply Transformer to vision tasks and have obtained promising results . In [ 10 , 55 , 48 , 4 , 7 , 45 , 46 , 27 , 3 , 54 , 14 , 15 , 44 , 53 ] , many novel architectures and optimizing strategies are proposed and achieve comparable or even better performance than CNNs , emphasizing the significance of extracting global features in vision tasks . Inspired by the verified efficacy of CNN and Transformer , many concurrent works , such as ContNet [ 47 ] a CvT [ 46 ] , attempt to introduce convolutions to vision transformers in different manners , hoping to combine both advantages . In CvT [ 46 ] , linear projection of self-attention block in the Transformer module are replaced with convolutional projection . In contrast , ContNet [ 47 ] performs 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) convolution on different token maps to build mutual connections , which is similar to Swin Transformer [ 27 ] . Attention maps ( a ) ( b ) Input image Feature maps ( c ) ( d ) Deit DS-Net Deit DS-Net However , some drawbacks of existing models still remain to be solved . Firstly , works mentioned above either perform convolution and attention mechanisms sequentially , or just replace linear projection with convolutional projection in attention mechanisms , which may not be the most ideal design . Additionally , the conflicting properties of such two operations , convolution for local patterns but attention for global patterns , might cause ambiguity during training , which prevents their merits from merging to the maximum extent . Furthermore , self-attention can capture long-range information via the built-in all-pair interaction in theory , but it is quite possible that attentions might be confused and disturbed by neighboring details in high resolution feature maps , fail to build up object-level global patterns ( see Fig . 1 ( a ) ) . Finally , the computation cost of self-attention is unaffordable due to the quadratic computation complexity of sequence length . Although PVT [ 45 ] and APNB [ 56 ] downsample the key-query features to improve the efficiency of self-attention operator , they both abandon fine-graind local details of the image , which greatly impairs their performance . In this paper , we address the these issues by introducing a Dual-stream Network ( DS-Net ) . Instead of single stream architecture as previous works , our DS-Net adopts Dual-stream Blocks ( DS-Blocks ) , which generates two feature maps with different resolutions , and retains both local and global information of the image via two parallel branches . We propose a Intra-scale Propagation module here to process two feature maps . Specifically , high-resolution features are used to extract fine-grained local patterns with depth-wise convolution , while low-resolution features are expected to summarize long-range global patterns . Considering low-resolution features themselves contain more integrated information , it would be much more easier for self-attention mechanism to capture object-level patterns rather than overwhelmed by trivial details ( see Fig . 1 ( b ) ) . Such dual-stream architecture disentangles local and global representations , which helps maximize both their merits , and thus generates better representations compared to DeiT baseline ( see ( c ) and ( e ) in Fig . 1 ) . Besides , low-resolution feature maps for self-attention dramatically reduce the memory cost and computation complexity . After parallelly processing dual streams , we present Inter-scale Alignment module based on co-attention mechanism at the end of DS-Blocks . This is because local details and global patterns capture different perspectives of the image , which are misaligned not only in pixel positions but also in semantics . Hence , this module is designed for modeling complex cross-scale relations and adaptively fuse local and global patterns . Besides , we apply our DS-Blocks to Feature Pyramid Networks for further feature refinement , named DS-FPN . In this way , multi-level features are capable of extracting contextual information from both local and global views , improving performance of downstream tasks . This demonstrates that our Dual Stream design could be utilized as a plug-in building block not only for image recognition , but for many other vision tasks . The contributions of this work are concluded as follows : 1 . We present a novel Dual-Stream Network , named DS-Net , which retains both local and global features in DS-Block . The independent propagation maximize the advantages of convolution and self-attention , thus eliminating the conflicts during training . 2 . We propose Intra-scale Propagation and Inter-Scale Alignment mechanism to achieve effective information flows within and between features of different resolutions , thus generating better representations of the image . 3 . We introduce Dual-stream Feature Pyramid Network ( DS-FPN ) to enhance contextual information for downstream dense tasks and achieves better performance with little extra costs . 4 . Without bells and whistles , the proposed DS-Net outperforms DeiT baseline by significant margins in terms of top-1 accuracy on ImageNet-1k and achieves state-of-the-art performance over other Vision Transformers and CNN-based networks on image classification and downstream tasks , including object detection and instance segmentation . 2 Related work . Vision Transformers . Motivated by the great success of Transformer in natural language processing , researchers are trying to apply Transformer architecture to Computer Vision tasks . Unlike mainstream CNN-based models , Transformer is capable of capturing long-distance visual relations by its selfattention module and provides the paradigm without image-specific inductive bias . ViT [ 10 ] views 16 × 16 image patches as token sequence and predicts classification via a unique class token , which shows promising results . Subsequently , many works , such as DeiT [ 42 ] and PVT [ 45 ] achieve further improvement on ViT , making it more efficient and applicable in downstream tasks . ResT [ 50 ] proposed a efficient transformer model . Besides , models based on Transformer also give leading performance in many other vision tasks such as object tracking and video analysis . Local and Global Features in Transformers . Despite the Transformer ’ s superiority upon extracting global representations , image-level self-attention is unable to capture fine-grained details . To tackle this issue , previous works assert to obtain the local information at the same time and then fuse features of the two scales . TNT [ 16 ] proposes intra-patch self-attention block to model local structures and aggregates them with global patch embeddings . Swin Transformer [ 27 ] extracts local features within each partitioned window and fuses them by self-attention in successive shifted windows . Similarly , shifted windows are replaced by convolutions in ContNet [ 47 ] . Following such design of local windows , Twins [ 6 ] uses inter-window attention to aggregate global features . Swin Transformer and Twins both give advanced performance , but still rely on local-window self-attention to capture fine-level features , which works intuitively worse than CNN . Besides , Swin ’ s shifted windows are complex for device optimization and Twins [ 6 ] recurrently local and global attentions would constrain each other ’ s representation learning . Equip Transformers with Convolution . Compared to self-attention mechanism , CNN has its unique advantages of local modeling and translation invariance . Therefore , some existing transformers explore the hybrid architecture to incorporate both merits for better visual representation . T2T [ 48 ] progressively aggregate neighboring tokens to one token to capture local structure information , similar to convolution operation . Followingly , CvT [ 46 ] designs Convolutional Token Embedding and Convolutional Transformer Block for capturing more precise local spatial context . CPVT [ 7 ] utilizes one convolutional layer to dynamically generate positional encodings , adaptive for varying image scales . Conformer [ 33 ] combines transformer with an independent CNN network for specializing local features and flows information via lateral connections . Nonetheless , none of the aforementioned models maximizes the potential of the hybrid architecture . They either use CNN for marginal operations to boost Transformer , e.g. , tokens generation in CvT [ 46 ] and positional encodings in CPVT [ 7 ] , or view CNN as a relatively separate branch , like Conformer . Different from previous works , in proposed DS-Blocks , we treat self-attention and convolution as dual resolution processing paths via Intra-scale Propagation module , where the former aims to extract local fine-grained details , and the latter focus on exploring features from a global view . On top of that , their information is reasonably fused by Inter-scale Alignment module based on co-attention , which addresses the issue of feature misalignment . By this design , the local and global patterns could be simultaneously extracted and the capability of CNN is fully exerted in Transformer architecture . 3 Method . 3.1 Dual-scale Representations . The overall pipeline of our proposed DS-Net is shown in Fig . 2 . Motivated by stage-wise design in previous networks like ResNet [ 18 ] , we set 4 stages in the architecture , whose down-sampling factors are 4 , 8 , 16 , 32 , respectively . Within each stage , Dual-stream Blocks ( DS-Blocks ) are adopted to generate and combine dual-scale representations . Our key idea is to keep local features in a relatively high resolution to reserve local details , while represent global features with a lower resolution ( 1/32 of the image size ) to retain global patterns . Specifically , in each DS-block , we split the input feature map into two parts at the channel dimension . One is for extracting local features , denoted as fl , the other is for summarizing global features , denoted as fg . Notably , we keep the size of fg unchanged in all stages across the network by downsampling it with a proper factor , W32 × H 32 × Ci 2 , where W and H represents the width and height of the input image , and Ci represents the channel number of input features in current stage . Aided by the high resolution of fl , the local patterns could be much reserved for subsequent extraction , but thanks to the low resolution of fg , the exploration of non-local and object-level information is greatly benefited . 3.2 Intra-scale Propagation . Local and global features are representations of one image from two totally different views . The former focuses on fine-grained details , essential for tiny-object detection and pixel-level localization , while the latter aims at modeling object-level relations between long-range parts . Therefore , given dual-scale features fl and fg , we perform Intra-scale Propagation module to parallelly process them . Local representation . For high-resolution fl with the size of Wi ×Hi × Cl , where Cl equals Ci2 , we perform 3×3 depth-wise convolution to extract local features as follows and obtain fL : fL ( i , j ) = M , N∑ m , n W ( m , n ) ⊙ fl ( i+m , j + n ) , ( 1 ) where W ( m , n ) , ( m , n ) ∈ ( −1 , 0 , 1 ) represents the convolution filters , W ( m , n ) and fl ( i , j ) are both Ci 2 dimensional vectors , ⊙ denotes element-wise product . By the power of depth-wise convolution , fL is able to contain fine-grained local details of the input image . Global representation . For low-resolution representation fg with the fixed size of W32 × H 32 × Ci 2 as illustrated in Section 3.1 , we first flatten fg to a sequence with the length of lg , the product of W32 and H 32 , in which each element is a Ci 2 dimensional vector . By doing this , each vector in the sequence is treated as a visual token without spatial information . Here , the dependencies between different token pairs are unrelated with their spatial positions in the feature map , which is totally different with the convolution . Then , we summarize the global information and model the object-level coupling relation via self-attention mechanism : fQ = fgWQ , fK = fgWK , fV = fgWV , ( 2 ) where WQ , WK , WV denote matrixes to generate queries , keys and values respectively . By calculating the similarity between fQ and fV , we obtain attention weights for aggregating information from different locations of fg . Finally , we calculate the weighted sum of attention weights and fV , thus obtaining integrated features : fG = softmax ( fQf T K√ d ) fV , ( 3 ) where d equals Ci 2 N , N denotes the number of attention head that we set to 1 , 2 , 5 , 8 for 4 stages respectively in DS-Net . Our dual-stream architecture disentangles fine-grained and integrated features in two pathways , which significantly eliminates the ambiguity during training . Additionally , the Intra-scale Propagation module processes feature maps with dual resolutions by two domain-specifically effective mechanisms separately , extracting local and global features to the maximum extent .

Summary: The authors propose a new architecture for visual recognition. Like standard convolutional architectures, the representation is structured in a multi-scale hierarchy, with deeper layers being coarser and more abstract. At every level of the hierarchy, a coarse, global representation is appended and simultaneously processed with the more fine-grained, local one. Local features are processed with depth-wise convolutions, and global ones with multi-head self-attention. In addition to this separate processing of local and global features, they are co-processed by cross-attending between local and global features.   In addition to this recognition backbone, the authors adapt Feature Pyramid Networks in a similar spirit. Whereas FPN's use dense convolutions to fuse an intermediate representation with a coarser one, the authors instead use the proposed "Dual-Stream" block, combining local depth-wise convolutions, global self-attention, and local/global cross-attention.   The authors evaluate the recognition backbone for classification on ImageNet and detection and instance segmentation on COCO, reporting large gains over standard convolutional architectures such as ResNet, and reasonable gains over more recent transformer-based ones.