On the Connection between Local Attention and Dynamic Depth-wise Convolution

1 INTRODUCTION . Vision Transformer ( Chu et al. , 2021b ; d ’ Ascoli et al. , 2021 ; Dosovitskiy et al. , 2021 ; Guo et al. , 2021 ; Han et al. , 2020 ; Khan et al. , 2021 ; Touvron et al. , 2020 ; Wang et al. , 2021b ; Wu et al. , 2021 ; Xu et al. , 2021 ; Yuan et al. , 2021b ) has shown promising performance in ImageNet classification . The improved variants , Local Vision Transformer ( Chu et al. , 2021a ; Liu et al. , 2021b ; Vaswani et al. , 2021 ) , adopt the local attention mechanism , which partitions the image space into a set of small windows , and conducts the attention over the windows simultaneously . Local attention leads to great improvement in memory and computation efficiency and makes the extension to downstream tasks easier and more efficient , such as object detection and semantic segmentation . We exploit the network regularization schemes ( Goodfellow et al. , 2016 ) , sparse connectivity that controls the model complexity , and weight sharing that relaxes the requirement of increasing the training data scale and reduces the model parameters , as well as dynamic weight prediction that increases the model capability , to study the local attention mechanism . We rephrase local attention as a channel-wise spatially-locally connected layer with dynamic connection weights . The main properties are summarized as follows . ( i ) Sparse connectivity : there is no connection across channels , and each output position is only connected to the input positions within a local window . ( ii ) Weight sharing : the connection weights are shared across channels or within each group of channels . ( iii ) Dynamic weight : the connection weights are dynamically predicted according to each image instance . We connect local attention with depth-wise convolution ( Chollet , 2017 ; Howard et al. , 2017 ) and its dynamic variants that are also a channel-wise spatially-locally connected layer with optional dynamic connection weights . They are similar in sparse connectivity . The main differences lie in ( i ) weight sharing - depth-wise convolution shares connection weights ( kernel weights ) across spatial positions and attention shares the connection weights across channels , and ( ii ) dynamic weight computation manners - local attention is based on dot-products between pairwise positions in the local window , and dynamic convolution is based on linear projections conducted on the center representation or the globally pooled representation . We further present the empirical verification for the connection . We take the recently-developed Local Vision Transformer , Swin Transformer ( Liu et al. , 2021b ) , as an example , and study the empirical performance of local attention and ( dynamic ) depth-wise convolution in the same training settings as Swin Transformer . We replace the local attention layer with the ( dynamic ) depth-wise convolution layer , keeping the overall structure unchanged . The results show that the ( dynamic ) depth-wise convolution-based approaches achieve comparable or slightly higher performance for ImageNet classification and two downstream tasks , COCO object detection and ADE semantic segmentation , and ( dynamic ) depth-wise convolution takes lower computation complexity . The ablation studies imply that weight sharing and dynamic weight improves the model capability . Specifically , ( i ) for Swin Transformer , weight sharing across channels is beneficial mainly for reducing the parameter ( attention weight ) complexity , and the attention-based dynamic weight scheme is advantageous in learning instance-specific weights and block-translation equivalent representations ; ( ii ) for depth-wise convolution , weight sharing across positions is beneficial for reducing the parameter complexity as well as learning translation equivalent representations , and the linear projection-based dynamic weight scheme learns instance-specific weights . 2 CONNECTING LOCAL ATTENTION AND DEPTH-WISE CONVOLUTION . 2.1 LOCAL ATTENTION . Vision Transformer ( Dosovitskiy et al. , 2021 ) forms a network by repeating the attention layer and the subsequent point-wise MLP ( point-wise convolution ) . The local Vision Transformer , such as Swin Transformer ( Liu et al. , 2021b ) and HaloNet ( Vaswani et al. , 2021 ) , adopts the local attention layer , which partitions the space into a set of small windows and performs the attention operation within each window simultaneously , to improve the memory and computation efficiency . The local attention mechanism forms the keys and values in a window that the query lies in . The attention output for the query xi ∈ RD at the position i is the aggregation of the corresponding values in the local window , { xi1 , xi2 , . . . , xiNk } , weighted by the corresponding attention weights { ai1 , ai2 , . . . , aiNk } 1 : yi = ∑Nk j=1 aijxij , ( 1 ) where Nk = Kw ×Kh is the size of the local window . The attention weight aij is computed as the softmax normalization of the dot-product between the query xi and the key xij : aij = e 1√ D x > i xij Zi where Zi = ∑Nk j=1 e 1√ D x > i xij . ( 2 ) The multi-head version partitions the D-dimensional query , key and value vectors into M subvectors ( each with DM dimensions ) , and conducts the attention process M times , each over the corresponding subvector . The whole output is the concatenation of M outputs , yi = [ y > i1 y > i2 . . . y > iM ] > . The mth output yim is calculated by yim = ∑Nk j=1 aijmxijm , ( 3 ) where xijm is the mth value subvector and aijm is the attention weight computed from the mth head in the same way as Equation 2 . 2.2 SPARSE CONNECTIVITY , WEIGHT SHARING , AND DYNAMIC WEIGHT . We give a brief introduction of two regularization forms , sparse connectivity and weight sharing , and dynamic weight , and their benefits . We will use the three forms to analyze local attention and connect it to dynamic depth-wise convolution . 1For presentation convenience , we ignore the linear projections conducted to the queries , the keys and the values . In vision applications , the value and the corresponding key are from the same feature possibly with different linear projections , and we denote them using the same symbol xij . Sparse connectivity means that there are no connections between some output neurons ( variables ) and some input neurons in a layer . It reduces the model complexity without decreasing the number of neurons , e.g. , the size of the ( hidden ) representations . Weight sharing indicates that some connection weights are equal . It lowers the number of model parameters and increases the network size without requiring a corresponding increase in training data ( Goodfellow et al. , 2016 ) . Dynamic weight refers to learning specialized connection weights for each instance . It generally aims to increase the model capacity . If regarding the learned connection weights as hidden variables , dynamic weight can be viewed as introducing second-order operations that increase the capability of the network . The connection to Hopfield networks is discussed in ( Ramsauer et al. , 2020 ) . 2.3 ANALYZING LOCAL ATTENTION . We show that local attention is a channel-wise spatially-locally connected layer with dynamic weight computation , and discuss its properties . Figure 1 ( c ) illustrates the connectivity pattern . The aggregation processes ( Equation 1 and Equation 3 ) for local attention can be rewritten equivalently in a form of element-wise multiplication : yi = ∑Nk j=1 wij xij , ( 4 ) where is the element-wise multiplication operator , and wij ∈ RD is the weight vector formed from the attention weight aij or { aij1 , aij2 , . . . , aijM } . Sparse connectivity . The local attention layer is spatially sparse : each position is connected to the Nk positions in a small local window . There are also no connections across channels . The element-wise multiplication in Equation 4 indicates that given the attention weights , each output element , e.g. , yid ( the ith position for the dth channel ) , is only dependent on the corresponding input elements from the same channel in the window , { xi1d , xi2d , . . . , xiNkd } , and not related to other channels . Weight sharing . The weights are shared with respect to channels . In the single-head attention case , all the elements { wij1 , wij2 , . . . , wijD } in the weight vector wij are the same : wijd = aij , 1 6 d 6 D. In the multi-head attention case , the weight vector wij is group-wise same : wij is partitioned to M subvectors each corresponding to one attention head , { wij1 , wij2 , . . . , wijM } , and the elements in each subvector wijm are the same and are equal to the mth attention weight , aijm . Dynamic weight . The weights , { wi1 , wi2 , . . . , wiNk } , are dynamically predicted from the query xi and the keys { xi1 , xi2 , . . . , xiNk } in the local window as shown in Equation 2 . We rewrite it as : { wi1 , wi2 , . . . , wiNk } = f ( xi ; xi1 , xi2 , . . . , xiNk ) . ( 5 ) Each weight may obtain the information across all the channels in one head , and serves as a bridge to deliver the across-channel information to each output channel . Translation equivalence . Different from the convolution which satisfies translation equivalence through sharing weights across positions , the equivalence to translation for local attention , depends if the keys/values are changed , i.e. , the attention weights are changed , when the feature map is translated . In the case of sparsely-sampled window ( for run-time efficiency ) , e.g. , ( Hu et al. , 2019 ; Liu et al. , 2021b ; Ramachandran et al. , 2019 ; Vaswani et al. , 2021 ) , local attention is equivalent to block-wise translation , i.e. , the translation is a block or multiple blocks with the block size same as the window size Kw × Kh , and otherwise not equivalent ( as kyes/values are changed ) . In the case that the windows are densely sampled ( e.g. , ( Zhao et al. , 2020 ) ) , local attention is equivalent to translation . Set representation . The keys/values for one query are collected as a set with the spatial-order information lost . This leads to that the spatial correspondence between the keys/values across windows is not exploited . The order information loss is partially remedied by encoding the positions as embeddings ( Dosovitskiy et al. , 2021 ; Touvron et al. , 2020 ) , or learning a so-called relative position embedding ( e.g. , ( Liu et al. , 2021b ) ) in which the spatial-order information is preserved as the keys/values in a local window are collected as a vector . 2.4 CONNECTION TO DYNAMIC DEPTH-WISE CONVOLUTION . Depth-wise convolution is a type of convolution that applies a single convolutional filter for each channel : X̄d = Cd ⊗Xd , where Xd and X̄d are the dth input and output channel maps , Cd ∈ RNk is the corresponding kernel weight , and ⊗ is the convolution operation . It can be equivalently written in the form of element-wise multiplication for each position : yi = ∑Nk j=1 woffset ( i , j ) xij . ( 6 ) Here , offset ( i , j ) is the relative offset , from the 2D coordinate of the position j to the 2D coordinate of the central position i . The weights { woffset ( i , j ) ∈ RD ; j = 1 , 2 , . . . , Nk } are reshaped from C1 , C2 , . . . , CD . The Nk weight vectors are model parameters and shared for all the positions . We also consider two dynamic variants of depth-wise convolution : homogeneous and inhomogeneous2 . The homogeneous dynamic variant predicts the convolution weights using linear projections from a feature vector that is obtained by globally-pooling the feature maps : { w1 , w2 , . . . , wNk } = g ( GAP ( x1 , x2 , . . . , xN ) ) . ( 7 ) Here , { x1 , x2 , . . . , xN } are the image responses . GAP ( ) is the global average pooling operator . g ( ) is a function based on linear projection : a linear projection layer to reduce the channel dimension with BN and ReLU , followed by another linear projection to generate the connection weights . The inhomogeneous dynamic variant predicts the convolution weights separately for each position from the feature vector xi at the position ( the center of the window ) : { wi1 , wi2 , . . . , wiNk } = g ( xi ) . ( 8 ) This means that the weights are not shared across positions . We share the weights across the channels in a way similar to the multi-head attention mechanism to reduce the complexity . We describe the similarities and differences between ( dynamic ) depth-wise convolution and local attention . Figure 1 ( c ) illustrates the connectivity patterns and Table 1 shows the properties between local attention and depth-wise convolution , and various other modules . Similarity . Depth-wise convolution resembles local attention in sparse connectivity . There are no connections across channels . Each position is only connected to the positions in a small local window for each channel . Difference . One main difference lies in weight sharing : depth-wise convolution shares the connection weights across spatial positions , while local attention shares the weights across channels or within each group of channels . Local attention uses proper weight sharing across channels to get better performance . Depth-wise convolution benefits from the weight sharing across positions to reduce the parameter complexity and increase the network capability . The second difference is that the connection weights for depth-wise convolution are static and learned as model parameters , while the connection weights for local attention are dynamic and predicted from each instance . The dynamic variants of depth-wise convolution also benefit from the dynamic weight . One more difference lies in window representation . Local attention represents the positions in a window by utilizing a set form with spatial-order information lost . It explores the spatial-order 2The homogeneous version follows and applies the dynamic convolution to depth-wise convolution . The inhomogeneous version is close to involution ( Li et al. , 2021 ) and lightweight depth-wise convolution ( Wu et al. , 2019 ) information implicitly using the positional embedding or explicitly using the learned so-called relative positional embedding . Depth-wise convolution exploits a vector form : aggregate the representations within a local window with the weights indexed by the relative position ( see Equation 6 ) ; keep spatial correspondence between the positions for different windows , thus exploring the spatial-order information explicitly .

Summary: Recently local attention based vision transformers achieved state-of-the-art results on various visual recognition tasks. This paper rephrases local attention as a channel-wise spatially-locally connected layer with dynamic connection weights. By analyzing local attention form the view of sparse connectivity, weight sharing and dynamic weight computation, the paper discusses the similarities and differences between local attention and dynamic depth-wise convolution. Motivated by this connection, the paper experimentally compares local attention with depth-wise convolution and its dynamic variants on three vision tasks using the macro architecture of Swin-Transformer. The results show that dynamic depth-wise convolution-based models perform on par or better when compared to local attention while being computationally more efficient.