PKCAM: Previous Knowledge Channel Attention Module

1 INTRODUCTION . Over the years , CNN architectures have evolved with many ideas to better deal with spatial image features . Moreover , their localized nature makes such features lack the global view of the image . Deeper architectures emerged that stack multiple convolution layers , known with different names ; backbone , bottleneck , feature extractor , or encoder . The main feature of such architectures is their ability to cover spatial features at multiple scales . As we go deeper , the feature maps get smaller , while their content represents a wider region in the space , which gets us closer to better semantics of the image contents Luo et al . ( 2016 ) . With the emergence of AlexNet Krizhevsky et al . ( 2012 ) , many kinds of research investigate to further improve the performance of deep CNNs . Simonyan & Zisserman ( 2014 ) He et al . ( 2016 ) Szegedy et al . ( 2017 ) Szegedy et al . ( 2016 ) Srivastava et al . ( 2015 ) have sought to strengthen the CNNs by making it deeper and deeper as they have shown that increasing the depth of a network could significantly increase the quality of the learned representations . Many researchers are continuously investigating to further improve the performance of deep CNNs by incorporating attention mechanisms to exploit its ability to cover the relationship between the learned spatial features . Attention modules , in general , are designed to suppress noise while keeping useful information by refining the learned features using attention scaling . By quoting from the human perception process Mnih et al . ( 2014 ) where the high-level information is used in guiding the bottom-up learning process by capturing more sophisticated features while disregarding irrelevant details . Human perception and visual attention Beck & Kastner ( 2009 ) Desimone ( 1998 ) Mnih et al . ( 2014 ) Desimone & Duncan ( 1995 ) is enhanced by top-down stimuli and non-relevant neurons will be suppressed in feedback loops . Referencing to human visual system , various different attention mechanisms Cao et al . ( 2019 ) Wang et al . ( 2018 ) Zhao et al . ( 2020 ) Vaswani et al . ( 2017 ) Wang et al . ( 2017 ) have been explored and integrated into deep CNNs . Attention mechanisms were introduced in the context of CNNs to capture the relations between features , either across the spatial dimension as in Carion et al . ( 2020 ) Hu et al . ( 2018a ) , or across channel-wise dimension as in Wang et al . ( 2018 ) Cao et al . ( 2019 ) Hu et al . ( 2018b ) Lee et al . ( 2019 ) Wang et al . ( 2020 ) or across both dimensions as in Park et al . ( 2018 ) Woo et al . ( 2018 ) Fu et al . ( 2019 ) Wang et al . ( 2017 ) Linsley et al . ( 2018 ) Roy et al . ( 2018 ) Chen et al . ( 2017 ) . Although these attention methods have achieved higher accuracy than their counterpart baselines which do not invoke any attention mechanisms in their architectures , they often bring higher model complexity and exploit only the current feature map while refining it , that ’ s why we call it local attention mechanisms . Exploiting previous knowledge has been applied to image classification Huang et al . ( 2017 ) Iandola et al . ( 2014 ) , image segmentation Ronneberger et al . ( 2015 ) , tracking Ma et al . ( 2015 ) , and human pose estimation Newell et al . ( 2016 ) where they obtain enhanced performance . DenseNets Huang et al . ( 2017 ) encourage feature reuse by connecting each layer to every other layer in a feed-forward fashion . U-Net Ronneberger et al . ( 2015 ) consists of two paths , which are contracting path to capture context and a symmetric expanding path that enables precise localization , where feature reuse is introduced through using skip connection between two paths . Driven by the significance of employing feature reuse while learning different tasks Szegedy et al . ( 2015 ) Huang et al . ( 2017 ) Iandola et al . ( 2014 ) Chen et al . ( 2016 ) Ma et al . ( 2015 ) Newell et al . ( 2016 ) , a question arises : How can one incorporate previous knowledge aggregation while learning channel attention more efficiently ? To answer this question , we introduce PKCAM , a novel feature recalibration module based on channel attention , which improves the quality of the representations produced by a network using the global information to selectively emphasize informative features and suppress less useful ones . In contrast to the aforementioned attention mechanisms , our global context aware attention block obtains additional inputs from all preceding attention blocks , that have the same depth , and passes on its refined feature-maps to all subsequent blocks , creating global awareness from exploiting previous knowledge aggregation from earlier layers that can capture fine-grained information which is useful for precise localization while attending to features from earlier layers that can encode abstract semantic information , which is robust to target appearance changes . The contributions of this paper are summarized as follows : • We propose a simple and effective attention module , PKCAM , which can be integrated easily with any CNNs and applied across all it ’ s blocks due to the lightweight computation of our novel architecture . • We verify the effectiveness and robustness of PKCAM throughout extensive experiments with various baseline architectures on multiple tasks and datasets . • Through detailed analysis along with ablation studies , we examine the internal behavior and validity of our method . The rest of the paper is organized as follows , first , we discuss the related work , followed by the details of the proposed model . Then we present detailed ablation studies to settle on the best architectural design , and finally , illustrate the experimental setup for the various experiments we conducted for every contribution . 2 RELATED WORK . Channel attention Hu et al . ( 2018b ) proposed SE block , squeeze , and excitation block , which comprises a lightweight gating mechanism that focuses on enhancing the representational power of the network by modeling channel-wise relationships using two fully connected layers . ECA-Net Wang et al . ( 2020 ) empirically shows avoiding dimensionality reduction in Hu et al . ( 2018b ) by using a simple 1-D convolution layer , is important for learning channel attention and appropriate cross-channel interaction . SRM Lee et al . ( 2019 ) proposes a Style-based Recalibration Module , which adaptively recalibrates intermediate feature maps by exploiting their styles . Zhao et al . ( 2020 ) explore two variations of self-attention , pairwise and patchwise , that produce more powerful refined features . The basic non-local block ( NLB ) Wang et al . ( 2018 ) aims at strengthening the features of the query position via aggregating information from other positions . GC-NetCao et al . ( 2019 ) introduces an abstract global context modeling framework , that could be summarized into two blocks : context modeling and transform block besides proposing a simplified local network as the context modeling and use SENet Hu et al . ( 2018b ) as the transform block . GSoP Gao et al . ( 2019 ) obtain a covariance matrix by exploiting holistic image information using global second-order pooling , which is used for tensor scaling along channel dimension . Spatial attention GENet Hu et al . ( 2018a ) consists of two operators that follow also the context modeling framework Cao et al . ( 2019 ) , gather and excite operators . GENet Hu et al . ( 2018a ) uses stridden depth-wise convolution which acts as the gather operator , which applies spatial filters to independent channels of the input , and a simple excite operator consists of sigmoid function and multiplication . Spatial Transformer Networks Jaderberg et al . ( 2015 ) tackle the lack of CNN ability to be spatially invariant to the input , by integrating a learnable module , the Spatial Transformer , which can be inserted into CNNs , giving neural networks the ability to actively spatially transform feature maps , conditional on the feature map itself . DETR Carion et al . ( 2020 ) stacks a spatial transformer after the CNN backbone to learn the interaction between each spatial position and its effect on different vision tasks , object detection , and instance segmentation . Spatial and channel attention BAM Park et al . ( 2018 ) , CBAM Woo et al . ( 2018 ) , DANet Fu et al . ( 2019 ) , Residual attention network Wang et al . ( 2017 ) , SCA-CNN Chen et al . ( 2017 ) , scSE Roy et al . ( 2018 ) and GALA Linsley et al . ( 2018 ) show that taking the spatial axis into consideration besides channel axis boost the attention module accuracy . Given an intermediate feature map , they sequentially infer attention maps along two separate dimensions , channel and spatial , then the attention maps are multiplied to the input feature map for adaptive feature refinement . Table 1 summarizes the existing attention modules in terms of whether previous knowledge crosschannel interaction , attention type where C means channel attention mechanism is used , S means spatial attention mechanism is used and C + S indicates that both attention mechanisms are used , and lightweight model . 3 METHODOLOGY . In this section , we first demonstrate an abstracted overview of our PKCAM . Then , we demonstrate the motivations to adopt the feature reuse concept via exploiting the previous knowledge to create a global aware attention block ( i.e. , PKCAM ) . Finally , dissecting our PKCAM by detailing it main blocks . 3.1 ABSTRACTED OVERVIEW OF OUR PKCAM . By scrutinizing the aforementioned channel attention techniques , as presented in Table 1 , previous knowledge aggregation was not explored from the channel attention module perspective . Therefore we studied the previous knowledge cross-channel interaction by proposing PKCAM . The left part in Fig. ? ? demonstrates an abstracted overview of our PKCAM , which exploits both local and global feature maps while recalibrating the current feature map . PKCAM consists of two stacked modules : previous knowledge aggregation ( PKA ) and global cross channel interaction ( GCCI ) . The PKA module covers the channel interactions across different preceding aggregated feature maps , while the GCCI module utilizes the refined features produce by the PKA module to model channel-wise relationships in a computationally efficient manner . GCCI modules could be any one of the on-theshelf channel attention techniques which are studied in Section 2 . Algorithm 1 demonstrates a pseudo algorithm for our PKCAM module to show how easily it could be implemented and integrated to any CNN architecture . 3.2 PREVIOUS KNOWLEDGE AGGREGATION BLOCK . Algorithm 1 PKCAM module algorithm Input : X : Current feature map . P : List of preceding feature maps . Output : Z1 : Learned channel scales 1 : R← Length ( Y ) 2 : for r in range ( R ) do 3 : P [ r ] ← SDA ( P [ r ] ) 4 : P [ r ] ← CDA ( P [ r ] ) 5 : end for 6 : S ← Stack ( X , Y ) 7 : Y ← 1DConv ( S , Kernel = R ) 8 : Z1 ← 1DConv ( Y , Kernel = 3 ) 9 : return Z1 In contradiction to the aforementioned channel attention techniques which relies on the current output of an arbitrary CNN block , our proposed PKCAM exploits both the current CNN block output , x0 ∈ RH0×W0×C0 , and a range of earlier CNN blocks output , Xp = [ x1 , x2 , ... , xR ] , where R is the coverage region that delimits how many previous CNN blocks output will be consolidated along side the current CNN block , x1 ∈ RH1×W1×C1 , x2 ∈ RH2×W2×C2 , and xR ∈ RHR×WR×CR . Channel dimension alignment In general , the earlier features Xp have different channel dimensions , as the conventional is as we go deeper the depth is increased . Therefore , the first operation in our Previous Knowledge Aggregation ( PKA ) block is aligning the channel dimension among different CNN blocks . As C0 ≥ C1 ≥ C2 ≥ CR , aligning operation can be done be learnable upsampling techniques or a simple repeating operation to align with the channel dimension of the current CNN block C0 , producing channel aligned feature maps , x′0 ∈ RH0×W0×C0 , x′1 ∈ RH1×W1×C0 , x′2 ∈ RH2×W2×C0 , and x′R ∈ RHR×WR×C0 . Spatial dimensions alignment Analogous to aligning the channel dimensions , the spatial dimensions ; H and W , is aligned through squeeze operation by adapting the general global average pooling equation as follows , X̃ = 1RWH ∑R k=1 ∑W i=1 ∑H j=1 Xkij , where X̃ ∈ RR×1×1×C0 and represents the squeezed feature maps from the channel aligned aggregated feature maps x′i , where i = 0 , 1 , . . . , R , producing X̃ = [ x̃0 , x̃1 , ... , x̃R ] , x̃0 ∈ R1×1×C0 , x̃1 ∈ R1×1×C0 and x̃R ∈ R1×1×C0 . Previous knowledge cross-channel attention Given the aggregated feature X̃ , previous knowledge cross-channel attention can be learned by Y = f ( X̃ ) , where Y ∈ R1×1×C0 , f ( X̃ ) = W ′X̃ , and W ′ could take one of the following forms , W ′ = W ′ 1 = W ′ 1,1 · · · W1 , RC0 ... . . . ... W ′ RC0,1 · · · W ′RC0 , RC0 W ′ 2 = W ′ 1,1 0 · · · 0 0 W ′ 2,2 · · · 0 ... ... . . . ... 0 0 · · · W ′RC0 , RC0 ( 1 ) where W ′1 is a RC0 × RC0 parameter matrix which learns previous knowledge interaction in conjunction with cross-channel interaction . In contrast W ′2 is a 1 × RC0 parameter matrix which learns previous knowledge interaction and channel interaction neglecting the cross channel relations . The key difference between W ′1 and W ′ 2 is that W ′ 1 considers previous knowledge cross-channel interaction while W ′2 does not , leading W ′ 1 to be more complex than W ′ 2 . Interpreting Eq . 1 to neural networks W ′1 and W ′ 2 can be regarded as a fully connected layer and depth-wise separable convolution layer respectively . However , obviously from Eq . 1 , W ′1 and W ′ 2 have a tremendous number of parameters , driving to high model complexity , especially for large channel numbers as mainly C0 > > R. Therefore , we divide learning the previous knowledge cross-channel interaction into two sub-modules as shown in Fig . 1 , learning previous knowledge interaction , and exploiting the cross-channel interaction . Consequently , in contrast to Eq . 1 , f ( X̃ ) is splitted into two cascaded functions , f1 ( X̃ ) and f2 ( X̃ ) , where f1 ( X̃ ) is responsible to learn the previous knowledge channel interaction and f2 ( X̃ ) is responsible to learn the cross-channel interaction . Previous knowledge channel interaction Previous knowledge channel attention can be learned by Eq . 2 , where for each channel the global information is aggregated using simple summation operation , where no learnable parameters are invoked . Y = f1 ( X̃ ) = C0∑ L=1 R∑ K=1 X̃LK ( 2 ) A possible compromise between Eq . 1 and Eq . 2 is Eq . 3 , where a tiny number of parameters are used whereas W ′ ∈ R1×1×R compared to the tremendous number of parameters that are invoked in Eq . 1 while learning the previous knowledge channel interaction . From the perspective of the convolution neural network , Eq . 3 could be readily interpreted to a 1-D convolution layer with kernel k = W̃ . Y = f1 ( X̃ ) = C0∑ L=1 W ′ X̃L ( 3 ) Global cross-channel interaction Global cross-channel interaction could be learned by adopting one of the local channel attention modules Hu et al . ( 2018b ) Cao et al . ( 2019 ) Wang et al . ( 2018 ) Lee et al . ( 2019 ) Wang et al . ( 2020 ) Gao et al . ( 2019 ) producing Z1 = f ( Y ) where Z1 ∈ R1×1×C0 . Wang et al . ( 2020 ) Gao et al . ( 2019 ) Hu et al . ( 2018b ) Cao et al . ( 2019 ) Wang et al . ( 2018 ) Lee et al . ( 2019 ) refer to the term global as they are taking into consideration the whole spatial dimension from the fed features using GAP - Global Average Pooling . In contrast we refer to the term global as previous knowledge aggregation .

Summary: This paper computes channel attention by considering feature maps across different layers, namely previous knowledge channel attention module (PKCAM).  The developed PKCAM is achieved by two steps: (1) the previous knowledge aggravation module is used to aggregate channel information of multiple feature images; and (2) the global cross channel interaction module models the correlation among channels to compute attention weights. The experiments are conducted on image classification and object detection tasks.