PKCAM: Previous Knowledge Channel Attention Module

1 INTRODUCTION . Over the years , CNN architectures have evolved with many ideas to better deal with spatial image features . Moreover , their localized nature makes such features lack the global view of the image . Deeper architectures emerged that stack multiple convolution layers , known with different names ; backbone , bottleneck , feature extractor , or encoder . The main feature of such architectures is their ability to cover spatial features at multiple scales . As we go deeper , the feature maps get smaller , while their content represents a wider region in the space , which gets us closer to better semantics of the image contents Luo et al . ( 2016 ) . With the emergence of AlexNet Krizhevsky et al . ( 2012 ) , many kinds of research investigate to further improve the performance of deep CNNs . Simonyan & Zisserman ( 2014 ) He et al . ( 2016 ) Szegedy et al . ( 2017 ) Szegedy et al . ( 2016 ) Srivastava et al . ( 2015 ) have sought to strengthen the CNNs by making it deeper and deeper as they have shown that increasing the depth of a network could significantly increase the quality of the learned representations . Many researchers are continuously investigating to further improve the performance of deep CNNs by incorporating attention mechanisms to exploit its ability to cover the relationship between the learned spatial features . Attention modules , in general , are designed to suppress noise while keeping useful information by refining the learned features using attention scaling . By quoting from the human perception process Mnih et al . ( 2014 ) where the high-level information is used in guiding the bottom-up learning process by capturing more sophisticated features while disregarding irrelevant details . Human perception and visual attention Beck & Kastner ( 2009 ) Desimone ( 1998 ) Mnih et al . ( 2014 ) Desimone & Duncan ( 1995 ) is enhanced by top-down stimuli and non-relevant neurons will be suppressed in feedback loops . Referencing to human visual system , various different attention mechanisms Cao et al . ( 2019 ) Wang et al . ( 2018 ) Zhao et al . ( 2020 ) Vaswani et al . ( 2017 ) Wang et al . ( 2017 ) have been explored and integrated into deep CNNs . Attention mechanisms were introduced in the context of CNNs to capture the relations between features , either across the spatial dimension as in Carion et al . ( 2020 ) Hu et al . ( 2018a ) , or across channel-wise dimension as in Wang et al . ( 2018 ) Cao et al . ( 2019 ) Hu et al . ( 2018b ) Lee et al . ( 2019 ) Wang et al . ( 2020 ) or across both dimensions as in Park et al . ( 2018 ) Woo et al . ( 2018 ) Fu et al . ( 2019 ) Wang et al . ( 2017 ) Linsley et al . ( 2018 ) Roy et al . ( 2018 ) Chen et al . ( 2017 ) . Although these attention methods have achieved higher accuracy than their counterpart baselines which do not invoke any attention mechanisms in their architectures , they often bring higher model complexity and exploit only the current feature map while refining it , that ’ s why we call it local attention mechanisms . Exploiting previous knowledge has been applied to image classification Huang et al . ( 2017 ) Iandola et al . ( 2014 ) , image segmentation Ronneberger et al . ( 2015 ) , tracking Ma et al . ( 2015 ) , and human pose estimation Newell et al . ( 2016 ) where they obtain enhanced performance . DenseNets Huang et al . ( 2017 ) encourage feature reuse by connecting each layer to every other layer in a feed-forward fashion . U-Net Ronneberger et al . ( 2015 ) consists of two paths , which are contracting path to capture context and a symmetric expanding path that enables precise localization , where feature reuse is introduced through using skip connection between two paths . Driven by the significance of employing feature reuse while learning different tasks Szegedy et al . ( 2015 ) Huang et al . ( 2017 ) Iandola et al . ( 2014 ) Chen et al . ( 2016 ) Ma et al . ( 2015 ) Newell et al . ( 2016 ) , a question arises : How can one incorporate previous knowledge aggregation while learning channel attention more efficiently ? To answer this question , we introduce PKCAM , a novel feature recalibration module based on channel attention , which improves the quality of the representations produced by a network using the global information to selectively emphasize informative features and suppress less useful ones . In contrast to the aforementioned attention mechanisms , our global context aware attention block obtains additional inputs from all preceding attention blocks , that have the same depth , and passes on its refined feature-maps to all subsequent blocks , creating global awareness from exploiting previous knowledge aggregation from earlier layers that can capture fine-grained information which is useful for precise localization while attending to features from earlier layers that can encode abstract semantic information , which is robust to target appearance changes . The contributions of this paper are summarized as follows : • We propose a simple and effective attention module , PKCAM , which can be integrated easily with any CNNs and applied across all it ’ s blocks due to the lightweight computation of our novel architecture . • We verify the effectiveness and robustness of PKCAM throughout extensive experiments with various baseline architectures on multiple tasks and datasets . • Through detailed analysis along with ablation studies , we examine the internal behavior and validity of our method . The rest of the paper is organized as follows , first , we discuss the related work , followed by the details of the proposed model . Then we present detailed ablation studies to settle on the best architectural design , and finally , illustrate the experimental setup for the various experiments we conducted for every contribution . 2 RELATED WORK . Channel attention Hu et al . ( 2018b ) proposed SE block , squeeze , and excitation block , which comprises a lightweight gating mechanism that focuses on enhancing the representational power of the network by modeling channel-wise relationships using two fully connected layers . ECA-Net Wang et al . ( 2020 ) empirically shows avoiding dimensionality reduction in Hu et al . ( 2018b ) by using a simple 1-D convolution layer , is important for learning channel attention and appropriate cross-channel interaction . SRM Lee et al . ( 2019 ) proposes a Style-based Recalibration Module , which adaptively recalibrates intermediate feature maps by exploiting their styles . Zhao et al . ( 2020 ) explore two variations of self-attention , pairwise and patchwise , that produce more powerful refined features . The basic non-local block ( NLB ) Wang et al . ( 2018 ) aims at strengthening the features of the query position via aggregating information from other positions . GC-NetCao et al . ( 2019 ) introduces an abstract global context modeling framework , that could be summarized into two blocks : context modeling and transform block besides proposing a simplified local network as the context modeling and use SENet Hu et al . ( 2018b ) as the transform block . GSoP Gao et al . ( 2019 ) obtain a covariance matrix by exploiting holistic image information using global second-order pooling , which is used for tensor scaling along channel dimension . Spatial attention GENet Hu et al . ( 2018a ) consists of two operators that follow also the context modeling framework Cao et al . ( 2019 ) , gather and excite operators . GENet Hu et al . ( 2018a ) uses stridden depth-wise convolution which acts as the gather operator , which applies spatial filters to independent channels of the input , and a simple excite operator consists of sigmoid function and multiplication . Spatial Transformer Networks Jaderberg et al . ( 2015 ) tackle the lack of CNN ability to be spatially invariant to the input , by integrating a learnable module , the Spatial Transformer , which can be inserted into CNNs , giving neural networks the ability to actively spatially transform feature maps , conditional on the feature map itself . DETR Carion et al . ( 2020 ) stacks a spatial transformer after the CNN backbone to learn the interaction between each spatial position and its effect on different vision tasks , object detection , and instance segmentation . Spatial and channel attention BAM Park et al . ( 2018 ) , CBAM Woo et al . ( 2018 ) , DANet Fu et al . ( 2019 ) , Residual attention network Wang et al . ( 2017 ) , SCA-CNN Chen et al . ( 2017 ) , scSE Roy et al . ( 2018 ) and GALA Linsley et al . ( 2018 ) show that taking the spatial axis into consideration besides channel axis boost the attention module accuracy . Given an intermediate feature map , they sequentially infer attention maps along two separate dimensions , channel and spatial , then the attention maps are multiplied to the input feature map for adaptive feature refinement . Table 1 summarizes the existing attention modules in terms of whether previous knowledge crosschannel interaction , attention type where C means channel attention mechanism is used , S means spatial attention mechanism is used and C + S indicates that both attention mechanisms are used , and lightweight model . 3 METHODOLOGY . In this section , we first demonstrate an abstracted overview of our PKCAM . Then , we demonstrate the motivations to adopt the feature reuse concept via exploiting the previous knowledge to create a global aware attention block ( i.e. , PKCAM ) . Finally , dissecting our PKCAM by detailing it main blocks . 3.1 ABSTRACTED OVERVIEW OF OUR PKCAM . By scrutinizing the aforementioned channel attention techniques , as presented in Table 1 , previous knowledge aggregation was not explored from the channel attention module perspective . Therefore we studied the previous knowledge cross-channel interaction by proposing PKCAM . The left part in Fig. ? ? demonstrates an abstracted overview of our PKCAM , which exploits both local and global feature maps while recalibrating the current feature map . PKCAM consists of two stacked modules : previous knowledge aggregation ( PKA ) and global cross channel interaction ( GCCI ) . The PKA module covers the channel interactions across different preceding aggregated feature maps , while the GCCI module utilizes the refined features produce by the PKA module to model channel-wise relationships in a computationally efficient manner . GCCI modules could be any one of the on-theshelf channel attention techniques which are studied in Section 2 . Algorithm 1 demonstrates a pseudo algorithm for our PKCAM module to show how easily it could be implemented and integrated to any CNN architecture . 3.2 PREVIOUS KNOWLEDGE AGGREGATION BLOCK . Algorithm 1 PKCAM module algorithm Input : X : Current feature map . P : List of preceding feature maps . Output : Z1 : Learned channel scales 1 : R← Length ( Y ) 2 : for r in range ( R ) do 3 : P [ r ] ← SDA ( P [ r ] ) 4 : P [ r ] ← CDA ( P [ r ] ) 5 : end for 6 : S ← Stack ( X , Y ) 7 : Y ← 1DConv ( S , Kernel = R ) 8 : Z1 ← 1DConv ( Y , Kernel = 3 ) 9 : return Z1 In contradiction to the aforementioned channel attention techniques which relies on the current output of an arbitrary CNN block , our proposed PKCAM exploits both the current CNN block output , x0 ∈ RH0×W0×C0 , and a range of earlier CNN blocks output , Xp = [ x1 , x2 , ... , xR ] , where R is the coverage region that delimits how many previous CNN blocks output will be consolidated along side the current CNN block , x1 ∈ RH1×W1×C1 , x2 ∈ RH2×W2×C2 , and xR ∈ RHR×WR×CR . Channel dimension alignment In general , the earlier features Xp have different channel dimensions , as the conventional is as we go deeper the depth is increased . Therefore , the first operation in our Previous Knowledge Aggregation ( PKA ) block is aligning the channel dimension among different CNN blocks . As C0 ≥ C1 ≥ C2 ≥ CR , aligning operation can be done be learnable upsampling techniques or a simple repeating operation to align with the channel dimension of the current CNN block C0 , producing channel aligned feature maps , x′0 ∈ RH0×W0×C0 , x′1 ∈ RH1×W1×C0 , x′2 ∈ RH2×W2×C0 , and x′R ∈ RHR×WR×C0 . Spatial dimensions alignment Analogous to aligning the channel dimensions , the spatial dimensions ; H and W , is aligned through squeeze operation by adapting the general global average pooling equation as follows , X̃ = 1RWH ∑R k=1 ∑W i=1 ∑H j=1 Xkij , where X̃ ∈ RR×1×1×C0 and represents the squeezed feature maps from the channel aligned aggregated feature maps x′i , where i = 0 , 1 , . . . , R , producing X̃ = [ x̃0 , x̃1 , ... , x̃R ] , x̃0 ∈ R1×1×C0 , x̃1 ∈ R1×1×C0 and x̃R ∈ R1×1×C0 . Previous knowledge cross-channel attention Given the aggregated feature X̃ , previous knowledge cross-channel attention can be learned by Y = f ( X̃ ) , where Y ∈ R1×1×C0 , f ( X̃ ) = W ′X̃ , and W ′ could take one of the following forms , W ′ = W ′ 1 = W ′ 1,1 · · · W1 , RC0 ... . . . ... W ′ RC0,1 · · · W ′RC0 , RC0 W ′ 2 = W ′ 1,1 0 · · · 0 0 W ′ 2,2 · · · 0 ... ... . . . ... 0 0 · · · W ′RC0 , RC0 ( 1 ) where W ′1 is a RC0 × RC0 parameter matrix which learns previous knowledge interaction in conjunction with cross-channel interaction . In contrast W ′2 is a 1 × RC0 parameter matrix which learns previous knowledge interaction and channel interaction neglecting the cross channel relations . The key difference between W ′1 and W ′ 2 is that W ′ 1 considers previous knowledge cross-channel interaction while W ′2 does not , leading W ′ 1 to be more complex than W ′ 2 . Interpreting Eq . 1 to neural networks W ′1 and W ′ 2 can be regarded as a fully connected layer and depth-wise separable convolution layer respectively . However , obviously from Eq . 1 , W ′1 and W ′ 2 have a tremendous number of parameters , driving to high model complexity , especially for large channel numbers as mainly C0 > > R. Therefore , we divide learning the previous knowledge cross-channel interaction into two sub-modules as shown in Fig . 1 , learning previous knowledge interaction , and exploiting the cross-channel interaction . Consequently , in contrast to Eq . 1 , f ( X̃ ) is splitted into two cascaded functions , f1 ( X̃ ) and f2 ( X̃ ) , where f1 ( X̃ ) is responsible to learn the previous knowledge channel interaction and f2 ( X̃ ) is responsible to learn the cross-channel interaction . Previous knowledge channel interaction Previous knowledge channel attention can be learned by Eq . 2 , where for each channel the global information is aggregated using simple summation operation , where no learnable parameters are invoked . Y = f1 ( X̃ ) = C0∑ L=1 R∑ K=1 X̃LK ( 2 ) A possible compromise between Eq . 1 and Eq . 2 is Eq . 3 , where a tiny number of parameters are used whereas W ′ ∈ R1×1×R compared to the tremendous number of parameters that are invoked in Eq . 1 while learning the previous knowledge channel interaction . From the perspective of the convolution neural network , Eq . 3 could be readily interpreted to a 1-D convolution layer with kernel k = W̃ . Y = f1 ( X̃ ) = C0∑ L=1 W ′ X̃L ( 3 ) Global cross-channel interaction Global cross-channel interaction could be learned by adopting one of the local channel attention modules Hu et al . ( 2018b ) Cao et al . ( 2019 ) Wang et al . ( 2018 ) Lee et al . ( 2019 ) Wang et al . ( 2020 ) Gao et al . ( 2019 ) producing Z1 = f ( Y ) where Z1 ∈ R1×1×C0 . Wang et al . ( 2020 ) Gao et al . ( 2019 ) Hu et al . ( 2018b ) Cao et al . ( 2019 ) Wang et al . ( 2018 ) Lee et al . ( 2019 ) refer to the term global as they are taking into consideration the whole spatial dimension from the fed features using GAP - Global Average Pooling . In contrast we refer to the term global as previous knowledge aggregation .

Summary: This paper introduces an attention module, namely PKCAM for Previous Knowledge Channel Attention Module, which can be integrated into CNNs. The proposed PKCAM mainly aggregates the feature maps of earlier CNN blocks for aggregating previous knowledge. The ablation study is conducted for validating the effect of PKCAM in CNNs as well as its different design. Image classification and object detection experiments are considered for demonstrating the effectiveness of PKCAM in different ResNet architectures.