PKCAM: Previous Knowledge Channel Attention Module
Authors: Eslam Mohamed BAKR
Ahmad A. Al Sallab
Mohsen Rashwan
Venue: ICLR 2022 Submitted (2022)
Keywords: Channel Attention
Attention
Deep Learning
Computer Vision
Neural Networks.

Abstract: Attention mechanisms have been explored with CNNs, both across the spatial and channel dimensions. 
However, all the existing methods devote the attention modules to capture local interactions from the current feature map only, disregarded the valuable previous knowledge that is acquired by the earlier layers. 
This paper tackles the following question: Can one incorporate previous knowledge aggregation while learning channel attention more efficiently? To this end, we propose a Previous Knowledge Channel Attention Module( PKCAM), that captures channel-wise relations across different layers to model the global context. 
Our proposed module PKCAM is easily integrated into any feed-forward CNN architectures and trained in an end-to-end fashion with a negligible footprint due to its lightweight property. We validate our novel architecture through extensive experiments on image classification and object detection tasks with different backbones. 
Our experiments show consistent improvements in performances against their counterparts. We also conduct experiments that probe the robustness of the learned representations.

Under review as a conference paper at ICLR 2022
PKCAM: PREVIOUS KNOWLEDGE CHANNEL ATTEN- TION MODULE
Anonymous authors Paper under double-bling review
ABSTRACT
Attention mechanisms have been explored with CNNs, both across the spatial and channel dimensions. However, all the existing methods devote the attention modules capture local interactions from the current feature map only, disre- garded the valuable previous knowledge that is acquired by the earlier layers. This paper tackles the following question: Can one incorporate previous knowledge aggregation while learning channel attention more efficiently? To this end, we propose a Previous Knowledge Channel Attention Module(PKCAM), that cap- tures channel-wise relations across different layers model the global context. Our proposed module PKCAM is easily integrated into any feed-forward CNN architectures and trained in end-to-end fashion with negligible footprint due to its lightweight property. We validate our novel architecture through extensive experiments on image classification and object detection tasks with different back- bones. Our experiments show consistent improvements in performances against their counterparts. We also conduct experiments that probe the robustness of the learned representations.
INTRODUCTION
Over the years, CNN architectures have evolved with many ideas to better deal with spatial image features. Moreover, their localized nature makes such features lack the global view of the image. Deeper architectures emerged that stack multiple convolution layers, known with different names; backbone, bottleneck, feature extractor, or encoder. The main feature of such architectures their ability to cover spatial features at multiple scales. As we go deeper, the feature maps get smaller, while their content represents wider region in the space, which gets us closer to better semantics of the image contents Luo et al. (2016). With the emergence of AlexNet Krizhevsky et al. (2012). many kinds of research investigate to further improve the performance of deep CNNs. [Simonyan & Zisserman (2014) He et al. (2016) Szegedy et al. (2017) Szegedy et al. (006)) Srivastava et al. (2015) have sought to strengthen the CNNs by making it deeper and deeper as they have shown that increasing the depth of network could significantly increase the quality of the learned representations. Many researchers are continuously investigating to further improve the performance of deep CNNs by incorporating attention mechanisms to exploit its ability to cover the relationship between the learned spatial features.
Attention modules, in general, are designed to suppress noise while keeping useful information by refining the learned features using attention scaling. By quoting from the human perception process Mnih et al. (2014) where the high-level information is used in guiding the bottom-up learning process by capturing more sophisticated features while disregarding irrelevant details. Human perception and
visual attention BBek & Kastner (2009) Desimone/ (1998) Mnih et al. (2014) Desimone & Dunean (1995) is enhanced by top-down stimuli and non-re relevant neurons will be suppressed in feedback loops. Referencing to human visual system, various different attention mechanisms Cao et al. (2019)
Wang et al. (2018) Zhao et al. (2020) Vaswani et al. (2017) Wang et al. (2017) have been explored and integrated into deep CNNs. Attention mechanisms were introduced in the context of CNNs to
capture the relations between features, either across the spatial dimension as in Carion et al. (2020)
Hu et al. (2018a) ,or across channel-wist dimension as in Wang et al. 2018 Cao et al. (2019) Hu
et al. (2018b) Lee al. (2019) Wang et al. (2020) across both dimensions as in Park et al. (2018)
Woo et al. (2018) Fu et al. (2019) Wang et al. (2017) Linsley et al.) (2018) Roy et al. (2018) Chen
al." 2017). Although these attention methods have achieved higher accuracy than their counterpart

---

Under review as a conference paper at ICLR 2022
Table 1: Comparison of existing attention modules in terms whether previous knowledge cross- channel interaction (PKCCT), attention dimension, where C indicates channel attention and S indicates spatial attention, and lightweight or not.
Methods
SE
ECA
SRM
GSop GC GE
CBAM
BAM
DAN
GALA
RAN
PKCAM
PKCCI
X
X
x
<nx
44
X
x
X
x
X
x
/
Atlention
C+S
s
C+S
C+S
C+S
C+S
C
Lightweight
N/A
X
X
X
x
K
/
baselines which do not invoke any attention mechanisms in their architectures, they often bring higher model complexity and exploit only the current feature map while refining it, that's why we call it local attention mechanisms. Exploiting previous knowledge has been applied image classification Huang iet al. (2017) landola et al. (2014), image segmentation Ronneberger et al. (2015), tracking Ma et al. (2015). and human pose estimation Newell al. (0166) where they obtain enhanced performance. DenseNets Huang et al. (2017) encourage feature reuse by connecting each layer to every other layer in a feed-forward fashion. U-Net Ronneberger et al. (2015) consists of two paths, which are contracting path to capture context and symmetric expanding path that enables precise localization, where feature reuse is introduced through using skip connection between two paths. Driven by the significance of employing feature reuse while learning different tasks Szegedy et al. al.] (2015) Huang (ct al. (2017) landola et al. (2014) Chen et al. (2016) Ma et al. (2015) Newell et al. (2016), a question arises: How can one incorporate previous knowledge aggregation while learning channel attention more efficiently?
To answer this question, introduce PKCAM, novel feature recalibration module based on channel attention, which improves the quality of the representations produced by a network using the global information to selectively emphasize informative features and suppress less useful ones. In contrast to the aforementioned attention mechanisms, our global context aware attention block obtains additional inputs from all preceding attention blocks, that have the same depth, and passes on its refined feature-maps to all subsequent blocks, creating global awareness from exploiting previous knowledge aggregation from earlier layers that can capture fine-grained information which useful for precise localization while attending to features from earlier layers that can encode abstract semantic information, which is robust to target appearance changes.
The contributions of this paper are summarized as follows:
We propose simple and effective attention module, PKCAM, which can be integrated easily with any CNNs and applied across all it's blocks due to the lightweight computation of our novel architecture.
We verify the effectiveness and robustness of PKCAM throughout extensive experiments with various baseline architectures on multiple tasks and datasets.
Through detailed analysis along with ablation studies, we examine the internal behavior and validity of our method.
The rest of the paper is organized as follows, first, we discuss the related work, followed by the details of the proposed model. Then we present detailed ablation studies to settle the best architectural design, and finally, illustrate the experimental setup for the various experiments we conducted for every contribution.
RELATED WORK
Channel attention Hu et al. (201&b) proposed SE block, squeeze, and excitation block, which
comprises a lightweight gating mechanism that focuses on enhancing the representational power of
the network by modeling channel- relationships using two fully connected layers. ECA-Net Wang et al. (2020) empirically shows avoiding dimensionality reduction in Hu et al. (2018b) by
using a simple I-D convolution layer, is important for learning channel attention and appropriate cross-channel interaction. SRM Lee et al. (2019) proposes a Style- Recalibration Module, which adaptively recalibrates intermediate feature maps by exploiting their styles. Zhao et al. (2020) explore two variations of self-attention, pairwise and patchwise, that produce more powerful refined
2

---

Under review as a conference paper at ICLR 2022
EDA
Stage (I)
Stage (I+1)
fxtx
M
12126
TXRXC
Presiuu Knowfedge Aggrogation
Global Charnal Interection
Diagram of our Previous Knowledge Channel Attention Module (PKCAM). Given R aggregated features, PKCAM generates global aware channel weights by performing fast ID convolution of size R, accompanied by another ID convolution, which represents global cross channel interaction, then fused with the standard local cross channel interaction.
Figure 1:
features. The basic non-local block (NLB) Wang et al. (2018) aims at strengthening the features of the query position via aggregating information from other positions. GC-Ne(Cao et al. (2019) introduces abstract global context modeling framework, that could be summarized into blocks: context modeling and transform block besides proposing a simplified local network as the context modeling and SENet Hu al. (2018b) the transform block. GSoP al. (2019) obtain covariance matrix by exploiting holistic image information using global second-order pooling, which used for tensor scaling along channel dimension.
Spatial attention GENet Hu al. (2018a) consists of two operators that follow also the context modeling framework Cao et al. (2019), gather and excite operators. GENet Hu et al. (201 8a) uses stridden depth-wise convolution which acts as the gather operator, which applies spatial filters to independent channels of the input, and a simple excite operator consists of sigmoid function and multiplication. Spatial Transformer Networks Jaderberg al. (2015) tackle the lack of CNN ability to be spatially invariant to the input, by integrating a learnable module, the Spatial Transformer, which be inserted into CNNs, giving neural networks the ability to actively spatially transform feature maps, conditional on the feature map itself. DETR Carion et al. (2020) stacks a spatial transformer after the CNN backbone learn the interaction between each spatial position and its effect on different vision tasks, object detection, and instance segmentation.
Spatial and channel attention BAM Park al. (2018). CBAM Woo et al. (2018), DANet Fu et al. (2019), Residual attention network Wang et al. (2017), SCA-CNN
al. (2017), scSE Roy ctt al. (2018) and GALA Linsley et al. (2018) show that taking the spatial axis into consideration besides channel axis boost the attention module accuracy. Given intermediate feature map, they sequentially infer attention maps along two separate dimensions, channel and spatial, then the attention maps are multiplied to the input feature map for adaptive feature refinement.
Table T summarizes the existing attention modules in terms whether previous knowledge cross- channel interaction, attention type where C means channel attention mechanism is used, $ means spatial attention mechanism used and + S indicates that both attention mechanisms are used, and lightweight model.
METHODOLOGY
In this section, we first demonstrate an abstracted overview of PKCAM. Then, we demonstrate the motivations to adopt the feature reuse concept via exploiting the previous knowledge to create global aware attention block (i.e., PKCAM). Finally, dissecting our PKCAM by detailing it main blocks.
3.1 ABSTRACTED OVERVIEW OF OUR PKCAM
By scrutinizing the aforementioned channel attention techniques, as presented in Table previous knowledge aggregation not explored from the channel attention module perspective. Therefore we studied the previous knowledge cross-channe interaction by proposing PKCAM. The left part in Fig.?? demonstrates abstracted overview of our PKCAM, which exploits both local and global
3

---

Under review as a conference paper at ICLR 2022
feature maps while recalibrating the current feature map. PKCAM consists of two stacked modules: previous knowledge aggregation (PKA) and global cross channel interaction (GCCI. The PKA module covers the channel interactions across different preceding aggregated feature maps, while the GCCI module utilizes the refined features produce by the PKA module to model channel-wise relationships in a computationally efficient manner. GCCI modules could be any one of the on-the- shelf channel attention techniques which are studied in Section 2 Algorithm demonstrates a pseudo algorithm for our PKCAM module show how easily could be implemented and integrated to any CNN architecture.
3.2 PREVIOUS KNOWLEDGE AGGREGATION BLOCK
In contradiction to the aforementioned channel atten- tion techniques which relies on the current output of an arbitrary CNN block, our proposed PKCAM exploits both the current CNN block output To Ro and a range of carlier CNN blocks out- the coverage region that delimits how many previous CNN blocks output will be consolidated along side the current CNN block, I1 E 12 x2 22 IR H2 W2XC2 and ø E IR HaxWxCCa
Algorithm PKCAM module algorithm
Input: X: Current feature map.
put, X, [21, I2, -2,,,,,, ...; ...;,.., Øn]. where
P: List of preceding feature maps.
Output: Z1: Learned channel scales
1: R <- Length(Y)
2: for r in range (R) do
P(r) SDi(((r)
4: Pir + CDA(P(T)
Channel dimension alignment In general, the car-
end for
lier features Xp have different channel dimensions, as the conventional is as we go deeper the depth increased. Therefore, the first operation in our Previ- ous Knowledge A Aggregation(PKA block is aligning the channel dimension among different CNN blocks.
6: S + Stack Y)
1DConw(S (S, Kernel R) 8: Zi +- IDConw(Y, Kernel 3)
return 7.1
As C 2 C1 2 C2 2 CR. aligning operation can be done be learnable upsampling techniques simple repeating operation align with the channel dimension of the current CNN block Co. producing channel aligned feature maps, a' E HHaxWoxCG oi HHH, 2/2 E H2xxxxXCo and n'R E T TRHaxWaxxe
Spatial dimensions alignment Analogous to aligning the channel dimensions, the spatial dimen- sions; H and W, is aligned through squeeze operation by adapting the general global average pooling equation as follows, X = RWH E Σ Σ Xaig, where X E Rx1xIXC, and represents the squeezed feature maps from the channel aligned aggregated feature maps x/,, where i = 0, I..... R. producing X Fo ...... Fnl, To R RIxIxOo T1 RIXIIXC and žn E IR xxxxxo
Previous knowledge cross-channel attention Given the aggregated feature X, previous knowl- edge cross-channel attention can be learned by Y = f(X), where Y E RlxxxCo f(X) W and W" could take one of the following forms.
W
WI.CoO
W
W
W Wro.nne nca
RC W
W'=
1,
0
(1)
W
0
W
0
W RC..
where Wi is a RC, x RCs parameter matrix which learns previous knowledge interaction in conjunction with cross-channel interaction. In contrast W is 1 x RC, parameter matrix which learns previous knowledge interaction and channel interaction neglecting the cross channel relations. The key difference between W/ and W/2 is that W/ considers previous knowledge cross-channel
4

---

Under review as a conference paper at ICLR 2022
Table 2: Comparison of different previous knowledge Aggregation(PKA) techniques on the Tiny- ImageNet dataset. Where 1-D Conv., Sum and FC stands for one dimensional convolution layer Eq summation 2
and fully connected layer respectiveey.
I-D Cony.
1-D Conv.
1-D Conv.
PKA
Sum
FC
Sum
FC
Sum
FC
ResNet-18
ResNet-34
ResNet-50
55.70
55.28
54.63
56.94
56.26
56.52
57.89
56.18
56.41
H.P (M)
10.749
10.749
11.413
20.389
20.389
23.049
22.824
824
65.387
GFLOPss
2.075
2.075
2.076
4.329
4.329
4.331
4.878
4.878
4.878
interaction while W/y does not, leading W1 to be more complex than W Interpreting Eq. neural networks Wi and Wi2 can be regarded fully connected layer and depth-wise separable convolution layer respectively. However, obviously from Eq. [] W and W22 have tremendous number of parameters, driving high model complexity, especially for large channel numbers as mainly Co >> R.
Therefore, we divide learning the previous knowledge cross-channel interaction into two sub-modules as shown in Fig. [ learning previous knowledge interaction, and exploiting the cross-channel interaction. Consequently, in contrast to Eq. 1 f(X) is splitted into two cascaded functions, fi (X) and fx(X,, where (X) is responsible to learn the previous knowledge channel interaction and fe(X) is responsible to learn the cross-channel interaction.
Previous knowledge channel interaction Previous knowledge channel attention can be learned by Eq. 2 where for each channel the global information is aggregated using simple summation operation, where learnable parameters invoked.
Co
Y =
Σ
XLK
possible compromise between Eq. and Eq. is Eq. where a tiny number of parameters are used whereas W' I compared the tremendous number of parameters that are invoked in Eq. [ while learning the previous knowledge channel interaction. From the perspective of the convolution neural network, Eq. 3 could be readily interpreted a I-D convolution layer with kernel = W.
Co Y =(()
Σ
w W
X
Global cross-channel interaction
Global cross-channel interaction could be learned by adopting
one the local channel attention modules Hu et al. (2018b) Cao et al. (2019) Wang et al. (2018) Lee et al. (2019) Wang et al.
2020) Gao et al. (019 producing Z1
where Z E R Xxx
Wang et al. (2020) Gao et al.
2019) Hu et al. (2018b) Cao et al. (2019) Wang et al. (2018) Lee et al.
(2019) refer the term global they are taking into consideration the whole spatial dimension from
the fed features using GAP Global Average Pooling. In contrast we refer to the term global as
previous knowledge aggregation.
EXPERIMENTS
In this section, we perform controlled ablation experiments to settle on the best design for our proposed module and assess its sub-modules. Then we evaluate the performance of the proposed Previous Knowledge Attention module, on series of benchmark datasets across different tasks including Tiny- Image.NttNN Le & Yang (2013), i.e., discussed in the appendix, and ImageNet Deng et al. (2009) for the classification task. and KITTI Geiger al. (2012) for detection. Finally, We conduct empirical experiments that probe the robustness of the representations learned by PKCAM, compared to convolutional baselines and other attention mechanisms.
5

---

Under review as a conference paper at ICLR 2022
Table 3: Comparison of the various basic attention modules in our proposed module PKCAM.
Basic Allention Module
SE
SRM ECA
SE
SRM
ECA
Tiny-em
ImageNet
Resnet-18
54.07
55,10
55.70
70.65
70.75
70.83
Resnet-34
56.61
56.52
56.94
73.98
74.48
74.39
Resnct-50
57.42
57.12
57.89
76.86
77.42
77.56
4.1 IMPLEMENTATION DETAILS
For classification task two datasets are used, i.e., Tiny-ImageNet dataset Le & Yang (2015) and ImageNet dataset Deng et al. (2009), to evaluate our proposed module and show its effectiveness, where the same data augmentation and hyper-parameter settings in Hu et al. (2018b) are adopted. For the Imageee dataset Le & Yang (2013), input images are randomly cropped to 64 x 64 with random horizontal flipping. For the ImageNet dataset Deng et al. (2009). ) a 224 x 224 crop is randomly sampled from image or its horizontal flip, with the per-pixel RGB mean value subtracted. All models are trained for 100 epochs from scratch, using the weight initialization strategy described in He al. (2015) and the initial learning rate is set to 0.1 and decreased by factor of 10 every 30 epochs. Stochastic gradient descent (SGD) with weight decay of 1c-4, the momentum of 0.9, and mini-batch size of 32 is used for Imay- Le Yang (2015), and 256 for ImageNet Deng et al. (2009). Our module is implemented in Python using the Py Torchh framework using four PCs with Intel Xeon(R) 4108 1.8GHz CPU, 64G RAM, Nvidia Titan-XP.
4.2 ABLATION STUDIES FOR INTERNAL DESIGN
We have conducted two to settle on the best internal design and analyze the effectiveness of each component in our PKCAM. As shown in Fig. I our proposed module consists of two cascaded blocks, i.e., previous knowledge aggregation block (PKA) and global cross channel interaction block (GCCI). Therefore two ablations studies are conducted to settle on the best internal design and analyze the effectiveness of each one of them. The first ablation investigating the different approaches for previous knowledge channel interaction that are discussed in Section 3.2 Then, we assess the choice of basic attention modules that are used the global cross channel interaction modules as shown in Fig.
4.2.1 PREVIOUS KNOWLEDGE AGGREGATION
We have investigated empirically the different global channel interaction techniques that were described in detail in Sectior3.2] As shown in Tabld& exploiting the previous knowledge channel interaction through 1-D Conv. layer by following Eq is the best compromise to cope with the trade-off between performance and complexity, where shares almost the same model complexity (i.e., network parameters and FLOPs) with the original ResNet while the same time it boosts the accuracy. Based on the atorementioned results in Table 8 our novel approach follows the compromised combination while exploiting the previous knowledge cross channel interaction by following Eq to capture the previous knowledge channel interaction.
4.2.2 GLOBAL CROSS CHANNEL INTERACTION
We next assess the choice of basic attention modules that are used in the global cross channel interaction module as shown Fig Three channel attention mechanisms are evaluated on the Tiny- Imagenet dataset Le & Yang (2015) and ImageNet dataset, including SE-Net et al. (2018b), SRM Lee et al. (2019). and ECA-Net Wang et al. (2020). As shown in Table B ECA-Net achieves the best accuracy across different ResNet backbones. However, building up PKCAM using other channel attention mechanism boost the accuracy as shown in Table compared to their original results that mentioned in Table 9 and Table

---

Under review as a conference paper at ICLR 2022
Table 4: Comparison of the various ways to integrate PKCAM into CNNs using the ImageNet dataset.
Integration Methodology
All blocks
Last block
LCAM-PKCAM
PKCAM
Top-1
FPS
Top-1
FPS
Top-1
FPS
Top-1
FPS
Resnet-1k 18
70.78
112
70.78
138
70.78
138
70.83
160
Resnet-34
74.29
69
74.36
83
74.36
83
74.39
93
Resnet-50
77.19
42
77.22
69
77.22
69
77.56
74
Table 5: Comparisons with state-of-the-art attention modules on ImageNett in terms of the number of parameters (#P.) in millions, GFLOPs, top-1, and top-5 accuracy. Top-1 relative improvement results are reported between parentheses w.r. SENet improvement over Vanilla Resnet.
A.P.MM GFLOPs
Top-1
Top-5 +.P.MM: GFLOPs FPS
Top-1
Top-5 GFLOPs
Top-1
Top-5
ResNet-18
ResNet-34
ResNet-50
ReNNet
3.427
139
3.56
110
+SHNet
11:23
1,700)
70.59
89.78
20.93
3.428
79
73.87
91.65
26.77
1.87
76.71
93.38
-CBAM
11.23
1,700
92 (0.73(182%)
$9.91
20.94
3.428
48 74.01(125%)
91.76
26.77
3.57
36
77.34(1411)
93.69
-ECANet
11.14
1,699
179 (0.75(194%)
89.74
20.78
3.427
100 74.13(146%)
91.68
24.37
3.86
36
(7.39
93.60
PKCAM
11.14
70.83
89.96
20.78
3.427
93 74.39
24.17
3,86
77.56(156%)
4.3
ABLATION STUDIES FOR INTEGRATING PKCAM
4.3.1 DO WE NEED TO INJECT PKCAM AT EACH BLOCK?
The majority of CNN backbones have the same convention in their structure that consists of cascaded blocks that form a stage, then the basic structure of the stage is repeated with some modifications to form the backbone architecture. Driven by this, empirically demonstrate that, placing PKCAM in the last block of each stage only is sufficient to boost the accuracy by learning more powerful representations, and there is no need inject our module after each block re-calibrate its channels. Fig. ?? demonstrates the two different approaches. Injecting PKCAM at the last block will be sufficient it aggregate the whole previous knowledge within the same stage, exploiting this feature enables us to overcome all local attention modules in terms of model speed and complexity. Table 4 shows that injecting PKCAM the last block only superior injecting the whole blocks in terms of Top-1 accuracy and the inference time.
4.3.2 PKCAM Vs. LCAM
Local channel attention modules (LCAM) showed their ability in learning powerful representations, however exploiting the previous knowledge reinforcements the learned scales to be more representa- tive. Table demonstrates that our module powerful enough to totally replace the known LCAM, that are discussed at Sec. 2 instead of leveraging both of them. Upper part in Fig. ?? shows the fusion approach between ICAM and PKCAM, while the right part in Fig shows using PKCAM only while integrating an arbitrary CNN.
4.4 IMAGE CLASSIFICATION
In this section, we evaluate the performance of proposed PKCAM network on classification bench- mark datasets including ImageNet Deng et al. (2009) and Imay-mmaeNNee[Le & Yangl (2013). that is mentioned in the appendix. All the classification experiments follows the same training procedure that is discussed in Sec. [..] The evaluation metrics incorporate both efficiencies (i.e.. network parameters (#P.) in millions, inference frame rate per second (FPS), and floating point operations per second (FLOPs) in Gigas, and effectiveness (i.e., Top-1 accuracy).
ImageNet LSVRC 2012 dataset Deng et al. (2009), which contains 10s classes with 1.2 million training images, 50 x 10x validation images, and 100 103 test images. The evaluation is measured on the non-blacklist images of the ImageNet LSVRC 2012 validation set.
We compare our PKCAM module with several state-of-the attention methods using ResNet family. Efficiency and effectiveness are measured, and the results are reported in Table Table their original papers besides reproducing ECA-Net results as we notice there are difficulties in reproducing and
7

---

Under review as a conference paper at ICLR 2022
Table 6: Zero-shot testing: Analyzing the robustness of trained networks on
Robustnevs
Vanilla
SE
ECA
PKCAM
Vanilla
SE
ECA
PRCAM
Vanilla
SE
ECA
PRCAM
ResNei-18
ResNet-34
ReaNet-50
No-Rotation
53.33
53.71
53.76
55.70
55.90
56.08
55.66
56.94
56.11
57.78
56.59
57.89
90
20.56
20.60
21.80 80
21.83
21.68
21.75
22.70
22.71
23.04
25.42
25,53
25.68
180
25.85
26.59
26.60
27.57
27.57
31.74
270"
20.71
20.98
21.49
22.26
21.47
21.74
22.48
23.59
22.85
25.69
25.85
26./4
Table 7: Comparisons with state-of-the-art attention modules on KITTI-RGB in terms of mAP using YOLOV3 on Resnet-18 and 34 backbones.
Vanilla
ECA
CBAM
BAM
SRM
PKCAM
R-18
57.87
59.32
58.55
57.90
59.61
59.20
59.66
R-50
64.19
65.08
64.34
64.18
65.10
64.82
65.21
verifying their results] We adopt the same training setup He et al. 2016) Hu et al. (2018b) for fair comparison as discussed at Section 4.4 Results show that our proposed PKCAM achieves the best accuracy besides be the lightest model compared to other attention modules. Top-1 relative improvement results is reported between parentheses in green w.r.t SENet improvement over Vanilla Resnet.
4.5 OBJECT DETECTION
We evaluate our proposed PKCAM on object detection task using Faster R-CNN Ren et al. (2015) on the MS COCO dataset Lin al. (2014) and using YOLOV3 Redmon Farhadi (2018) on the KITTI-RGB dataset. KITTi-RGB Geiger et al. (2012) consists of 7,481 training images and 7,518 88 test images, comprising a total of 80,256 labeled objects of eight different classes. Each image has RGB color channels and pixel dimensions 1242 375 which is resized to 224 224. We follow the same training setup as mentioned at Section 4.4 As shown Table 7 PKCAM considerably improves the accuracy
than other attention modules compared the baseline He al.) (2016). The MMDetection Chen et al. (2019) framework used to guarantee fair comparison between different channel attention mechanisms
4.6 ROBUSTNESS
We have conducted experiments to probe the robustness of the representations learned by our proposed module PKCAM, compared to other channel attention mechanisms on Tiny-ImaeeNer (2015), where the testing images are rotated deliberately in one of three ways: clockwise 90", clockwise 180", clockwise 270", where these rotations were not scrutinized at the training. As shown in 6 PKCAM is less vulnerable than other attention modules.
Yang
DISCUSSION
5.1 TOP-1 I VALIDATION ACCURACY CURVES
To provide some insight into influence of our PKCAM module on the optimisation of these mod- els, example training curves for runs of the baseline architectures and their respective PKCAM counterparts are depicted in Fig. 2 We observe that PKCAM blocks yield a steady improvement throughout the optimisation procedure. Moreover, this trend is fairly consistent across range of network architectures considered as baselines.
5.2 DISSECTING THE PRODUCED LEARNABLE SCALES
To further analyze the effect of our PKCAM module on learning channel attention, we visualize the scales learned by novel PKCAM modules and compare it against ECA module. For this
Referring issues numher 21
52
62
24
58 from the official ECA-Net implementation.
8

---

Under review as a conference paper at ICLR 2022
Table 8: Comparisons with state-of-the-art attention modules on the MS-COCO dataset using Faster R-CNN detector based on Resnet-50 backbones.
Method
*.P.(M)
GFLOPs
FPS
AP
AP1s
APs
APM
APL
ResNet-50
41.53
207.07
36.4
58.2
39.2
21.8
40.0
46.2
+SENct
44.02
207.18
37.7
60.1
40.9
22.9
41.9
48.2
+ECANet
41.53
207.18
38.0
60.6
40.9
23.4
42.1
48.0
+PKCAM
41.53
207.18
10
38.3
60.9
41.0
23,9
42.4
48.2
Has Not-18
Hest 34
Rashet-50
Noae
9014
Epeth
Opech
Coech
Figure 2: Training ResNet, and local channel attention modules (LCA) baseline architectures and their PKCAM counterparts ImageNet validation set. PKCAM exhibit improved optimisation characteristics and produce consistent gains in performance which are sustained throughout the training process.
experiment, we adopt ResNet-18 18 as backbone models, and illustrate scales of first block only for each stage as discussed in Sec. 4.2 in the main script. In contrast to ECA setup, where a random sample consists of four classes only from ImageNet dataset are used, which are hammerhead shark, ambulance, medicine chest and butternut squash, we have used more generic and fair way to analyze the learned scales by averaging them over the whole validation dataset instead of using only four selected classes. Fig. 3 visualizes the channel learned scales for each first block from each stage for both modules; PKCAM in blue and ECA in orange.
As shown in Fig. ECA scales have larger variance than our learned scales, where ECA module's authors claim that indicates better discriminative ability which necessarily reflect the quality of the learned scales. Accordingly, a question is arising, is our main focus is to learn a discriminative scales, to learn more representative ones? We argue the claim that says the more discriminative scales indicates a more useful ones. As the most important a more representative scales not discriminative ones. This could be shown as all the learned channels important and we can't exclude any of them unlike the learned ECA scales which indicates that we can totally prune some of the learned scales, this shows deficiency in the learning process where not the full network capacity was exploited. Our scales are shown empirically that they are more representative as they boost the accuracy in consistent manner over different architectures and different tasks.
CONCLUSION
In this paper, we concentrate on determining an effective channel attention module with low model complexity. To this end, we propose efficient channel attention (PKCAM). Because of the lightweight
Mear
Magr
Mear
110
Figure 3: Comparison for the learned channel scales by our novel PKCAM module against ECA modules. Better view with zooming in.

---

Under review as a conference paper at ICLR 2022
computation of the PKCAM block, it can be integrated into all modern CNN architectures across the whole layers and trained end-to-end. While most previous works utilized uni-scale features, PKCAM is designed to employ the ability of global information while recalibrating feature maps. Our experiments demonstrate that simply inserting PKCAM into standard CNN architectures boosts the performance across different tasks. Furthermore, verify the robustness of the representations learned by PKCAM and its generalization ability via zero-shot experiments to rotated images.
AUTHOR CONTRIBUTIONS
you'd d like to, you may include a section for author contributions as is done in many journals. This optional and at the discretion of the authors.
ACKNOWLEDGMENTS
Use unnumbered third level headings for the acknowledgments. All acknowledgments, including those to funding agencies, go at the end of the paper.
REFERENCES
Diane Beck and Sabine Kastner. Top-down and bottom-up mechanisms in biasing competition in the human brain. Vision research, 154-)11154-15,,,,, 2009.
Yue Cao, Jiarui Xu, Stephen Lin, Fangyun Wei, and Han Hu. Genet: Non-local networks meet squeeze- excitation networks and beyond. In Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, pp. 0-0, 2019.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision, pp. 213-229. Springer, 2020.
Kai Chen, Jiaqi Wang, Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijie Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: 19990000155. 2019.
Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, and Alan L Yuille. Attention to scale: Scale-aware semantic image segmentation. In Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 3640-3649, 2016
Long Chen, Hanwang Zhang, Jun Xiao, Liqiang Nie, Jian Shao, Wei Liu, and Tat-Seng Chua. Sca-cnn: Spatial and channel- attention in convolutional networks for image captioning. In Proceedings of the IEEE conference computer vision and pultern recognition, pp. 5659-5667, 2017.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. Ieee, 2009.
Robert Desimone. Visual attention mediated by biased competition in extrastriate visual cortex. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 353 12353):1455-1255, 1998.
Robert Desimone and John Duncan. Neural mechanisms of selective visual attention, Annual review of neuroscience, 18(1):193-222, 1995.
Jun Fu, Jing Liu, Haijie Tian, Yong Li, Yongjun Bao, Zhiwei Fang, and Hanqing Lu. Dual attention network for scene segmentation. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 2146-3154, 2019.
Zilin Gao, Jiangtao Xie, Qilong Wang, and Peihua Li. Global second-order pooling convolutional net- works. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3024-3033, 2019.
10

---

Under review as a conference paper at ICLR 2022
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In Conference on Computer Vision and Pattern Recognition (CVPR), 2012.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Jie Hu, Li Shen, Samuel Albanie, Gang Sun, and Andrea Vedaldi. Gather-excite: Exploiting feature context in convolutional neural networks. arXiv preprint / 12348, 2018a.
Jie Hu, Shen, and Gang Sun. squeeze-and-excitane conference on computer vision and pattern recognition, pp. 7132-7141, 2018b.
networks. In Proceedings of the IEEE
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Forrest landola, Matt Moskewicz, Sergey Karayev, Ross Girshick, Trevor Darrell, and Kurt Keutzer. Densenet: Implementing efficient convnet descriptor pyramids. arXiv preprint arXiv: 1869, 2014.
Max Jaderberg. Karen Simonyan, Andrew Zisserman, al. Spatial transformer networks. In Advances in neural information processing systems, pp. 2017-2025, 2015.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015.
HyunJae Lee, Hyo-Eun Kim, and Hyconseob Nam. Srm: A style-based recalibration module for convolutional neural networks. In Proceedings the IEEE/CVF International Conference Computer Vision, pp. 1854-1862, 2019.
Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Drew Linsley, Dan Shiebler, Sven Eberhardt, and Thomas Serre. Learning what and where to attend. arXiv preprini arXiv: arXiv:1805.00 (e0::080559999 2018
Wenjie Luo, Yujia Li, Raquel Urtasun, and Richard Zemel. Understanding the effective receptive field in deep convolutional neural networks. Proceedings of the 30th International Conference on Neural Information Processing Systems, pp. 4905-4913, 2016.
Chao Ma, Jia-Bin Huang, Xiaokang Yang, and Ming-Hsuan Yang. Hierarchical convolutional features for visual tracking. In Proceedings of the IEEE international conference on computer vision, pp. 3074-3082, 2015.
Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu. Recurrent models of visual attention. arXiv preprini arXiv: 4406.6247, 2014.
Alejandro Newell, Kaiyu Yang, and Jia Deng. Stacked hourglass networks for human pose estimation. In European conference on computer vision, pp. 483-499. Springer, 2016.
Jongchan Park, Sanghyun Woo, Joon- Young Lee, and In So Kweon. Bam: Bottleneck attention module. arXiv preprint 1 2018.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv: (00::8804.000 02767, 2018.
11

---

Under review as a conference paper at ICLR 2022
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprint arXiv: 2015.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer. 2015.
Abhijit Guha Roy, Nassir Navab, and Christian Wachinger. Recalibrating fully convolutional networks with spatial and channel "squeeze and excitation" blocks. IEEE transactions medical imaging, 38(2):500-549 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.155: 1556, 2014.
Rupesh Kumar Srivastava, Klaus Greff, and Jürgen Schmidhuber. Training very deep networks. arXiv preprint arX1vv/507.062228, 2015.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 28 1-28266, 2016.
Christian Szegedy, Sergey Ioffe, Vincent Vanhoucke, and Alexander Alemi. Inception-v4, v4, inception- resnet and the impact of residual connections on learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 31, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Fei Wang, Mengqing Jiang, Chen Qian, Shuo Yang. Cheng Li, Honggang Zhang, Xiaogang Wang, and Xiaoou Tang. Residual attention network for image classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3156-3164, 2017.
Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu. Eca-net: Efficient channel attention for deep convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7794-7803, 2018.
Sanghyun Woo, Jongchan Park, Joon- attention module. In Proceedings of the European conference on computer vision (ECCV), pp. 3-19,2018.
Joon-Young Young Lee, and In So Kweon. Cham: Convolutional block
Hengshuang Zhao, Jiaya Jia, and Vladlen Koltun. Exploring self-attention for image recognition. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10076-10085, 2020.
A APPENDIX
In this section we will demonstrate experiments Iiny-ImageeNe dataset.
Tmay-ImageNet Le & Yang 2013) that is also called Micro-ImageNet, is a classification challenge that is similar to the full ImageNet ILSVRC challenge. Micro- Imcge-I comprises 200 classes. Each class has 500 images for training. The test set includes 10,000 images. 64 x 64 crop is randomly sampled from image or its horizontal flip, with the per-pixel RGB mean value subtracted.
12

---

Under review as a conference paper at ICLR 2022
Table 9: Comparisons with state-of-the-ar attention modules on Tiny-ImageNet in terms of the
number of parameters (ff..) in millions, GFLOPs, FPS, and top-1 accuracy.
Methods
#.P (M)
GFLOPs
FPS
Top-1
#.P (M)
GFLOPs
FPS
Top-1
#.P (M)
GFLOPs FPS
Top-1
ResNet-18
ResNet-34
ResNet-50
Vanilla
10.749
2.074
352
53.33
20.389
4.327
195
$5.90
22.802
4.861
SE
10.832
2.075
202
53.71
20.539
4.329
131
56.50
25.201
4.870
95
57.78
ECA
10.749
2.075
242
53.76
20.389
4.329
143
55.66
22.803
4.868
56.59
SRM
10.753
2.074
168
53.39
20.396
4.327
107
55.34
22.831
4.861
92
57.02
CBAM
10.835
2.075
132
$5.06
20.544
4.329
68
55.27
25.218
4.867
49
56. 76
BAM 10.771
2.080
201
55.00
20.411
4.333
145
55.97
23.144
4,962
119
58.37
PKCAM
10.749
2.075
215
55.70
20.389
4.329
136
56.94
22.803
4.868
101
57.89
We compare PKCAM module with several state-of-the-ard attention methods using three variants of ResNet backhone He et al. (2016) which are ResNet- 18, ResNet-34, and ResNet-50 on the Tiny- Imagenet dataset Le Yang 2015), including SENe(Hu LLe et al. (2019). CBAM Woo et al. (2018), and BAMPark et al. (018).. For a fair comparison, we ran all experiments by following the same setup that was mentioned above. As shown Table 9 our PKCAM module shares almost the same model complexity with the original ResNets variants He al. (016)) ResNet-18, ResNet-34, and ResNet-50, while achieving 2%, 0.7%, and 1.78% gains Top-1 1 accuracy. respectively. Comparing with state-of-the-art counterparts (i.e., SENe(Hu et al. (20T8b), ECANeeWar et al. 2020), SRM[Lee et al. (2019), CBAM al. (2018), and BAMPark al. (2018)), PKCAM obtains better competitive results while availing lower model complexity. Inference time measured on a PC equipped with TITAN Xp GPU and an Intel(R) Xcon Silver 4112
(2018b). ECANe (Wan al. (2020),
Please note that, Our code will be published once the paper is accepted, also it is attached in the supplementary material compressed file.
13


Reviews:
[{"review_id": "t243j0nI1G", "reviewer": "Reviewer_HH21", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is not well prepared for review with many incomplete and inconsistent expressions, e.g., the Fig.?? in Pages 3 and 7, the EMCA in Figure 2 and Figure 3 but the PKCAM in the captions, the CDA and SDA in Figure 1 without any explanation in the caption and text, the first SDA then CDA operation in both Figure 1 and Algorithm 1 but inverse order in Section 3.2, etc.\n\nThe key contribution of this work is to consider aggregating the previous feature maps from earlier CNN blocks, but this idea has already been proposed for CNNs such as the DenseNet. Besides, the implementation of PKCAM is not special and actually the commonly used spatial and channel attention. Thus the paper has limited technical novelty.\n\nFor the ablation study, most of the analysis is actually related to the current techniques used in attention design (see Table 2). Besides, most of the comparison results show that the PKCAM is not superior to the compared counterparts a lot (refer to Tables 4, 5, 7, and 8 as well as Figure 2). Therefore, the advantage of PKCAM is not obvious, that is, this work is not convincing empirically.\n\nFrom Figure 3, it can be seen that the PKCAM has a big difference with ECA, however, the quantitative comparison shows very small difference between PKCAM and ECANet, while the explanation of the last paragraph in Section 5 is not reasonable, especially how to evaluate the discriminative and representative ability of an attention module. \n\nThe paper slightly exceeds the paper limit of ICLR, and the writing needs to be carefully revised for typos and grammar errors.", "summary_of_the_paper": "This paper introduces an attention module, namely PKCAM for Previous Knowledge Channel Attention Module, which can be integrated into CNNs. The proposed PKCAM mainly aggregates the feature maps of earlier CNN blocks for aggregating previous knowledge. The ablation study is conducted for validating the effect of PKCAM in CNNs as well as its different design. Image classification and object detection experiments are considered for demonstrating the effectiveness of PKCAM in different ResNet architectures.", "summary_of_the_review": "The paper is not ready for review, also the contribution and analysis are limited. Thus, I vote for rejecting.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iAxDsbcJZXQ", "reviewer": "Reviewer_cAgA", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the descriptions about the core PKCAM module are unclear and seem to be incorrect. As shown on Page 4,  Y=W\\widetilde(X) and thus Y should be a feature vector with the dimension of RC_0. However, the authors claim that Y is a feature vector with dimension of C_0. Do I miss something? Besides, the authors state to split the function f() into two functions f_1() and f_2(). However, I can not find the definition of f_2(). \n\nThe authors split the PKCAN module into a summation followed by a linear mapping. It can indeed reduce the complexity. But why such simplicity works? More analysis should be provided.\n\nThe authors claim that the module can be easily integrated into different network architectures. However, they merely conduct experiments with the Reset series as baselines. I think the experiments with other architecture s such as mobile net should also be provided. Or the experiments can not support the statements.\n\nMore recent works [1, 2] also adopt channel and spatial attention for feature enhancement. These works should also be included for analysis and comparisons.\n\nI think the work is somewhat a draft for the current version. There are many reference errors throughout the paper, e.g., fig. ?? on Page 3 and fig. ?? on Page 7.\n[1] Hou et al., Coordinate Attention for Efficient Mobile Network Design Coordinate Attention, in CVPR, 2021.\n[2] Zhang et al., ResNeSt: Split-Attention Networks, in arXiv 2020.", "summary_of_the_paper": "This work proposes a previous knowledge channel attention module (PKCAM) that captures channel relations across the different layers to help enhance feature representation. The module can be integrated into the current ResNet series and show reasonable performance improvement over the baseline network on some benchmark datasets.", "summary_of_the_review": "I think the introduction about the PKCAM module is very unclear. The experiments are not sufficient and convincing.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Krjnv1xYj2X", "reviewer": "Reviewer_2Qrr", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n1. The previous feature maps are used to compute channel attention for the current feature map, showing some differences with existing methods. Besides, the proposed PKCAM is only applied to the end of the last stages, leading small parameters and FLOPs.\n2.\tThe proposed method seems easy to implement. \n\nWeaknesses:\n1.\tThe technical contribution of this work seems limited. The core of PKCAM is to exploit feature maps from previous layers for computing channel attention of the current feature map. For me, it is still not very clear why previous feature maps can help to compute channel attention of the current feature map. In other word, what are the merits brought by the previous feature maps? Furthermore, the core components of PKCAM (i.e., SDA and CDA) show similar with existing works, e.g., SE and ECA blocks.\n2.\tThe experimental results are not very convincing. Specifically,\n(1)\tThe improvement of PKCAM over existing methods is very marginal in terms of accuracy and model complexity, which hardly verify the effectiveness of previous knowledge.\n(2)\tExperiments could be further strengthened. More stronger backbones (e.g., ResNet-101 for classification and Faster R-CNN/ Mask R-CNN for object detection) could be used to further evaluate performance of the proposed method.\n(3)\tSome experimental results need further discussions. For example, why FPS of ECANet increases two times from ResNet-34 to ResNet-50, while others only increase less 1.5 times? Why LCAM+PKCAM is inferior to PKCAM? How to combine LCAM with PKCAM? What are the details of LCAM?\n(4)\tWhy PKCAM is only applied to the end of each stage? Why PKCAM only inserted into the last stage achieves the best results?\n3.\tThe writing could be further polished. Specifically,\n(1)\tIn Section 3.2, the details on how to select the previous knowledge seem missing, i.e., How to select R feature maps? and How about effect of number of R?\n(2)\tWhich method is used for channel dimension alignment?\n(3)\tThe orders of channel dimension alignment and spatial dimension alignment described in Eqn.(2) (from 1 to R) are inconsistent with those in Figure 1 (from 0 to R).\n(4)\tThe dimension of fully-connected layer in Eqn.(1) seems confusing.\n(5)\tThe sum operations in Eqn.(2) and Eqn.(3) are confusing.\n(6)\tMany citations are missing, e.g., Figure ???\n", "summary_of_the_paper": "This paper computes channel attention by considering feature maps across different layers, namely previous knowledge channel attention module (PKCAM).  The developed PKCAM is achieved by two steps: (1) the previous knowledge aggravation module is used to aggregate channel information of multiple feature images; and (2) the global cross channel interaction module models the correlation among channels to compute attention weights. The experiments are conducted on image classification and object detection tasks. ", "summary_of_the_review": "In my opinion, the technical contribution of this work seems limited, while experiments and writing could be further polished. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JnE2ujxRtn", "reviewer": "Reviewer_BFC2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposes an attention module with the knowledge from the previous layers also included. I have two concerns about this idea.\n\nFirstly, the idea to use the previous knowledge is not new. In the works of self-knowledge distillation, they use the previous layers knowledge to do the distillation (i.e. Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillation). So, the idea of utilizing previous knowledge is not new and thus the contribution, for me, is not enough.\n\nSecondly, the authors use the previous knowledge to build a better attention module. This idea is too intuitive. I don’t think the attention module has long-term relationship. \n\n\nSome small typos in writing:\n- The last line of page 3, there is something wrong with the figure (actually, all the citations for figures are wrong)\n- The second row of sec 3.2, which rely on…\n", "summary_of_the_paper": "This paper explores a new kind of attention mechanism, which can be easily integrated into any cnn models.\n\nThe proposed method is called previous knowledge channel attention module, whose aim is to make use of the knowledge from the previous layers.\n\nThe methods are validated on the image classification and object detection tasks. \n", "summary_of_the_review": "-\tWhat is the main contribution or motivation of this paper. If it is only about using the previous knowledge, it is not creative enough.\n-\tTry to explain or proof the working mechanism of the proposed PKCAM, which should be made more solid.\n-\tTry to polish the paper.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper computes channel attention by considering feature maps across different layers, and named it the previous knowledge channel attention module (PKCAM). The reviewers find the proposed idea too straightforward and naive. Lack of technical contribution is one of the major criticisms. There are also correctness concerns with the submission. The authors have not provided any rebuttal.

We recommend rejecting the paper.
Decision: Reject