GhostNetV2: Enhance Cheap Operation with Long-Range Attention
Authors: Yehui Tang
Kai Han
Jianyuan Guo
Chang Xu
Chao Xu
Yunhe Wang
Venue: NeurIPS 2022 Accept (2022)

Abstract: Light-weight convolutional neural networks (CNNs) are specially designed for applications on mobile devices with faster inference speed. The convolutional operation can only capture local information in a window region, which prevents  performance from being further improved. Introducing self-attention into convolution can capture global information well, but it will largely encumber the actual speed. In this paper, we propose a hardware-friendly attention mechanism (dubbed DFC attention) and then present a new GhostNetV2 architecture for mobile applications. The proposed DFC attention is constructed based on fully-connected layers, which can not only execute fast on common hardware but also capture the dependence between long-range pixels. We further revisit the expressiveness bottleneck in previous GhostNet and propose to enhance expanded features produced by cheap operations with DFC attention, so that a GhostNetV2 block can aggregate local and long-range information simultaneously. Extensive experiments demonstrate the superiority of GhostNetV2 over existing architectures. For example, it achieves 75.3% top-1 accuracy on ImageNet with 167M FLOPs, significantly suppressing GhostNetV1 (74.5%) with a similar computational cost. The source code will be available at https://github.com/huawei-noah/Efficient-AI-Backbones/tree/master/ghostnetv2_pytorch and https://gitee.com/mindspore/models/tree/master/research/cv/ghostnetv2. 

GhostNetV2: Enhance Cheap Operation with Long-Range Attention
Yehui Tang Kai Han', Jianyuan Guo Chang Xu', Chao Xu', Yunhe Wang"* School of Artificial Intelligence, Peking University Huawei Noah's Ark Lab "School of Computer Science, University of Sydney yhtang pku.edu..nn (kai.han, yunhe. wang) @huawei.com
Abstract
Light-weight convolutional neural networks (CNNs) are specially designed for applications mobile devices with faster inference speed. The convolutional operation can only capture local information in a window region, which prevents performance from being further improved. Introducing self-attention into convolu- tion can capture global information well, but it will largely encumber the actual speed. In this paper, we propose hardware-friendly attention mechanism (dubbed DFC attention) and then present a new GhostNetV2 architecture for mobile applica- tions. The proposed DFC attention is constructed based fully-connected layers, which can not only execute fast on common hardware but also capture the depen- dence between long-range pixels. We further revisit the expressiveness bottleneck in previous GhostNet and propose to enhance expanded features produced by cheap operations with DFC attention, so that GhostNetV2 block can aggregate local and long-range information simultaneously. Extensive experiments demonstrate the su- periority of GhostNetV2 over existing architectures. For example, it achieves 75.3% top-1 accuracy on ImageNet with 167M FLOPs, significantly suppressing Ghost- NetVI (74.5% with a similar computational cost. The source code will be avail- able at https: thub. hww.w/wwwi.....cotoet.tw.......a -tt.e//eaaaiiiiiitt------------- comptiiseremmmmiinppopre
tree/ htree/masscrr/ggotttteeeppppooor and hidel///teeenaserr///esrrc//////
1 Introduction
In computer vision, the architecture of deep neural network plays vital role for various tasks, such as image classification 19 10. object detection [27 26), and video analysis [18]. the past decade,
the network architecture has been evolving rapidly, and series of milestones including AlexNet [19]. GoogleNet [29], ResNet [10] and EfficientNet [32] have been developed. These networks have
pushed the performances of wide range of visual tasks to high level.
To deploy neural networks on edge devices like smartphone and wearable devices, need to consider not only the performance of a model, but also its efficiency especially the actual inference speed. Matrix multiplications occupy the main part of computational cost and parameters. Developing light- weight models is a promising approach to reduce the inference latency. MobileNet [13] factorizes a standard convolution into depthwise convolution and point-wise convolution, which reduces the computational cost drastically. MobileNerV2 (28) and MobileNerV3 [12] further introduce the inverted residual block and improve the network architecture. ShuffleNet utilizes the shuffle operation to encourage the information exchange between channel groups, GhostNet [8] proposes
the cheap operation to reduce feature redundancy channels. WaveMLP 33 replaces the complex
"Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

78-
78
77-
77
GhostMitV2
Accuray (80)
76
GhostNett1
Accuray (%)
NobileNetv
GhostNetvz
75
EfficietNN
Motiloleava
Shuffehat22
x
-E) MobiliNewv
9 Mnashet
-+ FBNct
FBNet
Misssett
ProxylesMMS
ProcylessNAS
450
500
50
60
70
FLOPs
Lateney (ms)
Figure 1: Top-1 accuracy VL.PS on ImageNet Figure 2: Top-1 accuracy vs. latency on ImageNet dataset.
dataset.
self-attention module with simple Multi-Layer Perceptron (MLP) to reduce the computational cost. These light-weigh, neural networks have been applied in many mobile applications.
Nevertheless, the convolution-bass light-weight models are weak in modeling long-range depen- dency, which limits further performance improvement. Recently, transformer-like models are intro- duced to computer vision, in which the self-attention module capture the global information. The typical self-attention module requires quadratic complexity w.n.t. the size of feature's 's shape and is not computationally friendly. Moreover, plenty of feature splitting and reshaping operations are required to calculate the attention map. Though their theoretical complexity is negligible, these operations incur more memory usage and longer latency in practice. Thus, utilizing vanilla self-attention in wighttteight models is not friendly for mobile deployment. For example, MobileViT with massive self-attention operations more than 7x slower than MobileNetV2 on ARM devices 23
In this paper, we propose a new attention mechanism (dubbed DFC attention) to capture the long- range spatial information, while keeping the implementation efficiency of light- convolutional neural networks. Only fully connected (FC) layers participate in generating the attention maps for simplicity. Specifically, FC layer decomposed into horizontal FC and vertical FC aggregate pixels in 2D feature map of CNN. The two FC layers involve pixels in long range along their respective directions, and stacking them will produce global receptive field. Moreover, starting from ate-of-the-art GhostNet, we revisit its representation bottleneck and enhance the intermediate features with the DFC attention. Then we construct new light-weight vision backbone, GhostNetV2. Compared with the existing architectures, it can achieve better tread-off between accuracy and inference speed (as shown in Figures and 2p.
2
Related Work
is a challenge to design a light-weight neural architecture with fast inference speed and high performance simultaneously [16] (1)41333 40 35). SqueezeNet [16] proposes three strategies to design a compact model, i.e., replacing 3 x filters with x filers, decreasing the number of input channels 3z3 filters, and down-sampling late in the network to keep large feature maps. These principles are constructive, especially the usage of 1 x 1 convolution. MobileNetV] (13 replaces almost all the 3 x filers with 1 kernel and depth-wise separable convolutions, which dramatically reduces the computational cost. MobileNetV2 (28) further introduces the residual connection to the light-weight model, and constructs an inverted residual structure, where the intermediate layer of block has more channels than its input and output. To keep representation ability, part of non-linear functions are removed. MobileNeXt 44 rethinks the necessary of inverted bottleneck, and claims that the classic bottleneck structure can also achieve high performance. Considering the 1 x 1 convolution account for a substantial part of computational cost, ShuffleNet (42 replace with group convolution. The channel shuffle operation to help the information flowing across different groups. By investigating the factors that affect practical running speed, ShuffleNet V2 (22 proposes a hardware-friende new block. By leveraging the feature's 's redundancy, GhostNet g replaces half channels in 1 x 1 convolution with cheap operations. Until now, GhostNet has been the SOTA light-weight model with good trade-off between accuracy and speed.

---

Besides manual design, series of methods try to search for wight- architecture. For example, FBNet [3 designs hardware-aware searching strategy, which can directly find a good trade-off between accuracy and speed on a specific hardware. Based on the inverted residual bottleneck, MnasNet [31, MobileNetV3 [2 search the architecture parameters,such as model width, model depth, convolutional filter's size, etc. Though NAS based methods achieve high performance, their success is based on well-designed search spaces and architectural units. Automatic searching and manual design can be combined to find better architecture.
3 Preliminary
3.1 A Brief Review of GhostNet
GhostNet [8] is SOTA light-weight model designed for efficient inference mobile devices. Its main component is the Ghost module, which can replace the original convolution by generating more feature maps from cheap operations. Given input feature X E I xxxx xxx with height H, width W and channel's 's number C, a typical Ghost module can replace a standard convolution by two steps. Firstly, a 1 x 1 convolution used generate the intrinsic feature, i.e.
x
(1)
where * denotes the convolution operation. Fix1 is the point-wise convolution, and Y' R W x.u. is the intrinsic features, whose sizes are usually smaller than the original output features, Le., C.us < Cout- Then cheap operations (e.g,, depth-wise convolution) used to generate more features based on the intrinsic features. The two parts of features are concatenated along the channel dimension, i.e.,
Y == Concat((Y', Y' -Fap)).
(2)
where Fáp is the depth-wise convolutional filter, and Y E |#*xxCout is the output feature. Though Ghost module can reduce the computational cost significantly, the representation ability is inevitably weakened. The relationship between spatial pixels is vital to make accurate recognition. While in GhostNet, the spatial information is only captured by the cheap operations (usually implemented by 3 x 3 depth-wise convolution) for half of the features. The remaining features are just produced by 1 x point-wise convolution, without any interaction with other pixels. The weak ability to capture the spatial information may prevent performance from being further improved.
block GhostNet constructed by stacking two Ghost modules (shown in Figure a)). Similar to a)). MobileNetV2 (28). it is also an inverted bottleneck, i.e., the first Ghost module acts as an expansion layer to increase the number of output channels, and the second Ghost module reduces the channels' number to match the shortcut path.
3.2 Revisit Attention for Mobile Architecture
Originating from the NLP field (36). attention-based models are introduced to computer vision recently 6 9 34
Table I: The comparison of theoretical FLOPs and practical
latency.
Z7. For example. ViT 66] uses the standard transformer model stacked by self-attention modules and MLP modules. Wang el il.insert the self- attention operation into convolutional neural networks to capture the non-
Top-1 Acc. (%)
FLOPs
Latency
Model
(M)
(ms)
GhostNet
73.9
141
31.1
+ Self Attention
23
74.4
172
72.3
DFC Atlention (Ours)
75.3
167
37.5
local information [37]. A typical at-
tention module usually has quadratic complexity w.r.t the feature's size, which is unscalable to high-resolution images in downstream tasks such as object detection and semantic segmentation.
mainstream strategy to reduce attention's 's complexity is splitting images into multiple windows and implementing the attention operation inside windows or crossing windows. For example, Swin Transformer [21] splits the original feature into multiple non-overlapped windows, and the self- attention is calculated within the local windows. MobileVIT [ also unfolds the feature into non-overlapping patches and calculates the attention across these patches. For the 2D feature map in CNN, implementing the feature splitting and attention calculation involves plenty of tensor reshaping

---

Down-sample
Up-sample
Input feature
Vertical FC
Horizontal FC
Attention map
Figure 3: The information flow of DFC attention. The horizontal and vertical FC layers capture the long-range information along the two directions, respectively.
and transposing operations. whose theoretical complexity negligible. In large model (e.g., Swin-B [21] with several billion FLOPs) with high complexity, these operations only occupy few portions of the total inference time. While for the light-weight models, their deploying latency cannot be overlooked.
For an intuitive understanding, we equip the GhostNet model with the self-attention used in Mo- bileViT [23] and measure the latency on Huawei P30 (Kirin 980 CPU) with TFLite tool. We use the standard input's resolution of ImageNet, i.e., 224 x 224, and show the results in Table [ The attention mechanism only adds about 20% theoretical FLOPs. but requires 2x inference time mobile device. The large difference between theoretical and practical complexity shows that is necessary to design hard-ware friendly attention mechanism for fast implementation on mobile devices.
Approach
4.1 DFC Attention for Mobile Architecture
In this section, we will discuss how design an attention module for mobile CNNs. desired attention is expected to have the following properties:
Long-range. It vital to capture the long-range spatial information attention to enhance the representation ability, as light- CNN (e.g., MobileNet [13]. GhostNet (8) usually adopts small convolution filters (e.g., x convolution) to save computational cost.
Deployment-efficient The attention module should be extremely efficient to avoid slowing the inference down. Expensive transformations with high FLOPs or hardware-unfriende operations are unexpected.
Concept-simple. To keep the model's generalization on diverse tasks, the attention module should be conceptually-simple with little dainty design.
model the long- range dependence well, they not deployment-efficcce as discussed in the above section. Compared with them, fully-connected (FC) layers with fixed weights are simpler and easier to implement, which can also be used to generate attention maps with global receptive fields. The detailed computational process is illustrated as follows.
Though self-attention operations 66 24 21
Given a feature {211, #12, =HW A direct implementation of FC layer to generate the attention map is formu- lated as:
HxxCC can be seen as HW tokens z; E RCC i.e., Z =
Σ
Frow N' 2k'w's
where @ is element-wise multiplication, is the learnable weights in the FC layer, and A {a11, .. a arw) is the generated attention map. Eq 3 can capture the global information by aggregating all the tokens together with learnable weights, which much simpler than the typical self-attention (66] as well. However, its computational process still requires quadratic complexity w.n.t. feature's 's size (i.e. (iee,O(H' (H2 which unacceptable practical scenarios especially when the input images of high resolutions. For example, the 4-th layer of GhostNet has feature map
TTe computational complexity channel's number C' omitted for brevity.

---

Add
Sigmaii
Add
BN
Ghost module
Vertical FC
BN
Ghost module
Ghost module
Strido-2
Horizontal FC
module
Stride=?
Mui
Mui
BN
RelU
RetUI
Ixl Convelution
Ghost module
Ghost module
Ghost module
DFC attention
Ghost module
attention
Striden1
Striden2
Striden1
Striden2
DFC attention
(a) GhostNerV1 bottleneck
(b) GhostNerV2 bottleneck
Figure 4: The diagrams of blocks in GhostNerV1 and GhostNetV2. Ghost block is an inverted residual bottleneck containing two Ghost modules, where DFC attention enhances the expanded features to improve expressiveness ability.
with 3136 (56 56) tokens, which incurs prohibitively high complexity calculate the attention map. Actually, feature maps in CNN are usually of low-rank 17), it is unnecessary to connect all the input and output tokens in different spatial locations densely. The feature's 's 2D shape naturally provides a perspective to reduce the computation of FC layers, i.e., decomposing Eq. inno two FC layers and aggregating features along the horizontal and vertical directions, respectively. It can be formulated as:
F h, Zh/no 1,2, H, 1,2,
(4)
W=1
W
//w
FW whww hhw .H. w H,, 1,2, ,,,
(5)
where Fdf and FW are transformation weights. Taking the original feature 2 as input, Eq. 4 and Eq are applied the features sequentially, capturing the long-range dependence along the directions, respectively. We dub this operation as decoupled fully connected (DFC) attention, whose information flow is shown Figure Owing to the decoupling of horizontal and vertical transformations, the computational complexity of the attention module can be reduced to O(HPW + HW?). In the full attention (Eq. 3). all the patches in a square region participate in the calculation of the focused patch directly. In DFC attention, patch is directly aggregated by patches in its vertical/horizontall lines, while other patches participate the generation of those patches the vertical/horizontal lines, having indirect relationship with the focused token. Thus the calculation of patch also involves all the patches in the square region.
Eqs. 4 and 5 denote the general formulation of DFC attention, which aggregates pixels along horizontal and vertical directions, respectively. By sharing part of transformation weights, it can be conveniently implemented with convolutions, leaving out the time-consuming tensor reshaping and transposing operations that affect the practical inference speed. To process input images with varying resolutions, the filter's size can be decoupled with feature map's 's size, i.e., two depth-wise convolutions with kernel sizes Kи and Kw x are sequentially applied on the input feature. When implemented with convolution, the theoretical complexity of DFC attention is denoted as O(K,) n HW + Kw HW). This strategy well supported by tools such as TFLite and ONNX for fast inference mobile devices.
5

---

4.2 GhosetNet V2
In this section, we use the DFC attention to improve the representation ability of lightweight models and then present the new vision backbone, GhostNetV2.
Enhancing Ghost module. As discussed 331 only half of features in Ghost module (Eqs. a interact with other pixels, which damages its ability to capture spatial information. Hence we use DFC attention to enhance Ghost module's output feature Y for capturing long-range dependence among different spatial pixels.
The input feature X IR HXxCC is sent two branches, i.e., one the Ghost module to produce output feature Y (Eqs.) [and D) and the other is the DFC module to generate attention map A (Eqs. 4 and Recalling that a typical self-attention [36]. linear transformation layers are used transform input feature into query and key for calculating attention maps. Similarly, we also implement a 1 x 1 convolution convert module's imput X into DFC's input Z. The final output "= of the module is the product of two branch's 's output. i.e..
O Sigmoid ()) V(X),
where @ is the element-wise multiplication and Sigmoid the scaling function to normalize the attention map A into range (0, 1).
The information aggregation process is shown in Figure 5 With the same input, the Ghost module and DFC attention are two parallel branches extracting information from different perspectives. The output is their weisent-wise product, which contains information from both features of the Ghost module and attentions of the DFC attention module. The calculation of each attention value involves patches in a large range so that the output feature can contain information from these patches.
As Ghost mod- ule (Eqs. aad ( is an extremely efficient oper- ation, directly paralleling the DFC attention with it will introduces extra computational cost. Hence we reduce the feature's size by down-sampling both horizontally and vertically, that all the operations in DFC attention can be conducted on the smaller features. By default, the width and height are both scaled to half of their original lengths, which reduces 75% FLOPs of DFC at- tention. Then produced feature map is then up- sampled to the original size to match the feature's s size in Ghost branch. We naively use the average pooling and bilinear interpolation for downsam- pling and upsampling, respectively. Noticing that directly implementing sigmoid (or hard sigmoid) function will incur longer latency, also deploy the sigmoid function on the downsampled features to accelerate practical inference. Though the value of attention maps may not be limited in range (0,1) strictly, we empirically find that its impact on the final performance is negligible.
Feature downsampling.
Ghost branch
Multiply
Input
Output
Attention branch
Focused patch
Participated patches in Ghost branch
Participated patches Attention branch
Figure 5: The information aggregation process of different patches.
V2 bottleneck. GhostNet adopts inverted residual bottleneck containing two Ghost modules, where the first module produces expanded features with more channels, while the second one reduces channel's number to get output features. This inverted bottleneck naturally decouples the "cxpressiveness" and "capacity" of a model (28). The former is measured by the expanded features while the latter is reflected by the input/output domains of block. The original Ghost module generates partial features via cheap operations, which damages both the expressiveness and the capacity. By investigating the performance difference of equipping DFC attention on the expanded features or output features (Table in Section 5.4). we find that enhancing 'expressiveness' is more effective. Hence only multiply the expanded features with DFC attention.
Figure 4b) shows the diagram of GhostV2 bottleneck. DFC attention branch parallel with the first Ghost module to enhance the expanded features. Then the enhanced features are sent to the second Ghost module for producing output features. It captures the long-range dependence between pixels in different spatial locations and enhances the model's expressiveness.

---

Table 2: Comparison of SOTA light-weight models over classification accuracy, the number of
parameters and FLOPs on ImageNet dataset.
Model
Params (M)
FLOPs (M)
Top-1 Acc. (%)
Top-S Acc. (%)
MobileNetVI 0.5x
1.3
63.3
84,9
MobileNetV2 0.6x
28
2,2
141
66.7
ShuffleNerV 1.0x (g=3)
42
1.9
138
67.8
87.7
ShuffleNett2 1.0x
20
2.3
146
69.4
88.9
MobileNetV3-LL 0.75x
11
4.0
73.3
GhostNerV1 1.0x
g
5.2
141
73,9
91.4
GhostNetV1 1.1x
g
5.9
168
74.5
92.0
GhostNetV2 1.0x
6.1
167
75.3
92.4
MobileNetVI 1.0x
13
4.2
575
70.6
MobileNetV2 1.0x
28
3,5
300
72.8
90.8
ShuffloNetV2 1.5x
22
3,5
299
72.6
90.6
FB-Net 1.0x
3
3.7
301
72.9
FBNet-B 39
4.5
295
74.1
ProxylessNAS
4.1
320
74.6
92.2
MnasNet-A1 31
3.9
312
75.2
92.5
MnasNet-A2 BI
4.8
340
75.6
92.7
MobileNetV3-LL 1,0x
12
5.4
219
75.2
MobileNeXt 1.0x
[44]
3,4
300
74.0
MobileNeXt+ 1.0x
44
3.94
330
76.1
GhostNetV1 1.3x
7.3
226
75.7
92.7
GhostNetV1 1.4x
8.2
76.1
92.9
GhostNetV2 1.3x
8.9
269
76.9
93,4
FBNet-C 39
5.5
375
74.9
EfficientNet-Bi B0
3.2
5.3
390
77.1
93.3
MnasNet-A3 31
5.2
403
76.7
93.3
MobileNetW3--L 1.25x
12
7.5
355
76.6
MobileNeXt++ 1.1x
44
4.28
76.7
VibileViT-XS 23
2.3
700
74.8
GhostNetV1 1.7x
8
11.0
378
77.2
93.4
GhostNetV2 1.6x
12.3
399
77.8
93.8
5 Experiments
In this section, we empirically investigate the proposed GhostNetV2 model. We conduct experiments on the image classification task with the large-scale ImageNet dataset [5] To validate its general- ization, we use GhostNetV2 as backbone and embed it into light-weigh object detection scheme YOLOV3 226. Models with different backbone are compared on MS COCO dataset 200 last, we conduct extensive ablation experiments for better understanding GhostNetV2. The practical latency measured on Huawei P30 (Kirin 980 CPU) with TFLite tool.
5.1 Image Classification on ImageNet
Setting. The classification experiments are conducted on the benchmark ImageNet (ILSVRC 2012) dataset, which contains 1. 1.28M training images and 50K validation images from 1000 classes. We follow the training setting in [8] and report results with single crop on ImageNet dataset. All the experiments are conducted with PoToc [25] and MindSpore [15).
Results. The performance comparison different models ImageNet is shown in Table Figure | and Figure 2 Several light- models are selected as the competing methods. GhostNet [8] MobileNetV2 (28). MobileNetV3
[12), and ShuffleNet 422 are widely-used light- CNN models with SOTA performance. By combing CNN and Transformer, Mobile (24 is new backbone presented recently. Compared with them, GhostNetV2 achieves significantly higher performance with lower computational cost. For example, GhostNerV2 achieves 75.3% top-1 accuracy with only 167 FLOPP, which significantly outperform GhostNet VI (74.5% with similar computational cost (167M FLOPs).

---

Table 3: Results of object detection on MS COCO dataset. YOLOv3 26 is used as the detection head.
Backbone
Resolution
Backbone FLOPs (M)
AP
AP50
AP7s
APg
APM M
AP,, t
MobileNetV2 1.0x
613
22.2
41.9
21.4
23.6
35.8
GhostNet VI 1.1x
320 x 230
338
21.8
41.2
20.8
5.7
22.3
37.3
G2ostNetV2 1.0x
342
22.3
41.4
21.9
6.0
22.8
38.1
MobileNetV2 1.0x (28
1035
23.9
45.4
22.6
10.6
25.1
34.9
GhostNet VI 1.1x
416 x 416
567
23.4
45.2
21.9
9.8
24.4
34.9
GhostNetV2 1.0x
571
24.1
45.7
23.0
10.4
25.0
36.1
Table 4: Effectiveness of DFC attention with MobileNetV2 on ImageNet dataset.
Model
Params (M)
FLOPs (M)
Top- Acc. (%)
Top-5 Acc. (%)
MobileNetV2 1.0 X
3,5
300
72.8
90.8
MobileNetV2
4.1
338
73.0
90.0
MobileNetV2 + SE 14
4.0
338
73.8
MobileNetV2 1.1 X + CBAM 38
4.0
338
74.0
91.4
MobileNetV2 1.1 X CA II]
4.1
350
74.5
91.8
MobileNetV2 1.0 DFC (Ours)
4.3
344
75.4
92.4
Practical Inference Speed. Considering the light-weight model designed for mobile applications, arm-based mobile phone, using the TFLite tool [4]. Owing to the deploying efficiency of DFC attention, GhostNetV2 also achieves good trade-off between accuracy and practical speed. For example, with similar inference latency (e.g., s.. 37 ms), GhostNetV2 achieves 75.3% top-1 accuracy, which is obviously GhostNet V1 with 74.5% top-1 accuracy.
practically measure the inference latency of different models
5.2 Object Detection on COCO
Setting. To validate the generalization of GhostNetV2, we further conduct experiments on the object detection task. The experiments conducted MS COCO 2017 dataset, composing of 118k training images and 5k validation images. We embed different backbone into a widely-used detection head, YOLOv3 (26) and follow the default training strategy provided by MMDetection E Specifically, based on the pre-trained weights on ImageNet, the models are fine-tuned with SGD optimizer for 30 epochs. The batchsize set to 192 and initial learning 0.003. The experiments are conducted with input resolutions 320 x 320.
Results. Table 3 compares the proposed GhostNerV2 model with GhostNet V1. With different input resolutions, GhostNetV2 shows obvious superiority to the GhostNet V1. For example, with similar computational cost (i.e., 340M FLOPs with 320 320 input resolution), GhostNetV2 achieves 22.3% mAP, which suppresses GhostNet V1 by 0.5 mAP. We conclude that capturing the long-range dependence is also vital for downstream tasks, and the proposed DFC attention can effectively endow a large receptive field to the Ghost module, and then construct more powerful and efficient block.
5.3 Semantic Segmentation on ADE20K
We conduct semantic segmentation experiments on ADE20K 443. which contains 20k training, 2k validation, and 3k testing images with 150 semantic categories. We use the DeepLabV3 2 model as the segmentation head, and follow the default training setting MMSSegmentation From the pre- pre-trained weights on ImageNet, the models are fine-tuned for 160000 iterations with crop size 512 x 512. Table show the results with different backbones. In the semantic tasks, GhostNetV2 also achieves significantly higher performance than GhostNetV1, which illustrates the university of GhostNetV2 over different tasks.

---

Table Results of semantic segmentation on ADE20K dataset.
Backbone
Method
Backbone FLOPs (M)
mloU (%)
MobileNetV2 1.0x
28
300
34.08
GhostNet VI 1.1x
DeepLabV3
168
34.17
GhostNetV2 1.0x
167
35.52
Table 7:
The location for implementing DFC attention.
Table 8: Enhancing expressiveness or capac-
ity.
Topl-Ace. (%)
Params (M)
FLOPs
Stage
Topl-Acc. (%)
Params (M)
FLOPs
(M)
Model
(M)
None
73.9
141
Baseline
73.9 (+0.0)
5.2
141
1
74.8
5.3
150
Expressiveness
75.3 (+1.4)
6.1
167
2
75.0
5.4
152
Capacity
74.8 (+0.9)
162
74.7
5.8
147
Both
75.5
7.0
188
All
75.3
5.8
168
5.4
Ablation Studies
In this section, we conduct extensive experiments to investigate the impact of each component in VhostNetV2. The experiments are conducted with GhostNetV2 1 xx ImageNet.
Experiments with other models. As universal module, the DFC attention can also be embedded into other architectures for enhanc- ing their performance. The resultsof MobileNetV2 with different attention modules are shown in Table SE [14] and CBAM (38 are two widely-used attention modules, and CA
Table 6: The impact of kernel size in DFC attention.
Top1- (%)
[ a SOTA method presented recently. The proposed DFC attention achieves higher performance than these existing methods. For example, the pro- posed DFC attention improves the top-1 accuracy of MobileNetV2 by 2.4%, which suppresses CA (1.5%) by a large margin.
Kernel sizes
(3, 3, 3)
74.8
(7. 5, 5)
75.0
7,
74.2
(9, 7, 5)
75.3
The impact of kernel size in DFC attention. We split the Ghost- NetV2 architecture into 3 stages by the feature's size, and apply DFC attention with different kernel size (Table(6). The kernel sizes 1 x 3 and 3 x 1 cannot capture long-range dependence well, which results the worst performance (i.e., 74.8%). Increasing the kernel size capture the longer range information can significantly improve the performance.
(11,9.7)
75.3
The location for implementing DFC tention. The GhostNetV2 model can be split into stages by the feature's 's size, and we empirically investigate how the imple- menting location affects the final perfor- mance. The results are shown in Table 7 which empirically shows that the DFC tention can improve performance when im- plementing it on any stage. Exhaustively adjusting searching for proper locations has the potential to further improve the trade-off between accuracy and computa- tional cost, which exceeds the scope of this paper. By default, we deploy the DFC at- tention on all the layers.
Table 9: The impact of scaling function. 'BF' and 'AF' denote implementing the scaling function before or after the up-sampling operation, respectively.
Topl-Ace. (%)
FLOPs (M)
Latency (ms)
Scaling function
Sigmoid (BF)
75.3
167
37.5
Hard simoid (BF)
75.2
167
36.8
Clip (BF)
74.9
167
36.7
Sigmoid (AF)
75.3
167
40.7
Hard simoid (AI)
75.2
167
39.6
Clip (AF)
75.0
167
38.5
The impact of scaling function. For an attention model, it is necessary to scale the feature maps into range (0,1) 1), which can stabilize the training process. Though the theoretical complexity negligible, these element- wisen operations still incur extra latency. Table investigates how the scaling function affects the final performance and latency. Though sigmoid and hard sigmoid functions bring obvious performance improvement, directly implementing them the large feature maps incur long latency.

---

Implementing them before up-sampling is much more efficient but results in similar accuracy. By default, we use the sigmoid function and put before the up- up-sampling operation.
Enhancing expressiveness or capacity. We implement the DFC attention on two Ghost modules and show the results in Table As discussed in Section 4.2] the former enhances expanded features (expressiveness) while the latter improves the block's ss capacity. With similar computational costs, enhancing the expanded features brings 1.4% top-1 accuracy improvement, which is much higher than enhancing the output feature. Though enhancing both of the features can further improve the performance, the computational cost also increases accordingly. By default, we only enhance the expanded features in an inverse residual bottleneck.
The resizing functions for up-sampling and down-sampling. Multiple functions conduct the up-sampling and down- sampling operations, and we investigate several widely-used functions, i.e., aver- age pooling, max pooling, bilinear inter- polation for down-sampling, and bilinear, bicubic interpolations for up-sampling (Ta- ble I0p. The performance of GhostNetV2 is robust to the choice of resizing functions, i... all of these methods achieve similar accuracies in ImageNet. Their differences mainly lie in practical deploying efficiency on mobile devices. Maxing pooling is slightly more efficient than average pooling (37.5 ms vs.38.4
Table 10: The resizing functions for down- somollig and up-sampling, denoted as 'D' and 'U", respectively.
Topl-Ace. (%)
FLOPs
Latency (ms)
Resizing function
(M)
Average Pooling (D)
75.4
167
38.4
Max Pooling (D)
75.3
167
37.5
Bilinear (D)
75.3
167
38.7
Bilinear (U)
75.3
167
37.5
Bicubic (U)
75.4
167
39.9
ms), and bilinear interpolation faster than the bicubic one (37.5 ms vs.39.9 ms). Thus we choose the maxing pooling for down-sampling and bilinear interpolation for up-sampling by default.
Visualization of decoupled attention and full at- tention. We visualize the decoupled attention pro- duced by stacking vertical and horizontal attentions and compare it with full attention. In low layers, the decoupled attention shows some cross-shaped pat- terns, indicating patches from the vertical/horizootal lines participate more. As the depth increases, the pattern of the attention map diffuses and becomes more similar to the full attention.
Vartica/HHarizontt
Attention
Full Attention
Layor
Layer 10
Layar
Figure 6: Visualization of attention maps.
Conclusion
This paper proposes hariware-friendly DFC attention and presents a new GhostNetV2 architecture for mobile applications. The DFC attention capture the dependence between pixels in long-range spatial locations, which significantly enhances the expressiveness ability of light-weight models. It decomposes FC layer into horizontal FC and vertical FC, which has large receptive fields along the two directions, respectively. Equipped this computation-effiiien and deployment-simpl modules, ChostNetV2 2 can achieve better trade-off between accuracy and speed. Extensive experiments benchmark datasets (e.g., ImageNet, MS COCO) validate the superiority of GhostNetV2.
Acknowledgment. This work is supported by National Natural Science Foundation of China under Grant No.61876007, Australian Research Council under Project DP2 10101859 and the University of Sydney SOAR Prize. We gratefully acknowledge the support of MindSpore, CANN(Compute Architecture for Neural Networks) and Ascend Al Processor used for this research.
References
[1] Han Cai, Ligeng Zhu, and Song Han. Proxylessnas: Direct neural architecture search target task and hardware. /CLR, 2019.
[2] Liang- Chiih Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint 1706: 2017.
10

---

[3] Weijie Chen, Di Xie, Yuan Zhang, and Shiliang Pu. All you need is few shifts: Designing efficient convolutional neural networks for image classification. In CVPR, 2019.
[4] Robert David, Jared Duke, Advait Jain, Vijay Janapa Reddi, Nat Jeffries, Jian Li, Nick Kreeger, Ian Nappier, Meghna Natraj, Tiezhen Wang. et al. Tensorflow lite micro: Embedded machine learning for tinyml systems. Proceedings of Machine Learning and Systems. 3:800-811, 2021.
[5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. / /929,, 2020.
[7] Kai Han, Jianyuan Guo, Ychui Tang, and Yunhe Wang. Pyramidmt: Improved transformer-ing senpffommrriin-traaffooo transfforrrr basclines with pyramid architecture. arXiv preprint (rXiv:2201.00977 2022.
[8] Kai Han, Yunhe Wang. Qi Tian, Jianyuan Guo, Chunjing Xu, and Chang Xu. Ghostnet: More features from cheap operations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 1580-1589, 2020.
[9] Kai Han, An Xiao, Enhua Wu, Jianyuan Guo, Chunjing Xu, and Yunhe Wang. Transformer in transformer. Advances in Neural Information Processing Systems, 34:15908-11 15919,, 2021.
[10] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[11] Qibin Hou, Daquan Zhou, and Jiashi Feng. Coordinate attention for efficient mobile network design. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13713- 13722, 2021.
[12] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al. Scarching for mobilenetv3. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1314-1324, 2019.
[13] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: arXiv:170..0466,, 2017.
[14] Hu, Li Shen, and Gang Sun. Squeezz-andd-excitaton networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-71411 2018.
[15] Huawei. Mindspore. https: /ttp:://wwwwwwee http:/ww...m.. www.w..indspor....m cn/ 2020.
[16] Forrest N landola. Song Han, Matthew W Moskewicz. Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint arXiv:1602.073660 2016.
[17] Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint 1XXv:1440538866 2014.
[18] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pages 1725 1732, 2014.
[19] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[20] Tsung- Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[21] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[22] Ningning Ma, Xiangyu Zhang, Tai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. In Proceedings of the European conference on computer vision (ECCV), pages 116-131. 2018
11

---

[23] Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprini 1rX:1:2110.0217. 2021.
[24] Sachin Mehta and Mohammad Rastegari. Mobilevit: Light-weight, gemeral-purpose, and mobile-friendly vision transformer. In International Conference on Learning Representations, 2022.
[25] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[26] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint 1rXiv.180.00276 2018.
[27] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[28] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEFF conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[29] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[30] Cheng Tai, Tong Xiao, Yi Zhang, Xiaogang Wang, et al. Convolutional neural networks with low-rank regularization, arXiv preprint arXiv:1511.0600 2015.
[31] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-awar aware neural architecture search for mobile. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2828,2828,, 2019.
[32] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference on machine learning. pages 6105-61 14. PMLR, 2019.
[33] Yehui Tang, Kai Han, Jianyuan Guo, Chang Xu, Yanxi Li, Chao Xu, and Yunhe Wang. An image patch is wave: Phase-aware vision mlp. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 10935-10944 2022.
[34] Yehui Tang, Kai Han, Chang Xu, An Xiao, Yiping Deng, Chao Xu, and Yunhe Wang. Augmented shortcuts for vision transformers. Advances in Neural Information Processing Systems, 34: 1511611733,,, 2021.
[35] Ychui Tang, Yunhe Wang, Yixing Xu, Dacheng Tao, Chunjing Xu, Chao Xu, and Chang Xu. Scop: Scientific control for reliable neural network pruning. Advances in Neural Information Processing Systems, 30:10936- 1094661097,,400 2020.
[36] Ashish Vaswani, Noam Shazeer. Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[37] Xiaolong Wang, Ross Girshick. Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7794-7803, 2018.
[38] Sanghyun Woo, Jongchan Park, Joon-Young Lee, and In So Kweon. Cham: Convolutional block attention module. In Proceedings of the European conference computer vision (ECCV), pages 3-19, 2018
[39] Bichen Wu, Xiaoliang Dai, Peizhao Zhang, Yanghan Wang, Fei Sun, Yiming Wu, Yuandong Tian, Peter Vajda, Yangqing Jia, and Kurt Keutzer. Fbnet: Hardware-aware efficient convnet design via differentiable neural architecture search. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10734-10742, 2019.
[40] Yixing Xu, Kai Han, Chang Xu, Ychui Tang, Chunjing Xu, and Yunhe Wang. Learning frequency domain approximation for binary neural networks. Advances in Neural Information Processing Systems, 2-225553-25565. 2021.
[41] Yixing Xu, Yunhe Wang, Hanting Chen, Kai Han, Chunjing Xu, Dacheng Tao, and Chang Xu. Positive- unlabeled compression on the cloud. Advances in Neural Information Processing Systems, 32, 2019.
[42] Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on compuder vision and pattern recognition, pages 6848 6856, 2018
12

---

[43] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler. Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings the IEEE conference computer vision and pattern recognition. pages 633-641, 2017.
[44] Daquan Zhou, Qibin Hou, Yunpeng Chen, Jiashi Feng, and Shuicheng Yan. Rethinking bottleneck structure efficient mobile network design. In European Conference Computer Vision, pages 680-697. Springer. 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section 1.
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No] No potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you's 're using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "u6iftgwjhb", "reviewer": "Reviewer_L8vJ", "summary": "This paper improves a previous work, named GhostNet. The original GhostNet aims to eliminate the effect of those uninformative feature maps by introducing the ghost module. However, the drawback of the original GhostNet is the lack of the ability to capture long-range relationships among pixels. This paper is motivated by this and designs an edge device friendly attention mechanism, which runs fast and performs well on ImageNet classification.\n\nOverall speaking, the quality of this paper is good. The novelty of this paper is significant. Thorough experiments are also conduct to demonstrate the effectiveness of the proposed approach.", "questions": "Tables 9 and 10 could be redesigned to make the arrangement look better.", "limitations": "Not found.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The strengths  of this paper is clear.\n\n- The proposed light-weight attention mechanism is interesting. It simplifies the standard self-attention and the performance does not drop. I think it could be considered as a promising way to encode global context for mobile networks.\n\n- The analysis of this paper is sufficient. The authors carefully analyze why self-attention is not friendly to edge devices and show lots of results to support this.\n\n- The results on ImageNet are great. Compared to most of previous models for mobile devices, this paper performs better.\n\nWeaknesses:\n\nI do not think there are any major red flags, but some minor to moderate concerns that should be addressed.\n\n- In Sec. 3.1, the authors say that only half of the channels are used for encoding spatial information with a depthwise conv in a ghost module and claim that this may be a performance bottleneck of GhostNet. How could you prove this?\n\n- The performance on downstream tasks, like COCO object detection, seems not that surprising but considering the low computations, it is acceptable.\n\n- The English should be improved, especially the usage of the definite articales.\n\n- The usage of abbreviations. L168 and L183, \"Eq 4, 5\" should be \"Eqs. 4 and 5.\" ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "oURXecLNmwd", "reviewer": "Reviewer_BfQb", "summary": "This paper proposes a cheap attention module that focuses on mobile settings, e.g. limited FLOPs and latency. The cheap attention module is implemented as two depthwise convolutions, followed by sigmoid attention. The module is mainly used to augment GhostNet and achieves improvements on ImageNet classification and downstream tasks, e.g. detection, and segmentation.", "questions": "Minor: should the latter $a$ be $a\\prime$ in equation 5?\n\nAlso, in equation 4, for each h, the equation does not depend on h. Does it mean that the output $a\\prime$ is the same for each h? If so, equation 5 does not depend on w?", "limitations": "Yes.", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n1. There are many ablation studies that validate the model design choices.\n2. Extensive experiments on multiple large scale tasks and datasets.\n\n### Weaknesses\n1. The proposed solution, Equation 4,5, is not depthwise convolution with kernel $K_H$ and $K_W$. In equation 4 and 5, the weight matrix F has HWC parameters. These parameters are not shared for each column or row. This is more like a batch matrix multiplication, but is different from depthwise convolution which has $K_H$ and $K_W$ parameters. According to table 5, it seems that depthwise convolution is being used in practice.\n2. Limited novelty with height-width decoupled depthwise convolutions [spatially separable convolution] and SE-like spatial-channel attention. Both of the techniques have been heavily explored in the community.\n3. Marginal performance gain when FLOPs is controlled. For example, Auto-NL [1] also studied lightweight self-attention, achieved 77.7% on ImageNet with 353M FLOPS and 5.6M parameters two years ago, compared with this paper, 77.8% with 399M FLOPs and 12.3M parameters. Although it is not clear what latency Auto-NL needs, this related work is not discussed or compared.\n4. Downsampling and upsampling part of the feature map has been explored too in [2]. And it leads to gains as well. So it is not clear why DFC works.\n5. Latency is important to the paper, so instead of saying “an ARM-based mobile device”, is it possible to specify which device exactly, then it is possible for others to reproduce the latency results and make meaningful comparisons.\n6. Related to the point above, FBNet reports a latency of 28.1ms, but this paper reports around 70ms. Is this mainly caused by device difference and optimization? Also, [3] reports multiple models on multiple mobile devices, e.g. 76.1% with 22ms on Samsung Note 10, or 76.9% with 58ms on Pixel 1. Is it possible to compare the latency results with the literature? Similarly, MobileViT reports a 17.86ms latency on iPhone 12, which is much faster than the GhostNet+self attention baseline in Table 1, probably due to a larger feature resolution in the comparison.\n\n[1] Neural Architecture Search for Lightweight Non-Local Networks, CVPR 2020.\n[2] ELASTIC: Improving CNNs with Dynamic Scaling Policies, CVPR 2019.\n[3] Once for All: Train One Network and Specialize it for Efficient Deployment, ICLR 2020.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "SmKnmcr_lAL", "reviewer": "Reviewer_zuKb", "summary": "This paper proposes a hardware friendly attention module and then present a light-weight neural architecture for general vision tasks. It finds that light-weight neural networks have weak ability to capture the global information, which is the bottleneck restricting the representation ability.  The attention module is only constructed by fully-connected layers, which can efficiently capture the global information without complex operations. Then a light-weight architecture is constructed, which achieves SOTA performance on various vision tasks, such as image classification and object detection. ", "questions": "Please see the weaknesses. Further investigating the impact of non-linear functions will make the paper stronger.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+It is vital to design light-weight neural networks with low latency on edge devices (e.g., ARM CPU) for implementing AI models. Due to the strict constraint on practical latency, improving its performance is a very challenging task. Some complex operations may have low theoretical complexity, but will incur high practical latency as they are not hard-ware friendly. This paper finds the performance bottleneck of light-weight models and presents an efficient DFC attention to capture the global information, which can significantly improve performance. \n\n+The empirical results are impressive. Based on DFC attention, the GhostNetV2 model achieves significantly higher performance (about 1 point) than the existing architectures. Both with attention mechanism, GhostNetV2 achieves much higher performance than MobileViT (77.8% v.s. 74.8%) with lower computational cost (399M v.s. 700M). \n\n+The proposed architecture has strong generalization ability and can be used in diverse tasks (such as image classification and object detection), as it does not introduce prior knowledge of a specific task. Considering its SOTA performance, easy implementation, and strong generalization ability, it can play as the backbone to improve the model’s performance on various tasks.\n\nThough the proposed architecture is novel and effective, I still have some suggestions to further improve its impact on the community. \n\n- The DFC attention captures the long-range dependence with two sequential FC layers, which aggregate information along with horizontal and vertical directions, respectively. Adding a non-linear activation function between them may reflect a more complex relationship between different pixels. It is interesting to investigate the effectiveness of this non-linear function empirically.\n\n-For intuitive presentation, it is better to describe the proposed architectures in the captions of Figure 4.\n\n+This paper explores an interesting direction to leverage the attention mechanism, which may inspire the community. Transformer achieves high performance owing to its strong ability for capturing global information. However, it suffers high computational complexity and complex formulation, which is not hard-ware friendly when implemented on edge devices. The proposed method uses fully-connected layers to implement the attention operation, which is both effective and efficient. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "L8OIDMusXmS", "reviewer": "Reviewer_Fb8Z", "summary": "In this paper, the authors propose a \"decoupled fully connected\" (DFC) attention module which can significantly improve the performance of image classification models such as GhostNet and MobileNet at a much lower cost than typical self-attention like those used in transformers. This is achieved by aggregating information on the horizontal and vertical axis (instead of over the entire image) in a downsampled feature space and combining the output of this module with the output of standard network block.\n\nExperiments show that this approach empirically achieves significant improvement in accuracy at a modest cost in throughput, and outperforms previous state of the art for efficient mobile-friendly image classification networks.\n\nNits:\n\n69-70: Until now, GhostNet is still the SOTA light-weight model with a good trade-off between accuracy and speed. -> Until now, GhostNet has been the SOTA light-weight model with a good trade-off between accuracy and speed.\n\n71: Besides manual design, a series of methods try to search a light-weight architecture. -> Besides manual design, a series of methods try to search for a light-weight architecture.\n\n199: practica -> practical\n\n215: Experiment -> experiments", "questions": "- This is not quite global attention because you attend over patches aligned horizontally or vertically, and hope that previous blocks captured sufficient global context in their respective patches. one way to test whether they do is to make the network deeper, but narrower (to control for number of params) and see if the accuracy improves. Have you considered this, along with visualizing attention maps to see what is actually being attended\n\n- I am a bit confused about figure 4 and equation 6 and the preceding paragraph. Is it saying that you compute your the horizontal/vertical attention (via a linear projection) followed by sigmoid followed by elementwise multiplication with the output of the ghost module (which is of dim HxWxC). Does this not mean you are just rescaling the output of the ghost module? if not can you please explain and also possibly clarify in the paper how exactly you are attending over/aggregating info from different patches?", "limitations": "limitations were addressed adequatly", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Experiments and ablation studies are well designed and show the value of this work\n- The results are convincing\n- The approach appears to be novel\n\nWeaknesses:\n- The description and diagram of the actual approach is somewhat confusing (see the question section)\n- It would be good to study what this attention ends up attending over (and compared for vertical/horizontal att vs full att)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept