Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes a cheap attention module that focuses on mobile settings, e.g. limited FLOPs and latency. The cheap attention module is implemented as two depthwise convolutions, followed by sigmoid attention. The module is mainly used to augment GhostNet and achieves improvements on ImageNet classification and downstream tasks, e.g. detection, and segmentation.
strengths_and_weaknesses: ### Strengths
1. There are many ablation studies that validate the model design choices.
2. Extensive experiments on multiple large scale tasks and datasets.

### Weaknesses
1. The proposed solution, Equation 4,5, is not depthwise convolution with kernel $K_H$ and $K_W$. In equation 4 and 5, the weight matrix F has HWC parameters. These parameters are not shared for each column or row. This is more like a batch matrix multiplication, but is different from depthwise convolution which has $K_H$ and $K_W$ parameters. According to table 5, it seems that depthwise convolution is being used in practice.
2. Limited novelty with height-width decoupled depthwise convolutions [spatially separable convolution] and SE-like spatial-channel attention. Both of the techniques have been heavily explored in the community.
3. Marginal performance gain when FLOPs is controlled. For example, Auto-NL [1] also studied lightweight self-attention, achieved 77.7% on ImageNet with 353M FLOPS and 5.6M parameters two years ago, compared with this paper, 77.8% with 399M FLOPs and 12.3M parameters. Although it is not clear what latency Auto-NL needs, this related work is not discussed or compared.
4. Downsampling and upsampling part of the feature map has been explored too in [2]. And it leads to gains as well. So it is not clear why DFC works.
5. Latency is important to the paper, so instead of saying “an ARM-based mobile device”, is it possible to specify which device exactly, then it is possible for others to reproduce the latency results and make meaningful comparisons.
6. Related to the point above, FBNet reports a latency of 28.1ms, but this paper reports around 70ms. Is this mainly caused by device difference and optimization? Also, [3] reports multiple models on multiple mobile devices, e.g. 76.1% with 22ms on Samsung Note 10, or 76.9% with 58ms on Pixel 1. Is it possible to compare the latency results with the literature? Similarly, MobileViT reports a 17.86ms latency on iPhone 12, which is much faster than the GhostNet+self attention baseline in Table 1, probably due to a larger feature resolution in the comparison.

[1] Neural Architecture Search for Lightweight Non-Local Networks, CVPR 2020.
[2] ELASTIC: Improving CNNs with Dynamic Scaling Policies, CVPR 2019.
[3] Once for All: Train One Network and Specialize it for Efficient Deployment, ICLR 2020.
questions: Minor: should the latter $a$ be $a\prime$ in equation 5?

Also, in equation 4, for each h, the equation does not depend on h. Does it mean that the output $a\prime$ is the same for each h? If so, equation 5 does not depend on w?
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: In this paper, the authors propose a "decoupled fully connected" (DFC) attention module which can significantly improve the performance of image classification models such as GhostNet and MobileNet at a much lower cost than typical self-attention like those used in transformers. This is achieved by aggregating information on the horizontal and vertical axis (instead of over the entire image) in a downsampled feature space and combining the output of this module with the output of standard network block.

Experiments show that this approach empirically achieves significant improvement in accuracy at a modest cost in throughput, and outperforms previous state of the art for efficient mobile-friendly image classification networks.

Nits:

69-70: Until now, GhostNet is still the SOTA light-weight model with a good trade-off between accuracy and speed. -> Until now, GhostNet has been the SOTA light-weight model with a good trade-off between accuracy and speed.

71: Besides manual design, a series of methods try to search a light-weight architecture. -> Besides manual design, a series of methods try to search for a light-weight architecture.

199: practica -> practical

215: Experiment -> experiments
strengths_and_weaknesses: Strengths:
- Experiments and ablation studies are well designed and show the value of this work
- The results are convincing
- The approach appears to be novel

Weaknesses:
- The description and diagram of the actual approach is somewhat confusing (see the question section)
- It would be good to study what this attention ends up attending over (and compared for vertical/horizontal att vs full att)
questions: - This is not quite global attention because you attend over patches aligned horizontally or vertically, and hope that previous blocks captured sufficient global context in their respective patches. one way to test whether they do is to make the network deeper, but narrower (to control for number of params) and see if the accuracy improves. Have you considered this, along with visualizing attention maps to see what is actually being attended

- I am a bit confused about figure 4 and equation 6 and the preceding paragraph. Is it saying that you compute your the horizontal/vertical attention (via a linear projection) followed by sigmoid followed by elementwise multiplication with the output of the ghost module (which is of dim HxWxC). Does this not mean you are just rescaling the output of the ghost module? if not can you please explain and also possibly clarify in the paper how exactly you are attending over/aggregating info from different patches?
limitations: limitations were addressed adequatly
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes a hardware friendly attention module and then present a light-weight neural architecture for general vision tasks. It finds that light-weight neural networks have weak ability to capture the global information, which is the bottleneck restricting the representation ability.  The attention module is only constructed by fully-connected layers, which can efficiently capture the global information without complex operations. Then a light-weight architecture is constructed, which achieves SOTA performance on various vision tasks, such as image classification and object detection. 
strengths_and_weaknesses: +It is vital to design light-weight neural networks with low latency on edge devices (e.g., ARM CPU) for implementing AI models. Due to the strict constraint on practical latency, improving its performance is a very challenging task. Some complex operations may have low theoretical complexity, but will incur high practical latency as they are not hard-ware friendly. This paper finds the performance bottleneck of light-weight models and presents an efficient DFC attention to capture the global information, which can significantly improve performance. 

+The empirical results are impressive. Based on DFC attention, the GhostNetV2 model achieves significantly higher performance (about 1 point) than the existing architectures. Both with attention mechanism, GhostNetV2 achieves much higher performance than MobileViT (77.8% v.s. 74.8%) with lower computational cost (399M v.s. 700M). 

+The proposed architecture has strong generalization ability and can be used in diverse tasks (such as image classification and object detection), as it does not introduce prior knowledge of a specific task. Considering its SOTA performance, easy implementation, and strong generalization ability, it can play as the backbone to improve the model’s performance on various tasks.

Though the proposed architecture is novel and effective, I still have some suggestions to further improve its impact on the community. 

- The DFC attention captures the long-range dependence with two sequential FC layers, which aggregate information along with horizontal and vertical directions, respectively. Adding a non-linear activation function between them may reflect a more complex relationship between different pixels. It is interesting to investigate the effectiveness of this non-linear function empirically.

-For intuitive presentation, it is better to describe the proposed architectures in the captions of Figure 4.

+This paper explores an interesting direction to leverage the attention mechanism, which may inspire the community. Transformer achieves high performance owing to its strong ability for capturing global information. However, it suffers high computational complexity and complex formulation, which is not hard-ware friendly when implemented on edge devices. The proposed method uses fully-connected layers to implement the attention operation, which is both effective and efficient. 

questions: Please see the weaknesses. Further investigating the impact of non-linear functions will make the paper stronger.
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper improves a previous work, named GhostNet. The original GhostNet aims to eliminate the effect of those uninformative feature maps by introducing the ghost module. However, the drawback of the original GhostNet is the lack of the ability to capture long-range relationships among pixels. This paper is motivated by this and designs an edge device friendly attention mechanism, which runs fast and performs well on ImageNet classification.

Overall speaking, the quality of this paper is good. The novelty of this paper is significant. Thorough experiments are also conduct to demonstrate the effectiveness of the proposed approach.
strengths_and_weaknesses: The strengths  of this paper is clear.

- The proposed light-weight attention mechanism is interesting. It simplifies the standard self-attention and the performance does not drop. I think it could be considered as a promising way to encode global context for mobile networks.

- The analysis of this paper is sufficient. The authors carefully analyze why self-attention is not friendly to edge devices and show lots of results to support this.

- The results on ImageNet are great. Compared to most of previous models for mobile devices, this paper performs better.

Weaknesses:

I do not think there are any major red flags, but some minor to moderate concerns that should be addressed.

- In Sec. 3.1, the authors say that only half of the channels are used for encoding spatial information with a depthwise conv in a ghost module and claim that this may be a performance bottleneck of GhostNet. How could you prove this?

- The performance on downstream tasks, like COCO object detection, seems not that surprising but considering the low computations, it is acceptable.

- The English should be improved, especially the usage of the definite articales.

- The usage of abbreviations. L168 and L183, "Eq 4, 5" should be "Eqs. 4 and 5." 
questions: Tables 9 and 10 could be redesigned to make the arrangement look better.
limitations: Not found.
review_title: null

Meta-review: metareview: This paper aims to augment efficient CNNs with self-attention. However, since the naive approach to self-attention is computationally expensive and would contradict the point of efficient CNNs, the authors introduce a new attention mechanism which captures long-range information without substantially added computation cost. The paper demonstrates that GhostNetV2 exhibits markedly better performance at various compute limits as compared to previously proposed efficient networks. Three of the reviewers were quite positive on this paper, noting the novelty of the approach and the strength of the empirical results. One reviewer had several concerns, primarily regarding comparison to NAS based approaches and the novelty of the approach. I agree with the other reviewers that it is not reasonable to compare NAS approaches to non-NAS approaches, and agree that there are marked differences between this work and the previous work cited. I therefore recommend acceptance. I think this will be a valuable contribution to the efficient network community.
confidence: Certain
Decision: Accept