RaCNN: Region-aware Convolutional Neural Network with Global Receptive Field

Authors: Yuhang Pei, Shenqi Lai, Zhanwei Zhang, Dan Deng, Wei Qian, Zheng Yang, Liang Xie, Haifeng Liu, Deng Cai
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Convolutional Neural Network, Global Receptive Field, Backbone

Abstract:
Recent Convolutional Neural Networks (CNNs) utilize large-kernel convolutions (e.g., 101 kernel convolutions) to simulate a large receptive field of Vision Transformers (ViTs). 
    However, these models introduce specialized techniques like re-parameterization, sparsity, and weight decomposition, increasing the complexity of the training and inference stages. 
    To address this challenge, we propose Region-aware CNN (RaCNN), which achieves a global receptive field without requiring extra complexity, yet surpasses state-of-the-art models. 
    Specifically, we design two novel modules to capture global visual dependencies. 
    The first is the Region-aware Feed Forward Network (RaFFN). 
    It uses a novel Region Point-Wise Convolution (RPWConv) to capture global visual cues in a region-aware manner. 
    In contrast, traditional PWConv shares the same weights for all spatial pixels and cannot capture spatial information. 
    The second is the Region-aware Gated Linear Unit (RaGLU). 
    This channel mixer captures long-range visual dependencies in a sparse global manner and can become a better substitute for the original FFN. 
    Under only 84\% computational complexity, RaCNN significantly outperforms the state-of-the-art CNN model MogaNet (83.9\% vs. 83.4\%). 
    It also demonstrates good scalability and surpasses existing state-of-the-art lightweight models. 
    Furthermore, our RaCNN shows comparability with state-of-the-art ViTs, MLPs, and Mambas in object detection, instance segmentation, and semantic segmentation.  
    All codes and logs are released in the supplementary materials.

Decision: reject
Reviewer scores: 5, 5, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730098784895

thread_type:
official_review

<thread_object>
creator: reviewer_pz92

<content>
summary:
This paper proposes a novel convolutional neural network (CNN) model called Region-aware CNN (RaCNN), designed to simulate a large receptive field with minimal computational overhead. RaCNN achieves this by replacing the standard feedforward neural network (FNN) with RaGLU and the original depthwise convolution (DWConv) with Region PWConv (RPWConv), allowing the model to capture long-range visual dependencies. The paper demonstrates the effectiveness of RaCNN through experiments on classification, detection, and segmentation tasks.

strengths:
The motivation of the paper is valuable. The high computational complexity of large kernel size is indeed a barrier in the field of computer vision.

The proposed RaCNN model shows effectiveness across various tasks, as evidenced by the experimental results.

weaknesses:
The contribution of this work is not particularly groundbreaking. Compared to previous methods like [], the advantages of RaCNN are not significantly clear.

Some expressions in the paper lack precision. For example, the distinction between “FLOPS” and “FLOPs” on page 8, line 421, is not handled correctly. I recommend a thorough review of the manuscript.

Equation (6) appears to be incorrect; “x·x” cannot be computed as written. Please verify all equations.

questions:
The paper lacks detailed explanation regarding the advantages of dilated windows in Region PWConv. Could you provide more clarification on this?

What would be the impact of using Dynamic PWConv instead of Region PWConv? Additional ablation studies would strengthen the validation of your method.

In my view, Figure 2(e) does not effectively demonstrate RaCNN’s ability to capture long-range dependencies.

In Table 8, row 4 (which uses both ReFFN and RaGLU) reports fewer FLOPs and parameters than rows 2 and 3, which is puzzling. Could you explain this discrepancy?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730645212059

thread_type:
official_review

<thread_object>
creator: reviewer_zcme

<content>
summary:
This paper presents RaCNN, a convolutional neural network designed to capture both long-range dependencies and local contextual information in images. The architecture introduces two key components: the Region-aware Feed Forward Network (RaFFN) and the Region-aware Gated Linear Unit (RaGLU). RaFFN employs a specialized Region Point-Wise Convolution (RPWConv), which divides spatial feature maps into global regions and applies dynamic weights within each, allowing for the capture of both global and local features. RaGLU enhances spatial information mixing through region-specific attention mechanisms. The authors claim that RaCNN outperforms state-of-the-art models such as MogaNet and Swin Transformer in tasks like image classification, object detection, and instance segmentation, while maintaining computational efficiency. Extensive experiments and ablation studies are provided to validate RaCNN's effectiveness.

strengths:
RaFFN and RaGLU introduce a novel region-based dynamic weighting approach in CNNs, allowing RaCNN to capture multiscale spatial dependencies compared to traditional convolutional methods more effectively.

The experimental results across multiple vision tasks, such as object detection and instance segmentation, demonstrate RaCNN's robustness. This approach efficiently captures global dependencies while maintaining computational efficiency, representing an advancement over previous CNNs and ViTs.

RaCNN demonstrates scalability across various model sizes and tasks.

weaknesses:
The claim that RaCNN "can capture long-range dependencies and local context features simultaneously without excessive noise" is based on a single ERF visualization. However, visualizations from a single layer are insufficient to support such broad conclusions about noise reduction and contextual feature capture, as ERF patterns vary across layers in complex networks. A more comprehensive analysis across multiple layers or the use of additional metrics would be necessary to fully substantiate this claim.

Although the paper highlights RaCNN’s computational efficiency, it lacks a detailed comparison of how this efficiency holds up against similar models in real-world scenarios, such as inference time or memory usage.

The RPWConv design, which divides the spatial feature map into predefined regions, may limit flexibility across different image types. Fixed partitioning could reduce generalizability for diverse visual tasks that benefit from more adaptive receptive fields. Further investigation into alternative partitioning strategies, like adaptive region sizes, would strengthen the proposed approach.

While the ablation studies focus on the impact of individual components (e.g., RPWConv, RaGLU), they do not sufficiently explore the interactions between these modules. Expanding the ablation studies to analyze how these components interact could provide deeper insights into RaCNN’s overall design.

The paper’s focus on FLOPs as the primary metric for efficiency may overlook other crucial factors, such as memory usage and inference time. These metrics are critical for real-world deployment, and their inclusion would provide a more balanced perspective on RaCNN’s efficiency.

questions:
1. Could the authors provide additional evidence to support RaCNN’s claims of noise resistance and contextual feature capture? Specifically, how does the ERF visualization change across different layers of RaCNN, and would deeper layer results reinforce the conclusions drawn from Figure 2?

2. How does RaCNN perform in terms of inference speed and memory usage compared to models like Swin Transformer and MogaNet in real-time applications?

3. Have the authors considered exploring alternative region partitioning strategies for RPWConv, such as using adaptive region sizes, to enhance flexibility across different image types?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730652589243

thread_type:
official_review

<thread_object>
creator: reviewer_cksh

<content>
summary:
This paper presents the RaCNN for vision recognition. The proposed RaCNN achieves a global receptive field with two designs: the region-aware FFN that uses a variant of dilated attention to model global relations, and the region-aware GLU that uses DWConv and a regional SE module to improve model capacity. The model is evaluated on widely used visual recognition benchmarks and compared to recent SoTA methods.

strengths:
- The method achieves competitive performance on various visual recognition tasks including classification and dense prediction. The throughput also looks good.

- The authors conducted extensive experiments to verify the model.

weaknesses:
- The proposed model actually is not a convolutional neural network. A convolution operation should be transition-equivariant. The proposed operation is closer to a variant of local attention instead of convolution. The authors may consider changing the name of the model and some claims in the paper to avoid misunderstanding.

- The proposed design is a mixture of some existing techniques.  The Region Point-Wise Convolution is not a convolutional operation and it is very similar to [r1]. The RA operation is a variant of the SE module. Although the paper shows that the mixture can achieve good performance, the technical contribution is quite limited. 

[r1] Glance-and-Gaze Vision Transformer, NeurIPS 2021. 

- The motivation of the paper is not quite clear. According to the abstract, the paper wants to propose a new solution because "these models introduce specialized techniques like re-parameterization, sparsity, and weight decomposition, increasing the complexity of the training and inference stages". However, I think the proposed solution also introduces many complex operations and may make the solution less general. 

- The presentation of the paper needs improvement. Many claims are not clear. For example, in the caption of Figure 1, it is mentioned that some existing methods "could capture long-range dependency but introduce excessive visual noises". I don't found there is evidence to show these methods can introduce extra noise. Besides, the descriptions of the core contribution: RPWConv and RA, are quite confusing. Figure 4 didn't clearly the RPWConv operation. Equations 5-7 are also misleading since both "x \dot x" and "xw" actually are matrix multiplication. I need to check the provided code to clearly understand how the operation actually works.

questions:
Please refer to my comments above. Considering there are many issues about the presentation, positioning, technical contribution, and motivation, I cannot recommend acceptance for this paper.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720775587

thread_type:
official_review

<thread_object>
creator: reviewer_eqag

<content>
summary:
The paper titled "RaCNN: Region-Aware Convolutional Neural Network with Global Receptive Field" proposes a new CNN model, RaCNN, that enhances the receptive field to capture global visual dependencies without increasing computational complexity. The key innovations of RaCNN include two modules: the Region-aware Feed Forward Network (RaFFN) and the Region-aware Gated Linear Unit (RaGLU).

RaFFN uses Region Point-Wise Convolution (RPWConv) to capture global cues by dividing spatial feature maps into sparse global regions, allowing it to dynamically adjust weights per region. Meanwhile, RaGLU serves as a channel mixer, capturing long-range dependencies in a sparse, global manner, improving spatial information aggregation. Compared to conventional CNNs, RaCNN demonstrates state-of-the-art performance, surpassing models like MogaNet, while requiring fewer computational resources.

strengths:
1.	The paper proposes region-aware CNN. The author suggest RaCNN captures both local and global information to better feature extraction
2.	Experiments show RaCNN achieves better results on various tasks, including image classification (ImageNet-1K), object detection (COCOval2017), instance segmentation (COCOval2017), and semantic segmentation (ADE20K).

weaknesses:
1.	It is confused to figure out how RaCNN captures global information. For instance, Region PWConv in figure 4c takes dilated windows and Region Attention in figure 5 averages spatial feature in a dilated manner. However, dilation is not equivalent to the so-called ‘global’. From my point of view, it is more proper to say it captures a larger local receptive field. 
2.	The author suggests RaCNN is better because it captures local and global information. However, in figure 2 we can see SLak, UniRepLKNet, and InceptionNeXT all captures local and global receptive field, even with a more ‘global’ field. Therefore, I am suspecting local and global receptive field is not the true reason for RaCNN performance. 
3.	For figure 1, there is no margin between the figure 1 caption and main text. 
4.	More tasks [1] should be added to verify the effectiveness of RaCNN, such as 2D and 3D human pose estimation, and video prediction. 
[1] Li, Siyuan, Zedong Wang, Zicheng Liu, Cheng Tan, Haitao Lin, Di Wu, Zhiyuan Chen, Jiangbin Zheng, and Stan Z. Li. "Moganet: Multi-order gated aggregation network." In The Twelfth International Conference on Learning Representations. 2023.

questions:
Question: While RaCNN claims to achieve a global receptive field without additional complexity, it would be beneficial to understand the exact computational trade-offs compared to recent models with large-kernel convolutions. Could the authors provide further detail on any specific optimizations that contribute to this efficiency?
Suggestion: Including a breakdown of computational complexity relative to specific design choices (like RPWConv and RaGLU) and a comparison against representative models would enhance clarity on how RaCNN maintains low computational cost.
Scalability Across Tasks and Architectures:

Question: RaCNN demonstrates strong performance across multiple vision tasks. How does its performance and efficiency vary when scaling to larger or smaller model variants, particularly in dense prediction tasks? Are there configurations where RaCNN’s advantages diminish?
Suggestion: It would be helpful if the authors provided more details on the model's performance when scaled to different architectures and task demands, particularly in cases where efficiency gains may be less pronounced.
Impact of Region Size and Sparse Global Regions:

Question: RaCNN’s RPWConv operates on sparse global regions, which raises questions about the impact of different region sizes on performance and the possibility of information loss. Could the authors clarify how they determine optimal region sizes and the robustness of RaCNN to varying region granularity?
Suggestion: An ablation study on the impact of different region sizes and sparsity levels on accuracy and computation might help clarify the balance between global context capture and computational efficiency.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730883174616

thread_type:
official_review

<thread_object>
creator: reviewer_ppek

<content>
summary:
The paper introduces Region-aware CNN (RaCNN), a novel architecture that achieves a global receptive field without added complexity, outperforming state-of-the-art models. RaCNN features two key modules: the Region-aware Feed Forward Network (RaFFN) using Region Point-Wise Convolution (RPWConv) to capture global visual cues, and the Region-aware Gated Linear Unit (RaGLU) for capturing long-range dependencies. With only 84% of the computational complexity, RaCNN surpasses the performance of MogaNet (83.9% vs. 83.4%) and demonstrates strong scalability, excelling in object detection, instance segmentation, and semantic segmentation.

strengths:
1. RaCNN includes two critical modules: the Region-aware Pointwise Convolution (RPWConv) for capturing global visual information and the Region-aware Gated Linear Unit (RaGLU) for capturing long-range dependencies.
2. RaCNN outperforms state-of-the-art models like MogaNet, achieving higher performance (83.9% vs. 83.4%) with only 84% of the computational complexity.
3.  The model demonstrates excellent performance across various tasks, including object detection, instance segmentation, and semantic segmentation.The paper provides a thorough comparison with other methods across multiple downstream tasks, showcasing the robustness and effectiveness of RaCNN.

weaknesses:
1. While RaCNN shows improvements over existing models, the performance gains might be considered incremental. 
2. The ablation experiments are insufficient, lacking further analysis on the effectiveness of the proposed modules. For example, the dynamic PWconv use cosine distance to measure similarity instead of inner-product, how about the difference affect the performance ?
3. From the definition of the dynamic PWconv and RaGLU in the paper, and its implementation in the code, the proposed method does not capture the global receptive field of imagesm which is inconsistent with the paper's title.  Since the dynamic PWconv are improvement upon the basic of Pointwise Convolution, the dynamic wieghts generated for different region just provide different ways for gatherring features along the channel dimension.  The region attention in RaGLU is also a kind of channel attention .All the operation about the spatial gathering remains conventional 3x3 convolution. So where the global receptive field exhibits?

questions:
The questions have been listed in the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732503548777

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response (part 1)

comment:
We are grateful to all the reviewers for their efforts and valuable feedback!

We appreciate all the reviewers for their positive evaluation as below:

+ **Valuable motivation:** Our effort to overcome the high computational complexity of large kernel is valuable (Reviewer `PZ92`).
+ **Novel methods:** The proposed RPWConv and RaGLU incorporate a novel region-based dynamic weighting approach (Reviewer `zCME`), helping RaCNN to better capture global cues and local information (Reviewer `pPeK`, `EQAG`, `zCME`).
+ **Outstanding empirical performance:** Extensive experiments (Reviewer `CKsH`) effectively prove that RaCNN outperforms state-of-the-art CNNs and ViTs with lower computational complexity (Reviewer `pPeK`, `zCME`). The efficiency is an advantage of RaCNN (Reviewer `CKsH`).
+ **Versatility and scalability:** RaCNN achieves excellent performance across various tasks (Reviewer `pPeK`, `EQAG`, `CKsH`, `zCME`, `PZ92`) and different variants (Reviewer `zCME`), demonstrating robustness and effectiveness (Reviewer `pPeK`, `zCME`).

Here we answer some common questions shared by several reviewers.

**[Common Question 1] Some reviewers have questions about how RaCNN achieves global receptive fields with RPWConv and RaGLU**

Global receptive fields refer to the ability to focus on the entire image. In both modules, **sparse global view** plays a crucial role to obtain global receptive fields. Each spatial position in the output features is computed based on global sparse spatial features (i.e. positions with the same color in Figure 4(c)), enabling global receptive fields.

RPWConv: Given a feature map with shape $C\times H\times W$, RPWConv firstly divides it into $\frac{H}{rs}\times \frac{W}{rs}$ square spatial regions, where $rs$ is the region size. Then, pixels at the same relative position within all regions are sampled to form a **sparse global window** (renamed from "dilated window" in the original manuscript to avoid discrimination mentioned by Reviewer `EQAG`) and these pixels are aggregated into a vector in each channel. In other words, positions with the same color form a sparse global window. Dynamic PWConvs are applied in each sparse global window to generate dynamic weights tailored for that window and perform channel mixing. **The global perception capability of RPWConv stems from the sparse global window**, as each window gathers information from a global range and offers a global view of the image. Moreover, compared to Dynamic RWPConv, which shares the same dynamic weights across all positions, RPWConv generates unique weights for each sparse global window.

RaGLU: **Its ability to capture long-range dependencies derives from the sparse global pooling**. In the Region Attention, **sparse global average pooling** is employed to downsample spatial features. Each position in the resulting low-resolution features is averaged from scattered spatial positions in the input features, thus embedding global spatial cues.

To clarify this common question, we have provided additional explanations in the revised PDF. Figures 4 and 5 have been updated for better clarity, and we have added captions to Figure 5 to better explain the sparse global pooling in Region Attention.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes  a Region-aware CNN that achieves a global receptive field without requiring extra complexity. It shows good scalability and surpasses some of existing sota lightweight models. The reviewers raised concerns about limited technical contributions, incremental improvement, insufficient ablation experiments, mechanism to capture global information, confusion about convolution and local attention. Despite the authors' efforts in the rebuttal, some of the concerns remain unresolved. The final scores of the paper are unanimous negative. Therefore, the paper is rejected based on the reviewers' feedback.