Exploring contextual modeling with linear complexity for point cloud segmentation

Authors: Yong Xien Chng, Xuchong QIU, Henry Zheng, Yizeng Han, Yifan Pu, Jiewei Cao, Gao Huang
Primary area: applications to robotics, autonomy, planning
Keywords: point cloud segmentation, efficient, contextual modeling

Abstract:
Point cloud segmentation is an important topic in 3D understanding that has traditionally been tackled using either the CNN or Transformer. Recently, Mamba has emerged as a promising alternative, offering efficient long-range contextual modeling capabilities without the quadratic complexity associated with Transformer's attention mechanisms. However, despite Mamba's potential, early efforts have all failed to achieve better performance than the best CNN-based and Transformer-based methods. In this work, we address this challenge by identifying the key components of an effective and efficient point cloud segmentation architecture. Specifically, we show that: 1) Spatial locality and robust contextual understanding are critical for strong performance, and 2) Mamba features linear computational complexity, offering superior data and inference efficiency compared to Transformers, while still being capable of delivering strong contextual understanding. Additionally, we further enhance the standard Mamba specifically for point cloud segmentation by identifying its two key shortcomings. First, the enforced causality in the original Mamba is unsuitable for processing point clouds that have no such dependencies. Second, its unidirectional scanning strategy imposes a directional bias, hampering its ability to capture the full context of unordered point clouds in a single pass. To address these issues, we carefully remove the causal convolutions and introduce a novel Bidirectional Strided SSM to enhance the model's capability to capture spatial relationships. Our efforts culminate in a novel architecture named MEEPO that effectively integrates the strengths of CNN and Mamba. MEEPO surpasses the previous state-of-the-art method, PTv3, by up to +0.8 mIoU on multiple key benchmark datasets, while being 42.1\% faster and 5.53$\times$ more memory efficient. Our code will be released.

Decision: reject
Reviewer scores: 8, 5, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729172074533

thread_type:
official_review

<thread_object>
creator: reviewer_fevg

<content>
summary:
Point cloud segmentation is crucial for 3D understanding, traditionally tackled using CNNs or Transformers. Recently, Mamba has emerged as an efficient solution for contextual modeling, though it has struggled to outperform leading CNN and Transformer methods. This work identifies key requirements for effective segmentation: strong spatial locality and robust contextual understanding. Enhancing Mamba, the authors remove causality and introduce a Strided Bidirectional SSM to address directional biases in unordered point clouds. The resulting architecture, MEEPO, merges the strengths of CNNs and Mamba, achieving up to +0.8 mIoU over state-of-the-art methods on benchmarks like ScanNet and nuScenes, with notable gains in speed and memory efficiency.

strengths:
1. The paper is well-structured with a clear analysis-driven approach.
2. The experimental results are impressive.

weaknesses:
1. In lines 238-241, it's unclear why Mamba is considered effective for local processing, given its design goal of modeling long sequences with linear complexity. The reference to a "locally-biased forget gate" needs further explanation and detailed analysis or statistical data beyond visualization.

2. In lines 300-302, the statement that long-range attention is unnecessary may only partially reflect the issue. The core reason might be the sparsity of 3D point clouds. Transformers generally require extensive data and prolonged training to surpass CNNs, so it might be more accurate to say that insufficient data limits the full potential of Transformers rather than implying long-range attention is irrelevant.

3. The innovations appear somewhat limited, with minimal structural changes. The modifications, such as Causal-Free Mamba and Bidirectional Strided SSM, feel more like tricks.

4. Presenting results on downstream perception tasks like segmentation and detection would add further value.

Minor Errors:
- Line 251: Fig.4 should be Fig.5.
- clarify the point cloud processing order in Mamba. Is it similar to PTv3's order?

questions:
see above weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1729937594982

thread_type:
official_review

<thread_object>
creator: reviewer_xay8

<content>
summary:
This paper analyzes several popular neural network architectures, including CNN-based and Transformer-based designs, as well as the recently introduced Mamba model. It proposes a new point cloud segmentation architecture, MEEPO, which combines the strengths of CNNs and Mamba, surpassing previous state-of-the-art methods like PTv3 on multiple key benchmark datasets.

strengths:
1. The model is highly efficient and achieves accuracy that exceeds prior state-of-the-art methods.
2. The design of the Strided Bidirectional SSM effectively enhances the model's understanding of spatial relationships.

weaknesses:
1. The proposed hyper-structure is based on the analysis of existing architectures, which makes the technical innovation somewhat limited.

questions:
I am unsure whether the authors of this paper are aware of *Point Cobra*. I would like the authors to address the numerous similarities between the two papers and provide an explanation.

1. While the new model structure aligns with the previous version (NIPS2024), there are differences in latency and memory usage. I am interested in understanding the source of these improvements.

2. This paper introduces an additional CNN module to capture the local features of the point cloud. Could the Causal-Free Conv1D in the Mamba block be removed, given that these two modules appear to serve the same role in the model?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730616961818

thread_type:
official_review

<thread_object>
creator: reviewer_wi14

<content>
summary:
This paper presents a novel architecture named MEEPO, designed for point cloud segmentation with a focus on efficient contextual modeling. It introduces Mamba, a state-space model (SSM) that achieves linear complexity compared to traditional Transformers, which have quadratic complexity. The authors identify Mamba's limitations, such as enforced causality and directional bias, and propose solutions through causal-free convolutions and bidirectional strided state-space modeling. MEEPO outperforms previous models, like PTv3, in accuracy, latency, and memory efficiency across several benchmark datasets (e.g., ScanNet, S3DIS, nuScenes).

strengths:
1.	MEEPO introduces a new method by integrating CNN and Mamba components, achieving good contextual modeling with reduced computational costs.
2.	The paper thoroughly explores the limitations of existing architectures (CNN, Transformer, and SSM) and provides comprehensive ablations and comparisons to validate its design choices.
3.	The introduction of causal-free convolutions and bidirectional strided SSM in Mamba addresses some original limitations in Mamba for this task,.

weaknesses:
1.	In Section 4.2, the authors claim that they propose a 'Bidirectional Strided SSM' method, however, Bidirectional SSMs have already been proposed in works like [1][2] to address the issue of forgetting unidirectional sequences. I believe that not referencing these articles while claiming authorship is not rigorous. Additionally, what baffles me is that in Table 7, a comparison is made for strides ranging from 2 to 16, showing a decreasing trend in performance. So, why not similarly compare and discuss the results for a stride of 1?
2.	The manuscript presents in Fig. 4 the mIoU performance of the Transformer-based PTv3 under different range windows. The advantage of the Mamba model lies in its better memory capabilities for long sequences. The authors emphasize this advantage in line 240, but they do not provide specific data to demonstrate the validation of this conclusion in point cloud data  (the author can show a similar format figure as Fig.4).
3.	Table 7 in the article indicates that the parameter volume of CNN+Mamba is smaller than that of pure Mamba. However, based on the author's description, compared to pure Mamba, the author has added a CNN module to Mamba and employed a bidirectional mechanism for computation. Why the parameter count is lower than pure Mamba needs further clarification from the author.
4.	In line 251, the referred figure is not Fig.4, it should be Fig. 5.


[1] Bi-Mamba+: Bidirectional Mamba for Time Series Forecasting
[2] Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Model

questions:
please refer to the weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642447535

thread_type:
official_review

<thread_object>
creator: reviewer_hj9v

<content>
summary:
This paper focuses on the 3D point cloud segmentation task. It first compares the performance of three types of blocks: CNN, Transformer, and Mamba, and then improves the existing Mamba architecture. The authors provide extensive visualization results and ablation experiments.

strengths:
[S1] It is meaningful to compare the performance of the three types of blocks—CNN, Transformer, and Mamba—on the same task, point cloud segmentation.
[S2] The paper is detailed and easy to read.
[S3] The paper conducts detailed ablation experiments on the newly proposed module.
[S4] The paper achieves state-of-the-art results on multiple datasets.

weaknesses:
[W1] The comparison of CNN, Transformer, and Mamba blocks lacks strong evidence. I believe that concluding "CNN is more effective for local modeling and Transformer is better at handling contextual information" based on just one example for each is insufficient.
[W2] The proposed method is more like a small fix to Mamba-based method. 
[W3] The two solutions proposed in this paper: 1) the casual-free block is described in very little detail and lacks a clear explanation; 2) bidirectional SSM seems to have already been introduced in VisionMamba (Zhu 2024), and this paper's work only adds n-stride.

questions:
[Q1] Could the authors provide more detailed explanations of W1, such as whether there is theoretical evidence to support it or if there are sufficiently extensive experiments across multiple datasets?
[Q2] Is the Bidirectional SSM mentioned by the authors consistent with VisionMamba (Zhu 2024)? I do not see a bidirectional structure in Fig(6b), only the Strided SSM.
[Q2.5] If the answer of Q2 is yes, the result you reported in the second line of Tab(7d) shows that this structure can only bring 0.1% increasement, which seems inconsistent with the effects reported in VisionMamba (Zhu 2024) regarding this structure. From an efficiency perspective, is the introduction of the indirection necessary if it only brings a 0.1% improvement?
[Q3] The casual-free conv block is not clearly explained. If this is something new you are proposing, it would be helpful to include a more detailed explanation or illustration.
[Q4] Standardize the notation for SSM. In the abstract and introduction, it is referred to as "Strided Bidirectional SSM," while later it is called "Bidirectional Strided SSM."
[Q5] There’s something wrong of your hyperlink in 3.3. It should be Fig 5 but yours is Fig 4.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730681455160

thread_type:
official_review

<thread_object>
creator: reviewer_u9m8

<content>
summary:
This paper focuses on a simple yet practical target in 3D understanding: how to enhance the accuracy of a Mamba-based framework while preserving its efficiency rooted in linear complexity. The paper seeks to achieve this through an in-depth analysis of the components that contribute to the strong performance of PTv3, summarized as contextual understanding and spatial locality. These findings lead to the two major design elements of this work: Causal-free Mamba and Bidirectional Strided SSM. Overall, it is encouraging to see the method achieve solid performance on major scene-level point cloud semantic segmentation benchmarks.

strengths:
1. **[Analysis-driven Methodology]** First of all, I appreciate the paper's analysis-driven approach: rather than being experiment-driven, it presents insights based on previous work and derives its methodology from this understanding. This makes the paper easier to follow and the proposed method more convincing.

2. **[Strong performance]** The paper achieves strong performance on several major point cloud semantic segmentation benchmarks and appears to be efficient as well.

weaknesses:
The insightful analysis, convincing approach, and solid performance demonstrate that this manuscript makes **a valuable empirical contribution**. However, the following weaknesses **limit its broader impact**:

1. **[Scaling up]**  
**(a)** Many observations and analyses (such as the ablation on PTv3 window size) are based on training from scratch on ScanNet (1,500 samples), which is relatively small in scale. However, many model properties may change significantly when scaled up with more data. For instance, using the PTv3 window size ablation as an example, attention mechanisms are inherently adaptive to kernel size, meaning accuracy should not be negatively impacted by the window size. The degradation observed when increasing the window size beyond 1024 is likely due to insufficient data to support this adaptive capacity.  
**(b)** It is good to see the Mamba framework achieve both higher accuracy and efficiency compared to previous SOTA. However, the true value of superior scratch accuracy and efficiency lies in its capacity for further scaling. I am particularly curious about the model's accuracy and efficiency when scaling up training through multi-dataset joint training, as well as its performance when scaling up parameters with larger data volumes.

2. **[Go beyond semantic segmentation]**
Why do point cloud perception and representation learning prioritize semantic segmentation? This is because it is the simplest way to evaluate the quality of learned representations using a single linear layer. However, this does not mean that research on point cloud backbones should be limited to semantic segmentation alone. The claims of the paper would be stronger if more downstream tasks, such as instance segmentation and object detection, were included. It may not be necessary to achieve the highest performance; instead, demonstrating more properties of the proposed method could be more informative.

questions:
1. In Figure 1(b), PTv3 consumes even more memory compared to PTv2, which seems unusual since window-based attention should be significantly more memory-efficient than neighborhood attention. The only reason I can imagine is that FlashAttention may be disabled while using a large kernel, resulting in larger matrix multiplications. Could you provide a more detailed explanation of this setup?

2. Maybe a shorter, more impactful title could make it easier for readers to remember? The current version is too lengthy.

3. Maybe the table arrangement could be improved? I don’t recommend using resizebox, as it can lead to uneven text sizes. You might consider referring to the LaTeX source code from the PTv3 paper for tips on adjusting table formatting.

4. It might be better to reduce the use of bold text? For instance, consider changing `\textbf{Proposed Solution:}` to` \textit{Proposed solution:}`, and avoid bolding certain numbers in the main text.

(Minor suggestions for reference only)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732685605975

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
Dear Reviewers and Area Chair,

We sincerely thank you for your valuable time and effort in reviewing our paper.

In this paper, we provide a comprehensive analysis of architectural properties of commonly used point cloud segmentation operators, investigate the reasons behind Mamba-based networks' under-performance in this task, and propose simple yet effective solutions to address these issues. These insights culminate in the development of a novel architecture for point cloud segmentation, named MEEPO, which achieves state-of-the-art performance. Our research offers many valuable insights for designing efficient and effective point cloud segmentation networks.

All the reviewers have acknowledged several strengths of our paper, including its **clear structure and readability (HJ9v, fEVg)**, **meaningful and thorough analysis (HJ9v, U9M8, wi14)**, and **comprehensive ablation experiments (HJ9v, wi14)**. Additionally, there is unanimous agreement regarding the **impressive performance and efficiency** of our proposed network.

Most weaknesses mentioned are minor issues related to writing presentations and clarity. In response to the constructive feedback given, we have carefully revised the manuscript, making the following key improvements:

1. Enhanced the explanation of PTv3's performance degradation with increasing window size (lines 234 and 298)
2. Added a stride=1 comparison in Table 7(e) (line 446)
3. Incorporated a Strided SSM ablation study in Table 7(d) (line 441)
4. Clarified parameter count details in Table 7(a) (lines 196 and 469)
5. Provided more detailed explanation of Mamba's point cloud processing order (line 189)
6. Expanded the discussion of Mamba's local bias (line 822 in the Appendix)
7. Included data demonstrating MEEPO's effectiveness in processing increasingly larger point sets (line 841 in the Appendix)
8. Reduced bold text usage (lines 357 and 370)
9. Corrected the Figure 5 reference (line 251)
10. Added citation to Bi-Mamba+ (line 518)

We are pleased to note that all the engaged reviewers expressed satisfaction with these revisions, awarding positive scores (8, 6, 6, 6) after their evaluation. We have also carefully addressed the weaknesses raised by reviewer HJ9v but was unable to obtain his/her response. All in all, we strongly believe that our revised paper has met the rigorous standards required for an ICLR submission.

Thank you again for your thoughtful feedback and support. 

Sincerely,

The Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper receives 4 positive and 1 negative rating after rebuttal. Although the paper has some merits like competitive results with faster runtime and lower memory cost, the reviewers pointed out a few critical concerns about 1) technical contributions compared to other Mamba-based approaches, and 2) results other than semantic segmentation. After taking a close look at the paper, rebuttal, and discussions, the AC agrees with reviewers' feedback, especially on the minor architectural change of the existing Mamba methods but applying to the studied task. Without further explanations about the main contributions or additional results, e.g., other point cloud tasks, it is not convincing to show enough technical novelty, and hence the AC suggests the rejection decision. The authors are encouraged to improve the paper based on the feedback for the next venue.