State Space Model Meets Transformer: A New Paradigm for 3D Object Detection
Authors: Chuxin Wang
Wenfei Yang
Xiang Liu
Tianzhu Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Point Cloud; 3D Object Detection; State Space Model
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: This study introduces a novel paradigm, DEST, which significantly enhances 3D object detection performance on two challenging datasets through the proposed interactive state space model.

Abstract: DETR-based methods, which use multi-layer transformer decoders to refine object queries iteratively, have shown promising performance in 3D indoor object detection. However, the scene point features in the transformer decoder remain fixed, leading to minimal contributions from later decoder layers, thereby limiting performance improvement. Recently, State Space Models (SSM) have shown efficient context modeling ability with linear complexity through iterative interactions between system states and inputs. Inspired by SSMs, we propose a new 3D object DEtection paradigm with an interactive STate space model (DEST). In the interactive SSM, we design a novel state-dependent SSM parameterization method that enables system states to effectively serve as queries in 3D indoor detection tasks. In addition, we introduce four key designs tailored to the characteristics of point cloud and SSM: The serialization and bidirectional scanning strategies enable bidirectional feature interaction among scene points within the SSM. The inter-state attention mechanism models the relationships between state points, while the gated feed-forward network enhances inter-channel correlations. To the best of our knowledge, this is the first method to model queries as system states and scene points as system inputs, which can simultaneously update scene point features and query features with linear complexity. Extensive experiments on two challenging datasets demonstrate the effectiveness of our DEST-based method. Our method improves the GroupFree baseline in terms of $\text{AP}_{50}$ on ScanNet V2 (+5.3) and SUN RGB-D (+3.2) datasets. Based on the VDETR baseline, Our method sets a new state-of-the-art on the ScanNetV2 and SUN RGB-D datasets.

Published as a conference paper at ICLR 2025
STATE SPACE MODEL MEETS TRANSFORMER: A NEW PARADIGM FOR 3D OBJECT DETECTION
Chuxin 2 Wenfei Yang Xiang Liu", Tianzhu Zhang 12.33
Universityy of Science and Technology of China Nationaa Key Laboratory of Deep Space Exploration, Deep Space Exploration Laboratory Hainan Aerospace Technology Innovation Center *Dongguan University of Technology { (wex0602, yangwf] @ mail.ustc.edu.co liuxiang @ dgut.edu.cn, tzzhang @ustc.edu.cn
ABSTRACT
DETR-based methods, which use multi-layer transformer decoders to refine ob- ject queries iteratively, have shown promising performance in 3D indoor object detection. However, the scene point features in the transformer decoder remain fixed, leading to minimal contributions from later decoder layers, thereby limit- ing performance improvement. Recently, State Space Models (SSM) have shown efficient context modeling ability with linear complexity through iterative inter- actions between system states and inputs. Inspired by SSMs, we propose a new 3D object DEtection paradigm with interactive STate space model (DEST). In the interactive SSM, we design a novel state-dependent SSM parameterization method that enables system states to effectively serve queries in 3D indoor detection tasks. In addition, we introduce four key designs tailored to the char- acteristics of point cloud and SSM: The serialization and bidirectional scanning strategies enable bidirectional feature interaction among scene points within the SSM. The inter-state attention mechanism models the relationships between state points, while the gated feed-forward network enhances inter-channel correlations. To the best of our knowledge, this is the first method to model queries as sys- tem states and scene points as system inputs, which can simultaneously update scene point features and query features with linear complexity. Extensive exper- iments on two challenging datasets demonstrate the effectiveness of our DEST- based method. Our method improves the GroupFree baseline terms of APso on ScanNet V2 (+5.3) and SUN RGB-D (+3.2) datasets. Based the VDETR base- line, Our method sets a new SOTA on the ScanNetV2 and SUN RGB-D datasets.
INTRODUCTION
With the widespread application of LiDAR and depth cameras, it is becoming easier to obtain 3D point clouds of real scenes. The large amounts of 3D scene data provide rich geometric information for 3D scene understanding in fields such as autonomous driving, robotics, and augmented reality. As a fundamental task in 3D scene understanding, 3D indoor object detection has garnered sig- nificant attention from both academia and industry. Unlike 3D object detection (Shi et al., 2019; Yin et al., 2021; Lang et al., 2019; Shi et al., 2020b;a) autonomous driving scenarios, 3D indoor object detection involves objects with more diverse categories and shapes, posing more significant challenges for the model design and training.
To address the above challenges, numerous 3D indoor object detection methods have been proposed, which can be roughly divided into three categories: vote-based methods (Qi et al., 2019; Xie et al., 2020; Zhang et al., 2020), expansion-based methods (Gwak et al., 2020; Rukhovich et al., 2022; Wang et al., 2022a), and DETR-based methods (Misra et al., 2021; Liu et al., 2021; Wang et al., 2023; Shen et al., 2024). Vote-based methods (Qi et al., 2019; Xie et al., 2020; Zhang et al., 2020) use a voting mechanism to shift surface points toward the object center and then generate candi- date points by clustering the points that have shifted the same regions. Although these methods
"Corresponding author.

---

Published as a conference paper at ICLR 2025
Somo Frinss
2007
VOVEA
Deodle lijer
Transfeas Doosker lazer
Dousinn ltcd
Theesteae
Trasslenze
Quayy Poiza
ISSM bood
1SSV-booc
Decsterr loper
busd
Deistee Hord
Danala
Quary Pozza
(o)Comparin of Trarafommerad and Deener Layers
(b) Performence Improvemennt Arnoss Pafforrnt Layan
Figure 1: (a): Transformer decoder solely updates the features of the query points, while our
ISSM-based decoder simultaneously updates the features of scene points and query points. (b): The DETR- models show only slight accuracy enhancements in the later layers, whereas the DEST-based methods significantly boost the performance in the later layers.
have achieved great success in 3D object detection, The vote mechanism is performed in a category- independent manner, leading to the shifted points that are adjacent but belong to different cate- gories being grouped, which limits the model detection capability. Expansion- et al., 2020; Rukhovich et al., 2022: Wang et al., 2022a) use generative sparse decoders to generate high-quality proposals based on object surface voxel features with the same semantic prediction. Compared with vote-based methods, expansion-based methods consider the semantic consistency of voxels within the same group and achieve better performance. However, expansion-based meth- ods require a carefully designed proposal generation module and involve numerous manually set thresholds, which limits the model versatility. Recently, DETR-based methods (Misra et al., 2021; Liu et al., 2021; Wang al., 2023; Shen et al., 2024) have shown promising performance in 3D object detection. Unlike the above methods, or points as the initial object queries and use the scene point features to refine these queries. The query refinement module is simple in design and retains the original geometric structure of the input 3D point cloud. Based on the query refinement module, DETR-basce methods have achieved the best performance in indoor object detection tasks. However, DETR- methods still face a key issue that limits their performance. These methods employ multi-layer transformer decoders (Car- ion et al., 2020) to iteratively refine the object queries. While the transformer decoder layers update the query point features, they do not simultaneously update the scene point features as shown in Figure (a). As result, each decoder layer uses the same scene point features refine the queries, leading to only marginal improvements from the later layers. As shown in Figure (b), we evaluate the detection accuracy improvements of different decoder layers in two DETR-based models (Liu et al., 2021: Shen et al., 2024) on the ScanNet V2 (Dai et al., 2017) dataset. GroupFree (Liu et al., 2021) achieves only 2.12 performance improvement in the last six layers, while VDETR (Shen et al., 2024) shows a mere 0.64 improvement in the last four layers. Based on the above analysis, the fixed scene point features constrain the potential performance enhancement of the models.
methods (Gwak
BETR-based methods select a small group of voxels
To address this issue, an intuitive idea is to introduce self-attention mechanism (Vaswani, 2017) be- tween different decoder layers to update the scene point features. However, the quadratic complexity of the self-attention mechanism significantly reduces the model efficiency, making this approach im- practical. Recently, state space moddl-besed methods (Gu et al., 202 Gu & Dao, 2023; Dao & Gu, 2024) have shown efficient context modeling with linear complexity through interactions between system states and inputs. The pioneer works lead us to think: Is it possible to design a State Space Model (SSM) to replace the transformer decoder, enabling the simultaneous update of scene features and query point features? SSM is used to describe the evolution of system states and to predict future states and system outputs based on system inputs. Therefore, if model the query point features as the system states and the scene point features as the system inputs at different time steps, we can simultaneously obtain the final system states (updated query point features) and the system outputs at each time step (updated scene point features). However, existing SSMs (Gu et al., 2021a; Gu & Dao, 2023; Dao & Gu, 2024) are not suitable for modeling queries as system states, for two main reasons: (1). Update the states solely based on the system inputs. Existing SSMs (Gu & Dao, 2023; Dao & Gu, 2024) adjust the SSM parameters (A, B, C) solely using system inputs without considering the system states. Therefore, the state points cannot adaptively select system in- puts to update themselves, while different query points need to focus on distinct regions of the scene. (2). Cannot directly process 3D point cloud. Existing SSMs are inherently designed to process sequential data, and their unidirectional modeling and sensitivity to input order pose challenges for feature modeling of scene points and query points.
2

---

Published as a conference paper at ICLR 2025
Based on the above discussion, we propose new 3D object DEtection paradigm with STate space model (DEST) to address the performance limitation caused by the fixed scene point features. The proposed DEST consists of two core components: novel Interactive State Space Model (ISSM) and an ISSM-based decoder. In the ISSM, we model the query point features as the system states and the scene point features the system inputs at different time steps. Unlike previous SSMs (Gu et al., 2021a; Gu & Dao, 2023; Dao & Gu, 2024), the proposed ISSM determines how to update the system states based on both the system states and system inputs. Specifically, we modify the SSM parameters (A, B. C) to be dependent on the system states and design a spatial correlation module to model the relationship between state points and scene points. Therefore, the system states in the ISSM can effectively fulfill the role of queries in complex 3D indoor detection tasks. In the ISSM-based decoder, four modules are designed for feature modeling of scene points and query points: Hilbert-based point cloud serialization strategy, ISSM-based Bidirectional Scan (IBS) module, Inter-state attention module, and Gated Feed-Fore werd-ro Network (GFFN). The proposed se- rialization strategy is designed to serialize the scene points based on the Hilbert curve (Hilbert & Hilbert, 1935), benefiting from its locality-preserving properties. The IBS module is designed to achieve bidirectional interaction among different scene points, while the inter-state attention mod- ule is designed to capture the relationships between state points. Lastly, the GFFN is designed to enhance inter-channel correlations through gated linear unit. The ISSM-based decoder can replace the transformer decoder in BETR-based methods to address the performance limitations caused by fixed scene point features. As shown in Figure 1 (b), our DEST- method significantly enhances the performance in the later layers.
In summary, the core contributions of this paper are follows: (I). We propose novel SSM-based 3D object detection paradigm DEST to overcome the performance limitations caused by fixed scene point features during the query refinement process. To the best of our knowledge, this is the first method to model queries as system states within an SSM framework. (2). We design novel ISSM whose system states can effectively function as queries in complex 3D indoor detection tasks. In addition, develop ISSM-based decoder tailored to the characteristics of 3D point clouds, fully harnessing the potential of the ISSM for 3D object detection. (3). Extensive experimental results demonstrate that the proposed SSM-based 3D object detection method consistently enhances the performance of baseline detectors on two challenging indoor datasets, i.e., ScanNet V2 (Dai et al., 2017) and SUN RGB-D (Song et al., 2015). Moreover, comprehensive ablation studies validate the effectiveness of each designed component.
2 RELATED WORK
2.1 3D OBJECT DETECTION.
The goal of 3D object detection is to estimate oriented 3D object bounding boxes with their category labels from a point cloud. According to the application scenario, 3D object detection is typically divided into outdoor and indoor detection tasks. Outdoor 3D object detection is commonly used in autonomous driving scenes, where objects are primarily distributed across a wide 2D plane. There- fore, outdoor 3D detection methods typically project the 3D point cloud into bird's-eye view (BEV) and utilize 2D convolutional networks to detect 3D objects. For instance, MV3D (Chen al., 2017) directly projects the point cloud onto 2D grid for feature processing and detection. VoxeINet (Zhou & Tuzel, 2018) first converts the point cloud into a 3D volumetric grid and uses 3D CNN for feature extraction. Then, it projects the 3D voxels into a BEV for bounding box prediction. PointPillars (Lang et al., 2019) employs PointNet (Qi et al., 2017a) to learn point cloud representations organized in vertical columns (pillars), then uses a 2D convolutional neural network to process flattened pillar features in the BEV.
In contrast, indoor 3D object detection involves handling a more diverse set of object categories and shapes, as well more complex spatial relationships between objects. Existing indoor 3D object detection methods can be broadly categorized into three groups: vote-based methods, expansion- based methods, and bated methods. For vote-based methods, VoteNet (Qi et al., 2019), as a pioneering work, designs& a novel 3D proposal mechanism based on deep Hough voting. ML- CVNet (Xie al., 2020) introduces three context modules in the voting and classifying stages of VoteNet to encode contextual information at different levels. BRNet (Cheng et al., 2021) backtraces representative points from the voting centers better capture the fine local structural features sur-
3

---

Published as a conference paper at ICLR 2025
rounding the potential objects from the raw point clouds. H3DNet (Zhang et al., 2020) predicts a hybrid set of geometric primitives and converts the predicted geometric primitives into object proposals. For expansion-basee methods, GSDN (Gwak et al., 2020) proposes generative sparse tensor decoder to generate virtual center features from surface features while discarding unlikely object centers. FCAF3D (Rukhovich et al., 2022) further introduces fully convolutional anchor- free indoor 3D object detection method. CAGroup3D (Wang et al., 2022a) generates high-quality 3D proposals by leveraging a class-aware local grouping strategy on object surface voxels with con- sistent semantic predictions. The above methods require carefully designed proposal generation modules and involve several manually set thresholds. DETR (Carion et al., 2020) is a pioneering work that applies Transformers (Vaswani, 2017) to 2D object detection, eliminating many hand- crafted components such as Non-Maximum Suppression (Neubeck Van Gool, 2006) or anchor boxes (Girshick, 2015; Ren, 2015; Lin, 2017). Currently, DETR and its variants (Zhu et al., 2020; Dai et al., 2021; Liu et al., 2022) have achieved state-of-the-art results in various 2D object detec- tion tasks. Inspired by these works, numerous DETR-based 3D object detection methods have been explored. 3DETR (Misra et al., 2021) is the first to introduce an end-to-end Transformer model for 3D object detection, achieving promising results. GroupFree (Liu et al., 2021) employs key point sampling strategy to select candidate points and utilizes the attention mechanism to update query point features. LeadNet (Wang et al., 2023) further improves the transformer decoder by introduc- ing dynamic object query sampling module and dynamic Gaussian weight map. Most recently, VDETR (Shen et al., 2024) proposes a novel 3D vertex relative position encoding method, which di- rects the model to focus on points near the object, achieving these DETR- based methods use fixed scene point features in different decoder layers, which limits the detection capabilities of later layers. Unlike the above methods, we propose an ISSM-based decoder that simultaneously updates both scene point and query point features.
fffte-of-theaale performance. However,
2.2 STATE SPACE MODELS (SSMS).
Recently, SSMs (Kalman, 1960; Gu et al., 2021a;b) have become prominent research focus. S4 (Gu et al., 20) demonstrates the capability of capturing long-range dependencies with linear complex- ity. Mamba (Gu & Dao, 2023) further enhances S4 by introducing selection mechanism, specifi- cally parameterizing the SSM based on the system input. The selection mechanism allows Mamba to selectively retain information, facilitating the efficient processing of long sequence data. Inspired by Mamba, Vision Mamba (Zhu et al., 2024) introduces an JSM-based visual model. VMamba (Liu al., 2024) furthermore incorporates a 2D selective scan module, enabling the model to perform selective scanning of two-dimensional images. In the field of point cloud understanding. numer- ous Mamba-hased works have emerged. PointMamba (Liaagg et al., 2024) proposes a simple yet effective Mamba-based baseline, while PCM (Zhang et al., 2024) develops diverse point cloud seri- alization methods that significantly improve performance. These methods have achieved promising results by leveraging the efficient context modeling and linear complexity of Mamba. Unlike these methods that use SSMs to design feature encoders, we design an SSM-basee decoder to address the performance limitations caused by fixed scene point features in the transformer decoder.
METHOD
Below, we first briefly review the existing SSMs (Section 3.1), followed by overview of the proposed DEST (Section 3.2). Subsequently, we offer detailed explanations of the two core compo- nents: the Interactive State Space Model (Section 3.3) and the IbsM-based decoder (Section 3.4). Lastly, we outline the model setups for the two baseline models (Section 3.5).
3.1 PRELIMINARIES
SSM is used describe the evolution of system states h(t) E R and predict future states h'(t) and system outputs vt) based on system inputs st).. The system can be defined as follows:
N((1) = At(t) + Bx((), v(t) = Ct'()),
(I)
where A E IR K represents the state transition matrix that describes how the system states evolve, B E IR xx1 denotes the control matrix that describes the influence of the system inputs on the system states, and C E I is the observation matrix characterizing the impact of the system states on
4

---

Published as a conference paper at ICLR 2025
K" Boxes
or
M
State
KxC
ISSM-based Decoder
Sampling
Input
Encoder
Figure 2: The overall framework of the DEST-based method for 3D object detection. We first utilize an encoder to extract 3D features, followed by state sampling module to select state points, referred to as queries in DETR architecture. Subsequently, we imput both the scene points and state points into the ISSM-based decoder for simultaneous updates. Finally, the updated state points fed into a detection head to predict the 3D bounding boxes.
the system outputs. To handle discrete-tinge sequence data imputs, the Zero-Order Hold is typically used to discretize the SSM. The discretized version of equation is as follows:
hi Aht-1 + BIG, Y/ Cli,,
(2)
== exp AA), B = (AA) (exp((A)))))))))) I)AB, C=C
(3)
where A represents the timescale from system states h 1 the next heg The entire sequence transformation can also be represented in a convolutional form:
K = (CB, CAB, .... CA
B), y=IKKK
where N is the length of the input sequence x, and K RN denotes a global convolution kernel, which can be efficiently pre-computed. However, due to the Linear Time-Invarian (LTI) nature of SSM, the parameters (A, A, B, C) remain fixed across all time steps, which limits their ability to handle varying imput sequences.
Recently, Mamba (Gu & Dao, 2023) introduced selection mechanism that treats the parameters (A, B, C) functions of the input, effectively transforming the SSM into a time-varying model:
h4 = (=(x)())--11 + (g(()) Yt = (0(2)),,
(5)
where (2x), (Z)) and ()())) denote the parameter matrices are dependent the system inputs Zt.- While the selection mechanism addresses the limitations of the LTT model, also does not allow for parallel computation using equation 4. To tackle this challenge, Mamba introduced hardware-aware selective scanning, achieving near-linear complexity. Mamba2 (Dao & Gu, 2024) propose refinement version of the selective SSM by leveraging structured semiseparable matrices and the state space dual framework, further enhancing performance and efficiency. In this paper, we model the relationship between the system states and system inputs based on Mamba and Mamba2, adapting it to more challenging point cloud tasks.
3.2 OVERVIEW
Figure presents the overall framework the DEST-based method for 3D object detection. In 3D object detection point clouds, given set containing N points, the objective is generate set of 3D oriented bounding boxes with classification scores to cover all ground -ruth objects. The pro- posed DEST-based detector primarily consists of three components: features, a sampling module for generating the initial system states, and an ISSM-based decoder to refine system states and predict the 3D oriented bounding boxes. In this paper, we focus primarily on the decoder design, leveraging the SSM to facilitate the simultancous updating of query points and scene points, thereby mitigating the performance limitations.
encoder for extracting point
3.3 INTERACTIVE STATE SPACE MODEL.
In the ISSM, model the query points as the initial system states ho KxC and the scene points as the system inputs z E CC As shown Figure 3, we provide the overview of the ISSM. Compared to Mamba, expand the dimension of A to make it state-dependent and design a spatial correlation module to generate the SSM parameters (A,BC))
Extension of A. In Mamba, the important parameter A MxC controls the balance between how much to focus on or ignore the current input. Specifically, a larger A resets the states h,- to focus on the current input 24, while smaller A retains the states h and disregards the current
5

---

Published as a conference paper at ICLR 2025
Hg
RM
Spatial
Br
Conceltion
34
Figure 3: Overview of the Interactive State Space Model. In the ISSM, we model the query points as the system states and the scene points as the system inputs. We design a spatial correlation module to parameterize the SSM based the initial system states and inputs.
input T4.- However, in both Mamba and Mamba2, A only considers the system inputs without accounting for differences in initial system states ho. In 3D object detection tasks, query points have different positions across varying scenes, and different query points focus on different system inputs. Updating the states h by considering only the system inputs I prevents the state points from adequately focusing on their respective regions. To address this issue, we modify A to have distinct values for each system input and state, expanding A E R RMX to A E RR Mx XC Although the expansion of A affects the discretization formula as equation 3, does not impact the discretized state space model as equation 2. Therefore, the proposed ISSM can achieve efficient parallel computation by utilizing the hardware-awar selective scanning strategy (Gu & Dao, 2023) or the state space dual framework (Dao & Gu, 2024).
Spatial Correlation. The success of Mamba demonstrates that the state selection mechanism is crucial for SSMs. For 3D object detection tasks, the proposed ISSM needs address more complex state updating problem. A key challenge lies in enabling state points to select the ap- propriate scene points for self-updating. DETR-based methods (Wang et al., 2023; Shen et al., 2024) have inspired us that query points should primarily focus on points surrounding the rel- evant 3D bounding boxes. Therefore, we design spatial correlation module that encodes the spatial relationships between scene points x and initial state points hg to derive the parameters (A E RM Xx B RM C ) in the ISSM. Specifically, for each state point hi we first predict a rotated 3D bounding box. We then calculate the relative offsets A E IR M K x8x3 between the scene points and the eight vertices of the bounding box. Finally, we use an MLP to map these positional relationships to the parameters in the ISSM:
St Σ MLP(AP)). A = Linear.(S"). BA_ (inear((s)) C Linear (
(6)
where Lineara a parameterized projection to dimension d. Since there are numerous background points within the scene points, we need to prevent these from interfering with the system states. Therefore, we use the features the scene points to modify the parameters ISSM:
A BC (Linear.(x)) A s: B = (Linearr +B C = "B==Cia([ann (Lineari (x)) (7)
where BC₃ denotes broadcasting the values to K dimensions. Apart from modeling positional correlations and background information, design an explicit delay kernel for A
A
exp(a min(R(h)) - 0))
g
where R(ht) is the circumscribed sphere radius of the bounding box predicted by the state point hig P. denotes the position of the scene points and a is learnable parameter. With the parameters (Ai B', C)), the state points h can select the appropriate scene points x for updating, while the scene points 2 can simultaneously acquire surrounding structural information from the state points h. In practice, we observe that the numerous combinations between state points h and scene points result in significant memory overhead. Therefore, following VDETR (Shen et al., 2024), we utilize smaller predefined 3D table I e IR 10x10x10 to obtain St of equation 6 through grid sampling. In Appendix, we further discuss the mathematical relationship between the system states in the state space model and the query points in the transformer decoder.
3.4 ESSM-BASED DECODER
Based on the above ISSM, we further design an ISSM-based decoder suitable for 3D point cloud detection shown in Figure 4 (a). The ISSM- decoder consists of four core components:

---

Published as a conference paper at ICLR 2025
[ Linear
Limear
15SM-bbased
Bidirectional Scan
ACT
Foeward ISSM
Layer Norm
Layer Norm
ACT
Forward DW-Cone
ISSM
DW CORY
Lincer
Serializariin
Inier-atatc
Backward DW-Coom
Linear
Linear
Scene Points
State Points
Scane Poinnts
State Points
Points
(a) ISSM-based Decoder
(b) ISSM-based Bidirectional Scan
(0) Food-Forward Network
Figure 4: (a): Illustration of (SSM-based decoder architecture. (b): Detailed Structure of the ISSM-
based Bidirectional Scan. (c): Detailed Structure of the Gated Feed-Forward Network (GFFN).
a Hilbert-based point cloud serialization strategy, an inter-state attention module, an "SSM-based Bidirectional Scan (IBS) module, and Gated Feed-Forward Network (GFFN).
Hilbert-based point cloud serialization strategy. SSMs are designed for ordered ID sequences, which are not suitable for unordered point clouds. To model the scene points as the system inputs of the ISSM, we need to serialize the scene points. Following PTv3 (Wu et al., 2024), we lever- age space-filling curves to serialize point clouds. Among these space-filling curves, the Hilbert curve (Hilbert & Hilbert, 1935) is renowned for its efficient locality preservation. Thus, we generate six different scrialized results for the point cloud by reordering the x, y. and z axes of the Hilbert curve, aiming for comprehensive observation of the point cloud. The detailed serialization process shown in Appendix A.3. Additionally, unlike serialized attention in PTv3, the sequential feature modeling ISSM is more sensitive changes in the serialization method. Therefore, we apply dif- ferent serialization methods for each decoder layer without the shuffle order strategy used in PTv3, ensuring that the ISSM-based decoder can comprehensively capture scene point features.
Inter-state attention module. In 3D object detection, objects in a scene often exhibit strong cor- relations. For example, tables and chairs commonly appear together, while beds and toilets rarely coexist in the same room. In DETR-based decoder layers (Carion et al., 2020; Misra et al., 2021; Liu et al., 2021: Shen et al., 2024), the self-attention mechanism for queries is employed to model the correlations between different objects. However, in the ISSM, there is no design specifically for interactions between state points. To capture such relationships, we employ a standard self-attention mechanism for state points. The inter-state attention module allows states to capture richer features, particularly enhancing the detection performance for objects with ambiguous boundaries or those that are challenging to distinguish from the background.
IBS module. To facilitate bidirectional interaction among different scene points in a single pass, we follow Vision Mamba (Zhu et al., 2024) and introduce a bidirectional scanning mechanism into ISSM. Figure 4 (b) illustrates the detailed structure of the IBS module. Firstly, we input the forward- ordered and backward-ordered scene points into their respective forward and backward ISSMs. Both the forward and backward ISSMs use state points as system states, with each ISSM generating the updated features of the scene points and state points as outputs. We then fuse the output using a linear layer to obtain the final features of the scene points and state points. Additionally, we incorporate depthwise convolution (Chollet, 2017) for local feature extraction of the scene points. In Appendix A.2, we provide detailed algorithmic procedures of the IBS module.
The gating mechanism (Dauphin et al., 2017) introduces gated linear units, which allows the model to dynamically select activation paths based on the input. This selectivity improves the flexibility of the model, enabling model to better capture complex patterns. Thus, we design the GFFN to replace the standard FFN, shown in Figure 4 (c). For the ordered scene points, we also incorporate a depth-wise convolution to fully leverage the spatial structural information of the point cloud.
GFFN.
3.5
MODEL SETUPS
For fair comparison, build our DEST-based 3D detector based two baseline DETR-based models: GroupFree (Liu et al., 2021) 1) and VDETR (Shen et al., 2024), respectively. For GroupFree baseline, the input points P.,. Nx3 contain only position information. We replace all trans-

---

Published as a conference paper at ICLR 2025
former decoder layers with our proposed ISSM-based decoder layers while retaining the original spatial encodings and the original detection heads. For the training loss, we employ the same detection loss with GroupFree and introduce an additional objectness loss for the scene points. Specifically, we use an MLP head to determine whether a scene point is a foreground point and apply a binary focal loss to supervise the prediction results. For VDETR baseline, the input points P. E N include both position and RGB information. In the decoder, we also replace all trans- former decoder layers with our ISSM-based decoder layers while retaining the original detection heads. Regarding the spatial encoding, we generate them for the corresponding state points using the predicted 3D bounding boxes and for the scene points using their point positions. For the train- ing loss, we employ the same detection loss with VDETR and introduce the same objectness loss as above for the scene points. Due to space limitations, more details can be found in the Appendix.
4 EXPERIMENTAL RESULTS
4.1 DATASETS AND METRICS
Dataset. We evaluate our DEST-based detector on two challenging 3D indoor object detection datasets, including ScanNet V2 (Dai al., 2017) and SUN RGB-D (Song et al., 2015). ScanNer V2 dataset contains 1201 training samples and 312 validation samples, each annotated with per-point instance and semantic labels, as well as axis-aligned 3D bounding boxes across 18 categories. SUN RGB-I) dataset is a monocular dataset, containing over 10000 indoor RGB-D images annotated with per-point semantic labels and oriented 3D bounding boxes across 37 categories. We follow previous methods (Qi et al., 2019; Liu et al., 2021) to evaluate our approach on the 10 most common classes of objects. The training and validation splits contain 5285 and 5050 point clouds, respectively.
Evaluation Metries. Following the standard evaluation protocol (Qi et al., 2019), we evaluate ISSe- detector performance with the mean Average Precision (AP25 and AP30) under two different Intersections over Union (IoU) thresholds of 0.25 and 0.5. Since the input point clouds are obtained through random sampling, both the training and testing processes are stochastic. To ensure the reliability of the test results, we the training 5 times and independently test each trained model 5 times. We report both the highest performance and the average results under 5 x 5 trials.
4.2 COMPARISON WITH STATE-OF-THE-ARE METHODS
Different 3D detection models employ various techniques in terms of encoder architecture and bounding box parameterization, with some methods using point clouds with color information as model inputs. Therefore, is unfair to compare the performance of different methods directly. Among these 3D detection methods, GroupFree (Liu et al., 2021) is a pioneer in designing the DETR-based method for point clouds, demonstrating strong performance across multiple datasets. VDETR (Shen et al., 2024) builds upon 3DETR (Misra et al., 2021) by introducing the positional encoding in the decoder, achieving state-of-ther f-thhee-tt art performance. To demonstrate the effectiveness of our method, we implement the DEST- based detector on top of the two DETR-based methods.
As shown in Table 1, we compare our DEST-based detector with the previous 3D object detec- tion methods on the ScanNet V2 and SUN RGB-D datasets. The results indicate that method significantly outperforms the baseline methods on both datasets, whether measured by the highest performance or the average results over multiple trials. For the GroupFree baseline, our DEST-based detector demonstrates substantial performance improvements across different decoder scales. For example, on the ScanNetV2 dataset, our method achieves a 4.3 increase in AP30 based on GroupFree (S) and 5.3 increase based on GroupFree (L). For the VDETR bascline, our DEST-based detector achieves new state-of-the- art performance on both datasets. Specifically, our approach reaches 78.8 in AP25 and 67.9 in AP30 on the ScanNet V2 dataset, which is 1.0 and 1.9 better than the base- line model. Additionally, our method achieves 69.2 in AP25 and 52.2 in AP50 on the SUN RGB-D dataset with the gains of 1.2 and 1.1. In Appendix, we further present visual comparison of the prediction results with different baseline methods. The experimental results clearly demonstrate the effectiveness of our method. Our DEST-based decoder addresses the performance limitations caused by fixed scene point features during the query refinement process, resulting in a significant performance boost.
8

---

Published as a conference paper at ICLR 2025
Table 1: Comparison the ScanNet V2 and SUN RGB-D datasets. We report both the highest performance (H) and the average results (A) under multiple trials. 'RGB' indicates that the input point clouds of the methods include color information. GroupFree(S) denotes model with 6-layer decoder and 256 object candidates. GroupFree(L)) denotes a model with a 12-layer decoder and 512 object candidates. TTA is the test-time augmentation used in VDETR.
ScanNet V2(II)
ScanNet V2(A) SUN RGB-D(II) SUN RGB-D(A)
Method
RGB
AP2s
APsa
AP25
APse
AP2s
AP
AP25
APs0
VoteNet (Qi al., 2019)
X
62.9
39.9
57.7
HGNet (Chen al., 2020)
X
61.3
34.4
61.6
3D-MPA (Engelmann et al., 2020)
x
64.2
49.2
MLCVNet (Xie et al., 2020)
X
64.5
41.4
59.8
GSDN (Gwak et al., 2020)
X
62.8
34.8
H3DNet (Zhang et al., 2020)
X
64.4
43.4
60.1
39.0
BRNet (Cheng et al., 2021)
X
66.1
50.9
61.1
43.7
3DETR (Misra et al., 2021)
X 65.0
47.0
59.1
32.7
VENet (Xie al., 2021)
X
67.7
62.5
39.2
GroupFree(S
al., 2021)
X 67.3
48.9
66.3
48.5
63.0
45.2
62.6
44.4
(Lou et al., 2021)
X 69.1
52.8
68.6
51.8
RBGNet (Wang et al., 2022h)
X 70.6
55.2
69.9
54.7
64.1
47.2
63.6
46.3
HyperDet3D (Zheng al., 2022)
X 70.9
57.2
63.5
47.3
LeadNet (Wang ct al., 2023)
X 68.0
51.3
63.4
45.8
FCAF3D (Rukhovich et al., 2022)
71.5
57.3
70.7
56.0
64.2
48.9
63.8
48.2
TR3D (Rukhovich et al., 2023)
72.9
59.3
72.0
57.4
67.1
50.4
66.3
49.6
CAGroup3D (Wang al., 2022a)
75.1
61.3
74.5
60.3
66.8
50.2
66.4
49.5
VDETR (Shen al., 2024)
77.4
76.8
64.5
67.5
66.8
49.7
VDETR(TTA) (Shen et al., 2024)
77.8
66.0
77.0
65.3
68.0
51.1
67.5
50.0
GroupFree(S)Liu et al., 2021)
X
67.3
48.9
66.3
48.5
63.0
45.2
62.6
44.4
DEST()urs)
X
68.8
53.2(+4.3)
67. 07.9(+1.))
52.7
65.3
48
4.4.22
64.7
47.6
GroupFree(L)(Lin) et al., 2021)
X
69.1
52.8
68.6
51.8
DEST(ours)
X
71.3
58.
(45.))
.0.5
56.8...0)
VDETR (Shen et al., 2024) DEST(ours)
/
77.4
65.0
76.8
64.5
67.5
50.4
66.8
49.7
/
78.5 66.6..1.
77.8
66.2 mm
68.400.9
1.84 67. 7..4) 50.9
VDETR(TTA) (Shen et al., 2024)
/ 77.8
66.0
77.0
65.3
68.0
51.1
67.5
50.0
DEST(ours)
78.8
67.9
78.3
66.9
69.2.... 2::22
52.2
68.8
51.6.1..
4.3 ABLATION EXPERIMENTS
In this section, we verify the key design modules of our DEST-based detector. All ablation ex- periments are conducted on the ScanNet V2 dataset with the GroupFree(S) baseline. Following GroupFree (Liu et al., 2021), report the average performance of 25 trials by default.
Effect of the designed modules. As shown in Table 2, we incrementally add each designed mod- ule in SSM-based baseline. The SSM-hased baseline is also built GroupFree(SS), using the standard Mamba2 block and FFN as the decoder components. The SSM-hased baseline lacks the adaptability the state points and cannot handle unordered point cloud inputs, resulting signif- icant performance drop. Introducing the serialization and bidirectional scan strategies improves the model performance, but still lags behind GroupFree(S)) A substantial performance boost is observed when the proposed ISSM replaces the Mamba2 block. This boost is attributed to the adaptive ability of the ISSM, allowing state points select appropriate scene points for feature updating. Further incorporating an inter-state mechanism and gated linear unit achieves the best performance.
Effect of the ISSM. ISSM is the core the proposed DEST and responsible for the information exchange and updating between scene and state points. Compared with the selective SSM, ISSM extends the dimension A and introduces spatial correlation module and a delay kernel to assist scene points in generating the SSM parameters. To analyze the impact of parameter generation, evaluate different combinations of parameter generation methods. As shown the second and third of Table 3, using the spatial correlation module and delay kernel to generate SSM parameters leads to a significant performance improvement. The results indicate that allowing state points to focus on their relevant regions is crucial for designing SSMs for 3D object detection. Moreover,

---

Published as a conference paper at ICLR 2025
Table 2: Effect of the designed modules. We progressively add the proposed modules to the SSM-based baseline to verify the con- tribution of each module.
Table 3: Effect of the ISSM. Here, ((x) represents generating the SSM parameters using the scene points x. ((S) indicates the incorporation of spa- tial correlation S to generate the parameters. R(h) denotes the addition of a delay kernel for A.
Method
AP25
AP50
GroupFrec(SS
66.3
48.5
(xx)
Φ(S)
R(h)
AP25
AP50
Bascline
60.2
41.6
X
X
64.3
46.1
w/ serialization
62.8
43.5
7
/
67.2
51.3
w/ bidirectional scan
63.7
44.8
/
67.4
50.8
w/ ISSM
67.1
50.6
66.7
49.4
inter-state attention
67.6
51.9
67.9
52.7
w/ GFFN
67.9
52.7
Effect of the simultaneous updat- ing. "Fixed" denotes that the scene point fea- tures used in each decoder layer remain as those outputted by the encoder. "GroupFree" indi- cates that we introduced a GFFN in each de- coder layer to update the scene point features.
Table 4:
Table Comparison of the model param- eters and inference speed on ScanNet V2. "Param." " denotes the total parameters.
Method
Param. Latency
AP25
AP50
GroupFree(S)
13.8 M
21 ms
67.3
48.9
(OST((urs)
19.6 M
34 ms
68.8
53.2
Method
Fixed
AP2s
APsu
GroupFree(L)
28.2 M
68
69.1
52.8
GroupFree(S)
66.3
48.5
(OST()urs)
39.8 M
98 ms
71.3
58.1
GroupFree"
X
66.8
49.0
FCAF3D
67.2 M
138 ms
71.5
57.3
(EST(ours)
V
66.6
49,3
CAGroup3D
120.7 M
472 ms
75.1
61.3
(EST(ours)
x 67.9
52.7
VDETR
75.6 M
238 ms
77.8
66.0
+DEST(Our))
80.1 M
263 ms
78.8
67.9
comparing the fourth and fifth rows, we find that considering only spatial relationships also results in drop in performance. When the above modules are combined generate SSM parameters, the DEST-based detector achieves the best performance.
Effect of the scene point updating. To evaluate the impact of scene point feature updating, we conduct ablation experiments as shown in Table 4. In the second row, we add the proposed GFFN to GroupFree(S) to update the scene point features. However, due to its limited receptive field, only achieves a slight performance improvement. In the third row, we fix the scene point features in the proposed DEST-based detector, which results a significant performance decline. These results demonstrate the effectiveness of the simultaneous updating in DEST-based methods. In the DEST-based methods, scene points can capture global contextual information through state points for self- self-updating. thereby providing more effective information for subsequent decoder layers.
4.4 COMPARISON OF PARAMETERS AND INFERENCE SPEED
The model complexity of both
and DEST-based methods is determined by the number of scene points and query points. To ensure fair comparison, we compare the model parameters and inference speed with different DETR- baseline methods. All experiments are conducted on Tesla V100 GPU. As shown in Table 5, our method demonstrates comparable model param- eter efficiency and inference speed while enabling simultaneous updates of scene points and state points. Although there is slight increase in parameter count and computational cost compared to baseline methods, this overhead is acceptable given the substantial performance gains.
5 CONCLUSION
In this paper, we identify crucial issue in DETR-based decoders: the fixed scene point features lead to suboptimal refinement of query points in the later layers. To address this issue, we introduce novel DEST-based method that simultaneously updates scene and query point features. The key contribution lies in designing the ISSM, which models query points as system states and scene points system inputs, allowing simultaneous updates with linear complexity. Our DEST-based method demonstrates significant advantages through comparison experiments with two DETR-base based methods, and comprehensive ablation studies validate the effectiveness of each designed module.
10

---

Published as a conference paper at ICLR 2025
6 ACKNOWLEDGEMENT
This work was supported by Hainan Province Science and Technology Special Fund (Grant No. ATTC-20230100044, National Natural Science Foundation of China (No. 12150007, No. 62071122), Basic Strengthening Program Laboratory Fund (No. NKLDSE20234009) and Youth Innovation Promotion Association of CAS.
REFERENCES
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020. 2, 7.7
Jintai Chen, Biwen Lei, Qingyu Song. Haochao Ying, Danny Chen, and Jian Wu. A hierarchical graph network for 3d object detection on point clouds. In Proceedings of the IEEE/CVF confer- ence on computer vision and puttern recognition, pp. 392-401, 2020.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In Proceedings of the IEEE conference Computer Vision and Pattern Recognition, pp. 1907-1915, 2017. 3
Bowen Cheng, Lu Sheng, Shaoshuai Shi, Ming Yang, and Dong Xu. Back-tracing representative points for voting- -aaedd 3d object detection in point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 3,9
François Chollet. Xeeption: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1251-1258, 2017.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5828-5839, 2017. 3.
Xiyang Dai, Yinpeng Chen, Jianwei Yang. Pengchuan Zhang, Lu Yuan, and Lei Zhang. Dynamic detr: End-to-ent end object detection with dynamic attention. In Proceedings of the IEEE/CVF inter- national conference on computer vision, pp. 2988-29229999 2997,2997, 2021.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality. In Forty-first International Conference Machine Learning, 2024.
2, 5, 6
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory- efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, 15316944-16339, 2022. 20
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks. In International conference on machine learning, pp. 933-941. PMLR, 2017. 7
Francis Engelmann, Martin Bokeloh, Alireza Fathi, Bastian Leibe, and Matthias Nießner. 3d- mpa: Multi- Aulti-proposal aggregation for 3d semantic instance segmentation. In Proceedings of the IEFE/VVF conference on computer vision and pattern recognition, pp. 9031-9040, 2020. 9
R Girshick. Fast r-cnn. arXiv preprint arXiv:1504.0880 1504.08838, 2015. 4
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00075,, 2023. 2, 3, 4, 5, 20
Albert Gu, Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint wrXiv:2111.003990 202 2,
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Com- bining recurrent, convolutional, and continuous-time models with linear state space layers. Ad- vances in neural information processing systems, 34:572-585, 2021b.

---

Published as a conference paper at ICLR 2025
Jun Young Gwak, Christopher Choy, and Silvio Savarese. Generative sparse detection networks for 3d single-shot object detection. In Computer Vision- ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pp. 297-313. Springer, 2020. 1, 2, 4,
David Hilbert and David Hilbert. Neubegründung der mathematik. ersle mitteilung. Springer, 1935.
3,7
Rudolph Emil Kalman. A new approach to linear filtering and prediction problems. Journal of Basic Engineering, 1960.
Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Point- pillars: Fast encoders for object detection from point clouds. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12697-12705, 2019. 1,3
Dingkang Liang Xin Zhou, Xinyu Wang, Xingkui Zhu, Wei Xu, Zhikang Zou, Xiaoqing Ye, and Xiang Bai. Pointmamba: A simple state space model for point cloud analysis. arXiv preprint arXiv:2402.10773 10739, 2024. 4
T Lin. Focal loss for dense object detection. arXiv preprint arXiv: 020080 2017. 4
Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang. Dab-detr: Dynamic anchor boxes better queries for detr. arXiv preprint arXiv:2201. 12329, 2022. 4
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint arXiv:2401.
2024. 4
Ze Liu, Zheng Zhang, Yue Cao, Han Hu, and Xin Tong. Group-free 3d object detection via trans- formers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 2949-2958, 2021. 1, 2, 4, 7, 8, 9, 17
I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017. 17
Ishan Misra, Rohit Girdhar, and Armand Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF international conference computer vision, pp- 2906-2917, 2021. 1, 7,
Alexander Neubeck and Luc Van Gool. Efficient non-maximum suppression. In 18th international conference on pattern recognition (ICPR 066), volume 3, pp. 850-855. IERE, 2006.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 652-660, 2017a.
Charles R Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. In Proceedings of the IEEF/CV International Conference on Computer Vision, pp. 9277-9286, 2019. 1, 8,9
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical fea- ture learning point sets in metric space. Advances in neural information processing systems, 30, 2017b. 17
Shaoqing Ren. Faster r-cnn: Towards real-time object detection with region proposal networks. arXiv preprini orXiv: 156.0055 2015.
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Feaf3d: Fully convolutional anchor-free 3d object detection. In European Conference Computer Vision, pp. 477-493. Springer, 2022.
1, 2. 4,
Danila Rukhovich, Anna Vorontsova, and Anton Konushin. Tr3d: Towards real-time indoor 3d
object detection. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 281-
285. IEFF, 2023. 9
12

---

Published as a conference paper at ICLR 2025
Yichao Shen, Zigang Geng, Yuhui Yuan, Yutong Lin, Ze Liu, Chunyu Wang, Han Hu, Nanning Zheng, and Baining Guo. V-DETR: DETR with vertex relative position encoding for 3d object detection. In The Twelfth International Conference Learning Representations, 2024. URL https: /eppnreviiiiee ew net / forum? inruumidd=iae G=NDkpxG44K 1,2, 4, 6, 7, 8, 9, 17
Shaoshuai Shi, Xiaogang Wang. and Hongsheng Li. Pointrenn: 3d object proposal generation and detection from point cloud. Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 770-779, 2019.
Shaoshuai Shi, Chaoxu Guo, Li Jiang, Zhe Wang, Jianping Shi, Xiaogang Wang. and Hongsheng
Li. Pv-renn: Point-voxel feature set abstraction for 3d object detection. In Proceedings of the IEEE/CVE conference on computer vision and patlern recognition, pp. 10529- 10538, 2020a.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts:
3d object detection from point cloud with part-aware and part-aggregation network. IEEE trans- actions on pattern analysis and machine intelligence, 43(8):2647-2664, 2020b.
Shuran Song, Samuel P Lichtenberg, and Jianxiong Xiao. Sun rgb-d: A rgb-d scene understand- ing benchmark suite. In Proceedings of the IEEE conference on computer vision and pattern recogmition, pp. 567-576, 2015. 3,8
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
2,4
Chuxin Wang, Jiacheng Deng, Jianfeng He, Tianzhu Zhang, Zhe Zhang, and Yongdong Zhang. Long-short range adaptive transformer with dynamic sampling for 3d object detection. IEEE Transactions on Circuits and Systems for Video Technology, 33(12):7616-762, 2023. 1, 2, 4, 6,
Haiyang Wang, Lihe Ding, Shaocong Dong, Shaoshuai Shi, Aoxue Li, Jianan Li, Zhenguo Li, and Liwei Wang. Cagroup3d: Class-aware grouping for 3d object detection point clouds. Advances in Neural Information Processing Systems, 0222975-299888 2022a. 1,2,4,
Haiyang Wang, Shaoshuai Shi, Ze Yang, Rongyao Fang, Qi Qian, Hongsheng Li, Bernt Schiele, and Liwei Wang. Rbgnet: Ray-based grouping for 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1110-1119, 2022b. 9
Xiaoyang Wu, Li Jiang, Peng-Shuai Wang, Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang, Tong He, and Hengshuang Zhao. Point transformer v3: Simpler faster stronger. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 4840 4851, 2024.
Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Yiming Zhang, Kai Xu, and Jun Wang. Mlevnet: Multi-level context votenet for 3d object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10447-10456, 2020. 1,3,9
Qian Xie, Yu-Kun Lai, Jing Wu, Zhoutao Wang, Dening Lu, Mingqiang Wei, and Jun Wang. Venet: Voting enhancement network for 3d object detection. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 3712-3721, 2021. 9
Tianwei Yin, Xingyi Zhou, and Philipp Krahenbuhl. Center-based 3d object detection and tracking. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp- 11784-1 1793, 2021.1
Tao Zhang, Xiangtai Li, Haobo Yuan, Shunping Ji, and Shuicheng Yan. Point could mamba: Point cloud learning via state space model. arXiv preprint arXiv:2403.00766. 2024. 4
Zaiwei Zhang. Bo Sun, Haitao Yang, and Qixing Huang. H3dnet: 3d object detection using hybrid geometric primitives. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XII 16, pp. 311-329, Springer, 2020. 1, 4,
Yu Zheng, Yueqi Duan, Jiwen Lu, Jie Zhou, and Qi Tian. Hyperdet3d: Learning cecne-conditioned 3d object detector. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5585-5594 2022.
13

---

Published as a conference paper at ICLR 2025
Yin Zhou and Oncel Tuzel. Voxelnet: End-to-end learning for point cloud based 3d object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4490- 4499, 2018. 3
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vi- sion mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024. 4,
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. RELATIONSHIP BETWEEN QUERY AND STATE
In this section, we further analyze the relationship between queries in the attention mechanism and the states in the State Space Model (SSM). For 3D object detection, the transformer decoder employs an attention mechanism facilitate the interaction between object candidate point features ho E I k xC and scene point features E R RMxC. Specifically, the object candidate point features ho serve as the query Qo. while the scene point features x act as the key and value V. The attention mechanism is then used to update ho:
En sim(@o K3)V, 2i ΣØ= sim (Qi: K2)
(9)
where sim(,) ) denotes the feature similarity calculation, Q'n represents the i-th query updated using the first m scene point features, Q4 denotes the initial i-th query, and K and V correspond to the j-th key and value, respectively. The above equation can be reformulated as follows:
j=
m-1 sim (V, K,) + [sim( Qi Vm)
Σ sim(Qi, Kj)
sim (Qm; Km) sim(Q'. K3)V;
Vn
j= sim K2)
Σド= sim (Q): Kÿ)
(10)
sim (bg: Kz)
Σ
m-1
j=1 sQm K2)V;
sim(Q:: Km)
j=1
Vm
sim simm(0) Kj)
m- sim (Q: Kj)
2 sim Qú: Kj)
sim Qú, Kj)
Qm sim (Q): Km) Ey sim Kj)
Vn.
(im K,)
Based on the above equation, the query feature Qi updated with the first m points can be regarded as linear combination of the query feature O', Q'm11 updated with the first m points and the feature Vn of the m-th scene point. Furthermore, we rewrite the equation 10 in the following form:
2 = A O' + Bi, Vm, A
m- sim (Qm K2)
-==
B
(Qm Km)) -/=1 sim Q K2)
(II)
Ej= sim Qi: Kj)
In the discrete SSM, we observe that the interaction process between system states and scene points shares a similar form with the equation 11:
hin = Am km + B Tm. Tm
(12)
where hi denotes the i-th system state updated using the first m scene points, 2m denotes the m- input of scene point features, while A", and B'n, are the parameters of the SSM. Therefore, we consider the attention mechanism in the transformer decoder to be type of SSM, which designs the query update strategy through the feature similarity sim(,). The previous SSMs only utilize Fm
generate the SSM parameters A", and B',, making them unsuitable for modeling query points in 3D object detection tasks. However, the update strategy based on feature similarity sim(,) ) requires significant computational overhead. Consequently, in the proposed ISSM, we utilize spatial cor- relation module and scene point features to generate SSM parameters as equation 7 and equation 8, reducing the model overhead without compromising performance. Based on the above analysis, we believe that the system states in our ISSM can effectively serve the role of the queries with the appropriate parameters 4 and B m
A.2 ISSM-BASED BIDIRECTIONAL SCAN MODULE
In the ISSM- decoder, the core design is the ISSM-based Bidirectional Scan (IBS) module. The IBS module integrates an interactive state space model with bidirectional sequential modeling tailored for point clouds. To illustrate the process of the IBS module more clearly, we provide the
15

---

Published as a conference paper at ICLR 2025
Algorithm ISSM-based Bidirectional Scan Module Process
Require: scene points x: (B, M. C), initial state points ho: (B, K, C)
Ensure: updated scene points y: (B, M, C), final state points hMr: (B, K, C)
1* normalize the input scene points x */
2: x': (B, M. C) < Norm(z).. Rg: (B, K. C) + Norm(ho)
3: #: (B, M. E) + Linear" (a'), , z: (B, M. E) + Linear (x/)
4: he: (B, K. E) + Linear" (h%)
5: /M process with different direction */
6: for oin oin,forward, backward) do
7: Go: (B, M. E) < SiL (S..U/DW.CCovvdd((e))
8: / S is the spatial correlation of scene points and state points: (B, M. K, D) */
9: Bo: (B, M, K) < Broadcast (Tinear (T
(a.)) ) Linear B. (S)
C, ($.)) Lineerr ((())
C.: (B, M, K) Broadcast (Lineari
11: 1* softplus ensures positive A,, detog is the proposed delay kernel: (B, M. K) #/
12: Ao: (B, M. K, E) < log([1 + exp (xp(raaaaasss (Linear ((t)) + Linear E (8))) @ delay
13: 1# Parameter is learnable parameter: (M, E) */
14: Ao: (B, M, K, E) <- A @ Parameter
Bo: (B, M. K. B) < A. B.
No: (B, E), hg: (B, E) (S,(A,, Bo, Co) ho)
17: end for
1* gated linear unit and residual connection
19: D Morwardd (B, M, E) + Vtorward O Si)U(2)
20: #backward' (B, M, E) + Moackward O (z)
21: y: (B, M, C) f- Linear (Voorward + Vnackwardd ) +
bacckward) ) + ho hM: (B, K, C) + Linear" ( forwaadd + NN
23: return y and hM
detailed operations of the IBS module in Algorithm 1. The input scene points and initial state points ho are first normalized by the normalization layer. Next, we linearly project the normalized features x' to è and 2, and the Nn to lig Before we process the & from the forward and backward directions, we calculate the spatial correlation S of scene points and state points as equation 6 and the explicit delay kernel A deley equation 8. For each direction, we first apply the depthwise convolution to the and get the &.. The SSM parameters (B,, Co. Ao) are generated based on the t.. S and A delay. Then use the A. transform the A... B,, Finally, we get the updated scene points do and updated state points h3, through the SSM. For the scene points Moowardd Pbackward we gated them by the 2 and add them together. Similarly, for the state points heforward Je add them directly. The updated scene and state points are then connected to x and hg using residual connections, respectively, to produce the final outputs y and has.
hackwaard we
A.3 POINT CLOUD SERIALIZATION
To enable unordered point clouds to be inputted into the SSM in an ordered manner, we use Hilbert space-filling curves to sort the scene points. With the different priority of the x, y, and axes, we can get six different space-filling curves, which we denote as "xyz", "xzy", "yxz", "yzx", "zxy", and "zyx". To better illustrate the point cloud serialization process, we visualize the 2D Hilbert space- filling curves and the serialization results of the 2D point set in Figure 5. Specifically, we divide the space into uniformly sized grids and then sort each grid based on the space-filling curve. Finally, we sort the point set according to the indices of the grids where the points are located. As shown in Fig- ure 5, different space- filling curves yield varying ordered results for the point sets. These different serialization results represent observations of point clouds from different perspectives. Therefore, we utilize six space-filling curves in 3D space, employing different serialization methods at different decoder layers.
16

---

Published as a conference paper at ICLR 2025
Hilbert Curve (xy)
Ordered Points (xy)
Hilbert Curve (yx)
Ordered Points (yx)
Figure 5: Visualization of Hilbert space-filling curves and Hilbert-based point cloud serializa- tion. To facilitate analysis, we visualize all images in 2D space. By swapping the priority of the coordinate axes, we can obtain different Hilbert space-filling curves. Different spatial filling curves can yield distinct serialization results for the same point set.
A.4 MORE IMPLEMENTATION DETAILS
A.4.1 MODEL SETUP DETAILS
GroupFree baseline. Given the input points P... E Nx3 without RGB information, we employ PointNet++ (Qi al., 2017b) the point cloud encoder for fair comparison. In addition, utilize the same initial object candidate sampling module as employed in GroupFree (Liu et al., 2021). However, unlike GroupFree, where the object candidate points used the initial query points, they are treated the imitial state points in our ISSM-based model. In the decoder, we replace all transformer decoder layers with our proposed ISSM-basee decoder layers while retaining the original spatial encoding and the original detection heads.
VDETR baseline. Given the input points P.n E N6 with RGB information, we employ the same 3D sparse convolution network of VDETR (Shen et al., 2024) as the point cloud encoder for a fair comparison. Additionally, we utilize the same initial object query sampling module as employed in VDETR (Shen et al., 2024) to sample the initial state points. In the decoder, also replace all transformer decoder layers with our proposed ISSM-based decoder layers while retaining the original detection heads. Regarding the spatial encoding, generate them for the corresponding state points using the predicted 3D bounding boxes and for the scene points using their point positions.
A.4.2 TRAINING DETAILS
Training with GroupFree baseline.
50k points input. In the training phase, we adopt the same data augmentation as in GroupFree (Liu et al., 2021). including random flip, random rotation along the z-axis [-50 59), and random scaling [0.9, 1.1]. The encoder consists of four set abstraction layers and two feature propagation layers. After feature extraction by the encoder, a total of 1024 points are output as scene points. The detector is trained from scratch using the Adam W (Loshchilov, 2017) optimizer (.1 = 0.9, B2 0.999) for 400 epochs. The weight decay is set to 5e-4. The initial learning rate is 6e-3, which decays by a factor of at the 280th and 340th epochs. The learning rate of the decoder is set as TO of that in the encoder.
For the ScanNet V2 dataset,
For the SUN RGB-D dataset, we use 20k points as input. The encoder architecture and data augmen- tation the same those used for ScanNet V2. Following GroupFree, include an additional orientation prediction branch in all decoder layers. During training. the detector is trained from scratch using the AdamW optimizer (81 0.9, 32 0.999) with 600 epochs. The weight decay set to le-7. The initial learning rate 4e-3, which decays by factor of 10 at the 420th epoch, the 480th epoch, and the 540th epoch. The learning rate of the decoder is set I of that in the encoder.
Training with VDETR baseline.
17

---

Published as a conference paper at ICLR 2025
Table Effect of the serialization strategy. "xyz"represeni" the axis priority order of the Hilbert space-filling curve.
Strategy
AP25
("xyz")
66.6
51.1
("xyz" "xzy") x
67.1
51.7
("xyz", "yzx", "zxy") x 2
67.4
52.1
("xyz", "yxz", "yzx", "zxy", " "zyx"}
67.9
52.7
Table 7: Effect of the bidirectional scan. "Uni-Scan" denotes using the unidirectional scan while
"Bi-Scan" denotes using the bidirectional scan. "Channel Flip" refers to reversing along the feature channels.
Strategy
AP25
AP50
Uni-Scan
67.1
50.9
Bi-Scan
67.9
52.7
Bi-Scan w/ Channel Flip
67.7
52.4
For the ScanNet V2 dataset, we use 100k points as input. In the training phase, we adopt the data augmentation, including random cropping, random sampling, random flipping. random rota- tion along the z-axis [-5", ,5), random translation [-0.4, 0.4]. and random scaling [0.6, 1.4]. After feature extraction by the encoder, total of 4096 points are output as scene points. The detector trained from scratch using the AdamW optimizer (3, = 0.9, Bg = 0.999) for 540 epochs. The weight decay is set to 0.1. The initial learning rate is 7c-4, which is warmed up for 9 epochs and then is dropped to 1e-6 using the cosine schedule during the entire training process.
For the SUN RGB-D dataset, we 100k points input. The encoder architecture, data augmen- tation and training settings are the same as those used for ScanNet V2. In the decoder, we include an additional orientation prediction branch in all decoder layers.
A.5 MORE ABLATION EXPERIMENTS
In this section, we provide additional ablation experiments focusing on several modules designed in model: the serialization strategy, bidirectional scanning strategy, gated fced-forward network, and depth-wise convolution. All ablation experiments are conducted on the ScanNet V2 dataset with the GroupFree(S)) baseline, and report the average performance of 25 trials by default.
Effect of the serialization strategy. We conduct ablation experiments on the serialization strategy used in the based decoder. As shown in Table 6, employing single space-fillingg curve for serialization leads to a decrease in model performance. This is because single serialization method ensures that adjacent points in the sequence are spatially adjacent, but it cannot guarantee that all spatially adjacent points remain adjacent in the sequence. Observing from multiple perspectives essential for better modeling the local relationships high nigh-diimnsiooal data. The experimental results also demonstrate that using different types of space- filling curves at different layers achieves the best performance.
Effect of the bidirectional scan. To validate the effect of the bidirectional scanning on the proposed ISSse method, we conduct ablation experiments as shown Table It is evident that bidi- rectional scanning significantly improves model performance compared to unidirectional scanning. In unidirectional scanning, scene points later in the sequence can access information from earlier points, but earlier points cannot access information from later ones. The bidirectional scanning strategy enables bidirectional interaction between scene points, leading to enhanced performance. Additionally, consider using channel flip strategy, which enhances inter-channel correlations by reversing feature channels. However, the channel flip strategy unsuitable for our proposed method, resulting in slight performance decrease.
Ablation on the model size. For 3D object detection models, the feature dimension, decoder depth, and the number of initial object candidates all significantly impact model performance. To design
18

---

Published as a conference paper at ICLR 2025
Table Evaluation on different model size. "Encoder width" represents the feature dimension of the encoder output, "# of layers" denotes the decoder depth, and "# of object candidates" denotes the number of system states in the
decoder.
Encoder width
# of layers
# of object candidates
AP25
AP50
288
2
256
66.2
45.9
288
4
256
67.1
49.3
288
6
256
67.9
52.7
288
8
256
68.3
54.1
288
12
256
68.6
54.8
288
256
67.9
$2.7
288
6
512
68.7
53.6
288
6
1024
68.8
53.3
288
6
256
67.9
52.7
576
6
256
68.4
54.2
576
12
256
69.8
56.5
576
12
512
70.5
56.8
Table 9: Effect of the GFEN "GLU on x" denotes using the gated linear unit on scene points x while "GLU on h" denotes using the gated linear unit state points h.
GLU on æ
GLU on
AP25
AP30
67.6
51.9
/
67.8
52.5
X
67.6
52.3
V
67.9
52.7
a model with better performance under a smaller model overhead, we conduct ablation experiments on model size, As shown in Table 8, we experiment with different combinations of encoder width, decoder depth, and the number of object candidates. First, increasing the number of decoder layers leads continuous and significant performance improvement, particularly in APs0- This result demonstrates that the ISSM-based decoder effectively updates the features of scene points, aiding the query points in more accurately locating object positions. Second, moderately increasing the number of initial object candidates also enhances model performance. However, when the number becomes too large, numerous background points are introduced as negative samples, reducing de- tection accuracy. Lastly, increasing the encoder feature dimension provides the model with richer information, thereby improving performance. By selecting a larger encoder width, a deeper decoder, and appropriate number of query points, our model achieves optimal performance.
Effect of the GFFN. Introducing gated linear unit (GLU) in the feed-forward network aims to enhance feature modeling for both scene points and state points, allowing the model selectively pass or suppress certain features. This dynamic gating mechanism strengthens the ability to capture complex patterns, improves training efficiency, and makes more effective in handling long-range dependencies. As shown in Table 9, including the gated linear unit positively impacts model perfor- mance. Applying the GLU on both scene and state points leads the best detection performance.
Effect of the local feature aggregation. In the ISSM-based decoder layer, we employ two depth- wise convolutions in ISSM and GFFN for local feature aggregation of scene points. To analyze the impact of this local feature aggregation on model performance, we conduct the ablation experiments shown in Table 10. Eliminating the depth-wise convolutions from the ISSM-based decoder layer results in significant decrease model performance. Furthermore, adding depth- convolutions to the ISSM is more effective in improving detection accuracy than adding them to the GFFN. We also analyze the impact of the kernel size in depth-wise wiith-wise convolutions on model performance. We find that when the kernel size reaches 8, the model performance does not continue to increase with larger kemee sizes. Therefore, in our ISSM-based decoder, we select a kereel size of to balance performance and computational cost.
19

---

Published as a conference paper at ICLR 2025
Table 10: Effect of the local feature aggregation. "Kernel" denotes the kernel size in depth convolution.
in ISSM
in GFEN
Kernel
AP25
AP50
X
66.8
51.8
X
8
67.6
52.4
/
8
67.2
52.3
/
/
8
67.9
52.7
/
4
67.4
52.2
/
12
68.1
52.6
/
16
67.8
52.6
Table 11: Comparison of different scene point feature update methods.
Method
AP25
AP30
GroupFree
66.3
48.5
w standard Transformer
66.7
49.1
w FlashAttention
66.5
49.2
selective SSM
67.0
50.1
DEST (Ours)
67.9
52.7
A.6 VISUAL COMPARISON
In this section, we present a visual comparison of the prediction results of our method and DETR- based methods. To ensure a fair comparison, we apply the same post-processing steps to the de- tection results. First, we filter out low low-quality predictions using a class confidence threshold of 0.3, followed by class-specific Mon-Maximum Suppression with an IoU threshold 0.5 remove redundant bounding boxes. As shown in Figure 6, Figure 7, and Figure 8, we first compare the pre- diction results of our method with three baseline methods on the ScanNet V2 dataset. Our method successfully detects most target objects across various scenes, with the predicted bounding boxes being more closely aligned with the ground truth labels. Subsequently, we present a visual com- parison of the prediction results the SUN RGB-D dataset. Since VDETR does not provide an official implementation for the SUN RGB-D dataset, we only include a visual comparison with the GroupFree baseline as shown in Figure 9. Since VDETR does not provide official implementa- tion for the SUN RGB-D dataset, we only include visual comparison with the GroupFree baseline. Unlike the ScanNet dataset, the SUN RGB-D dataset is generated from single images, which results in point cloud data with more pronounced occlusions and uneven density, making the detection task more challenging. Nevertheless, our method still achieves visually satisfactory detection results.
A.7 DISCUSSIONS ABOUT THE CHOICE FOR SSM
In this subsection, we discuss the choice of SSM for the DEST framework. To address the per- formance bottleneck caused by fixed scene point features transformer decoders, a straightforward approach is to introduce a module in the decoder to update scene point features. To validate the effec- tiveness of SSM updating scene point features, we provide numerical comparisons with standard self-attention. FlashAttention (Dao et al., 2022), and selective SSM (Gu & Dao, 2023). As shown in methods improve the detection accuracy of the baseline model but still fall
Table 11, attention- short compared to methods employing SSMs. This limitation arises because global attention is less effective at handling local relationships in point clouds and may introduce noise that disrupts scene point feature updates. In contrast, selective SSM uses ID convolution model local relationships and incorporates a hidden state selection mechanism to focus on extracting relevant features. There- fore. we adopt SSM to address the issue of fixed scene point features in the decoder. In our proposed DEST method, query points representing foreground objects serve as state points, providing richer and more task-specific foreground information enhance point feature updates.
20

---

Published as a conference paper at ICLR 2025
Ground-truth
GroupFree (S)
DEST (ours)
Figure 6: Visual comparison with the GroupFree (S) baseline on ScanNet V2 dataset. The ground truth is displayed in the first column, the baseline detection results in the second column, and our detection results in the third column.
A.8 DISCUSSIONS ABOUT THE LIMITATIONS AND FUTURE RESEARCH
In this section, discuss the limitations of this work and potential future research. We present novel framework, DEST, for 3D object detection, which utilizes the proposed ISSM to achieve joint updates of scene points and query points during the decoding process. Although method over- comes the performance bottleneck caused by fixed scene features in transformer decoders and signif- icantly improves the performance of DETR-based methods, there still instances of missed object
21

---

Published as a conference paper at ICLR 2025
Ground-truth
GroupFree (L)
DEST (ours)
Figure 7: Visual comparison with the GroupFree (L) baseline on ScanNet V2 dataset. The ground truth is displayed in the first column, the baseline detection results in the second column, and our detection results in the third column.
detections. Through layer-by-layer analysis the model, we find that the root cause of these missed detections is the absence of initial object candidate points. This study primarily focuses on the de- coder design and does not delve into designing an initial state point sampling method. Therefore, effective state point sampling strategy to reduce the number of missed detections a promising direction for future research. Beyond improving the proposed DEST framework, ex- ploring its effectiveness in other tasks, such as 3D point cloud segmentation, tracking, and even 2D)
designing
22

---

Published as a conference paper at ICLR 2025
Ground-truth
VDETR
DEST (ours)
Figure 8: Visual comparison with the VDETR baseline on ScanNet V2 dataset. The ground truth displayed in the first column, the baseline detection results in the second column, and detection results in the third column.
vision tasks, is an interesting research direction. In the future, will further explore the potential of the DEST framework achieving unified point cloud perception tasks.
23

---

Published as a conference paper at ICLR 2025
Ground-truth
GroupFree
DEST (ours)
Figure 9: Visual comparison with the GroupFree (S) baseline on SUN RGB-D dataset. The ground truth is displayed in the first column, the baseline detection results in the second column, and our detection results in the third column.
24


Reviews:
[{"review_id": "saTfD2ahcW", "reviewer": "Reviewer_Ho6d", "summary": "The paper presents a novel approach to 3D object detection, particularly for indoor environments, by integrating State Space Models (SSMs) with Transformer architectures. The key contributions of the paper are:\n\n1. **Introduction of DEST (State Space Model-based 3D Object Detection)**: The paper proposes a new paradigm called DEST, which addresses the limitations of fixed scene point features during the query refinement process in existing methods. This is the first method to model queries as system states within an SSM framework.\n\n2. **Design of an Interactive State Space Model (ISSM)**: The authors develop an ISSM that effectively functions as queries in complex 3D indoor detection tasks. The ISSM-based decoder is tailored to the characteristics of 3D point clouds, fully harnessing the potential of the ISSM for 3D object detection.\n\n3. **Enhanced Performance**: Extensive experimental results demonstrate that the proposed SSM-based 3D object detection method consistently enhances the performance of baseline detectors on two challenging indoor datasets, ScanNet V2 and SUN RGB-D. Comprehensive ablation studies validate the effectiveness of each designed component.\n\n4. **ISSM-based Decoder**: The paper introduces an ISSM-based decoder that replaces the transformer decoder in DETR-based methods, addressing the performance limitations caused by fixed scene point features. This decoder simultaneously updates both scene point and query point features.\n\n5. **Comparison with State-of-the-Art Methods**: The DEST-based detector significantly outperforms previous 3D object detection methods on the ScanNet V2 and SUN RGB-D datasets, demonstrating substantial performance improvements.\n\nOverall, the paper introduces a new state-space model-based approach to 3D object detection that overcomes the limitations of existing methods, particularly in handling complex indoor environments, and achieves state-of-the-art performance.", "questions": "Please refer to section Weaknesses.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. Writing: The writing is clear and effectively conveys the content of the paper, making it easy to understand.\n\n2. Innovative SSM Integration: The paper creatively combines State Space Models with transformers, offering a unique approach to 3D object detection that reduces complexity while enhancing performance.\n\n3. Performance Improvement: The model achieves substantial performance gains on benchmarks, specifically outperforming existing models on AP metrics for key datasets (ScanNet V2 and SUN RGB-D).\n\n4. Detailed Experimentation: Extensive experiments, including ablation studies, provide strong evidence for the model’s effectiveness and validate each component’s contribution.", "weaknesses": "1. According to Table 5, the proposed method results in a noticeable increase in latency.\n2. What is the motivation for introducing SSM in this paper? Why is SSM suitable for 3D detection tasks? I believe this is a critical point that requires detailed explanation from the authors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mJAdFhvN3h", "reviewer": "Reviewer_uYMo", "summary": "The paper address an issue that the ﬁxed scene point features lead to suboptimal reﬁnement of query points in the later layers of transformer and then introduce a novel DEST-based method for 3D indoor object detection. The key design is ISSM, which models query points as system states and scene points as system inputs, allowing simultaneous updates with linear complexity. Experiments in ScanNetv2 and SUN RGB-D show its effectiveness.", "questions": "1.\tAre there other methods to solve “fixed scene point” from previous works? \n2.\tIs is helpful to simultaneously update both scene point and query point features for most model, or just for DEST?\n3.\tState Space Models (SSM) has an efﬁcient context modeling ability with linear complexity through iterative interactions between system states and inputs. Is it possible to combine SSM and Transformers in 3D object detection?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tMeaningful topic：The paper starts from this problem that scene point features in the transformer decoder remain ﬁxed, leading to minimal contributions from later decoder layers, thereby limiting performance improvement for DETR-based methods.\n2.\tConvincing experiments: The paper tries its method on different baseline and datasets to verify its effectiveness. Ablation studies are also comprehensive.\n3.\tCreative design of method: The paper designs a novel state-dependent SSM parameterization method that enables system states to effectively serve as queries in 3D indoor detection tasks.", "weaknesses": "1.\tFor VDETR, improvement is not big enough.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rfO7GzOYDq", "reviewer": "Reviewer_Y9XD", "summary": "In this paper, the authors observe that the performance of DETR-based 3D object detection methods increases slowly in the last few Transformer decoder blocks. Based on the observation, this paper introduces state space models into 3D object detection and proposes an interactive state space model to update both 3D scene features and queries. Specifically, the authors design a state-dependent SSM parameterization to make each query interact with its respective scene features. Besides, the authors propose four key designs in an SSM-based decoder (*i.e.*, Hilbert-based serialization, inter-state attention, IBS module, and gated FFN). Experimental results show that the proposed method can be combined with multiple 3D detectors and further improve the detection accuracy on both Scannet V2 and SUN RGB-D datasets.", "questions": "Lots of typos should be modified.\n- Line 28, imporoves -> improves\n- line 107, perpormance -> performance\n\nAlso, in line 84, 0.64 -> 6.4 according to figure 1.\n\nThe authors should carefully examine the article.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors observe that the performance improves slowly in the last few blocks in the DETR-based 3D detector which is interesting.\n- Experimental results show that the proposed methods can be combined with multiple DETR-based detectors and achieve SOTA performance.", "weaknesses": "- Questions about the proposed method\n  - The first question is about computational complexity, in the proposed **Extension of $\\Delta$**, the authors expand $\\Delta$ $K$ times which will largely increase the computational complexity. The author should provide some results about the computational cost such as FLOPs.\n  - Secondly, the design of **Spatial Correlation** Module, especially the relative offsets is similar to the 3D Vertex Relative Position Encoding used in VDETR.\n  - Further, the ISSM-based decoder stacks lots of elements with few explanations. the authors should explain the importance of each component in the method section. Besides, more illustrations should be provided, for example, the structure of the inter-state attention module. Designs such as Hilbert serialization have been used in Point Transformer v3. The bidirectional scanning mechanism is also common in vision Mamba. The authors need to explain their rationale to avoid being mistaken for a simple stacking of modules.\n\n- Questions about the experiments\n  - In table 5, do the results in Param. represent the total parameters or the added parameters when using DEST?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ALbSfoh9xd", "reviewer": "Reviewer_bWrW", "summary": "Based on the motivation that sharing same scene representations in different decoder layers limits the overall performance of DETR-based 3D detectors, the paper proposes a mamba-based decoder architecture for indoor 3D detection. To fit the mamba design to 3D detection, the paper treats queries as states and scene representations as inputs. Experiments show that the proposed method improves existing baseline models.", "questions": "1. As shown in Fig. 4 (a), the difference between the proposed method mainly replaces the cross-attention layers in the standard transformer decoder architecture with the proposed ISSM blocks. Is it possible to also replace the inter-state attention module with bi-directional mamba?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The idea of updating scene features and query point features simultaneously is well-motivated.\n\n2. With sufficient and extensive evaluations and ablations, the authors validate that DEST surpasses previous baseline methods with reasonable computational cost.", "weaknesses": "1. As mentioned by the authors in line 87 - 89, it is possible to \"update the scene feature throughout the decoder layers with transformers\". Compared to the input point cloud, processing the encoded points (1024/4096 for GroupFree3D/V-DETR) with quadratic time complexity is somewhat acceptable. It is also possible to implement the attention layers with modern flash-attention approaches [R1] for efficiency. The authors are encouraged to present numerical comparisons to support the choice for SSMs.\n\n2. Missing citations. The Hilbert-based point cloud serialization strategy is closely related to PTv3[R2]. The authors are encouraged to state the relation to [R2].\n\n[R1] Dao, Tri, et al. \"Flashattention: Fast and memory-efficient exact attention with io-awareness.\" Advances in Neural Information Processing Systems 35 (2022): 16344-16359.\n\n[R2] Wu, Xiaoyang, et al. \"Point Transformer V3: Simpler Faster Stronger.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)