MamBEV: Enabling State Space Models to Learn Birds-Eye-View Representations
Authors: Hongyu Ke
Jack Morris
Kentaro Oguchi
Xiaofei Cao
Yongkang Liu
Haoxin Wang
Yi Ding
Venue: ICLR 2025 Poster (2025)
Keywords: BEV
Bird’s Eye View
State Space Model
3D Computer Vision
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: 3D visual perception tasks, such as 3D detection from multi-camera images, are essential components of autonomous driving and assistance systems. However, designing computationally efficient methods remains a significant challenge. In this paper, we propose a Mamba-based framework called MamBEV, which learns unified Bird's Eye View (BEV) representations using linear spatio-temporal SSM-based attention. This approach supports multiple 3D perception tasks with significantly improved computational and memory efficiency. Furthermore, we introduce SSM based cross-attention, analogous to standard cross attention, where BEV query representations can interact with relevant image features. Extensive experiments demonstrate MamBEV's promising performance across diverse visual perception metrics, highlighting its advantages in input scaling efficiency compared to existing benchmark models.

Published as a conference paper at ICLR 2025
MAMBEV: ENABLING STATE SPACE MODELS TO LEARN BIRDS-EYE VIEW REPRESENTATIONS
Hongyu Ke! Jack Morris Kentaro Oguchi
Xiaofei Cao?
Yongkang Liu?
Haoxin Wang' Yi Ding'
'Georgia State University
PInfoTech Labs, Toyota Motor North America R&D
[hke3, jmorris 16, haoxinwang, yiding] @ gsu.edu Ktentaro.ogucht xiaofei.cao, yongkang. liu) @toyota.com
ABSTRACT
3D visual perception tasks, such 3D detection from multi-camera images, are essential components of autonomous driving and assistance systems. However, designing computationally efficient methods remains significant challenge. In this paper, we propose Mamba-based framework called MamBEV, which learns unified Bird's Eye View (BEV) representations using linear spatio-temporal SSM- based attention. This approach supports multiple 3D perception tasks with signifi- cantly improved computational and memory efficiency. Furthermore, we introduce SSM based cross- atteent,,n,,on, analogous standard cross attention, where BEV query representations can interact with relevant image features. Extensive ex- periments demonstrate MamBEV's promising performance across diverse visual perception metrics, highlighting its advantages input scaling efficiency compared existing benchmark models. The code is available at https://gthub..co/amaai gsu/MamBEV
INTRODUCTION
Automatically constructing bird's-cye-view (BEV) of an object's surrounding environment is beneficial for tasks such as autonomous driving and driver assistance systems (Wang et al.) 2023a). These methods typically integrate the signals received by multi-view cameras and transforms them into top-down view of the surrounding environment. Furthermore, as these systems operate in an mobile edge environment, construction accuracy (Ke et al. 2024).
important to consider the computational costs in conjunction with
Examples of deployed BEV systems can be seen in Tesla cars (Tesla 2021). These detailed construc- tions can be used for downstream perceptual, prediction, and planning tasks (Casas al. 2021 1 Hu et al. 2023). There are two predominant methods for building BEV models: push and pull. Push methods project 2D image features into 3D space using pixel-wise depth predictions, transforming flat images into spatially-aware 3D representations. Pull methods, on the other hand, sample points from 3D prior and project them onto the 2D image plane, allowing the model to extract 3D information without explicit depth predictions (Li et al., 2022a; Wang et al., 20235 Chen et al. 2022 2023). However, many of these methods rely on the of transtormers' costly attention mechanism to learn accurate representations. Thus, we are motivated to examine more efficient methods that can replace transformer-basee based architectures.
Recently, Gu et al. (2021) and Gu & Dao (2023) showed that state space models (SSMs) can replace the quadratic computational complexity transformers' attention through a linear approximation. These models have been shown to work comparably to transformers scale on large language models (Dao & Gu (2024) such as Codestral Mamba. There has also been evidence that SSM's attention can replace transformer attention to learn effective visual representations (Zhu et al. 2024 Liu et al.) 2024, Patro & Agneeswaran 2024). While Mamba, like self-attention, is effective at capturing intra-sqquence dependencies, struggles in BEV scenarios involving multiple input modalities, where dynamic information exchange is crucial.
"Equal Contribution.

---

Published as a conference paper at ICLR 2025
Canvar Frest vine
Tiata Irane Faateos
Upcind Quanis
Tkdba BEV Quelle
Like
KEY Quaka
Figure 1: We propose MamBEV, novel paradigm that leverages both SSM based Cross-Attention and Self-Attention mechanisms to generate BEV features from multi-camera inputs.
Motivated by these findings, we examine how SSMs can be used to generate a BEV representation. This is important because is costly to capture temporal and spatial relationships multiview videos. We examine how the linear attention inside SSMs can be used to address these issues. However, incorporating SSMs into 3D representation learning tasks is not well understood. Furthermore, it is unclear how to fuse distinct visual representations as this is crucial step in learning BEV representations.
Our paper attempts to address these issues through the following contributions:
We propose an SSM-based architecture, MamBEV, that can exceed the performance of existing Transformer- architectures.
We propose novel approach, Spatial Cross Mamba, analogous standard cross-attention, where a set mapping mechanism enables the association and fusion of two different modali- ties. In our case, BEV query representations to facilitate direct integration of information from both modalities.
matched with corresponding image features
A thorough set ablation studies is provided to showcase model scaling and other properties. We open-source code and provide strong baseline and evaluation framework for future experimentation.
2 BACKGROUND
2.1 LEARNING BEV REPRESENTATIONS
BEV representations are widely used in autonomous driving as they provide dense unified repre- sentation of scene which can be used for variety of tasks. One approach, based on LSS (Philion) Fidler 2020), involves projecting 2D images into a 3D space by leveraging the camera intrinsics. such et al. (2023b). Li et al. (203aa), Han et al. (2024). For example, VideoBEV (Han al.) 2024) adopts an LSS-based 3D BEV detection approach, incorporating a customized recurrent-style temporal fusion and temporal embedding block to leverage long-term information more effectively, However, this method heavily depends on accurate depth estimation to ensure the precision of feature projections. On the other hand, BEVFormer Li et al. (2022a) introduces direct approach by pulling features from 2D image space into a BEV space, collapsing all height information into a single BEV query. adopts deformable attention Zhu et al. (2020), sparse sampling method, as the core of the BEV encoder which maps multiview image features to BEV features. BEVFormerV2 Yang et al. (2023) added an additional head and image augmentations to improve supervision for the model's backbone. It also uses CNN based temporal encoder in place of the recurrent deformable temporal attention used BEVFormer.
Deformable attention offers the advantage of reduced memory consumption, but it also has notable limitations. For example, deformable attention provides only sparse and limited supervision for the backbone, it does not engage with all image features Yang et al. (2023). In terms of feature refine- ment, deformable attention relies heavily on the backbone to produce strong feature representations. For instance, in the Deformable Attention Transformer (DAT) Xia et al. (2022), multiheaded full
2

---

Published as a conference paper at ICLR 2025
Muili Carrea Insut
Learred BEV Query Erntwackigg Grdd
Spatally Algned BEV Gastry Grid from Provious Franets)
Leemedi Otject Guery Vectors 8 Reforence Poinss
Reetler 101 Backbore
Fenture Pyrarid Networ Neck
Enoodar
Pecklt Tempost Fus on
Ful-Aterrion Decoder
Pradicion Heads
Predictions
Key
Learne: BEV Queries Q
Convolution
SSM Altent
Image Festures
Cenoo-Vaanta
Croes Attention
Deformable Atamiion
Enoodingg
Figure 2: The overall pipeline of of our architecture (MamBEV-Smanl Small). We present novel method for incorporating SSMs into a BEV construction algorithm. Features are extracted from six egocentric multiview camera images over multiple frames. A ResNet backbone is used to extract camera features which are passed to SSM based encoder blocks. We found that it was necessary to use full attention during the decoding process, however this has limited impact on the computational complexity as the encoded feature sequence is relatively short.
attention remains necessary to refine features effectively, underscoring the limitations of deformable attention in refining spatial representations. From a hardware perspective, deformable attention is suboptimal for modern GPU architectures due to its reliance on random memory access during the sampling process, which negatively impacts model throughput Zhu et al. (2020).
2.2 LINEAR ATTENTION WITH STATE SPACE MODELS
State space models (SSMs), particularly structured SSMs, present an effective linear complexity alternative to the prevalent transformer architecture for sequence modeling. At their core, SSMs operate on data from each channel independently which updates a latent state that evolves across the input sequence. The evolution of this latent state is governed by a of learnable parameters, denoted as A, B, C. Mamba Gu & Dao (2033) introduced selective SSMs (S6) which enables the model to dynamically change the values of the A, B, parameter matrices based on the input, effectively acting as filter or gating mechanism. This mechanism better allows SSMs to model input sequences which have elements with varying degrees of information density, such text, as can selectively ignore tokens which are information sparse and selectively remember those which are information dense. Mamba also included significant hardware optimizations which allows for parallel computation of state updates using an associative scan.
Mamba-2 (Dao Gu 2024) leverages a novel understanding of the connection between SSMs and attention mechanisms, termed state space duality (SSD), overcome some key weaknesses of its predecessor, Mamba-1. SSD demonstrates that SSM computations can be expressed through a dual form involving structured matrix multiplication. This allows Mamba-2 to leverage highly optimized matrix multiplication units on GPUs, resulting in a significant speedup compared to Mamba-l I's scan-based implementation.
While Mamba-2 excels in autoregressive tasks, its underlying SSM framework inherently operates in a causal manner, limiting its applicability non-causal scenarios. Hydra (Hwang et al. (2024) addresses this limitation by leveraging guasiseparable matrix mixers which generalize the semiseparable matrix mixer found in Mamba-2 encompass both lower and upper triangular components. This change enables bidirectional information flow with a minimal increase computation and parameters. For simplicity, we use Mamba when discussing the Hydra block in the following sections of the paper, as Hydra uses the same block structure and SSM formulation as Mamba-2.
SSM Inner Function. State Space Models can be considered as systems that map signal æ(t) E R into yt) E IR through h(t) E N,N, which can be formulated as
h' h1(1) Ah(t) Br((), v(1) Ch(() Dz((),
(1)
3

---

Published as a conference paper at ICLR 2025
where A RRNN is the evolution parameter, B parameters.
E RNX, Frxxx,C CeRRI R and D E R are the projection
Discretized Inner Function. In machine learning applications, most inputs not continuous signals so to adapt these systems to discrete input sequences, the system itself must be discretized. The discretized version can be formulated
h Ahh-1 + Brg.
(2)
y/ Ch+ - Drt:
where timescale parameter A is used transform the continuous parameters A and B to discrete parameters and B:
A softplus(dt (dt + ddaias),
(4)
Ā cAA
B (AA)) D)AB
This operation explicitly ties the hidden state update parameters A and B together. In practice, this operation is not necessary since the model can learn the discretized system directly noted in (Gu Dao 2023). However, the inclusion of a discretization function provides a useful mechanism for interpreting how given input interacts with the hidden state well methods which enable the direct control over whether the hidden state is updated by a given token or class of tokens. Ali ctt al. (2024) notes that each channel having a distinct discretization parameter is equivalent to having multiple heads in attention where each head in this case corresponds to its own channel. In Mamba-2, the authors found that tying multiple channels together by sharing a discretization factor between them reduced computational complexity while retaining similar level of expressivity. When grouping multiple channels into a head, the matrix can be shared and the number of di which are computed as function of the inputs are also reduced. Additionally, they found that reducing A from a diagonal matrix in N X to a scalar reduced computational cost with minimal impact on expressivity. The SSM component in diagrams refer to the operations in equations 2 and
3 METHODS
The BEV construction problem is a supervised learning algorithm which maps camera images I, (m1,..., m of k camera views at particular time step i onto 2D planar Birds-Eye-Viee View of objects. A given BEV scene that contains I objects, has bounding boxes (b,,...y} E B, classes (ci..., a} E
well as trajectory information T. The approach is to first learn an encoding transformation Q = (Z,) for all é, to obtain a latent BEV query representation Q E 1e # x where H, W represents the spatial shape of the BEV grid and D is the latent dimension. decoding transformation is then learned to predict object information B, C, T = (Q) by sampling the BEV grid.
BEV Method Overview, A visualization of our overall method for learning a BEV construction provided Figure Many of the existing components follow the methodologies of Li al. (2022a). Li et al. (2023)). and Liu et al. (203). The encoding function fene is parameterized by the ResNet101 backbone, feature pyramid network (FPN), BEV encoder, and temporal fusion modules. The pipeline has three main stages follows. First, image feature maps of different scales Fo. ..., F;; are produces as intermediate backbone outputs. Then their channels are reduced to a uniform size D through horizontal convolutions in the FPN (Lin et al. 2017). Second, feature representations and interactions are modeled using novel SSM-based pipeline (discussed below). Lastly, the decoding function Jdne operates on this representation to make predictions. As in Zhu et al. (2020) Li et al.) (002a Yang et al. (2023) ) methods, the decoder uses alternating layers of grouped object query self-attention and object query BEV feature deformable cross-attention. Decoder heads each layer follow the masked decoder heads from Li et al. (2022b) which predict bounding box, its properties, and the object class. We do not use map segmentation head, and only use the detection head which is optimized using 3D hungarian set matching with smooth L, loss for the bounding boxes and focal loss for class prediction.
4

---

Published as a conference paper at ICLR 2025
We next discuss the novel encoder in Figure
BEV Queries
Image Feature Values
3.1 SPATIAL CROSS MAMBA
Win
Lineur
T-snsoon
The central challenge in using Mamba to BEV is how to perform cross attention in an ef- ficient manner. There simple cross attention adaptations for SSMs which are inefficient or not well suited for the problem. For example, by panding the state size N to equal the number of image features T Mamba-2 able store all information about previous tokens at the cost of returning to complexity of transformers [ao & Gu (2024). To perform this naive form of mamba cross attention, first compute the final hidden state lyr Ixx over the image features then find yQ using
Q+
QuBC
Noelisear
di, 1
Vatraa
10 Concalenation
Adátion
BEV Position Aware Merge
L
Depthwiie
Convolutien 113
F BC
XQSSM
(7) where C; RIXT is function of corre- sponding query E Q. Computing ht takes TNP = T FLOPs where P is the latent di- mension per head and TN = 2 memory which in this case equivalent to the FLOPs and mem- ory used to compute self-attention in a Trans- former, T and T To compute 3/1 requires TiQlaD FLOPP and TIQ) memory giving a total computational and memory complexity similar to Transformer cross attention. To reduce the com- putational cost from the naive implementation, we propose task specific adaptations as shown in Figure 3 which allow us to reduce the size of the hidden dimension N << T.
1/1 CjhT
pxtel
RMS
Qaam
max (1, Qeaes))
Woy
Q
RMS
Figure 3: Proposed Spatial Cross Mamba using XQSSM. Our novel method to fuse two distinct spatial representations: 1) BEV queries which top-down representation, and 2) image features which come from an egocentric view.
3.1.1 REDUCING STATE SIZE
Simply reducing N may still give acceptable re- sults if the image is information sparse, however this cannot be guaranteed. The resulting hy would likely lose information about image features near the start of the sequence. This results from the definition of the decay parameter exp(AA) E [0, 1], A < 0, A > 0, which guarantees that the impact of yr decreases as T increases except when A; 0, ... T. The solution we propose is to compute y/ using Cyhı, where image feature vector is likely to be relevant to q Since multiple
image regions may be relevant to single BEV query, use Z copies of C2 to attend to Z locations on the image feature map, then
Z
y: = Σ C. k=
(8)
To select the Z feature map locations for a query we lift the 2D BEV location (x, y) into 3D pillar (x,y.) and project Z evenly spaced pillar points onto each image Lang al. (018).
The resulting locations called reference points E IR xx correspond where object
a BEV location would appear in the image based on the ego vehicle's camera calibration and assuming there are no obstructions. The number of locations which fall inside the image bounds.
Ruit = {Tis Tij E [0, C R is significantly smaller than QZ. We refer to the size of Rpic as M [[[Q), Z, Opov) 2
2Q as most y; are only visible in a single view.
3.1.2 BEV POSITION AWARE MERGE
Q. and V are flattened to ID sequences so that they can be processed by Mamba. V is flattened according to traversal order T H XxD ب IR HWWxal Then Qui merged with the flattened
5

---

Published as a conference paper at ICLR 2025
Pg,10 M933
M3,10 9:
P9,30
Reaidual Connection
9GM
QL
22
Linear Layer
Σ
Biua
06.03
Output Camera for Q1x
Ourput Camera 91,
Figure 4: Spatial Cross Mamba Pre- and Post-processing. Illustration of the processing performed on the input and output of the SSM to merge sampled information from multiple query copies in the input sequence into the BEV query grid. Image features (denoted by Us,z) and their corresponding query vectors (qx) are first interleaved to enable causal attention. Processed outputs of the SSM are normalized and fused into an updated query matrix (BEV.
sequence of image features through the ID reference points RiD as follows:
P h| R [
(9)
IndexOffset RID) RID + argsort RID);
(10)
where Ru, R1 E [0, 1] are the normalized and coordinates of the reference location on the image. The argsort function returns the index of each element in the sorted order of RID in ascending order and is added back to Rip for index reordering. This formulation takes O log n)) time, though in practice this operation has negligible cost. The result is list of one dimensional indices which correspond to the location one column to the left and one row above the hit image location supposing the image features have been flattened using row major traversal. In cases where other traversals are used, the indices are updated with T. Lastly, from RiD we generate a mask 8makk ( +Naa)x1 which marks the insertion points for each query qi for hit camera view. To complete the merge operation copy the queries Quit to their corresponding reference points and the values F; to the masked output tensor. This operation is repeated for every camera and every traversal method. An example of an input sequence can be seen in Figure Ablation studies are also performed examining the traversal order effects in Table
3.1.3 CROSS QUASI-SEPARABLE STATE SPACE MODEL
Mamba-2 offered dual methods of computation for selective state space models: as a matrix mixer, SSD and an associative scan, SSM. The reframing of SSMs as structured matrix mixers, M, allows for fast parallel computation by way of batch matrix multiplication and a shortened associative sean. In practice, utilize SSD during both training and inference it is better optimized for the hardware used, however we did not adapt the kernel to reflect the true computational complexity of the Cross Quasi-separable State Space Model (XQSSM) module.
The simple sequential implementation Algorithm 2 (Appendix) shows the unique approach allowed (aaistmmm@ppppppepeee
by module formulation where the discretization factor of the query input, Qet, is set 0. No hidden state update occurs when a token with di = 0 is processed as shown below
h exp(dl, A 14-11 + dl, Bat.
(11)
h, hxp(0)m,-1 0,
(12)
h4-1-
(13)
This changes the number of operations per query from 2H 11 + aD(3N H + 1) to (N H + 1). Additionally, outputs from the XQSSM are only needed for query token inputs which reduces the per image feature complexity 2H(N + + D. In total the computational complexity of

---

Published as a conference paper at ICLR 2025
the XQSSM is (V +DD) MaD(N + H+1), where M is the number of queries which are added to the sequence. Additionally. the memory complexity in the sequential form is constant, though when parallelized it becomes linear with respect to the sequence length as shown in Dao & Gu (2024). The resulting matrix mixer, M goes from a (M + V x M + V x 2H) to (M V x 2H) similar to the matrix mixer for dot product cross attention of shape (Q x K x H).
3.1.4 QUERY AND FEATURE DOST-PROCESSING
After the merged input sequence is passed through the inner SSM block, queries must be extracted from the output Y RR (WW++M)xa obtain the updated BEV queries YBEV. The mask generated during the merge operation Smask is then applied to the output to obtain Yo E Mxx Then using RID each query in Yę is accumulated in Qy 0 R Nxo at its original position in QBEV Each element 4y E 22 has a magnitude which varies depending on the number of queries accumulated in that BEV location. The result is projected to Qont E Nx then added to the residual BEV query grid Q. To account for instability resulting from disparity in magnitude, two methods of normalization are considered: averaging before the out projection and RMS normalization after the out projection. Each method and their joint usage is ablated in Table 7
BEV Self-Attention. The normalized BEV grid Q is traversed with a single hydra layer (bi-directional mamba) in row-major order before being fed deformable layer incorporate multiscale features.
Deformable Attention. Deformable attention is a sparse attention method which excels in detection tasks while training in less time, with fewer flops, and better scaling than vanilla softmax attention Zhu et al. 2020)). The foundation of the method is to take an input set of image features and a set of coordinates which act as a reference locations then grid sample the area around the reference location based on using offsets predicted queries.
The computational cost of deformable attention is divided into main components: the cost of calculating the offsets for each query 0(3NMPRD) the cost of bilinear interpolation and the weighted sum of samples O(N D2 + NPRD2 + 5 PNDD),, and lastly the cost of computing the linear projection of the values H, W; D2 for each image feature level F; with shape (H; X W; X D). Here N refers to the number of queries, D is the latent dimension, P the number of reference points per query, and R is the number of offsets per reference point. The total computational 5ND + 3NMPRD) and complexity of the operation is O + min(HW' D2 NPRD2) + under the assumption that 5PR + 3MPR < D, the overall complexity is simplified to O(2ND
min( HW D2 N PRD2)). Notably, the computational complexity here is low relative to the size of the image the heaviest component of the bound derives from the number of queries, heads, and reference points used. In cases where HW is small relative to the number of queries, deformable attention is similar to or worse than linear attention alternatives like Mamba in computational complexity while giving weaker form of cross attention.
EXPERIMENTAL SETUP
We follow the methodologies of the previous work of Wang et al. (2022) Li et al. (2022a) and Yang et al. 2202).. We evaluate on two backbones: R0 and ResNet50 which are trained in a depth prediction task and COCO, respectively. During training, the first stage of the backbone frozen, and all other stages are trained at 10% learning rate to fine-tune their latent representations to the multiview autonomous driving setting.
4.1 DATASET AND METRICS
We conduct our experiments using the nuScenes dataset Caesar et al. (2020). The nuScenes dataset is a large-sale autonomous driving dataset containing 1000 driving scenes from Boston and Singapore. Each scene is approximately 20 seconds duration. 23 object classes with 3D bounding boxes are annotated at 2Hz for the entire dataset, of these 10 are used for the 3D detection task. Each scene is captured using 6 cameras with a FOV of 360 degrees, LiDAR, radar, GPS, sensor calibration, and IMU data. The evaluation metrics and framework for computing them are provided nuScenes devkit. The metrics used for evaluation are 1) the mean average precision (mAP), which evaluates both localization and classification performance of the predicted results over four different
part of the
7

---

Published as a conference paper at ICLR 2025
XOD ASE
Figure 5: Visualization results of MamBEV- on nuScenes val set. We show the 3D bboxes predictions in multi-camera images and the bird's see'--eee-view.
thresholds using center distance on the ground plane, and 5 types of true positive metrics: 2) average translation error (ATE), 3) average scale enror (ASE), 4) average orientation enror (AOE), 5) average velocity error (AVE), and 6) average attribute error (AAE). The nuScenes also defines nuScenes detection score (NDS) by combining mAp with five true positive metries for comprehensive assessment. In our method and experiments, only the camera frames, sensor calibration data, and GPS data are used in making predictions.
4.2 IMPLEMENTATION DETAILS
We used a learning rate of 8 x 10-4, 4, with linear warmup for 10% of the scheduled steps starting from 5 x 10-4 Following the warmup, the learning rate follows an epoch based cosine annealing schedule with a minimum learning rate of 8 x 10-7, We trained with an effective batch size of 32 with gradient accumulation on 8 A100s for 30 epochs, truncated at 24 epochs. Starting from step 100 an exponential moving average according to the function w/ = (1 0.0002)w;, + 0.0002w, is applied to all weights. An AdamW optimizer with 0.01 weight decay is used, and training employs an automatic mixed precision optimizer wrapper with an initial gradient scaling of 512. A 0.1 multiplier is applied to the learning rate of the backbone weights and the deformable attention sampling offsets Zhu et al. (2020). We train the models from scratch using a randomly initialized network for the encoder layers. The source code will be made available upon publication.
RESULTS
Method
Backbone
Frames
NDSt
mAPt
MATE,
mASE/
mAOEL
mAVE,
mAAEf
BEVForerV -Timy
ResNet50
0.354
0.252
0.900
0.294
0.655
0.657
0,216
BEVForerr T
ResNe150
0.397
0.469
0.195
MamBEV-Tay
ResNet50
3
0.399
0.266
0.794
0.298
0.575
0.469
0.199
FCOS3D
ResNet101
0.415
0.343
0.725
0.263
0.422
1,292
0.153
Focal-PETR
ResNer101
0.461
0.390
0.678
0.263
0.395
0.804
0.202
DETR3D
ResNet101
0.434
0.716
0.268
0.379
0.842
0.200
LEVFomerVV-Smman
ResNet101
0.448
0.375
0.725
0.272
0.391
0,802
0,200
BEVFomerV2
ResNet101
0.426
0.355
0.751
0.275
0.429
0.847
0.215
MamBEV-Smal
ResNet101
0.444
0.392
0.696
0.283
0.411
0.897
0,230
PolarDETR-T
ResNer101
0.488
0.383
0.707
0.269
0.344
0.518
0.196
BEV
ResNet101
0.436
BEVFomerVi Base
ResNet101
0.517
0.416
0.673
0.274
0.372
0.394
0.198
ResNet101
0.506
0.412
0.676
0.281
0.400
0.470
0.187
MamBEV-Sman
ResNet101
0.523
0.415
0.656
0.281
0.379
0.340
0.192
MamBEV-Small
ResNet101
0.525
0.423
0.662
0.280
0.386
0.354
0.183
Table Main Results. Our method is the best when accounting for temporal properties. Our model outperforms existing techniques while requiring fewer computational resources. The best results for each experimental setup is highlighted in bold. * indicates models trained by us. f indicates without deformable attention.
We first report main results followed by ablation studies to understand the efficacy of various model choices. Unless otherwise specified, we used single temporal frame in all ablation studies. Ablation models were trained for 12 epochs and used ResNet50 backbone pre-trained the COCO
8

---

Published as a conference paper at ICLR 2025
object detection dataset. While performance improved with more training, it did not differ relatively to different model parametrizations.
Main Results. We present comparison of our results in Table 1 against oftte-of-thearrr art methods at compatible parameter and image input scales. We only use camera features; additionally, we do not make of any auxiliary loss in the works of Yang et al. 2023). We aligned the definitions of tiny and small models with BEVFormerV1. We trained only the tiny and small versions of MamBEV. The CNN based temporal component of our model consists of 57M parameters for the small configuration and 32M parameters for the tiny configuration. Excluding the temporal portion, our MamBEV-Snin model has 65M parameters, while MamBEV-Tiny has 39M parameters. For comparison, BEVFormer's ' tiny, small, and base models have 34, 60, and 69 million parameters respectively.
We report comparable results (+.002 NDS). MamBEV-Tiny, over previous state-of-the-arr methods evaluated on similar conditions for the ResNet50 backbone. We report an increase of .046 (+9.6%), ManBEV-Small with frames, in NDS over the previous BEVFormerV 1-Small model and 0.008 (1.5%) increase over the previous BEVFormerV V-rnner model. This shows that our model can improve performance while reducing memory complexity.
Effectiveness of Spatial Cross Mamba. To verify the effectiveness of the Spatial Cross Mamba layer, train small model names MamBEV-Small- Pure without deformable layers in the encoder. This configuration displayed good performance NDS and mAAE metrics that outperforms PolarDETR-T and BEV FormerV CiamnerV--SSa 1-Small. However, it outperformed by the small model which made of deformable layers.
Efficiency. In table 2] test the memory and compute estimated FLOPs for model configurations which use our XQSSM, standard dot product attention, or deformable attention. XQSSM and deformable attention scale linearly in memory and computational complexity with respect the size of the inputs V and Q, though the coefficient factor of deformable attention is smaller.
Cross Attention Module
BEV Scale (Q)
Image Size (V)
Params (K)
GFLOPs
Memory (GB)
50x50
800x450
239
3.7
1.7
XQSSM
100
1280x720
239
14
3.5
200x200
1600x900
239
51
6.5
50x50
800x450
156
3.3
1.7
Deformable
100x100
1280x720
156
12.8
3.2
200x200
1600x900
156
49.5
4.9
50x50
800x450
263
23.9
2.2
Dot Product
100x 100
1280x720
263
228.8
9.7
200x2000
1600x900
263
1,432.5
>24
Table 2: Scaling of cross attention modules with respect to BEV grid and image sizes. All models were tested with simple R50 backbone and single encoder and decoder layer. Memory measurements were taken at train time.
Increasing Temporal Information. We conduct experiments on the effect of increasing temporal information by providing the model with additional previous frames. Utilizing higher number of frames requires attending to additional information during the encoding step. Since SSM-based attention can reduce this complexity, we can better utilize this information. We report our results in Table
# Frames
NDS
mAPt
MATE|
MASEJ
mAOFJ.
mAVEJ
mAAEJ
Params.
0.3128
0.2362
0.8882
0.3098
0.7158
0.9114
0.2269
39M
0.3512
0.2415
0.8657
0.3126
0.6743
0.6134
0.2292
53M
0.3730
0.2546
0.8480
0.3156
0.6576
0.5073
0.2143
71M
0.3747
0.2545
0.8614
0.3132
0.6464
0.4820
0.2227
96M
0.3874
0.2705
0,8400
0.3111
0.6323
0.4740
0.2209
128M
0,3825
0.2643
0.8372
0.3143
0.6298
0.4971
0.2184
266M
Table 3: Performance of model across all metrics NDS of models nuScenes validation set using different numbers of temporal frames.

---

Published as a conference paper at ICLR 2025
As can be observed, increasing frames helps overall performance until approximately 5 frames. The worst performance occurs when a single frame is used to make predictions. Unsurprisingly. the mAVE, measurement of velocity prediction error, is also significantly higher when only a single time step of multi-view video is available to the model.
SSM-hased Attention versus Deformable Attention. We compare formulation of spatial cross mamba against deformable attention. We present our results in Table 4 Spatial cross mamba can serve as a replacement for deformable attention there is minimal difference in performance. In our experiments, we found that using mixed spatial and deformable attention was helpful when training a larger network that learns representations over larger number of frames.
Method
NDSt
MAPt
MATE
MASEL mAOE
mAVE
mAAE
Deformable Attention
0.3140
0.2384
0.8891
0.3097
0.7166
0,9107
0,2255
Spatial Cross Mamba
0.3141
0.2386
0,8882
0.3098
0.7158
0,9114
0.2269
Mixed
0.3128
0.2362
0.8799
0.3086
0.7044
0.9349
0.2246
Table 4: Comparison of encoder layer submodule formulations. Minimal difference is observed between the methods. Mixed attention involves a single layer of spatial cross mamba followed by single layer of deformable cross attention.
Scaling Experiments. We conduct experiments to understand the effect of scaling the number of channels inside the Mamba module well the hidden state inside the state space model. We report the hidden state scaling experiments in Table [ and the channel scaling results in Table B Minimal effect observed in our experiments when adjusting these variables. The findings suggest that increasing the scales of channel features and hidden state may not necessarily lead to improved
performance.
Hidden State
NDSt
mAPt
MATEJ
MASEL
mAOE
mAVE
mAAE|
16
0.3167
0.2368
0.8812
0.3073
0.7012
0.8932
0,2339
32
0.3128
0.2362
0.8882
0.3098
0.7158
0.9114
0.2269
64
0.3111
0.2356
0.8861
0.3117
0.7015
0,9392
0,2288
128
0.3125
0.2371
0.8730
0.3106
0.7077
0,9410
0,2287
256
0,3164
0.2397
0,8790
0.3110
0.7151
0,8979
0,2319
Table 5: Effect of adjusting SSM hidden state size. Minimal difference in performance is observed.
Queries Insertion. To validate our BEV Position Aware Merge (Project), we conduct experiments to show the effectiveness in Table o The results indicate that Project the most effective query insertion method for our model, particularly in terms of detection performance and minimizing key errors. is reasonable that the performance of append and prepend worse as Mamba learns spatial relations mainly through the position of elements in sequence.
Method NDSt
mAPt
mATE)
mASEL
mAOEL
MAVEL
mAAE
Append 0.3051
0.2314
0.8897
0.3103
0.7560
0.9229
0.2221
Prepend
0.3073
0,2300
0.8917
0.3106
0.7695
0.8887
0,2265
Project
0.3128
0.2362
0.8882
0.3098
0.7158
0.9114
0.2269
Table 6: Performance comparison of our model using different query insertion methods. Ap- pend/prepend represents naively append/prepend the queries after the corresponding image feature maps. Project is our proposed BEV Position Aware Merge method.
CONCLUSION
Our work presents BEV construction model. We show that it improves performance over prior art when evaluated the similar experimental conditions. Extensive ablation studies demonstrate that the model is robust to various changes. Our source code will be open open-source for future research purposes.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Research was sponsored by the Army Research Laboratory and was accomplished under Cooper- ative Agreement Number W91 INF2232222222 and Toyota Motor North America. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the Army Research Laboratory or the U.S. Govemment. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation herein. The contents do not necessarily reflect the official views of Toyota Motor North America. This work is partially supported by the National Science Foundation (NSF) under Grant OAC-2017564, which has been instrumental in providing valuable training opportunities for the author(s).
ETHICS STATEMENT
Our motivation for studying this problem primarily driven by a desire to minimize the computa- tional costs associated with training and deployment of large deep learning systems. The potential impacts of this are two fold, first it may lead improved real-time performance properties, second, would reduce the carbon footprint of training large models.
REPRODUCIBILITY STATEMENT
Code will be publicly available through github. We will also include installation scripts as the libraries have complex dependencies and compilation needs. We hope this will facilitate faster future development.
REFERENCES
Ameen Ali, Itamar Zimerman, and Lior Wolf. The hidden attention of mamba models, 2024. URL https://arxiv:
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Osear Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 1621-11631, 2020.
Sergio Casas, Abbas Sadat, and Raquel Urtasun. Mp3: A unified model to map, perceive, predict and plan. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14403-14412. 2021.
Dian Chen, Jie Li, Vitor Guizilini, Rares Andrei Ambrus, and Adrien Gaidon. Viewpoint equivariance for milti-view 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9213- 9222, 2023.
Zehui Chen, Zhenyu Li, Shiquan Zhang, Liangji Fang, Qinhong Jiang, and Feng Zhao. Bevdistill: Eodsl bev distillation for multi-view 3d object detection. arXiv preprint arXiv:2211.093866 2022.
Tri Dao and Albert Gu. Transformers are ssms: Generalized models and efficient algorithms through structured state space duality, arXiv preprint arXiv:2405.2106 2024.
Albert Gu and Tri Dao. Mamba: Linear-time sequence modeling with selective state spaces. arXiv preprint arXiv:2312.00752 2023.
Albert Gu. Karan Goel, and Christopher Ré. Efficiently modeling long sequences with structured state spaces. arXiv preprint arXiv:2111.00300 2021.
Chunrui Han, Jinrong Yang, Jianjian Sun, Zheng Ge. Runpei Dong, Hongyu Zhou, Weixin Mao, Yuang Peng, and Xiangyu Zhang. Exploring recurrent long-term temporal fusion for multi-view 3d perception. IEEE Roboties and Automation Letters, 2024.

---

Published as a conference paper at ICLR 2025
Yihan Hu, Jiazhi Yang. Li Chen, Keyu Li, Chonghao Sima, Xizhou Zhu, Siqi Chai, Senyao Du,
Tianwei Lin, Wenhai Wang, et al. franning-orientee autonomous driving. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 17853-17862, 2023.
Sukjun Hwang, Aakash Lahoti, Tri Dao, and Albert Gu. Hydra: Bidirectional state space models
through generalized matrix mixers. arXiv preprint arXiv:2407.09941, 2024.
Hongyu Ke, Wanxin Jin, and Haoxin Wang. Carbonep: Carbon-aware dnn partitioning with conformal prediction for sustainable edge intelligence. arXiv preprint arXiv:2404. 16970, 2024.
Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpil- lars: Fast encoders for object detection from point clouds. CoRR, abs/18 181.05584, 2018. URL http://arxis wtt.oorgg//bb/a//oo 05784
Yinhao Li, Han Bao, Zheng Ge, Jinrong Yang, Jianjian Sun, and Zeming Li. Bevstereo: Enhancing depth estimation in multi-view 3d object detection with temporal stereo. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1486-1494 2023a.
Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang. Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1477-1485, 2023b.
Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird' s-eee-view representation from multi-camera images via spatiotemporal transformers. In European conference on computer vision, pp. 1-18. Springer, 2022a.
Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp-
1280-1281 1289,229,,200 2022b.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 21 17-2122, 2017.
Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Aqi Gao, Tiancai Wang, and Xiangyu Zhang. Petrv2: unified framework for 3d perception from multi-camera images. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3262-3272, 2023.
Yue Liu, Yunjie Tian, Yuzhong Zhao, Hongtian Yu, Lingxi Xie, Yaowei Wang, Qixiang Ye, and Yunfan Liu. Vmamba: Visual state space model. arXiv preprint 2011::2401.10166. 2024.
Badri N Patro and Vijay S Agneeswaran. Simba: Simplified mamba-base based architecture for vision and multivariate time series. arXiv preprint arXiv:2403.1 15360, 2024.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs
by implicitly unprojecting to 3d. In Computer Vision-FCCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XIV 16, pp. 194-210. Springer, 2020.
Tesla.
Tesla ai day 2021, 2021. 5044FweCy4M
URL
https: http///w.woooo. www.w.ouuubbeccom wwt.:b/..oooooooooocccooo vv=
Haoxin Wang, Ziran Wang, Dawei Chen, Qiang Liu, Hongyu Ke, and Kyungtae KT Han. Metamobil- ity: Connecting future mobility with the metaverse. IEEE Vehicular Technology Magazine, 18(3): 69-79, 2023a.
Shihao Wang, Yingfei Liu, Tiancai Wang, Ying Li, and Xiangyu Zhang. Exploring object-centric temporal modeling for efficient multi-view 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3621-3631, 2023b.
Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang. Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference Robot Learning, pp. 180-191. PMLR, 2022.
12

---

Published as a conference paper at ICLR 2025
Zhuofan Xia, Xuran Pan, Shiji Song, Li Erran Li, and Gao Huang. Vision transformer with deformable attention, In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4794-4803. 2022.
Chenyu Yang, Yuntao Chen, Hao Tian, Chenxin Tao, Xizhou Zhu, Zhaoxiang Zhang, Gao Huang, Hongyang Li, Yu Qiao, Lewei Lu, et al. Bevformer v2: Adapting modern image backbones to bird's eeeeviwww recognition via perspective supervision. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17830-17839, 2023.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.09417, 2024.
Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX (SUPPLEMENTARY)
A.1 ADDITIONAL ABLATION
Feature Normalization. We show the effect of various norms on the latent BEV queries. We demonstrate this in Table 7 Normalization and averaging features slightly improve the overall performance of the model, as evidenced by the highest NDS and mAP scores. Even without normalization or averaging, the model still performs relatively well, but the combination of these techniques is critical to have the best detection precision.
Method
NDS1
MAPt
mATE|
mASEJ
mAOE
mAVE!
mAAE|
Average
0.3125
0.2355
0.8886
0.3084
0.7174
0.9056
0.2327
RMSNorm
0.3086
0.2358
0.8931
0.3143
0.7337
0,9197
0.2322
Neither
0,3110
0.2336
0.8905
0.3092
0.6859
0.9361
0.2367
Both
0.3128
0.2362
0.8882
0.3098
0.7158
0.9114
0.2269
Table 7: Performance comparison of our model using combinations of feature normalization tech- niques. Using both normalization helped very slightly in performance.
Expand
NDSt
mAPt
MATE
mASEJ
mAOE
mAVEJ
mAAE,
0.5
0,3003
0.2129
0,9041
0.2999
0.7046
0.9255
0.2270
0.3040
0.2177
0,9160
0.3064
0.6842
0.9138
0.2283
0.3036
0.2177
0.9025
0.3044
0.6881
0.9225
0.2350
0.3065
0.2182
0,9064
0.3066
0.6906
0.8771
0.2451
0,3054
0.2216
0.8863
0.3044
0.6862
0.9309
0.2458
Table 8: Comparison of various expansion scales for the linear layer in our Mamba spatial cross mamba. While the performance differences between scales are minimal, the computational cost increases proportionally with the expansion scale.
Feature Traversal Order. We examine whether the traversal order on the input features of the spatial mamba affects the performance. This is because SSM attention operates on ID input sequence but our input is a 2D image feature map. A 2D feature representation is given by the matrix [[vo,o- .... W. .... [UH,o, ..., UH,w]] []. Flattening according changes this to the vector (e0.0. .... UH.0. v.. UH,, Row-major order flattens it to the form (vo.o...., Wo 01,0: ...]. The snake scan changes goes in reverse once it reaches the edge the matrix. u2........
column-mayj mmajor order
For example, for horizontal snake scan, the vector would be [vo,o...., Ho, , We also consider patch based local scans which work by first dividing the image feature map (H x W X D) into patches of shape (H, W, x D) where // and W are divisible by Hp and W respectively. The inner traversal order flattens the resulting P patches into ID sequences of shape (H,W, x D). The outer traversal determines the order of the flattened patches in the final sequence by mapping the P sequences of length H, Wp to one sequence with shape (PH,W Wp x D).
Traversal Method
NDS
Column-major
0.3176
kow-major
0,3128
Row-mat major column-mapj
0.3124
Column snake
0,3126
Row snake
0.3201
Row snake + column snake
0.3188
Row snake inner, column snake outer
0,3173
Row snake inner. aalumn-major outer
0.3201
Table 9: Effect of Traversal Order.
These patches are internally traversed and the patches are then traversed globally. We also evaluate combinations of scanning order which would lengthen the number of outputs during spatial cross
14

---

Published as a conference paper at ICLR 2025
mamba. Results visualized in Table 9 method for scanning and additional traversal orders in a single layer did not improve performance significantly.
We found that horizontal snake order is simple and effective
Method
NDSt
mAPT
mATEL
mASE
mAOEJ
mAVE
mAAE]
Params
Self- Attentttnnion
0.2991
0.2196
0,9054
0.3091
0.7498
0.9148
0.2274
37.5M
Deformable
0.2992
0.2246
0.9216
0.3114
0.7437
0.9092
0.2447
37.5M
Hydra 0.3061
0.2282
0.8962
0.3131
0.7167
0,9184
0.2358
38. 1M
Hydra a
0.3128
0.2362
0.8882
0.3098
0.7158
0.9114
0.2269
38.9M
Table 10: Performance comparison of different Self-Attention styles.
BEV Self-Attentio Styles. We conduct experiments to show the effect of various self-attention styles for the BEV query grid. We demonstrate this in Table I0 Along with a slight increase in parameters, the performance of hydra (bi-directional mamba-2) outperforms normal self attention and deformable self attention, as evidenced by the NDS and mAP scores. This validates the effectiveness of our BEV Self-Attention design.
Inner Cross Mamba Block Order. We fully explore the merge order and extract order inside the
Cross Mamba block. The results reported in Table 11. Merging after convld and extracting queries before gate with Bg = 0, Cy = 0, and dig = 0 have outstanding performance. While merging before convld and extracting queries after gate with Bg 0 and diq 0 also have not bad performance, this combination suffer from more computation as there are additional conv I operation for queries.
Merge Order
Extract Order
Zero Parameters(s)
NDS
mAP
0.2937
0.1835
dtQ
0.2921
0.1840
Before Conv ID
After Gate
BQ
0.2991
0.1870
BQ, dig
0.3010
0.1870
0.3035
0.1864
dtQ
0.2978
0.1842
After ConviD
Before Gate
Bq,dto
0.2985
0.1892
Bv.Cv,
0,3071
0.1946
0.1849
0.0614
After ConviD
After Gate
dtQ
0.21588
0.0895
Table 11: Ablation of different merge and extract operation orders tested on a model with 3 layers with alternating layers of SSM Spatial Cross Mamba and SSM Self attention with 2 temporal frames. Other settings of note: self attention uses hidden state expansion size of 2 instead of 4, with a state dimension of 64 instead of 128 with the same number of attention heads. Additionally, a grouped decoder is used with 6 groups. Trained for 12 epochs on the same schedule as other ablations.
A.2 ADDITIONAL EFFICIENCY ANALYSIS
In our assessment, the reported runtime numbers may not accurately reflect the potential of our method, as the implementation we used has not been optimized to the same extent as existing methods. Notably, the authors of Mamba and Mamba-2 have demonstrated that the inner SSM can be computed efficiently and have provided optimized implementations of this block in their works. However, we did not modify this block and instead repurposed the existing Mamba implementation as explained in Section 3.1.3 This highlights the possibility of further refinement and performance improvements our approach. Compared attention-baser methods, Mamba exhibits significantly better scaling, achieving linear complexity. While the full theoretical efficiency outlined in our work has not yet been fully realized, the estimated FLOPs and inference memory metrics serve as the most accurate representations of our method's performance to date. To underscore our runtime performance gains, we provide comparative analysis with full attention-ban based methods in the table
Our primary contribution and goal are not a minor improvement in performance in BEV detection, but rather method to show how Mamba can be used to achieve that performance while maintaining
15

---

Published as a conference paper at ICLR 2025
Method
FPSt
Memory(GBB
Transformer
3.7
9.8
Spatial Cross Mamba
4.7
2.2
Table 12: Both models above use 3 layers (cross attention, self attention, ffn), act only on the smallest feature map (23x40), use 100x100 BEV grid, with R101 backbone. The FPS is the average number of samples per second processed by the model in evaluation mode an RTX 4090 GPU. All cameras views are collapsed into a single sequence.
its advantages (linear-cost attention with parallel training). In the table T3 compare naïve implementation and our specialized implementation for BEV perception. In this case, the input sequence is relatively short (375 image feature vectors) so hidden state of size 256 < 375 is likely to capture all relevant information in the sequence. Compared to our result using projection, method we see small 0.3% increase in NDS performance while having per block increase in GFLOPs of 461 %, per block increase in parameters of 98%, and a total memory increase of 10%. The stark difference efficiency shows the benefit of our work to efficient applications of Mamba for cross attention in BEV and offers direction for future applications to other areas.
Method
NDS1
mAPt
Params(K) 4
GFLOPs (K) (Spatial Cross Mamba)
Memory(MB) (Total)
(Spatial Cross Mamba)
Append h=32
0.3051
0.2314
240
2.6
438
Append h=256
0.3138
0.2371
476
14.6
480
Project h=32
0.3128
0.2362
240
2.6
438
Table 13: Experiments used single temporal frame and were trained for 12 epochs and used ResNet50 backbone pre-trained on the COCO object detection dataset.
A.3 ALGORITHMS
The Pseudocode our proposed Spatial Cross Mamba shows Algorithm [ Cross Quasi-Separable State Space Model (XQSSM) show in Algorithm
The details of the
Algorithm 1 Spatial Cross Mamba
Input: 9: (B, Q, D). v: (B, C, (B (B, (,,2,
Output: y: (B, Q D)
1: A: (H,) 6- Parameter
2: b: C. Q, Z) < 0-5 1rri
Mask invalid reference points b =ZQ
3: (1, sum(b)
4: Q=ZBC (B, Q, 2aD+4NG) + MaskedL.near,, di)
5: Vznca (B, C, V, 4D++ < Maskedl (V,
6: VZBCu : (BCV, aD) + flatten( V VCca [0.1,2]
7: Qz (B, Q aD), QUBC: (B, Q. aD+4NG) + split(Q QC)
8: QuBCa: (B, Q, aD+4NG+2H) + concat([QuBe. repeat(O, 2H))
9: xBCdt (M+BCV, aD+4NG+2H), extract (M), Smask (M+BCV)
VZBCat b, ye : (M+BCV, aD) BCdlM A)
+
11: Vgaedd (M, aD) + RMSGated (/o [8mas) Q2)
Qy (B. Q, aD)
indexAdd (Q. extract)
13: count (B, Q) + clamp(sum/ [1. 3]) 1.0)
Qayg (B, Q aD) <- Qu//ountt
15: Q
(B, Q, D) <- FusddNomDrppRssddaa(iiaannn (Qavg),
16

---

Published as a conference paper at ICLR 2025
Algorithm 2 Cross Quasi-Separable State Space Model (XQSSM) Recurrent Form
Input: (2, M+BCV, aD), B (2, M+BCV, N), C': (2, M+BCV, N), A:HH), dl:(, M+BCV, H),
8mask (2, M+BCV)
Output: y (M, oD)
1: digias (2H) < Parameter
2: A: (2, BCV, H) < SoftPlus(dtH) [ pita) mmask 1 where Zij is image feature vector
3: dA (2, BCV, H) t- exp(A/ A [8mack]]
4: dBx (2, BCV, aND) A BB 8mask
5: h: (2, H, aD//H, N) + 0
6: for 0,..., do
Forward and backward scans
7: idx + 0
8: for == O.... M+BCC do
9:
if sij then
h4 f- dAight + dBwig
D Only update state if Tay E V
11:
else
12:
yi + Ciphs
b Only compute output if my E Q
13:
idx + idx + 1
14:
end
15:
end for
16:
end for
A.4 VISUALIZATION
As shown in Figure 7 we compare MamBEV-Small with BEVFormer-Bar MamBEV-Small successfully detects cars occluded by obstructions with a higher degree of accuracy compared to BEVFormer-Base The predictions MamBEV-Small exhibit a closer alignment with the ground truth in terms of both spatial positioning and bounding box dimensions. We also provide more detection results in Figure 8 9 where our model can successfully detect fully occluded vehicles as well as small and distant objects.
17

---

Published as a conference paper at ICLR 2025
USAA ToP partonoral
Preittin Groadd Tuth
Figure Comparison of MamBEV Small and BEVFormer Bass on nuScenes val set. We observe that model can detect highly occluded cars with higher degree of accuracy compared BEVFormer- Base.
Predetton Ground Toch
HI
Figure Comparison of MamBEV Small and BEVFormer Coamer-Baseeeee on nuScenes val set. We observe that model successfully detect objects missed in the prediction results of BEVFormer-Baar
18

---

Published as a conference paper at ICLR 2025
NONT
CAN PICTT
OMd HOUP RIAP
Prediction
Ground Truns
pa
Figure 8: Visualization results of MamBEV-Sman on nuScenes val set. We observe that our model can detect highly occluded objects.
Si PANTAET
TANT NCAT
Prediction Ground truth
Figure 9: Visualization results of MamBEV-Smal on nuScenes val set. We can see that our model can detect objects that are tiny and far away.
19


Reviews:
[{"review_id": "EJd8mhDiPn", "reviewer": "Reviewer_wMH8", "summary": "The main content of the paper is focused on enabling state space models (SSMs) to learn birds-eye-view (BEV) representations, specifically in the context of 3D representation learning tasks. \n\nThe authors propose a novel method called MAMBEV, which incorporates SSMs and linear attention to address the challenges of capturing temporal and spatial relationships in multiview video and fusing distinct visual representations. \n\nThey conduct experiments using the nuScenes dataset and achieve results comparable to the state of the art.", "questions": "Please refer to section Weaknesses.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This paper presents a method for applying Mamba to BEV detection.", "weaknesses": "1. **Writing Issues:**  \nThe paper has several writing issues:  \n* Many sentences are lengthy, which hinders comprehension. For example, lines 188, 191, and 201.  \n* Certain sections lack citations, such as on lines 206 and 207.  \n* Some areas seem to be missing punctuation, possibly in line 225.  \n\n2. **Experimental Results:**  \nThe overall improvement in the experimental results is quite limited, and the value of applying SSM to BEV detection is not clear.  \n\n3. **Comparison with Related Work:**  \nAlthough the authors argue that it is unnecessary to compare with some long-sequence works, I believe it would be best to provide such comparisons, especially given that the authors conducted multi-frame ablation studies. Including comparisons with related work, such as VideoBEV, would strengthen the paper’s claims.  \n\n4. **Efficiency Metrics:**  \nThe paper mentions reducing computational load and improving efficiency. The proposed method should provide relevant metrics related to training and testing, such as training time and model latency during testing. While the authors provide some data, these metrics do not demonstrate any significant advantage over more established methods like deformable attention. This lack of practical efficiency undermines the paper’s claims.  \n\n5. **Test Dataset Results:**  \nThe authors have not provided results on the test dataset. Their explanation for this omission is unconvincing and raises questions about the generalizability of the proposed method.  \n\n6. **Summary:**  \nOverall, this paper discusses the application of Mamba in BEV detection. However, from the perspective of the detection task, the introduction of Mamba does not demonstrate any practical significance. Combined with the above points, I believe the paper requires more comprehensive experiments to substantiate its contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fIhIgAjw4N", "reviewer": "Reviewer_knmh", "summary": "The paper presents MamBEV, a novel 3D object detection framework designed to improve BEV-based perception systems, particularly for autonomous driving. By introducing state-spatial separation and multi-axis multiview projection, MamBEV addresses the challenges in handling multi-camera BEV representation. The proposed method achieves promising results in complex driving scenarios, showcasing improvements in detection accuracy and computational efficiency.", "questions": "1. An important reason for using Mamba is to achieve lower computational cost; however, the paper only provides theoretical calculations without experimental validation, which is a crucial aspect.\n2. In the bottom experiments of Table 1, why do BEVFormerV1-Small and BEVFormerV1-Base use different frames?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Lower computational cost.\n2. Good performance, comparable to VideoBEV.", "weaknesses": "1.The paper could have provided more ablation studies to dissect the contributions of individual components of MamBEV.\n2. Limited discussion on potential limitations of the framework, such as scalability or adaptability to other domains beyond autonomous driving.\n3. limit novelty. The BEVFormer architecture introduces Mamba.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kZ6NSwOZdQ", "reviewer": "Reviewer_yMTq", "summary": "The paper proposes a new approach to building bird's-eye-view (BEV) maps from multiple cameras. With an aim of mobile deployment, the paper investigates approaches that depart from the existing transformer-based push (from images to 3D) or pull (from 3d to images) methods. In particular, recent work on state space models (SSM) is an inspiration to replace transformers and their quadratic attention computational complexity. Of particular interest here is to leverage ideas from the Mamba-2 and Hydra SSMs and adapt them to the BEV problem. The new SSM-based BEV model is well described and well tested, including several ablation studies.\n\nTwo potential caveats noted below are that only one dataset is used (as seems to be the case with other SOTA approaches, so likely not the authors' fault), and the improvements over SOTA are small and sometimes mixed depending on which metric one looks at.", "questions": "- only tested on nuScenes (like some of the other SOTA models, including DETR3D and BEVformerV2). Is there another suitable dataset that this could be tested on to better demonstrate generality?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- significant problem domain where much improvement remains possible\n\n- very well explained technical approach\n\n- great to see asymptotic complexity analysis of the various stages of the approach\n\n- convincing evaluation on the nuScenes dataset\n\n- experiments on increasing temporal information, SSM vs deformable attention, feature normalization, scaling, and feature traversal order are interesting and convincing.", "weaknesses": "- table 1, the proposed model does not outperform SOTA in all metrics. This table needs to be described better. If NDS is the main metric, then what about Focal-DETR? Should explain more clearly which table rows are compared in the main text lines 395-396\n\n- should include a figure with reconstruction samples, for qualitative evaluation by the reader.\n\n- lines 221-222: nonsense sentence", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WNaVxl9QRf", "reviewer": "Reviewer_4oR6", "summary": "The paper introduces the Mamba mechanism into 3D visual perception task to learn bird's-eye view (BEV) representations in order to address computational and memory efficiency problems, and designs a Spatial Cross Attention module for the task. The effectiveness of the design is demonstrated through extensive experiments and ablation studies, which would bring benefits to application scenarios such as autonomous driving and driver assistance systems.", "questions": "The main issues have been raised in Weaknesses section.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. MamBEV proposes a design based on SSM, which exceeds the performance of existing Transformer-based structures in several metrics.\n2. MamBEV provides sufficient experimental results to illustrate the effectiveness of the design.", "weaknesses": "1. Lack of visualization of results. I am quite sorry to say that I am not familiar with the nuScenes dataset, so could the authors provide some visual examples to show the advantages of MamBEV?\n2. Lack of more detailed description of the effectiveness of the Spatial Cross Attention module. Since the authors mention in the Introduction that \"we further demonstrate that our method can better capture longer dependencies in multiview video\", whether this word \"better\" means the module proposed in the paper comparing to other SSM Attention designs, or Mamba-2 comparing to Mamba-1, or Mamba comparing to Transformer, could it be supported and presented through other ways in addition to the accuracy results on the specific dataset?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uY2mUAZ3bK", "reviewer": "Reviewer_yyMe", "summary": "The paper presents MamBEV, a novel framework designed for 3D visual perception tasks like 3D detection from multi-camera images, which are crucial for autonomous driving and assistance systems. The framework leverages a Mamba-based approach to learn unified Bird’s Eye View (BEV) representations using linear spatio-temporal SSM-based attention, which improves computational and memory efficiency across multiple 3D perception tasks. The key is SSM-based cross-attention, which functions similarly to standard cross-attention, enabling BEV query representations to interact effectively with relevant image features. Experiments are conducted to demonstrate the effectiveness of the method. \n\nPos:\n- Enabling Mamba to BEV representation seems to be interesting.\n\nCons:\n- The authors claim that proposed method supports multiple perceptual tasks but has no proof.\n- The article keeps emphasizing computational efficiency, but without proofs.\n- The core figures are crude and difficult to understand\n- Insufficient experimentation and comparison methods", "questions": "Please refer to the above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- Enabling Mamba to BEV representation seems to be interesting.", "weaknesses": "Cons:\n- The authors claim that proposed method supports multiple perceptual tasks but has no proof.\n- The article keeps emphasizing computational efficiency, but without proofs.\n- The core figures are crude and difficult to understand\n- Insufficient experimentation and comparison methods", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)