HotBEV: Hardware-oriented Transformer-based Multi-View 3D Detector for BEV Perception
Authors: Peiyan Dong
Zhenglun Kong
Xin Meng
Pinrui Yu
Yifan Gong
Geng Yuan
Hao Tang
Yanzhi Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: Multi-view 3D detection
Hardware efficiency
Autonomous driving

Abstract: The bird's-eye-view (BEV) perception plays a critical role in autonomous driving systems, involving the accurate and efficient detection and tracking of objects from a top-down perspective. To achieve real-time decision-making in self-driving scenarios, low-latency computation is essential. While recent approaches to BEV detection have focused on improving detection precision using Lift-Splat-Shoot (LSS)-based or transformer-based schemas, the substantial computational and memory burden of these approaches increases the risk of system crashes when multiple on-vehicle tasks run simultaneously. Unfortunately, there is a dearth of literature on efficient BEV detector paradigms, let alone achieving realistic speedups.
Unlike existing works that focus on reducing computation costs, this paper focuses on developing an efficient model design that prioritizes actual on-device latency.
To achieve this goal, we propose a latency-aware design methodology that considers key hardware properties, such as memory access cost and degree of parallelism.
Given the prevalence of GPUs as the main computation platform for autonomous driving systems, we develop a theoretical latency prediction model and introduce efficient building operators.
By leveraging these operators and following an effective local-to-global visual modeling process, we propose a hardware-oriented backbone that is also optimized for strong feature capturing and fusing.
Using these insights, we present a new hardware-oriented framework for efficient yet accurate camera-view BEV detectors.
Experiments show that HotBEV achieves a 2\%$\sim$23\% NDS gain, and 2\%$\sim$7.8\% mAP gain with a 1.1$\times$$\sim$3.4$\times$ speedups compared to existing works on V100;
On multiple GPU devices such as GPU GTX 2080 and the low-end GTX 1080, HotBEV achieves 1.1$\times$$\sim$6.3$\times$ faster than others.

HotBEV: Hardware-oriented Transformer-based Multi-View 3D Detector for BEV Perception
Peiyan Dong' Zhenglun Kong',, Xin Meng',, Pinrui Yu', Yifan Gong', Geng Yuan?,
Hao Tang" Yanzhi Wang'
Northeastern University, University of Georgia, CCrnegie Mellon University (dong-pe, kong. yanz. wang/onortheastern.cc edu
Abstract
The bird's-eyeee s-eye-vieww (BEV) perception plays critical role in autonomous driving systems, involving the accurate and efficient detection and tracking of objects from top-down perspective. To achieve real-time decision-making in self-driving scenarios, low-latency computation essential. While recent approaches to BEV detection have focused improving detection precision using Lift-Splat- Shoot (LSS)- or transformer-bas based schemas, the substantial computational and mem- ory burden of these approaches increases the risk of system crashes when multiple on-vehicle tasks run simultaneously. Unfortunately, there a dearth of literature efficient BEV detector paradigms, let alone achieving realistic speedups. Unlike existing works that focus on reducing computation costs, this paper focuses on developing an efficient model design that prioritizes actual on-device latency. To achieve this goal, we propose latency-aware design methodology that considers key hardware properties, such memory access and degree of parallelism. Given the prevalence of GPUs as the main computation platform for autonomous driving systems, we develop theoretical latency prediction model and introduce efficient building operators. By leveraging these operators and following an ef- fective gocl-to-glooba visual modeling process, we propose hardware-oriented backbone that is also optimized for strong feature capturing and fusing. Using these insights, we present a new hardware-oriented framework for efficient yet accurate camera-view BEV detectors. Experiments show that HotBEV achieves a 2%%~23% NDS gain, and 2%.~7.8% mAP gain with a 1. x3,4% speedups compared to existing works on V100; On multiple GPU devices such as GPU GTX 2080 and low-end GTX 1080, HotBEV achieves 1. x.3x faster than others. The code is available at HotBEV
Introduction
Recently, there has been growing interest 3D object detection (also known as BEV perception tasks) from multi-camera images, especially in the context of autonomous driving. While LiDAR- based methods have made remarkable progress, camera-only-batt baled approaches have gained extensive attention due to their lower expenses and ability to identify volor-baseds road elements such as traffic lights and signs [Α]. Mainstream camera-only-Bea oouera--ol--baseed approaches for multi-camera 3D object detection tasks can be broadly classified into two research domains: 1) 3D reconstruction detectors, which project 2D features from the image view into the bird's-eye view (BEV) to extract intrinsic and extrinsic information 2 3
7 However, these approaches suffer from degradation due to the inaccuracy of the depth information predicted from 2D features. 2) 3D projection detectors,
Equaa Contribution Corresponding Author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

STU
HotBEY Design
CNN-based
Transformer--used
HoLDEV
(a)
(b)
Figure (a) Comparison with previous backbone methods in real-world object detection. Hot-
BEV captures the global/local information better than others. (b) HotBEV aims to search for specialized model designs for target GPU devices.
which encode information such as 3D position and object queries into 2D features and then sample
corresponding features for end-to-end 3D bounding box prediction [8. E D. These approaches overcome the problem of inaccuracies in the BEV features caused by the prediction of depth values or distributions. In particular, 3D projection detectors are considered one of the dominant future multi-view BEV perception techniques.
Despite the impressive performance exhibited by 3D projection detectors, their on-device in- ference speed is limited by various factors. Detectors employing CNN-based backbone of- ten demonstrate inferior detection performance compared to those with Transformrr-baser backbone ITC. Additionally, transformers are known to possess computation and memory complexities [Л. which obstruct scalability, particularly when deploying them on resource- limited devices. ii) Many prior works 7 prioritized efficient model design by only rely- ing on hardware-age agnoosti metrics like compu- tation FLOPs, neglecting account for the ac- tual hardware's 's performance, such as the actual inference latency. Current detectors lack opti- mizations for specific hardware deployments, including factors like memory access cost, de- gree of parallelism, and compiler characteristics, all of which significantly impact hardware throughput during inference. The quadratic memory com- plexity associated with original self-attention operations further exacerbates the disparity between theoretical FLOPs and actual hardware performance. Notably, most current in-vehicle chips adopt the GPU architecture, which poses greater challenges and requires further investigation despite its essential role. iii) To bridge the gap between the theoretical and practical efficiency of deep models, researchers have started considering the real latency in the network design phase. Hardware profil- ing 12 [3], which is generally utilized to estimate on-device speed, is dedicated and time-consuming. And profiling should be redone once the model structure, size, and hardware type change. Thousands of measurements are required for each operation to ensure data correctness. In contrast, there a need for practical, plug-and-pll model capable of efficiently estimating inference latency, specifically designed for general GPU architectures.
0.60
0.55
HOISEV (0.427 MAP)
HotBEV (0.407 NAP!
sEvformer (0-416 MAP)
PotarDETR (0.353 MAP
HotBE 385
BEVDet (0.393
NDS
0.45
PETM (U. mA
Focal-it (8.39 HAP)
DETR3D MAP)
0.40
0.35
PGD (0. 409 mAP)
FCO530
FPS
Figure 2: The trade-off between performance (mAP/NDS) and hardware efficiency (FPS) for different methods on the nuScenes test dataset. HotBEV outperforms SOTA efficient transformer- based detectors on both precision and efficiency. Models are evaluated on NVIDIA V100.
In this paper, we propose pardware-orientedr transformer-basee framework (HotBEV) for camera- only 3D detection tasks, which achieves both higher detection precision and remarkable speedup across both high-end GPUs and low-end GPUs (see Figure II. Firstly, we propose a theoretical latency prediction model by considering the algorithm, the scheduling strategy, and the hardware properties. Given a target GPU, we directly use the latency, rather than the computation FL.OPs, to guide our algorithm design. Then we perform latency breakdown of major modules classic camera-only detectors and figure out that the backbone is usually the speed bottleneck. After benchmarking

---

Canera-view Deleciors V100
T1
200
11
53
04
48
150
64
Petr
Pots2
DETISD
Besihel
Beclomer
BoNo55
Swii-T
Del-S
BOB masellB Tex
RooNet00
SwitsT
Dell-S
Rob samitiol Tiny
Total
Dicader
Pestion: Contehoatt
Teall
Poic
Eines
RepUNN
New
Olhers
Toader
Fiades
Bibhsiing
Figure 3: Device speed breakdown. Left: Detectors (Input resolution: 3x256x704 for PETRv2, others 3x900x I 6000). Middle and Left: Results are obtained on NVIDIA V100, GeForce GTX 2080 Ti, and GeForce GTX 1080 Ti. The on-device speed for frequently used backbone and various operators reported. (Input resolution: 3x224x224) standard backbones on high-end (V100) and low-end (GTX 1080 Ti) GPUs, we propose efficient operators and fusion techniques for model on-device implementation. Based these operators and the process of vision modeling, we design a hardware-orientee backbone with strong feature enhancement by information interaction between model stages. Then extend the latency-aware design methodology to other parts, such as image embedding and decoder, and propose the basic design paradigm of HotBEV. Finally, guided by the latency prediction model, generate the family of HotBEV through a standard search algorithm. Experiments show that our proposed framework can achieve 23% NDS gain and 2%-%7.8% mAP with 1.1 x x speedups compared existing works on V100 (see Figure D On multiple GPU devices such as GPU GTX 2080 HotBEV can reach 1.1x~663x faster than other models; for the low-end GTX 1080, our framework can achieve 1.1 x 44.9 x faster than others. Overall, our contributions are summarized as follows:
The latency breakdown of representative Camera-on -onl
We propose lawre design methodology for 3D object detection based BEV perception.
We propose a hardware-orientee backbone that excels at capturing and fusing features. According to analysis of efficient operators and strong feature modeling, we propose new hardware-oriented framework for the BEV detector.
We conduct experiments showcase the superior inference accuracy of HotBEV compared SOTA efficient BEV detectors while also highlighting its significant hardware efficiency.
2
Related Work
Current multi-view 3D object detectors be divided into two schemas: LSS-based schema and transformer- based schema.
LSS-based schema. BEVDet 2 is the first study that combines LSS and LiDAR detection head which uses LSS extract BEV feature and LiDAR detection head to propose bounding boxes. By acquires the ability of velocity prediction. For the above works, the models are complex, so many hardware platforms do not support some inside operators.
introducing previous frames, BEVDet4D [5]
Transformer-ber based schema. BEVDepth 2 uses LiDAR to generate depth GT for supervision and encodes camera intrinsic and extrinsic parameters enhance the model's ability of depth detection. DETR3D 9 extends DETR [14] into 3D space, using transformer to generate 3D bounding boxes. Following the DETR3D, PETR converts 3D coordinates into 3D position embedding to perceive the 3D scene. And BEVFormer images and cross attention to link the characteristics between different frames for velocity prediction. Although better performance is achieved through these works, the computation cost (e.g.. over 170 GFLOPs) or realistic speed has yet to be optimized. We are the first to work on mansformer-basee 3D detectors to explore comparable performance and fast inference speed diverse hardware platforms while maintaining acceptable detection accuracy.
|| uses deformable transformer 15 to extract features from
Hardware-aware network design. Several existing works have addressed the issue of realistic latency during the network design phase, exploring two distinct directions. Firstly, some researchers evaluate speed directly on targeted devices and derive guidelines for developing hand-designed effi- cient models, as demonstrated by [T6] Secondly, others employ the technique neural architecture search (NAS) to search for fast models, exemplified by [17]. Nevertheless, conducting speed tests on various hardware platforms for our proposed structures can be exceedingly labor-intensive due to the extensive range of candidates and their corresponding properties. Additionally, obtaining precise
3

---

results for small-scale structures on low-end devices presents its own set of challenges. In contrast, our latency prediction model directly estimates the inference speed on specific computing platforms, circumventing these difficulties.
3
Methodology
In this section, we present our approach to efficiently and accurately predict the on-device latency of an architecture on the target device using the latency prediction model. Additionally, we detail efficient operators and fusion implementation techniques. According to the process of vision modeling, we introduce an efficient embedding module and a hardware-oriented backbone that excels capturing and fusing features. Then our proposed basic design paradigm for HotBEV is based on these efficient operators, the process of vision modeling, and the detection flow of DETR3D 1. Finally, guided by the latency prediction model, we generate family of models known as HotBEV through a standard search algorithm [ Note that we also adapt the Temporal Aligned Module (TAM) in 19 to efficiently improve performance and robustness.
3.1 Latency-aware Design
3.1.1 Lateney Prediction Model
We introduce a novel latency prediction model, denoted by E, which enables direct prediction of the latency of runtime design choices on any target GPU. This allows for the efficient identifi- cation of optimal model settings on correspond- ing platforms. Specifically, the latency predictor E considers the hardware properties H, layer type T, channel dimension C, and input gran- ularity G as input for each design choice and outputs the predicted latency of the block as
Computation
Units
Momory
On-chip Memory CLC Buffer)
Oti-chip
Computalion
PE Memory
Memory
Units
(GDDR/
DDRHHBM
Computation
Units
PE Meroory
Figure 4: Hardware Mödeling, The memory sys- tem includes 1) off-chip memory, 2) on-chip global memory, and 3) memory in the PEs.
I = E,H, T, C, G).
Hardware model. We model hardware device as multiple processing engines (PEs), allowing for parallel computation with varying degrees of parallelism. As illustrated in Figure we represent the memory system using three-level structure (20 211,, which includes: 1) off-chip memory. 2) on-chip global memory, and 3) memory in the PEs. This hardware model enables accurate prediction of the latency of data movement and computation.
Latency prediction modeling. It includes three steps: 1) Input/output shape definition. The initial step is to calculate the input and output shapes. 2) Operation mapping to hardware. Based our hardware model, we first divide the output feature map into multiple tiles and assign each tile to separate PE for parallel processing. 3) Latency estimation. We evaluate the latency of each tile, which comprises both data movement and computation latency: (=laato commpute. For ddttas we leverage our hardware model (see Figure 4l and compute the sum of input and output data movement latencies as 1data=4nn +oout- These latencies are estimated based on hardware bandwidth and input granularity G (equivalent to resolution scale). We assume that each PE moves the required input feature maps and weights just once to compute an output tile. For leomputes we use the maximum throughput of FP32 computation total computation latency can be determined by considering the number of tiles and PEs. We test three types of hardware devices, NVIDIA V100, NVIDIA GTX 2080, and NVIDIA GTX 1080. For a more detailed description of our latency prediction model, please refer to Appendix A
PEs and the FL.OPs required compute a single output tile. The
3.1.2 Efficient Operators and Fusion Techniques of Model Implementation
The development of efficient network architectures for essource-limited devices has greatly benefited from reduced parameters and floating-poin operations (FLOPs) and improved accuracy. However, conventional efficiency metrics, such as FLOPS, overlook memory cost and degree of parallelism. In
this study, we aim to improve network runtime speed and detection performance by identifying and modifying building blocks that are not hardware-friendle To achieve this, we first perform a latency
breakdown of major modules in classic camera-only detectors, DETR3D, PETR g PETRv2 [9

---

BEVDeT E. and BEVFormer M. which reveals that the backbone is usually the speed bottleneck (66%-~78%) Therefore, we deploy common backbones on high-end (V (V100) and low-end (GTX 1080 Ti) GPUs and benchmark their speed, as illustrated in Figure 3 We then introduce efficient operators and fusion techniques to improve the speed and detection performance of these networks.
Convolutional modulation for efficient global modeling, Instead of calculating the similarity score matrix (attention matrix (2 ), we simplify self-attention by modulating the value V with convolutional features as Figure c). Our approach uses convolutional modulation rather than self- attention to build relationships since they are more memory-effie -efooreefficint,, particularly when processing high-resolution images. Due to the modulation operation, our method differs from traditional residual blocks and can adapt to the input content.
Fusing BatchNorm into the preceding fully-connected layer. After analyzing various backbones as shown in Figure 3 found that LayerNorm (L.N) accounts for approximately 10% to 15% of the network's s total latency. Dynamic normalization techniques like LN gather running statistics during inference, resulting in delays that impact speed. On the other hand, BatchNorm (BN) is more memory and computation-efficient, as it is fused with the preceding fully-connected layer. reducing data movement and computation load (see Figure 4. To this end, we modified the WMSA/SWMSA in Swin 100 by replacing LN-Linear with Linear-BN, which we refer to as /SWMS By switching to a based design, we achieved a 13% to 15% speedup with no significant loss in accuracy (less than 0.3%). Based on the accuracy analysis in Appendix A one advantage of the window-based self-attention on the efficient implementation compared to other transformer architectures: It can replace the LayerNorm with BatchNorm and then perform the fusing tech- nique with acceptable accuracy. And replacing the LayerNorm directly with the BatchNorm the original self-attention is difficult since the model training cannot converge. Therefore, we consider WMSAA,, /SMMSAA as our design candidates.
Fusing multiple branches into one single branch in reparameterized CNNs. Multi-branch structures come with increased data movement cost, the activation values of each branch are saved into PE memory or on-chip memory (if the PE memory is insufficient) to compute the subsequent tensor the graph. Additionally, the synchronization cost arising from multiple branches impacts the overall runtime [ To address these challenges, we use RepCNN [24 as network component, which fuses multiple branches into more single-branch substructures during inference. This approach enables even distribution of computation among multiple PEs, preventing imbalanced computation overheads associated with multiple branches. The resulting operator fusion improves memory access and parallel computation on multiple PEs.
3.2 Architecture Design
We present HotBEV. a hardware-orienter framework for multi-view 3D detection. Given a set of [=(1' E R3xH>W ,i == 1, NJ} our framework leverages a hardware-orienter backbone (HOB) to extract 2D multi-view features, =²2=(F?" E RRCHH, i generator generates 3D coordinates, which are aligned with the coordinate system of the frame I using the temporal aligned module (TAM) concerning the previous frame L - 1. Next, the 2D features and 3D coordinates from adjacent frames are concatenated and fed into the 3D position encoder to obtain 3D-aware features. Our hardware--rriened decoder uses these features as keys and values, which interacts with detection queries initialized from standard learnable 3D anchor points with small MLP network. Finally, the updated queries are input to the detection head for the final prediction. For coordinates generator and position encoder, please refer [
I................ I.... NN.. The coordinates
3.2.1 Hardware-Oriented Backbone with Strong Feature Enhancement
Hardware-oriented backbone. Based on our GPU breakdown B we found that the backbone consistently caused the most latency. To maintain detection precision while removing this speed bottleneck, we propose powerful transformer encoder for feature capturing and fusing, as shown in the backbone design in Figure
5 We divide the backbone into four stages S, following the granularity of data flow in ResNet (25. Since features scale from the local to the global visual receptive field, we introduce the HOB block design as shown in Figure aa.). In each HOB block, we use several consecutive local-wise attention mechanisms to extract local information (texture-level semantics), followed by global attention to enhance global information (abstract-level semantics) the feature map. Furthermore, we insert aamantic-augmenter module (consisting of an upsampling layer
5

---

(a) Overall Backbone
Stage
Stage
Stage
Stage
Malti-view images
#
16
16
H...R.
1897
Imaage
Emsicaling
1000
D00
100
Semsniie-A Augmeniog Module
SemetticAAggaattiee Module
Samanix-Augnanne
Module
(b) HOB Block
(e) Convolation
Decoder
RepCNNN
closs
bbox
Linerr
10001
Patch Meqsing
Hadamord Product
FC
Global Artention
2D Features
wise Attention
SWSSS
Seatch ScathhTfema
DCCon
Nx x
Query Generair
Rel.I
PC
Convolational modulation
x vt
3D Position-
Linear
Lincer
aware Fentures
Global Attomtion
Q.K
Self
Object
Query
Figure 5: The design of our proposed HotBEV. components include that (a) Overall backbone: includes four stages, where feature size scales along the stage. To enhance the semantic information of low-level features, one semantic-augmented module (SAM) is added at the end of each stage.
The workflow follows the DETR3D, and the main
(b) Design of the HOB block, which contains multiple local/global attention layers. (c) The convolutional modulation for efficient global modeling. (d) Hardware-Oriented Decoder.
and global attention) into every two consecutive HOB blocks (except between Stage and 2) to further enhance low-level semanties in the current stage, shown in Figure 12 Notably, unlike other methods such as ICC 25
we simultaneously enhance texture-level and abstract-level information in leverage information interaction between stages. not just between layers in the same stage. By using the efficient operators described in Section 3.1.2 provide the "two-phase design space" (DS) of the HOB backbone:
each stage. To reinforce texture-level semantics,
1oi, E (RepCNN', WMSA SWMSA
DS Loout,,444 E (WMSAi... SWMSA
(I)
1.2 iiglocal E (CoulMo
where local represents the candidates of local-wise attention while global for the global attention; the 1st phase covers Sj, S2. S3 of the backbone, and the 2nd phase for 54; i denotes the jth block.
Image embedding module. Figure 3 reveals that patch embedding is speed bottleneck multiple platforms due to its reliance on non-overlapping convolutional layer with large kernel size and stride. Unfortunately, most compilers and acceleration techniques, such as Winograd, do not support this type of layer well. To address this issue, we propose a faster downsampling method using a convolution stem, which consists of three hardware-efficien xx3 convolutions with stride of 2. To obtain input embeddings To of size G
for an input image ZER xxxx x3, we divide into HxW patches and feed them to the convolution stem:
GXG
x D1 (2 Xx333
3.2.2 Hardware-Oriented Decoder
To alleviate the convergence difficulties in the 3D scene, we initialize set of learnable 3D anchor points with a 0~1 uniform distribution. Then, we input the coordinates of the 3D anchor points into a small MLP network with two linear layers of Sigmoid function in between and generate the initial object query Qp. The decoder is applied to predict the final abstract semantics and generate the bounding box. Our decoder consists of six global attention blocks and is divided into two parts, as shown in FigureSidd The front three layers take object queries as the only input and perform self-attention computation, aiming at separating different objects [4]; The back three layers take object queries as queries and image features as keys and values. As shown in Figure 12 they perform the cross- attention between object queries and image features to extract the content and position of

---

Table 1: Comparison of different methods the nuScenes val set. FPS is tested V100 with INT8.
Method
Backhone
Resolation
Frames
NDS
mAP
mATE
mASE
mADE
mAVE.
mAAE.
FPS
BEVDet
ResNet50
BEVDeH
ResNet50
256 x 704
0.457
0.522
0.703
0.278
0.495
0.354
0.206
25.9
PETRY2
ResNet50
256x 704
2
0.456
0.349
0.7
0.275
0:58
01477
0.187
30.2
BEVDepth
ResNet50
256 704
0.475
0.351
0.639
0.267
0.479
0.428
0.198
24.3
FadREX-MS
ResNer50
256
0.485
0.647
0.282
0.225
SOL OFasion
ResNe150
256 704
16
0.534
0.427
0.567
0.274
0.511
0.252
0.181
20.2
SOLOFusion
ResNe150
256 704
0.494
0.362
0.607
0.304
0,539
0,293
0.19
21.4
HoIRKY
HOB-mano
x 704
0:455
0.35
0.636
0.271
0,449
0.542
0.36
39.2
HotBRY
HOB-liy
256x 704
0.487
0.362
0.628
0.27
0.438
0,412
0.19
31.7
HoLBEV
HOB-buse
256 704
0,506
0.369
0.625
0.264
0,362
0,364
0.153
27.2
BEVDei
ReNNa101-DCN
640 1600
0:472
0.393
0.608
0.259
0,366
0.822
0.391
2.95
ROOS3D
ReNNa101-DDNN
900 1600
0.395
0.572
0.806
0.266
0.511
1315
0.7
2.64
DETR3D
ResNet001DCCN
900 x 1600
0:434
0.349
0.716
0.268
0.379
0.8422
0.2
5.71
RosNa1U1-DCN
0.335
Focal-PETR
ResNe101--DCN
512 x 1408
0.461
0.39
0.678
0.263
0.395
0.804
0,202
10.23
PETR
ResNet101DDCN
512 x 1408
0.441
0.366
0.717
0.267
0,412
0.834
0.19
8.84
BEVFarmer
ResNeti01
900 x 1600
0.517
0.416
0.673
0.274
0.372
0.394
0.198
4.65
PolarDETR
ResNe1001DDNN
900
0:488
0.383
0.707
0.269
0.518
0.196
HoUBEV
BOB nano
512 1108
0.47
0.385
0.648
0.243
0.412
0.715
0.183
12.38
HotBEV
HOB-tiny
512 1408
0.512
0:407
0.634
0.235
0.408
0.632
0.175
11.39
HoIRKY
HOB-hise
512x
0.525
0.427
0.62
0.221
0.163
9:08
the object. For efficient modeling, we leverage the convolutional modulation layer as the decoder component (more details in Appendix A..
To generate temporal-aligned 3D coordinates, we adapted the technique TAM Subsequently, we combine the resulting 3D coordinates with the 2D features and feed them into a 3D position encoder [.. This allows to obtain temporal-aligned 3D-aware features, which improve the model's localization, attitude, speed estimation, and overall robustness.
3.3
Training
Our design space (Eq. (1). also as search space is selection of possible blocks, including RepCNN, WMSA, SWMA (for local-wise attention layer), and convolutional modulation (for global-wise attention layer). We propose simple, fast yet effective gradient-based search algorithm to obtain candidate network that just needs to train the supernet for once. To train our supernet, we adopt the Gumble Softmax sampling to get the importance score for the blocks within each search space/stage. During each step of training, a number of blocks are sampled to obtain subnet structure. The latency of this subnet can be estimated using our latency prediction model.
Supernet design. We use the two-phase design space (DS) as the search space and train the supernet for the HOB backbone. We only search the backbone's 's structures, dimensions C, and input granularity G, while the decoder uses fixed structures with dimensions adapted to the backbone.
Latency-aware model slimming. It has three steps:
1) Train the supernet with the Gumble Softmax sampling 26 to get the importance score for the blocks within each DS.
2) Use latency prediction model E (Section 3.1.1 to estimate the on-device latency of each candidate.
3) Perform lateney- model slimming on the supernet obtained from step 1) by FPS evaluated with predictor E. Specifically, we use the score of each candidates to define the importance score c- ResCNN +e
WMSA
of DS; as
for S1, S2, Ss, and
A for S4. We sum up all the scores of all DS within
SWMEA
that S and deduce the score for each S. Then we define the evolution process (all performed in the current least important S): a) remove the 1st SWMSA b) remove the 1st WMSA; c) reduce the width by multiples of 16. Then we predict the current FPS f and decide by FPS_NDS drop(-% (f)).. This process is repeated until the target throughput is reached.
4 Experiments
4.1 Datasets and Implementation Details
We conduct comprehensive experiments on the nuScenes dataset 277 which contains 1000 driving scenes of 20-second length for each. The scenes are officially split into 700, 150, and 150 scenes for training, validation, and testing. The dataset includes approximately 1.4M camera images.
7

---

Table 2: Comparison for large backbone and resolutions on the nuScenes val set (FPS on V100).
Method
Backbone
Resolution
NDS f
mAP
mATE
mASE
mAOE.
mAVE 1
mAAE
FPS
PETR
ResNec101
1600x>000
0.442
0.37
0.711
0.267
0.383
0.865
0.201
5.7
PETRv2
ResNer101
1600x640
0.524
0.421
0.681
0.267
0.357
0.377
0.186
BEVDer4D
Swin-B
1600x6440
0.515
0.396
0.619
0.26
0.361
0.399
0.189
BEVDepth
ResNer101
512x1408
0.535
0.412
0.565
0.266
0.358
0.331
0.19
2.3
BEVFomeer22
ResNet50
1600x640
0.529
0.423
0.618
0.273
0.413
0,333
0.181
PolatFormer-T
ReaNet101
1600x900
0.528
0:432
0.648
0.27
0.348
0.409
0.201
3.5
Sparso4D
ResNet101-D00
900x1600
0.541
0.436
0.633
0.279
0.363
0.317
0.177
4.3
HotBEV
HOB-base
512 1408
0.525
0.427
0.62
0.221
0.36
0.55
0.163
5.5
Evaluation metrics include mean Average Precision (mAP) and five types of true positive metrics (TP metrics): mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), mean Average Attribute Error (mAAE). We also report nuScenes Detection Score (NDS) to capture all aspects of the nuScenes detection tasks. We follow the training recipe from PETR but mainly report results with 24 training epochs compare with other detection models. Experiments were run on 8 NVIDIA V100.
4.2 Model Accuracy and Speed Performance
Main results. As delineated in Table our model is compared an array of existing camera-based
methodologies. These include FCOS3D 28 DETR3D 2 PGD 29 PETR 8 PETRv2 [T)],
Focal-PETR BDJ, BEVDet S. BEVDet4D 1., BEVFormer M. BEVDepth 2. and POIarDETR
B1I The table lists the backbone type, image resolution, number of frames, inference speed (FPS), and accuracy on the nuScenes validation set for each method. The backbone options include ResNet50 (25) Our streamlined models surpass other methods in both performance score
and ResNet101-DD and inference speed.
In particular, our compact model, HOB-tiny (256 x 704). with 0.487 NDS, 0.362 mAP, and 19.8 FPS, outperforms BEVDet (0.379 NDS, 0.298 mAP, 16.7 FPS), BEVDet4D (0.457 NDS, 0.322 mAP, 16.7 FPS), PETRv2 (0.456 NDS, 0.349 mAP, 18.9 FPS), and BEVDepth (0.475 NDS, 0.351 mAP, 15.7 FPS) with Net5e backbone, achieving a 2.5% 28.5% NDS gain, 3.1% 21.5% mAP gain and 4.8% ~ 26.1% 1% FPS gain. The reductions in the orientation error, velocity error, and attribute error all contribute continues to offer an unrivaled balance of accuracy and speed compared to extant work. For instance, our HOB- Bonoo model (512 1408) achieves 2% NDS and 13.6% FPS increase with 1.3% mAP and score difference when compared to Focal-PETR RReNNet011-DCN, 512 x 1408). Our HOB-tiny model (512 x 1408) surpasses PolarDETR (0.488 NDS, 0.383 mAP, 3.5 FPS) in both speed and accuracy. Our HOB-base model (512 x 1408) also surpasses BEVFormer (0.517 NDS, 0.416 mAP, 3 FPS) in both speed and accuracy. For detailed configurations of our model and detailed analysis of temporal modeling and robustness related to frame length, please refer to Appendix A
enhancing the NDS score. Even with larger input resolution, our model
Our research specifically targets small models, so our results are particularly favorable for these models compared to other studies. For comprehensive understanding, we also present a comparison with baseline models that possess larger backbones and increased input sizes in Table 2 Notably, we surpass baseline models in frames per second (FPS) while maintaining comparable accuracy levels.
Table 3: Comparison with 3D reconstruction-
Comparison with 3D reconstruction-base BEV detectors. Table 3 shows that 3D reconstruction- based BEV detectors are better than our methods by up to 19.8 NDS but with inferior on-device speed than ours by up to 19.4 FPS. Even though the de- tection performance of image-only detectors cannot be superior to Lidar-only methods, these results also motivate us implement our design into one of the commercial mainstream, post-fusion systems to im- prove hardware efficiency. Currently, the pre- pre-iusion family, such as BevFusion, is costly to deploy in practice. The current in-vehicle market is dominated by camera-only solutions, such as Tesla, or post-fusion of camera and lidar signals, such as XPeng Motors, whose camera part still deploys camera-only solutions 32) Hence, our research aims to explore the efficient and effective camera-only 3D detection series for practical implementation. So we choose state-of-te the-the-rt art frameworks, and multi-fram models as the starting point of our research.
based BEV detectors on the nuScenes val set.
Methods
NDS
mAPt
FPS
PointPillars
61.3
52.3
29
SECOND
63
52.6
14.3
CenterPoint
66.8
59.6
12.4
HotBEV-nano HotBEV-tiny HOB-bas base
47
38.5
31.8
51.2
40.7
20.4
52.5
42.7
16.1
8

---

NVIDIA GTX 2080
NVIIMA GTX 1080
HNDEV 12-27
#ac//eme 12.40 mAr
Accunacy (%)
AdarGET (0.333
PoleBETT (0.302yaa))
MODEY To nare
Pepvixt
PECSLPETS 20:35 m
Pocal 10,99 conrs
TE R. 13.00 1AA7T
DEFI3D 34550
640
640
0.5
T50 422 mA//1
1 7120
FPS
FPS
Lateney (us)
Figure 6: Left and middle: The trade-off between performance (NDS) and hardware efficiency (FPS) for different detection methods on the nuScenes val set with different GPUs. Right: Various granularity settings on the HOB backbone of HotBEV.
Moreover, according [R the camera processing modules occupy 33% 42% of the whole latency distribution for the post-fusion system. Our hardware-orientee oriented design with enhanced image feature representation can be leveraged in the camera encoder part of the current in-velicle detection system to improve on-device efficiency.
Performance on multiple GPU devices. To evaluate the hardware throughput, we implement the latency-aware model slimming on two other devices: NVIDIA GTX 2080 Tí which has a 5.5M size L2 cache with 448 Gbps memory bandwidth; NVIDIA GTX 1080 Ti which has a 5.5M size L2 cache with 325 Gbps memory bandwidth; We report the average FPS of over 1000 inferences. As depicted in Figure our approach surpasses current camera-only BEV frameworks in terms of both hardware efficiency and performance. Other methods typically overlook the constraints imposed by limited memory and parallelism in on-device runtimes, leading further degradation in speed performance on devices with limited resources. Without hardware optimization, e.g., int8 quantization, some methods fail to produce results within reasonable timeframe. In contrast, our models strike the optimal balance between speed and performance, making them the superior choice among existing approaches. For example, (0.393 mAP); our HotBEV (0.407 mAP) P) is 2.6x faster than BEVformer (0.416 mAP); our HotBEV (0.427 mAP) is 1.4x faster than PolarDETR (0.383 mAP).
GTX 2080, our HotBEV (0.385 mAP) is 4.5x faster than BEVDet
The speed-up effect superior to that Table T demonstrating our framework's s enhanced GPU generalization ability compared to other approaches. It's worth noting that logic extends beyond autonomous driving. Firstly, our backbone is designed to cater to general vision tasks. Secondly, the hardware model is specifically optimized for GeMM. In the appendix, showcase our implementa- tion on diverse general vision tasks such as classification and 2D detection.
Performance on multiple commercial Orin. It is necessary to test the speed on an actual commercial chip. As shown in Table g we test our HotBEV models on Orin to validate our framework. Before testing, we quantized our model into INT8 with a tensorRT engine. And then run the test 50 times for each model to obtain stable results.
Table 4: Our proposed results on Orin.
Methods
Backhone
Resolation
NDS;
mAP;
FPS
HoLBEV HaBEV
HOB-sano
512x1408
0.47
0.385
31.8
HOB-tin diny
512x1408
0.512
0.407
HoBEV
HOB-base
512x1408
0.525
0.427
16.1
4.3 Ablation Study
Table 5: Ablation study of HOB.
In this section, we conduct the ablations with HotBE trained 24 epochs. The backbone is pre-trained on ImageNet dataset (3 and trained on Nuscenes. The input image size is 256x 704, and the number of detection queries is set to 900.
Index
GA
SAM
NDS
mAP
FPS
0.396
0.328
21.3
@
0.443
0.337
21.4
@
GA
0.447
0.340
23.7
0.455
0.350
24.5
4.3.1 Analysis of Components in HotBEV
Major components of HOB. Table 5 studies how global attention (GA) and cemantic-augmentt module (SAM) contribute to HOB performance. We only modify the backbone network without disabling the 3D position encoder, TAM, and decoder modules. o and @ show that inserting our GA after the local-wise attention can improve 0.9% mAP and 4.7 4.7% NDS without significant impact on speed. @ and @ show that inserting one GA block costs 2.3 FPS yet only gains 0.3% mAP. Once we replace the GA with the SAM, mAP is increased from 34% to 35% while NDS is increased from 44.7% to 45.5%. So SAM can enhance performance and be better than simple global modeling.

---

4.3.2 Analysis of Prediction Modeling
Empirical validation. To evaluate the prediction model, we varied the input granularity G and used the 1-st block in Swin-T as a case study. Figure compares our predictions and the actual testing latency on two different GPUs, the NVIDIA V V100 and GTX 1080 Ti. The results demonstrate that the prediction model can accurately estimate the actual latency across a wide range of input granularities.
More granularities settings. We test various granularity settings on the HOB backbone of HouBEV-nano exam-
216
HVID V100
ine the effects of G. The results on the Tesla-V Tesla-V100 1100 GPU are presented in the left sub-figure of Figure O
NVVDAA GTX 3002
40
We increase
G' from 2 and 64, which consistently improves the realis- tic efficiency of both the NVIDIA V100 and GTX 1080 GPU. It can be found that the finest granularity (G=2) causes substantial inefficiency despite the mAP improve- ment. Otherwise, the coarsest granularity (G-64) benefits the speedup with large detection precision degradation. specific, all networks are transformed into fixed matrix operations on GPU platforms (General Matrix Multiply, GcMM). The latency prediction model evaluates the speed of matrix multiplication and the corresponding data move- ment, so it is applicable to general networks. For example, this modeling can be generalized in other networks as the prediction results on the 1st block in Re in Appendix A
18
Latency
10
Gramularity
Figure 7: Latency prediction results. Results are tested on NVIDIA V100 and GTX 1080 Ti.
More discussion on latency predictor. Our proposed latency predictor provides some opportunities.
(1) Efficient model generation, which also proceeds with Al democratization. As the benchmarking- based approach needs one-day training, our proposed theoretical latency predictor is training-free For example, the benchmarking- baase approach requires 5 days to generate the dataset of 5 different devices if 5 target models are demanded. In contrast, our proposed is off-the-shelf. Our method provides the opportunity for inexpensive and efficient research for users who do not have access to target devices. For instance, when the in-vehicle Orin chip is not accessible, the related efficient model research the Orin chip can still be advanced. In conclusion,
approach makes sense for today's 's rapidly growing demand for autonomous driving. (2) The proposed latency predictor focuses modeling the latency of Matrix Multiplication (MM) with generalizability. Indeed, strong GPU simulators cannot accurately model the behavior of the latest NVIDIA GPUs. However, our purpose is not describe the behavior of GPUs. We want to reflect the relative performance of latency for different layer types and sizes on target GPUs. This is because our search goal is to minimize the relative time in the search space of the current device. For generalizability. our design focuses on latency modeling of MM, the typical computation operation in DNNs, which is mainly impacted by the computing performance of Tensor Core, not other specific operators, so the proposed predictor has generalizability, as shown in Figure 7 of our paper.
5 Conclusions and Limitations
We present a hardware-oriented transformer-basee framework (HotBEV) for camera-only 3D detec- tion tasks, which achieves higher detection precision and remarkable speedup across high-end and low-end GPUs Firstly, we propose a theoretical, plug-an soug-and-ple and-play latency prediction model. Given a tar- get GPU, we directly use the latency to guide our algorithm design. Based on the latency breakdown of classic camera-only detectors, we identify the backbone as the main speed bottleneck. Then, we propose efficient operators and fusion techniques for model on-device implementation. Based on these operators and the process of vision modeling, we design hardware-orienter backbone with strong feature enhancement. Then we propose the basic design paradigm of HotBEV. Finally, guided by the latency prediction model, we generate the family of HotBEV through a standard search algorithm. Experiments show the superior inference accuracy of HotBEV compared to SOTA BEV detectors with significant on-device speed. Notably, while our primary focus lies in the camera-only method for BEV perception, our latency-aware design methodology can also be applied to fusion-based BEV methods, enabling efficient algorithm design that aligns with real-world requirements.
Acknowledgment
This work is supported in part by National Science Foundation CCF-1937500.
10

---

References
[1] Zhiqi Li, Wenhai Wang, Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Qiao Yu, and Jifeng Dai. Bevformer: Learning sidds--eee representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203. 172.722233..7770,, 2022.
[2] Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang. Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection, arXiv preprint arXiv:2206. 10092, 2022.
[3] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In European Conference Computer Vision, pages 194-210. Springer, 2020.
[4] Junjie Huang, Guan Huang, Zheng Zhu, and Dalong Du. Bevdet: High-performance multi- camera 3d object detection in bird-eye-view. arXiv preprint arXiv:2112.11990, 2021.
[5] Junjie Huang and Guan Huang. Bevdet4d: Exploit temporal cues in multi-camera 3d object detection. arXiv preprint arXiv:2203. 17054 2022.
[6] Enze Xie, Zhiding Yu, Daquan Zhou, Jonah Philion, Anima Anandkumar, Sanja Fidler, Ping Luo, and Jose M Alvarez. M 2bev: Multi-camera joint 3d detection and segmentation with unified birds-eye view representation. arXiv preprint arXiv:2204005088 2022.
[7] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang, Cvt: Introducing convolutions vision transformers. arXiv preprint urXiv:2103. 15808, 2021.
[8] Yingfei Liu, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petr: Position embedding transfor- mation for multi-view 3d object detection. arXiv preprint arXiv:2203.05625, 2022.
[9] Yue Wang, Vitor Campagnolo Guizilini, Tianyuan Zhang, Yilun Wang, Hang Zhao, and Justin Solomon. Detr3d: 3d object detection from multi-view images via 3d-to-2d queries. In Conference on Robot Learning, pages 180-191, PMLR, 2022.
[10] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021.
[11] Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Bin Ren, Minghai Qin, Hao Tang, and Yanzhi Wang. Spvit: Enabling faster vision transformers via soft 2021.
token pruming. arXiv preprint
[12] Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural net- work quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543-7552. PMLR, 2019.
[13] Lei Deng, Guoqi Li, Song Han, Luping Shi, and Yuan Xie. Model compression and hardware acceleration for neural networks: A comprehensive survey. Proceedings of the IEEE, 188(4)4485- 532, 2020.
[14] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), pages 213-229. Springer, 2020.
[15] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2020.
[16) Ningning Ma, Xiangyu Zhang, Hai-Tao Zheng, and Jian Sun. Shufflenet v2: Practical guidelines for efficient enn architecture design. Proceedings of the European conference on computer vision (ECCV), pages 116-131, 2018.
11

---

[17] Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard. and Quoc Le. Mnasnet: Platform-ave aware neural architecture search for mobile. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2820-2828, 2019.
[18] Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVI 16, pages 544-560. Springer, 2020.
[19] Yingfei Liu, Junjie Yan, Fan Jia, Shuailin Li, Qi Gao, Tiancai Wang, Xiangyu Zhang, and Jian Sun. Petrv2: A unified framework for 3d perception from multi-camera images. arXiv preprint urXiv:2206.0/256, 2022.
[20] John L Hennessy and David A Patterson. Computer architecture: a quantitative approach. Elsevier, 2011.
[21] Peiyan Dong, Zhenglun Kong, Xin Meng, Peng Zhang, Hao Tang. Yanzhi Wang, and Chih-Hsier Chou. Speeddetr: Speed-aware transformers for end-to-end object detection. 2023,
[22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez. tukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, pages 1998-6000 2017.
[23] Jie Hu, Li Shen, and Gang Sun. gqueeze-and-excitatiinn networks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pultern Recognition (CVPR), pages 7132-7141, 2018.
[24] Xiaohan Ding, Xiangyu Zhang, Ningning Ma, Jungong Han, Guiguang Ding, and Jian Sun. Repvgg: Making vgg-style convnets great again. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13733-13743 2021.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and paltern recognition, pages 770-778, 2016.
[26] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. arXiv preprint 1rXiv:1806.000 1806.09055, 2018
[27] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621-11631. 2020.
[28] Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Fcos3d: Fully convolutional one-stage monocular 3d object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 913-922, 2021.
[29] Tai Wang, ZHU Xinge, Jiangmiao Pang, and Dahua Lin. Probabilistic and geometric depth: Detecting objects in perspective. In Conference on Robot Learning, pages 1475-1485. PMLR, 2022.
[30] Shihao Wang, Xiaohui Jiang, and Ying Li. Focal-petr: Embracing foreground for efficient multi-camera 3d object detection. arXiv preprint arXiv:2212.05505, 2022.
[31] Shaoyu Chen, Xinggang Wang, Tianheng Cheng, Qian Zhang, Chang Huang, and Wenyu Liu. Polar parametrization for vision-based surround-view 3d detection. arXiv preprint arXiv:2206. 10965, 2022.
[32] U.S.News. 10 Cars That Are Almost Self-Driving.
[33] Trung Pham, Mehran Maghoumi, Wanli Jiang, Bala Siva Sashank Jujjavarapu, Mehdi Saj- jadi Xin Liu, Hsuan-Chu Lin, Chen, et al. Nvautonet: Fast and accurate 360o 3d perception for self driving. arXiv preprint 2333 2023.
12

---

[34] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pages 248-255 leee, 2009.
[35] Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5270-52777 2022.
[36] Sachin Mehta and Mohammad Rastegari. Mobilevit: light, general-purpose, and mobile- friendly vision transformer. arXiv preprini arXiv:2110.02177 2021.
[37] Qibin Hou, Cheng-Ze Ze Lu, Ming-Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer-style convnet for visual recognition. arXiv preprint arXiv:2211.11943, 2022.
[38] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. arXiv preprint arXiv:2106.090) 2021.
[39] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for maasformer-baseer detector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2567-2575, 2022.
[40] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEF/CVF international conference on computer vision, pages 12259-
12269, 2021.
13


Reviews:
[{"review_id": "fbWbHHKfoF", "reviewer": "Reviewer_Z9GJ", "summary": "This paper presents HotBEV, a new model developed for 3D detection tasks. By prioritizing actual on-device latency and considering key hardware properties, HotBEV achieves impressive reductions in computational delay. This optimization allows for real-time decision-making in self-driving scenarios, making it a significant contribution to the field. The model's versatility, being compatible with both high-end and low-end GPUs, further underscores its practical value. Rigorous experimental validation showcases the model's superior performance in terms of speed and accuracy compared to existing solutions.", "questions": "1. To ensure an equitable comparison, the inclusion of SoloFusion with a four-frame input is indispensable.\n2. Comprehensive relevant literature should be integrated into this paper for completeness.", "limitations": "This paper discuss the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The model is compatible with both high-end and low-end GPUs, demonstrating a broad range of applicability.\n2. The proposed method successfully achieve a delicate balance between model speed and detection precision.\n3. They utilize a theoretical latency prediction model to guide their design, an innovative approach that differs from the typical focus on computational FLOPs.", "weaknesses": "1. The comparison is not fair in main experiments. The length of temporal fusion is critical to model performance. SoloFusion [1] suggests that a longer temporal sequence does not affect the model's FPS (Frames Per Second). This paper's methodology employs four frames for temporal fusion, while most comparative methods use only one to two frames.\n2. The exploration of utilizing convolutional modulation, as opposed to self-attention, for relationship building is already documented in certain works [2], which have not been incorporated into this paper's analysis.\n\n\n[1] Time Will Tell: New Outlooks and A Baseline for Temporal Multi-View 3D Object Detection.                                                                                     \n[2] You Only Segment Once: Towards Real-Time Panoptic Segmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReQ1:** For a fair comparison with SoloFusion, we tested a 4-frame version of SoloFusion, as shown in Table A. Across multiple benchmarks, our HotBEV consistently outperforms SoloFusion, with the exceptions of mATE and mAVE. Furthermore, HotBEV achieves a 35% faster FPS compared to SoloFusion.\n\nTable A. State-of-the-art Comparison on nuScenes val set.\n|Methods | Backbone | Resolution |Frames | NDS ↑ | mAP↑ | mATE↓ | mASE↓ | mAOE↓ | mAVE↓ | mAAE↓ | FPS |\n|------------|-----|-----|-----:|-----|-----|-----|-----|-----|-----|-----|-----|\n|SOLOFusion|\tResNet50|\t256 × 704|\t16|\t0.534|\t0.427|\t0.567|\t0.274|\t0.511|\t0.252|\t0.181|\t11.4|\n|SOLOFusion|\tResNet50|\t256 × 704|\t4|\t0.494|\t0.362|\t0.607|\t0.304|\t0.539|\t0.293|\t0.19|\t12.2|\n|HotBEV|\tHOB-base|\t256 × 704|\t4|\t0.506|\t0.369|\t0.625|\t0.264|\t0.362|\t0.364|\t0.153|\t16.5|\n\n\n**ReQ2:** Thank you for sharing this paper. Upon review, we observed that while both our methods employ convolution, our design demonstrates greater efficiency.\nYOSO incorporates a Separable Dynamic Decoder, specifically substituting the Multi-Head Cross-Attention with the Separable Dynamic Convolution, which consists of a depthwise convolution followed by a pointwise convolution. As detailed in Equation 10 of [2], its computational complexity is represented by $2ndt+2n^2d$. Notably, this complexity grows quadratically with increasing sequence length N.\nIn contrast, our approach employs depthwise convolution combined with the Hadamard product to determine the output, as illustrated in Equations 3, 4, and 5 of our appendix. Our computational demands rise linearly, rather than quadratically, as the image resolution escalates. Moreover, it's evident that smaller models derive greater benefits from the Hadamard product [1].\nWhen we substituted our design with YOSO's Separable Dynamic Convolution, the results are presented in Table B.\n\nTable B. Convolutional Modulation Comparison on nuScenes val set.\n|Methods | Backbone | Resolution |Frames | NDS ↑ | mAP↑ | mATE↓ | mASE↓ | mAOE↓ | mAVE↓ | mAAE↓ | FPS |\n|------------|-----|-----|-----:|-----|-----|-----|-----|-----|-----|-----|-----|\n|HotBEV\t|YOSO|\t256 × 704|\t4|\t0.498|\t0.36|\t0.643|\t0.277|\t0.371|\t0.375|\t0.162|\t15.1|\n|HotBEV\t|HOB-base|\t256 × 704|\t4|\t0.506|\t0.369|\t0.625|\t0.264|\t0.362|\t0.364|\t0.153|\t16.5|\n\n[1] Conv2Former: A Simple Transformer-Style ConvNet for Visual Recognition\n\n[2] You Only Segment Once: Towards Real-Time Panoptic Segmentation"}, {"review_id": "L6aVaTTCZW", "reviewer": "Reviewer_H5E6", "summary": "The paper proposes a novel hardware-efficient transformer-based framework called HotBEV for camera-only 3D detection tasks. The framework is designed to achieve high-speed inference on multiple devices, including resource-limited ones, by considering hardware properties such as memory access cost and degree of parallelism. Extensive results on nuScenes shows the effectiveness of HotBEV.", "questions": "Please respond to my concerns in the weaknesses section. New experiment results and more thorough comparisons with SOTA camera-only 3D object detection papers are expected. WOD results are helpful. Please also discuss how the proposed approach takes advantage of unique properties of the task. The current design seems to be assembling bag of tricks for general model design.", "limitations": "The authors have adequately addressed the limitations. \n\n===\n\nJustifications for final ratings:\n\nPros:\n\n+ Good experiment results presented in the rebuttal;\n\n+ Very solid latency measurement;\n\n+ Promised to release the code.\n\nCons:\n\n+ Too many tricks in the methodology (mentioned by more than 1 reviewer);\n\n+ Unclear advantage over transformer-based backbones;\n\n+ Relatively small advantage over FastBEV, a method that does not require a re-design of the backbone;\n\n+ Uncertain about the effectiveness of NAS (also pointed out by my colleague).\n\nOverall, I believe that the authors have made a good faith effort to address the limitations of their work, but I would like to see the AC carefully investigate the submission before making a final decision.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper is quite well written.\n\n- The authors have conducted a comprehensive evaluation of their approach on the large-scale nuScenes dataset, resulting in a notable balance between accuracy and efficiency when compared to previous publications.\n\n- The idea of latency prediction is easy to grasp and effective.", "weaknesses": "- Please double check whether the abstract matches with the paper. In the abstract the proposed method is called HETR, but in paper it is called HotBEV.\n\n- The paper seems to assemble a lot of engineering tricks to achieve the final performance. For example, using BN in FFN, or using RepCNN layers. These tricks are widely known to be effective.\n\n- Latency prediction is an interesting idea but I don't think it is quite different from existing designs in hardware-aware neural architecture search.\n\n- The authors did not make a fair comparison with stronger baselines such as (the larger variants of) PETRv2, BEVFormer / BEVFormer++, BEVDepth, BEVDet4D, etc.\n\n- The authors did not present experiment results on Waymo Open Dataset, on which BEVFormer achieves really good results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1:** We have not updated the abstract of the paper on the OpenReview interface. We sincerely apologize. The title of our submitted paper is “HotBEV: Hardware-oriented Transformer-based Multi-View 3D Detector for BEV Perception”. So our proposed method is ‘HotBEV’. We appreciate your valuable comments.\n\n**ReW2:** Thank you for inspiring us to clarify the differences between our design and the existing modules in the final version. \nRegarding the BN fusion technique, directly substituting Layernorm with Batchnorm in the original self-attention often proves challenging, as the model struggles to converge during training. One distinct advantage of our design, particularly with window-based self-attention, becomes evident when compared to other transformer architectures. Our approach can seamlessly replace Layernorm with Batchnorm, subsequently applying the fusion technique without sacrificing accuracy, as illustrated in Figure 9 of the appendix.\nIn our research, we incorporated RepCNN as a candidate within our proposed latency-aware architectural search. This was to analyze the balance between detection precision and the efficiency derived from using RepCNN. Through our experiments, we found that substituting RepCNN in both the 1st and 2nd stages yielded optimal results, as detailed in Table 9 of the paper.\n\n**ReW3:**  Traditional hardware-aware network search methods usually depend on the hardware deployment of each candidate within the search space to ascertain latency, a process that is both time-consuming and inefficient. A single candidate demands hundreds of inferences to generate an accurate latency, prolonging the search process. Some contemporary methods, like HAT[1], leverage a latency predictor. This predictor, pre-trained on thousands of real-world latency data points, serves as an offline tool to anticipate candidate latency, rather than determining real latency via inference during the search. \nHowever, it still necessitates hundreds, even thousands, of actual speed tests across diverse model structures to form a robust training dataset to train an accurate latency predictor. Moreover, is only applicable to a relatively small search space. For larger search spaces, an increased volume of measured latency data is required as a training set for the predictor, substantially raising the time cost. If the test set is inadequate, the predictor fails to estimate the latency accurately.\nOur methodology stands out in its accuracy and efficiency. Our latency prediction model is a training-free theoretical model, suitable for general-purpose hardware, GPU.  It considers the properties of the target hardware, the model type, the model size, and the data granularity. It then quantitatively captures both the computation latency and data movement latency, enabling it to precisely predict the actual throughput for each layer, as depicted in Figure 8 of the appendix.\n\n**ReW4:**  Our research specifically targets small models, which is why our results are particularly favorable for these models compared to other studies. For a comprehensive understanding, we also present a comparison with baseline models that possess larger backbones and increased input sizes in Table C.\nNotably, we surpass baseline models in frames per second (FPS) while maintaining comparable accuracy levels.\n\nTable C. Comparison on nuScenes val set.\n|Methods|Backbone|Resolution|NDS↑|mAP↑|mATE↓|mASE↓| mAOE↓| mAVE↓| mAAE↓|FPS|\n|-|-|-|-|-|-|-|-|-|-|-|\n|PETR|ResNet101|1600×900|0.442\t|0.37|0.711|0.267|0.383|0.865|0.201|5.7|\n|PETRv2|ResNet101|1600×640|0.524|0.421|0.681|0.267|0.357|0.377|0.186|-|\n|BEVDet4D|Swin-B|1600×640\t|0.515|0.396|0.619|0.26|0.361|0.399|0.189|-|\n|BEVDepth|ResNet101|512×1408|0.535|0.412|0.565|0.266|0.358|0.331|0.19|2.3|\n|BEVFormerv2|ResNet50|1600×640|0.529|0.423|0.618\t|0.273|0.413|0.333|0.181|-|\n|PolarFormer-T|ResNet101|1600×900|0.528|0.432|0.648|0.27|0.348|0.409|0.201|3.5|\t\t\t\t\n|Sparse4D|ResNet101-DCN|900×1600|0.541|0.436|0.633|0.279|0.363|0.317|0.177|4.3|\n|HotBEV|HOB-base|512×1408|0.525|0.427|0.62|0.221|0.36|0.55|0.163|5.5|\n\n**ReW5:**  Thank you for pointing this out. We ran our HotBEV on Waymo Open Dataset and compared it with State-of-the-art models BEVFormer++ and PETRv2. The results are shown in Table D.\n\nTable D. Comparison on the Waymo val set.\n|Methods | Backbone | mAPL↑ |mAP↑ | mAPH↑ |\n|------------|-----|-----|-----|-----|\n|BEVFormer++\t|ResNet101-DCN\t|0.361\t|0.522\t|0.481|\n|PETRv2 \t|ResNet101\t|0.366\t|0.519\t|0.479|\n|HotBEV\t|HOB-base\t|0.371\t|0.537\t|0.598|\n\n\n**ReQ1:** Thanks for your valuable suggestions.  Based on our analysis of GPU performance, we observed that the primary source of latency consistently stems from the backbone. In order to mitigate this speed bottleneck without compromising detection precision, we introduce a potent transformer encoder for feature capturing and fusion. We partition the backbone into four stages, mirroring the data flow granularity seen in ResNet architecture. As the features transition from local to global visual receptive fields, we introduce the HOB block design.\n\nWithin each HOB block, we employ a sequence of local-wise attention mechanisms to extract local information, i.e., texture-level semantics. This is followed by global attention to enhance abstract-level semantics across the feature map. Moreover, to further bolster low-level semantics within the current stage, we insert a semantic-augmented module comprising an upsampling layer and global attention after every two consecutive HOB blocks, excluding the interval between Stage 1 and 2. To amplify texture-level semantics, we foster information exchange not only within stages but also between them.\n\nBy leveraging the efficient operators outlined in Section 3.1.2, including Convolutional Modulation, BN Fusing, and Multiple Branch Fusing, we introduce the concept of a \"two-phase design space\" (DS) for the HOB backbone.\n\n[1] HAT: Hardware-Aware Transformers for Efficient Natural Language Processing"}, {"review_id": "73VwupLKO3", "reviewer": "Reviewer_BJ1v", "summary": "This paper proposed a latency-aware design strategy to search for an efficient network structure for BEV perception. To make this happen, this paper proposed a convolutional modulation layer for replacing native self-attention, and proposed to use BN to replace LN to make the network inference faster. Based on such basic module, a hardware-oriented backbone design is proposed for the following network searching process. The experiments demonstrate that the searched network achieved a good trade-off between inference efficiency and performance.", "questions": "1. For the hardware-oriented decoder, what's the difference compared with DETR3D and PETR? Did you just replace native cross-attention with the proposed convolutional modulation layer? \n2. Have you evaluated the predicted latency with the realistic latency? How about the accuracy with your latency prediction model? \n3. By comparing with native self-attention, how about the effectiveness of the proposed convolutional modulation layer? \n4. For L186-L190, can we directly replace the LN in window-based self-attention with BN? Or we must combine the proposed convolutional modulation layer with BN to make the network converge.", "limitations": "The main concern of this paper is still unclear writing and organization. Besides that, it is not clear whether the network search process is necessary and whether it can really achieve state-of-the-art performance with good inference latency.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The proposed idea of using latency prediction model for efficient neural network searching is reasonable. \n2. The finally searched result HoTBEV achieves a good trade-off between performance and efficiency.", "weaknesses": "1. The writing and organization of the paper are not easy to follow, since lots of technical details are not clearly presented. For example, (a) the detailed structure of the two-phase design space (Eq. 1) is not clear. (b) Fig 5(c) indicates a structure titled Convolution, and the correspondence between Fig.5(c) and HOB block (fig. 5(b)) is not clear. (c) For L230-L236, what do you mean by mentioning patch embedding? Is it path merging in Fig. 5(b)? Did not you simply replace convolutional with larger kernel size with a series of 3x3 convolutions? (d) The structure of the supernet is also simply mentioned without detailed information. \n2. The paper does not discuss the searched structure with the existing works, which can not verify whether it is necessary to use the latency prediction model and conduct the network search process. \n3. Table 1 lacks of comparison with some latest BEV works. For example, PETRv2 with ResNet101-DCN, PolarFormer, and Sparse4D, and it seems that all of them achieve similar or better performance than the searched results. \n4. The illustration of latency prediction model is not clear. In particular, how to get the latency of computation is not clear. Is it reasonable to use the maximum throughput of each PE?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1:** Thank you for highlighting areas in our article's expression that need improvement.\n\n(a) The detailed structure of the modules in equation 1 is provided in Figure 9 -11 of the appendix, also our searched structures are provided in Table 13-15 of the appendix. We will mention this in the updated version.\n\n(b) We replace the channel-wise attention in Fig. 5(b) and Fig.5(d) with our proposed Convolution modulation of Fig5(c). Our Proposed Hardware-Oriented Decoder is shown in Figure 11 of the appendix, we will move this figure to the main paper. \n\n(c) For patch embedding (L230-L236), a convolution stem, which consists of three 3x3 convolutions with a stride of 2, is more hardware-efficient than a larger kernel.\n\n(d) Our search space (Figure 5 (b)) is a selection of possible blocks, including RepCNN, WMSA, SWMA, and Convolutional modulation. We propose a simple, fast yet effective gradient-based search algorithm to obtain a candidate network that just needs to train the supernet for once. To train our supernet, we adopted the Gumble Softmax sampling to get the importance score for the blocks within each search space/stage. During each step of training, a number of blocks are sampled to obtain a subnet structure. The latency of this subnet can be estimated using our latency prediction model. The detailed training pipeline is discussed in A.10 and Algorithm 1 of the appendix. Our searched structures are provided in Table 13-15 of the appendix. \n\n**ReW2:** Leveraging network search allows us to pinpoint the optimal structure tailored to specific hardware constraints.  As shown in Table 12 of the appendix, we compare the performance of our search model against four randomly sampled networks, all operating under identical mapping latency.  Our searched HotBEV-nano has better latency or higher NDS/mAP on nuScenes than the randomly sampled networks.\nThe utilization of the latency prediction model is essential, especially when crafting hardware-aware structural designs, which need to consider factors such as inference time, energy consumption, and memory footprint.  Our method is accurate. It considers the properties of the target hardware, the model type, the model size, and the data granularity. Furthermore, it mathematically describes the computation latency and data movement latency to accurately predict the actual throughput of each layer (See Figure 8 of the appendix). This model serves as a ready-to-use theoretical framework for general GPU architectures.\n\n**ReW3:**  Our research specifically targets small models, which is why our results are particularly favorable for these models compared to other studies. For a comprehensive understanding, we also present a comparison with baseline models that possess larger backbones and increased input sizes. Due to the word limit, please see Table C in Reviewer **H5E6**’s response.\nNotably, we surpass baseline models in frames per second (FPS) while maintaining comparable accuracy levels.\n\n**ReW4:** Due to the length restriction of the paper, a comprehensive discussion on our latency prediction model can be found in Appendix A.1.2.\nIn summary, the inputs for the latency prediction model include: 1) the structure configuration of a candidate block, 2) the spatial granularity G, 3) the channel dimension C, and 4)  the hardware properties are shown in Table 3. \n\nThe latency of a candidate block is predicted according to the following three steps: 1)Input/output shape definition. 2) Operation-to-hardware mapping. 3) Latency estimation.\n\n\n**ReQ1:** Our contribution lies in three folds:\n\n(1) Motivation: Our methods use convolution instead of self-attention to create associations, which are more memory-efficient (linear memory complexity), especially when processing high-resolution images. Due to the modulation operation, our method differs from traditional residual blocks and can adapt to the input content.\n\n(2) Convolution Modulation Design: We use the convolutional features extracted by the depthwise convolutions to modulate the weights of the right linear branch via the Hadamard product operation, as illustrated in Equations 3, 4, and 5 of our appendix.\n\n(3) Implementation: The decoder module consists of six layers. we replace the attention with convolution modulation in the first three layers to allow for more efficient processing while still capturing important local dependencies. In the remaining three layers, we employ channel-wise attention to effectively capture and incorporate global information into the decoding process, enhancing its overall performance.\n\n**ReQ2:**  We randomly sample 100 structures with estimated FPS ranging between 15 and 25, then evaluate the actual latency of these structures on NVIDIA V100. As depicted in Figure A inside the pdf submitted for the rebuttal, the predicted latency closely aligns with the actual latency, with their relationship approximating the line y=x. \n\n**ReQ3:** Our proposed convolutional modulation is more memory-efficient than native self-attention, especially when handling high-resolution images. This is attributed to its linear memory complexity, as detailed in Equation 3 of the appendix. \nIn Table 5 of the appendix, we also demonstrate the effectiveness of running 2D detection baselines with or without Convolutional modulation.\n\n**ReQ4:**  We can directly replace the LN in window-based self-attention with BN.  Replacing the LayerNorm directly with the BatchNorm in the original self-attention can make the network hard to converge.\nThis approach is feasible even in the absence of convolutional modulation.  In Table 5 of the appendix, we ablation the two proposed methods individually on 2D detection baselines.\n\n**ReLimitation** Please refer to Reverviewer H5E6: ReW3 for the network search. And we can achieve the SOTA trade-off of performance and latency among the camera-only methods as Table 1 in the paper."}, {"review_id": "O9TSs3Wgbl", "reviewer": "Reviewer_Mv7q", "summary": "This submission introduces a carefully-crafted transformer model family (models with varying speed-accuracy trade-off) for 3D detection from multi-view camera data. The adopted model design methodology prioritises hardware-efficiency, customising the proposed architecture to the target GPU. For this purpose, an analytical performance model is developed and queried to estimate the inference latency and guide different design choices. Additionally, following a module-level benchmarking, targeted architectural changes are introduced to alleviate the main identified computational bottlenecks in the backbone, normalisation and fusion layer design. The proposed model is trained in the form of a super-network, from which targeted sub-models can be extracted, offering a speed-accuracy trade-off that dominates the pareto frontier between the examined baselines.", "questions": "- Please provide further discussion/results on how the proposed methods speed and accuracy compares with 3D reconstruction based BEV detectors.\n- How would the proposed methodology perform when targeting embedded GPUs, such as AGX Xavier or Orin.\n- The proposed latency modelling seems to study memory and computation in isolation. In the case of memory-bounded layers, does the proposed model capture the latency-penalty of PEs who are stalling due to data starvation (e.g. similar to Roofline model analysis) ?\n- Would an oracle/benchmarking-based approach, considering the actual latency of each architecture result in different design choices to those taken considering the estimation of the developed model ?\n\nTypos:\n- In Table 3 the HW properties of GTX2080 and GTX1080 appear to be identical. Is this correct ?\n- Line 213: [3] -> Figure [3]", "limitations": "The manuscript briefly discusses some limitations of the current approach, mostly in the form of future/active work undertaken by the authors. A more detailed discussion of the assumptions and corresponding limitations of the approach would add value to the manuscript.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The aim of the paper to design efficient 3D detection models that are able to achieve realistic latency gains, rather than theoretical workload reduction, is very important and well-motivated. \n- Additionally, offering a design methodology that is able to customise the model design to the target computation platform facilitates adaptability and is of great benefit to the community. \n- The results of the conducted benchmarking are insightful and offer detailed information for improving inference efficiency on similar tasks. \n- The proposed approach dominates the speed-accuracy optimality frontier on 3D detection from BEV data, compared to widely adopted baselines.", "weaknesses": "- The design choice to focus on multi-frame models instead of those adopting 3D representations is not sufficiently justified, nor backed by a corresponding discussion back by experimental evidence. Particularly since this design choice restricts the re-use of the proposed approach to data from other sensors, such as LiDAR.\n- The examined GPU devices are on the power-hungry end of the spectrum. For self-driving scenarios, it also makes sense to consider embedded GPU devices too.\n-The writing of the paper becomes a bit dense and hard to follow in certain sections of the methodology, leaving some unclear points.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**ReW1:**  Thank you for this question to give us a chance to clarify our motivations.\nCurrently, the pre-fusion family, such as BevFusion, is very expensive to deploy in practice. The current in-vehicle market is dominated by camera-only solutions, such as Tesla, or post-fusion of camera and lidar signals, such as XPeng Motors, whose camera part still deploys camera-only solutions [1]. \nHence, our research goal is to explore the efficient and effective camera-only 3D detection series for practical implementation. So we choose the state-of-the-art frameworks, multi-frame models as the starting point of our research.\nMoreover, according to [2], the camera processing modules occupy 33% ~ 42% of the whole latency distribution for the post-fusion system. Our hardware-oriented design with enhanced image feature representation can be leveraged in the camera encoder part of the current in-vehicle detection system to improve on-device efficiency.\n\n**ReW2:**  Thanks for your valuable suggestions. It is necessary to test the speed on an actual commercial chip. As shown in Table E, we test our HotBEV models on Orin to validate our framework. Before testing, we quantized our model into INT8 with a tensorRT engine. And then run the test 50 times for each model in order to obtain stable results. \n\nTable E. Results on Orin.\n|Methods | Backbone | Resolution| NDS↑ |mAP↑ | FPS|\n|------------|-----|-----|-----|-----|-----|\n|HotBEV\t|HOB-nano\t|512x1408\t|0.47\t|0.385\t|31.8|\n|HotBEV\t|HOB-tiny\t|512x1408\t|0.512\t|0.407\t|20.4|\n|HotBEV\t|HOB-base\t|512x1408\t|0.525\t|0.427\t|16.1|\n\n**ReW3:** Thanks for your valuable suggestions. We will put more related works and experiments in the appendix part. We will focus on methodology in the main body of our final version.\n\n**ReQ1:** Thank you for guiding us to make a more comprehensive comparison with other branch methods. Table F shows that 3D reconstruction-based BEV detectors are better than our methods by up to 19.8 NDS but with inferior on-device speed than ours by up to 19.4 FPS. Even though the detection performance of image-only detectors cannot be superior to Lidar-only methods, these results also provide us motivation to implement our design into one of the commercial mainstream, post-fusion systems to improve hardware efficiency, as mentioned in W1.\n\n\nTable F. Comparison with 3D reconstruction-based BEV detectors.\n|Methods | NDS↑ |mAP↑ | FPS|\n|------------|-----|-----|-----|\n|PointPillars \t|61.3\t|52.3\t|29|\n|SECOND\t\t|63\t|52.6\t|14.3|\n|CenterPoint\t|66.8\t|59.6\t|12.4|\n|HotBEV-nano\t|47\t|38.5\t|31.8|\n|HotBEV-tiny\t|51.2\t|40.7\t|20.4|\n|HOB-base\t|52.5\t|42.7\t|16.1|\n\n**ReQ2:** Please refer the results in W2.  Thanks again for your valuable suggestions to help up improve our paper presentation.\n\n**ReQ3:** This is really a meaningful comment. When we build up our latency modeling, we do not add the roofline analysis, which can help to analyze the memory-bound layers more accurately. Please note that the self-attention module in the core architecture of the transformer is a memory-bound operation due to softmax operations. This is because  Softmax operations follow a two-stage dataflow, requiring buffering of intermediate data since direct output generation is not possible. Additionally, the lack of input data reuse in Softmax further complicates efforts to amortize memory costs through computation. However, in our design, the use of window-based self-attention is reduced, and the global attention is replaced by convolution modulation, significantly reducing the number of softmax, thus mitigating the adverse effect of memory-bound operators on speed. Hence, we do not excuse the analysis of memory-bound. Also, through the latency profiling of our HoB-nano backbone, we do not observe that PEs are stalling due to data starvation. \nThanks again for your valuable suggestion. We will add one discussion part to clarify our latency modeling with memory IO analysis.  Figure B in the pdf submitted for the rebuttal displays the latency percentage of each module. \n\n**ReQ4:** Thanks for guiding us for a deeper analysis of the latency prediction model. Our prediction model is mainly utilized to estimate the on-device latency of matrix multiplication. The latency of other operations will not be included during the network search, which is the same as using the benchmarking-based approach to estimate the on-device latency. Based on Q2 of Reviewer BJ1v, the predicted results of the proposed latency model are consistent with the actual on-device latency. Hence, the architecture results of our proposed latency modeling can be similar as benchmarking-based approach.\n\n**ReLimitation**: Thanks for your comprehensive and insightful recommendations. Based on your questions, we analyze it from the point of current practical applications and compare our method with other branches of research. We will add a discussion part for these content.\n\n[1] U.S.News, “Vehicles That Are Almost Self-Driving in 202”. Online Available. “https://cars.usnews.com/cars-trucks/advice/cars-that-are-almost-self-driving”\n\n[2] Pham, Trung, et al. \"NVAutoNet: Fast and Accurate 360o 3D Perception For Self Driving.\" arXiv preprint arXiv:2303.12976 (2023).\n\n**Typo:** Thank you very much for pointing them out. We will correct the typos in the revision."}]

Author rebuttal:
We first sincerely thank every reviewer for your insightful and constructive feedback. Then, we will answer the specific questions from each reviewer. We upload a pdf file with figures (Figure A,B) which we will present in our rebuttal.


Meta-review: The paper proposes HotBEV that concentrates on deploying hardware-end BEV detectors. This is a very good research direction for feasible autonomous driving scenario. There are few literature on designing efficient BEV detector paradigms. 

The paper proposes novel techniques to achieve efficiency by way of latency-aware methodology. And experimental results show very promising speedup with performance improvement. Overall all reviewers reach concensus that this is a good paper to be accepted. There are a few concerns on experiments and technical details, authors in general did a good rebuttal to address most of the concerns. It is highly recommended to incorporate all comments and revised the manuscript accordingly before camera-ready.
Decision: Accept (poster)