SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Comparison of different methods on the nuScenes val set. FPS is tested on V100 with INT8.
Method	Backbone	Resolution	Frames	NDS ↑	mAP ↑	mATE ↓	mASE ↓	mAOE ↓	mAVE ↓	mAAE ↓	FPS
BEVDet	ResNet50	256 × 704	1	0.379	0.298	0.725	0.279	0.589	0.86	0.245	25.8
BEVDet4D	ResNet50	256 × 704	2	0.457	0.322	0.703	0.278	0.495	0.354	0.206	25.9
PETRv2	ResNet50	256 × 704	2	0.456	0.349	0.7	0.275	0.58	0.437	0.187	30.2
BEVDepth	ResNet50	256 × 704	2	0.475	0.351	0.639	0.267	0.479	0.428	0.198	24.3
FastBEV-MS	ResNet50	256 × 704	4	0.485	0.343	0.647	0.282	0.36	0.342	0.225	-
SOLOFusion	ResNet50	256 × 704	16	0.534	0.427	0.567	0.274	0.511	0.252	0.181	20.2
SOLOFusion	ResNet50	256 × 704	4	0.494	0.362	0.607	0.304	0.539	0.293	0.19	21.4
HotBEV	HOB-nano	256 × 704	4	0.455	0.35	0.636	0.271	0.449	0.542	0.36	39.2
HotBEV	HOB-tiny	256 × 704	4	0.487	0.362	0.628	0.27	0.438	0.412	0.19	31.7
HotBEV	HOB-base	256 × 704	4	0.506	0.369	0.625	0.264	0.362	0.364	0.153	27.2
BEVDet	ResNet101-DCN	640 x 1600	1	0.472	0.393	0.608	0.259	0.366	0.822	0.191	2.95
FCOS3D	ResNet101-DCN	900 x 1600	1	0.395	0.372	0.806	0.268	0.511	1315	0.7	2.64
DETR3D	ResNet101-DCN	900 x 1600	1	0.434	0.349	0.716	0.268	0.379	0.842	0.2	5.74
PGD	ResNet101-DCN	900 x 1600	1	0.335	0.409	0.732	0.263	0.423	1.285	0.172	2.17
Focal-PETR	ResNet101-DCN	512 x 1408	1	0.461	0.39	0.678	0.263	0.395	0.804	0.202	10.23
PETR	ResNet101-DCN	512 x 1408	1	0.441	0.366	0.717	0.267	0.412	0.834	0.19	8.84
BEVFormer	ResNet101-DCN	900 x 1600	4	0.517	0.416	0.673	0.274	0.372	0.394	0.198	4.65
PolarDETR	ResNet101-DCN	900 x 1600	2	0.488	0.383	0.707	0.269	0.344	0.518	0.196	5.43
HotBEV	HOB-nano	512 x 1408	4	0.47	0.385	0.648	0.243	0.422	0.715	0.183	12.38
HotBEV	HOB-tiny	512 x 1408	4	0.512	0.407	0.634	0.235	0.408	0.632	0.175	11.39
HotBEV	HOB-base	512 x 1408	4	0.525	0.427	0.62	0.221	0.36	0.55	0.163	9.08
the object. For efficient modeling, we leverage the convolutional modulation layer as the decoder component (more details in Appendix A).
To generate temporal-aligned 3D coordinates, we adapted the technique TAM [19]. Subsequently, we combine the resulting 3D coordinates with the 2D features and feed them into a 3D position encoder [8]. This allows us to obtain temporal-aligned 3D-aware features, which improve the model's localization, attitude, speed estimation, and overall robustness.
3.3 Training
Our design space (Eq. (1)), also as search space is a selection of possible blocks, including RepCNN, WMSA, SWMA (for local-wise attention layer), and convolutional modulation (for global-wise attention layer). We propose a simple, fast yet effective gradient-based search algorithm to obtain a candidate network that just needs to train the supernet for once. To train our supernet, we adopt the Gumble Softmax sampling to get the importance score for the blocks within each search space/stage. During each step of training, a number of blocks are sampled to obtain a subnet structure. The latency of this subnet can be estimated using our latency prediction model.
Supernet design. We use the two-phase design space (DS) as the search space and train the supernet for the HOB backbone. We only search the backbone's structures, dimensions  $C$ , and input granularity  $G$ , while the decoder uses fixed structures with dimensions adapted to the backbone.
Latency-aware model slimming. It has three steps:
1) Train the supernet with the Gumble Softmax sampling [26] to get the importance score for the blocks within each DS.
2) Use latency prediction model  $E$  (Section 3.1.1) to estimate the on-device latency of each candidate.
3) Perform latency-aware model slimming on the supernet obtained from step 1) by FPS evaluated with predictor  $E$ . Specifically, we use the score  $s$  of each candidates to define the importance score of  $\mathrm{DS}_i$  as  $\frac{s_i^{\mathrm{RepCNN}} + s_i^{\mathrm{WMSA}}}{s_i^{\mathrm{SWMSA}}}$  for  $S_1$ ,  $S_2$ ,  $S_3$ , and  $\frac{s_i^{\mathrm{WMSA}}}{s_i^{\mathrm{SWMSA}}}$  for  $S_4$ . We sum up all the scores of all DS within that  $S$  and deduce the score for each  $S$ . Then we define the evolution process (all performed in the current least important  $S$ ): a) remove the  $1^{st}$  SWMSA; b) remove the  $1^{st}$  WMSA; c) reduce the width by multiples of 16. Then we predict the current FPS  $f$  and decide by FPS_NDS_drop(-%*f). This process is repeated until the target throughput is reached.
We conduct comprehensive experiments on the nuScenes dataset [27], which contains 1000 driving scenes of 20-second length for each. The scenes are officially split into 700, 150, and 150 scenes for training, validation, and testing. The dataset includes approximately 1.4M camera images.
Table 2: Comparison for large backbone and resolutions on the nuScenes val set (FPS on V100).
Method	Backbone	Resolution	NDS ↑	mAP ↑	mATE ↓	mASE ↓	mAOE ↓	mAVE ↓	mAAE ↓	FPS
PETR	ResNet101	1600×900	0.442	0.37	0.711	0.267	0.383	0.865	0.201	5.7
PETRv2	ResNet101	1600×640	0.524	0.421	0.681	0.267	0.357	0.377	0.186	-
BEVDet4D	Swin-B	1600×640	0.515	0.396	0.619	0.26	0.361	0.399	0.189	-
BEVDepth	ResNet101	512×1408	0.535	0.412	0.565	0.266	0.358	0.331	0.19	2.3
BEVFormerv2	ResNet50	1600×640	0.529	0.423	0.618	0.273	0.413	0.333	0.181	-
PolarFormer-T	ResNet101	1600×900	0.528	0.432	0.648	0.27	0.348	0.409	0.201	3.5
Sparse4D	ResNet101-DCN	900×1600	0.541	0.436	0.633	0.279	0.363	0.317	0.177	4.3
HotBEV	HOB-base	512 × 1408	0.525	0.427	0.62	0.221	0.36	0.55	0.163	5.5
Evaluation metrics include mean Average Precision (mAP) and five types of true positive metrics (TP metrics): mean Average Translation Error (mATE), mean Average Scale Error (mASE), mean Average Orientation Error (mAOE), mean Average Velocity Error (mAVE), mean Average Attribute Error (mAAE). We also report nuScenes Detection Score (NDS) to capture all aspects of the nuScenes detection tasks. We follow the training recipe from PETR but mainly report results with 24 training epochs to compare with other detection models. Experiments were run on 8 NVIDIA V100.
4.2 Model Accuracy and Speed Performance
Main results. As delineated in Table [1], our model is compared to an array of existing camera-based methodologies. These include FCOS3D [28], DETR3D [9], PGD [29], PETR [8], PETRv2 [19], Focal-PETR [30], BEVDet [4], BEVDet4D [5], BEVFormer [1], BEVDepth [2], and PolarDETR [31]. The table lists the backbone type, image resolution, number of frames, inference speed (FPS), and accuracy on the nuScenes validation set for each method. The backbone options include ResNet50 and ResNet101-DCN [25]. Our streamlined models surpass other methods in both performance score and inference speed.
In particular, our compact model, HOB-tiny  $(256 \times 704)$ , with 0.487 NDS, 0.362 mAP, and 19.8 FPS, outperforms BEVDet (0.379 NDS, 0.298 mAP, 16.7 FPS), BEVDet4D (0.457 NDS, 0.322 mAP, 16.7 FPS), PETRv2 (0.456 NDS, 0.349 mAP, 18.9 FPS), and BEVDepth (0.475 NDS, 0.351 mAP, 15.7 FPS) with a ResNet50 backbone, achieving a  $2.5\% \sim 28.5\%$  NDS gain,  $3.1\% \sim 21.5\%$  mAP gain and  $4.8\% \sim 26.1\%$  FPS gain. The reductions in the orientation error, velocity error, and attribute error all contribute to enhancing the NDS score. Even with a larger input resolution, our model continues to offer an unrivaled balance of accuracy and speed compared to extant work. For instance, our HOB-nano model  $(512 \times 1408)$  achieves  $2\%$  NDS and  $13.6\%$  FPS increase with a  $1.3\%$  mAP and score difference when compared to Focal-PETR (ResNet101-DCN,  $512 \times 1408$ ). Our HOB-tiny model  $(512 \times 1408)$  surpasses PolarDETR (0.488 NDS, 0.383 mAP, 3.5 FPS) in both speed and accuracy. Our HOB-base model  $(512 \times 1408)$  also surpasses BEVFormer (0.517 NDS, 0.416 mAP, 3 FPS) in both speed and accuracy. For detailed configurations of our model and detailed analysis of temporal modeling and robustness related to frame length, please refer to Appendix A.
Our research specifically targets small models, so our results are particularly favorable for these models compared to other studies. For a comprehensive understanding, we also present a comparison with baseline models that possess larger backbones and increased input sizes in Table 2. Notably, we surpass baseline models in frames per second (FPS) while maintaining comparable accuracy levels.
Comparison with 3D reconstruction-based BEV detectors. Table 3 shows that 3D reconstruction-based BEV detectors are better than our methods by up to 19.8 NDS but with inferior on-device speed than ours by up to 19.4 FPS. Even though the detection performance of image-only detectors cannot be superior to Lidar-only methods, these results also motivate us to implement our design into one of the commercial mainstream, post-fusion systems to improve hardware efficiency. Currently, the pre-fusion
Table 3: Comparison with 3D reconstruction-based BEV detectors on the nuScenes val set.
Methods	NDS↑	mAP↑	FPS
PointPillars	61.3	52.3	29
SECOND	63	52.6	14.3
CenterPoint	66.8	59.6	12.4
HotBEV-nano	47	38.5	31.8
HotBEV-tiny	51.2	40.7	20.4
HOB-base	52.5	42.7	16.1
family, such as BevFusion, is costly to deploy in practice. The current in-vehicle market is dominated by camera-only solutions, such as Tesla, or post-fusion of camera and lidar signals, such as XPeng Motors, whose camera part still deploys camera-only solutions [32]. Hence, our research aims to explore the efficient and effective camera-only 3D detection series for practical implementation. So we choose state-of-the-art frameworks, and multi-frame models as the starting point of our research.
Figure 6: Left and middle: The trade-off between performance (NDS) and hardware efficiency (FPS) for different detection methods on the nuScenes val set with different GPUs. Right: Various granularity settings on the HOB backbone of HotBEV.
Moreover, according to [33], the camera processing modules occupy  $33\% \sim 42\%$  of the whole latency distribution for the post-fusion system. Our hardware-oriented design with enhanced image feature representation can be leveraged in the camera encoder part of the current in-vehicle detection system to improve on-device efficiency.
Performance on multiple GPU devices. To evaluate the hardware throughput, we implement the latency-aware model slimming on two other devices: NVIDIA GTX 2080 Ti which has a 5.5M size L2 cache with 448 Gbps memory bandwidth; NVIDIA GTX 1080 Ti which has a 5.5M size L2 cache with 325 Gbps memory bandwidth; We report the average FPS of over 1000 inferences. As depicted in Figure 6 our approach surpasses current camera-only BEV frameworks in terms of both hardware efficiency and performance. Other methods typically overlook the constraints imposed by limited memory and parallelism in on-device runtimes, leading to further degradation in speed performance on devices with limited resources. Without hardware optimization, e.g., int8 quantization, some methods fail to produce results within a reasonable timeframe. In contrast, our models strike the optimal balance between speed and performance, making them the superior choice among existing approaches. For example, on GTX 2080, our HotBEV (0.385 mAP) is  $4.5 \times$  faster than BEVDet (0.393 mAP); our HotBEV (0.407 mAP) is  $2.6 \times$  faster than BEVformer (0.416 mAP); our HotBEV (0.427 mAP) is  $1.4 \times$  faster than PolarDETR (0.383 mAP).
The speed-up effect is superior to that in Table I, demonstrating our framework's enhanced GPU generalization ability compared to other approaches. It's worth noting that our logic extends beyond autonomous driving. Firstly, our backbone is designed to cater to general vision tasks. Secondly, the hardware model is specifically optimized for GeMM. In the appendix, we showcase our implementation on diverse general vision tasks such as classification and 2D detection.
Performance on multiple commercial Orin. It is necessary to test the speed on an actual commercial chip. As shown in Table 4, we test our HotBEV models on Orin to validate our framework. Before testing, we quantized our model into INT8 with a
Table 4: Our proposed results on Orin.
Methods	Backbone	Resolution	NDS↑	mAP↑	FPS
HotBEV	HOB-nano	512x1408	0.47	0.385	31.8
HotBEV	HOB-tiny	512x1408	0.512	0.407	20.4
HotBEV	HOB-base	512x1408	0.525	0.427	16.1
tensorRT engine. And then run the test 50 times for each model to obtain stable results.
In this section, we conduct the ablations with HotBEV-nano trained 24 epochs. The backbone is pre-trained on ImageNet dataset [34] and trained on Nuscenes. The input image size is  $256 \times 704$ , and the number of detection queries is set to 900.
Table 5: Ablation study of HOB.
Index	GA	SAM	NDS	mAP	FPS
①	-	-	0.396	0.328	21.3
②	✓	-	0.443	0.337	21.4
③	✓	GA	0.447	0.340	23.7
④	✓	✓	0.455	0.350	24.5
4.3.1 Analysis of Components in HotBEV
Major components of HOB. Table 5 studies how global attention (GA) and semantic-augmented module (SAM) contribute to HOB performance. We only modify the backbone network without disabling the 3D position encoder, TAM, and decoder modules. ① and ② show that inserting our GA after the local-wise attention can improve  $0.9\%$  mAP and  $4.7\%$  NDS without significant impact on speed. ② and ③ show that inserting one GA block costs 2.3 FPS yet only gains  $0.3\%$  mAP. Once we replace the GA with the SAM, mAP is increased from  $34\%$  to  $35\%$ , while NDS is increased from  $44.7\%$  to  $45.5\%$ . So SAM can enhance performance and be better than simple global modeling.
4.3.2 Analysis of Prediction Modeling
Empirical validation. To evaluate the prediction model, we varied the input granularity  $G$  and used the 1-st block in Swin-T as a case study. Figure 7 compares our predictions and the actual testing latency on two different GPUs, the NVIDIA V100 and GTX 1080 Ti. The results demonstrate that the prediction model can accurately estimate the actual latency across a wide range of input granularities.
More granularities settings. We test various granularity settings on the HOB backbone of HotBEV-nano to examine the effects of  $G$ . The results on the Tesla-V100 GPU are presented in the left sub-figure of Figure 6. We increase  $G$  from 2 and 64, which consistently improves the realistic efficiency of both the NVIDIA V100 and GTX 1080 GPU. It can be found that the finest granularity ( $G = 2$ ) causes substantial inefficiency despite the mAP improvement. Otherwise, the coarsest granularity ( $G = 64$ ) benefits the speedup with large detection precision degradation. In specific, all networks are transformed into fixed matrix operations on GPU platforms (General Matrix Multiply, GeMM). The latency prediction model evaluates the speed of matrix multiplication and the corresponding data movement, so it is applicable to general networks. For example,
Figure 7: Latency prediction results. Results are tested on NVIDIA V100 and GTX 1080 Ti.
this modeling can be generalized in other networks as the prediction results on the 1st block in ResNet-50 in Appendix A.
More discussion on latency predictor. Our proposed latency predictor provides some opportunities. (1) Efficient model generation, which also proceeds with AI democratization. As the benchmarking-based approach needs one-day training, our proposed theoretical latency predictor is training-free. For example, the benchmarking-based approach requires 5 days to generate the dataset of 5 different devices if 5 target models are demanded. In contrast, our proposed is off-the-shelf. Our method provides the opportunity for inexpensive and efficient research for users who do not have access to target devices. For instance, when the in-vehicle Orin chip is not accessible, the related efficient model research on the Orin chip can still be advanced. In conclusion, our approach makes sense for today's rapidly growing demand for autonomous driving. (2) The proposed latency predictor focuses on modeling the latency of Matrix Multiplication (MM) with generalizability. Indeed, strong GPU simulators cannot accurately model the behavior of the latest NVIDIA GPUs. However, our purpose is not to describe the behavior of GPUs. We want to reflect on the relative performance of latency for different layer types and sizes on target GPUs. This is because our search goal is to minimize the relative time in the search space of the current device. For generalizability, our design focuses on latency modeling of MM, the typical computation operation in DNNs, which is mainly impacted by the computing performance of Tensor Core, not other specific operators, so the proposed predictor has generalizability, as shown in Figure 7 of our paper.
We present a hardware-oriented transformer-based framework (HotBEV) for camera-only 3D detection tasks, which achieves higher detection precision and remarkable speedup across high-end and low-end GPUs. Firstly, we propose a theoretical, plug-and-play latency prediction model. Given a target GPU, we directly use the latency to guide our algorithm design. Based on the latency breakdown of classic camera-only detectors, we identify the backbone as the main speed bottleneck. Then, we propose efficient operators and fusion techniques for model on-device implementation. Based on these operators and the process of vision modeling, we design a hardware-oriented backbone with strong feature enhancement. Then we propose the basic design paradigm of HotBEV. Finally, guided by the latency prediction model, we generate the family of HotBEV through a standard search algorithm. Experiments show the superior inference accuracy of HotBEV compared to SOTA BEV detectors with significant on-device speed. Notably, while our primary focus lies in the camera-only method for BEV perception, our latency-aware design methodology can also be applied to fusion-based BEV methods, enabling efficient algorithm design that aligns with real-world requirements.
This work is supported in part by National Science Foundation CCF-1937500.
[34] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248–255. IEEE, 2009.
[35] Yinpeng Chen, Xiyang Dai, Dongdong Chen, Mengchen Liu, Xiaoyi Dong, Lu Yuan, and Zicheng Liu. Mobile-former: Bridging mobilenet and transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5270-5279, 2022.
[36] Sachin Mehta and Mohammad Rastegari. Mobilevit: light-weight, general-purpose, and mobile-friendly vision transformer. arXiv preprint arXiv:2110.02178, 2021.
[37] Qibin Hou, Cheng-Ze Lu, Ming-Ming Cheng, and Jiashi Feng. Conv2former: A simple transformer-style convnet for visual recognition. arXiv preprint arXiv:2211.11943, 2022.
[38] Alaaeldin El-Nouby, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. arXiv preprint arXiv:2106.09681, 2021.
[39] Yingming Wang, Xiangyu Zhang, Tong Yang, and Jian Sun. Anchor detr: Query design for transformer-based detector. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2567-2575, 2022.
[40] Benjamin Graham, Alaaeldin El-Nouby, Hugo Touvron, Pierre Stock, Armand Joulin, Hervé Jégou, and Matthijs Douze. Levit: a vision transformer in convnet's clothing for faster inference. In Proceedings of the IEEE/CVF international conference on computer vision, pages 12259-12269, 2021.