SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Peiyan Dong $^{1*}$ , Zhenglun Kong $^{1*}$ , Xin Meng $^{1*}$ , Pinrui Yu $^{1}$ , Yifan Gong $^{1}$ , Geng Yuan $^{2}$ , Hao Tang $^{3\dagger}$ , Yanzhi Wang $^{1}$
$^{1}$ Northeastern University,  $^{2}$ University of Georgia,  $^{3}$ Carnegie Mellon University  $\{\text { dong.pe, kong.zhe, yanz.wang }\} @$ northeastern.edu
The bird's-eye-view (BEV) perception plays a critical role in autonomous driving systems, involving the accurate and efficient detection and tracking of objects from a top-down perspective. To achieve real-time decision-making in self-driving scenarios, low-latency computation is essential. While recent approaches to BEV detection have focused on improving detection precision using Lift-Splat-Shoot (LSS)-based or transformer-based schemas, the substantial computational and memory burden of these approaches increases the risk of system crashes when multiple on-vehicle tasks run simultaneously. Unfortunately, there is a dearth of literature on efficient BEV detector paradigms, let alone achieving realistic speedups. Unlike existing works that focus on reducing computation costs, this paper focuses on developing an efficient model design that prioritizes actual on-device latency. To achieve this goal, we propose a latency-aware design methodology that considers key hardware properties, such as memory access cost and degree of parallelism. Given the prevalence of GPUs as the main computation platform for autonomous driving systems, we develop a theoretical latency prediction model and introduce efficient building operators. By leveraging these operators and following an effective local-to-global visual modeling process, we propose a hardware-oriented backbone that is also optimized for strong feature capturing and fusing. Using these insights, we present a new hardware-oriented framework for efficient yet accurate camera-view BEV detectors. Experiments show that HotBEV achieves a  $2\% \sim 23\%$  NDS gain, and  $2\% \sim 7.8\%$  mAP gain with a  $1.1 \times 3.4 \times$  speedups compared to existing works on V100; On multiple GPU devices such as GPU GTX 2080 and the low-end GTX 1080, HotBEV achieves  $1.1 \times 6.3 \times$  faster than others. The code is available at HotBEV.
Recently, there has been a growing interest in 3D object detection (also known as BEV perception tasks) from multi-camera images, especially in the context of autonomous driving. While LiDAR-based methods have made remarkable progress, camera-only-based approaches have gained extensive attention due to their lower expenses and ability to identify color-based road elements such as traffic lights and signs [1]. Mainstream camera-only-based approaches for multi-camera 3D object detection tasks can be broadly classified into two research domains: 1) 3D reconstruction detectors, which project 2D features from the image view into the bird's-eye view (BEV) to extract intrinsic and extrinsic information [2,3,4,5,6,7]. However, these approaches suffer from degradation due to the inaccuracy of the depth information predicted from 2D features. 2) 3D projection detectors,
Figure 1: (a) Comparison with previous backbone methods in real-world object detection. HotBEV captures the global/local information better than others. (b) HotBEV aims to search for specialized model designs for target GPU devices.
which encode information such as 3D position and object queries into 2D features and then sample corresponding features for end-to-end 3D bounding box prediction [8, 9, 11]. These approaches overcome the problem of inaccuracies in the BEV features caused by the prediction of depth values or distributions. In particular, 3D projection detectors are considered one of the dominant future multi-view BEV perception techniques.
Despite the impressive performance exhibited by 3D projection detectors, their on-device inference speed is limited by various factors. i) Detectors employing a CNN-based backbone often demonstrate inferior detection performance compared to those with a transformer-based backbone [10]. Additionally, transformers are known to possess computation and memory complexities [11], which obstruct scalability, particularly when deploying them on resource-limited devices. ii) Many prior works [2,7,8] prioritized efficient model design by only relying on hardware-agnostic metrics like computation FLOPs, neglecting to account for the actual hardware's performance, such as the actual inference latency. Current detectors lack optimizations for specific hardware deployments, including factors like memory access cost, degree of parallelism, and compiler characteristics,
Figure 2: The trade-off between performance (mAP/NDS) and hardware efficiency (FPS) for different methods on the nuScenes test dataset. HotBEV outperforms SOTA efficient transformer-based detectors on both precision and efficiency. Models are evaluated on NVIDIA V100.
all of which significantly impact hardware throughput during inference. The quadratic memory complexity associated with original self-attention operations further exacerbates the disparity between theoretical FLOPs and actual hardware performance. Notably, most current in-vehicle chips adopt the GPU architecture, which poses greater challenges and requires further investigation despite its essential role. iii) To bridge the gap between the theoretical and practical efficiency of deep models, researchers have started considering the real latency in the network design phase. Hardware profiling [12, 13], which is generally utilized to estimate on-device speed, is dedicated and time-consuming. And profiling should be redone once the model structure, size, and hardware type change. Thousands of measurements are required for each operation to ensure data correctness. In contrast, there is a need for a practical, plug-and-play model capable of efficiently estimating inference latency, specifically designed for general GPU architectures.
In this paper, we propose a hardware-oriented transformer-based framework (HotBEV) for camera-only 3D detection tasks, which achieves both higher detection precision and remarkable speedup across both high-end GPUs and low-end GPUs (see Figure 1). Firstly, we propose a theoretical latency prediction model by considering the algorithm, the scheduling strategy, and the hardware properties. Given a target GPU, we directly use the latency, rather than the computation FLOPs, to guide our algorithm design. Then we perform a latency breakdown of major modules in classic camera-only detectors and figure out that the backbone is usually the speed bottleneck. After benchmarking
Figure 3: Device speed breakdown. Left: The latency breakdown of representative Camera-only Detectors (Input resolution: 3x256x704 for PETRv2, others 3x900x1600). Middle and Left: Results are obtained on NVIDIA V100, GeForce GTX 2080 Ti, and GeForce GTX 1080 Ti. The on-device speed for frequently used backbone and various operators is reported. (Input resolution: 3x224x224) standard backbones on high-end (V100) and low-end (GTX 1080 Ti) GPUs, we propose efficient operators and fusion techniques for model on-device implementation. Based on these operators and the process of vision modeling, we design a hardware-oriented backbone with strong feature enhancement by information interaction between model stages. Then we extend the latency-aware design methodology to other parts, such as image embedding and decoder, and propose the basic design paradigm of HotBEV. Finally, guided by the latency prediction model, we generate the family of HotBEV through a standard search algorithm. Experiments show that our proposed framework can achieve a  $2\% \sim 23\%$  NDS gain and  $2\% \sim 7.8\%$  mAP with a  $1.1 \times \sim 3.4 \times$  speedups compared to existing works on V100 (see Figure ②). On multiple GPU devices such as GPU GTX 2080 HotBEV can reach  $1.1 \times \sim 6.3 \times$  faster than other models; for the low-end GTX 1080, our framework can achieve  $1.1 \times \sim 4.9 \times$  faster than others. Overall, our contributions are summarized as follows:
- We propose a latency-aware design methodology for 3D object detection based on BEV perception.
- We propose a hardware-oriented backbone that excels at capturing and fusing features.
- According to our analysis of efficient operators and strong feature modeling, we propose a new hardware-oriented framework for the BEV detector.
- We conduct experiments to showcase the superior inference accuracy of HotBEV compared to SOTA efficient BEV detectors while also highlighting its significant hardware efficiency.
Current multi-view 3D object detectors can be divided into two schemas: LSS-based schema [3] and transformer-based schema.
LSS-based schema. BEVDet [2] is the first study that combines LSS and LiDAR detection head which uses LSS to extract BEV feature and LiDAR detection head to propose 3D bounding boxes. By introducing previous frames, BEVDet4D [5] acquires the ability of velocity prediction. For the above works, the models are complex, so many hardware platforms do not support some inside operators.
Transformer-based schema. BEVDepth [2] uses LiDAR to generate depth GT for supervision and encodes camera intrinsic and extrinsic parameters to enhance the model's ability of depth detection. DETR3D [9] extends DETR [14] into 3D space, using a transformer to generate 3D bounding boxes. Following the DETR3D, PETR [8] converts 3D coordinates into 3D position embedding to perceive the 3D scene. And BEVFormer [1] uses deformable transformer [15] to extract features from images and cross attention to link the characteristics between different frames for velocity prediction. Although better performance is achieved through these works, the computation cost (e.g., over 170 GFLOPs) or realistic speed has yet to be optimized. We are the first to work on transformer-based 3D detectors to explore comparable performance and fast inference speed on diverse hardware platforms while maintaining acceptable detection accuracy.
Hardware-aware network design. Several existing works have addressed the issue of realistic latency during the network design phase, exploring two distinct directions. Firstly, some researchers evaluate speed directly on targeted devices and derive guidelines for developing hand-designed efficient models, as demonstrated by [16]. Secondly, others employ the technique of neural architecture search (NAS) to search for fast models, exemplified by [17]. Nevertheless, conducting speed tests on various hardware platforms for our proposed structures can be exceedingly labor-intensive due to the extensive range of candidates and their corresponding properties. Additionally, obtaining precise
results for small-scale structures on low-end devices presents its own set of challenges. In contrast, our latency prediction model directly estimates the inference speed on specific computing platforms, circumventing these difficulties.
In this section, we present our approach to efficiently and accurately predict the on-device latency of an architecture on the target device using the latency prediction model. Additionally, we detail efficient operators and fusion implementation techniques. According to the process of vision modeling, we introduce an efficient embedding module and a hardware-oriented backbone that excels at capturing and fusing features. Then our proposed basic design paradigm for HotBEV is based on these efficient operators, the process of vision modeling, and the detection flow of DETR3D [9]. Finally, guided by the latency prediction model, we generate a family of models known as HotBEV through a standard search algorithm [18]. Note that we also adapt the Temporal Aligned Module (TAM) in [19] to efficiently improve performance and robustness.
3.1 Latency-aware Design
3.1.1 Latency Prediction Model
We introduce a novel latency prediction model, denoted by  $E$ , which enables direct prediction of the latency of runtime design choices on any target GPU. This allows for the efficient identification of optimal model settings on corresponding platforms. Specifically, the latency predictor  $E$  considers the hardware properties  $H$ , layer type  $T$ , channel dimension  $C$ , and input granularity  $G$  as input for each design choice and outputs the predicted latency of the block as  $l = E(H, T, C, G)$ .
Figure 4: Hardware Modeling. The memory system includes 1) off-chip memory, 2) on-chip global memory, and 3) memory in the PEs.
Hardware model. We model a hardware device as multiple processing engines (PEs), allowing for parallel computation with varying degrees of parallelism. As illustrated in Figure 4, we represent the memory system using a three-level structure [20, 21], which includes: 1) off-chip memory, 2) on-chip global memory, and 3) memory in the PEs. This hardware model enables accurate prediction of the latency of data movement and computation.
Latency prediction modeling. It includes three steps: 1) Input/output shape definition. The initial step is to calculate the input and output shapes. 2) Operation mapping to hardware. Based on our hardware model, we first divide the output feature map into multiple tiles and assign each tile to a separate PE for parallel processing. 3) Latency estimation. We evaluate the latency of each tile, which comprises both data movement and computation latency:  $l = l_{data} + l_{compute}$ . For  $l_{data}$ , we leverage our hardware model (see Figure 4) and compute the sum of input and output data movement latencies as  $l_{data} = l_{in} + l_{out}$ . These latencies are estimated based on hardware bandwidth and input granularity  $G$  (equivalent to resolution scale). We assume that each PE moves the required input feature maps and weights just once to compute an output tile. For  $l_{compute}$ , we use the maximum throughput of FP32 computation on PEs and the FLOPs required to compute a single output tile. The total computation latency can be determined by considering the number of tiles and PEs. We test three types of hardware devices, NVIDIA V100, NVIDIA GTX 2080, and NVIDIA GTX 1080. For a more detailed description of our latency prediction model, please refer to Appendix A.
3.1.2 Efficient Operators and Fusion Techniques of Model Implementation
The development of efficient network architectures for resource-limited devices has greatly benefited from reduced parameters and floating-point operations (FLOPs) and improved accuracy. However, conventional efficiency metrics, such as FLOPs, overlook memory cost and degree of parallelism. In this study, we aim to improve network runtime speed and detection performance by identifying and modifying building blocks that are not hardware-friendly. To achieve this, we first perform a latency breakdown of major modules in classic camera-only detectors, DETR3D, PETR [8], PETRv2 [19],
BEVDeT [4], and BEVFormer [1], which reveals that the backbone is usually the speed bottleneck  $(66\% \sim 78\%)$ . Therefore, we deploy common backbones on high-end (V100) and low-end (GTX 1080 Ti) GPUs and benchmark their speed, as illustrated in Figure 3. We then introduce efficient operators and fusion techniques to improve the speed and detection performance of these networks.
Convolutional modulation for efficient global modeling. Instead of calculating the similarity score matrix (attention matrix [22]), we simplify self-attention by modulating the value  $V$  with convolutional features as Figure 5(c). Our approach uses convolutional modulation rather than self-attention to build relationships since they are more memory-efficient, particularly when processing high-resolution images. Due to the modulation operation, our method differs from traditional residual blocks and can adapt to the input content.
Fusing BatchNorm into the preceding fully-connected layer. After analyzing various backbones as shown in Figure 3, we found that LayerNorm (LN) accounts for approximately  $10\%$  to  $15\%$  of the network's total latency. Dynamic normalization techniques like LN gather running statistics during inference, resulting in delays that impact speed. On the other hand, BatchNorm (BN) is more memory and computation-efficient, as it is fused with the preceding fully-connected layer, reducing data movement and computation load (see Figure 4). To this end, we modified the WMSA/SWMSA in Swin [10] by replacing LN-Linear with Linear-BN, which we refer to as  $\mathrm{WMSA}_{bn} / \mathrm{SWMSA}_{bn}$ . By switching to a BN-based design, we achieved a  $13\%$  to  $15\%$  speedup with no significant loss in accuracy (less than  $0.3\%$ ). Based on the accuracy analysis in Appendix A, one advantage of the window-based self-attention on the efficient implementation compared to other transformer architectures: It can replace the LayerNorm with BatchNorm and then perform the fusing technique with acceptable accuracy. And replacing the LayerNorm directly with the BatchNorm in the original self-attention is difficult since the model training cannot converge. Therefore, we consider  $\mathrm{WMSA}_{bn} / \mathrm{SWMSA}_{bn}$  as our design candidates.
Fusing multiple branches into one single branch in reparameterized CNNs. Multi-branch structures come with increased data movement cost, as the activation values of each branch are saved into PE memory or on-chip memory (if the PE memory is insufficient) to compute the subsequent tensor in the graph. Additionally, the synchronization cost arising from multiple branches impacts the overall runtime [23]. To address these challenges, we use RepCNN [24] as a network component, which fuses multiple branches into more single-branch substructures during inference. This approach enables even distribution of computation among multiple PEs, preventing imbalanced computation overheads associated with multiple branches. The resulting operator fusion improves memory access and parallel computation on multiple PEs.
3.2 Architecture Design
We present HotBEV, a hardware-oriented framework for multi-view 3D detection. Given a set of  $I = \{I^i \in \mathbb{R}^{3 \times H \times W}, i = 1, \dots, N\}$ , our framework leverages a hardware-oriented backbone (HOB) to extract 2D multi-view features,  $F^{2d} = \{F_i^{2d} \in \mathbb{R}^{C \times H \times W}, i = 1, \dots, N\}$ . The coordinates generator generates 3D coordinates, which are aligned with the coordinate system of the frame  $t$  using the temporal aligned module (TAM) concerning the previous frame  $t - 1$ . Next, the 2D features and 3D coordinates from adjacent frames are concatenated and fed into the 3D position encoder to obtain 3D-aware features. Our hardware-oriented decoder uses these features as keys and values, which interacts with detection queries initialized from standard learnable 3D anchor points with a small MLP network. Finally, the updated queries are input to the detection head for the final prediction. For coordinates generator and position encoder, please refer to [8].
3.2.1 Hardware-Oriented Backbone with Strong Feature Enhancement
Hardware-oriented backbone. Based on our GPU breakdown [3], we found that the backbone consistently caused the most latency. To maintain detection precision while removing this speed bottleneck, we propose a powerful transformer encoder for feature capturing and fusing, as shown in the backbone design in Figure 5. We divide the backbone into four stages  $S$ , following the granularity of data flow in ResNet [25]. Since features scale from the local to the global visual receptive field, we introduce the HOB block design as shown in Figure 5(a). In each HOB block, we use several consecutive local-wise attention mechanisms to extract local information (texture-level semantics), followed by global attention to enhance global information (abstract-level semantics) in the feature map. Furthermore, we insert a semantic-augmented module (consisting of an upsampling layer
(a) Overall Backbone
(b) HOB Block
Figure 5: The design of our proposed HotBEV. The workflow follows the DETR3D, and the main components include that (a) Overall backbone: includes four stages, where feature size scales along the stage. To enhance the semantic information of low-level features, one semantic-augmented module (SAM) is added at the end of each stage. (b) Design of the HOB block, which contains multiple local/global attention layers. (c) The convolutional modulation for efficient global modeling. (d) Hardware-Oriented Decoder.
(d) Decoder
and global attention) into every two consecutive HOB blocks (except between Stage 1 and 2) to further enhance low-level semantics in the current stage, as shown in Figure [12] Notably, unlike other methods such as [10, 25], we simultaneously enhance texture-level and abstract-level information in each stage. To reinforce texture-level semantics, we leverage information interaction between stages, not just between layers in the same stage. By using the efficient operators described in Section [3.1.2] we provide the "two-phase design space" (DS) of the HOB backbone:
$$
\mathrm {D S} _ {i, \text {l o c a l}, s = 1, 2, 3,} ^ {1} \in \left\{\operatorname {R e p C N N} ^ {i}, \operatorname {W M S A} _ {b n} ^ {i}, \operatorname {S W M S A} _ {b n} ^ {i} \right\},
$$
$$
\mathrm {D S} _ {i, \text {l o c a l}, s = 4,} ^ {2} \in \left\{\mathrm {W M S A} _ {b n} ^ {i}, \mathrm {S W M S A} _ {b n} ^ {i} \right\}, \tag {1}
$$
$$
\mathrm {D S} _ {i, g l o c a l} ^ {1, 2} \in \{\text {C o n v M o d u l a} \},
$$
where local represents the candidates of local-wise attention while global for the global attention; the 1st phase covers  $S_{1}$ ,  $S_{2}$ ,  $S_{3}$  of the backbone, and the 2nd phase for  $S_{4}$ ;  $i$  denotes the  $i^{th}$  block.
Image embedding module. Figure [3] reveals that patch embedding is a speed bottleneck on multiple platforms due to its reliance on a non-overlapping convolutional layer with large kernel size and stride. Unfortunately, most compilers and acceleration techniques, such as Winograd, do not support this type of layer well. To address this issue, we propose a faster downsampling method using a convolution stem, which consists of three hardware-efficient  $3 \times 3$  convolutions with a stride of 2. To obtain input embeddings  $L_{0}$  of size  $\frac{H}{G} \times \frac{W}{G} \times C$  for an input image  $x \in \mathbb{R}^{H \times W \times 3}$ , we divide it into  $\frac{H \times W}{G \times G}$  patches and feed them to the convolution stem:
$$
l _ {0} ^ {\frac {H}{G} \times \frac {W}{G} \times D _ {1}} = \operatorname {P a t c h E m b e d} \left(x ^ {H \times W \times 3}\right). \tag {2}
$$
3.2.2 Hardware-Oriented Decoder
To alleviate the convergence difficulties in the 3D scene, we initialize a set of learnable 3D anchor points with a  $0 \sim 1$  uniform distribution. Then, we input the coordinates of the 3D anchor points into a small MLP network with two linear layers of Sigmoid function in between and generate the initial object query  $Q_{0}$ . The decoder is applied to predict the final abstract semantics and generate the bounding box. Our decoder consists of six global attention blocks and is divided into two parts, as shown in Figure 5(d). The front three layers take object queries as the only input and perform the self-attention computation, aiming at separating different objects as [14]; The back three layers take object queries as queries and image features as keys and values. As shown in Figure 12 they perform the cross-attention between object queries and image features to extract the content and position of