Camera stream architecture: From multi-view images to BEV space
As our framework has the capability to incorporate any camera streams, we begin with a popular approach, Lift-Splat-Shoot (LSS) philion2020lift .
As the LSS is originally proposed for BEV semantic segmentation instead of 3D detection, we find out that directly using the LSS architecture has inferior performance, hence we moderately adapt the LSS to improve the performance (see Sec. 4.5 for ablation study).
In Fig. 2 (top), we detail the design of our camera stream in the aspect of an image-view encoder that encodes raw images into deep features, a view projector module that transforms these features into 3D ego-car coordinate, and an encoder that finally encodes the features into the bird’s eye view (BEV) space.

Image-view Encoder aims to encode the input images into semantic information-rich deep features. It consists of a 2D backbone for basic feature extraction and a neck module for scale variate object representation.
Different from LSS Philion2020LiftSS which uses the convolutional neural network ResNet HeZRS16 as the backbone network, we use the more representative one, Dual-Swin-Tiny liang2021cbnetv2 as the backbone.
Following Philion2020LiftSS , we use a standard Feature Pyramid Network (FPN) LinDGHHB17 on top of the backbone to exploit the features from multi-scale resolutions. To better align these features, we first propose a simple feature Adaptive Module (ADP) to refine the upsampled features. Specifically, we apply an adaptive average pooling and a 1\times 1 convolution for each upsampled feature before concatenating. See Appendix Sec. A for the detailed module architecture.

View Projector Module. As the image features are still in 2D image coordinate, we design a view projector module to transform them into 3D ego-car coordinate.
We apply 2D \rightarrow 3D view projection proposed in Philion2020LiftSS to construct the Camera BEV feature. The adopted view projector takes the image-view feature as input and densely predicts the depth through a classification manner. Then, according to camera extrinsic parameters and the predicted image depth, we can derive the image-view features to render in the predefined point cloud and obtain a pseudo voxel V\in R^{X\times Y\times Z\times C} .

BEV Encoder Module. To further encode the voxel feature V\in R^{X\times Y\times Z\times C} into the BEV space feature ( \mathbf{F}_{\text{Camera}}\in R^{X\times Y\times C_{\text{Camera}}} ), we design a simple encoder module.
Instead of applying pooling operation or stacking 3D convolutions with stride 2 to compress z dimension, we adopt the Spatial to Channel (S2C) operation abs-2204-05088 to transform V from 4D tensor to 3D tensor V\in R^{X\times Y\times(ZC)} via reshaping to preserve semantic information and reduce cost.
We then use four 3\times 3 convolution layers to gradually reduce the channel dimension into C_{\text{Camera}} and extract high-level semantic information.
Different from LSS Philion2020LiftSS which extracts high-level features based on downsampled low-resolution features, our encoder directly processes full-resolution Camera BEV features to preserve the spatial information.