LaRa: Latents and Rays for Multi-Camera Bird’s-Eye-View Semantic Segmentation
Authors: Florent Bartoccioni
Eloi Zablocki
Andrei Bursuc
Patrick Perez
Matthieu Cord
Karteek Alahari
Venue: CoRL 2022 Poster (2022)
Keywords: bird’s eye view semantic segmentation
encoder-decoder transformers
autonomous driving

Abstract: Recent works in autonomous driving have widely adopted the bird’seye-view (BEV) semantic map as an intermediate representation of the world. Online prediction of these BEV maps involves non-trivial operations such as multi-camera data extraction as well as fusion and projection into a common topview grid. This is usually done with error-prone geometric operations (e.g., homography or back-projection from monocular depth estimation) or expensive direct dense mapping between image pixels and pixels in BEV (e.g., with MLP or attention). In this work, we present ‘LaRa’, an efficient encoder-decoder, transformer-based model for vehicle semantic segmentation from multiple cameras. Our approach uses a system of cross-attention to aggregate information over multiple sensors into a compact, yet rich, collection of latent representations. These latent representations, after being processed by a series of selfattention blocks, are then reprojected with a second cross-attention in the BEV space. We demonstrate that our model outperforms the best previous works using transformers on nuScenes. The code and trained models are available at https://github.com/valeoai/LaRa.


LaRa: Latents and Rays for Multi-Camera Bird's-Eye View Semantic Segmentation
Florent Bartoccioni Valco.ai Inria
Éloi Zablocki Valco.ai
Andrei Bursuc
Valco.ai
Patrick Pérez
Matthieu Cord
Karteek Alahari
Valeo.ai
Valeo.ai
Inria
Sorbonne Université
Abstract: Recent works in autonomous driving have widely adopted the bird's- eye-view (BEV) semantic map as an intermediate representation of the world. Online prediction of these BEV maps involves non-trivial operations such multi-camera data extraction as well as fusion and projection into a common top- view grid. This is usually done with emrr--prnne geometric operations (e.g., ho- mography or back-projection from monocular depth estimation) or expensive di- rect dense mapping between image pixels and pixels in BEV (e.g.. with MLP or attention). In this work, we present 'LaRa', an efficient encoder-decoder, transformer-based model for vehiele semantic segmentation from multiple cam- eras. Our approach uses system of cross-attention to aggregate information over multiple sensors into compact, yet rich, collection of latent representa- tions. These latent representations, after being processed by a series of self- attention blocks, are then reprojected with second cross-attention in the BEV space. We demonstrate that our model outperforms the best previous works us- ing transformers on nuScenes. The code and trained models are available https: //gi thub.
Keywords: bird's eye view semantic segmentation; encoder-decoder transformers
Introduction
To plan and drive safely, autonomous cars need accurate 360-degree perception and understanding of their surroundings from multiple and diverse sensors, e.g., cameras, RADARs, and LiDARs. Most of the established approaches tardily aggregate independent predictions from each sensor [1, 2, 3]. Such a late fusion strategy has limitations for reasoning globally at the scene level and does not take advantage of the available prior geometrie knowledge that links sensors. Alternativel, the bird's "od''-eye-view'ss 's (BEV) representational space, a.k.a. top-view occupancy grid, recently gained considerable interest within the community. BEV appears
suitable and natural space fuse multiple views [4, 5] or sensor modalities [6, 7] and to capture semantic, geometric, and dynamic information. Besides, is widely adopted choice for downstream driving tasks including motion forecasting [5, 8, 9, 10] and planning [11, 12, 13, 14]. In this paper, we focus on BEV perception from multiple cameras. The online estimation of BEV representations is usually done by: (i) impos- ing strong geometric priors such as flat world [15] or correspondence between pixel columns and BEV rays [16], (ii) predicting depth probability distribution project back in BEV [4, 5], a system subject to compounding errors, or, (iii) learning a costly dense mapping between multi-camera features and the BEV grid pixels [17].
pixels to lift from 2D 3D and
Here, we depart from these dominant strategies and introduce 'LaRa', a novel transformer-base model for vehicle segmentation from multiple cameras. In contrast prior works, propose to use a latent 'internal representation' instantiated as a collection of vectors. Fusing multiple views into a
Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK 38000 Grenoble, France
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Imsgge
BEY
init.
MLP
Riyer
35G
CA
BEV
Latest
ML.Paxy
representation
BEV sgmentation
aed dircetio
Rays ego-frame
Semantic features (green) are extracted from the images with a shared CNN and concatenated with ray embeddings (multi-colored) that inform about geometric infor- mation to spatially relate pixels within and across cameras. This representation is then fused into compact latent representation through one cross-attention (CA) and L self-attention (SA) layers (yellow). The final BEV map is obtained by querying the latent representation with a cross-attention and then refined with BEV CNN (red). @ denotes concatenation. The orange letters indicate tensor dimensions. K Q, and V are the Key, Query, and Value of the cross-attentions.
Figure 1:
LaRa overview.
compact latent space comes with several benefits. First, it provides an explicit control on the memory and computation footprint of the model, instead of the quadratic scaling of the full mapping between multi-camera features and the BEV grid pixels [17]. By design, the number of latents that we use much smaller compared to the spatial resolution of the BEV grid, enabling a highly-efficient aggregation of information at the latent-level while exploiting spatial cues within and across camera views. Moreover, we also hypothesize that discarding error-prone modules the pipeline such as depth estimation [4. 5] boost model accuracy and robustness. Finally, at the full-scale BEV resolution bypassing noisy upsampling operations. This is infeasible, within reasonable computational budget, for prior works restricted to coarser BEV grids they map densely between all the image and BEV grid pixels [17]. Besides, as an orthogonal contribution, we augment input features with ray embeddings that encode geometric relationships within and across images. We show that such spatial embeddings, encoding prior geometric knowledge, help guide the cross-attention between input features and the latent vectors.
directly predict
Our approach extensively validated against prior works on the nuScenes [10] dataset. We sig- nificantly improve the performance the vehicle segmentation task, outperforming recent high- performing models [4, 17]. Moreover, we show interesting properties our cross-atee cross-attention, which naturally stitches multiple cameras together. We also perform several ablation and sensitivity studies of our architecture with respect to hyper-parameter changes. Overall, LaRa is a novel model that lears the mapping from camera views to bird's s-eee-veew for the task of vehicle semantic segmen- tation. In summary, our contributions are as follows:
We encode multiple views into compact latent space that enables precise control on the model's memory and computation footprint, decoupled from the input size and output resolution.
We augment semantic features with spatial embeddings derived from cameras' calibration param- eters and show that strongly helps the model learn to stitch multiple views together.
Our architectural contributions are validated on nuScenes where we reach new SOTA results.
Related work
2.1 BEV semantic segmentation
Models for BEV segmentation are typically structured in two parts. They first extract features of each camera and then project them into a common top-view grid, called the bird's-eye-views There are different strategies for this projection, which can be grouped into the following categories.

---

IPM-based. Inverse perspective mapping (IPM) defines the correspondence between the camera and the ground planes as homography matrix. IPM makes strong assumptions that the world is planar and the cameras" horizontal axes are parallel to the ground. Early works [18, 19] apply it directly to raw camera pixels or features. This approach suffers from blurring and stretching artifacts for distant objects (as they have fewer pixels in the camera view) and objects with height (as they violate the planar world assumption). To alleviate these shortcomings, a generative adversarial network [20] or training a BEV decoder with synthetic ground-truth [15] has been used to refine the IPM projection.
'Lift-splat" -based:: guiding with depth. Using depth information to lift features from 2D to 3D and then 'splatting' them in BEV space recently gained popularity for its effectiveness and sound geo- metric definition. Among the formulations of depth estimation for BEV projection [2, 4, 5, 21, 22], estimating depth probabilities along camera rays appears to perform the best [4, 5]. However, such a strategy, depth being the most influential factor [23], subject to compounding errors. Inaccuracies in depth prediction will propagate into the BEV features, which themselves can be erroneous.
Implicitly learned with dense networks. An alternative to explicit geometric projection is to learn the mapping from data. For instance, VPN [24] uses an MLP to make a dense correspondence between pixels in the camera views and BEV. These methods rely on such expensive operations and do not use readily available spatial information given by the calibrated camera rig capturing the images. The BEV projection must be entirely learned, and as determined by training data, can hardly apply to new settings with slightly different camera calibrations. Alternatively, PON [16] builds on the observation that a column in the camera image contains all the information of the corresponding ray in BEV: it first encodes each column into a feature vector, which is then decoded into ray along the depth dimension. However this relies on two implicit assumptions: (i) the camera follows pinhole projective model, and (ii) it is horizontally aligned with the ground plane.
Implicitly learned with transformer architectures. The attention system the core of transformer architectures [25, 26, 27, 28] allows learning of long-range dependencies and correspondences ex- plicitly. These architectures have recently been employed for the BEV semantic segmentation task, yielding among the best pest-performing methods [17, 29, 30J. Nonetheless, a direct cross- -ross-attention [25] between camera images and the BEV grid is computationally expensive. BEVFormer [29] allevi- ates this issue by only gross-attending BEV pixels with cameras in which the BEV pixel is visible and by replacing the heavier multi-head attention [25] with deformable attention [31]. CVT [17]
keeps the vanilla multi-head cross-attention [25] but applies it between low-resolution camera fea-
ture maps and a small BEV grid which is then upsampled to reach the final resolution. GitNet [30]
restrains the cross-attentions column-ray pairs making the same original implicit assumptions as PON [16]. Our proposed model LaRa belongs to this category as learns the BEV representation with a transformer architecture. On the other hand, our attention scheme does not impose strong geometric assumptions while still being efficient enough to attend to a full-resolution BEV grid.
2.2 Incorporating geometric priors in Transformers
Since transformer architectures are permutation-invarian spatial relationships between image re- gions are lost if no precautions are taken. A standard practice to retain this spatial knowledge is to add a positional embedding to the input of attention layers [25]. A popular approach is to encode the position of pixels with sine and cosine functions of varying frequencies [25, 27, 28] applied over the horizontal and vertical axes. An alternative solution to induce spatial awareness the model is to concatenate x. y positions to feature maps fed to convolutional layers [32].
Related our ray embedding proposition, recent works [33, 17] embed the parameters of the cali- brated cameras in the image features, improving training efficiency and segmentation performance. Similar LaRa, IIB [33] also encodes the camera center and ray direction in the input feature sequence, but it applies it to depth estimation on image pairs in an indoor environment. Further- more Yifan et al. [33] embed the origin and direction of rays into Fourier features, which can become memory intensive depending on the number of frequency bands and also introduces addi- tional hyper-parameters to tune. CVT [17] adds up a ray direction embedding to the input feature sequence, but, differently from ours, uses the camera center embedding in the BEV query. This requires a BEV query and 'cross-view attention' operation per camera, increasing the memory and computational footprint, thus limiting the maximum resolution of the BEV query.
3

---

3 LaRa: Our Latents and Rays Model
Given multiple cameras observing the scene, our goal is estimate a binary occupancy grid [34] ÿ E {0, 1 X of size fbber X Wbev E N² for vehicles in the surroundings of the ego car. We propose 'LaRa' a transformer-De based architecture to efficiently aggregate information gathered from multiple cameras into compact latent representation before expanding back into the BEV space. Besides, as we believe that the geometric relationship between cameras should guide the fusion across each camera view, we propose to augment each pixel with the geometry of the ray that cap- tured it. The LaRa architecture is illustrated in Figure
3.1 Input modeling with geometric priors
We consider C' cameras described by (42, Kx, RR tx 1x) with L E I2xxxxx3 x3 the image pro- duced by camera k. Kg E R3x3 the intrinsics, R₂ E R3x3 and ig E R the extrinsic rotation and translation respectively. From these inputs, complementary types of information are extracted: semantic information from raw images and geometric cues from the camera calibration parameters.
Semantic information from raw images. A shared image-encoder E extracts feature maps for each image FE= = E((xk) E /xxxee Following [4, 5], we instantiate E with a pretrained Efficient- Net [35] backbone to produce the multi-camert features. These spatial feature maps in Cxxxxxxxx then rearranged sequence of feature vectors, in (C
Leveraging geometric priors. To enrich camera features with geometric priors, commonly used sine and cosine spatial embeddings [25, 27, 28] are ambiguous in presence of multiple cameras. A straightforward solution would be to use camera-dependant learnable embeddings in addition to the Fourier embeddings disambiguate between cameras. However, in setting, argue that the geometric relationship between cameras, which is defined by the structure of the camera rig, is crucial to guide the fusion of the views. This motivates our choice to leverage the cameras' extrinsics and intrinsics to encode the position and orientation of each pixel in the vehicle ago-frame.
More precisely, we encode the camera calibration parameters by constructing the viewing ray for each pixel of the cameras. Given a pixel coordinate x E 2R22 within a camera image Tt, the direction d..rr) E R of the ray that captured x is computed with:
d (x) = R -
(1)
where are the homogeneous coordinates of x, and (x(x)) expressed in ego-coordinates. The origin of the ray da (z) is the camera center given by 4x-
Then, to fully describe the position and the orientation of the ray that captured pixel x, the embedding ray, (z) E Rd computed as follows:
rayy (a) MLP my (IN dx(x)),
where @ is a concatenation operation and MLPray a 2-layer MLP with GELU activations [36]. The computation is consistent within and across cameras and exhibits an interesting property: overlap- ping regions for two cameras with the same optical center have the same ray embedding. Note that the intrinsics are scaled according to the difference in resolution between 44 and Fk.
As shown in Figure 1, the final input vector sequence, in ((Chw))(d+t) is produced by concatenat- ing each the how feature vectors F& (x) E R with its geometric embedding ray, (x) E Rd.
3.2 Building latent representations and deep fusion
To control the computational and memory footprint of the image-to-BEV block, we leverage find- ings from general-purpose architectures [28] and propose use an intermediate fixed-sized latent space instead of learning the quadratic all-to-all correspondence between multi- features and BEV space [17]. Formally, the visual representations F& from all cameras, along with their cor- responding geometric embeddings ray,, are compressed by rross-attention [25] into a collection of N learnable latent vectors of dimension M E and processed by a series of self-attention blocks [25] (see yellow elements in Figure 1). We stress that N & Ch which enables to fuse and process efficiently the visual information coming from all the cameras regardless of the input feature resolution or the number of cameras. Thanks to latent-based querying, this formulation decouples

---

Table 1: Intersection-ove-Uniion (IoU) for vehicle segmentation on nuScenes. 'Setting 1' refers to 100mx50m grid with a 25cm resolution and 'Setting 2' to 100m 100m grid with 50cm resolution. For training and validation, vehicles are considered only if their visibility level is above predefined threshold (either 0% or 40%). To compare against other works, we refer the reader to Lift-splat [4] and CVT (17].
visibility > 0%
visibility > 40%
Method
Conference
Setting 2
Setting
Setting 2
Lift-splat [4] CVT [17]
ECCV'20 CVPR'22
32.1
37.5
36.0
LaRa (ours)
35.4
41.4
38.9
the network's 's deep multi-view processing from the input and output resolution. Our architecture can thus take advantage of the full resolution of the BEV grid.
3.3 Generating BEV output from latents
The final step is to decode the binary segmentation prediction ÿ E {0, 1] A XWher from the la- tent space. In practice, the latent vectors are cross-attended [25] with BEV 'query" grid Q e Re XWr Xches at the final prediction resolution, with dbev E a hyper-parameter (illustrated by the red blocks in Figure 1). Each element of the query grid is feature vector encoding the spatial posi- tion the bird s-ee which specifies what information the cross-attention would extract from the latent representations. This last cross-attention yields a feature map in BEV space, in dimension hhev x Whev x 256, that further refined with a small convolutional encoder-decoder U-Net ('BEV CNN' in Figure 1) finally predict the binary aadd'seee-viees semantic map ÿ E f0, 1 J a the
Specifically, we consider a combination two types of queries: normalized coordinates in the BEV space and radial distance. Normalized coordinates encode ego-centered normalized coordinates of the BEV plane. They are obtained with:
2i
2j
1)
Qcnends [e
1,
Vi (O..... N.eev 1} x (0,
Whev - 1}. (3)
Jibev
Ubev
Normalized radial distances are simply Fuclidean distances of pixels w.r.t. the origin:
Qudiali, Qcoonts
[i, (i,
While the network could produce a similar embedding from Qeoers using MLPbcv. we find that introducing these radial embeddings along Qooods empirically improves results. Moreover, this query decoding choice compares favorably against more classical Fourier embeddings [25, 28, 33] and learned query embeddings [25, 27], as shown in Table 2.
Experiments
Dataset. We conduct experiments on the nuScenes dataset [10], which contains 34k annotated sets of frames captured by C=6 synchronized cameras covering the 360 field of view around the ego vehicle. The extrinsics and intrinsics calibration parameters are given for all cameras in every scene. Raw amnotations come in the form of 3D bounding boxes that are simply rendered in the discretized top-down view of the scenes to form the ground-truth for our binary semantic segmentation task.
Precise settings for training and validation. With no established benchmarks to precisely compare model's 's performances, there are almost as many settings as there are previous works. Differences are found at three different levels: The resolution of the output grid, the level of visibility used to select objects as part of the ground-truth, and the task considered. task of binary semantic segmentation of all vehicles (cars, bicycles, trucks, etc.) [4, 17). This choice made have fair and consistent comparisons with our baselines [4, 17], however, it should be noted that our model is not constrained this setting. To enable and case future comparison,
this paper, we address the
5

---

Table Ablation study for the input and output query embedding. Training and evaluation are done in Setting (100mx 100m at 50em resolution), with a visibility > 0%.
Input geometry embedding
Output query embedding
Cam, rays
Cam, idx
Fourier
IoU
Radial dist. Norm, coords
Fourier Learned
IoU
X
35.4
x
x
35.4
V
34.4
V
x
x
35.1
X
32.3
X
X
V
x
30.6
X
30.5
x
x
v
21.8
have published our code?. We also present additional settings in the supplementary material. In all the settings we considered, models are evaluated with the IoU metric.
Training and implementation details. We train our model by optimizing the Binary Cross Entropy with predicted soft segmentation maps and the binary ground-truth. Images are processed at resolution 224 x 480. We use the Adam [37] optimizer with a constant learning rate of 5e-4 and weight decay of 1e-7. We train model on 4 Tesla V100 16GB GPUs with total batch size of for 30 epochs. Training takes on average 11 hours. We use an EfficientNet- B4 [35] with an output stride of 8 our CNN image encoder. For the BEV CNN we follow Philion and Fidler [4]. MLP a 2-layer MLP producing dnev = 228-dimensional features.
4.1 Comparison with previous works
In Table 1, compare the IoU performances of LaRa against two baselines Lift-Splat [4] and CVT [17] on vehicle BEV segmentation in their respective training and evaluation setups. In all cases, improve results by significant margin. More precisely, improve by 10% compared to Lift-Splat in their settings, by 10% and 8% compared to CVT respectively in Setting and Setting 2. This suggests that model can better extract the geometric and semantic information from all cameras with a very general architecture that does not necessitate any strong geometric assumptions. Besides, when compared with CVT, we observe that LaRa obtains better results in the setting with finer resolution (+10% in Setting vs. +8% in Setting 2).
Since our attention mechanism does not rely on all-to-all attention between camera images and BEV map as CVT does, LaRa can directly decode to the final BEV resolution which helps for fine prediction high resolution.
4.2 Model ablation and sensitivity to hyper-parametee
Input and Output-level embeddings. To assess the contribution of the geometric embeddings that use, compare the different choices at both the input and output level in Table 2. As hypothesized, embedding the geometric relationship between cameras in the input is better suited for task than the generic sine and cosine spatial embeddings. The additional camera index, while performing better than Fourier feature alone, is not enough to link pixels across cameras. For the output query embedding, the combination of normalized coordinates and radial distance gives the best results. This simple choice outperforms both the Fourier features [25, 28) and learned embeddings [25, 27) that also have the disadvantage of increasing the number of parameters.
Sensitivity to hyper-parameters. To delve into the influence of sensitivity analysis in Figure where we vary the number N of latent vectors, their dimension M and the number of self-attention blocks L. We clearly observe that the performance increases with the number of latent vectors used. This is expected it is the main parameter controlling the attentional bottleneck between input and output. Such a parametrization allows for an easy tuning of the performance/memory trade-off. We observe clear correlation between the dimension M of latent vectors, the number I self-attention layers, and the obtained IoU performance. This indicates that architecture is not too sensitive to these hyper-parameters and can work efficiently with a wide range of values for these parameters. Although we obtain better results with 512 latent vectors,
we conduct
maximum of 256 stay in the same computational regime the baseline compare against; training with 512 latent vectors requires 32GB GPUs.
hhttps://giituu..co//aaaa/

---

N=84
N=128
latent dim
512
256
36
34
(1 SA layers)
[ layers)
layers)
layers)
Figure 2: Sensitivity study of LaRa to hyper-parameters. We vary the number of latent vectors
(N), their dimension (M), and the number of self-attention layers (T. and report IoU performances.
4.3 Study of attention
As quantitatively studied in Section 4.2, embedding camera rays impacts significantly the perfor- mance of LaRa. By analyzing the input-to-latent attention map, further investigate the geometric reasoning of LaRa in Figure 3. In this figure, we show two representations the attention: a re- projection of the attention in the camera-space (left) and top-view projection of the attention in polar coordinates by collapsing, i.e., averaging the vertical dimension (right). the latter, the radial distance is proportional to the attention level and shows the directions the network attends the most.
The study is conducted at three different levels. First, for a couple of one latent vector and one attention head (n 10, h = 5 and 50, h = 30), among N 256 possible latents and H 32 possible attention heads. Second, for one latent vector and the averaged attention from all attention heads (n 10, h avg and n 50, h avg). Third, for one attention head and the averaged attention over latents (n = avg, = and n = avg. h = 30). From these three settings, we note the followings: First, the attention map between one latent vector and one attention head targets a specific direction (about a 90º field view). Additionally, it can be clearly observed that the attention is continuous across cameras, proving the network is able to retrieve the pixel relationships between views. Second, while one attention head fires in a specific direction, the attention averaged over all the heads for latent vector spans over half of the scene. This allows one latent vector to extract long-range context between views with the capacity to disambiguate them. Third, the attention for one head aggregated over all the latent vectors covers all directions, suggesting that the latent vectors contain all of the directional information and that the whole scene attended across the latents. To summarize, by integrating early multi-view geometric cues instantiated by camera rays embedding (Section 3.1), we show that LaRa learns to reason across views. We also provide quantitative evidence in the supplementary material.
froat lect
Trent night
back night
10 hravg
maaii h:S5
m: h:5
front
Teont might
hack
Right
D:50 SU
haag
0:50 h:30
Figure 3: Input-to-latent attention study. Attention maps analysis for network using 256 latents and 32 attention heads. The attention for one attention head and one latent is shown on the left superimposed with RGB images. The polar plots represent the directional attention intensity for one (or the average) attention head with one (or the average) latent vector. The radial distance proportional to the attention level and shows the directions the network attends the most.
7

---

front
front
back
back
back Jeft
Sift Vis>0%
CVT vis>40%
Our vis>40%
front left
front
front right
back right
back
back
Lift-Splat vis-0%
Our vis>0%
CVT vis>40%
Our vis>40%
GT
Qualitative results on complex scenes. We show the six camera views surrounding the vehicle along with segmentation map ground-truth for reference. In the ground-truth (GT) map, vehicles are shown in blue (visibility > 40%) or purple (visisibility < 40%). The ego vehicle is located in the center and facing downwards. We show results for our two baselines [4, 17]. For fair comparison, we always compare using their respective level of visibility. Setting 2 is used.
Figure 4:
4.4 Qualitative Results
We show the segmentation results of two complex scenes in Figure For fair comparison, use our model trained with visiblity > 40% against CVT and >> 0% against Lift-Splaat Compared long distance and the other in the dark (red box). We
LaRa, CVT missed objects, one also estimate the boundaries of the vehicles better than Lift-Splat (green box). Interestingly, models trained on all vehicles (visibility > 0%) tend to hallucinate cars in occluded distant regions (highlighted with black circles in the figure).
Conclusion
We presented LaRa, which leverages wansformer-based architectures and encoder-decoder models, with respectively efficient deep cross- and self-attentions as well as an explicit control on the com- putation and memory footprint thanks to decoupling the bulk of the processing from the input and output resolution. By incorporating ray embeddings into LaRa, we augment semantic features with geometric cues of the scene and show that this leads multi-view stitching.
Limitations. Our model operates on camera inputs only. Thus, adverse conditions, e.g., with glares and darkness, its performance remains limited. To better handle these challenging situations, one avenue of improvement would be the extension of LaRa to handle complementary modalities, e.g., coming from LiDARs radars.
Broader impacts. LaRa demonstrates that the geometry and semantics complex scene can be compacted in small collection of latent vectors. We believe that this formulation would allow for efficient temporal reasoning- Currently, the temporal modeling is done the BEV space, which high resolution and mostly represents empty space [5, 29].

---

Acknowledgments
This work was supported in part by the ANR grants AVENUE (ANR-18-CE2330001)) VISA DEEP (ANR-20-CHIA-002) --000220- and MultiTrans (ANR-21-CE23-0032) was granted access to the HPC resources of IDRIS under the allocation 2021-101766 made by GENCI.
References
[1] Z. Liu, Z. Wu, and R. Toth. SMOKE: Single-stage monocular 3D object detection via keypoint estimation. In CVPR Workshop, 2020.
[2] Roddick, Kendall, and Cipolla. Orthographic feature transform for monocular 3D) object detection. In BMVC, 2019.
[3] T. Wang, X. Zhu, J. Pang, and D. Lin. FCOS3D: Fully convolutional one-stage monocular 3D object detection. In ICCV Workshop. 2021.
[4] J. Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3D. In ECCV, 2020.
[5] A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, V. Badrinarayanan, R. Cipolla, and
A. Kendall. FIERY: Future instance segmentation in bird's-eye view from surround monocular cameras. In ICCV, 2021.
[6] N. Hendy, C. Sloan, F. Tian, Duan, N. Charchut, Y. Xie, C. Wang, and Philbin. FISHING net: Future inference of semantic heatmaps in grids. In CVPR Workshop, 2020.
[7] X. Bai, 7. Hu, X. Zhu, Q. Huang, Y. Chen, H. Fu, and C.-L.. Tai. Transfusion: Robust LiDAR- camera fusion for 3D object detection with transformers. In CVPR, 2022.
[8] Ettinger, S. Cheng, B. Caine, C. Liu, Zhao, Pradhan, Y. Chai, B. Sapp. C. Qi,
Y. Zhou, Z. Yang, A. Chouard, P. Sun, J. Ngiam, V. Vasudevan, A. McCauley, J. Shlens, and
D. Anguelov. Large scale interactive motion forecasting for autonomous driving: The waymo
open motion dataset. In ICCV, 2021.
[9] M.-F. Chang, J. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays. Argoverse: 3D tracking and forecasting with rich maps. In CVPR, 2019.
[10] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. E. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuScenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
[11) W. Zeng, W. Luo, Suo, A. Sadat, B. Yang, S. Casas, and R. Urtasun. End-to-end interpretable neural motion planner. In CVPR, 2019.
[12] Casas, Sadat, and R. Urtasun. MP3: unified model to map, perceive, predict and plan. In CVPR, 2021.
[13] K. Chitta, A. Prakash, and A. Geiger. NEAT: Neural attention fields for end-to autonomous driving. In JCCV, 2021.
[14] H. Caesar, J. Kabzan, K. Tan, W. K. Fong, E. M. Wolff, A. H. Lang, L. Fletcher, O. Beijbom, and S. Omari. NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles. arXiv 2/06. (006.118/0 1110, 2021.
[15] L. Reiher, B. Lampe, and L. Eckstein. A sim2real deep learning approach for the transforma- tion of images from multiple rehicle-mounted cameras to semantically segmented image in bird's eye view. In IEEE ITSC, 2020.
[16] T. Roddick and R. Cipolla. Predicting semantic map representations from images using pyra- mid occupancy networks. In CVPR, 2020.
[17] B. Zhou and P. Krähenbühl, Cross-view transformers for real-time map-view semantic seg- mentation. In CVPR, 2022.

---

[18] M. Bertozzi, A. Broggi, G. Conte, and A. Fascioli. Experience of the ARGO autonomous vehicle. In Enhanced and Synthetic Vision, 1998.
[19] S. Sengupta, Sturgess, L. Ladicky, and P. H. S. Torr. Automatic dense visual semantic mapping from street-level imagery. In IROS, 2012.
[20] X. Zhu, Z. Yin, J. Shi, H. Li, and D. Lin. Generative adversarial frontal view bird view synthesis. In 3DV, 2018.
[21] S. Srikanth, J. A. Ansari, K. Ram, S. Sharma, J. Krishna Murthy, and K. Madhava Krishna. INFER: Intermediate representations for future prediction. In IROS, 2019.
[22] M. H. Ng, K. Radia, Chen, D. Wang. Gog, and J. E. Gonzalez. BEV-Seg: Bird's eye view semantic segmentation using geometry and semantic point cloud. arXiv 2006. 2020.
[23] A. Simonelli, S. R. Bulo, L. Porzi, P. Kontschieder, and E. Ricci. Are missing confidence in pseudo-LiDAR methods for monocular 3D object detection? ICCV, 2021.
[24] B. Pan, J. Sun, H. Y. T. Leung. A. Andonian, and B. Zhou. Cross-view semantic segmentation for sensing surroundings. In IROS, 2020.
[25] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L.. Kaiser, and I. Polo- sukhin. Attention is all you need. In NeurIPS, 2017.
[26] A. Dosovitskiy. L. Beyer. A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. De- hghami, M. Minderer, G. Heigold, S. Gelly, Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[27] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[28] A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, D. Zoran, A. Brock, E. Shelhamer, J. Henaff, M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira. Perceiver 10: A general architecture for structured inputs & outputs. In ICLR, 2022.
[29] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai. BEVFormer: Learn- ing bird's seeye-veww representation from multi-camera images via spatiotemporal transformers. arXiv 2203. 2022.
[30] S. Gong, X. Ye, X. J. Wang, E. Ding, Y. Zhou, and X. Bai. GitNet: Geometric prior-based transformation for birds-eye-view segmentation. arXiv 2204.07733. 2022.
[31] X. Zhu, W. Su, L. Lu, B. Li, Wang, and J. Dai. Deformable DETR: Deformable transformers for end-to-end object detection. In ICLR, 2021.
[32] R. Liu, J. Lehman, Molino, F. P. Such, E. Frank, A. Sergeev, and J. Yosinski. An intriguing failing of convolutional neural networks and the coordcony solution. In NeurIPS, 2018.
[33] W. Yifan, C. Doersch, R. Arandjelović, J. Carreira, and A. Zisserman. Input-level inductive biases for 3D reconstruction. In CVPR, 2022.
[34] A. Elfes. Occupancy grids: A stochastic spatial representation for active robot perception. UAI, 1990.
[35] M. Tan and Q. V. Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[36] D. Hendrycks and K. Gimpel. Gaussian error linear units (GELUs). arXiv 1606.08415, 2016.
[37] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
10


Reviews:
[{"review_id": "zYJqK4NcR_", "reviewer": "Reviewer_u5Wy", "summary": "This paper proposes an efficient transformer-based encoder-decoder model for BEV vehicle semantic segmentation from camera-only inputs. It first encodes the semantic features from input images and camera intrinsic information into a set of latent variables with a system of cross-attention. These latent features are then processed with self-attention blocks and cross-attention with BEV queries to obtain the final semantic segmentation results. Experimental results demonstrate the efficacy and efficiency of the proposed method.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and the basic idea is easy to follow.\n- Encoding the input images into a set of latent vectors can avoid the significantly increasing computational burden caused by high-resolution PV and BEV features.\n- Related works are comprehensively analyzed and compared.\n- The qualitative visualization in Sec. 4.3 is interesting.\n- Experiments show state-of-the-art results compared to previous works.\n\nWeaknesses:\n- This method is general and should be able to be applied to other tasks, such as map segmentation (not only vehicles) and 3D detection. There are more competitive entries and more baselines that can be compared in those benchmarks. More experimental results on those tracks can distinguish this paper from others better and make the conclusion more convincing.\n- Questions about the encoding of camera calibration parameters:\n\na. There is another related work required to be compared and discussed: PETR: Position Embedding Transformation for Multi-View 3D Object Detection. This work also encodes similar information from the input perspective and achieves improvements.\n\nb. How is x formulated in Eq. (1)? Except for the pixel coordinates, the depth should also be considered to make it capable of being transformed to 3D space? And why here disentangle the rotation and translation when constructing such embeddings in Eq. (1) and (2)? It should be more common to use both to directly calibrate all these points of different camera views into a unified 3D space? Here may need more explanations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes an effective framework for BEV vehicle semantic segmentation. It comprehensively analyzes previous works and improves several technical aspects over them. Experimental results can demonstrate its efficacy. Nevertheless, there are still questions in terms of the design details of camera calibration embeddings and more experiments on other relevant competitive benchmarks. I would vote for weak accept temporarily and would consider adjusting the score according to the authors' feedback on my questions.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "See weaknesses.", "reviewer_expertise": "5: The reviewer is absolutely certain that the evaluation is correct and very familiar with the relevant literature"}, "extra_text": {}}, {"review_id": "gc2GohRZa-q", "reviewer": "Reviewer_T5RB", "summary": "This paper presents a method for computing Bird's Eye View (BEV) occupancy maps. Unlike methods that explicitly use geometric information or that compare all pairs of pixels, they use an encoder-decoder transformer model that contains a number of design decisions that allow it to be efficient and accurate. They fuse camera rays and what they term \"semantic features\" from a CNN. The rays incorporate geometric information by applying an MLP to the direction and camera center. Their model uses a fixed-size latent representation inspired by general-purpose architectures like the Perceiver. This avoids the quadratic penalty incurred by naive models. They evaluate on nuScenes vehicle segmentation, where they outperform several recent works. Through ablations, they find that the different aspects of the geometry and query embeddings improve results. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Their approach is simple. The fixed-size latent is a nice way of addressing the quadratic complexity incurred by prior work.\n- The ablation experiments validate the different components of their approach.\n- The approach outperforms previous work such as CVT and Lift-splat.\n- They seem to have made an attempt to control for differences in resolution and other experimental details (although I'm not an expert on this benchmark and other reviewers may be better-suited to evaluating this). \n\nWeaknesses:\n- The evaluation is a bit limited, covering only one task (vehicle segmentation). It would be helpful to expand this to other tasks considered in prior work (eg. the other comparisons in CVT [17]).\n- I found the description of the latent representation (Section 3.2) to be sparse on details, although it is described more precisely in the supplemental material. It would be helpful to provide slightly more detail in the main text, given that this is one of the main sources of novelty.\n- The approach is somewhat incremental. Most of the improvement comes from fairly well-known architectural changes.\n- The conclusions in the attention study (Section 4.3) seem slightly overstated (eg. the claim \"proving the network is able to retrieve the pixel relationships between views\"). This study does not seem to have been conducted very systematically, and consists mostly of qualitative results. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I'm borderline on the paper but lean towards acceptance. While the approach is incremental, the architecture's design decisions seem reasonable, and this is an interesting application of a fixed-size latent space. The experiments suggest the model works well, though I will defer to other reviewers in assessing the soundness of the experimental design.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "If time permits, I would be interested in seeing how the model performs on other BEV prediction tasks.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "4qIput-92fw", "reviewer": "Reviewer_Zycj", "summary": "This paper a method for vehicle segmentation in BEV space for use in autonomous driving. The system takes in images from multiple cameras, feeds them through a CNN to extract features, and then combines the features with ray embeddings (a representation of the ray that captured each pixel). The ray embeddings encode the spatial relationship between the cameras. The fused representation goes into a transformer, and outputs a BEV map which is further refined using a CNN. The authors show the method outperforms baselines, and also conduct several interesting ablation studies justifying various design choices.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The latent representation is less computationally expensive compared to methods that map densely between input image pixels and BEV pixels\n- Does not require BEV projection, which can lead to compounding errors\n- The model does not require a hardcoded BEV resolution or geometric assumptions\n- Interesting analysis showing that the attention heads of the learned model cover many directions\n\nWeaknesses:\n- The performance of the system does not seem ready for use in such a safety critical application area\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper proposes a novel method for BEV vehicle segmentation that outperforms other methods with a simpler and less computationally intensive approach. Also, the use of ray embeddings in this system seem likely to be of interest for other systems tackling this problem.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "- In multiple places, need to change 32Go to 32Gb", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)