CroMA: Cross-Modality Adaptation for Monocular BEV Perception
Authors: Yunze Man
Liangyan Gui
Yu-Xiong Wang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Multi-Modality
Cross-Modality
Domain Adaptation
Autonomous Driving

Abstract: Incorporating multiple sensor modalities, and closing the domain gaps between training and deployment are two challenging yet critical topics for self-driving. Existing adaption works only focus on visual-level domain gaps, overlooking the sensor-type gaps which exist in reality. A model trained with a collection of sensor modalities may need to run on another setting with less types of sensors available.  In this work, we propose a Cross-Modality Adaptation (CroMA) framework to facilitate the learning of a more robust monocular BEV perception model, which transfer the point clouds knowledge from LiDAR sensor during training phase to the camera-only testing scenario. The absence of LiDAR during testing negates the usage of it as model input. Hence, our key idea lies in the design of a LiDAR-teacher and Camera-student knowledge distillation model, as well as a multi-level adversarial learning mechanism, which adapt and align the features learned from different sensors and domains. This work results in the first open analysis of cross-domain perception and cross-sensor adaptation model for monocular 3D tasks in the wild. We benchmark our approach on large-scale datasets under various domain shifts and show state-of-the-art results against various baselines.

Under review as a conference paper at ICLR 2023
CROMA: CROSS-MODALITY DOMAIN ADAPTATION FOR MONOCULAR BEV PERCEPTION
Anonymous authors
Paper under double-blind review
ABSTRACT
Incorporating multiple sensor modalities and closing the domain gap between training and deployment are two challenging yet critical topics for self self-driving. Existing adaption work only focuses on the visual-level domain gap, overlook- ing the sensor-type gap which exists in reality. A model trained with collection of sensor modalities may need to run on another setting with less types of sen- sors available. In this work, we propose Cross-Modality Adaptation (CroMA) framework to facilitate the learning of a more robust monocular bird's- s-eee-view (BEV) perception model, which transfers the point clouds knowledge from Li- DAR sensor during the training phase to the camera-only testing scenario. The ab- sence of LiDAR during testing negates the usage of it as model input. Hence, key idea lies in the design of (i) a LiDAR-teacher and Camera-student knowledge distillation model, and (ii) multi-level adversarial learning mechanism, which adapts and aligns the features learned from different sensors and domains. This work results in the first open analysis of cross-domain perception and cross-sensor adaptation for monocular 3D tasks in the wild. We benchmark our approach on large-scale datasets under a wide range of domain shifts and show state-of-the-art results against various baselines.
INTRODUCTION
In recent years, multi-modality 3D perception has shown outstanding performance and robustness over its single-modality counterpart, achieving leading results for various 3D perception tasks (Vora et al., 2020; Qi et al., 2020; Jaritz et al., 2020; Park et al., 2021; Weng et al., 2020) on large-scale multi-sensor 3D datasets (Caesar et al., 2020; Kesten et al., 2019; Sun et al., 2020). Despite the su- periority in information coverage, the introduction of more sensor modalities also poses additional challenges to the perception system. On one hand, generalizing the model between datasets becomes hard because each sensor has its unique domain gap, such field-of-view (FoV) for cameras, den- sity for LiDAR, etc. On the other hand, the operation of the model is conditioned on the presence and function of more sensors, making it hard to work on autonomous agents with less sensor types or under sensor failure scenarios.
More specifically, transferring knowledge among different data domains is still an open problem for autonomous agents in the wild. In the self-driving scenario, training the perception models offline in source domain with annotations while deploying the model in another target domain without annotations is very common in practice. As a result, the model will have to consider the domain gap between source and target environments or datasets, which usually involves different running locations, different sensor specifications, different illumination and weather conditions, etc.
Meanwhile, the domain shift lies not only the visual perspective, but also in the sensor-modalin perspective. Previous methods assume less realistic setting where all sensors are available during training, validation, and deployment time, which is not always true in reality. Due the cost and efficiency trade-off, available in the target domain during testing than what we have in the source domain during training. A typical scenario is having camera and LiDAR sensors in the large-scale training phase while only having cameras for testing, as shown in Figure 1. It is not clear how to facilitate the camera-only 3D inference with the help of a LiDAR sensors only in the source domain during training.
sensor missing and failure, in many scenarios
can have fewer sensors
The challenges above raise an important question and task: Can we achieve robust 3D perception under both the visual domain gap and sensor modality shift??

---

Under review as a conference paper at ICLR 2023
Fixed-Mod ModdliMtyy Adaptation (Existing)
Cross Modality Adaptation (Ours)
Scraace Soeuatio
Fmaage
adaptation
image
kmagg
Image
Single trainiing (Source)
types deplayment (Tarrpe)
Sahoe rapertitnn, Worse nisttr
Differeut (unually [cves) azzzon in Ceplayyment
Sunsorr Scauario
LIDAR
adaptation
Fmagge
EIDAR
image
LIDAR
Multiiple
(couraly mun)))
LitAAR-Theubbrr Disetiniiatoor
Multiple traininng
typer deployment (Mbrget)
sensors training
Figure 1: Left & Middle: Existing models assume fixed sensor modalities during training and testing phases. Right: We introduce a more realistic setting which considers possible cross moddlltyy domain shift between model training and deployment. Surprisingly, using source-only LiDAR as depth supervision leads worse performance (6.3 in IoU) than the image-only model (6.7). Thus, we propose CroMA reduce the domain discrepancy with knowledge distillation and feature alignment, achieving state-of-the-ar performance (17.0).
This task is challenging for several main reasons. (1) Ill-posed. The monocular 3D perception in the target domain is inherently ill-posed, due to the lack of direct 3D measurement from the camera- only sensor input. (2) Partial availability of LiDAR data. Most prior work multi-modal 3D) perception (Vora et al., 2020; Man et al., 2021) and 3D domain adaptation (Zhang et al., 2021; Yang al., 2021) makes LiDAR point clouds the input signals their models for providing accurate range measurement. However, these methods fail when the LiDAR sensor is unavailable during the evaluation, as no LiDAR data can be used as input into the model. Hence, our new setting requires model to leverage information from the LiDAR point clouds without making it an input to the model. (3) Naive LiDAR supervision leads to worse performance. in the community that introducing additional sensors is bound to increase the overall performance. Surprisingly, our experiments showed 0.3 IoU decrease when we naively introducing LiDAR to supervise the depth estimation. This is because the source and target domain gap becomes larger with the additional sensor-type shift. As we will discuss in Sec. 3.2, our new problem setting requires novel methodology in using LiDAR without increasing the domain discrepancy.
is generally believed
To tackle the above challenges, we propose CroMA, a cross-modality domain adaptation framework for bird's veew (BEV) perception. Our model addresses the monocular 3D perception task be- tween different domains, and utilizes additional modalities in the source domain to facilitate the evaluation performance. Motivated by the fact that image and BEV frames are bridged with 3D rep- resentation, we first design an efficient backbone to perform 3D depth estimation followed by BEV projection. Then, to learn from point clouds without explicitly taking them as model inputs, we pro- pose an implicit learning strategy, which distills 3D knowledge from LiDAR-Teacher to help the learn better 3D representation. Finally, in order address the visual domain shift, we introduce adversarial learning on the student to align the features learned from source and target domains. Supervision from the teacher and feature discriminators are designed at multiple layers to ensure effective knowledge transfer. By considering the domain gap and effectively leveraging LiDAR point clouds in the source domain, our proposed method is able to work reliably in more complicated, uncommon, and even unseen environments. Our model achieves state-of-the-at per- formance in four very different domain shift settings. Extensive ablation studies are conducted to investigate the contribution of our proposed components, the robustness under different changes, as well as other design choices.
The main contributions of this paper are follows. (1) We introduce modality mismatch, over- looked but realistic problem setting in 3D domain adaptation in the wild, leading to a robust camera- only 3D model that works complicated and dynamic scenarios with minimum sensors available. (2) We propose a novel Teacher and Camera-Student knowledge distillation model, which considerably outperforms state-of-the-art LiDAR supervision methods. (3) Extensive experiments in challenging domain shift settings demonstrate the capability of our methods in leveraging source domain point cloud information for accurate monocular 3D perception.
RELATED WORK
2.1 MULTI-MODALITY AND CROSS- 3D PERCEPTION
Considerable research has examined leveraging signals from multiple modalities, especially images and point clouds, for 3D perception tasks. Frustum PointNet (Qi et al., 2018) uses 2D detection
2

---

Under review as a conference paper at ICLR 2023
on images to generate bounding boxes, which further guide the 3D detection with point clouds. Ku et al. (2018); Chen et al. (2017); Liang et al. (2019) project point clouds to BEV frame, and then fuse 2D RGB features with BEV features to generate proposals and regress bounding boxes. Alternatively, Pointpainting (Vora et al., 2020) proposes to augment point clouds with image seman- tic segmentation results leverage color information. Recent work (Zhu et al., 2021; Yoo et al., 2020) starts to explore deep feature-level fusion between points and images. CLOCs (Pang et al., 2020) proposes a post-detection fusion mechanism combine the candidate boxes from RGB and LiDAR inputs. Ye et al. (2020) propose to use feature alignment between point clouds and im- ages to improve monocular 3D object detection. Under the umbrella of the cross-modality setting, 2DPASS (Yan et al., 2022) transfers features learned from images to the Lidar during training to help the Lidar model to perform 3D semantic segmentation during inference. BEVDepth (Li et al., 2022) proposes to obtain reliable depth estimation for 3D object detection by exploiting the camera parameters together with the image features during training. On the contrary, our method explores more realistic yet challenging setting, where use Lidar during training to help the camera-only model in 3D perception. Despite outperforming single-modality methods, most prior work assumes identical domain distribution during training and inference, which is not always true in reality. As result, the actual usefulness of additional sensors is still unclear when domain shift exists.
2.2 CROSS-DOMAIN 3D PERCEPTION
While extensive research has been conducted on domain adaptation for 2D tasks, the field of domain adaptation for 3D perception in the real world has relatively small literature. Some prior work adapts depth estimation from synthetic to real image domains (Kundu et al., 2018; Zhao et al., 2019). Working classification. For 3D semantic segmentation, Wu et al. (2019) project the point clouds to 2D view, while some other work (Jaritz et al., 2020; Peng et al., 2021; Gong et al., 2021) proposes to leverage point clouds and images data together. Recent work (Zhang et al., 2021; Yang et al., 2021; Luo et al., 2021) starts to explore cross-domain 3D object detection from point clouds. Tarvainen & Valpola (2017) employ Mean Teacher to generate pseudo-label for the target domain. SRDAN (Zhang et al., 2021) employs adversarial learning to align the features between different domains. Although prior work (Jaritz et al., 2020; Li et al., 2021) explores various domain adaptation techniques for different sensor modalities, these methods only adopt the same modalities learn the domain shift between source and target data. In contrast, our approach achieves robust 3D perception in more general scenario, where the model can perform accurate 3D inference in the target domain by adapting information encoded in source-exclusive modalities.
point clouds, Qin et al. (2019) design multi-scale adaptation model for 3D
2.3 3D INFERENCE IN BIRD'S-ENS S-EEE- VIEW FRAME
Inferring 3D scenes from the BEV perspective has recently received a large amount of interest due to its effectiveness. MonoLayout (Mani et al., 2020) estimates the layout of urban driving scenes from images in the BEV frame and uses an adversarial loss to enhance the learning of hidden objects. Can et al. (2021) propose employ graphical representation and temporal aggregation for better inference in the driving scenarios using on-board cameras. Recently, using BEV representation to merge images from multiple camera sensors has become popular approach (Hendy et al., 2020; Pan et al., 2020). Following the monocular feature projection proposed by Orthographic Feature Transform (OFT) (Roddick et al., 2018), Lift-Splat-Shoot (Philion & Fidler, 2020) disentangles feature learning and depth inference by learning a depth distribution over pixels to convert camera image features into BEV. Unlike the above work performing BEV analysis in settings with more controlled premises, we are the first to explore cross-domain and cross-sensor settings, leading to more robust and more realistic 3D inference methodology.
3 APPROACH
In this work, we consider the task of learning BEV representation of scenes with domain shift and modality mismatch. Specifically, the model will be given annotated LiDAR point clouds and cam- cras images in the source domain, but only unannotated camera images in the target domain. And the model seeks achieve highest performance setting is common and worthwhile, especially considering the existence of many existing public multi-modality datasets and the rise of many camera-only vehicle scenarios.
the unsupervised target domain. This problem
Formally. for the source domain, we are given labeled data with N multi-modality samples, D {(X;, Pf, yt) N where represents the source domain. Here X = {=f) consists of
3

---

Under review as a conference paper at ICLR 2023
BEY Project
Source X
Encoder
Soutge BDAR
Teacher BFV
Decoder
Source Domain Branch
LiDAR-Teacher
Ldp
l
Target
Domain Branch
Camera-Student
BEV Mape
Source
Depth Estimation
Encoder
Image
Project
BEV Seatures
BEV Decoder
LGT
Tanget image
Embeddings
Disclminator
Dicciminattor
In,
Figure 2: Overview of CroMA framework. CroMA includes three components. (1) LiDAR- Teacher uses voxelized LiDAR point clouds to transform the image features to BEV frame. It pro- vides essential knowledge how guide image learning given LiDAR information. (2) Camera- Student leverages the same architecture as the teacher, but the depth is estimated. It is supervised by teacher model as well as the LiDAR ground truth. (3) Discriminators are used to align features from source and target domains.
camera images F 3x The number of cameras n can take any integer as small as one, depending on the dataset used cameras deployed on the vehicle. In addition, each camera image has an intrinsic matrix and an extrinsic matrix. P. is point cloud containing multiple unordered points p E RR represented by 3D coordinate values. And label y represents rasterized representa- tion of the scenes in the BEV coordinate. For the target domain, No
given unlabeled data with (image) samples, D' [X where f represents the target domain, and would like to estimate (vl) ==1 )= the BEV representation of the scenes in the target domain.
An overview of our method CroMA is illustrated in Figure 2. CroMA is designed to extract features from monocular images and project the features into BEV frame (Section 3.1), using estimated or ground truth 3D depth information. The model is composed of LiDAR-teacher and Camera- student (Section 3.2), where the teacher encodes how to learn better representation given LiDAR point clouds, and transfers that knowledge to the camera-only student using multi-level teacher- student supervision. Finally, to bridge the domain gap between source and target domains, we lever- age adversarial discriminators different feature layers to align the distributions across domains in the camera-student model (Section 3.3). Finally, we describe the overall learning objective and loss designs (Section 3.4).
3.1 LEARNING BEV FROM IMAGES
In order to achieve 3D perception under the cross-modality setting, our first challenge is to unify the image coordinates, point cloud coordinates, and BEV coordinates into joint space. We follow LSS (Philion & Fidler. 2020) to transform the image features from perspective view into the BEV view. More specifically, we tackle this problem by constructing a 3D voxel representation of the scene for each input image. We discretize the depth axis into Na bins and lift each pixel of the images into multiple voxels (frustums), where each voxel is represented by the 3D coordinate of its center location. For given pixel px (h, w) of Na voxels at different depth bins.
of the camera image it corresponds
(I) where M is camera matrix and d; is the depth of the i-th depth bin. The feature vector of each voxel ii in Vpx is the base feature fpx of pixel px scaled by the depth value a; More specifically, feyEV, a; fpx. where the pixel feature fpx is extracted by an image encoder. And the depth value a; obtained either from LiDAR point clouds or by estimation, in the teacher and student model, respectively. The acquirement of a, is introduced in Sec. 3.2.
Vpx = (tr - dsh, daw, dil (i
After getting the features for each the voxels, we project the voxels onto the BEV and aggregate the features get the BEV feature map. The BEV frame is rasterized into (X, Y) 2D grids, and for each grid, its feature is constructed from the features of all the 3D voxels projected into it using mean pooling. This projection allows to transform arbitrary number of camera images into unified
4

---

Under review as a conference paper at ICLR 2023
BEV frame. Finally, we obtain an image-like BEV feature embedding, which is used to estimate the final representation using a convolutional neural network (CNN) decoder.
This architecture design bridges the image and LiDAR modalities through intermediate 3D vox- clized representation. Hence, we can input LiDAR point clouds into the model to directly guide the BEV projection without having change the overall pipeline. This further enables the distillation of knowledge from the point clouds to images using a teacher-student model.
3.2 CROSS-MODALITY TRANSFER WITH -STUDENT DISTILLATION
The co-existence of domain and modality gaps poses additional challenges to the adaptation task. Although the Lidar sensor in the source domain provides 3D knowledge to the model, it also in- creases the domain discrepancy between the source and the target, which could hurt the model adaptation (as we will see Sec, 4.3 and Table 5) '. Hence, the unique difficulty of our work lies in exploiting the Lidar point cloudy during training to guide the camera model for better 3D estimation.
Depth Supervision by Point Clouds. The main advantage of point clouds over the image modality is the accurate 3D positional information coming from the depth measurement. Due to the lack of LiDAR during cvaluation, we cannot use point clouds as direct input of the model. Hence, one alternative approach to use point clouds is supervise the depth estimation in the model. As in Eq. I, for each pixel, our model calculates the feature of its corresponding voxels by multiplying the pixel voxel with depth value aş, We use another depth head predict depth distribution Oxpx - -aa1,000 , ). over Na depth bins for each pixel px.
The ground truth depth supervision for this estimation task is generated by LiDAR point clouds as follows: When projected to the image frame, the points corresponding to one pixel can have three conditions. If the pixel has, (1) no point inside: the ground truth depth distribution of it is omitted. (2) only one point inside: the ground truth depth distribution of this point is a one-hot vector, with value one being in the voxel that the point lies in. (3) multiple points inside: the ground truth depth distribution og of this point is calculated by counting the number of points in each depth bin, and dividing them by the total number of points: a, Number of points in depth hin og
Total number of points in Vpx
Using distribution- of different depth occur in one pixel. This happens at the boundary of the objects, and becomes more severe when images get downsampled and pixels become large during feature encoding. Moreover, a probabilistic depth representation considers uncertainty during depth estimation, and degenerates to pseudo-LiDAR methods (Weng & Kitani, 2019) if the one-hot constraint is added.
depth representation effectively accounts for the ambiguity when objects
Learning from LiDcReTeacher Despite being intuitive and straightforward, direct depth su- pervision is not optimal for two reasons. First, LiDAR supervision is only the intermediate feature layer, providing no supervision on the second half of the model. Also, while LiDAR pro- vides accurate depth measurement, "depth estimation" is still different from our overall objective on BEV representation. Motivated by this, as shown in Figure 2, we propose to use pretrained LiDAR oracle model to supervise the image model the final BEV feature embedding, such that the supervision of LiDAR is provided to the whole model, and is closer to the final objective. We call the model using ground truth point cloud information supervised Stamera-Student. This boils down to knowledge distillation problem where the 3D inference knowledge of the LiDAR-teacher classic problem of "better teacher, worse student" (Cho & Hariharan, 2019; Mirzadeh et al., 2020; Zhu & Wang, 2021) in knowledge distillation due to capacity mismatch does not cxist in this model, because the -DDAR-Teache and Camera-Student models in CroMA are almost identical.
and the model to be
distilled to the camera-only student. Note that the
Overall, this teacher-student mechanism allows the camera model to learn better 3D representation from the point clouds, achieving more complete LiDAR supervision at different stages, while still keeping the model image-centric for image-only inference.
3.3
ADAPTATION WITH ADVERSARIAL FEATURE ALIGNMENT
Since the BEV annotations and the LiDAR ground truth are only available in the source data, the model will be heavily biased to the source distribution during teacher-student supervision. Hence, we bridge the target and source domains using adversarial training. Specifically, we place one
We provide more detailed analysis over this perspective in the Appendix.
5

---

Under review as a conference paper at ICLR 2023
discriminator D1 BEV decoder CNN blocks, and another D2 at the image encoder CNN blocks, to align the features two domains by optimizing over discriminator losses. While the final-layer discriminator D1 is constantly useful to align features learned from the LiDAR-Teacher and final ground truth, we find that the middle-layer discriminator D2 is exceptionally effective under certain domain gaps where images have great changes but LiDAR remains robust.
To achieve adversarial learning, given a feature encoder E and input sample X, domain discrim- inator is used to discriminate whether the feature ( comes from the source domain or the larget domain. The target and source domain samples are given the label d = and d = 0, re- spectively. And D (X)) outputs the probability of the sample X belonging to the target domain. Hence, the discriminator loss is formulated by cross-entropy loss:
Lais = log ((E(X))
d) log(1 D(E(X))).
Moreover, in order to learn domain-invarian features, our feature encoder E should try to extract features that fool the discriminator D, while the discriminator D tries to distinguish the right domain label of the samples. This adversarial strategy can be formulated as a "min-max" optimization problem: LD ming max Lais- The "min-max" problem is achieved by Gradient Reverse Layer (GRL) (Ganin & Lempitsky, 2015), which produces reverse gradient from the discriminator D to learn the domain-invariant encoder E. The loss form is the same for both D1 and D2.
3.4 FULL OBJECTIVE AND INFERENCE
The overall objective of our model is composed of the supervision from the BEV ground truth, the LiDAR-Teacher, and the domain alignment discriminators. Given the output rasterized BEV representation map y E XX'x"'C, the ground truth (GT) loss term LoT can be formulated as a cross-entropy loss between the estimated source domain BEV map ÿ" and the GT label y":
Lor(Ø" )²) ΣΣ>
M(..a log ((..)))
The supervision from the LiDAR-Teachhe is composed of a direct depth estimation loss Lap and a teacher feature supervision LT. As described in Sec. 3.1, given the 3D depth volume a E TRHxxxN,, the direct depth supervision term Lap formulated as a cross entropy loss between the estimated 3D depth distribution volume ā" in the source domain, and the GT depth volume a calculated from LiDAR point clouds as described in Sec. 3.2:
11
W N₂
Cāp (a)(("a")))
ΣΣΣ
alasa log a(..,,),
And for the LiDAR-Teacte feature supervision: (T F = L2(Fte, Fut) is an L2 loss, where Fte and Fst are the feature maps of teacher and student models, respectively. Finally, the domain adaptation loss contains LD, and LD, with the form described in Eq. 2.
The final objective is formulated as multi-task optimization problem:
LCCOMA = Le + ArCr + AtpLap + ÅD, Cp, + ADzCDa, where År. Adp- ÄD, and ÅD. are weights for the corresponding loss terms. The CroMA model is trained end-to-end using the loss term Eq. During inference, target samples go into the Camera- Student model to output the final BEV representation. More training details are provided in Sec. 4.
(5)
4 EXPERIMENTS
4.1 DATASETS AND DOMAIN SETTINGS
We evaluate CroMA with four unique domain shift settings, constructed from two large-scale datasets, nuScenes (Caesar et al., 2020) and Lyft (Kesten et al., 2019). The first three domain set- tings subsampled from the nuScenes dataset, including city-to-city, day-to-night, and dry-to-rain adaptation. In addition, we also perform another inter-dataset domain transfer: dataset-to-datave adaptation, where the source and target domain data are from nuScenes and Lyft datasets. All adap- tation settings follow the assumption that the source has access to cameras and Lidar sensors, while the target only has cameras. We use all six cameras provided by the nuScenes dataset. We also analyze surprising observations on cross mods-ity performance in the ablation study. More details regarding the domain settings and dataset splits are in the Appendix Sec. B.

---

Under review as a conference paper at ICLR 2023
Table 1: CroMA achieves best performance all classes under city-to-city domain gaps in IoU. DA and CM denote whether model considers domain adaptation and cross-modality in design.
Boston Singapare
CM
Vchcle
Lanc
Singapore.
Monolagour (Ma e 2020
x
14.2
35.
7.5
Monollayour (Mani et al.. 2000
14.4
38.7
5.8
OFF (Roiklick 2018)
x
16.8
9.6
OFT (Roollic al, 2018)
x
16.1
LSS (Puilion & Fidler. 2020)
x
17.6
38.2
10.6
LSS (Philion & Filler, 2020)
19.5
42.3
10.7
Wilk-ange Ang
7
17.9
40.5
124
Wilk-rang Aug.
7
19.3
45.7
11.3
Vanilla DA
13.0
31.4
9.1
Vanilla DA
21.2
45.7
12:5
Depch-Sapy DA
4
19.0
42.8
14.9
Depth-Supe DA
4
22.5
47.1
12.9
Irgut-Fasion Teacher
18.6
42.5
14.1
Impie- fuvion Teacher
22.0
47.5
13.1
CroMA (ours)
28.5
433
136
CroMA (ours)
25
48.9
14:
Table 2: CroMA leads significant improvements under day-to-night domain shift, and also achieves best results under dry-to-rain domain shift in IoU.
Day - Night
7M
C.W
Vehicle
Reod
Lane
Day - Rain
Da
CM
Vehicle
Road
Lane
MinsoLayoon Mam atl (2020)
5.9
27.7
5.5
MonoLayor (Mail et a 2020))
X
20.5
08.7
13.T
OFT (Koddick et al. 2018)
X
X
0.6
90.5
6.0
OFT (Rodlick al. 2018)
X
24.1
79%
16.2
LSS (Philion & Filles, 2020)
41.2
7.1
LSS (Philins & Filler, 2020)
X
27.8
71.
16.8
Wide-ragge
10.3
46.0
10.4
Widee rango Aug.
/
18.2
71.2
17.2
Vanilla
Vanilla DA
/
39.1
708
18.3
Depth DA
Depth DA
29.6
Input-fuiion Tencher
fusion Teacher CroMA
7
29.5
7
4.2 RESULTS AND COMPARISONS
Baselines We compare our method with state-of-the-art BEV 3D layout perception work MonoLay- out (Mani et al., 2020), OFT (Roddick et al., 2018), LSS (Philion & Fidler, 2020), as well other baseline methods in domain adaptation and cross-modality learning. Wide-range Aug. means using wide range of random scaling augmentation which potentially includes the target domain scale. For Vanilla DA, we adapt camera-only DA-Faster (Chen et al., 2018) to our BEV perception setting. Depth-Supy DA stands for depth supervised domain adaptation. We use source domain LiDAR as ground truth to supervise the depth estimation during training, without LiDAR-Teacher supervision (only Láp without LT). Input-fusion Teacher is an alternative way of designing the where we directly fuse point (I, y, coordinates into their corresponding image pixels as additional channels in the teacher model, similar to Pointpainting (Vora et al., 2020). We use DA and CM to denote whether model considers domain adaptation and cross- modss-mddlllyy in design. Results are reported on vehicle, drivable roads, and lane marking classes using intersection-over-mmee
(loU).
City-to-City Adaptation As shown in Table 1, we observe that our CroMA model achieves the best performance on all classes for inter-city transfer settings. Without domain adaptation, baseline approaches Monol OFT, and LSS all suffer from performance degradation. By considering the visual domain shift, wide-range augmentation and vanilla domain adaptation methods improve the baseline up to 11.2. Direct depth supervision and alternative input-fusion teacher models do not bring as much improvement as CroMA. The results clearly demonstrate the effectiveness of our method by distilling and aligning the LiDAR information for cross modolity domain adaptation 3D BEV perception.
Day-to-Night Adaptation As shown in Table 2 on the left, we observe that our CroMA model achieves the best performance in all classes. Different from the previous setting, we notice that the improvement on Day > Night setting is exceptionally high. This is because the initial domain gap between day and night scenarios very large the camera modality space. Moreover, the LiDAR sensor is robust under illumination changes due to its active imaging mechanism opposed to camera's passive one. Thus, incorporating LiDAR point cloud information helps the model to learn more robust illumination-invarit representation from the image inputs.
Dry-to-Rain Adaptation As shown in Table on the right, under this setting we also observe that our CroMA model achieves the best performance on all classes. However, we notice that the improvement under Dry - Rain setting not as big as previous settings, and only on par with other alternative cross-modality baselines. This is because the domain gap between dry and rain scenarios not big in image modality. Hence, baseline methods OFT and LSS are already able to obtain decent results even without domain adaptation. Furthermore, rainy weather is known to cause great domain shift in the LiDAR modality (Xu et al., 2021). As a result, the knowledge learned from exclusve LiDAR suffers from unknown domain shift which can be larger than the image modality domain shift. This can potentially cancel out the benefit of 3D information learned from point clouds and explains for the smaller improvement.
7

---

Under review as a conference paper at ICLR 2023
Table 3: Our model achieves the hest perfor-
Table Our proposed LIDR--Teache also leads
mance under dataset-toddatase domain gaps.
to best performance without domain shift, which
demonstrate the robustness of our CroMA model.
muSeenes -+ Lyft
DM
CM
Vehicle
Monnl ayout Mani al., 2020)
11.8
wio domain shift
CM
Vahicle
Road
Lanc
OFT (Roddick et al., 2018)
X
16.5
Monol (Mami
1.8 (Philion Fidler, 2020)
X
19.9
FishingNet (Hendy et al., 2020)
30.0
Widc-range Aug.
21.9
OFT (Roddick et al., 2018)
30.1
72.2
16.9
Vanilla DA
22.5
1SS (Philion Hidler, 2020)
18.8
Depth-Supy
23.4
Depth-Supv
34.8
75.8
[9.]
Input fusion Teacher
22.8
Inper fasion Teacher
35.1
76.5
18.7
CroMA
24.4
CroMA (ours)
38.8
76.8
20.5
Frod Car
Rack Crr
Vanilla
Depth-Sups
CioMA (Ounl
Ground Truth
Figure 3: Qualitative Results Day Night setting (model trained with daytime data, and validated with night data). We notice that CroMA performs significantly better than other baselines for vehicles, drivable roads, and lane marking classes. From left to right: (1) Vanilla adversarial learning; (2) LiDAR depth
supervision with adversarial learning; (3) CroMA model: (4) Ground Truth. Best viewed in color.
Maanset-to-Datasee Adaptation As shown in Table 3, we can also observe that our CroMA model achieves best performance nuScenes -> Lyft setting. Following Philion & Fidler (2020), because Lyft does not provide road segment and lane marking information the HD map. we report results on the vehicle class. Note that there is not pre-defined train and validation data split for Lyft, choose a new split and report the number of our re-implemented results for the baselines. Compared with baselines with and without domain adaptation cross-modality learning, our CroMA model does a better job in leveraging and adapting LiDAR information. We include Lyft + nuScenes results in the Appendix Sec. C.
Qualitative Results As shown in Figure 3, under Day -> Night domain shift setting, our model achieves significantly better monocular 3D perception than the vanilla DA and Depth-Supv DA baselines. We observe that CroMA provides more clearly defined road boundaries and lane mark- ings. The depth and size of the vehieles and the road the right side are also predicted more accurately. CroMA only misses some vehicles that are hardly visible in camera due to occlusion and distance. Overall, the qualitative results validate the effectiveness CroMA in closing the gap between data domains and leveraging point clouds information for better 3D inference.
4.3 ANALYSIS OVER SURPRISING OBSERVATIONS AND ABLATION STUDY
Naive Usage of More Sensors Leads to Worse Performance is naturally believed that introduc- ing multiple sensors in the perception model is bound increase the model performance. Surpris- ingly, experiments shown in Table 5 negates this naive intuition. When we introduce LiDAR sensor in the source domain as depth supervision, the result decreases for 0.3. As we described in Sec. 3.2, the domain distribution divergence increases after introducing the sensor-modality shift. As a result, we propose multiple components in CroMA to account for the visual and sensor domain shift. Ex- periments show that while wide augmentation strategy and adversarial discriminator both achieve better results than baseline (11.2 vs. 6.7 in loU), our LiDAR Teacher further boosts the result to 17.0 leveraging effective LiDAR knowledge distillation and alignment.
8

---

Under review as a conference paper at ICLR 2023
Table 5: Ablation study shows that our proposed components all contribute the final state-of state-of-the-art the-art perfor- mance. We report results on vehicle class under day-to-night domain gap in loU.
Backbone
Augmentation
Wide
Diseriminators
Adversarial
Supervision
LIDAR
LiDAR-Teacher Tencher
LiDAR:
Results
6.4
11.2 (44.5)
15.7 (43.0)
17.0 (+10.3)
Figure 4: The proposed progressive learning strategy effec- tively addresses the challenge caused by mixed domain gap scenario Poaston-to-Sinappe nuScenes. This shows that CroMA can function in
mixed with day-to-night)
CrdMA
Only
more realistic domain shift setting.
No LIDAR Baadine
Mixed Domain Gap
Vehicle
Road
Lane
Direct Inference
17.6
38.2
10.6
73
Vanilla DA
13.0
31.4
9.1
20
Progressive DA
18,8
41.5
13.2
Figure Results of CroMA improve as the
CroMA (ours)
20.5
43.1
15.6
number of LiDAR sweeps increases.
Dealing with Mixed Domain Shift Another common but underexplored question we observe in the 3D domain adaption setting is the mixed domain shift problem, where multiple types of gaps between source and target domains often occur concurrently. For example, in the nuScenes dataset, the Boston data only captured during the day, while the Singapore data is captured both day and night. This leads to the fact that some of the data in the Singapore domain only has a city-wise gap the source, while the other part data has both city-wise and illumination-winn shown in Table 4, we find that directly leveraging adversarial discriminator in this scenario leads to worse performance than direct inference (model trained without domain adaptation), because mixed domain in the target domain confuses the domain discriminator. Hence, we propose a progressive learning mechanism, where we first perform adaptation with city-wise data for 100K steps, and then train the model the full target domain dataset for another 150K steps. This effectively alleviates the mixed domain shift problem, and helps CroMA achieve leading results than other baselines.
gaps. As
Results within Same Domain In addition to the domain shift setting, we further validate that our proposed LiDAR-Teacher is able to distill knowledge and achieve the best performance within the same domain. As shown in Table 4, the results are obtained in the original nuScenes train/val split.
Effect on LiDAR Density & Comparison with Oracle Model As shown in Figure 5, we vali- date that our model can achieve higher performance when denser LiDAR is available. This can be achieved by grouping continuous scans of LiDAR point clouds (from to 5) into a whole, to have denser 3D representation of the scene. We observe that other cross- mooss-myd baselines including Input-Fusion Teacher and Depth-Supy models cannot effectively leverage the LiDAR knowledge, with dense point clouds available. We also compare (target domain also has LiDAR modality) and find that the gap between the upper bound result and No-LiDAR baseline is significantly reduced. The remaining performance gap is caused by the unknown LiDAR domain gap which we hope to further reduce in future work.
model with the LiDAR oracle model
More results in Sec. C We also find that the different types of domain gaps react unevenly to different model designs, and conduct more ablation on model designs, including depth ground truth signals and
These can be found in the Appendix.
CONCLUSION
In this paper, we proposed CroMA to estimate 3D scene representation BEV under domain shift and modality change. To achieve this, construct iIDAR-Teache and distill knowledge from into an Camera-Student by feature supervision. And we further propose to align feature space between the domains using multi-stage adversarial learning. Results large-scale datasets with various challenging domain gaps demonstrated the effectiveness of our approach, which marks a significant step towards robust 3D scene perception in the wild.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wort- man Vaughan. A theory of learning from different domains. Machine learning, 79(1):151-175, 2010.
Garrick Brazil and Xiaoming Liu. M3d-rpn: Monocular 3d region proposal network for object detection. In ICCV, 2019.
Holger Caesar, Varun Bankiti, Alex Lang. Sourabh Vora, Venice Erin Liong, and Qiang Xu. nuscenes: A multimodal dataset for autonomous driving. In CVPR, 2020.
Yigit Baran Can, Alexander Liniger, Danda Pani Paudel, and Luc Van Gool. Structured bird's-eye view traffic scene understanding from onboard images. In ICCV, 2021.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In CVPR, 2017.
Yuhua Chen, Wen Li, Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Domain adaptive faster r-enn for object detection in the wild. In CVPR, 2018.
Jang Hyum Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In ICCV, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In CVPR, 2009.
Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In ICML, 2015.
Rui Gong, Dengxin Dai, Yuhua Chen, Wen Li, and Luc Van Gool. mdalu: Multi-souree domain adaptation and label unification with partial datasets. In ICCV, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Noureldin Hendy, Cooper Sloan, Feng Tian, Pengfei Duan, Nick Charchut, Yuesong Xie, Chuang Wang, and James Philbin. Fishing net: Future inference of semantic heatmaps in grids. In CVPR, 2020.
Maximilian Jaritz, Tuan-Hung Vu, Raoul de Charette, Emilie Wirbel, and Patrick Pérez. xmuda: Cross-modal unsupervised domain adaptation for 3d semantic segmentation. CVPR, 2020.
R. Kesten, M. Usman, J. Houston, T. Pandya, K. Nadhamuni, A. Ferreira, M. Yuan, B. Low, A. Jain, P. Ondruska, S. Omari, S. Shah, A. Kulkarni, A. Kazakova, C. Tao, L. Platinsky, W. Jiang, and V. Shet. Level perception dataset 2020. 2019.
data/,
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14t1::11126680.. 2014.
Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3d proposal generation and object detection from view aggregation. In IROS, 2018.
Jogendra Nath Kundu, Phani Krishna Uppala, Anuj Pahuja, and R Venkatesh Babu. Adadepth: Unsupervised content congruent adaptation for depth estimation. In CVPR, 2018.
Yinhao Li, Zheng Ge, Guanyi Yu, Jinrong Yang, Zengran Wang, Yukang Shi, Jianjian Sun, and Zeming Li. Bevdepth: Acquisition of reliable depth for multi-view 3d object detection. arXiv, 2022.
Yu-Jhe Li, Xiaoliang Dai, Chih-Yau Ma, Yen-Cheng Liu, Kan Chen, Bichen Wu, Zijian He. Kris Kitani, and Peter Vadja. Cross-domain object detection via adaptive self training. arXiv preprint arXiv:2111.13221 2021.
10

---

Under review as a conference paper at ICLR 2023
Ming Liang, Bin Yang, Yun Chen, Rui Hu, and Raquel Urtasun. Multi-task multi-sensor fusion for 3d object detection. CVPR, 2019.
Zhipeng Luo, Zhongang Cai, Changqing Zhou, Gongjie Zhang, Haiyu Zhao, Shuai Yi, Shijian Lu, Hongsheng Li, Shanghang Zhang, and Ziwei Liu. Unsupervised domain adaptive 3d detection with multi-level consistency. In ICCV, 2021.
Yunze Man, Xinshuo Weng, Prasanna Kumar Sivakumar, Matthew O'Toole, and Kris M Kitani. Multi-echo lidar for 3d object detection. In ICCV, 2021.
Kaustubh Mani, Swapnil Daga, Shubhika Garg, Sai Shankar Narasimhan, Madhava Krishna, and Krishna Murthy Jatavallabhula. Monolayout: Amodal scene layout from single image. In WACV, 2020.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In AAAI, 2020.
Bowen Pan, Jiankai Sun, Ho Yin Tiga Leung. Alex Andonian, and Bolei Zhou. Cross-view semantic segmentation for sensing surroundings. RA-L, 2020.
Su Pang, Daniel Morris, and Hayder Radha. Clocs: Camera-lidar object candidates fusion for 3d object detection. In IROS, 2020.
Jinhyung Park, Xinshuo Weng, Yunze Man, and Kris Kitani. Multi-modality task cascade for 3d object detection. In BMVC, 2021.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performance deep learning library. In NeurIPS, 2019.
Duo Peng, Yinjie Lei, Wen Li, Pingping Zhang, and Yulan Guo. Sparse-to-dense feature matching: Intra and inter domain cross-modal learning in domain adaptation for 3d semantic segmentation. In ICCV, 2021.
Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
Charles Qi. Wei Liu, Chenxia Wu, Hao Su, and Leonidas J Guibas. Frustum pointnets for 3d object detection from rgb-d data. In CVPR, 2018.
Charles R Qi, Xinlei Chen, Or Litany, and Leonidas J Guibas. Imvotenet: Boosting 3d object detection point clouds with image votes. CVPR, 2020.
Can Qin, Haoxuan You, Lichen Wang, C-C Jay Kuo, and Yun Fu. Pointdan: A multi-scale 3d domain adaption network for point cloud representation. In NeurlPS, 2019.
Thomas Roddick, Alex Kendall, and Roberto Cipolla. Orthographic feature transform for monocular 3d object detection. arXiv preprini arXiv: 1.001///111111111 1.08188, 2018.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for au- tonomous driving: Waymo open dataset. In CVPR, 2020.
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In ICML, 2019.
Antti Tarvainen and Harri Valpola. Mean teachers better role models: Weight-averaged consis- tency targets improve semi-supervised deep learning results. In NeurIPS, 2017.
Sourabh Vora, Alex H Lang, Bassam Helou, and Oscar Beijhom. Pointpainting: Sequential fusion for 3d object detection. In CVPR, 2020.
Tai Wang, Xinge Zhu, Jiangmiao Pang, and Dahua Lin. Feos3d: Fully convolutional one-stage monocular 3d object detection. In ICCV, 2021.

---

Under review as a conference paper at ICLR 2023
Tai Wang, ZHU Xinge, Jiangmiao Pang, and Dahua Lin. Probabilistic and geometric depth: Detect- ing objects in perspective. In CoRL, 2022.
Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Wein- berger. Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In CVPR, 2019.
Xinshuo Weng and Kris Kitani. Monocular 3d object detection with pseudo-lidar point cloud. In ICCVW, 2019.
Xinshuo Weng, Yongxin Wang, Yunze Man, and Kris M Kitani. Gnn3dmot: for 3d multi-object tracking with 2d-3d multi-feature learning. In CVPR, 2020.
Graph neural network
Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer. Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from lidar point cloud. ICRA, 2019.
Bin Xu and Zhenzhong Chen. Multi-level fusion based object detection from monocular images. In CVPR, 2018.
Qiangeng Xu, Yin Zhou, Weiyue Wang, Charles R Qi, and Dragomir Anguelov. Spg: Unsupervised domain adaptation for object detection via semantic point generation. ICCV, 2021.
Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation on lidar point clouds. ECCV, 2022.
Jihan Yang, Shaoshuai Shi, Zhe Wang. Hongsheng Li, and Xiaojuan Qi. St3d: Self-training for unsupervised domain adaptation on 3d object detection. In CVPR, 2021.
Xiaoqing Ye, Liang Du, Yifeng Shi, Yingying Li, Xiao Tan, Jianfeng Feng, Errui Ding, and Shilei Wen. Monocular 3d object detection via feature domain adaptation. In ECCV, 2020.
Jin Hyeok Yoo, Yeocheol Kim, Ji Song Kim, and Jun Won Choi. 3d-cvf: Generating joint camera and lidar features using cross-view spatial feature fusion for 3d object detection. In ECCV, 2020.
Weichen Zhang, Wen Li, and Dong Xu. Srdan: Scale-aware and range-aware domain adaptation network for dross-dataset 3d object detection. In CVPR, 2021.
Shanshan Zhao, Huan Fu, Mingming Gong, and Dacheng Tao. Geometry-aware symmetric domain adaptation for monocular depth estimation. CVPR, 2019.
Ming Zhu, Chao Ma, Pan Ji, and Xiaokang Yang. Cross-modality 3d object detection. In WACV, 2021.
Yichen Zhu and Yi Wang. Student customized knowledge distillation: Bridging the gap between student and teacher. In ICCV, 2021.
12

---

Under review as a conference paper at ICLR 2023
APPENDIX
A VISUALIZATION WITH VIDEO CLIPS
For video visualization, please see the ICLR-supp-video.mp4 in the supplementary files.
B DATASET AND IMPLEMENTATION DETAILS
B.1 IMPLEMENTATION DETAILS
Following Philion & Fidler (2020), we use EfficientNet (Tan & Le, 2019) pretrained on Ima- geNet (Deng et al., 2009) as our image encoder backbone. Two heads are applied to estimate pixel features and pixel-wise depth distribution from the 8x downsampled feature map. The 3D feature maps are projected to the bird's s-eye-view frame using mean pooling. For the bird''-eeye-verr de- coder use ResNet-18 18 (He et al., 2016) as backbone, and upsample the features learned from the first three meta-layers of ResNet to the final BEV output. The D1 and D2 domain discriminators applied to the output feature layers of EfficientNet and ResNet backbone, respectively. We use a light weight discriminator architecture, which is composed of a global averaging pooling layer, followed by two fully connected layers, and outputs the domain label. For input, resize and crop input images to size 128 x 352. For output, we consider a 100 meters x 100 meters range centered at the ego-vehicle, with the grid size set to be 0.5 meters x 0.5 meters. The depth bin is set to be 1.0 meter between 4.0 meters and 45.0 meters range. The whole model is trained end-to-end, with Ar = 1.0, Adp = 0.05, ÅD, 0.1, ÀD. 0.01. We train CroMA using the Adam (Kingma & Ba, 2014) optimizer with learning rate 0.001 and weight decay 1e-7 for 50K steps for the teacher model, and 200K for the student model. We use horizontal flipiigg, random cropping, rotation, and color jittering augmentation during training. The whole model is implemented using the PyTorch framework (Paszke et al., 2019).
B.2 DATASET DETAILS
For this section, we explain our dataset split in more details. For our experiments, we always split the target domain into two subsets for fair comparison. One of them can be accessed during training for adversarial learning and domain alignment, and the other is held out exclusively for validation.
We follow existing Lidar-based domain adaptation work, including SRDAN (Zhang et al., 2021), ST3D (Yang et al., 2021), UDA3D (Luo et al., 2021), and xMUDA (Jaritz et al., 2020), to create the domain split strategies. Specifically, for the day-to-night, city-to-city, and dry-to-rain settings, as described in prior approaches, we use the sentence the nuScenes dataset and filter the keywords to split the dataset into corresponding subsets. For the dataset-to-dataset setting, we use the official split of the nuScenes dataset, and the split provided in ST3D (Yang et al., 2021) for the Lyft dataset. More details are provides follows.
City to City Adaptation For the city-to-city adaptation scenario, we sub-sample the trainval split of the large-scale dataset nuScenes (Caesar et al., 2020) captured in Boston and Singapore city. We treat one city as the source domain and the other as the target domain. The Boston part of data has 467 scenes in total, which is separated into 350 scenes for training and 117 scenes for validation. And the Singapore part of data has 383 scenes in total, which is separated into 287 scenes for training and 96 scenes for validation.
Day to Night Adaptation For the day-to-night adaptation scenario, we also sub- sub-sample the trainval split of the large-scale dataset nuScenes (Caesar et al., 2020). Every scene in the nuScenes dataset has sentence of description, which can be parsed and used to categorize it into certain class. In this way, we create a day scene subset and night scene subset out of the whole dataset. We treat day as the source domain and night as the target domain, because the night subset has significantly fewer samples than the day subset. The day split has 751 scenes which are all used for training. And the night part has 99 scenes in total, which is separated into 74 scenes for training and 25 scenes for validation.
13

---

Under review as a conference paper at ICLR 2023
Table 6: CroMA achieves great inference time compared with the baseline methods.
#Params (M)
Frame-pet per-Second (FPS)
OFT (Roddick al., 2018) LSS (Philion & Fidler, 2020)
22
14
35
CroMA (Ours)
15
33
Dry to Rain Adaptation For the dry-to-rain adaptation scenario, we also sub-sample the trainval split of the large-scale dataset nuScenes (Caesar et al., 2020). Similarly, using the scene description sentence, we create a dry (non-rainy) scene subset and rainy scene subset out of the whole dataset. We treat dry as the source domain and rain as the target domain. because the rain subset has sig- nificantly fewer samples than the dry subset. The dry split has 685 scenes which are all used for training. And the rain part of data has 165 scenes in total, which is separated into 124 scenes for training and 41 scenes for validation.
Dataset Dataset Adaptation For the cataset-to-datasee adaptation scenario, we use the two large-scale autonomous driving datasets nuScenes (Caesar et al., 2020) and Lyft (Kesten et al., 2019). We treat one dataset as the source domain and the other as the target domain. For the nuScenes dataset, we use the original train and validation split, which has 700 scenes and 150 scenes, respec- tively. The Lyft dataset does not have an original split, so we sub-sample 132 scenes for training and 48 scenes for validation.
B.3 COMPUTATIONAL COMPLEXITY
Table 6 summarizes the number of parameters and inference speed for prior baselines and our model. We can see that our Lidar-Teacher distillation and multi-level adversarial learning modules do not affect the inference efficiency of CroMA compared with the baseline model. Our total number of parameters 15M, and our inference time is 33 Frame-per -enee-ee-SSeoodd (FPS) on a V100 GPU, which on par with the baseline LSS (Philion & Fidler, 2020). The training time for our model is around 20 hours on 4x V100 GPUs.
C ADDITIONAL RESULTS AND ANALYSIS
Unique Challenge due Problem Formulation We provide another perspective understand the challenges brought by the co-existence cross- modolltyy and cross-domann gaps, and thus further motivating the design of our architecture. It is proved that in domain adaptation, the target domain error (h) can be upper-bounded by the inequality (Ben-David et al., 2010):
('hh) s ch)h) dHAH (x) +C,
(6)
where the bound is composed of the source-domain error e (h), source-target distribution divergence dHAH (Px, )x) and another term which is considered constant in our case. Existing domain adap- tation work mostly focuses on reducing the source-domain error. However, while the introduction of LiDAR sensor reduces the first term, it increases the second term, because the source and target domains have an additional sensor-type gap. It also demonstrated in Sec. 4 that naively us- ing source domain Lidar even hinder the target performance rather than improve it. Hence, the unique difficulty of our work lies in leveraging Lidar to reduce the source-domain error ch)),, and in the meanwhile, preventing distribution divergence dHAH (P* P from increasing too much.
Different Depth GT Signal Generation Methods As shown in Table 7, we validate CroMA's de- sign choice in depth ground truth signal generation (Eq. 2 in main file) by comparing with different alternative methods. Note that all the methods have the same output when there no point or only one point projected pixel. The difference only comes from the behaviour when multiple points are projected to one pixel.
Majority Voting means to generate one-hot GT vector by assigning "1" to the depth bin with the most number of points inside, and random select one bin when more than one bin with the most number of points.
14

---

Under review as a conference paper at ICLR 2023
Table 7: We validate the design choices of CroMA comparing with various depth GT generation methods and different choices of teacher supervision signals. Results show that depth supervision generated by Point Number Distribution (Eq. in main file), and feature-level supervision from LiDAR-Teacher Teacher help the model achieve the best performance.
Design choices in Day 7 Night Setting
Vehicle
Road
Lane
No LiDAR
11.2
48.8
11.1
Depth Ground Truth Signal (From Point Clouds)
Majority Voting
14.1
50.0
13.7
Random Selection
13.8
49.5
12.6
Softmax Point Number Distribution
13.1
49.9
11.9
Point Number Distribution (Ours)
17.0
51.8
16.9
No LiDAR-Teacher
15.7
50.5
14.2
LIDAR-Teacher Supervision
Soft-label Supervision
15.2
50.7
13.9
Feature Supervision (Ours)
17.0
51.8
16.9
Table CroMA model achieves best performance all classes under Lyft-to-nuSeenes domain gaps in IoU. DA and CM means domain adaptation and cross modality.
Lyft + nuScenes
DA
CM
Vehicle
ayout (Mani et al., 2020)
X
X
7.1
OFT (Roddick et al., 2018)
X
X
11.9
LSS (Philion Fidler, 2020)
X
X
13.8
Wide-range Aug.
/
X
14.6
Vanilla DA
/
X
15.1
Depth-Supv DA
/
16.5
Input-fusion Teacher
15.1
CroMA (ours)
19.2
Random Selection means to generate a one-hot GT vector by randomly selecting one of all the projected points, and assigning "I" to the depth bin in which the selected point lies.
Softmax Point Number Distribution means to count the number of points inside every depth bin, and then use softmax to turn this vector into a distribution.
direct Point Number Distribution, where count the number of points inside every depth bin, and divide the vector by the total number of points to get a depth distribu- tion. Results show that direct point number distribution outperforms other counterparts in ground truth generation. One possible reason is because CroMA downsamples the feature map 8x when estimating the 3D depth volume. This makes it much more common for multiple points with dif- ferent depth values to fall in the same pixel. By contrast, the other two methods, Majority Voting and Random Selection, will lose valuable information in these cases. On the other hand, softmax activate over-smooth the distribution, and also assign small values to zero bins (depth bins with no point inside). Hence, we find that preserving the native depth distribution the points a better way to supervise CroMA in 3D evaluation.
Finally, in CroMA
Different GDAR-Teecche Supervision Design As shown in Table 7, also validate CroMA's design choice a LiDAR- Supervision by comparing with different alternative methods. In addition to the feature-level supervision used in CroMA, another commonly used teacher supervi- sion is the soft label output. Specifically, Soft-label Supervision means to use the class distribution output of the teacher model as the supervision of the student model, as opposed to the one-hot vec- tor from the ground truth annotation. We find that feature-level supervision performs better than the soft-label supervision. One reason because we use small number classes, which makes the supervision from the soft-label less informative. Moreover, because the teacher and student models in CroMA have almost identical architecture and capacity, enforcing the corresponding feature level similarity between the teacher and student model provides stronger supervision than the soft-label output, without harming the model learning. As future work, we will explore whether soft-label may be more useful when the number of classes in the model is larger.
Lyft to nuScenes Adaptation Due to page limit, Table 3 in the main file only shows the results under the nuScenes Lyft setting. Here also present the results under the Lyft nuScenes adaptation setting for completeness. As shown in Table 8, we can also observe that our CroMA model achieves the best performance. Like previous scenarios, baseline approach MonoLayout,
15

---

Under review as a conference paper at ICLR 2023
Table 9: Different types of domain gaps react unevenly different model designs. Direct depth supervision and the middle layer feature alignment block provide a larger improvement under Day - Night setting than Dry - Rain setting.
CroMA Designs
Day
Night
Dry Rain
Image-only Baseline
11.2
28.3
LiDAR Teacher Feature Supervision
14.9 (+3.7)
29.5 (+1.2)
LiDAR Teacher Feature + Depth Supervision
17.0 (+5.8)
29.6 (+1.3)
Without Domain Alignment
7.1
28.1
Feature Alignment Final layer
12.2 (+5.1)
29.6 (+1.5)
Feature Alignment Mid Final layer
17.0 (49.9)
29.3 (+1.2)
Table 10: CroMA model achieves great performance with scaling augmentation and FoV matching for maScenes-to- Lyft domain gaps in vehicle loU.
nuScenes >> Lyft
Sealing Augmentation
Match FoV
Vehicle IoU
None
23.5
with scaling augmentation
23.8
with both
24.4
OFT, and LSS without domain adaptation suffer from performance degradation due to domain shift. And compared with other baselines with domain adaptation cross- wross-modality learning, CroMA does a better job in leveraging and adapting LiDAR information.
Effeet of Different Designs Different Domain Gaps We find in our experiments that different types of domain gaps react unevenly to different model designs. As shown Table 9, for Day + Night setting which has less domain shift in LiDAR point cloud modality than camera modality, the depth supervision, and the middle layer feature alignment block provide a large improvement on top of other modules. As opposed to Dry Rain setting, where domain gap is larger for point clouds than images, and no significant improvement is achieved by using these two components. This further validates the necessity of using multiple modalities under domain adaptation settings, which can effectively improve the algorithm robustness under different domain shifts.
Solving Scale Problem Scaling ambiguity is inherent problem for monocular depth estimation. We solve this problem by using the random cropping, scaling augmentation strategy during training, and also by matching the FoV (Field-of-Viee) augmentation increases the robustness of the depth prediction model in scale difference. And the FoV matching scales the images in the target domain match the FoV of the source domain. This makes sure that one object looks approximately the same size in images, if it is of the same distance the ego vehicle in source and target domains, thus reducing the scale ambiguity. provide an ablation study of scale augmentation and FoV matching in nuScenes-to. to-fft adaptation. As we can see, they both improve our final model performance.
of domains using their intrinsic matrices. The
Table 10, we
From Table 10, we also notice that even without the two methods mentioned above, the model still performs fairly well, compared with the baseline methods in Table 3. This is because in driving scenarios, the camera FoV, the context in the images and the depth distribution of the images have relatively strong (domains). This is different to more general depth estimation scenarios, where objects can have drastically different depth distribution and the intrinsic matrix can have big differences from image to image.
do not have strong discrepancy even across different driving scenarios
Ablation in Semantic Segmentation and Depth Prediction From the task perspective, there are two domain gaps in this task, one in the semantic segmentation task and the other in the depth prediction task. In Table 11, we show the result of depth estimation and semantic segmentation alone in the Day to Night scenario.
For depth estimation, we report the cross entropy error (CEE) with the ground truth we described in Sec 3.2, because each pixel will have multiple depth values. For this task the Lidar teacher supervision refers to Lap in our main pipeline. We observe that the method we propose significantly improves the depth estimation metrics by 44.9%.
16

---

Under review as a conference paper at ICLR 2023
Table 11: Our proposed modules achieve great performance in both depth estimation and semantic segmentation tasks.
Day + Night
Depth Estimation (CEE)
Semantic Segmentation (IoU)
Direct Inference
2.56
27.5
Adversarial Learning (AL.)
1.97
31.8
Tidchreeacher Supervision + AL
1.41
32.1
Table 12: Naive LiDAR supervision cannot help the final perception under cross-domain and cross-modality scenario.
Method
Boston + Singapore
Dry - Rain
nuScenes +Lyft
LSS (Philion & Fidler. 2020) Baseline Baseline + naive LiDAR supervision CroMA (Full Model)
17.5
27.9 28.2 (+0.3)
20.6
17.5 (-0.0)
20.3 (-0.2)
20.5 (+3.0)
29.6 (+1.7)
24.4 (+3.9)
For semantic segmentation, we report the IoU for vehicle class. To remove the effect of depth estimation, use pre-trained depth estimation model and fix its weight. For this task the Lidar teacher supervision refers to LT in our main pipeline. We observe that the CroMA architecture also improves the baseline method by 16.7%.
Results of Naïve LiDAR Supervision Under Other Adaptation Settings In Table 12, we present the ablation study of direct Lidar supervision under different adaptation scenarios. We observe that in all scenarios, naïve Lidar supervision cannot lead to better performance against the baseline.
When the source and target domains have large visual gaps (Dah-Niihh, nuScenes the naive supervision leads to worse results, and when the gap is smaller (Boston-+Singape Drin+Rain), the Lidar supervision results in on-par or only slightly better performance. The reason is that although the Lidar sensor in the source domain provides 3D knowledge to the model, it also increases the domain discrepancy between the source and the target (the model has to adapt to the additional modality shift), which hurts the model performance instead.
D VISUALIZATION ON FAILURE CASES
As shown in Figure 6, we visualize the failure cases of our CroMA model. We can notice that most failure cases come from far distance, heavy occlusion (which are typical failure cases for baselines as well). Faraway objects are known be typically hard cases for monocular 3D percep- tion (Xu & Chen, 2018; Wang et al., 2019; Brazil & Liu, 2019; Wang et al., 2021; 2022), because ambiguity of object depth from images becomes larger as the distance increases. This can be potentially alleviated by using smaller downsampling rate when extracting the image features. As for objects inside the red dashed boxes in Figure they can be seen the LiDAR sensor due to the higher deployment position. But in cameras, the objects are almost invisible due to the occlusion of vegetation, structures, or other vehicles. The occlusion problem can be potentially addressed if we have access additional sensors during evaluation. As future work, will also try alleviate the occlusion problem in monocular settings by leveraging temporal information.
17

---

Under review as a conference paper at ICLR 2023
Oulput Vehide Segmenisiinn
Output Merid Segmenlul ReadiSegnerstton
Segmerbat
Oulgul Lans Marking Segmeriisiion Veniele
Oulpur Late Wanking Segments on Virticke
CroMA (Ours)
Ground Truth
Outpul Velicle Segmentation
Ourput Veh Segmentation
Ourpul Segmantation
Curpul
Ourpul Lane Marking Segmentation
Ougput Lame Marking Segmentation
Ego
Fgo Vihiidee
CroMA (Ours)
Ground Truth
Figure 6: Visualization of failure cases CroMA (model is trained with daytime data, and validated with night data). We notice that major failure cases CroMA are Far distance and Occlusions of objects and regions.
18


Reviews:
[{"review_id": "uqOU0Iikpj", "reviewer": "Reviewer_5QJE", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an approach for learning a monocular bird's-eye-view that transfers knowledge from Lidar data (used in training) to the testing scenario where only camera images are used. The application is self-driving. The approach is based on a Lidar-Teacher and Camera-Student knowledge distillation model. Therefore multiple sensor modalities are used (LiDAR and cameras) and domain adaptation is also handled. The point clouds knowledge from Lidar sensor is used only during the training phase. A multi-level adversarial learning mechanism is used to adapt and align features learned from different sensors and domains. Therefore there is cross-domain perception and cross-sensor adaptation for monocular 3D tasks.", "summary_of_the_review": "Interesting work that handles cross-domain and cross-modality adaptation. This paper contributes to the robust estimation of 3D scene representation in BEV under both domain shift and modality change. The results show improvements in the estimation of BEV compared with competing approaches.", "strength_and_weaknesses": "The strengths of the work results from the approach that is able to learn 3D BEV representations of scenes with both domain shift and modality mismatch. Another strength is the alignment of feature space between domains. The main weakness results from the lack of results concerning the computational complexity.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iIvsRHMuF6", "reviewer": "Reviewer_Dv6y", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims to transfer the point cloud knowledge from a LiDAR sensor during the training phase to the camera-only testing scenario. The authors apply knowledge distillation with a LiDAR teacher and a camera student. They also use the multi-level adversarial learning mechanism to adapt the features learned from different sensors and domains. The proposed CroMA delivers fairly good domain adaptation performance.", "summary_of_the_review": "The current recommendation is primarily based on the limited novelty and insufficient evaluation. However, I would love to see the authors' response before making the final recommendation.", "strength_and_weaknesses": "---\nStrengths:\n* The paper is well-written and easy to follow. The authors have provided sufficient background on monocular BEV perception, which is helpful for readers outside this domain.\n* The proposed solution is technically sound and achieves good empirical performance on domain adaptation.\n\n---\nWeaknesses:\n* The technical novelty is very limited. The proposed techniques in this paper are not new: (1) 2DPASS has explored cross-modality knowledge distillation, and (2) many domain adaptation methods apply adversarial learning. Combining existing methods is a good engineering effort but cannot be considered a solid contribution.\n* All experimental results target BEV segmentation. It is essential to present some results on 3D object detection benchmarks.\n* Despite the good overall paper writing, I do not quite appreciate the theoretical insights provided in Section 3.2. The domain error bound seems far-fetched and not very related to the proposed method.\n* Section 3.1 describes the exact method from LSS without any proper citation.\n\n---", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P3pUCJ_3gC", "reviewer": "Reviewer_DeZJ", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a CroMA framework to incorporate multiple sensor modalities and close the domain gap between training and deployment for self-driving. They utilize LiDAR sensor during the training phase with knowledge distillation paradigm to enhance the camera-only model. Adversarial learning is adopted to address the issues of domain gap. They perform experiments to justify aspects of their claims.", "summary_of_the_review": "This paper seems to be a step to address the issues of cross-domain perception and cross-sensor adaptation. However, there are missing complementary experiments that I feel are required to fully validate the claims. The comparisons are performed on fully customized setting, which damages the credibility of some of the claims. The cross-modality adaptation has been validated in previous works, which should be clearly discussed in the related work. Therefore, I am leaning towards a reject for this version of the work. ", "strength_and_weaknesses": "Strength:\n1.\tProviding diagrams highlighting the differences between the proposed cross-modality adaptation and the existing fixed-modality adaptation.\n2.\tPerforming experiments validating aspects of their claims.\n\nWeaknesses:\n1.\tThe proposed cross-modality adaptation has been validated in BEVDepth [1], 2DPASS [2]. This paper did not discuss them in related work.\n2.\tThis paper proposes new dataset split setting to investigate the cross-domain perception and cross-modality adaptation, and re-runs the baselines on the proposed custom setting, making the comparisons less convinced.\n3.\tThis paper claims that naïve LiDAR supervision leads to worse performance, which is counter-intuitive. But the paper only validates the claim under Day-to-Night Adaptation. How about City-to-City Adaptation, Dry-to-Rain Adaptation, and Dataset-to-Dataset Adaptation?\n4.\tThis paper seems to change LSS [3] from 6 perspective camera setting to monocular setting.\n\n[1] Li, Yinhao et al. “BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection.” ArXiv abs/2206.10092 (2022): n. pag.\n[2] Yan, Xu et al. “2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds.” ArXiv abs/2207.04397 (2022): n. pag.\n[3] Philion, Jonah and Sanja Fidler. “Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D.” ArXiv abs/2008.05711 (2020): n. pag.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "FTl8gSpVokz", "reviewer": "Reviewer_Lf7W", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new domain adaptation setting where multi-modalities are provided in source domain while only visual images are provided in unseen target domain. Specifically, the authors focus on the BEV prediction problem where perspective image is given as input and model would output Bev semantics. Assuming that images and LiDAR are known in source domain, the goal of this paper is to achieve accurate Bev prediction in unseen target domain that only images are provided.\n\nTo achieve that, the authors propose to leverage the depth information rather than directly using LiDAR. And LiDAR-teacher Camera student architecture is used to bridge the gap introducing by missing modality, in both image-encoder and Bev-decoder modules.\n\nTo validate their ideas, the authors conduct experiments on two datasets, NuScenes and Lyft, with four shift settings. Results on these settings show that the proposed method achieves better segmentation results compared to segmentation only methods, or method with simple depth or image level adaptation. ", "summary_of_the_review": "Overall, I feel this paper is well written and well motivated. But more explanations and experiments should be done to support the claims of the authors.", "strength_and_weaknesses": "Pros:\n1. Well motivated and very interesting problem\n2. For semantic BEV prediction, adapting depth rather than directly adapting LiDAR is intuitive and well-motivated.\n3. Paper itself is well written and easy to follow\n\nCons:\n1. Scale problem in depth prediction: Depth prediction itself suffers a lot with scale problem. I have concerns about the reliability of predicted depth during test time and am wondering why the proposed method can solve the scale problem thus bridge the gap between two domains.\n2. Sparsity in LiDAR: Given the fact that LiDAR points are usually sparse, how does the proposed method handle the sparsity in depth prediction? Will the feature aggregation process be affected by the sparse depth?\n3. Others and more details see below", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject