Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes an approach for learning a monocular bird's-eye-view that transfers knowledge from Lidar data (used in training) to the testing scenario where only camera images are used. The application is self-driving. The approach is based on a Lidar-Teacher and Camera-Student knowledge distillation model. Therefore multiple sensor modalities are used (LiDAR and cameras) and domain adaptation is also handled. The point clouds knowledge from Lidar sensor is used only during the training phase. A multi-level adversarial learning mechanism is used to adapt and align features learned from different sensors and domains. Therefore there is cross-domain perception and cross-sensor adaptation for monocular 3D tasks.
strength_and_weaknesses: The strengths of the work results from the approach that is able to learn 3D BEV representations of scenes with both domain shift and modality mismatch. Another strength is the alignment of feature space between domains. The main weakness results from the lack of results concerning the computational complexity.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and the description of the technical work is easy to follow and understand. The experimental validation is extensive and there is novelty in that the testing data is not annotated and there is evaluation of dataset-to-dataset adaptation. There are also results evaluation cross-domain adaptation. There is some lack of detail that may hamper the reproducibility of the results. 
summary_of_the_review: Interesting work that handles cross-domain and cross-modality adaptation. This paper contributes to the robust estimation of 3D scene representation in BEV under both domain shift and modality change. The results show improvements in the estimation of BEV compared with competing approaches.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper aims to transfer the point cloud knowledge from a LiDAR sensor during the training phase to the camera-only testing scenario. The authors apply knowledge distillation with a LiDAR teacher and a camera student. They also use the multi-level adversarial learning mechanism to adapt the features learned from different sensors and domains. The proposed CroMA delivers fairly good domain adaptation performance.
strength_and_weaknesses: ---
Strengths:
* The paper is well-written and easy to follow. The authors have provided sufficient background on monocular BEV perception, which is helpful for readers outside this domain.
* The proposed solution is technically sound and achieves good empirical performance on domain adaptation.

---
Weaknesses:
* The technical novelty is very limited. The proposed techniques in this paper are not new: (1) 2DPASS has explored cross-modality knowledge distillation, and (2) many domain adaptation methods apply adversarial learning. Combining existing methods is a good engineering effort but cannot be considered a solid contribution.
* All experimental results target BEV segmentation. It is essential to present some results on 3D object detection benchmarks.
* Despite the good overall paper writing, I do not quite appreciate the theoretical insights provided in Section 3.2. The domain error bound seems far-fetched and not very related to the proposed method.
* Section 3.1 describes the exact method from LSS without any proper citation.

---
clarity,_quality,_novelty_and_reproducibility: This paper has good clarity, moderate quality but poor novelty.
summary_of_the_review: The current recommendation is primarily based on the limited novelty and insufficient evaluation. However, I would love to see the authors' response before making the final recommendation.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new domain adaptation setting where multi-modalities are provided in source domain while only visual images are provided in unseen target domain. Specifically, the authors focus on the BEV prediction problem where perspective image is given as input and model would output Bev semantics. Assuming that images and LiDAR are known in source domain, the goal of this paper is to achieve accurate Bev prediction in unseen target domain that only images are provided.

To achieve that, the authors propose to leverage the depth information rather than directly using LiDAR. And LiDAR-teacher Camera student architecture is used to bridge the gap introducing by missing modality, in both image-encoder and Bev-decoder modules.

To validate their ideas, the authors conduct experiments on two datasets, NuScenes and Lyft, with four shift settings. Results on these settings show that the proposed method achieves better segmentation results compared to segmentation only methods, or method with simple depth or image level adaptation. 
strength_and_weaknesses: Pros:
1. Well motivated and very interesting problem
2. For semantic BEV prediction, adapting depth rather than directly adapting LiDAR is intuitive and well-motivated.
3. Paper itself is well written and easy to follow

Cons:
1. Scale problem in depth prediction: Depth prediction itself suffers a lot with scale problem. I have concerns about the reliability of predicted depth during test time and am wondering why the proposed method can solve the scale problem thus bridge the gap between two domains.
2. Sparsity in LiDAR: Given the fact that LiDAR points are usually sparse, how does the proposed method handle the sparsity in depth prediction? Will the feature aggregation process be affected by the sparse depth?
3. Others and more details see below
clarity,_quality,_novelty_and_reproducibility: Clarity:
- Paper is well written in general. One thing I would like to note is that though the authors mentioned that “we provide a brief theoretical insight” in the beginning of Sec.3.2, I did not feel much theory is shared.

Quality:
- This paper is of good quality as the authors provide good motivation and showcase SOTA performance. Together with the cons I described above, I have the following concerns:
1. It seems to me there are two domain gaps in this task, one in semantic segmentation space and one in depth prediction space. I did not see the ablations in these two spaces, individually. What if some generic depth estimator would give us some depth prediction? Can the results be improved when combined with existing DA methods for semantic segmentation in perspective view?
2. Why the L2 loss between BEV features extracted by student and teacher networks is a good idea? What is truly going on when training the entire model? Can the author provide some insights?
3. How are the weights in Eq.6 learnt?
4. How can the proposed method avoid degenerate cases? For instance, what if all model parameters that shared by teacher and student models are the same and the discriminator is weak?   

Novelty:
- It addresses an interesting yet novel problem.

Reproducibility:
- The authors provided enough details in supplementary and code bases are available in existing work. This paper can be re-produce with some time. But other details, such as dataset split in some settings or number of cameras uses are missing.
summary_of_the_review: Overall, I feel this paper is well written and well motivated. But more explanations and experiments should be done to support the claims of the authors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a CroMA framework to incorporate multiple sensor modalities and close the domain gap between training and deployment for self-driving. They utilize LiDAR sensor during the training phase with knowledge distillation paradigm to enhance the camera-only model. Adversarial learning is adopted to address the issues of domain gap. They perform experiments to justify aspects of their claims.
strength_and_weaknesses: Strength:
1.	Providing diagrams highlighting the differences between the proposed cross-modality adaptation and the existing fixed-modality adaptation.
2.	Performing experiments validating aspects of their claims.

Weaknesses:
1.	The proposed cross-modality adaptation has been validated in BEVDepth [1], 2DPASS [2]. This paper did not discuss them in related work.
2.	This paper proposes new dataset split setting to investigate the cross-domain perception and cross-modality adaptation, and re-runs the baselines on the proposed custom setting, making the comparisons less convinced.
3.	This paper claims that naïve LiDAR supervision leads to worse performance, which is counter-intuitive. But the paper only validates the claim under Day-to-Night Adaptation. How about City-to-City Adaptation, Dry-to-Rain Adaptation, and Dataset-to-Dataset Adaptation?
4.	This paper seems to change LSS [3] from 6 perspective camera setting to monocular setting.

[1] Li, Yinhao et al. “BEVDepth: Acquisition of Reliable Depth for Multi-view 3D Object Detection.” ArXiv abs/2206.10092 (2022): n. pag.
[2] Yan, Xu et al. “2DPASS: 2D Priors Assisted Semantic Segmentation on LiDAR Point Clouds.” ArXiv abs/2207.04397 (2022): n. pag.
[3] Philion, Jonah and Sanja Fidler. “Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D.” ArXiv abs/2008.05711 (2020): n. pag.

clarity,_quality,_novelty_and_reproducibility: This paper clarifies its contribution well and provides details to reproduce the experiments. The contributions are somewhat new. Some of the claims need to be validated by complementary experiments.
summary_of_the_review: This paper seems to be a step to address the issues of cross-domain perception and cross-sensor adaptation. However, there are missing complementary experiments that I feel are required to fully validate the claims. The comparisons are performed on fully customized setting, which damages the credibility of some of the claims. The cross-modality adaptation has been validated in previous works, which should be clearly discussed in the related work. Therefore, I am leaning towards a reject for this version of the work.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary: The paper proposes a Cross-Modality Adaptation (CroMA) framework for monocular BEV perception.  The work explores the setting of when there are different sensors available during training and test.  Specifically, they consider the case when LiDAR is available during training but during testing, only a RGB camera is available.  The work proposes training a Camera-student model to project BEV features with a LiDAR-teacher.  The LiDAR-teacher had depth information that allows it to directly transform images features to the BEV frame, and serves as an guide for the Camera-student.  The model is trained using an adversarial schemes to align the features of the BEV and image domains and allows for robustness to domain shifts.  Domain-shift (city-to-city, day-to-night, dry-to-rain, dataset-to-dataset) experiments are conducted on the nuSenes and Lyft datasets,  Results show that the proposed CroMA method outperforms prior methods for most of the scenarios.

Strengths
- The paper clearly explains the cross-modality setting where the sensors available during test time may be a subset of what is available during training time.  This is an important setting that should be encouraged.
- Experiments show that the proposed method can handle domain shifts 
- Proposed solution is technical sound

Weaknesses
- Missing discussion and comparison against recent relevant work on distillation for 3D detection.  Upon the request of reviewers Dv6y and DeZj, the authors have added citations and discussion for 2DPASS, BEVDepth, and Lift-Splat.  However, the AC believe that there are relevant recent work using 3D-to-2D distillation that are not mentioned including:
  - MonoDistill: Learning Spatial Features for Monocular 3D Object Detection, Chong et al. ICLR 2022
  - LIGA-Stereo: Learning LiDAR Geometry Aware Representations for Stereo-based 3D Detector, Guo et al. ICCV 2021
  - Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection, Hong et al. ECCV 2022

Some of these recent methods, such as MonoDistill (which only uses the student RGB model during test time) should be included for comparison as well.
- Limited technical novelty as there has been prior work in cross-modality adaptation that the paper did not discuss in detail 

justification_for_why_not_higher_score: Three out of the four reviewers are negative, and the fourth reviewer (while giving the work a high rating) also agrees that the paper has weaknesses and does not object to reject.  The AC feel that the paper is missing discussion and comparison of key work (MonoDistill).
Decision: Reject