Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper tackles the problem of sim2real domain adaptation for the bird's-eye-view vehicle segmentation task. The authors trained existing models (Lift Splat and Point Pillar) with carefully sampled synthetic data,  pseudo-labels on real data (eq. 6), and gradient-reversal layers with f-DAL inspired loss. On the nuScenes dataset, the authors found that the these tricks improve models' (Lift Splat and Point Pillar) sim2real generalization accuracy of the vehicle segmentation IOU (Table 1 and Table 2).
main_review: Strengths:
- The authors study a few recent sim2real techniques (e.g., pseudo-labels on real data, gradient-reversal layers with f-DAL inspired loss).
- The authors carefully sampled synthetic data from simulators to match the real labels marginals of the nuScenes datasets (section 3).
- Using the above tricks, two models' (Lift Splat and Point Pillar) sim2real generalization accuracies improve on the vehicle segmentation IOU metrics (Table 1 and Table 2).

Weaknesses:
- The paper combines a few existing techniques to train baseline models on synthetic data. The techniques and baseline models are all from existing literatures, so the novelty is lacking.
- The authors didn't include comparison with prior works on the nuScenes datasets. What are the current SOTA models' accuracies (trained on real data or sim-real mixed)? Are there other sim2real works that evaluate on nuScenes?
limitations_and_societal_impact: Yes (see: section 7)
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors propose a technique for sampling data from a self-driving car simulator. They sample in a simulator-agnostic way in an effort to reduce the distance between the label marginals and therefore reduce the sim2real gap.They compare their approach to standard domain adapation techniques. They compare their techniques to standard data adaption frameworks and baselines and show promising results. They also demonstrate that domain adversarial training outperform simple style transfer approaches. 
main_review: # Originality:
The paper follows in a long line of adapting self driving car simulator data to the real world. The individual ideas in the paper are different from the related works mentioned since it relies on regenerating the datasets using parameters from the simulator. In this way, it's slightly different from DANN or MUNIT. The lack of related work that is actually compared to in the paper is my biggest concern. Note: not many papers focus on what parts of the simulator are actually relevant to transfer between datasets or the limitations of the simulator itself and this is is a very valuable and understudied problem.

# Quality
The paper does appear to include  proper comparisons against the other techniques. The claims are well supported technically (with the exception of the technique being simulator agnostic. That particular point is not demonstrated by the paper.

Truly showing that would require more simulators than just CARLA, but that is a minor issue.  The baselines compared against seem adequate. The weakest portion of the paper is the limited number of baselines. They only compare to DANN or MUNIT despite those baselines being rather old. More modern state of the art domain adaption and style transfer strategies. DANN was published 5 years ago in 2016.  Without recent and adequate baselines, the paper feels incomplete. The fact that only rather dated two baselines were chosen is rather concerning as reviewer as it seems possible that more recent work may outperform this approach. The shear lack of baselines here is my greatest concern for this paper.

# Clarity
The clarity of the paper is adequate. Note that many of the figures are so small that they are hard to read the text on, but this  is easily fixable. (In fact a lot of the figures are small enough that they are bit hard to read the underlying text). The paper is well written.

# Significance
This paper would prove helpful in the domain adaption self-driving cars. As self-driving car research requires accurate simulators that transfer well, this submission would prove useful in that space. More importantly though, this paper is very relevant to demonstrating which types of variation (ie. map design, and number of vehicles) actually matter to self-driving car simulator allowing the creators to focus on the types of variation that are most important to impactful real world performance. For instance, they state that variation in NPC color does not matter for performance so there is no need to build a randomization strategy for those NPC colors. On the other hand, they showing matching the NPC positions with a dataset like nuscene is rather important. 

#Update
Given the rebuttals from the authors, I am willing to raise my score to a 7. However, I would recommend the do not include their claims about the method being simulator agnostic as while possible, this claim has not be empirically proven in the paper through the use of multiple simulators.
limitations_and_societal_impact: Yes
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a method for both generating data in simulation and training with the generated data for the bird's-eye-view (BEV) vehicle segmentation task. It takes inspiration from recent theoretical advances in domain adaptation (DA) and frames learning from simulation as DA. It also points out special properties of using simulated data for adaptation, i.e., we can control the simulated data (label) distribution. Inspired by risk upperbound derived in prior work, it proposes a synthetic data generation strategy that samples data based on human-designed spatial priors and priors of the target domains (if some target labels are given). Once the data is generated, it employs a training method that extends recent work on adversarial DA to the dense prediction task of BEV vehicle segmentation and aims to learn invariant representation across domains. For experiments, the paper synthesizes data using the CARLA simulator and evaluates the adapted model on the nuScenes dataset. Results show that the method outperforms other data synthesis strategies and training methods such as style transfer-based DA.
main_review: **Strength:**

- The idea of optimizing synthetic data distribution to match that of target domain distribution is sensible and appealing. The paper follows a rigorous DA framework to analyze the need for optimizing data sampling distribution in order to minimize the target domain risk.
- Although the theoretical results and DA approach used in the paper are not new, the paper uses them effectively to solve practical real-world tasks such as BEV vehicle segmentation, and shows that the proposed framework outperforms prior data synthesis and DA methods typically used in these tasks. The method also shows good robustness against different types of domain gaps such as maps, vehicle assets, and weather. So this seems like a good empirical contribution.
- The paper is generally well-written. Most techniques are well-motivated from a theoretical standpoint.

**Weakness:**

- When no target label is provided (direct transfer learning), the spatial priors used in Sec 3.1 are rather heuristic, which makes the method less principled or general. Using these human priors is what people typically do when generating synthetic data, so the practice itself is not new, although the paper does formalize it inside a DA framework.
- The paper claims to be "agnostic of the network architecture and the choice of a simulator", however, the paper has only used one simulator, one architecture, and one hand-designed prior. Thus, it is difficult to verify this claim. It would be more interesting if the authors can try other architecture and experiment with other autonomous driving tasks and use other priors.
- In the semi-supervised learning setting, the performance of the proposed method over Lift Splat seems small. So it seems like the method is most effective for direct transfer learning. One could argue that the performance gain is mainly coming from the strong hand-designed spatial priors.

**Additional Comments:**

typo:

- line 245: "defficiencies" → "deficiencies"
- line 331: "it's" → "its"
limitations_and_societal_impact: The paper adequately discussed its limitations and potential negative societal impact. The limitations listed in the papers are mainly typical failure modes of the proposed method, such as mistaking pedestrians for vehicles.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This work targets the sim2real domain adaptation problem and approaches the problem from two angles - data collection and transfer method. The authors discuss the importance of how data is generated from the source domain and propose ways to effectively design scenarios (placement of other vehicles) that lead to better transfer. This work also introduces a novel method for domain adaptation that utilizes adversarial-training and learning with pseudo-labels.
They measure transfer performance across domains (CARLA to NuScenes) on the bird’s-eye-view vehicle segmentation from multi-view images task.
main_review: This work addresses important questions in domain adaptation for sim2real transfer with limited amounts of labeled data in the target domain.

This paper provides valuable discussion and experiments on how data collection in the source domain should be performed i.e. how does diversity of weather and vehicle assets in the simulator affect downstream transfer? These are important design decisions when developing simulators and I found the experiments on data collection Fig. 7-12 to be well-designed and insightful.

**Questions/Concerns**:
* The authors use only BEV vehicle segmentation for evaluation. I would like to see some discussion on why they chose this over full perception task presented in the Lift-Splat-Shoot (vehicle segmentation, driveable area, lane boundary …).
* I find the ablations on data sampling quite thorough (weather/car color diversity/...) but the current set of experiments provide no insight in how transfer varies across simulator photorealism, a major factor to consider in simulation.
* The proposed training objectives are described in Eq 4, 5, 6 and Figure 14 in the supplementary material. I highly recommend moving Figure 14 to the main paper, as well as including the training objective in the figure.

**Typo/Prose/Figures**:
* Line 245: “deficiencies”
* Line 277: “obtained”
* Line 331: “its”
* Figure 13: images are quite difficult to see

limitations_and_societal_impact: yes
review_title: null

Meta-review: This paper studies the problem of sim2real domain adaptation, focusing on the bird's-eye-view vehicle segmentation task and on strategies for sampling data from a self-driving car simulator. The authors trained existing models and sampled data in a simulator-agnostic way in an effort to reduce the sim2real gap, compared their approach to standard domain adaptation techniques and demonstrated that domain adversarial training outperformed simple style transfer approaches.

Reviewers have praised the clarity of the paper and thoroughness of the experiments, and noted that this research was “very relevant to demonstrating which types of variation (ie. map design, and number of vehicles) actually matter to self-driving car simulators”.
Reviewers’ claims about “novelty” are easy to make and to dismiss, and I note that the authors provide an in-depth mathematical formalism and analysis of the simulation-reality gap. 
Reviewers also had concerns about missing evaluation and comparison of SOTA models on nuScenes (TDjR and z4FW) - the authors have addressed these by running additional experiments comparing their method with “HorizonLiDAR3D" from the 3D detection track and the domain adaptation track in Waymo Open Dataset Challenge at CVPR 2020, training the models on CARLA; moreover reviewer z4FW praised the “focus on what parts of the simulator are actually relevant to transfer between datasets”. Reviewers gzio and LRhN had additional small concerns about photorealism, focus on bird eye view, heuristics and spatial priors, and the authors committed to answer them.

Reviewers TDjR and z4FW did not answer the authors’ rebuttal but I believe they should and would increase their score. I am therefore willing to challenge these two reviewers and promote this paper to an acceptance.

Decision: Accept (Poster)