Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation
Authors: David Acuna
Jonah Philion
Sanja Fidler
Venue: NeurIPS 2021 Poster (2021)
Keywords: Self-Driving
Domain Adaptation
Computer Vision
Transfer Learning

Abstract: Autonomous driving relies on a huge volume of real-world data to be labeled to high precision.  Alternative solutions seek to exploit driving simulators that can generate large amounts of labeled data with a plethora of content variations. However, the domain gap between the synthetic and real data remains,  raising the following important question: What are the best way to utilize a self-driving simulator for perception tasks?. In this work, we build on top of recent advances in domain-adaptation theory, and from this perspective, propose ways to minimize the reality gap. We primarily focus on the use of labels in the synthetic domain alone. Our approach introduces both a principled way to learn neural-invariant representations and a  theoretically inspired view on how to sample the data from the simulator. Our method is easy to implement in practice as it is agnostic of the network architecture and the choice of the simulator.   We showcase our approach on the bird's-eye-view vehicle segmentation task with multi-sensor data (cameras, lidar) using an open-source simulator (CARLA), and evaluate the entire framework on a real-world dataset (nuScenes). Last but not least, we show what types of variations (e.g. weather conditions, number of assets, map design and color diversity) matter to perception networks when trained with driving simulators, and which ones can be compensated for with our domain adaptation technique. 

Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation
David Acuna, Jonah Philion; Sanja Fidler
NVIDIA, University of Toronto, Vector Institute (dacunamarrer, jphilion, sfidler) @nvidia.com com
Abstract
Autonomous driving relies a huge volume of real-world data to be labeled to high precision. Alternative solutions seek to exploit driving simulators that can generate large amounts of labeled data with plethora of content variations. However, the domain gap between the synthetic and real data remains, raising the following important question: What are the best way to utilize a self-driving simulator for perception tasks? In this work, we build on top of recent advances in domain domain-adaptann adaptation theory, and from this perspective, propose ways to minimize the reality gap. We primarily focus on the use of labels in the synthetic domain alone. Our approach introduces both principled way to learn representations and theoretically inspired view on how to sample the data from the simulator. Our method easy to implement in practice as it agnostic of the network architecture and the choice of the simulator. We showcase our approach on the bird's "ord's-eye-views vehicle segmentation task with multi-sensor data (cameras, lidar) using an open-source simulator (CARLA), and evaluate the entire framework on a real-world dataset (nuScenes). Last but not least, we show what types of variations (e.g. weather conditions, number of assets, map design and color diversity) matter to perception networks when trained with driving simulators, and which ones can be compensated for with our domain adaptation technique.
Introduction
The dominant strategy in self-driving for training perception models is to deploy cars that collect massive amounts of real-world data, hire large pool of annotators label and then train the models on that data using supervised learning. Although this approach is likely to succeed asymptotically, the financial cost scales with the amount of data being collected and labeled. Furthermore, changing sensors may require redoing the effort to large extent. Some tasks such as labeling ambiguous far away or occluded objects may hard or even impossible for humans.
In comparison, sampling data from self-driving simulators such as 46 First, one has control over the content inside a simulator which makes all self-driving scenarios equally efficient to generate, indepedent of how rare the event might be in the real world. Second, full world state information is known in simulator, allowing one to synthesize perfect labels for information that humans might annotate noisily, as well as labels for fully occluded objects that would be impossible label outside of simulation. Finally, one has control over sensor extrinsics and intrinsics in a simulator, allowing one to collect data for a fixed scenario under any chosen sensor rig.
6 13
has several benefits.
Given these features of simulation, it would be tremendously valuable be able train deployable perception models on data obtained from a self-driving simulation. There are, however, two critical challenges in using driving simulator out of the box. First, sensor models in simulation do not mimic real world sensors perfectly; even with exact calibration of extrinsics and intrinsics, it is extremely
denotes equal contribution, httpp//httt///t//aabbbbbb htt.::llwwwiiiiuuiiiiiimaaioooso
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Figure I: nuScenes vs CARLA Visualization of real-world data & labels from nuScenes 7 (top) and synthetic data & labels we obtain from CARLA [1] (bottom). As in nuScenes, we scrape 6 cameras, LiDAR, ego-motion (for temporal fusion of LiDAR seans), and 3D bounding boxes from CARLA. On the right, show the target binary image for the bird's sind's-eye-view vehicle segmentation task that we consider in this paper
22
hard to perfectly label all scene materials and model complicated interactions between sensors and objects at least today [34 g 55 42 Second, real world content is difficult to incorporate into simulation; world layout may be different, object assets may not be diverse enough, and synthetic materials, textures and weather may all not align well with their real world counterparts [51] 6 19 (55. To make matters worse, machine learning models and more specifically neural networks are very sensitive these discrepancies, and are known to generalize poorly from virtual (source domain) to real world (target domain) [52] 14
Contributions and Outline. In this paper, we seek to find the best strategies for exploiting a driving simulator, that can alleviate some of the aforementioned problems. Specifically, we build on recent advances in domain adaptation theory, and from this perspective, propose a theoretically inspired method for synthetic data generation. We also propose a novel domain adaptation method that extends the recently proposed f-DAL (3] to dense prediction tasks and combines it with pscudo-labels.
We start by formalizing our problem from DA perspective. In Section 2 we introduce the mathe- matical tools, generalization bounds and highlight main similarities and differences between standard DA and learning from a simulator. Our analysis leads to a simple but effective method for sampling data from the self-driving simulator introduced in Section 3 This technique is simulator agnostic, simple to implement in practice and builds on the principle of reducing the distance between the labels' marginals, thereby reducing the domain-gan gap and allowing adversarial frameworks to learn meaningfully in a representation space. Section 4 generalizes recently proposed discrepancy mini- mization adversarial methods using Pearson x² 3 to dense prediction tasks. It also shows novel way to combine domain adversarial methods with pseudo-labels. In Section 6 we experimentally validate the efficacy of our method and show the same approach can be simply applied to different sensors and data modalities (oftentimes implying different architectures). Although we focus on the scenario where labeled data is not available in the target domain, show our framework also provides gains when labels in the target domain are available. Finally, we show that variations such as number of vehicle assets, map design and camera post-processing effects can be compensated for with our method, thus showing what variations are less important to include in the simulator.
2 Learning from a Simulator: A Domain Adaptation Perspective
We can learn great deal about a model's generalization capabilities under distribution shifts by analyzing its corresponding binary classifier. Therefore, building on the existing work in domain- adaptation theory (5] 4 59 3. we assume the output domain be y {0, 1} and restrict the mathematical analysis to the binary classification setting. Section O shows experiments in more general settings, validating the usefulness of this perspective and our algorithmic solution which is inspired by the theoretical analysis presented in this section. Specifically, here, we interpret learning from simulator as a domain adaptation problem. We first formulate the problem and introduce the notation that will be used throughout the work. Then, we introduce mathematical tools and generalization bounds that this work builds upon. We also highlight main similarities and differrences between standard DA and learning from simulator.

---

We start by assuming the self-driving simulator can automatically produce labels for the task in hand (as is typical), and refer data samples obtained from the simulator as the synthetic dataset (S) with labeled datapoints S = [xff yf ) Let's also assume we have access to another dataset (T) with unlabeled examples T ((x{) Nt
== that are collected the real world. We emphasize that no labels are available for T. Our goal then is to learn a model (hypothesis) h for the task using data from the labeled dataset S such that / performs well in the real-world. Intuitively, one should incorporate the unlabeled dataset T during the learning process it captures the real world data distribution.
Certainly, there are different ways in which the dataset S can be generated (e.g. domain randomization [52 38) D. and also several algorithms that practitioner can come up with in order to solve the task (e.g. style transfer). In order to narrow the scope and propose a formal solution to the problem. we propose to interpret this problem from DA perspective. The goal of this learning paradigm is to deal with the lack of labeled data in target domain (e.g real world) by transferring knowledge from a labeled source domain (e.g. virtual word). Therefore, clearly applicable to our problem setting.
In order to properly formalize this view, we must add few extra assumptions and notation. Specifi- cally, we assume that both the source inputs x and target inputs x are sampled i.i.d. from distributions P. and P3 respectively, both over X. We assume the output domain be y' = {00, 1] and define the indicator of performance to be the risk of a hypothesis h : X + )' w.r.t. the labeling function f. using a loss function f Y' y + R., under distribution D, or formally Ry (h. f) := Ep (()(())) f()) We let the labeling function be the optimal Bayes classifier fzz) = arg max Ply = u [35]. where P(y[z) denotes the class conditional distribution for either the source-domain (simulator)
(P)( or the target domain (real-world) (Pi (v) For simplicity, we define R'g(h) == R'p, (h. and (h) == R2y, st).
For reasons that will become obvious later (Sec. 4. will interpret the hypothesis h (model) the composition of h h g with g X + Z, and h: Z + V, where Z is a representation space. Thus, we define the hypothesis class H == {h hhgg g: h E H. g E g} such that h E H. With this in hand, we can use the generalization bound from B to measure the generalization performance of a model h:
Theorem (Acuna et al. 3 ). Suppose e: y x y + [0, 1] and denote := minKEH R6
+
(h). We have:
(1) S (b) + DD. (P.(IP) + X", where
(1)
where: D DI. (P.IIP)) suPWEH PB ((h()))) M -B ((((()))) and the func- tion o: R+ + IR defines a particular f-divergence with o² being its (Fenchel) conjugate.
Theorem important for our analysis as it shows what we have take into account such that a model can generalize from the virtual to the real-world. Intuitively, the first term in the bound accounts for the performance of the model on simulation, and shows us, first and foremost, that we must perform well on S (this is intuitive). The second term corresponds to the discrepancy between the marginal distributions (x) and Pi(z). in simpler words, how dissimilar the virtual and real world are from an observer's view (this is also intuitive). The third term measures the ideal joint hypothesis (X*) which incorporates the notion of adaptability and can be tracked back to the dissimilarity between the labeling functions 3 [5 4 In short, when the optimal hypothesis performs poorly in either domain, we cannot expect successful adaptation. We remark that this condition is
necessary for adaptation 5
3 (59].
Comparison vs standard DA. In standard DA, two main lines of work dominate. These mainly depend on what assumptions are placed on X The first group (1) assumes that the last term in equation [ negligible thus learning is performed in an adversarial fashion by minimizing the risk in source domain and the domain discrepancy in Z [16] 3 (3] (5]. The second group (2) introduces reweighting schemes to account for the dissimilarity between the label marginals (x()) and P((y). This can be either implicit [24] or explicit (32 50, 48) In our scenario the synthetic dataset is sampled from the self-driving simulator. Therefore, we have control over how the data generated, what classes appear, the frequency of appearance, what variations could be introduced and where the assets are placed. Thus, Theorem [ additionally tell us that the data generation process must be done in such a way that X is negligible, and if so, we could focus only on learning swarriantrppresennaaiion We exploit this idea in Section [3] for the data generation procedure, and in Section 4 for the training algorithm. We emphasize the importance of controlling the dissimilarity between the label marginals
3

---

Samsing
(b) Map Samping Siawagy
(c) Siuaaal Pil0
(4) SuSconss Treas
(e) rudcomnss Vakdabon Ai
COCOET
00077
Figure 2: Marginals induced by different sampling strategies for the task of BEV vehicle segmentation. Figures (a) and (b) show induced marginal (y( when the sampling strategy follows standard approaches: e.g. the NPC locations are sampled based on the structure of the road or the map's drivable area (pseudocode appendix). Figure (c) shows the induced marginal when the NPC locations are sampled using our spatial prior (see See 3. Figure (d) is for reference as it assumes access to target labeled data. It shows the induced marginal when NPC locations are sampled using a target prior estimated on nuScenes training set (see SecB.2]. Figure (e) shows the estimated label marginal of the nuScenes validation set for the task of vehicle BEV segmentation P(y). The number on the top- the figures corresponds to the JSD between the induced marginal, and the nuScenes validation set estimated marginal (lower is hetter). In comparison with standard approaches, our sampling strategy minimizes the divergence hetween the task label marginals (e.g. 5.7e vs 1.40e-9 and leads to more diverse synthetic dataset. Notably, comparing (c) and (d), we observe their distances are in the same order, yet sampling based on the spatial prior does not require access to labeled data.
(y( and P((2), as it determines the training strategy that we could use. This is illustrated by the following lower bound from [59]:
Theorem (Zhao et al. (59)) Suppose that Das (P.(W)I)P()) 2 Das (P:())))P((E)) We have:
(h) + (b) 2 (VDis (P.U)) -Doo) (P. (P.())) where Das corresponds to the tensen-Shanon divergence.
Theorem 2 is important since it is placing lower bound the joint risk, and it is particularly applicable to algorithms that aim to learn domain-invariant representations. In our case, it illustrates the paramount importance of the data generation procedure in the simulator. If we deliberately sample data from the simulator and position objects in way that creates a mismatch between real and virtual world, simply minimizing the risk in the source domain and the discrepancy between source and target domain in the representation space may not help. Simply put, failing to align the marginals may prevent from using recent SoTA adversarial learning algorithms.
3 Synthetic Data Generation
In the previous section, analyzed learning from simulator from a domain adaptation perspective, and showed the importance of the data sampling strategy as this could create mismatch between the label marginals. In this section, we take insipiration from this analysis and propose two simple, but effective methods to sample data from the self-driving simulator. The first one proposes the use of spatial priors and is targeted a regime where there a complete absence of real-world labeled data. The second one assumes that few datapoints are labeled in the target domain and exploits these labels estimate a prior for the positions of the vehicles in the map.
3.1 Sampling with Spatial Priors
If labels in the target domain are not available, we cannot measure the divergence between the labels marginals P(y) between the generated and the real world data. That said, given the birde view segmentation map, it is not hard for human to design spatial priors representing locations with high probability of finding a vehicle. As visualized in Figure [] we design simple prior that samples locations for non- non-player characters (NPCs) proportional to the longi- tudinal distance of the vehicle to the ego car. For position (21, .22) E TR22 in meters relative to the ego car:
Figure 3: Sampling Methods (Left) Stan- dard approaches sample NPC locations from the map's 's drivable area based on the struc- ture of the road. (Right) We sample the NPC locations based on our spatial prior (indepen- dent of the structure of the road) which aims for diversity the generated data and mini- mizes the divergence between the task label marginals (see also Fig 2
22
0.6
|x2||
<
12.5
P spattil (22) x
125
(2)
[|zz)
50)
12.5
75

---

Vidicctss Per Scene
Camera Field of View (etriinic))
Camera Yow (extrinsic))
Under Height
1000
1700
200
800
470
100
Oounts
500
8CO
400
E00
100
200
100
23 10 70 102
1.20 100
2:243
3.812
1.5
(degrees)
Height
Figure 4: Matching nuScenes Statistics From left right, we show the distribution of number of vehicles per scene nuScenes, the distribution of field of view the nuScenes cameras, the yaw relative to ego coordinate frame of the cameras (6 peaks for the 6 different camera directions), and the height of the LiDAR (roughly the same across all scenes). These statistics are matched when sampling data from CARLA.
These numbers correspond to linearly interpolating between a probability proportional 0.0 for [æ2| = 50 meters, 0.5 for |2y] = 12.5 meters, and 0.6 for [z22 = 0 meters. Intuitively, the prior models the fact that other vehicles are generally located along the length-wiss axis of the ego vehicle. The density is independent of æ1 which models our prior that on straight road, vehicle is equally likely to be ; meters front or behind the ego, for any &t E R.
3.2 Sampling based on Target Priors
Often in practical applications, a small proportion of data is available in the target domain. In such scenarios a natural prior for sampling would be to compute Pv(y) (see Figure 2 for an example), and generate the data based on that. Depending on how large the amount of data in the target domain is, we could also create blend a Psspaiall + (1 - P; and sample NPC locations from it, for any [0, 1]. We emphasize that we do not sampling based on target priors in our experiments since our work focuses on the challenging setting where labeled data is not available in target domain.
4 Training Strategy
In the previous section, we designed simulator-agnostic sampling strategy that builds on the intuition of minimizing the distance between the label marginals. From Theorem [ we can also observe that in order to improve performance in the real world. we must minimize the distance between the input distributions P. (x) and Pz(z). In this section, we extend the training algorithm from [3] to dense prediction tasks, with the aim to learn domain-invarianl representations and minimize the divergence between virtual and real world in latent space Z. Minimizing the divergence in representation space allows domain adaptation algorithms be sensor and architecture agnostic. We additionally take inspiration from [7] and incorporate the use of pseudolabels into the f-DAL framework.
We remark in our scenario we cannot effectively learn invariant representations using adversarial learning unless the data sampling strategy from Section 3 is used because otherwise there may be a misalignment between the task label marginals (see Section 2 and 3 59 for more details).
Learning Domain-Invariant Representations. Our training algorithm can be interpreted as simul- tancously minimizing the source domain error and aligning the two distributions (virtual and real worlds) in some representation space Z. Formally, we aim at finding a hypothesis h that jointly minimizes the first terms of Theorem 1 Let h X y with y R. which corresponds to pixel-wise binary segmentation map of dimensions and w respectively, with X' corresponding to an unordered set of multi-vier images for camera-based based bird's s-eye-view segmentation X' N-3 H.W and point cloud for the lidar-based bird's pond's-eye-viess segmentation X' C R-3 Our objective function is then formulated in adversarial fashion by minimizing the following objective:
min max heg MER
Er.y-p, (6hh o 9, y)) + dst;
(3)
where f: R. Rh.-w + R, is defined as: E(p, q) == Ae Σ Xeo 3p, log (1 pc) log(1 q;),
h-w
and p3 and qi E [0, 1] (the averaged binary-cross entropy loss). B is hyperparameter that weights the importance of positive pixels in the pixel-wise binary segmentation map. We define dee as:
1
dst = Eanp, En Sh o g)]
Exnpe
En (h' o g)t + h g(x))
(4)
5

---

i== Ty. Different from the original formulation of f-DAL Pearson [3]. dat can be interpreted in our case as a per-location- domain-classifer. For simplicity, we use the gradient reversal layer from 16 15 to deal with the min-max objective single wrwaaddbbckwward pass.
which corresponds to the Pearson x² divergence, and B/ 1 h-w Σ A-w
Pseudo-Labels. In addition to the objective in Equation
3
we use the following pseudo-loss:
Epocudo (p, Pmng)
1 pp 2 =|13 log Pange
1 Pi S log(1 Pange)
(5)
where [0. 1). [xx corresponds to the indicator function, p h g(x²) and Paug == h g aug (x²) with aug X' - X' a function that produces a strong augmentation on the input data point. For camera sensors, we let aug be version of RandAugment [10] that additionally incorporates camera dropping. For lidar, we follow the same idea from [TO] but replace the image augmentations by random noise over the points positions as well as points dropout. We let 0.9 all our experiments. More details are provided in the supplementary material.
The objective in Equation 5 inspired by 47). Intuitively, it is based on the generation of pseudo- labels using the confident pixels in the model's predictions, as determined by т, on the target domain, non-strongly augmented version of the same data point. Therefore, it explicitly exposes the model during training to real-world data. We experimentally observed that using pseudo-labels without enforcing invariant representations performs worse than a simple vanilla model trained only on the source dataset, likely because as opposed to [47) in our scenario the target/unlabeles data comes from different distributions. The use of pseudo-labels and adversarial learning in our algorithm can be justified through the results of [45] and Theorem In summary. our training algorithm jointly minimizes equations
3 and
Other theoretically inspired training strategies. In principle several other training algorithms that minimize the discrepancy between source and target can be used. For example, style transfer
using MUNIT [23]. the original formulation of [16] or f-DAL 3]. Experimentally. we found our generalization of f f-DAL-Pearson and the use of pseudo-labels to be more performant (see Table 3 We hypothesise the reason for domain-invariant representation methods being better is because these minimize the dissimilarity between source and target in a low dimensional space. In contrast, style transfer approaches operate directly on higher dimensional input data (e.g. images). Moreover, the use of pseudo- labels further exposes the model to training examples on real-world data.
Figure 5: Style Transfer Baseline (MUNIT) our style transfer baseline, we sample style indepedently for each image and "translate" the CARLA images into nuScenes as shown above. The Lift Splat model receives images like those on the right input during training.
5 Related Work
Synthetic Data Generation. The use of synthetic data as alternative to dataset collection and annotation has received significant interest in recent years. Synthetic data with labels can be generated in different ways, e.g., with generative models such GANs [58] 31], data-driven reconstruction
with sensor simulation [34] g 27]. or via the use of graphics simulators. We here focus on the latter.
Various semantic segmentation, and home roboties among many others (39 36 4 AT. Several techniques
synthetic datasets have been created for tasks such as object detection,
have been proposed to generate useful labeled data from the graphics engine, informed by the target real data. For example, [52] proposed to randomize the parameters of the simulator in noa-realistic ways with the aim of forcing the neural network to learn the essential features of the object of
interest. In a similar vein, (38] proposes to procedurally generate synthetic images while preserving the structure, or context, of the problem at hand. Instead of randomizing, (25
30 proposed
data-driven approach. Specifically, the authors aim to resemble a target dataset by searching over set of parameters of surrogate function that interfaces with synthesizer. These methods are all

---

Lift Splat
Lift Adipt
Oracee
Figure 6: Lift Splat Adapt We visualize the output of Lift Splat (LS) for single timestep from nuScenes val. The model takes as input the 6 images shown on the left, outputs the heatmap shown the right, and we project the prediction back onto the input images colorized according depth. Top L. model without adaptation, middle is model trained with our method, and hottom is model trained directly on nuScenes.
different from us because 1) they require access to the procedural model of the simulator, thus they not easy apply to off-the-self self-driving simulators, they are focused on the generation of single dashcam RGB images instead of a sensor suite, and 3) data-driven approaches could require a significant amount of data in the target domain. Most importantly, the transfer performance that practitioner may observe lacks justification and guidance, c.g. methods such as domain randomization may increase the divergence between the label marginals not carefully done.
Domain Adaptation on Synthetic Data. There is significant body of work on domain adaptation ranging from theory and algorithms to applications 4 [5 3 [6] [5] some of them applicable to sim-to-real datasets such 41 and tasks such semantic segmentation and object detection [[8] 26 (33. Most of the literature in this direction however analyzes the problem with a fixed dataset perspective. They assume that the source and target datasets are given and aim find way to minimize the gap. In our formulation, we have access to the self-driving simulator. Therefore, we have control over how the data is generated, what variations are introduced and where the objects placed. Our method thus is a unified view that minimizes the discrepancy between source and target distribution by building previous work such as [17 R and ensuring the effectiveness of them through a sampling strategy that aims at minimizing the distance hetween the labels' marginals.
6 Experimental Section
In this section we quantify the performance of the proposed data-generation and training strategies using open source self-driving simulator (Carla, MIT license (13) and real-world dataset (nuScenes, Apache license (7]) We first introduce the self-driving simulator and the data generation setup. We then discuss the methods in comparison and our proposed basclines. In Section 6.2 show our main experimental results on the task bird-eye-vier (BEV) vehicle segmentation. Finally, we provide extensive experimental analysis where ablate data and training strategy, and aim to understand what defficiencies the self-driving simulator can be compensated with our approach.
6.1 Self-Driving Simulator and Methods in Comparison
Self-Driving Simulator. We use CARLA version 0.9.1 the self-driving simulator. For the synthetic data generation, we sample nuScenes-like datasets from CARLA such that dataset consists of 4000 episodes; each episode lasts 4 seconds during which all vehicles are controlled by CARLA's
7

---

default traffic manager. 3D bounding boxes and images are stored synchronously at 2hz [54]. and LiDAR scans are stored at 20hz as in nuScenes 77. We also store the ego pose facilitate accumulating LiDAR scans across multiple timesteps. Each CARLA dataset is 260 GB. We split each dataset into a training set of 28k timesteps (which matches the number of timesteps in the nuScenes training set) and validation set of 4k timesteps. The datasets and scripts for generating them will be publicly released. In the base case for all datasets, we match the distribution number of vehicles per episode and the distribution of camera and lidar extrinsics and intrinsics nuScenes (see Fig 4).
Methods in Comparison. For the Synthetic Data Generation strategy, we propose a base- line where we sample data based on the road structure, and randomize what is possible in the simulator, e.g. sampling the color of vehicle assets or weather parameters independently and uniformly. This method inspired by (52] 38 but within the realistic restriction of the self- driving simulator. We refer to this strategy as RS (road structure). We also add extra base- lines on top of RS to account for the domain- gap: e.g. style transfer with MUNIT (23] and using domain adaptation method inspired by [6]. Details on the MUNIT architecture can be found in the supplementary. We refer to these as US-Style-Trannter and RS-DANN respectively. If adaptation is not used refer to it RS- No-Adaptation. Finally, we have two additional baselines inspired by [12] which we call RS- Ensemble and RS-Ensemble + Test-time Aug. Since these ensemble baselines to choose the prediction, they represent an upper-bound on the performance of ensemble base- lines used in the Waymo adaptation challenge [12). More details in the supplementary material.
Table 1: Lift- Lift-Splat Sim - Real. We compare the per- formance of our method RS. account for the reality gap. we also show results of RS with different adaptation techniques.
Method
IOU
Ro-No Adaptation
9.76
RS- Ensemble (Inspired by (12)
11.95
RS-Style Transfer (MUNIT)
13.33
RS-DANN
13.76
RS- 35-Fnsemble + Test-time Aug (Inspired by 12]
13.76
Ours
17.84
ground-truth
6.2 Bird's-Eyees S-ees View Vehicle Segmentation
Table 2: Point Pillars Sim +
Real. We compare performance of our method vs RS.
Our main experiments and analysis are based on the task of BEV vehicle segmentation from multi-camera sensor data. We additionally compare to baselines using Lidar observations.
Method
IOU
Camera. In this experiment, the input data corresponds to an un- ordered set of multi-view images. We use Lift Splat method (37. We
RS-No Adaptation
15.09
RS-DANN
14.41
use the same backbone [49 and training scheme
228 as in 37
Ours
17.20
Lidar. In this experiment the input data corresponds to point cloud obatained from the lidar scan. We additionally show our method can be used in a completely different sensor such as Lidar Data. For the network architecture, we follow Point Pillars [29]. standard LiDAR architecture consisting of shallow pointnet [40] followed by a deep 2D CNN based on resnet18 18 (20).
Semi-Supervised Scenario. We also show experiments if labeled real world data is available. In Figure we plot transfer performance when we use 2%, 5%, and 25% of nuScenes training set.
6.3 Analysis
In this section, we first ablate the choice of the training strategy and then analyze the correlation between transfer performance and the distance between the labels' marginals as this constitutes the main motivation behind our sampling strategy. We then analyze functionality of the self-driving simulator to improve transfer performance and what deficiencies can be compensated with adaptation.
Training Strategy In Table we keep the data generation strategy fixed and evaluate the performance of different train- ing strategy such as Style-Transfer, DANN, our generalized version of f-DAL, and f-DAL-PParson with pseudo-labele as proposed in section [ Our method achieves the best results. Similar to what was found [3) we observe f-ALL performs better than DANN. We also observe that domain-adversarte approaches perform better than Style-Transfer We believe this is because it is easier to close the reality gap in a latent space rather than in the very high dimensional image space.
Table 3: Ablation on the Training Strat-
egy. In this scenario, we fix the data- generation strategy to he the best.
Name
IOU
No Adaptation
10.33
Style Transfer (MUNIT)
14.32
DANN
14.45
I.DAL.Jensen
16.19
F-DAL Pearson
17.03
Ours
17.84
8

---

muScendd Valdason IOU
DiScoman
Jensen-Shhamn Divirgee genno
Town
Toan bwih Sarnsing Prod
Figure 7: IOU vs. JSD Transfer performance is negatively correlated with the distance between P.(y) and Pi(y) as expected from Theorem
Training (9)
Figure Better Sampling Improves Town Robust- ness We compare transfer performance of Lift Splat models when NPCs are sampled according to geometry of the town roads (left) vs the hard-coded prior (right). Performance greatly improves when using sampling strategy that doesn't 't condition on road structure.
Figure 8: Semi-Super. Learning Our method im- proves over baselines when 5% of the labeled data is availahle training.
Carera OH0F2A DE Na5o (regress)
Garees www.Food of New Noce (Ongrees)
14 Stoggh
Wotthor Divesitt
Figure 10: Camera Intrinsics and Extrinsics Instead of sampling cam- era extrinsics and intrinsics that exactly correspond to extrinsics and intrin- sics from nuScenes. we sample yaw (left) and field of view (middle) from nuScenes with some variance.
Figure 11: Weather varia- tion does appear he a limiting factor in our setting.
Transfer performance vs JSD We also evaluate the distance between the label marginals computed as 1as ((()))))((() vs transfer performance (loU). Datasets with a smaller distance perform better.
Maps When simulating with CARLA, different "towns" can be chosen. Each town has different road topology and different static obstacles like buildings and foliage. In Figure 9 we show that performance is sensitive to the town if simple sampling strategy based on the road structure is used. However, by sampling NPC locations to minimize the JSD in the marginals, we increase performance by large amount.
Vehicle Assets An intuitive goal for self-driving simulators has been to improve the quantity and quality of vehicle assets [55] 19). In Figure 12] we show that performance is sensitive to the number of vehicle assets used in the simulator and the number of NPCs sampled per episode, but is not so sensitive to the variance in the colors chosen for the NPCs.
Tital
LIT M
Validation OOU
nuSoones Validation (QU
nuBieen Valdiatoon lou
0.00
nuSoenes
0.00
0.00
0.6
0.000
0.00
0.0
Vehiide Assets
Samplnng Prior)
Vehicles Per Episade
Car Color Diversity
Figure 12: Vehicle Assets (Left) We vary the number of vehicle assets used when sampling CARLA data. Interestingly, Lift-Splat-Aaapt (ours) is able to compensate for performance when very few assets are used. (Middle) We compare performance when the number of NPCs per episode sampled from (n, 40), fixed 1/10/20/30/40 NPCs per episode, or sampled from the distribution in Figure 4 Finally, we compare performance on datasets in which car colors are independently sampled from Unif 2) vs. default car colors.
Weather nuScenes contains sunny, rainy, and nighttime scenes. We test the extent to which it's important for CARLA data to also have variance weather in Figure [T] CARLA has 15 different "standard" weather settings as well as controls for cloudiness, precipitation, precipitation deposits (e.g. puddles), wind intensity, wetness, fog density, and sun altitude angle (which controls nighttime vs. daytime on a sliding scale). We randomly sample these controls from x ~ (ni v) for V. E {0.2, 0.4, 0.6, 0.8, 1.0) and compare performance against sampling categorically from the 15 preset weather settings. We also compare against exclusively using the "sunny" weather setting but sampling the location of the NPCs according to the hard-coded prior. We find that our method is able to compensate for much of the loss in weather diversity and using only the sunny weather setting on
9

---

data where NPCs are sampled according to the prior achieves higher performance than any amount of weather diversity with NPCs sampled according the town.
Camera Post-Processing The CARLA "scene final" camera provides a view of the seene after applying some post-processing effects create vignette, grain jitter, bloom, auto exposure, lens flares and depth of field as post-procens effects [ In figure 7 we evaluate the importance of this post-processing stage. We can see adaptation is able compensate for the loss in realism by a large amount.
realistic feel. Specifically, CARLA applies
7 Limitations and Societal Impact
LP-500
215
15.
Limitations In Figure we qualitatively demonstrate cer- tain limitations of our method. It is unlikely that adaptation can be used to fix errors for vehicles that look significantly different from the vehicle assets used in CARLA. We also
BuScenes Valcat
12
found that our adapted model misdetects pedestrians for
vehicles occasionally. perhaps because it has Iearned non-causal correlation between legs and bicicyles. Finally, the adapted model occasionally midpredicts the depth of vehicles.
Figure 13: Traffic Model Performance is unaffected by whether not traffic model is used when generating synthetic data.
Societal Impact In this paper, we primarily seek to opti- mize the intersection-over-uiion of bird's sird's-eye-view seg-
mentation models by leveraging synthetic data rendered in CARLA. While simulated data can controlled mit-
nuBeenes Vallattonio
igate some of the biases of real world data (for instance by uniformly sampling vehicles of any color), synthetic data also brings it's own biases in that not all materials, lighting conditions, or weather patterns can be simulated equally realistically; snow is notoriously difficult to sim- ulate correctly, for instance, so simulated data may not be the solution for improving perception in snowy climates. We believe using simulation will be crucial for training and testing safe self-driving systems that have the potential make public roads safer and more efficient.
Figure 14: Camera Post-Proee Post-Processing
CARLA uses post-processing improve the realism of images. We evaluate the impor- tance of this post-proeessig by toggling "en- able postprocess effects". Adaptation is able to compensate for the loss in realism by large amount.
Limitations Predictions the hest Lift Splat Adapt model vs. ground-truth when the model makes an error (Left) The model does not detect the orange bus in the garage. (Middle) The model predicts a vehicle where there are pedestrians. (Right) The model predicts behind is much closer than it actually is.
Figure 15:
8 Conclusion
Using recent advances in domain adaptation, we motivate methods for both sampling and for training on synthetic data such that models transfer well to real world data. We demonstrate the effectiveness of our method by training camera-based and lidar-based bird's ded''-eee- vehicle segmentation models on data sampled from the CARLA simulator and evaluated real world data from nuScenes.
10

---

References
[1] Rgh camera, carla simulator documentation. http:://errla.rrdddhecoss
0.9. 0.9.11/ref sensors/ Accessed: 2021-10-20.
[2] Nvidia drive sim, powered by omniverse. https: Ttttp::/www.ntt self-ariving-carss htw./aarsssmmlmaaiinn.ooooo Accessed: 2021-05-20.
[3] David Acuna, Guojun Zhang, Mare Law, and Sanja Fidler. Adomain-adversariaa learning: Theory and algorithms. In Proceedings of the 38th International Conference on Machine Learning, pages 66-75. PMLR, 2021.
[4] Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jen- nifer Wortman Vaughan. A theory learning from different domains. Machine learning, 79 (1-2):151-175, 2010.
[5] Shai Ben-David, Tyler Lu, Teresa Luu, and Dávid Pál. Impossibility theorems for domain adaptation. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pages 129-136. 2010.
[6] Alex Bewley, Jessica Rigley, Yuxuan Liu, Jeffrey Hawke, Richard Shen, Vinh-Dieu Lam, and Alex Kendall. Learning drive from simulation without real world labels. In 2019 International Conference on Robotics and Automation (ICRA), pages 4818-4824, 2019, doi:
10. (100//CCAA2019.8879999
[7] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. CoRR, abs/1903. 1027, 2019. URL httphttpp//axxee
abs/ / 11027
[8] Yun Chen, Frieda Rong, Shivam Duggal, Shenlong Wang, Xinchen Yan, Sivabalan Mani- vasagam, Shangjie Xue, Ersin Yumer, and Raquel Urtasun. Geosim: Photorealistic im- age simulation with geometry-aware composition. CoRR, abs/2101.06543, 2021. URL
https::/aaxxv.orrg/ abs http://aas/211.....0. 06543
[9] Dami Choi, Christopher J. Shallue, Zachary Nado, Jaehoon Lee, Chris J. Maddison, and George E. Dahl. On empirical comparisons of optimizers for deep learning. CoRR, abs/1910.05446, 2019. Utt.httt:///axxivv.o
abs 911/1191..00 05446
[10] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition Workshops, pages 702-703, 2020.
[11] Jeevan Devaranjan, Amlan Kar, and Sanja Fidler. Meta-sim2: Unsupervised learning of scene structure for synthetic data generation. CoRR, Abs/2008.09092, 2020. URL https
09092
[12) Zhuangzhuang Ding, Yihan Hu, Runzhou Ge, Li Huang, Sijia Chen, Yu Wang, and Jie Liao. 1st place solution for waymo open dataset challenge 3d detection and domain adaptation. arXiv preprint arXiv:2006. 15505, 2020.
[13] Alexey Dosovitskiy, Germán Ros, Felipe Codevilla, Antonio M. López, and Vladlen Koltun. CARLA: an open urban driving simulator. CoRR. abs/1711.00992 2017. URL http://
arxiv. http:////aabbs/1111/ 03938
[14] Adrien Gaidon, Qiao Wang, Yohann Cabon, and Eleonora Vig. Virtual worlds as proxy for multi-object tracking analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4340-4349, 2016.
[15] Yaroslav Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In International conference on machine learning, pages 1180-1189. PMLR, 2015.
[16] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural networks. The Journal of Machine Learning Research, 17(1):2096-2033 2016.

---

[17] Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. -aneainnadvesariilll training of neural networks, 2016.
[18] Muhammad Ghifary, W Bastiaan Kleijn, and Mengjie Zhang. Domain adaptive neural networks for object recogmition. In Pacific Rim international conference on artificial intelligence, pages 898 1044 Springer, 2014.
[19] Jiayuan Gu, Wei-Chiu Ma, Sivabalan Manivasagam, Wenyuan Zeng. Zihao Wang. Yuwen Xiong, Hao Su, and Raquel Urtasun. Weakly-supervised 3d shape completion in the wild. CoRR, abs/2008.09110 2020. TRL/Attss//aaxxv..oo org ab00g///b.//... 0
[20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. CoRR, abs/1512.03385, 2015. URL
ab12
03385
[21] Judy Hoffman, Eric Tzeng, Taesung Park, Jun- Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pages 1989-1998. PMLR, 2018.
[22] Anthony Hu, 7ak Murez, Nikhil Mohan, Sofía Dudas, Jeff Hawke, Vijay Badrinarayanan, Roberto Cipolla, and Alex Kendall. FIERY: future instance prediction in bird's-eye view from surround monocular cameras. CoRR, abs/2104.10490 2021. URL https::/arxiv..r.
abs/2104.0 10490
[23] Xun Huang, Ming- Yu Liu, Serge J. Belongie, and Jan Kautz. Multimodal unsp:/eiidduuuuuuuurieieiddde imagett-- image translation. CoRR, abs/1 180.0404333 2018. URL
04132
[24] Xiang Jiang, Qicheng Lao, Stan Matwin, and Mohammad Havaei. Implicit class-conditioned domain alignment for unsupervised domain adaptation. In Hal Daumé III and Aarti Singh, editors, Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 4816-4827. PMLR, 13-18 Jul 2020. URL http://proceedinm
m. www...pp...vvv... wttpsssvv11// ang html
[25] Amlan Kar, Aayush Prakash, Ming- Yu Liu, Eric Cameracei, Justin Yuan, Matt Rusiniak, David Acuna, Antonio Torralba, and Sanja Fidler. Meta-sim: Learning to generate synthetic datasets. CoRR, abs/1904. 1621, 2019.
11621
[26] Mehran Khodabandeh, Arash Vahdat, Mani Ranjbar, and William G Macready. A robust learning approach to domain adaptive object detection. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 480-490, 2019.
[27] Seung Wook Kim, Jonah Philion, Antonio Torralba, and Sanja Fidler. Drivegan: Towards controllable high-quality neural simulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5820-5822 2021.
[28] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2014. URL http: /ttt:///rrxaa org/ abs 11l22 6980, cite uxxiv:1412 199..mmmtt Published as conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.
[29] Alex H. Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. CoRR, abs/18 111.01782. 2018. URL http:://axxii....../////// org/ waapgg/a/////// /112//112.... 05784
[30] Daiqing Li, Amlan Kar, Nishant Ravikumar, Alejandro F Frangi, and Sanja Fidler. Federated simulation for medical imaging. In MICCAI, 2020.
[31] Daiqing Li, Junlin Yang, Karsten Kreis, Antonio Torralba, and Sanja Fidler. Semantic segmenta- tion with generative models: remi-supervised learning and strong oft-o-nd generalization. arXiv preprint arXhv:2104.05883 2021.
12

---

[32] Zachary Lipton, Yu-Xiang Wang, and Alexander Smola. Detecting and correcting for label shift with black box predictors. In International conference on machine learning, pages 3122-3130. PMLR, 2018.
[33] Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael I Jordan. Conditional adversarial domain adaptation. Advances in Neural Information Processing Systems, pages 1640-1650, 2018.
[34] Sivabalan Manivasagam, Shenlong Wang. Kelvin Wong, Wenyuan Zeng, Mikita Sazanovich, Shuhan Tan, Bin Yang, Wei-Chiu Ma, and Raquel Urtasun. Lidarsim: Realistic lidar simulation by leveraging the real world. CoRR, abs/2006.09348. 2020.
org/
09348
[35] Mehryar Mohri, Afshin Rostamizadch, and Ameet Talwalkar. Foundations of machine learning. MIT press, 2018.
[36]
Xingchao Peng, Ben Usman, Necla Kaushik, Judy Hoffman, Dequan Wang, and Kate Saenko. Visda: The visual domain adaptation challenge. arXiv preprint arXiv:1710.06924, 2017.
[37] Jonah Philion and Sanja Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. CoRR, abs/2008.05711, 2020. URL https:t::///ax
org/aba 2008. 05711
[38] Aayush Prakash, Shaad Boochoon, Mark Brophy, David Acuna, Eric Cameracci, Gavriel State, Omer Shapira, and Stan Birchfield. Structured domain randomization: Bridging the reality gap by context-aware synthetic data. In 2019 International Conference on Robotics and Automation (ICRA), pages 7249-72477 7255. IEEE, 2019.
[39] Xavier Puig, Kevin Ra, Marko Boben, Jiaman Li, Tingwu Wang, Sanja Fidler, and Antonio Torralba. Virtualhome: Simulating household activities via programs. In CVPR, 2018.
[40] Charles Ruizhongtai Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. CORR, abs/1612.00593, 2016. URL
attt:://arxis org/ awwpaall....00 00593
[41] Stephan R Richter, Vibhav Vineet, Stefan Roth, and Vladlen Koltun. Playing for data: Ground truth from computer games. In European conference on computer vision, pages 102-118. Springer, 2016.
[42] Stephan R. Richter, Hassan Abu AlHaija, and Vladlen Koltun. Enhancing photorealism en- hancement, 2021.
[43] Thomas Roddick and Roberto Cipolla. Predicting semantic map representations from images using pyramid occupancy networks. CoRR, abs/2003.13403 2020. URL htttttt////xiiii
htg/abs/2003..0 13402
[44] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez. The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2016.
[45] Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. In Doina Precup and Yee Whye Teh, editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2988 -997.. PMLR, 06-11 Aug 2017. URRL/tttp:/pproceeeeee fress/v70/4 wtt/saaitoll wttp:///aaaool.aat...... html
mlr
[46] J. Scanlon, Kristofer D. Kusano, Tom Daniel, C. Alderson, A. Ogle, Trent Victor, and Waymo. Waymo simulated driving behavior in reconstructed fatal crashes within an autonomous vehicle operating domain. 2021.
[47] Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying semi-supervised learning with consistency and confidence. arXiv preprint arXiv:2001.07688 2020.
13

---

[48] Remi Tachet, Han Zhao, Yu-Xiang Wang. and Geoff Gordon. Domain adaptation with con- ditional distribution matching and generalized label shift. arXiv preprint SkIv:20003004475 2020.
[49] Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural networks. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 6105-6114. PMLR, 09-15 Jun 2019. URL http:::/prooceeiiggg... mlr
ptess/v97/tan19a.ii
[50] Shuhan Tan, Xingchao Peng, and Kate Saenko. Class-imbalanced domain adaptation: An empirical odyssey. In European Conference on Computer Vision, pages 585-602. Springer, 2020.
[51] Shuhan Tan, Kelvin Wong, Shenlong Wang, Sivabalan Manivasagam, Mengye Ren, and Raquel Urtasun. Scenegen: Learning to generate realistic traffic scenes. CoRR, abs/2101.06541, 2021.
URL htR.htttps///xxii..orr abs /201.. 06541
[52] Jonathan Tremblay, Aayush Prakash, David Acuna, Mark Brophy, Varun Jampani, Cem Anil, Thang To, Eric Cameracci, Shaad Boochoon, and Stan Birchfield. Training deep networks with synthetic data: Bridging the reality gap by domain randomization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops, pages 969-977, 2018.
[53] Haoran Wang, Tong Shen, Wei Zhang, Ling- Yu Duan, and Tao Mei. Classes matter: A fine- grained adversarial approach to cross-domain semantic segmentation. In European Conference on Computer Vision, pages 642-659. Springer, 2020.
[54] Anqi Joyce Yang, Can Cui, Ioan Andrei Bårsan, Raquel Urtasun, and Shenlong Wang. Asyn- chronous multi-view SLAM. CoRR, abs/2101 2/s/2101.06622 2021. URL /tlp:tttps///a/v..oor
abs/2101. 06562
[55] Ze Yang. Siva Manivasagam, Ming Liang, Bin Yang. Wei-Chiu Ma, and Raquel Urtasun. Recovering and simulating pedestrians in the wild. CoRR, abs/2011.08106 2020. URL
https:://axxis org/ /ttgg//b///0//0/////
[56] Zhenpei Yang, Yuning Chai, Dragomir Anguelov, Yin Zhou, Pei Sun, Dumitru Erhan, Sean Rafferty, and Henrik Kretzschmar. Surfelgan: Synthesizing realistic sensor data for autonomous driving. CoRR, abs/2005.032 003844, 2020. URL
[57] Yuchen Zhang, Tianle Liu, Mingsheng Long, and Michael Jordan. Bridging theory and algorithm for domain adaptation. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 7404-7413, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
[58] Yuxuan Zhang, Huan I. Ling, Jun Gao, Kangxue Yin, Jean-Francois Lafleche, Adela Barriuso, Antonio Torralba, and Sanja Fidler. Datasetgan: Efficient labeled data factory with minimal human effort. arXiv preprint arXiv:2104.066
2021.
[59] Han Zhao, Remi Tachet des Combes, Kun Zhang, and Geoffrey J Gordon. On learning invariant representation for domain adaptation. XrXiv preprint arXiv: 1901.0945,, 2019.
14


Reviews:
[{"review_id": "f315L96wJ7A", "reviewer": "Reviewer_z4FW", "summary": "The authors propose a technique for sampling data from a self-driving car simulator. They sample in a simulator-agnostic way in an effort to reduce the distance between the label marginals and therefore reduce the sim2real gap.They compare their approach to standard domain adapation techniques. They compare their techniques to standard data adaption frameworks and baselines and show promising results. They also demonstrate that domain adversarial training outperform simple style transfer approaches. ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Originality:\nThe paper follows in a long line of adapting self driving car simulator data to the real world. The individual ideas in the paper are different from the related works mentioned since it relies on regenerating the datasets using parameters from the simulator. In this way, it's slightly different from DANN or MUNIT. The lack of related work that is actually compared to in the paper is my biggest concern. Note: not many papers focus on what parts of the simulator are actually relevant to transfer between datasets or the limitations of the simulator itself and this is is a very valuable and understudied problem.\n\n# Quality\nThe paper does appear to include  proper comparisons against the other techniques. The claims are well supported technically (with the exception of the technique being simulator agnostic. That particular point is not demonstrated by the paper.\n\nTruly showing that would require more simulators than just CARLA, but that is a minor issue.  The baselines compared against seem adequate. The weakest portion of the paper is the limited number of baselines. They only compare to DANN or MUNIT despite those baselines being rather old. More modern state of the art domain adaption and style transfer strategies. DANN was published 5 years ago in 2016.  Without recent and adequate baselines, the paper feels incomplete. The fact that only rather dated two baselines were chosen is rather concerning as reviewer as it seems possible that more recent work may outperform this approach. The shear lack of baselines here is my greatest concern for this paper.\n\n# Clarity\nThe clarity of the paper is adequate. Note that many of the figures are so small that they are hard to read the text on, but this  is easily fixable. (In fact a lot of the figures are small enough that they are bit hard to read the underlying text). The paper is well written.\n\n# Significance\nThis paper would prove helpful in the domain adaption self-driving cars. As self-driving car research requires accurate simulators that transfer well, this submission would prove useful in that space. More importantly though, this paper is very relevant to demonstrating which types of variation (ie. map design, and number of vehicles) actually matter to self-driving car simulator allowing the creators to focus on the types of variation that are most important to impactful real world performance. For instance, they state that variation in NPC color does not matter for performance so there is no need to build a randomization strategy for those NPC colors. On the other hand, they showing matching the NPC positions with a dataset like nuscene is rather important. \n\n#Update\nGiven the rebuttals from the authors, I am willing to raise my score to a 7. However, I would recommend the do not include their claims about the method being simulator agnostic as while possible, this claim has not be empirically proven in the paper through the use of multiple simulators.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "M12X8iOAXTd", "reviewer": "Reviewer_gzio", "summary": "This work targets the sim2real domain adaptation problem and approaches the problem from two angles - data collection and transfer method. The authors discuss the importance of how data is generated from the source domain and propose ways to effectively design scenarios (placement of other vehicles) that lead to better transfer. This work also introduces a novel method for domain adaptation that utilizes adversarial-training and learning with pseudo-labels.\nThey measure transfer performance across domains (CARLA to NuScenes) on the bird’s-eye-view vehicle segmentation from multi-view images task.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This work addresses important questions in domain adaptation for sim2real transfer with limited amounts of labeled data in the target domain.\n\nThis paper provides valuable discussion and experiments on how data collection in the source domain should be performed i.e. how does diversity of weather and vehicle assets in the simulator affect downstream transfer? These are important design decisions when developing simulators and I found the experiments on data collection Fig. 7-12 to be well-designed and insightful.\n\n**Questions/Concerns**:\n* The authors use only BEV vehicle segmentation for evaluation. I would like to see some discussion on why they chose this over full perception task presented in the Lift-Splat-Shoot (vehicle segmentation, driveable area, lane boundary …).\n* I find the ablations on data sampling quite thorough (weather/car color diversity/...) but the current set of experiments provide no insight in how transfer varies across simulator photorealism, a major factor to consider in simulation.\n* The proposed training objectives are described in Eq 4, 5, 6 and Figure 14 in the supplementary material. I highly recommend moving Figure 14 to the main paper, as well as including the training objective in the figure.\n\n**Typo/Prose/Figures**:\n* Line 245: “deficiencies”\n* Line 277: “obtained”\n* Line 331: “its”\n* Figure 13: images are quite difficult to see\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "F1HCfmybI3w", "reviewer": "Reviewer_LRhN", "summary": "The paper proposes a method for both generating data in simulation and training with the generated data for the bird's-eye-view (BEV) vehicle segmentation task. It takes inspiration from recent theoretical advances in domain adaptation (DA) and frames learning from simulation as DA. It also points out special properties of using simulated data for adaptation, i.e., we can control the simulated data (label) distribution. Inspired by risk upperbound derived in prior work, it proposes a synthetic data generation strategy that samples data based on human-designed spatial priors and priors of the target domains (if some target labels are given). Once the data is generated, it employs a training method that extends recent work on adversarial DA to the dense prediction task of BEV vehicle segmentation and aims to learn invariant representation across domains. For experiments, the paper synthesizes data using the CARLA simulator and evaluates the adapted model on the nuScenes dataset. Results show that the method outperforms other data synthesis strategies and training methods such as style transfer-based DA.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Strength:**\n\n- The idea of optimizing synthetic data distribution to match that of target domain distribution is sensible and appealing. The paper follows a rigorous DA framework to analyze the need for optimizing data sampling distribution in order to minimize the target domain risk.\n- Although the theoretical results and DA approach used in the paper are not new, the paper uses them effectively to solve practical real-world tasks such as BEV vehicle segmentation, and shows that the proposed framework outperforms prior data synthesis and DA methods typically used in these tasks. The method also shows good robustness against different types of domain gaps such as maps, vehicle assets, and weather. So this seems like a good empirical contribution.\n- The paper is generally well-written. Most techniques are well-motivated from a theoretical standpoint.\n\n**Weakness:**\n\n- When no target label is provided (direct transfer learning), the spatial priors used in Sec 3.1 are rather heuristic, which makes the method less principled or general. Using these human priors is what people typically do when generating synthetic data, so the practice itself is not new, although the paper does formalize it inside a DA framework.\n- The paper claims to be \"agnostic of the network architecture and the choice of a simulator\", however, the paper has only used one simulator, one architecture, and one hand-designed prior. Thus, it is difficult to verify this claim. It would be more interesting if the authors can try other architecture and experiment with other autonomous driving tasks and use other priors.\n- In the semi-supervised learning setting, the performance of the proposed method over Lift Splat seems small. So it seems like the method is most effective for direct transfer learning. One could argue that the performance gain is mainly coming from the strong hand-designed spatial priors.\n\n**Additional Comments:**\n\ntypo:\n\n- line 245: \"defficiencies\" → \"deficiencies\"\n- line 331: \"it's\" → \"its\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The paper adequately discussed its limitations and potential negative societal impact. The limitations listed in the papers are mainly typical failure modes of the proposed method, such as mistaking pedestrians for vehicles.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "8CCZvs_NKjG", "reviewer": "Reviewer_TDjR", "summary": "This paper tackles the problem of sim2real domain adaptation for the bird's-eye-view vehicle segmentation task. The authors trained existing models (Lift Splat and Point Pillar) with carefully sampled synthetic data,  pseudo-labels on real data (eq. 6), and gradient-reversal layers with f-DAL inspired loss. On the nuScenes dataset, the authors found that the these tricks improve models' (Lift Splat and Point Pillar) sim2real generalization accuracy of the vehicle segmentation IOU (Table 1 and Table 2).", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The authors study a few recent sim2real techniques (e.g., pseudo-labels on real data, gradient-reversal layers with f-DAL inspired loss).\n- The authors carefully sampled synthetic data from simulators to match the real labels marginals of the nuScenes datasets (section 3).\n- Using the above tricks, two models' (Lift Splat and Point Pillar) sim2real generalization accuracies improve on the vehicle segmentation IOU metrics (Table 1 and Table 2).\n\nWeaknesses:\n- The paper combines a few existing techniques to train baseline models on synthetic data. The techniques and baseline models are all from existing literatures, so the novelty is lacking.\n- The authors didn't include comparison with prior works on the nuScenes datasets. What are the current SOTA models' accuracies (trained on real data or sim-real mixed)? Are there other sim2real works that evaluate on nuScenes?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes (see: section 7)", "review_text": "", "extra_scores": {"ethical_concerns": "There is no ethical concerns", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper studies the problem of sim2real domain adaptation, focusing on the bird's-eye-view vehicle segmentation task and on strategies for sampling data from a self-driving car simulator. The authors trained existing models and sampled data in a simulator-agnostic way in an effort to reduce the sim2real gap, compared their approach to standard domain adaptation techniques and demonstrated that domain adversarial training outperformed simple style transfer approaches.

Reviewers have praised the clarity of the paper and thoroughness of the experiments, and noted that this research was “very relevant to demonstrating which types of variation (ie. map design, and number of vehicles) actually matter to self-driving car simulators”.
Reviewers’ claims about “novelty” are easy to make and to dismiss, and I note that the authors provide an in-depth mathematical formalism and analysis of the simulation-reality gap. 
Reviewers also had concerns about missing evaluation and comparison of SOTA models on nuScenes (TDjR and z4FW) - the authors have addressed these by running additional experiments comparing their method with “HorizonLiDAR3D" from the 3D detection track and the domain adaptation track in Waymo Open Dataset Challenge at CVPR 2020, training the models on CARLA; moreover reviewer z4FW praised the “focus on what parts of the simulator are actually relevant to transfer between datasets”. Reviewers gzio and LRhN had additional small concerns about photorealism, focus on bird eye view, heuristics and spatial priors, and the authors committed to answer them.

Reviewers TDjR and z4FW did not answer the authors’ rebuttal but I believe they should and would increase their score. I am therefore willing to challenge these two reviewers and promote this paper to an acceptance.

Decision: Accept (Poster)