Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation

1 Introduction . The dominant strategy in self-driving for training perception models is to deploy cars that collect massive amounts of real-world data , hire a large pool of annotators to label it and then train the models on that data using supervised learning . Although this approach is likely to succeed asymptotically , the financial cost scales with the amount of data being collected and labeled . Furthermore , changing sensors may require redoing the effort to a large extent . Some tasks such as labeling ambiguous far away or occluded objects may be hard or even impossible for humans . In comparison , sampling data from self-driving simulators such as [ 46 , 2 , 6 , 13 ] has several benefits . First , one has control over the content inside a simulator which makes all self-driving scenarios equally efficient to generate , indepedent of how rare the event might be in the real world . Second , full world state information is known in a simulator , allowing one to synthesize perfect labels for information that humans might annotate noisily , as well as labels for fully occluded objects that would be impossible to label outside of simulation . Finally , one has control over sensor extrinsics and intrinsics in a simulator , allowing one to collect data for a fixed scenario under any chosen sensor rig . Given these features of simulation , it would be tremendously valuable to be able to train deployable perception models on data obtained from a self-driving simulation . There are , however , two critical challenges in using a driving simulator out of the box . First , sensor models in simulation do not mimic real world sensors perfectly ; even with exact calibration of extrinsics and intrinsics , it is extremely ∗denotes equal contribution , https : //nv-tlabs.github.io/simulation-strategies 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hard to perfectly label all scene materials and model complicated interactions between sensors and objects – at least today [ 34 , 8 , 56 , 42 ] . Second , real world content is difficult to incorporate into simulation ; world layout may be different , object assets may not be diverse enough , and synthetic materials , textures and weather may all not align well with their real world counterparts [ 51 , 6 , 19 , 55 ] . To make matters worse , machine learning models and more specifically neural networks are very sensitive to these discrepancies , and are known to generalize poorly from virtual ( source domain ) to real world ( target domain ) [ 52 , 14 , 44 ] . Contributions and Outline . In this paper , we seek to find the best strategies for exploiting a driving simulator , that can alleviate some of the aforementioned problems . Specifically , we build on recent advances in domain adaptation theory , and from this perspective , propose a theoretically inspired method for synthetic data generation . We also propose a novel domain adaptation method that extends the recently proposed f -DAL [ 3 ] to dense prediction tasks and combines it with pseudo-labels . We start by formalizing our problem from a DA perspective . In Section 2 , we introduce the mathematical tools , generalization bounds and highlight main similarities and differences between standard DA and learning from a simulator . Our analysis leads to a simple but effective method for sampling data from the self-driving simulator introduced in Section 3 . This technique is simulator agnostic , simple to implement in practice and builds on the principle of reducing the distance between the labels ’ marginals , thereby reducing the domain-gap and allowing adversarial frameworks to learn meaningfully in a representation space . Section 4 generalizes recently proposed discrepancy minimization adversarial methods using Pearson χ2 [ 3 ] to dense prediction tasks . It also shows a novel way to combine domain adversarial methods with pseudo-labels . In Section 6 , we experimentally validate the efficacy of our method and show the same approach can be simply applied to different sensors and data modalities ( oftentimes implying different architectures ) . Although we focus on the scenario where labeled data is not available in the target domain , we show our framework also provides gains when labels in the target domain are available . Finally , we show that variations such as number of vehicle assets , map design and camera post-processing effects can be compensated for with our method , thus showing what variations are less important to include in the simulator . 2 Learning from a Simulator : A Domain Adaptation Perspective . We can learn a great deal about a model ’ s generalization capabilities under distribution shifts by analyzing its corresponding binary classifier . Therefore , building on the existing work in domainadaptation theory [ 5 , 4 , 59 , 3 ] , we assume the output domain be Y = { 0 , 1 } and restrict the mathematical analysis to the binary classification setting . Section 6 shows experiments in more general settings , validating the usefulness of this perspective and our algorithmic solution which is inspired by the theoretical analysis presented in this section . Specifically , here , we interpret learning from a simulator as a domain adaptation problem . We first formulate the problem and introduce the notation that will be used throughout the work . Then , we introduce mathematical tools and generalization bounds that this work builds upon . We also highlight main similarities and differences between standard DA and learning from a simulator . We start by assuming the self-driving simulator can automatically produce labels for the task in hand ( as is typical ) , and refer to data samples obtained from the simulator as the synthetic dataset ( S ) with labeled datapoints S = { ( xsi , ysi ) } nsi=1 . Let ’ s also assume we have access to another dataset ( T ) with unlabeled examples T = { ( xti ) } nti=1 that are collected in the real world . We emphasize that no labels are available for T. Our goal then is to learn a model ( hypothesis ) h for the task using data from the labeled dataset S such that h performs well in the real-world . Intuitively , one should incorporate the unlabeled dataset T during the learning process as it captures the real world data distribution . Certainly , there are different ways in which the dataset S can be generated ( e.g . domain randomization [ 52 , 38 ] ) , and also several algorithms that a practitioner can come up with in order to solve the task ( e.g . style transfer ) . In order to narrow the scope and propose a formal solution to the problem , we propose to interpret this problem from a DA perspective . The goal of this learning paradigm is to deal with the lack of labeled data in a target domain ( e.g real world ) by transferring knowledge from a labeled source domain ( e.g . virtual word ) . Therefore , clearly applicable to our problem setting . In order to properly formalize this view , we must add a few extra assumptions and notation . Specifically , we assume that both the source inputs xsi and target inputs x t i are sampled i.i.d . from distributions Ps and Pt respectively , both over X . We assume the output domain be Y = { 0 , 1 } and define the indicator of performance to be the risk of a hypothesis h : X → Y w.r.t . the labeling function f , using a loss function ` : Y × Y → R+ under distribution D , or formally R ` D ( h , f ) : = ED [ ` ( h ( x ) , f ( x ) ) ] . We let the labeling function be the optimal Bayes classifier f ( x ) = argmaxŷ∈Y P ( y = ŷ|x ) [ 35 ] , where P ( y|x ) denotes the class conditional distribution for either the source-domain ( simulator ) ( Ps ( y|x ) ) or the target domain ( real-world ) ( Pt ( y|x ) ) . For simplicity , we defineR ` S ( h ) : = R ` Ps ( h , fs ) and R ` T ( h ) : = R ` Pt ( h , ft ) . For reasons that will become obvious later ( Sec . 4 ) , we will interpret the hypothesis h ( model ) as the composition of h = ĥ ◦ g with g : X → Z , and ĥ : Z → Y , where Z is a representation space . Thus , we define the hypothesis classH : = { ĥ ◦ g : ĥ ∈ Ĥ , g ∈ G } such that h ∈ H. With this in hand , we can use the generalization bound from [ 3 ] to measure the generalization performance of a model h : Theorem 1 ( Acuna et al . [ 3 ] ) . Suppose ` : Y × Y → [ 0 , 1 ] and denote λ∗ : = minh∈HR ` S ( h ) + R ` T ( h ) . We have : R ` T ( h ) ≤ R ` S ( h ) + Dφh , H ( Ps||Pt ) + λ∗ , where ( 1 ) where : Dφh , H ( Ps||Pt ) : = suph′∈H |Ex∼Ps [ ` ( h ( x ) , h′ ( x ) ) ] −Ex∼Pt [ φ∗ ( ` ( h ( x ) , h′ ( x ) ) ) | , and the function φ : R+ → R defines a particular f -divergence with φ∗ being its ( Fenchel ) conjugate . Theorem 1 is important for our analysis as it shows us what we have to take into account such that a model can generalize from the virtual to the real-world . Intuitively , the first term in the bound accounts for the performance of the model on simulation , and shows us , first and foremost , that we must perform well on S ( this is intuitive ) . The second term corresponds to the discrepancy between the marginal distributions Ps ( x ) and Pt ( x ) , in simpler words , how dissimilar the virtual and real world are from an observer ‘ s view ( this is also intuitive ) . The third term measures the ideal joint hypothesis ( λ∗ ) which incorporates the notion of adaptability and can be tracked back to the dissimilarity between the labeling functions [ 3 , 59 , 4 ] . In short , when the optimal hypothesis performs poorly in either domain , we can not expect successful adaptation . We remark that this condition is necessary for adaptation [ 5 , 4 , 3 , 59 ] . Comparison vs standard DA . In standard DA , two main lines of work dominate . These mainly depend on what assumptions are placed on λ∗ . The first group ( 1 ) assumes that the last term in equation 1 is negligible thus learning is performed in an adversarial fashion by minimizing the risk in source domain and the domain discrepancy in Z [ 16 , 3 , 33 , 57 ] . The second group ( 2 ) introduces reweighting schemes to account for the dissimilarity between the label marginals Ps ( y ) and Pt ( y ) . This can be either implicit [ 24 ] or explicit [ 32 , 50 , 48 ] . In our scenario the synthetic dataset is sampled from the self-driving simulator . Therefore , we have control over how the data is generated , what classes appear , the frequency of appearance , what variations could be introduced and where the assets are placed . Thus , Theorem 1 additionally tell us that the data generation process must be done in such a way that λ∗ is negligible , and if so , we could focus only on learning invariant-representations . We exploit this idea in Section 3 for the data generation procedure , and in Section 4 for the training algorithm . We emphasize the importance of controlling the dissimilarity between the label marginals Ps ( y ) and Pt ( y ) , as it determines the training strategy that we could use . This is illustrated by the following lower bound from [ 59 ] : Theorem 2 ( Zhao et al . [ 59 ] ) Suppose that DJS ( Ps ( y ) ‖Pt ( y ) ) ≥ DJS ( Ps ( z ) ‖Pt ( z ) ) We have : R ` T ( h ) +R ` S ( h ) ≥ 12 ( √ DJS ( Ps ( y ) ‖Pt ( y ) ) − √ DJS ( Ps ( z ) ‖Pt ( z ) ) ) 2 where DJS corresponds to the Jensen-Shanon divergence . Theorem 2 is important since it is placing a lower bound on the joint risk , and it is particularly applicable to algorithms that aim to learn domain-invariant representations . In our case , it illustrates the paramount importance of the data generation procedure in the simulator . If we deliberately sample data from the simulator and position objects in a way that creates a mismatch between real and virtual world , simply minimizing the risk in the source domain and the discrepancy between source and target domain in the representation space may not help . Simply put , failing to align the marginals may prevent us from using recent SoTA adversarial learning algorithms .

Summary: This work targets the sim2real domain adaptation problem and approaches the problem from two angles - data collection and transfer method. The authors discuss the importance of how data is generated from the source domain and propose ways to effectively design scenarios (placement of other vehicles) that lead to better transfer. This work also introduces a novel method for domain adaptation that utilizes adversarial-training and learning with pseudo-labels. They measure transfer performance across domains (CARLA to NuScenes) on the bird’s-eye-view vehicle segmentation from multi-view images task.