Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation

1 Introduction . The dominant strategy in self-driving for training perception models is to deploy cars that collect massive amounts of real-world data , hire a large pool of annotators to label it and then train the models on that data using supervised learning . Although this approach is likely to succeed asymptotically , the financial cost scales with the amount of data being collected and labeled . Furthermore , changing sensors may require redoing the effort to a large extent . Some tasks such as labeling ambiguous far away or occluded objects may be hard or even impossible for humans . In comparison , sampling data from self-driving simulators such as [ 46 , 2 , 6 , 13 ] has several benefits . First , one has control over the content inside a simulator which makes all self-driving scenarios equally efficient to generate , indepedent of how rare the event might be in the real world . Second , full world state information is known in a simulator , allowing one to synthesize perfect labels for information that humans might annotate noisily , as well as labels for fully occluded objects that would be impossible to label outside of simulation . Finally , one has control over sensor extrinsics and intrinsics in a simulator , allowing one to collect data for a fixed scenario under any chosen sensor rig . Given these features of simulation , it would be tremendously valuable to be able to train deployable perception models on data obtained from a self-driving simulation . There are , however , two critical challenges in using a driving simulator out of the box . First , sensor models in simulation do not mimic real world sensors perfectly ; even with exact calibration of extrinsics and intrinsics , it is extremely ∗denotes equal contribution , https : //nv-tlabs.github.io/simulation-strategies 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . hard to perfectly label all scene materials and model complicated interactions between sensors and objects – at least today [ 34 , 8 , 56 , 42 ] . Second , real world content is difficult to incorporate into simulation ; world layout may be different , object assets may not be diverse enough , and synthetic materials , textures and weather may all not align well with their real world counterparts [ 51 , 6 , 19 , 55 ] . To make matters worse , machine learning models and more specifically neural networks are very sensitive to these discrepancies , and are known to generalize poorly from virtual ( source domain ) to real world ( target domain ) [ 52 , 14 , 44 ] . Contributions and Outline . In this paper , we seek to find the best strategies for exploiting a driving simulator , that can alleviate some of the aforementioned problems . Specifically , we build on recent advances in domain adaptation theory , and from this perspective , propose a theoretically inspired method for synthetic data generation . We also propose a novel domain adaptation method that extends the recently proposed f -DAL [ 3 ] to dense prediction tasks and combines it with pseudo-labels . We start by formalizing our problem from a DA perspective . In Section 2 , we introduce the mathematical tools , generalization bounds and highlight main similarities and differences between standard DA and learning from a simulator . Our analysis leads to a simple but effective method for sampling data from the self-driving simulator introduced in Section 3 . This technique is simulator agnostic , simple to implement in practice and builds on the principle of reducing the distance between the labels ’ marginals , thereby reducing the domain-gap and allowing adversarial frameworks to learn meaningfully in a representation space . Section 4 generalizes recently proposed discrepancy minimization adversarial methods using Pearson χ2 [ 3 ] to dense prediction tasks . It also shows a novel way to combine domain adversarial methods with pseudo-labels . In Section 6 , we experimentally validate the efficacy of our method and show the same approach can be simply applied to different sensors and data modalities ( oftentimes implying different architectures ) . Although we focus on the scenario where labeled data is not available in the target domain , we show our framework also provides gains when labels in the target domain are available . Finally , we show that variations such as number of vehicle assets , map design and camera post-processing effects can be compensated for with our method , thus showing what variations are less important to include in the simulator . 2 Learning from a Simulator : A Domain Adaptation Perspective . We can learn a great deal about a model ’ s generalization capabilities under distribution shifts by analyzing its corresponding binary classifier . Therefore , building on the existing work in domainadaptation theory [ 5 , 4 , 59 , 3 ] , we assume the output domain be Y = { 0 , 1 } and restrict the mathematical analysis to the binary classification setting . Section 6 shows experiments in more general settings , validating the usefulness of this perspective and our algorithmic solution which is inspired by the theoretical analysis presented in this section . Specifically , here , we interpret learning from a simulator as a domain adaptation problem . We first formulate the problem and introduce the notation that will be used throughout the work . Then , we introduce mathematical tools and generalization bounds that this work builds upon . We also highlight main similarities and differences between standard DA and learning from a simulator . We start by assuming the self-driving simulator can automatically produce labels for the task in hand ( as is typical ) , and refer to data samples obtained from the simulator as the synthetic dataset ( S ) with labeled datapoints S = { ( xsi , ysi ) } nsi=1 . Let ’ s also assume we have access to another dataset ( T ) with unlabeled examples T = { ( xti ) } nti=1 that are collected in the real world . We emphasize that no labels are available for T. Our goal then is to learn a model ( hypothesis ) h for the task using data from the labeled dataset S such that h performs well in the real-world . Intuitively , one should incorporate the unlabeled dataset T during the learning process as it captures the real world data distribution . Certainly , there are different ways in which the dataset S can be generated ( e.g . domain randomization [ 52 , 38 ] ) , and also several algorithms that a practitioner can come up with in order to solve the task ( e.g . style transfer ) . In order to narrow the scope and propose a formal solution to the problem , we propose to interpret this problem from a DA perspective . The goal of this learning paradigm is to deal with the lack of labeled data in a target domain ( e.g real world ) by transferring knowledge from a labeled source domain ( e.g . virtual word ) . Therefore , clearly applicable to our problem setting . In order to properly formalize this view , we must add a few extra assumptions and notation . Specifically , we assume that both the source inputs xsi and target inputs x t i are sampled i.i.d . from distributions Ps and Pt respectively , both over X . We assume the output domain be Y = { 0 , 1 } and define the indicator of performance to be the risk of a hypothesis h : X → Y w.r.t . the labeling function f , using a loss function ` : Y × Y → R+ under distribution D , or formally R ` D ( h , f ) : = ED [ ` ( h ( x ) , f ( x ) ) ] . We let the labeling function be the optimal Bayes classifier f ( x ) = argmaxŷ∈Y P ( y = ŷ|x ) [ 35 ] , where P ( y|x ) denotes the class conditional distribution for either the source-domain ( simulator ) ( Ps ( y|x ) ) or the target domain ( real-world ) ( Pt ( y|x ) ) . For simplicity , we defineR ` S ( h ) : = R ` Ps ( h , fs ) and R ` T ( h ) : = R ` Pt ( h , ft ) . For reasons that will become obvious later ( Sec . 4 ) , we will interpret the hypothesis h ( model ) as the composition of h = ĥ ◦ g with g : X → Z , and ĥ : Z → Y , where Z is a representation space . Thus , we define the hypothesis classH : = { ĥ ◦ g : ĥ ∈ Ĥ , g ∈ G } such that h ∈ H. With this in hand , we can use the generalization bound from [ 3 ] to measure the generalization performance of a model h : Theorem 1 ( Acuna et al . [ 3 ] ) . Suppose ` : Y × Y → [ 0 , 1 ] and denote λ∗ : = minh∈HR ` S ( h ) + R ` T ( h ) . We have : R ` T ( h ) ≤ R ` S ( h ) + Dφh , H ( Ps||Pt ) + λ∗ , where ( 1 ) where : Dφh , H ( Ps||Pt ) : = suph′∈H |Ex∼Ps [ ` ( h ( x ) , h′ ( x ) ) ] −Ex∼Pt [ φ∗ ( ` ( h ( x ) , h′ ( x ) ) ) | , and the function φ : R+ → R defines a particular f -divergence with φ∗ being its ( Fenchel ) conjugate . Theorem 1 is important for our analysis as it shows us what we have to take into account such that a model can generalize from the virtual to the real-world . Intuitively , the first term in the bound accounts for the performance of the model on simulation , and shows us , first and foremost , that we must perform well on S ( this is intuitive ) . The second term corresponds to the discrepancy between the marginal distributions Ps ( x ) and Pt ( x ) , in simpler words , how dissimilar the virtual and real world are from an observer ‘ s view ( this is also intuitive ) . The third term measures the ideal joint hypothesis ( λ∗ ) which incorporates the notion of adaptability and can be tracked back to the dissimilarity between the labeling functions [ 3 , 59 , 4 ] . In short , when the optimal hypothesis performs poorly in either domain , we can not expect successful adaptation . We remark that this condition is necessary for adaptation [ 5 , 4 , 3 , 59 ] . Comparison vs standard DA . In standard DA , two main lines of work dominate . These mainly depend on what assumptions are placed on λ∗ . The first group ( 1 ) assumes that the last term in equation 1 is negligible thus learning is performed in an adversarial fashion by minimizing the risk in source domain and the domain discrepancy in Z [ 16 , 3 , 33 , 57 ] . The second group ( 2 ) introduces reweighting schemes to account for the dissimilarity between the label marginals Ps ( y ) and Pt ( y ) . This can be either implicit [ 24 ] or explicit [ 32 , 50 , 48 ] . In our scenario the synthetic dataset is sampled from the self-driving simulator . Therefore , we have control over how the data is generated , what classes appear , the frequency of appearance , what variations could be introduced and where the assets are placed . Thus , Theorem 1 additionally tell us that the data generation process must be done in such a way that λ∗ is negligible , and if so , we could focus only on learning invariant-representations . We exploit this idea in Section 3 for the data generation procedure , and in Section 4 for the training algorithm . We emphasize the importance of controlling the dissimilarity between the label marginals Ps ( y ) and Pt ( y ) , as it determines the training strategy that we could use . This is illustrated by the following lower bound from [ 59 ] : Theorem 2 ( Zhao et al . [ 59 ] ) Suppose that DJS ( Ps ( y ) ‖Pt ( y ) ) ≥ DJS ( Ps ( z ) ‖Pt ( z ) ) We have : R ` T ( h ) +R ` S ( h ) ≥ 12 ( √ DJS ( Ps ( y ) ‖Pt ( y ) ) − √ DJS ( Ps ( z ) ‖Pt ( z ) ) ) 2 where DJS corresponds to the Jensen-Shanon divergence . Theorem 2 is important since it is placing a lower bound on the joint risk , and it is particularly applicable to algorithms that aim to learn domain-invariant representations . In our case , it illustrates the paramount importance of the data generation procedure in the simulator . If we deliberately sample data from the simulator and position objects in a way that creates a mismatch between real and virtual world , simply minimizing the risk in the source domain and the discrepancy between source and target domain in the representation space may not help . Simply put , failing to align the marginals may prevent us from using recent SoTA adversarial learning algorithms .

Summary: This paper tackles the problem of sim2real domain adaptation for the bird's-eye-view vehicle segmentation task. The authors trained existing models (Lift Splat and Point Pillar) with carefully sampled synthetic data,  pseudo-labels on real data (eq. 6), and gradient-reversal layers with f-DAL inspired loss. On the nuScenes dataset, the authors found that the these tricks improve models' (Lift Splat and Point Pillar) sim2real generalization accuracy of the vehicle segmentation IOU (Table 1 and Table 2).