CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation
Authors: Jingkang Wang
Sivabalan Manivasagam
Yun Chen
Ze Yang
Ioan Andrei Bârsan
Anqi Joyce Yang
Wei-Chiu Ma
Raquel Urtasun
Venue: CoRL 2022 Poster (2022)
Keywords: 3D Reconstruction
CAD models
Sensor Simulation
Self-Driving

Abstract: Realistic simulation is key to enabling safe and scalable development of self-driving vehicles. A core component is simulating the sensors so that the entire autonomy system can be tested in simulation. Sensor simulation involves modeling traffic participants, such as vehicles, with high-quality appearance and articulated geometry, and rendering them in real-time. The self-driving industry has employed artists to build these assets. However, this is expensive, slow, and may not reflect reality. Instead, reconstructing assets automatically from sensor data collected in the wild would provide a better path to generating a diverse and large set that provides good real-world coverage. However, current reconstruction approaches struggle on in-the-wild sensor data, due to its sparsity and noise. To tackle these issues, we present CADSim which combines part-aware object-class priors via a small set of CAD models with differentiable rendering to automatically reconstruct vehicle geometry, including articulated wheels, with high-quality appearance. Our experiments show our approach recovers more accurate shape from sparse data compared to existing approaches. Importantly, it also trains and renders efficiently. We demonstrate our reconstructed vehicles in a wide range of applications, including accurate testing of autonomy perception systems.

CADSim: Robust and Scalable in-the-wild 3D Reconstruction for Controllable Sensor Simulation
Jingkang Wang 1.2 Sivabalan Manivasagam' 1.2
Yun Chen 1.2
Ze Yang" 1.2
Ioan Andrei Bårsan 12
Anqi Joyce Yang!. Wei-Chiu Ma 1.3 Raquel Urtasun 1.2
Massachusetts Institute of Technology" (wangjk, mani www.maninaasammmmmm vasagam, zeyang, yun, yun,aa, ajyaan,ang urtasun (css toronto. edu weichiumemit.ccd edu
Waabi
University of Toronto²
Abstract: Realistic simulation is key to enabling safe and scalable development of self-driving vehicles. A core component is simulating the sensors so that the entire autonomy system can be tested simulation. Sensor simulation involves modeling traffic participants, such as vehicles, with high quality appearance and articulated geometry, and rendering them in real time. The self-driving industry has typically employed artists to build these assets. However, this is expensive, slow, and may not reflect reality. Instead, reconstructing assets automatically from sensor data collected in the wild would provide a better path to generating a diverse and large set with good real-world coverage. Nevertheless, current reconstruction approaches struggle on in-the-wild sensor data, due to its sparsity and noise. To tackle these issues, we present CADSim, which combines part-aware object-class priors via small set of CAD models with differentiable rendering automatically reconstruct vehicle geometry, including articulated wheels, with high- quality appearance. Our experiments show our method recovers more accurate shapes from sparse data compared to existing approaches. Importantly, it also trains and renders efficiently. We demonstrate our reconstructed vehicles in several applications, including accurate testing of autonomy perception systems.
Keywords: 3D Reconstruction, CAD models, Sensor Simulation, Self-Driving
Generic GAD models
LDAR Points
Camera Images
Controliable Simulation
CADSim (Fast reconstruction)
Novel View Synthesis
Mixed Realiy (Atee Insorion)
Realdime rendbring
360" Textured Assets
Vehicl RevArimation
Tecluna Transfer
Figure 1: CADSim recovers shape, appearance and illumination in robust and scalable way from sensor observations. The reconstructed assets are of high fidelity, part-aware, geometry- aligned and compatible with graphics engine, enabling efficient, realistic and controllable simulation.
Introduction
Robots, such as self-driving vehicles (SDVs), learn navigate environments and interact safely with other agents through experience. It is critical for robots to handle rare or safety safety-critical critical situations which are challenging or dangerous observe the real world [1]. Simulation provides solution for the robot to efficiently experience edge-case scenarios for evaluation in a safe and cost-effective manner. For proper testing that covers the full space of possible scenarios, the simulator should have diverse and large set of traffic participants. For example, vehicles have a wide variety of shapes, sizes, and appearances. We want to ensure that the perception and autonomy systems can detect them and act appropriately in all possible situations. Existing self self-driving simulators have a limited set of objects, as each 3D asset is typically designed manually, with fully specified shape, appearance, etc. This is time-consuming and expensive process that does not scale. In this work, we propose to leverage real sensor data collected by the SDV in the wild to automatically build an asset library
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

for realistic sensor simulation. In-the-wild data is cost-efficient and scalable to obtain, enabling the collection of a large and diverse set of objects.
Specifically, given a sequence of sensor data (camera images and LiDAR point clouds) collected by an SDV, our goal is to automatically create digital replicas for all nearby vehicles in the scene and directly use them in high-fidelity simulator to create many scenarios that never existed before. To achieve high-quality and scalable sensor simulation, the reconstructed assets should (i) have precise shape and photorealistic appearance; (ii) be easily editable to create new variations from the original assets, (iii) be controllable, such as the ability to manipulate the vehicle to generate new behaviors (e.g., wheels steering) and (iv) allow for real-time rendering, which is important for closed-loop training and large-scale evaluation [2]. Out of past works using various 3D representations for reconstruction, such as point clouds [3, 4, 5, 6], voxels [7, 8, 9], or neural implicit representations [10, 11, 12, 13), the mesh representation best achieves the desired properties (14, 15]. In this paper we utilize mesh representations since they are widely used throughout simulation [16, 17, 18) and content creation systems [19, 20), are easy to manipulate [21, 22], suitable for rigging and animation [23, 24, 25, 26], support texturing and material properties [27, 28), and are fast to render [29].
Generating meshes of nearby objects from in-the-wild sensor data is a challenging task. The sensor data is sparse (< 30 views) and may be noisy. Furthermore, the objects of interest are observed from limited viewpoints and portions of the objects are unobserved, whereas we want to reconstruct the complete shape for simulation. Previous mesh reconstruction methods tackle this task as an energy minimization problem, where given an initial sphere [30] or amean-shape mesh template [31, 32], the goal is to optimize the mesh to be consistent with sensor data, subject to some regularization. However, when operating on in-the-wild sensor data, these methods generate overly smoothed meshes that are inaccurate [30, 32), or have severe artifacts and self-intersecting edges [33]. We demonstrate in our experiments that this is due to poor initialization. Moreover, these approaches generate rigid meshes that cannot be controlled and lack articulation. We therefore propose a simple and effective solution-to leverage class-specific CAD models to provide priors for reconstruction. Not only do CAD models have high quality shape, they encode rich semantic information, such as where the wheels in a vehicle are and how they move. This enables better initialization and reconstruction of high-quality articulated meshes from real-world data.
CAD models alone are not sufficient for realistic sensor simulation, as we we want to create digital twins of any possible vehicle. Instead, we optimize our vehicle CAD model representation with a state-of-the -rrt differentiable renderer in an energy minimization framework. This allows us to faithfully reconstruct diverse set of real-world vehicles, including their moving parts (e.g., wheels), from very sparse and noisy observations. We compare against a wide range of reconstruction methods, including recent neural rendering- basedd approaches, and show improved reconstruction performance on both camera and LiDAR sensor data. We show that our reconstructed vehicles can be used for variety of applications such as novel view synthesis, texture transfer, mixed reality via injecting simulated actors into real data, and vehicle re-animation (Fig. 1). We also demonstrate that our approach creates more realistic simulation environment for robots such SDVs by evaluating perception models on simulated sensor data and showing small domain gap compared to real data.
2 Related Work
3D Representations for Sensor Simulation. Triangular meshes are currently the de facto standard in real-time graphics [14, 15, 16]. While fast, flexible, compact, and widely adopted in game engines and simulators [16], mesh-based assets are usually regarded as expensive to create due to the need for human artists. Surfels [34] have also been leveraged in LiDAR simulation for robotic applications [35, 36]. However, their lack of geometric structure makes them difficult to animate, edit, and compress. Recently. volumetric representations have experienced a surge in popularity [37, 38, 39] thanks to their simplicity, differentiability, and flexibility in modeling non- Lambertian effects [40, 41, 42]. These works can roughly be divided into neural volumetric rendering, such as NeRFs [40, 43, 44, 45, 46], and implicit surfaces such as signed distance fields [47, 48, 49] or occupancy fields [10, 50, 51]. Various methods have been proposed to speed up surface and volumetric rendering [52, 53, 54, 55,56), However, these approaches still lag behind traditional rasterization pipeline in terms of rendering speed and throughput. In our work, we use triangular meshes optimized from CAD models for efficient and accurate geometry modelling.
Multi- View 3D Reconstruction in the Wild. Despite the incredible success achieved by recent neural volumetric representations, these works are mainly demonstrated synthetic datasets with
2

---

Data Energy Eduts
Shope Energy Fshape
Bacosa
Chamfer
Mosh Inttalzzatin
Easter
Toxture Energy Elype
Diferentiable Renderer
Ensok
Z
PBR Manariii
Lighting Veep
Figure 2: Overview of CADSim for 3D Reconstruction on in- in-the-wild data.
hundreds of (noise-free) images and dense coverage on the camera viewpoints [40, 57, 30). However, the real world input is usually much sparser (e.g., few views of single object with similar view points) and noisier (e.g.. localization, calibration error and LiDAR noise) ) for real world applications such robotics and self-driving. To address this issue, recent works [58, 59, 60] proposed conditional models that require expensive pre-training on other scenes with multi-view images and camera pose annotations. Another recent line of work proposes learning mesh representations directly, either by deforming a generic mesh [61, 27, 62, 33, 63, 64], or by leveraging differentiable mesh extraction algorithm top of implicit representation [65, 66, 67]. Different from past works, show that simple mesh optimization with differentiable rendering is sufficient for high-fidelity reconstruction.
CAD Priors. Retrieving a 3D model conditioned on single observation has been shown to be strong baseline which outperforms more sophisticated early approaches to single-vien view reconstruc- tion [68]. This motivates the use of imperfect but readily available 3D CAD models as powerful prior for reconstruction [69, 31, 70, 71, 72, 73, 74, 75, 76]. [69, 31, 71] use the truncated SDF volumes to build the latent space for the CAD models for joint pose and shape estimation. Uy et al. (74, 75] encode sparse observations and use them to retrieve the most similar candidate from a CAD asset bank, which is then deformed to match the input. While promising, none of the approaches in this area have been demonstrated to work outside of synthetic [77] datasets. Most do not model realistic texture, nor they part-aware. Lu et al. [76] conduct two-stage CAD-based deformation and handle the wheels separately. However, they recover the appearance only from single images without physics based materials and lighting models, and do not optimize the articulation from data.
3 Method
Given camera images and/or LiDAR point clouds collected by self-driving car, our goal is to automatically create digital replicas for all nearby vehicles in the scene and use them in high-fidelity simulator. We build our model based on the observation that existing CAD models are equipped with detailed geometry and animatable parts, which can be used as prior during the reconstruction process. Towards this goal, we propose energy-based formulation that exploits CAD models, as well as visual and geometric cues from images and/or LiDAR for 3D reconstruction (Fig. 2). We first describe model representation and how it leverages CAD priors (Sec. 3.1). Then present energy-based model (Sec. 3.2), and how we conduct inference to generate the vehicle mesh (Sec. 3.3).
3.1 Model Representation
To faithfully simulate an object, we need to recover not only its geometry. but also its material. Towards this goal, exploit textured meshes underlying representation since they best meet the desired properties of modern high-fideling simulators, by being efficient to animate, edit, and render.
Mesh representation. A mesh M (V, F) is composed of of vertices V E R Vix3 and faces F FIF1 where the faces define the connectivity of the vertices. Our goal is deform mesh to match the observations from the sensor data. Typically, during deformation, the topology (i.e., connectivity) of the mesh is fixed and only the vertices are "moving" [23, 32, 78, 21]. This strategy greatly simplifies the deformation process, yet at the same time constrains its representation power. For instance, if the original mesh topology relatively simple (e.g., sphere [78] "mean-shape" mesh template [32]) and non-homeomorphic to the object of interest, the mesh may struggle to capture the fine-grained geometry (e.g.. side mirrors of car).
CAD models as priors. To circumvent such limitations, we propose to incorporate shape priors from CAD models into the mesh reconstruction. One straightforward approach is to directly exploit
3

---

the CAD model to initialize the mesh MCAD = (VCAD, FCAD). Since CAD models, by design, respect the topology of real-world objects, the mesh will be able model finer details. One obvious caveat, however, is that there is no structure among the vertices. Each vertex may move freely in the 3D space during the optimization; e.g., there is no guarantee that vertices belonging to a wheel will continue to form a cylinder. We thus further incorporate the part information from CAD models into the parameterization. This allows us to inherently preserve part geometry, as well as enable part manipulation (e.g., changing the steering of the wheels).
We leverage the semantic part infor- mation from CAD models to parti- tion the vehicle mesh MCAD into a vehicle body [V F bodyy and wheels { (v(6) F(k)
wheel> wheel 1=1- indi- cates the number of wheels vehicle has. K = 4 for most vehicles but may be 6 or more for large trucks. We employ the fact that the wheels are usually the same within the same vehicle, and model all wheels with the same underlying mesh (V F whecl) and their individual relative pose T to the vehicle origin. As there are a wide variety of vehicles with different wheel sizes and different relative positions to the vehicle body, we further add a scale factor r = [r.., Th Twol (wheel radius and thickness), and per-axle translation offsets Erront, back E R$ with respect the wheel origin. We can thus drastically reduce the degrees of freedom, enforce the shape of the wheel, and guarantee the relative positions of the wheels to be symmetric. Importantly, the front-axle wheels body. We hence parameterize the front wheels to have yaw-relative orientation p. The full vehicle mesh model (see Figure 3) can thus be written as:
Figure 3: Vehicle model used in CADSim. For simplicity, we only show the offsets along the axle axis.
be steered and do not necessarily align with the
V wheel (r, biront, P3 V Whheel) = Th (R wheel + (frons)
k E (1,2), (front wheels) 1 {3, 43., ..., K);
V wheel (r, bhack; V Whheel) (r wheel + thack)
(remaining wheels) (I)
V = (Voody, v-(k) wheal
}
k E (1....,K])
with V body., r, tbock, trront: being the free variables.
Extending to CAD library. While a CAD model provides a useful prior for mesh initialization, it likely that single template mesh may not sufficiently cover wide range of objects. We thus resort to CAD model library that consists of various vehicle types. Inspired by [31, 76]. we represent a large variety of vehicles with compact, low dimensional code, which allows us to quickly determine the best CAD model to initialize from. We first parameterize a collection of the CAD models with Eq. (1), simplify them so that they have the same number of vertices and topology, and align them in shared coordinate space with dense correspondences. Finally, we apply principal component analysis on vertex coordinates to obtain shared oow-dimensional code z. Note that since all CAD models are parameterized with Eq. (1), the deformed mesh will reflect this part structure as well. Please refer to the supp. matcrial for more details.
Appearance representation. In addition to geometry, our mesh representation must accurately capture how light interacts with its surface that we can realistically reproduce sensor observations. Towards this goal, we parameterize the mesh appearance using a physically-base representation. Specifically, we represent appearance using a micro-facet BRDF model with the differentiable split sum environment lighting [79] proposed by Munkberg et al. [67]. Since the topology of the mesh is fixed, we build one-to-one mapping relationship between each point on the mesh surface and each point in the 2D (u, v) space. We use a physically- (PBR) material model from Disney [80], which contains diffuse lobe k,, with an isotropic, specular GGX lobe [81].
appearance
3.2 Energy Formulation
The next step after the CAD model initialization is to optimize both the geometry and the appearance of the mesh such that it best matches the captured sensor data. Our approach builds upon recent success differentiable rendering [82, 83, 84, 85], leveraging a differentiable renderer that takes as input variables such as the sensor pose and our textured mesh representation, and outputs a realistic simulation of the object. Differentiable rendering allows all variables to be optimized based on the supervision provided by the input images.
4

---

Notation. Let I = fiiss be the images captured at different timestamps and P be the aggregated LiDAR point clouds captured by data collection platform, in our case self-driving vehicle driving in the real world. Let [M;) J N be the foreground segmentation mask of (T: obtained from an off-the- algorithm [30, 86). Let A = {D, R. be the variables directly related to the appearance model (i.e., the material and the lighting). We denote IT {Kş -.am clidar] as the intrinsics and the extrinsics of the sensors, where & E se(3) are elements of the Lie algebra associated with SE(3). We assume all cameras are pre-calibrated with known intrinsics. Let zi (M,A,II) - (Ip, Mo) be the differentiable renderer where Ly and My denote the rendered RGB image and object mask.
Overall energy. We design energy function with complementary terms which measure the geom- etry and appearance agreement between the observations and estimations (Edata), while regularizing the shape (Eshape) and appearance (Eapp) to obey known priors:
Edata (rgminnEaattaMMM, II, A:I, Eahape + Fupp (M, II, AII,
M.IL.A
We now describe each energy term in more detail.
Data term. This energy encourages the estimated textured mesh to match the sensor data as much as possible. Specifically, it consists of three components:
P) Ecotor (M, II;
Emask II; Andar Endar (M, P)
(3)
Feclor, Fmask both enforce the rendering of the textured mesh to be close to the image observations. They, however, are complementary. Ecolor encourages the appearance of the rendered image to match the RGB observation and propagates the gradients to variables including appearance variables A, while mask measures the mask difference and only depends on the shape of the mesh as well as the camera poses. Following previous work [30, 67], we exploit smooth-é, distance to measure the difference in RGB space and squared E2 for object masks:
Ecolor
1 ΣE (I,, I),
Emask
1 N
Σ
(M, E.) M. (4)
N
with & the smooth-l, norm [87] and N the number of images available.
Eidar encourages the geometry of our mesh to match the aggregated LiDAR point clouds. Since minimizing point-to-surface distance is computationally expensive in practice, adopt the popular Chamfer Distance (CD) to measure the similarity instead. We randomly select L points (P.) from the current mesh and compute the asymmetric CD of Ps with respect the aggregated point cloud P:
Epdar (D(P,P.)
1
Σ
Or min | -112.
(5)
IP
*EP.
ZEP
where is an indicator function representing which LiDAR point is outlier. We refer the reader to Sec. 3.3 for a discussion on how to estimate a.
Shape term. This energy encourages the deformed mesh to be smooth and the faces of the mesh to be uniformly distributed among the surfaces (so that the appearance would be less likely distorted). Specifically, Eshape is an addition of two shape regularizations [85, 63, 30]: a normal consistency term Enormal (V) and average edge length term Bedge (V):
Enormal V) NE PEF PEA(()
Σ Σ ||n(f) n(f')l1 Eedge (V) NE VEV v'EN(v)
EE
Σ
Σ
1/v - v'1 (6)
Here, f F and E V refer to single face and single vertex. .N(F) and N(vv denote the neighboring faces of f and the neighboring vertices of v respectively. Nr and Ng are the number of neighboring face pairs and edges.
Appearance term. This energy exploits the following facts: (1) the appearance of vehicle will not change abruptly in most cases; instead, varies in a smooth fashion; (2) neutral, white lighting dominates in the real world. Following prior art in intrinsic decomposition [88, 89], we adopt sparsity term penalize frequent color changes on the diffuse k,, and specular k, terms. We also add regularizer to penalize the environment light in gray scale.
3
Papp
Amat
(1) Vkall + | |Vk,ll) + Atight Elee 6,ll1,
|c
1=1
where Vk, and Vk, are the image gradients approximated by Sobel-Feldman operator [90]. and C are the light intensity values at G.G.B channels and the per-channel average intensities.
5

---

3.3 Inference
Our goal is to find the optimal mesh M appearance representation AA", and sensor poses II" such that the total energy is minimized. We adopt the following strategy for better inference convergence. Initialization. A good initialization is required for non-convex optimization- achieve good performance. If the energy model is initialized from a CAD asset that is very different from the observations, the mesh may not be able to fit the sensor data well. To handle a wide variety of vehicles, we leverage the learned low-dimensional latent code 2 to estimate the initial coarse mesh Minit iitt F). where iiit reconstructed from the optimized latent code z and
methods to
= argmin Amask Emassi (M, II; I) + Atidar Fridar (M, IT; P) + Ashape Eshape! (V)-
(8)
We note that we focus on the geometry and only optimize the latent code. The latent code is initialized from O, and the sensor poses are obtained from coarse calibration and fixed. We optimize using stochastic gradient descent with the Adam optimizer [91]. Please see the supp. material for additional inference details. Since the low-dimensional latent code already smooths out instance-specific details, the optimization is much more robust and less sensitive to initialization.
Efficient and robust inference. Given the initialization Minit we jointly optimize the vertices V. appearance variables A, and sensor poses II as described in Eq. (2). To save computation, we uniformly sample L points on the current mesh each iteration to compute the LiDAR energy Euaar = CD(P, Ps), where Pa M and [P] L. To handle LiDAR outliers, we estimate the indicator function a by only calculating the Chamfer distance for the top p percentage of point pairs with the smallest distance. We find it is sufficient to handle point outliers in most cases. For the sensor extrinsics, we optimize the 6D rotation representation [92] and 3D translation. issue that optimizing vertex positions with adverse gradient descent steps can cause self- eelf-intersections that is irrecoverable or even exacerbated in further steps [63, 33]. Therefore, following Nicolet et al. [33], we use preconditioned gradient descent steps to bias towards smooth solutions at each iteration, enabling faster convergence to higher-quality meshes with more fine-grained details.
a
4 Experiments
4.1 Experimental Setup
Datasets. We evaluate our approach on two in-the-wild datasets: Multi-View Marketplace Cars (MVMC) [30] and CandaVehick [93]. MVMC consists of a diverse set of vehicles captured under various illumination conditions. Each vehicle comes with around 10 views and rough camera pose. Following [30], we evaluate our model as well as the baselines on the selected 20 vehicles. Vendale is a dataset we derived from the PandaSet dataset [93]. It consists of 200+ vehicles, along with their corresponding multi-view images and aggregated LiDAR point clouds. We manually inspect and select 10 high-quality vehicles for evaluation based on the accuracy of LiDAR-camera calibration. On average, each vehicle is captured from 24 camera views.
Baselines. We compare our approach with three types of 31) reconstruction methods: (1) Neural radiance NeRF++ [43], and Instant-NGP [46]. (2) Implicit surface representations: NeRS [30], NVDiffRee [67], and NeuS [48]. (3) Explicit geometry- based approaches: single-image view-warping (SI-View [94], which warps the source image to the novel target view using depth estimated by Sl--iewWarrp) Warp), which blends multiple source images to the target view. We also compare against SAMP [31], CAD model mesh optimization approach which leverages aDF-aligned CAD models for joint pose and shape optimization.
LiDAR points; and multi-image view-warping
MiIVieeWWo
Evaluation settings and metries. We evaluate reconstruction quality by performing camera and LiDAR simulation with the reconstructed vehicle. For camera simulation (i.e., novel view syn- thesis (NVS)), we adopt mean aqeane error (MSE), peak tignil-o-noose ratio (PSNR), SSIM [95], LPIPS [96] and cleaned FID [97]. To focus on the reconstructed vehicle itself, we only evaluate the foreground pixels using the predicted segmentation mask [86]. For LiDAR simulation, we place the reconstructed vehicle mesh in its original location and perform ray-casting to generate a simulated point cloud, We compare against held out real LiDAR points. We evaluate what fraction of real LiDAR points do not have corresponding simulated point (i.e., coverage), the average per-ray & error, Chamfer and Hausdorff distance. See supp. for additional details.
4.2 Reconstruction Quality
Novel View Synthesis on Panda Vehicle and MVMC. We report NVS results on Panda Vehicle in Tab. 1 and MVMC in Tab. 2, along with reconstruction time (T) and rendering speed (i.e.,

---

Method
MSB 1
PSNR
SSIM 1
LPIPS 4
7 (hour)
FPS
NeRF++ [Zhang et al., 2020]
0.0138
20.86
0.611
0.300
4.70
0.05
Instant-NGP [Mäller al., 2022]
0.0095
21.68
0.641
0.319
0.05
1.14
NeRS [Zhang et al., 2021]
0.0176
18.49
0.562
0.265
1.37
3.23
NVDiffRec [Munkherg et al., 2021]
1.0114
20.46
0.593
0.396
1.07
51,2*
NeuS [Wang et al., 2021]
0.0115
21.37
0.640
0.247
6.25
0.02
SI- VieV Warp [Tulsiani et al., 2018]
0.0233
17.51
0.514
0.371
1.67
SAMP [Engelmann et al., 2017]
0.0144
19.52
0.628
0.283
0.09
71.4*
CADSim (ours)
0.0087
21.72
0.674
0.220
0.13
49.6*
Table 1: Evaluation of nivel-view synthesis (left -> front-left camera) on PandaSet. We report the average GPU hour 7 for reconstruction per actor and rendering FPS on single RTX A5000. Underline denotes fast reconstruction (i.e.. < 10 minutes) and real-time rendering (FPS > 30). NeRF-based
Implicit Surface
Geomebyybazed
Input
NaRF++
Instani-NGP
NeRS
NVDiMRac
Neu8
Warp
9AMP
Ours
Figure 4: Qualitative results on Panda Vehicle for novel view synthesis. Compared to existing reconstruction approaches, CADSim produces more robust and realistic results on large extrapolation.
frame per second (FPS)). For PandaVehick we train each approach using images only from the left camera, and evaluate on front-left and front cameras. For MVMC we evaluate each approach via a -ross-validation approach, where we hold out one image each time. CADSim achieves better performance across all image quality metrics, especially LPIPS. LPIPS measures how neural nets perceive images. This suggests that our simulation results are perceived by the ML models to be more similar to real-world images (see Sec 4.3 for more analysis).
With CAD priors, we can reconstruct from much fewer views compared to NeRF-based or implicit-surface based-approaches. Compared to geometry-based methods, our differentiable renderer and material model enable much higher photorealism. Qualitative results on Panda Vehi- cle in Fig. further confirm this. NeRF-based approaches usually lead to visible artifacts with large view extrapolation and sparse input obser- vations. Implicit surface approaches perform ambiguity
Method
MSE 4
PSNR f
SSIM 1
I.PIPS 4. FIDS +
NeRS [30] 0.0254
16.5
0.720
0.172
60.9
CADSim
0.0188
17.7
0.751
0.147
48.1
Table 2:
Evaluation of NVS on MVMC [30].
Geometry
Lg error
Hit rate
Chamfer.]
Hausdorff]
NeRS [30]
0.171
94.3%
0.249
1.028
NVDifiRec [67]
0,320
98.2%
0.439
1.708
NeuS [48]
0.367
90.3%
0.424
1.151
SAMP [31]
0.158
94.8%
0.256
1.043
CADSim (ours)
0.151
96.3%
0.245
0.972
better by alleviating shape-radiane [43], but still have artifacts due to lower quality geometries. The other geometry-based approaches have better geometry, but have blurry appearance or missing pixels compared to CADSim. To further demonstrate the value of our improved geome- try, we compare the NVS results using our reconstructed mesh against a NeRS mesh with various appearance representations in the supp. material (Tab. A1). Additionally, neural rendering based approaches such as NeuS and NeRF++ are usually slow to train and render. While Instant-NGP [46] reduces training time significantly. it does not support real-time rendering. We report the rendering time for NVDiffRec, SAMP and CADSim (highlighted with *) using the differentiable renderer Nydiffrast [98]. Faster rendering (> 100 FPS) is expected with other modern graphics engines.
Table 3: Comparison of LiDAR rendering metrics.
LIDAR simulation on Vehicle. We now evaluate the geometry performance of the recon- structed meshes for LiDAR simulation. As shown in Tab. 3, our approach is better than or comparable prior across all metrics, suggesting that our meshes are more complete and more accurate than existing methods. NVDiffRec has the highest hit rate, yet performs badly on the rest of the metrics. This is because their generated meshes are noisy and cover many LiDAR points.
Ablation study. To showcase the importance of CAD initalization, we initalize the mesh with the following alternatives: a unit sphere, rescaled ellipsoid, and geometries from either SAMP [31] or NeRS [30]. As shown in Fig. 5, exploiting CAD priors leads to faster convergence and better performance. Our reconstruction is able capture fine- fine-grained details such as wheel size and rotation,
7

---

Unit Sphere
Ellipsoid
NeRS
SAMP
Ours
Figure 5: CAD priors are crucial for achieving figh-fidelity shape in non-convex optimization.
Left camera
Front-loy left camera
Left camera
Front camera
Blending
Copy-
Blending [17]
Copy-Faste
Det. (IoU)
Segm. (loU)
Det. (loli)
Segm. (loU))
Det. (lolJ)
Segm. (IoU)
Det. (IoU)
Segm. (lolI)
Instant-NGP
86.77
86.86
80.57
80.81
66.37
65.80
41.65
40.13
NewS
93.97
94,22
92.82
66.26
66.01
66
SAMP
90.39
89,58
90.04
89.73
82.82
79.78
83.32
81.79
CADSim (ours)
94.15
93.92
93.71
93.72
83,99
80.94
83,45
81.42
Table 4: Evaluation of downstream perception tasks (i.e., object detection, instance segmentation) on camera simulation, metric agreement with real data. Our assets lead to smaller domain gaps.
rear-view mirrors, etc. We also evaluate our reflectance model choice [81] in the supp. material. Our thysics-based material and lighting model performs the best and generalizes well novel views.
4.3 Realistic and Controllable Simulation
We now showcase applying CADSim for accurate camera simulation to evaluate perception models, and to perform realistic vehicle insertion. CADSim also supports texture transfer naturally, allowing us to easily expand the asset library. Please see additional details and results in the supp. material.
Downstream evaluation on camera simulation. To verify if CADSim helps reduce domain gap for downstream perception tasks, we evaluate object detection and instance segmentation algo- rithms [99, 86] on simulated camera images at novel views. In particular, we consider two setups: rendered and blended [17] and copy sapyeeddedd. We compute the instance-leva Intersection over Union (IoU) of the predicted bounding box and segmentation mask between simulated and real images. This retection/segmeetatiinn agreement metric (the larger the better) indicates how well the simulated data align with the real data with respect to these two perception tasks and model architectures. This metric can provide guidance using simulation as one tool for evaluation. As shown in Tab. 4, using CADSim assets usually leads to the largest agreement with real images under different settings.
Realistic animated vehicle insertion. Given the original scenario, sensor data, and an inserted actor trajectory, we produce consistent multi-sensov simulation with the rendered asset. Fig. 6, we generate safety-critical scenarios by inserting a new actor turning into lane (left half) and an actor slowing down in the adjacent lane, preventing the SDV from lane-changing (right half). The LiDAR simulation is conducted on reconstructed meshes enriched with intensity values (see supp. A2). The rendered asset is seamlessly blended into the original scenario and has physically realistic wheel rotation and movement. This enables diverse scenario generation for end-to-end autonomy testing.
Figure 6: Multi-sensor simulation on safety-eritical scenarios with animated vehicle insertion. The original camera and LiDAR observations captured in the real world are at the top-left corners.
5 Limitations and Conclusion
CADSim's 's main assumption is that requires CAD models for the object class of interest. We note that CAD models are readily available for most object classes, and that our approach only requires encoding semantic priors for a single CAD asset, as our energy-model optimization allows for transfer of these priors to other assets of the same class. See supp. for non-vehicle classes. Additionally, approach relies on either good segmentation masks or LiDAR points to reconstruct accurate shapes.
In this paper, we proposed to leverage in-the-wild camera and LiDAR data to reconstruct objects such as vehicles. Towards this goal, we designed CADSim, which leverages geometry and semantic cues from CAD models with differentiable rendering, to generate meshes with high quality geometry and appearance. These cues also enable our approach to generate articulated and editable meshes, enabling endless creation of new shapes, textures, and animations for simulation. We demonstrated that integrating our meshes into a camera simulation system can more effectively evaluate perception algorithms than existing reconstruction methods, reducing the simulation-tor to-rcal domain gap.
8

---

Acknowledgement
We sincerely thank the anonymous reviewers for their insightful suggestions. We would like thank Chris Zhang and Yuwen Xiong for their feedback on the early draft and final proofreading. We also thank the Waabi team for their valuable assistance and support.
References
[1] S. Tan, K. Wong, S. Wang. S. Manivasagam, M. Ren, and R. Urtasun. SceneGen: Learning to generate realistic traffic scenes. In CVPR, 2021.
[2] B. Shacklett, E. Wijmans, A. Petrenko, M. Savva, Batra, Koltun, and K. Fatahalian. Large batch simulation for deep reinforcement learning. In ICLR, 2021.
[3] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert. PCN: Point completion network. In 3DV, pages 728-737. IEEE, 2018.
[4] E. Insafutdinov and A. Dosovitskiy. Unsupervised learning of shape and pose with differentiable point clouds. NewrIPS, 2018.
[5] C.-H. Lin, C. Kong, and S. Lucey. Learning efficient point cloud generation for dense 3d object reconstruction. In AAAI, 2018.
[6] Gu, W.-. Ma, S. Manivasagam, W. Zeng, Z. Wang, Y. Xiong, H. Su, and R. Urtasun. Weakly-supervised 3D shape completion in the wild. In ECCV, 2020.
[7] Wu, C. Zhang, T. Xue, B. Freeman, and J. Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-addeesarial modeling. NeurlPS, 2016.
[8] J. Wu, Y. Wang, T. Xue, X. Sun, B. Freeman, and J. Tenenbaum. MarrNet: 3D shape recon- struction via 2.5D sketches. NeurIPS, 2017.
[9] Wu, C. Zhang, X. Zhang, Z. Zhang, W. Freeman, and J. B. Tenenbaum. Learning shape priors for single-view 3d completion and reconstruction. In ECCV, 2018.
[10] I.. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[11] Q. Xu, W. Wang, D. Ceylan, R. Mech, and Neumann. Disn: Deep implicit surface network for high-quality single-view 3d reconstruction. 2019.
[12] C.-H. Lin, C. Wang, and S. Lucey. Sdf-srn: Learning signed distance 3d object reconstruction from static images. NeurlPS, 2020.
[13] S. Duggal,, Z. Wang, W.-C. Ma. S. Manivasagam, J. Liang, Wang. and Urtasun. Mending neural implicit modeling for 3d vehicle reconstruction in the wild. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022.
[14] A. Sanders. An introduction Unreal engine 4. AK Peters/CRC Press, 2016.
[15] B. O. Community. Blender 3D modelling and rendering package. Blender Foundation, Stichting Blender Foundation, Amsterdam, 2018. URL http: / //ww://wwwww/w/ blender org.
[16] A. Dosovitskiy, G. Ros, F. Codevilla, A. Lopez, and V. Koltun. CARLA: An open urban driving simulator. In CoRI., 2017.
[17] Y. Chen, F. Rong, S. Duggal, S. Wang, X. Yan, S. Manivasagam, S. Xue, E. Yumer, and R. Urtasun. Geosim: Realistic video simulation via geometry-aware composition for self- driving. In CVPR, 2021.
[18] J. Tu, M. Ren, S. Manivasagam, M. Liang, B. Yang, R. Du, F. Cheng, and R. Urtasun. Physically realizable adversarial examples for lidar object detection. CVPR, 2020.
[19] H.-T. Liu, M. Tao, and Jacobson. Paparazzi: surface editing by way of multi-view image processing. ACM TOG, 2018.

---

[20] H.-T. D. Liu and A. Jacobson. Cubic stylization. ACM TOG, 2019.
[21] W. Yifan, N. Aigerman, V. G. Kim, S. Chaudhuri, and O. Sorkine-Hornung. Neural cages for detail- preserrneg 3d deformations. In CVPR, 2020.
[22] T. Jakab, R. Tucker, A. Makadia, Wu, N. Snavely, and A. Kanazawa. Keypointdeformer: Unsupervised 3d keypoint discovery for shape control. In CVPR, 2021.
[23] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL: A skinned multi-person linear model. SIGGRAPH Asia, 2015.
[24] S. Zuffi, A. Kanazawa, D. W. Jacobs, and M. J. Black. 3D menagerie: Modeling the 3D shape and pose of animals. In CVPR, 2017.
[25] G. Pavlakos, V. Choutas, N. Ghorbani, T. Bolkart, A. A. A. Osman, D. Tzionas, and M. J. Black. Expressive body capture: 3D hands, face, and body from a single image. In CVPR, 2019.
[26] Z. Yang, S. Manivasagam, M. Liang, B. Yang. W.-C. Ma, and R. Urtasun. Recovering and simulating pedestrians in the wild. In CoRL, 2020.
[27] H. Kato, Y. Ushiku, and T. Harada. Neural 3D mesh renderer. In CVPR, 2018.
[28] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y. Lo, J. Johnson, and G. Gkioxari. Accelerating 3d deep learning with PyTorch3DD. arXiv preprint arXiv:2007.08500 2020.
[29] J. D. Foley, F. D. Van, A. Van Dam, S. K. Feiner, J. F. Hughes, and J. Hughes. Computer graphics: principles and practice. Addison- Wesley, 1996.
[30] J. Y. Zhang, G. Yang, S. Tulsiani, and D. Ramanan. NeRS: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild. In NeurlPS, 2021.
[31] F. Engelmann, J. Stückler, and B. Leibe. SAMP: shape and motion priors for 4d vehicle reconstruction. In WACV, 2017.
[32] A. Kanazawa, S. Tulsiani, A. A. Efros, and J. Malik. Learning sategory-speccin mesh recon- struction from image collections. In ECCV, 2018.
[33] B. Nicolet, A. Jacobson, and W. Jakob. Large steps in inverse rendering of geometry. ACM TOG, 40(6):1-13, 2021.
[34] H. Pfister, M. Zwicker, J. Van Baar, and M. Gross. Surfels: Surface elements as rendering primitives. In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 335-342, 2000.
[35] Z. Yang, Y. Chai, D. Anguelov, Y. Zhou, P. Sun, D. Erhan, S. Rafferty, and H. Kretzschmar. Surfelgan: Synthesizing realistic sensor data for autonomous driving. In CVPR, 2020.
[36] S. Manivasagam, S. Wang, K. Wong, W. Zeng, M. Sazanovich, S. Tan, B. Yang, W.-C. Ma, and
R. Urtasun. LiDARsim: Realistic lidar simulation by leveraging the real world. In CVPR, 2020.
[37] V. Sitzmann. Awesome implicit representations curated list of resources on implicit neural representations.
[38] A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, W. Yifan, C. Lassner, V. Sitzmann, R. Martin-Bruall S. Lombardi, et al. Advances in neural rendering. In Computer Graphics Forum, volume 41, pages 703-735. Wiley Online Library, 2022.
[39] Y. Xie, T. Takikawa, S. Saito, O. Litany, S. Yan, N. Khan, F. Tombari, J. Tompkin, V. Sitzmann, and S. Sridhar. Neural fields visual computing and beyond. Computer Graphics Forum, volume 41, pages 641-676. Wiley Online Library, 2022.
[40] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[41] V. Sizzmann, J. Martel, A. Bergman, D. Lindell, and G. Wetzstein, Implicit neural representations with periodic activation functions. NeurIPS, 2020.
10

---

[42] M. Boss, A. Engelhardt, A. Kar, Y. Li, D. Sun, J. T. Barron, H. Lensch, and V. Jampani. SAMURAI: Shape and material from unconstrained real-world arbitrary image collections. arXiv preprini arXiv:2205. 15768, 2022.
[43] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. NeRF++: Analyzing and improving neural radiance fields. urXiv, 2020.
[44] X. Zhang, P. P. Srinivasan, B. Deng, P. Debevec, W. T. Freeman, and J. T. Barron. NeRFactor: Neural factorization of shape and reflectance under unknown illumination. arXiv, 2021.
[45] M. Tancik, B. Mildenhall, T. Wang, D. Schmidt, P. P. Srinivasan, J. T. Barron, and R. Ng. Learned initializations for optimizing coordinate-based neural representations. In CVPR, 2021.
[46] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.05988 2022.
[47] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning continuous signed distance functions for shape representation. In CVPR, 2019.
[48] P. Wang, L. Liu, Y. Liu, C. Theobalt, T. Komura, and W. Wang. NeuS: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021.
[49] K. Zhang, F. Luan, Q. Wang. K. Bala, and N. Snavely. PhySG: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In CVPR, 2021.
[50] L.. Yariv, Y. Kasten, D. Moran, M. Galun, M. Atzmon, R. Basri, and Y. Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In NeurlPS, 2020.
[51] M. Oechsle, S. Peng, and A. Geiger. UNISURF: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In ICCV, 2021.
[52] R. Chabra, J. E. Lenssen, E. Ilg, T. Schmidt,, Straub, S. Lovegrove, and R. Newcombe, Deep local shapes: Learning local SDF priors for detailed 3d reconstruction. In ECCV, 2020.
[53] T. Takikawa, J. Litalien, K. Yin, K. Kreis, C. Loop, D. Nowrouzezahrai, A. Jacobson, M. McGuire, and S. Fidler. Neural geometric level of detail: Real-time rendering with implicit 3D shapes. In CVPR, 2021.
[54] P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec. Baking neural radiance fields for real-time view synthesis. ICCV, 2021.
[55] A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. ICCV, 2021.
[56] A. Yu, S. Fridovich-Keil M. Tancik, Q. Chen, Recht, and Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:21 2021.
[57] M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan. RegNeRF: Regularizing neural radiance fields for view synthesis from sparse inputs. 5480-5490, 2022.
CVPR, pages
[58] J. Chibane, A. Bansal, V. Lazova, and G. Pons-Moll. Stereo radiance fields (srf): Learning view synthesis for sparse views of novel scenes. In CVPR, pages 7911-7922 2021.
[59] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In ICCV, pages 14124-14133, 2021.
[60] A. Trevithick and B. Yang. GRF: Learning a general radiance field for 3d representation and rendering, In ICCV, pages 15182-15192, 2021.
[61] M. M. Loper and M. J. Black. Opendr: An approximate differentiable renderer. In European Conference on Computer Vision, pages 154-16616 Springer, 2014.
[62] W. Chen, Ling, J. Gao, F. Smith, I Lehtinen, A. Jacobson, and S. Fidler. Learning to predict 3d objects with an interpolation-baser differentiable renderer. NeurIPS, 32, 2019.
11

---

[63] W. Chen, J. Litalien, J. Gao, Z. Wang, C. F. Tsang, S. Khamis, O. Litany, and S. Fidler. DIB- R++: Learning predict lighting and material with a hybrid differentiable renderer. arXiv, 2021.
[64] T. Monnier, M. Fisher, A. A. Efros, and M. Aubry. Share with thy neighbors: Single-view reconstruction by cross-instance consistency. arXiv preprint arXiv:2204. 10310, 2022.
[65] M. Guo, A. Fathi, J. Wu, and T. Funkhouser. Object-centric neural scene rendering. 2021.
[66] T. Shen, J. Gao, K. Yin, M.-Y. Liu, and S. Fidler. Deep marching tetrahedra: a hybrid representation for high-resolution 3D shape synthesis. NeurlPS, 34, 2021.
[67] J. Munkbberg, Hasselgren, T. Shen, J. Gao, W. Chen, A. Evans, Müller, and S. Fidler. Extract- ing triangular 3D models, materials, and lighting from images. arXiv preprint arXiv:2111.12500 2021.
[68] M. Tatarchenko, S. R. Richter, R. Ranftl, Z. Li, V. Koltun, and T. Brox. What do single-view 3d reconstruction networks learn? In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 3405-3414, 2019.
[69] F. Engelmann, J. Stückler, and B. Leibe. Joint object pose estimation and shape reconstruction in urban street scenes using 3d shape priors. In German Conference on Pattern Recognition, pages 219-230. Springer, 2016.
[70] Chabot, M. Chaouch, J. Rabarisoa, C. Teuliere, and T. Chateau. Deep manta: A coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 2040-2049, 2017.
[71] R. Wang, N. Yang. J. Stueckler, and D. Cremers. Directshape: Direct photometric alignment of shape priors for visual vehicle pose and shape estimation. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 1067-11073. IEEE, 2020.
[72] L. Koestler, N. Yang, R. Wang, and D. Cremers. Learning monocular 3d vehicle detection without 3d bounding box labels. In DAGM German Conference on Pattern Recognition, pages 116-129. Springer, 2020.
[73] C. Jiang, J. Huang, A. Tagliasacchi, and L. J. Guibas. ShapeFlow: Learnable deformation flows among 3d shapes. NeurlPS, 1119745-9757) 2020.
[74] M. A. Uy. J. Huang. M. Sung. T. Birdal, and L. Guibas. Deformation-aware 3d model embedding and retrieval. In ECCV, pages 397-413. Springer, 2020.
[75] M. A. Uy, V. G. Kim, M. Sung, N. Aigerman, Chaudhuri, and Guibas. Joint learning of 3d shape retrieval and deformation. In CVPR, pages 11713-11722, 2021.
[76] F. Lu, Z. Liu, X. Song, D. Zhou, W. Li, H. Miao, M. Liao, L. Zhang, B. Zhou, R. Yang, et al. PerMO: Perceiving more at once from single image for autonomous driving. arXiv preprint arXiv:200..08116. 2020.
[77] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang. Z. Li, S. Savarese, M. Savva, S. Song, H. Su, et al. ShapeNet: An information-rich 3d model repository. arXiv preprint 1rXiv:1512.030/1 2015.
[78] S. Goel, A. Kanazawa, and J. Malik. Shape and viewpoint without keypoints. ECCV, 2020.
[79] B. Karis. Real shading in unreal engine 4. Proc. Physically Based Shading Theory Practice, (3):1, 2013.
[80] S. McAuley, Hill, N. Hoffmman, Y. Gotanda, B. Smits, B. Burley, and A. Martinez. Practical physically-baset shading in film and game production. In ACM SIGGRAPH 2012 Courses, pages 1-7. 2012.
[81] B. Walter, S. R. Marschner, H. Li, and K. Torrance. Microfacet models for refraction through rough surfaces. Rendering techniques, 18th: 2007.
12

---

[82] H. Kato, Y. Ushiku, and T. Harada. Neural 3d mesh renderer. In CVPR, 2018.
[83] S. Liu, T. Li, W. Chen, and H. Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning, ICCV, 2019.
[84] W. Chen, J. Gao, H. Ling, E. Smith, Lehtinen, A. Jacobson, and S. Fidler. Learning to predict
3d objects with interpolation-bassee differentiable renderer. In NeurIPS, 2019.
[85] N. Ravi, J. Reizenstein, D. Novotny, T. Gordon, W.-Y Lo, J. Johnson, and G. Gkioxari. Accelerating 3D deep learning with PyTorch3D. wrXiv:200708850/ 2020.
[86] A. Kirillov, Y. Wu, K. He, and R. Girshick. PointRend: Image segmentation as rendering, In CVPR, 2020.
[87] Huber. Robust estimation of location parameter. In Breakthroughs statistics, pages 492-518. Springer, 1992.
[88] E. H. Land and J. J. McCann. Lightness and retinex theory. Josa, 1971.
[89] W.-C. Ma, H. Chu, B. Zhou, R. Urtasun, and A. Torralba. Single image intrinsic decomposition without a single intrinsic image. In ECCV, 2018.
[90] I. Sobel and G. Feldman. A 3x3 isotropic gradient operator for image processing, a talk at the Stanford Artificial Project, pages 271-272 1968.
[91] D. Kingma and J. Ba. Adam: method for stochastic optimization. 2015.
[92] Y. Zhou, C. Barnes, J. Lu, J. Yang, and H. Li. On the continuity rotation representations in neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5745-5753, 2019.
[93] P. Xiao, Z. Shao, S. Hao, Z. Zhang, X. Chai, J. Jiao, Z. Li, J. Wu, K. Sun, K. Jiang, et al. Pandaset: Advanced sensor suite dataset for autonomous driving. In ITSC, 2021.
[94] Tulsiani, R. Tucker, and N. Snavely. Layer-structured 3d scene inference via view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), pages 302-317, 2018.
[95] Z. Wang, A. C. Bovik, H. R. Sheikh, and E. P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4):600-612, 2004.
[96] G. Parmar, R. Zhang, and J.-Y. Zhu. On aliased resizing and surprising subtleties in gan evaluation. In Computer Vision and Pattern Recognition, 2022.
[97] M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs trained by two time-scale update rule converge to a local nash equilibrium. NeurlPS, 2017.
[98] S. Laine, J. Hellsten, T. Karras, Y. Seol, J. Lehtinen, and T. Aila. Modular primitives for perhorerffrm differentiable rendering. ACM Transactions on Graphics, 39(6), 2020.
[99] K. He, G. Gkioxari, Dollár, and R. Girshick. Mask r-cnn, ICCV, pages 2961 1669,,96,,,01 2017.
13


Reviews:
[{"review_id": "wmB5Q8VzcYy", "reviewer": "Reviewer_iXu8", "summary": "The paper proposes CADSim, a system that reconstructs vehicle geometry and appearance from multi-view images and optionally multi-view LiDAR pointclouds. The mesh-based model is initialized from a database of CAD models and then fitted to the data (images and pointclouds) by minimizing an energy function in the vertex locations, camera/LiDAR poses, and appearance. The energy function is comprised of data terms capturing the color, mask, and pointcloud consistency as well as multiple regularisation terms. The obtained vehicle models feature articulate wheels and show comparable or better performance than SOTA methods for novel view synthesis and LiDAR rendering. Finally, the obtained models are evaluated wrt. a down-stream perception tas.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The paper is well-written, clearly understandable, and of high visual quality. This is especially respectable given that the paper touches many fields, e.g. (neural) rendering, geometric losses, mesh & appearance parametrization.\n- While the idea of fitting mesh-based representations to data using CAD models as priors is not very novel in itself, the proposed method is sound and delivers clear advantages wrt. the final performance and practical usability such as fast rendering, editability, and compatibility with current rendering/game engines.\n- The paper tackles real-world data and shows high-quality results.\n- The supplementary material and the paper's references are extensive.\n\n\n**Weaknesses**:\n\n- The paper does not promise to publish open-source code. Given the very involved architecture of the system, it is likely that reproduction by a third party would either take considerable time or would fail to achieve similar results. It would be very beneficial for the community if the paper's code could be released.\n- Given the intricate nature of the proposed system, it is likely less extendable than, for example, current neural-fields-based systems.\n- Given the intricate nature of the proposed system and required prior knowledge, the paper is very information-dense and hence not an easy read. While I do not generally see this as a weakness, because this can be the nature of research, this might limit the number of follow-up works building on the proposed method.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is of very high quality and a very good contribution. The strengths clearly outweigh the weaknesses.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- In Eqn. (5) every point on the current mesh is assumed to have a close point in the point cloud. However, the back half of the car is usually not in the pointcloud. This seems like the \"direction\" of the asymmetric CD is exactly the opposite of what it should be?\n- In 3.3 Inference, Initialization (line 223 ff.) it is not mentioned how the optimization problem over $z$ is solved.\n- Instant-NGP should deliver more than 1.14 FPS (Table 1)\n- For a simulator I consider real-time rendering to be less crucial (lines 5 and 39).\n- There are a few works that could be cited in the related work:\n  - Chabot et al. (2017). “Deep manta: A coarse-to-fine many-task network for joint 2d and 3d vehicle analysis from monocular image.” (Also uses a CAD database)\n  - Engelmann et al. (2016). “Joint object pose estimation and shape reconstruction in urban street scenes using 3D shape priors.” (Uses a latent code based on CAD models)\n  - Wang et al. (2019). “DirectShape: Photometric Alignment of Shape Priors for Visual Vehicle Pose and Shape Estimation.” (similar as above but w/ photometric loss)\n  - Koestler et al. (2020). \"Learning monocular 3D vehicle detection without 3D bounding box labels.\" (similar as above but in a DL framework)\n  - Loper, M. M. and M. J. Black (2014). “OpenDR: An approximate differentiable renderer.” (early differentiable renderer)\n- For the smooth L1 loss one should cite prior work instead of Fast R-CNN: Huber, P. J. (1964). “Robust Estimation of a Location Parameter.”\n- E_LiDAR is capitalized inconsistently."}}, {"review_id": "mquDXtXqLDE", "reviewer": "Reviewer_wSHy", "summary": "This paper presents a new pipeline to provide realistic and controllable sensor (camera and Lidar) simulation to generate in-the-wild data for autonomous driving scenarios.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is clearly written and easy to follow and understand the contributions.\n* This paper is a system paper. The method in section 3.2 is not that novel, but the combination works well and shows improvement compared to previous approaches. The semantic information-driven model decomposition in section 3.1 is useful to add variety to the data generation.\n* The training time required for the system is not large, and it can achieve real-time rendering speed.\n* The supplementary materials are very informative and contain sufficient explanations to help understand this work.\n\n\nWeakness:\n* Several issues need to be addressed. They are raised below.\n* One motivation emphasised in the paper is that the exisiting works are sensitive to sensor noise and distortion. However, the experiments did not focus on this point. The input images seem to all have good lighting conditions, and LiDAR motion distortion is not obvious in the test data.\n* The apperance term in L216-218 poses a strong assumption on the surrounding lighting condition. It raises the question of whether the proposed simulation pipeline can be generalised to different scenes. For example, light reflection and shading on the vehicles at night would probably violate this term.\n* The tests are conducted mainly on the dataset benchmark. It is unclear what the required data inputs are for the system. For example, are sensor pose and object annotations required to generate data in this pipeline? It will be better if some real-world experiments can be conducted on an SDV platform.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper is a good system paper and presents a realistic sensor simulation tool for in-the-wild data generation. However, I still have a few concerns regarding this work after reading it. It will be better if those concerns can be addressed in the rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Table 1 reports training time and rendering time for the system and its comparison methods. Is the computation time of all these methods tested on the same platform as the proposed method or taken from the respective paper/report?\n\n2. In section 3.2, it is explained that the shape is initialized from learned low-dimensional latent code. Does the zero-code initiation always work? From the supplementary video, the pickup truck and the shape generated from the mean code seem quite different. Can the zero code still be optimised for the truck shape? Or is it necessary to have a network to predict a latent code?\n\n3. Generalization: The model representation in 3.1 is only applicable to the vehicle category. It is totally fine as the supplementary document also shows the reconstruction on motorcycles and cones. However, I am not sure if it is correct that a motorcycle only has 3 parts. The reconstruction quality of motorcycles also seems much worse than vehicles. I am curious if the authors have dug into the reasons behind it.\n\n4. It is a bit unclear to me how the pose of the vehicle model is initiliazed. I believe a correct alignment of the CAD model to a detected shape is important for the proposed optimisation pipeline to work. However, there seems to be no analysis on this point. \n\n4. Dynamic scenes: the supplementary material shows 3D reconstruction in static scenes and mixed reality in dynamic scenes.  Does it mean the proposed method can not currently generate 3D reconstruction in dynamic scenes yet? It is unclear in the current version.\n\nAnother issue that does not need to be addressed but is worth mentioning:\\\n* I am not sure if high-fidelity reconstruction is the only metric to care about in the data generation. Sensor noise and distortion are also important when simulating real-world data and testing whether the self-driving vehicles can be robust to those outliers and distortion."}}, {"review_id": "jGqoL_GOhh", "reviewer": "Reviewer_NPXi", "summary": "This paper proposes a 3D reconstruction pipeline motivated by the need to simulate different sensors in data-driven simulated environments. The method employs CAD models as priors in improving 3D reconstruction quality. The result is shown generating more complete models that are compatible with the gamut of visual processing augmentations including re-animation and texture transfer.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper tackles an important problem that should be of interest to a substantial portion of the robotics community.\n\nThe results are visually impressive, including recovery of complete 3D geometry from incomplete scans and simulation of camera and LiDAR in complex scenes.\n\nThe task this paper has set out to accomplish is fundamentally that of reconstructing 3D scenes from sensor data. This is a very well explored space. Note that all methods capable of building a model and rendering novel views are carrying out sensor simulation (of cameras) and are generally trivially extended to simulate LiDAR.  \n\nAs divulged briefly in the literature review (lines 95-104), extensive work employing 3D CAD model priors has appeared throughout the graphics and vision communities. The revised paper expands its treatment of related work, especially around use of CAD priors, and positions itself relative to existing work in this very active area of research. It also includes in the supplement a consideration of robustness to data noise including, importantly for a paper about sensor simulation, robustness to sensor noise.\n\nThe system-level approach here is novel, and introducing techniques from the graphics and computer vision communities into robotics is also of value.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The system-level approach here is novel, and introducing techniques from the graphics and computer vision communities into robotics is also of value. I therefore intend to upgrade my review to a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "-", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "BekRLyWw8yt", "reviewer": "Reviewer_gqRW", "summary": "This paper presents a novel method for extracting 3D meshes for a specific category from videos taken in the wild. Instead of relying on a sphere as a starting point for their reconstruction, they use the LIDAR data to find a close matching CAD model of the current category, which is then adapted to be as close as possible to the real object. This reliance on CAD models allows them further to predict each model's texture and lighting mode. Lastly, through the use of these decomposed CAD models, they manage to extract the wheel rotation independent of the car orientation. The evaluation is done by comparing the reconstruction performance with typical object and scene reconstruction methods like NERF++ or NeRS.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n* The paper is well written and nicely presented\n* The supplementary video is outstanding and provides a nice overview with an excellent presentation of the results\n* I particular like the decoupling of the model into different parts, allowing for later manipulation of those\n\n\nWeaknesses:\n* While the evaluation compared to other methods is done, the paper claims that these models can be used to train other machine learning methods. However, this claim was not evaluated in the paper. But I can see how the manifold of different evaluations already was enough for one publication.\n* I am confused about 4.3, where the downstream evaluation on camera simulated data is done. Here, the extracted 3D models are rendered back into images and tested on different DL methods. Their performance is then compared between the real images and the simulated ones. They then state: \"how well we can use the simulation to test existing perception systems\". The problem with this is that this claim seems to state that we could use the simulated data to evaluate any future system and avoid using real data, which is of course, the dream. However, I do not believe this claim can be made based on the presented results, here the test is limited to two scenarios, which only shows that the used networks do well in this task, even for simulated data. Not that any other method would also do equally well on this simulated data. Could you please rephrase that statement.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Even though I am not happy with a few things in the paper. I overall like the presentation and the concept of the paper very much. I think it would be a valuable addition to CORL. I would still love to hear an answer to my two single weaknesses, but I can understand that a paper can not include every test possible under the sun.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "* clarify the statement in 4.3", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)