3DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Views

## Abstract

Abstract 3D cars are commonly used in self-driving systems, virtual/augmented reality, and games. However, existing 3D car datasets are either synthetic or low-quality, presenting a significant gap toward the high-quality real-world 3D car datasets and limiting their applications in practical scenarios. In this paper, we propose the first large-scale 3D real car dataset, termed 3DRealCar, offering three distinctive features. (1) High-Volume : 2,500 cars are meticulously scanned by 3D scanners, obtaining car images and point clouds with real-world dimensions; (2) High-Quality : Each car is captured in an average of 200 dense, high-resolution 360-degree RGB-D views, enabling high-fidelity 3D reconstruction; (3) High-Diversity : The dataset contains various cars from over 100 brands, collected under three distinct lighting conditions, including reflective, standard, and dark. Additionally, we offer detailed car parsing maps for each instance to promote research in car parsing tasks. Moreover, we remove background point clouds and standardize the car orientation to a unified axis for the reconstruction only on cars without background and controllable rendering. We benchmark 3D reconstruction results with state-of-the-art methods across each lighting condition in 3DRealCar. Extensive experiments demonstrate that the standard lighting condition part of 3DRealCar can be used to produce a large number of high-quality 3D cars, improving various 2D and 3D tasks related to cars. Notably, our dataset brings insight into the fact that recent 3D reconstruction methods face challenges in reconstructing high-quality 3D cars under reflective and dark lighting conditions. Our dataset is available here.

## 1 Introduction

Cars, as both daily objects and vehicles, are of significant interest to researchers, especially in the field of autonomous driving.
Autonomous perception systems are typically trained on daily scene datasets that are collected frequently. However, these datasets often exhibit long-tailed distributions, with far fewer instances of corner-case scenarios, like car accidents.
Consequently, this imbalance leads to the autonomous perception system generalizing well in the most frequently occurring scenes. This means that the system is likely to perform poorly in rare situations, posing significant safety risks to drivers. To build a reliable system, it is essential to have a simulator that can simulate photorealistic hazardous scenes. Moreover, high-quality 3D cars are necessary for a realistic simulator.

Recent 3D car reconstruction methods [, , ] mainly reconstruct cars from self-driving datasets [, , ].
To apply reconstructed cars to real-world scenes, the reconstructed 3D car should be high-quality.
However, it is very challenging to obtain such high-quality 3D cars for the following reasons: (1) Previous 3D car reconstruction methods produce low-quality 3D cars, primarily because they train on self-driving datasets with low-resolution car images and a limited number of trainable views. (2) Manually crafting a high-quality 3D car model requires specialized artists, which is time-consuming. (3) There is no large-scale 3D real car dataset that can be utilized to produce a bulk of 3D cars.

Moreover, existing 3D car datasets are either synthetic or only contain a few posed images, as shown in Figure 2. SRN-Car [] and Objaverse-Car [] collect 3D car computer-aided design (CAD) models from the Internet, but these models are synthetic and contain non-photorealistic texture. Although MVMC [] is a real car dataset, it collects only ten views on average for each car.
On the contrary, our collected 3DRealCar dataset provides an average of 200 dense RGB-D views per car for high-quality 3D car reconstruction.

We also show that the recent state-of-the-art 3D generative method, MVDream [], as depicted in Figure 2, fails to generate high-quality cars due to the multi-view inconsistency introduced by generative models [, , , ].
Thus, the existing 3D generation methods cannot be employed to generate high-quality 3D real car assets.

In this work, we collect a large-scale 3D real car dataset in the wild, termed 3DRealCar, which contains dense high-quality views and rich diversity. During data collection, we employ 3D scanners to scan cars parked on roadsides or parking lots, obtaining posed RGB-D images and point clouds of cars. In particular, we scan around the cars in three loops to obtain dense views. Note that we collect car data with the consent of owners. In Table 1 and Figure 1, we show our dataset possesses striking characteristics compared with previous 3D car datasets. We capture dense RGB-D images in high resolution, which promotes the reconstruction of high-quality 3D cars. Furthermore, we scan cars under three different lighting conditions, resulting in the surfaces of cars having different lighting effects, such as reflective, standard, and dark, where we denote the standard as the smooth lighting condition without obvious specular highlight. Figure 2 shows some examples of three lighting conditions in our dataset.
Note that the number of instances in our dataset is the largest in existing datasets.
Therefore, our collected 3DRealCar dataset has a rich diversity in terms of car types, colors, brands, and lighting conditions.
We also provide car parsing map annotations with thirteen classes for each instance, which enable our dataset to be applied in car component understanding tasks.

To construct a high-quality dataset, we filter out the images that are out of focus, occluded, or blurred.
To facilitate the 3D reconstruction solely on cars, we remove the point clouds of the background. We also adjust the orientation of the car facing along the x-axis before the reconstruction for controllable rendering.
Based on the high-quality posed RGB-D images, point clouds, and multi-grained annotations, we can apply the dataset to various tasks related to cars.
Figure 3 shows our dataset supports over 10 tasks, including several popular 2D and 3D tasks to promote the advancement of car-related research.

We leverage existing state-of-the-art methods to benchmark 3D reconstruction and car parsing tasks of our 3DRealCar dataset.
We also conduct extensive experiments to demonstrate that the reflective and dark lighting conditions in our dataset are challenging to existing methods, which brings a new challenge for 3D reconstruction in awful lighting conditions. Furthermore, we demonstrate that our 3DealCar dataset can bring real-car prior and enhance existing 3D generation and downstream methods.
Overall, the contributions of this work can be summarized below:

$\bullet$ We propose the first large-scale 3D real car dataset, named 3DRealCar, which contains 2,500 car instances and their point clouds with actual sizes in real-world scenes.

$\bullet$ 3DRealCar contains RGB-D images and point clouds with detailed annotations, supporting researchers to investigate various tasks in both 2D and 3D.

$\bullet$ We conduct 3D reconstruction and car parsing benchmarks to advance car-related tasks. Notably, we observe that existing methods face challenges under the extreme lighting conditions of 3DRealCar.

$\bullet$ Extensive experiments demonstrate our 3DRealCar dataset can enhance real-car prior and improve the performance of existing 3D generation and novel view synthesis methods.

Figure: Figure 3: The applicable tasks of our dataset. Our proposed 3DRealCar dataset containing RGB-D images and point clouds can be applied to many popular 2D and 3D tasks.
Refer to caption: x3.png

## 2 Related Work

3D Car Datasets. There are several well-known large-scale autonomous driving datasets so far, such as Nuscenes [], KITTI [], Waymo [], Pandaset [], ApolloScape [], and Cityscape []. These datasets are captured by multi-view cameras and lidars mounted on ego cars. Various works [, , , ] attempt to reconstruct 3D cars in these datasets. However, these methods fall short of reconstructing high-quality 3D cars due to the lack of sufficient and dense training views.
SRN-Car [] and Objaverse [] collect
3D car models from existing repositories and Internet sources. However, these datasets only contain synthetic cars, which cannot produce realistic textures and geometry. MVMC [] is collected from car advertising websites, which contain a series of car images, especially multi-view images of each car. However, the views of images per car in MVMC are unposed and sparse, which is adverse to reconstructing high-quality 3D car models.
In this paper, we collect a high-quality 3D real car dataset to fill the above gaps.

3D Reconstruction with Neural Field. 3D reconstruction aims to create a 3D structure digital representation of an object or a scene from its multi-view images, which is a long-standing task in computer vision.
One of the most representative works in 3D reconstruction is Neural Radiance Fields (NeRFs) [], which demonstrates promising performance for novel view synthesis.
Afterward, this method inspires a new wave of 3D reconstruction methods using the volume rendering method, with subsequent works focusing on improving its quality [, , , , , , , ], efficiency [, , , , , , ], applying artistic effects [, , , ], and generalizing to unseen scenes [, , , , , ]. Particularly, Kilonerf [] accelerates the training process of NeRF by dividing a large MLP into thousands of tiny MLPs. Furthermore, Mip-NeRF [] proposes a conical frustum rather than a single ray to ameliorate aliasing. Mip-NeRF 360 [] further improves the application scenes of NeRF to the unbounded scenes. Although these NeRF-based methods demonstrate powerful performance on various datasets, the training time always requires several hours even one day more. Instant-NGP [] uses a multi-resolution hash encoding method, which reduces the training time by a large margin. 3DGS [] proposes a new representation based on 3D Gaussian Splatting, which reaches real-time rendering for objects or unbounded scenes. 2DGS [] proposes a perspective-accurate 2D splatting process that leverages ray-splat intersection and rasterization to further enhance the quality of the reconstructions. Scaffold-GS [] proposes an anchor growing and pruning strategy to accelerate the scene coverage, which effectively reduces redundant Gaussians and improves rendering quality. However, there is not yet a large-scale 3D real car dataset so far. Therefore, we present a 3D real car dataset, named 3DRealCar in this work.

3D Generation with Diffusion Prior.
Some current works [, ] leverage a 3D diffusion model to learn the representation of 3D structure. However, these methods lack generalization ability due to the scarcity of 3D data.
To facilitate 3D generation without direct supervision of 3D data, image or multi-view diffusion models are often used to guide the 3D creation process.
Notable approaches like DreamFusion [] and subsequent works [, ] use an existing image diffusion model as a scoring function, applying Score Distillation Sampling SDS loss to generate 3D objects from textual descriptions.
These methods, however, suffer from issues such as the Janus problem [, ] and overly saturated textures.
Inspired by Zero123 [], several recent works [, , , , , , , , ] refine image or video diffusion models to better guide the 3D generation by producing more reliable multi-view images. However, these generative methods fail to generate high-quality cars due to the lack of the prior of real cars.

Figure: Figure 4: Illustration of our data collection and preprocessing. We first circle a car three times while scanning the car with a 3D scanner for the attainment of RGB-D images and its point clouds. Then we use Colmap [] and SAM [] to obtain poses and remove the background point clouds. Finally, we use the 3DGS [] trained on the processed data to obtain 3D car model.
Refer to caption: x4.png

## 3 Proposed 3DRealCar Dataset

### 3.1 Data Collection and Annotation

As shown in Figure 4, our dataset is collected using smartphones, specifically iPhone 14 models, equipped with a 3D scanner application to scan cars for their point clouds and RGB-D images. The data collection process is conducted under three distinct lighting conditions, such as standard, reflective, and dark. These lighting conditions represent the lighting states of vehicle surfaces. It is important to note that all data collection is performed with the consent of owners. During the scanning process, the car should be stationary while we meticulously circle the car three times to capture as many views as possible. For each loop, we adjust the height of the smartphone to obtain images from different angles. Furthermore, we try our best to make sure captured images contain the entire car body without truncation. To preserve the privacy of owners, we make license plates and other private information obfuscated. To construct a high-quality dataset, we filter out some instances with blurred, out-of-focus, and occluded images. We also provide detailed annotations for car brands, types, and colors. Particularly, we provide the car parsing maps for each car with thirteen classes in our dataset as shown in Figure 1 for the advancement of car component understanding tasks.

### 3.2 Data Preprocessing

Background Removal. Since we only reconstruct cars for the 3D car reconstruction task, the background should be removed.
Recent Segment Anything Model (SAM) [] demonstrates powerful context recognition and segmentation performance. However, SAM needs a bounding box, text, or point as a driving factor for accurate segmentation. Therefore, we employ Grounding DINO [] as a text-driven detector with a detection prompt with “car” for the attainment of car bounding boxes. With these bounding boxes, we use SAM to obtain the masks from captured images.
The point cloud initialization is demonstrated useful for the convergence of 3D Gaussian Splatting [].
Except for the removal of the background in 2D images, we still need to remove the background point clouds. Therefore, we first project the 3D point clouds into 2D space with camera parameters. Then, we can eliminate background point clouds with masks and save them for further processing.

Orientation Rectification. As shown in Figure 4, we utilize Colmap [] to reconstruct more dense point clouds and obtain accurate camera poses and intrinsics because we find that the estimated poses by the 3D scanner are not accurate. However, after the removal of the background point clouds, we find that the car orientation of the point cloud is random, which leads to the subsequent render task being uncontrollable. Given camera poses $P=\{p_{i}\}_{1}^{\mathcal{N}}$, where $\mathcal{N}$ is the number of poses, we use Principal Component Analysis (PCA) [] to obtain a PCA component $\mathcal{T}\in\mathbb{R}^{3\times 3}$.
The PCA component is the principal axis of the data in 3D space, which represents rotation angles to each axis. Therefore, we leverage it to rectify the postures of cars parallel to the x-axis. However, this process cannot guarantee cars facing along the x-axis. Therefore, in some failure cases, we manually interfere and adjust the orientation along the x-axis. With the fixed car orientation, we can control rendered poses for the subsequent tasks.

Point Cloud Rescaling. The size of the point clouds reconstructed by Colmap [] does not match the real-world size, which inhibits the reconstruction of a practically sized 3D car. To address this, we calculate the bounding box of the scanned foreground point clouds to obtain its actual size in the real-world scene. Then, we rescale the rectified point clouds into the real size. In addition to the rescaling of the point clouds, we also need to adjust the camera poses. We rescale translations of camera poses using a scale factor calculated by the ratio of scanned point cloud size and Colmap point cloud size. After these rescaling processes, we use rescaled point clouds to reconstruct a 3D car model through recent state-of-the-art methods, like 3DGS [].

Figure: Figure 5: The distributions of our 3DRealCar dataset. We show distributions of car types, lighting conditions, captured views, car colors, and car size. We try our best to capture cars with various colors and types for the diversity of our dataset.
Refer to caption: x5.png

### 3.3 Data Statistics

In our 3DRealCar, we provide detailed annotations for researchers to leverage our dataset for different tasks. During the data annotating, we discard the data with the number of views less than fifty. As we can observe in Figure 1 and 2, we collect our dataset under real-world scenes and meticulously scan dense views. Therefore, cars in our dataset possess dense views and realistic texture, which is necessary for the application in a real-world setting.

As shown in Figure 5, we conduct detailed statistical analysis to show the features of our dataset. Our dataset mainly contains six different car types, such as Sedan, SUV, MPV, Van, Lorry, and Sports Car. Among them, sedans and SUVs are common to collect in real life, so their volume dominates in our dataset. We also count the number of different lighting conditions on cars. The standard condition means the car is well-lighting and without strong specular highlights. The reflective condition means the car has strong specular highlights. Glossy materials bring huge challenges to recent 3D reconstruction methods. The dark condition means the car is captured in an underground parking so not well-lighting. To promote high-quality reconstruction, we save the captured images in high resolution (1920$\times$1440) and also capture as many views as possible. The number of captured images per car is an average of 200. The number of views ranges from 50 to 400. To enrich the diversity of our dataset, we try our best to collect as many different colors as possible. Therefore, our dataset contains more than twenty colors, but the white and black colors still take up most of our dataset. In addition, we also show the distribution of car size, in terms of their length, width, and height. We obtain their sizes by computing the bounding boxes of the scanned point clouds. Thanks to different car types, the sizes of cars are also diverse.

## 4 Overview of 3DRealCar Tasks

### 4.1 2D tasks

Corner-case scene 2D Detection []: Given a serial of images $I=\{I_{i}\}_{1}^{\mathcal{N}}$, this task aims to detect vehicles as accurately as possible. However, in some corner cases, like car accidents, detectors sometimes fail to detect target vehicles since this kind of scene is rare or not in the training set.

2D Car Parsing [, , , ]: Given a serial of images $I=\{I_{i}\}_{1}^{\mathcal{N}}$ and their car parsing maps $S=\{S_{i}\}_{1}^{\mathcal{N}}$, this task aims to segment car components. With annotated parsing maps, we can train a model to understand and segment each component of cars, which is very important in self-driving.

### 4.2 3D Tasks

3D Reconstruction [, , ]: Given a serial of images $I=\{I_{i}\}_{1}^{\mathcal{N}}$ and matched poses $P=\{p_{i}\}_{1}^{\mathcal{N}}$, where $\mathcal{N}$ is the number of images and poses, the task of 3D reconstruction aims to reconstruct 3D model of a object or a scene. The reconstructed 3D model is usually used to render 2D images with different views for the evaluation of the performance of the 3D model.

Novel View Synthesis [, , ]: Given a serial of reference images $I^{ref}=\{I_{i}^{ref}\}_{1}^{\mathcal{N}}$, reference poses $P^{ref}=\{p_{i}^{ref}\}_{1}^{\mathcal{N}}$, target images $I^{target}=\{I_{i}^{target}\}_{1}^{\mathcal{N}}$, and target poses $P^{target}=\{p_{i}^{target}\}_{1}^{\mathcal{N}}$, recent 3D generative models, such as Zero123 [], Syncdreamer [], and Stable-Zero123 [], take relative poses and reference images as inputs and generate target images. However, these models cannot generalize well to real car objects since they are trained on large-scale synthetic datasets [, ]. In this work, we will demonstrate that our dataset can improve the robustness of these generative models to real cars.

3D Generation [, , ]: Given a text prompt or single image, recent 3D generation methods generate 3D objects with Score Distillation Sampling (SDS) [] and diffusion generative models [, , ]. However, these methods cannot generate high-quality 3D cars due to the lack of the prior of real cars in 3D-based diffusion models. Therefore, we would demonstrate the value of our dataset by improving recent 3D generation for real cars.

Figure: Figure 6: The simulated corner-case scenes. We use a red rectangle to highlight the simulated vehicles. These corner-case scenes show some vehicles have potential risks to traffic safety.
Refer to caption: x6.png

**Table 2: Benchmark results on 2D car parsing of our 3DRealCar dataset. We use recent advanced image segmentation methods [, , , ] to benchmark our dataset.**
| Method | DeepLabV3 | PointRend | DDRNet | SegFormer |
| --- | --- | --- | --- | --- |
| mIOU $\uparrow$ | 0.556 | 0.562 | 0.603 | 0.613 |
| mAcc $\uparrow$ | 0.616 | 0.619 | 0.659 | 0.663 |

## 5 Experiments

### 5.1 Experimental Setup

Corner-case 2D Detection. In this task, we leverage the reconstructed cars to simulate rare and corner-case scenes. To be specific, we use Nuscenes [] as background to simulate corner-case scenes with reconstructed cars and leverage recent popular detectors, like YOLOv8 [], as detectors for evaluation. To evaluate the robustness of detectors in corner-case scenes, we use the test part of the corner-case dataset, CODA [] as a testing set. Since we focus on the corner-case scenes of cars, so we only evaluate a car class.

2D Car Parsing. In this task, we utilize DeepLabV3[], DDRNet [], SegFormer [], and PointRend [] to benchmark our dataset. To be specific, we split 80% of our car parsing maps in 3DRealCar as the training set and the rest of 20% as the testing set.

3D Reconstruction.
In this task, we randomly choose 100 instances from each lighting condition in our dataset and split 80% of the views per instance as the training set and the rest of 20% as the testing set. Specifically, we employ Instant-NGP [], 3DGS [], GaussianShader [], and 2DGS [] to benchmark our dataset.

Novel View Synthesis.
We finetune Zero123-XL [] on our 3DRealCar dataset to enhance its generalization to real cars. Note that since the training of diffusion-based models needs entire objects centered on images, we use the images rendered by our trained 3D models as training images.

3D Generation.
In this task, we exploit Dreamcraft3D [] as our baseline.
Dreamcract3D exploits Stable-Zero123 [] as a prior source for providing 3D generative prior. By fine-tuning Stable-Zero123 on our dataset, we enable it to obtain car-specific prior so it generalizes well to real cars.

Figure: Table 4: Detection improvements by simulated data for corner-case scenes. We report the metric by calculating mAP@0.5 on CODA dataset [].

### 5.2 2D Tasks

Figure: Figure 7: Qualitative comparisons among recent advanced image segmentation methods. We select the inputs from the testing set of our images and evaluate the capacity of car component understanding for each method.
Refer to caption: x7.png

Corner-case 2D Detection.
As shown in Table 4, we employ four variants of YOLOv8 serial models as our detectors for training. To evaluate the performance of models in corner-case scenes, we leverage the test part of the CODA dataset [] as our testing set.
In particular, when we increase the training simulated data from 500 to 5,000, the performance of detectors is also improved by a large margin. This phenomenon demonstrates that our simulated data is effective in improving a detector robust to corner-case scenes.
We provide the visualizations of simulated corner-case scenes in Figure 6. The detailed simulation process and more visualizations can be seen in the supplementary.

2D Car Parsing. We conduct benchmarks for car parsing maps of our dataset using recent image segmentation methods, such as DeepLabV3[], PointRend[], DDRNet[], and SegFormer[]. The quantitative performance for these methods on our dataset is summarized in Table 3. Visual comparisons are provided in Figure 7. Our high-quality dataset enables these methods to achieve promising performance, highlighting its potential for application in self-driving systems.

### 5.3 3D Tasks

3D Reconstruction.
As depicted in Table 5, we show benchmark results of recent state-of-the-art 3D reconstruction methods, such as Instant-NGP [], 3DGS [], GaussianShader [], and 2DGS [] on our dataset. To the standard lighting condition, we can find that recent methods are capable of achieving PSNR more than 27 dB, which means these methods can reconstruct relatively high-quality 3D cars from our dataset. However, the reflective and dark condition results are lower than the standard. These two parts of our 3DRealCar bring two challenges to recent 3D methods. The first challenge is the reconstruction of specular highlights. Due to the particular property of cars, materials of car surfaces are generally glossy, which means it would produce plenty of specular highlights if cars are exposed to the sun or strong light. The second challenge is the reconstruction in a dark environment. The training images captured in the dark environment lose plenty of details for reconstruction. Therefore, how to achieve high-quality reconstruction results from these two extremely lighting conditions is a challenge to recent methods. 3D visualizations can be found on our project page.

Novel View Synthesis.
As illustrated in the left part of Figure 8, we show visual comparisons of Zero123-XL [] and our improved version by training on our dataset. As we can see, given input images, we use Zero123-XL and our improved version to synthesize novel views. In this figure, we can find that Zero123-XL prefers to generate synthetic results with unrealistic texture and geometry, due to the lack of prior for real objects. In contrast, our improved version of Zero123-XL can generate photorealistic geometry and texture, which demonstrates the effectiveness of our dataset.

3D Generation.
As depicted in the right part of Figure 8, we visualize 3D generation results of the recent state-of-the-art single-image-to-3D method, Dreamcraft3D [], along with its improved version by our dataset. This figure shows that Dreamcraft3D sometimes fails to generate complete geometry or realistic texture, due to the scarcity of the real car prior. As shown in Table 3, we also show quantitative comparisons of Dreamcraft3D and its improved version. CLIP-I means the similarity of rendered images with the original input.
The quantitative and qualitative results indicate our dataset significantly improves 3D generation performance.

**Table 5: Benchmark results on 3D reconstruction of our 3DRealCar dataset. We present the 3D reconstruction performance of recent state-of-the-art methods in three lighting conditions, standard, reflective, and dark, respectively. The best results are highlighted.**
| Method | Standard | Reflective | Dark |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| PSNR $\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ | PSNR$\uparrow$ | SSIM$\uparrow$ | LPIPS$\downarrow$ |  |
| Instant-NGP [] | 27.31 | 0.9315 | 0.1264 | 24.37 | 0.8613 | 0.1962 | 23.17 | 0.9152 | 0.1642 |
| 3DGS [] | 27.47 | 0.9367 | 0.1001 | 24.58 | 0.8647 | 0.1852 | 23.51 | 0.9181 | 0.1613 |
| GaussianShader [] | 27.53 | 0.9311 | 0.1109 | 25.41 | 0.8684 | 0.1423 | 23.39 | 0.9172 | 0.1631 |
| 2DGS [] | 27.58 | 0.9361 | 0.1084 | 23.19 | 0.8509 | 0.2041 | 22.63 | 0.9148 | 0.1681 |

Figure: Figure 8: Visualizations of novel view synthesis (left) and image-to-3D generation (right). we compare the results of the recent state-of-the-art method, Zero123-XL [], Dreamcraft3D [], and their improvement by training on our dataset.
Refer to caption: x8.png

## 6 Conclusion

In this paper, we propose the first large-scale high-quality 3D real car dataset, named 3DRealCar.
The collected dense and high-resolution 360-degree views for each car can be used to reconstruct a high-quality 3D car.
Extensive experiments demonstrate the efficacy and challenges of our 3DRealCar in 3D reconstruction. Thanks to the reconstructed high-quality 3D cars from our dataset and car-part level annotations, our dataset can be utilized to support various tasks related to cars.
In addition, the benchmarking results can serve as baselines for prospective research.
Although 3DRealCar currently only has car exterior views, we intend to provide both exterior and interior views in the future to further promote the reconstruction of more intact 3D cars.

## Checklist

- 1.
For all authors…
(a)
Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?
[Yes]
(b)
Did you describe the limitations of your work?
[Yes]
(c)
Did you discuss any potential negative societal impacts of your work?
[Yes] We ensure our dataset would not bring negative societal impacts.
(d)
Have you read the ethics review guidelines and ensured that your paper conforms to them?
[Yes]
- 2.
If you are including theoretical results…
(a)
Did you state the full set of assumptions of all theoretical results?
[No]
(b)
Did you include complete proofs of all theoretical results?
[No]
- 3.
If you ran experiments (e.g. for benchmarks)…
(a)
Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)?
[Yes] We provide a URL in the abstract for researchers to download our dataset with detailed instructions.
(b)
Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)?
[Yes] Training details are described in Section 4. More details can be found in supplementary materials.
(c)
Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)?
[No]
(d)
Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)?
[Yes]
- 4.
If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…
(a)
If your work uses existing assets, did you cite the creators?
[Yes]
(b)
Did you mention the license of the assets?
[No] They are public and we only use them for academic study.
(c)
Did you include any new assets either in the supplemental material or as a URL?
[No]
(d)
Did you discuss whether and how consent was obtained from people whose data you’re using/curating?
[Yes] Our dataset is collected under the consent of owners.
(e)
Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content?
[No]
- 5.
If you used crowdsourcing or conducted research with human subjects…
(a)
Did you include the full text of instructions given to participants and screenshots, if applicable?
[Yes]
(b)
Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable?
[No] Our dataset does not contain any potential participant risks.
(c)
Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation?
[No] We collect our dataset by ourselves. We do not need to pay for participants.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - [Q1] Is the proposed dataset restricted to academia or is commercial use
  allowed?
- [Q2] I looked at the sample car reconstruction provided on the 3DRealCar
  dataset, and noticed that the COLMAP output mesh, =textured_output.obj=, is
  missing the upper half of the car. Is this expected?
- [Q3] Are any of the vehicles in the dataset captured in more than one
  environment? It is valuable to have the same vehicle captured in multiple
  environments, with varying lighting conditions, because it creates ground
  truth for relighting. Relighting is challenging and it is otherwise very
  difficult to get non-synthetic ground truth, so having some of the cars
  captured in 2+ lighting conditions would be valuable. However, I cannot tell
  whether this is the case in the proposed dataset.
- [Q3] Will the dataset also include pre-computed dense reconstructions like
  splats, or dense meshes extracted from a NeRF-like method? As mentioned above, it
  seems like the COLMAP mesh outputs can be incomplete.
- [Q4] On L346, "2D Car Parsing", why are car parsing maps given as input to
  segmentation? Or does "S" refer to just a binary mask of the vehicle? (Vehicle
  vs. background?)
soundness: 4
strengths: - [S1] The dataset includes diverse car types, including sedans, sports cars,
  and small trucks, captured in high resolution, and annotated with part
  information (e.g., doors, hood, wheels). This has applications in benchmarking
  3D reconstruction and in developing simulators for autonomous driving.
- [S2] The experimental section analyses multiple tasks such as neural rendering
  and 3D generative modeling, as well as camera-based perception in corner-case
  scenarios. In the latter example, the authors demonstrate that synthesizing
  corner-case images (e.g., cars swerving off road) using their proposed 3D car
  assets can help improve real data perception on these scenarios.
- [S3] The dataset is already openly available online!
summary: The paper presents a new dataset of 2.5k real cars scanned using modern
iPhones, resulting in high-resolution images as well as sparse LiDAR point
clouds (~200 views per car). This setting provides additional training and
benchmarking opportunities in novel view synthesis (NVS).

Key tasks which can be researched and benchmarked with this dataset include 3D
reconstruction, relighting, and parsing of car parts.

The dataset is further motivated with experiments demonstrating that using its
3D cars to perform image data augmentation can lead to perception improvements
in unusual scenarios. The dataset is also shown to help with part segmentation,
3D generative modeling, and 3D neural rendering.

One of my main questions regarding the paper, as elaborated in the Questions
Section, is whether any car is captured in more than one lighting condition. I
think this is a major research gap, so the presence of this data is pivotal in
assessing the impact of this dataset.
weaknesses: - [W1] The LiDAR used is the iPhone 14 model, which is sparser than typical
  automotive LiDARs. While denser LiDAR can of course be re-simulated from dense
  3D models, this will require additional engineering effort while also
  suffering from some domain gap. This should be clarified in the intro: "3D
  scanner" makes me think of automotive or survey-grade LiDAR, not smartphone.
- [W2] Some minor suggestions for additional references and discussions: Please
  consider mentioning DeepMANTA [0] as related work since, while old, it also
  proposed similar part-level annotations.
- This isn't a weakness but as a tip: It can be helpful to provide some usage
  examples and tutorials for the dataset next to its SDK. For example "how to
  run gaussian splatting on 3DRealCar", since this can help more people learn to
  be familiar with the dataset, which can increase its impact.
- Other small suggestions:
  - For the "Stable Zero123" reference please surround the author name in curly
    braces in the .bib file, so the citation shows up as "Stability AI, 2023"
    instead of just "AI, 2023".
  - For those curious (such as myself!) it would be helpful to add some
    low-level details on the data collection in the appendix. For example, I
    have no idea how to get raw LiDAR points from an iPhone, so a brief
    discussion would be interesting.
  - L325: "the car is well-lighting" -> "the car is well-lit"
  - L417: "Dreamcract3D" contains a typo
- References:
  - [0]: Chabot, Florian, et al. "Deep manta: A coarse-to-fine many-task network
    for joint 2d and 3d vehicle analysis from monocular image." Proceedings of
    the IEEE conference on computer vision and pattern recognition. 2017.

---

confidence: 5
contribution: 1
initial_rating: 5
presentation: 3
questions: See weakness.
soundness: 2
strengths: 1. The structure of the paper is well-organized and the content is clearly presented;
2. The effectiveness of the dataset is demonstrated through its application to various downstream tasks, including various 2D and 3D tasks.
3. This paper introduces a novel 360-degree real car dataset. Previous literature has not extensively covered. Both the quality and the diversity of the dataset are commendable.
summary: This paper introduces a comprehensive 3D car dataset featuring high-quality 360-degree scans. The dataset comprises 2,500 cars, each scanned with 3D scanners to produce 200 high-resolution RGB-D views. Additionally, the dataset includes variations across three different lighting conditions: reflective, standard, and dark.

The authors detail the data collection methodology extensively. Data capture involved the use of RGB-D sensors, followed by the recovery of camera poses using Colmap Structure from Motion (SfM), and mask extraction utilizing Grounding-DINO. The 3D models were then reconstructed with 3DGS.

The paper discusses extensive 2D and 3D downstream experiments conducted using the dataset, which include 2D detection and segmentation, depth estimation, point cloud completion, and 3D car generation. The experiment results demostrate the effectiveness of the collected data on various tasks.
weaknesses: 1. The data collection and processing method is not novel and is a standard approach to reconstructing 3D assets.
2. The paper does not highlight its uniqueness and irreplaceability, especially in terms of improving 2D parsing and 2D detection performance.
3. The results of the NVS task (Fig.8 and 9) are considerably inferior to the state of the art.
4. Many previous papers, including CADSim (https://arxiv.org/pdf/2311.01447), GeoSim (Chen et al., CVPR'21), and other related works, have demonstrated the capability to reconstruct cars from real-captured data, which I believe is a more extensible way to reconstruct car assets. The authors claim that their data is of higher quality, but the paper does not demonstrate the necessity of such high quality, especially for downstream tasks.

Although the paper provides a very useful dataset and makes a significant contribution, the downstream tasks and data collection processes are not novel and do not meet the acceptance standards of ICLR.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Data of the same car under different lighting conditions could be added to enhance the analysis of lighting effects. Additionally, including metadata such as environmental maps or more detailed car materials could improve the dataset's versatility.
soundness: 3
strengths: 1. This paper addresses the limitations of current car datasets by capturing real-world car data with diverse samples, high-quality images, and point clouds.
2. The idea of introducing 3 different lighting conditions is interesting and makes up for the shortcomings of existing datasets
3. The extensive experiments demonstrate the effectiveness of this dataset on various tasks.
summary: The paper proposes 3DRealCar, a large-scale 3D real car dataset with RGB-D images and point clouds of 2,500 cars captured in real-world environments. It addresses the limitations of existing car datasets that usually use synthetic data or have low-quality data. This dataset includes cars under three lighting conditions (standard, reflective, and dark), promoting research in 3D car reconstruction, parsing, and novel view synthesis. Benchmarks with state-of-the-art methods demonstrate that existing methods struggle in reflective and dark lighting conditions, emphasizing the dataset’s value for improving 3D reconstruction methods.
weaknesses: 1. Though collecting data is costly, the contribution of this work is only on the dataset and lacks technical contribution for 3D/2D car understanding. 
2. This dataset captures cars under three lighting conditions, including reflective, standard, and dark; however, it seems to lack data of the same car under all three conditions, which limits the exploration of the effects of lighting on car appearance.

Meta-review: The paper introduces a large-scale, real-world 3D car dataset containing 2500 vehicles, each captured with around 200 RGB images and sparse LiDAR point clouds, using modern iPhones. The goal of the dataset is to advance research in vehicle reconstruction and modeling (e.g., NVS, part parsing, relighting, etc). The major strengths of the paper are the scale and diversity of the data. Comparing to existing real world dataset, it features more vehicles, more brands (vehicle types), and fine-grained part annotations. The data has the potential to improve existing perception systems, especially in corner cases. That being said, while the dataset itself is potentially interesting, the reviewers are also concerned about the core contributions of the paper --- the data curation pipeline is very standard and nothing innovative has been proposed, except applying it to more vehicles. Furthermore, while the authors did provide preliminary results on using the proposed data for certain downstream tasks, the experimental setups are not that comprehensive, casting doubts on the necessity of the dataset. The overall review of the paper is on the fence. While the reviewers appreciate the new data, they also point out that there are multiple additional experiments that can be done to make the paper more convincing and better justify the need of the data. After extensive discussion, the ACs decided to reject the paper. The authors are encouraged to re-submit the paper to a future venue.

Decision: Reject