3DRealCar: An In-the-wild RGB-D Car Dataset with 360-degree Views

Authors: Xiaobiao Du, Yida Wang, Haiyang Sun, Zhuojie Wu, Hongwei Sheng, Shuyun Wang, Jiaying Ying, Ming Lu, Tianqing Zhu, Kun Zhan, Xin Yu
Primary area: datasets and benchmarks
Keywords: 3D reconstruction, Car reconstruction, Car dataset

Abstract:
3D cars are commonly used in self-driving systems, virtual/augmented reality, and games. However, existing 3D car datasets are either synthetic or low-quality, presenting a significant gap toward the high-quality real-world 3D car datasets and limiting their applications in practical scenarios. In this paper, we propose the first large-scale 3D real car dataset, termed 3DRealCar, offering three distinctive features. (1) \textbf{High-Volume}: 2,500 cars are meticulously scanned by 3D scanners, obtaining car images and point clouds with real-world dimensions; (2) \textbf{High-Quality}: Each car is captured in an average of 200 dense, high-resolution 360-degree RGB-D views, enabling high-fidelity 3D reconstruction; (3) \textbf{High-Diversity}: The dataset contains various cars from over 100 brands, collected under three distinct lighting conditions, including reflective, standard, and dark. Additionally, we offer detailed car parsing maps for each instance to promote research in car parsing tasks. Moreover, we remove background point clouds and standardize the car orientation to a unified axis for the reconstruction only on cars without background and controllable rendering. We benchmark 3D reconstruction results with state-of-the-art methods across each lighting condition in 3DRealCar. Extensive experiments demonstrate that the standard lighting condition part of 3DRealCar can be used to produce a large number of high-quality 3D cars, improving various 2D and 3D tasks related to cars. Notably, our dataset brings insight into the fact that recent 3D reconstruction methods face challenges in reconstructing high-quality 3D cars under reflective and dark lighting conditions. 
\textcolor{red}{\href{https://3drealcar.github.io/}{Our dataset is available here.}}

Decision: reject
Reviewer scores: 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729021698283

thread_type:
official_review

<thread_object>
creator: reviewer_fi6m

<content>
summary:
The paper proposes 3DRealCar, a large-scale 3D real car dataset with RGB-D images and point clouds of 2,500 cars captured in real-world environments. It addresses the limitations of existing car datasets that usually use synthetic data or have low-quality data. This dataset includes cars under three lighting conditions (standard, reflective, and dark), promoting research in 3D car reconstruction, parsing, and novel view synthesis. Benchmarks with state-of-the-art methods demonstrate that existing methods struggle in reflective and dark lighting conditions, emphasizing the dataset’s value for improving 3D reconstruction methods.

strengths:
1. This paper addresses the limitations of current car datasets by capturing real-world car data with diverse samples, high-quality images, and point clouds.
2. The idea of introducing 3 different lighting conditions is interesting and makes up for the shortcomings of existing datasets
3. The extensive experiments demonstrate the effectiveness of this dataset on various tasks.

weaknesses:
1. Though collecting data is costly, the contribution of this work is only on the dataset and lacks technical contribution for 3D/2D car understanding. 
2. This dataset captures cars under three lighting conditions, including reflective, standard, and dark; however, it seems to lack data of the same car under all three conditions, which limits the exploration of the effects of lighting on car appearance.

questions:
Data of the same car under different lighting conditions could be added to enhance the analysis of lighting effects. Additionally, including metadata such as environmental maps or more detailed car materials could improve the dataset's versatility.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732628439

thread_type:
official_review

<thread_object>
creator: reviewer_tjmk

<content>
summary:
This paper introduces a comprehensive 3D car dataset featuring high-quality 360-degree scans. The dataset comprises 2,500 cars, each scanned with 3D scanners to produce 200 high-resolution RGB-D views. Additionally, the dataset includes variations across three different lighting conditions: reflective, standard, and dark.

The authors detail the data collection methodology extensively. Data capture involved the use of RGB-D sensors, followed by the recovery of camera poses using Colmap Structure from Motion (SfM), and mask extraction utilizing Grounding-DINO. The 3D models were then reconstructed with 3DGS.

The paper discusses extensive 2D and 3D downstream experiments conducted using the dataset, which include 2D detection and segmentation, depth estimation, point cloud completion, and 3D car generation. The experiment results demostrate the effectiveness of the collected data on various tasks.

strengths:
1. The structure of the paper is well-organized and the content is clearly presented;
2. The effectiveness of the dataset is demonstrated through its application to various downstream tasks, including various 2D and 3D tasks.
3. This paper introduces a novel 360-degree real car dataset. Previous literature has not extensively covered. Both the quality and the diversity of the dataset are commendable.

weaknesses:
1. The data collection and processing method is not novel and is a standard approach to reconstructing 3D assets.
2. The paper does not highlight its uniqueness and irreplaceability, especially in terms of improving 2D parsing and 2D detection performance.
3. The results of the NVS task (Fig.8 and 9) are considerably inferior to the state of the art.
4. Many previous papers, including CADSim (https://arxiv.org/pdf/2311.01447), GeoSim (Chen et al., CVPR'21), and other related works, have demonstrated the capability to reconstruct cars from real-captured data, which I believe is a more extensible way to reconstruct car assets. The authors claim that their data is of higher quality, but the paper does not demonstrate the necessity of such high quality, especially for downstream tasks.

Although the paper provides a very useful dataset and makes a significant contribution, the downstream tasks and data collection processes are not novel and do not meet the acceptance standards of ICLR.

questions:
See weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730819218350

thread_type:
official_review

<thread_object>
creator: reviewer_6ons

<content>
summary:
The paper presents a new dataset of 2.5k real cars scanned using modern
iPhones, resulting in high-resolution images as well as sparse LiDAR point
clouds (~200 views per car). This setting provides additional training and
benchmarking opportunities in novel view synthesis (NVS).

Key tasks which can be researched and benchmarked with this dataset include 3D
reconstruction, relighting, and parsing of car parts.

The dataset is further motivated with experiments demonstrating that using its
3D cars to perform image data augmentation can lead to perception improvements
in unusual scenarios. The dataset is also shown to help with part segmentation,
3D generative modeling, and 3D neural rendering.

One of my main questions regarding the paper, as elaborated in the Questions
Section, is whether any car is captured in more than one lighting condition. I
think this is a major research gap, so the presence of this data is pivotal in
assessing the impact of this dataset.

strengths:
- [S1] The dataset includes diverse car types, including sedans, sports cars,
  and small trucks, captured in high resolution, and annotated with part
  information (e.g., doors, hood, wheels). This has applications in benchmarking
  3D reconstruction and in developing simulators for autonomous driving.
- [S2] The experimental section analyses multiple tasks such as neural rendering
  and 3D generative modeling, as well as camera-based perception in corner-case
  scenarios. In the latter example, the authors demonstrate that synthesizing
  corner-case images (e.g., cars swerving off road) using their proposed 3D car
  assets can help improve real data perception on these scenarios.
- [S3] The dataset is already openly available online!

weaknesses:
- [W1] The LiDAR used is the iPhone 14 model, which is sparser than typical
  automotive LiDARs. While denser LiDAR can of course be re-simulated from dense
  3D models, this will require additional engineering effort while also
  suffering from some domain gap. This should be clarified in the intro: "3D
  scanner" makes me think of automotive or survey-grade LiDAR, not smartphone.
- [W2] Some minor suggestions for additional references and discussions: Please
  consider mentioning DeepMANTA [0] as related work since, while old, it also
  proposed similar part-level annotations.
- This isn't a weakness but as a tip: It can be helpful to provide some usage
  examples and tutorials for the dataset next to its SDK. For example "how to
  run gaussian splatting on 3DRealCar", since this can help more people learn to
  be familiar with the dataset, which can increase its impact.
- Other small suggestions:
  - For the "Stable Zero123" reference please surround the author name in curly
    braces in the .bib file, so the citation shows up as "Stability AI, 2023"
    instead of just "AI, 2023".
  - For those curious (such as myself!) it would be helpful to add some
    low-level details on the data collection in the appendix. For example, I
    have no idea how to get raw LiDAR points from an iPhone, so a brief
    discussion would be interesting.
  - L325: "the car is well-lighting" -> "the car is well-lit"
  - L417: "Dreamcract3D" contains a typo
- References:
  - [0]: Chabot, Florian, et al. "Deep manta: A coarse-to-fine many-task network
    for joint 2d and 3d vehicle analysis from monocular image." Proceedings of
    the IEEE conference on computer vision and pattern recognition. 2017.

questions:
- [Q1] Is the proposed dataset restricted to academia or is commercial use
  allowed?
- [Q2] I looked at the sample car reconstruction provided on the 3DRealCar
  dataset, and noticed that the COLMAP output mesh, =textured_output.obj=, is
  missing the upper half of the car. Is this expected?
- [Q3] Are any of the vehicles in the dataset captured in more than one
  environment? It is valuable to have the same vehicle captured in multiple
  environments, with varying lighting conditions, because it creates ground
  truth for relighting. Relighting is challenging and it is otherwise very
  difficult to get non-synthetic ground truth, so having some of the cars
  captured in 2+ lighting conditions would be valuable. However, I cannot tell
  whether this is the case in the proposed dataset.
- [Q3] Will the dataset also include pre-computed dense reconstructions like
  splats, or dense meshes extracted from a NeRF-like method? As mentioned above, it
  seems like the COLMAP mesh outputs can be incomplete.
- [Q4] On L346, "2D Car Parsing", why are car parsing maps given as input to
  segmentation? Or does "S" refer to just a binary mask of the vehicle? (Vehicle
  vs. background?)
</content>
</thread_object>

</thread>
<thread>
creation date:
1732066235446

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thanks for All Reviewers and AC

comment:
We appreciate the reviewers' insightful feedback. It is encouraging that all the reviewers recognize and agree that this paper presents a valuable 3D real car dataset including

**Community Contribution**:
- This dataset contains diversified car types (Reviewer 6ons, tJmk)
- This dataset provides detailed part-level annotations (Reviewer 6ons)
- The extensive experiments demonstrate the effectiveness of this dataset on various tasks.  (Reviewer 6ons,  tJmk, fi6m)
- This paper is well-organized and clearly presented  (Reviewer  tJmk)
- The proposed dataset is high-quality with comprehensive meta-data (Reviewer fi6m)
- The proposed dataset contains different lighting conditions to make up for the shortcomings of existing datasets (Reviewer fi6m)

**Since we are continually extending the scale of our dataset, we have collected more than 3000 cars in our dataset.  We will select 2500 high-quality data to release to the public.
We confirm our dataset and all codes will be released to the public to facilitate the development of the community.**
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a large-scale, real-world 3D car dataset containing 2500 vehicles, each captured with around 200 RGB images and sparse LiDAR point clouds, using modern iPhones. The goal of the dataset is to advance research in vehicle reconstruction and modeling (e.g., NVS, part parsing, relighting, etc). The major strengths of the paper are the scale and diversity of the data. Comparing to existing real world dataset, it features more vehicles, more brands (vehicle types), and fine-grained part annotations. The data has the potential to improve existing perception systems, especially in corner cases. That being said, while the dataset itself is potentially interesting, the reviewers are also concerned about the core contributions of the paper --- the data curation pipeline is very standard and nothing innovative has been proposed, except applying it to more vehicles. Furthermore, while the authors did provide preliminary results on using the proposed data for certain downstream tasks, the experimental setups are not that comprehensive, casting doubts on the necessity of the dataset. The overall review of the paper is on the fence. While the reviewers appreciate the new data, they also point out that there are multiple additional experiments that can be done to make the paper more convincing and better justify the need of the data. After extensive discussion, the ACs decided to reject the paper. The authors are encouraged to re-submit the paper to a future venue.