RGM: Reconstructing High-fidelity 3D Car Assets with Relightable 3D-GS Generative Model from a Single Image

## Abstract

Abstract The generation of high-quality 3D car assets is essential for various applications, including video games, autonomous driving, and virtual reality. Current 3D generation methods utilizing NeRF or 3D-GS as representations for 3D objects, generate a Lambertian object under fixed lighting and lack separated modelings for material and global illumination. As a result, the generated assets are unsuitable for relighting under varying lighting conditions, limiting their applicability in downstream tasks. To address this challenge, we propose a novel relightable 3D object generative framework that automates the creation of 3D car assets, enabling the swift and accurate reconstruction of a vehicle’s geometry, texture, and material properties from a single input image. Our approach begins with introducing a large-scale synthetic car dataset comprising over 1,000 high-precision 3D vehicle models. We represent 3D objects using global illumination and relightable 3D Gaussian primitives integrating with BRDF parameters. Building on this representation, we introduce a feed-forward model that takes images as input and outputs both relightable 3D Gaussians and global illumination parameters. Experimental results demonstrate that our method produces photorealistic 3D car assets that can be seamlessly integrated into road scenes with different illuminations, which offers substantial practical benefits for industrial applications.

## 1 Introduction

Generating high-quality 3D car assets plays a vital role in various applications like digital city, autonomous driving, and virtual reality, among others. Traditionally, the 3D modeling of vehicle assets is hand-crafted by skilled artists using Computer-Aided Design (CAD) software. The modeled cars are then integrated into rendering engines such as Blender Blender () or Unreal Engine for photorealistic rendering. However, establishing such a 3D asset can be a cumbersome task, especially when there is a need to model 3D assets based on images of real-world cars. In such cases, the modeling artists are required to manually design and adjust the shape, texture and material of cars in detail to match the reference image accurately. To alleviate the burden on artists and facilitate complex car modeling, in this work, we focus on generating 3D car assets automatically with a data-driven method. Specifically, given an input image of a random 3D car, we aim to swiftly and precisely reconstruct its 3D geometry, texture, and material like roughness and metallic. This allows applications like relighting and holds substantial promise for industrial design.

In recent years, there has been a revolution in 3D generation with the rise of diffusion models. Some works Poole et al. (); Lin et al. (); Ornek et al. () integrate NeRF Mildenhall et al. () or 3D GS Kerbl et al. () with diffusion models to generate 3D objects using text prompts. However, these optimized-based methods typically face challenges like slow generation speed and unstable training processes. Other methods Shi et al. (); Liu et al. (; ) extend 2D diffusion models to generate consistent multi-view images by introducing viewpoint awareness. For example, Wonder3dLong et al. () generates consistent multi-view RGB images with corresponding normal maps via a cross-domain diffusion model, with the input of a reference image and camera parameters. However, existing multi-view image generation models only map a single image to multiple views, necessitating supplementary modules like NeuSWang et al. () to reconstruct the 3D shape and texture. These object-specific models often demand a lot of training time, ranging from minutes to hours. Recently, several works Tang et al. (); Xu et al. (); Tochilkin et al. (); Hong et al. () have leveraged transformer architecture for a feed-forward 3D generation that takes images as input and outputs 3D representation like NeRF or 3D-GS. These methods offer rapid 3D model generation through fast feed-forward inference and provide more precise control over the generated outputs.

Despite these advancements, the 3D generation of vehicles still struggles with issues such as insufficient detail and inconsistent geometry. These limitations are primarily due to the lack of large-scale car datasets that could support the learning of generalizable priors for vehicles. Besides, current 3D generation methods primarily focus on the geometry and texture of objects, often neglecting material modeling. In these methods, objects’ shading colors are typically baked into the surface, showing only the lighting effect under specific illumination. As a result, the generated objects are unsuitable for relighting with different illumination and require manual material assignment for downstream applications, thereby limiting their practicality for industrial 3D asset generation.

To address the aforementioned problems, we propose a large-scale synthetic car dataset and a novel relightable 3D object generation pipeline, offering efficient solutions for rapid and precise 3D car asset creation. First, we collect and construct a synthesized vehicle dataset named Carverse, which contains over 1,000 high-precision 3D vehicle models. These models are rendered with different materials and under different lighting to obtain multi-view images and camera poses. With the dataset, we introduce a novel relightable 3D-GS generative model (RGM) that: (1) We model a 3D scene with global illumination and a set of relightable 3D Gaussians, where each Gaussian primitive is associated with a normal direction and BRDF parameters including albedo, roughness, and metallic. (2) we adopt a U-Net Transformer as an image-conditioned model, which outputs both the relightable 3D-GS and global illumination parameters. With this pipeline, we successfully reconstruct the geometry, appearance, and material of a 3D car from an image, enabling the realistic insertion of the car into different scenes. We further present an autonomous driving scene simulation pipeline to demonstrate the practicality of RGM.

Fig. 1 provides qualitative results of the reconstructed cars with normal and material maps. Experiments demonstrate that our method outperforms previous approaches both quantitatively and qualitatively. Additionally, we showcase the effectiveness of relighting and integrating the generated car assets into road scenes. The results achieve photo-realistic performance, providing significant benefits for a wide range of AR/VR applications.

## 2 Related Works

### 2.1 Generative Model

The goal of generative model is to produce new samples with similar statistical characteristics to the training data by learning the underlying distribution. In recent years, there has been a revolution in generative model, with groundbreaking networks named diffusion models Sohl-Dickstein et al. (). Diffusion models have demonstrated significant success in image synthesis Rombach et al. (); Zhang et al. (); Ye et al. (); Podell et al. () by generating new samples through the gradual denoising of a normally distributed variable. They have also inspired numerous works to generate 3D assets. Many works Poole et al. (); Lin et al. (); Tang et al. (); Ornek et al. () integrate NeRF Mildenhall et al. () or 3D Gaussian Kerbl et al. () with diffusion models to generate 3D objects using text prompts. Additionally, several works Shi et al. (); Tang et al. (); Liu et al. (; ) extend 2D diffusion models to generate consistent multi-view images by introducing viewpoint awareness. Moreover, some works Richardson et al. (); Chen et al. (); Youwang et al. (); Zeng () focus on generating texture for 3D assets by extending text-conditional generative models. However, these works typically faces challenges like slow generation speed and unstable optimization process. Recently, servral works Tang et al. (); Xu et al. (); Tochilkin et al. (); Hong et al. () utilize transformer architectures for feed-forward processes, converting images into 3D representations such as NeRF or 3D-GS . These approaches enable rapid 3D model generation through efficient feed-forward inference. Nevertheless, these methods primarily concentrate on reconstructing shape and texture, which restricts their utility in practical industrial applications. In this study, we investigate a fast 3D generation pipeline capable of producing high-quality 3D car assets as well as material properties.

### 2.2 3D Gaussian Splatting

3D Gaussian splatting (3DGS) employs 3D Gaussian primitives to represent the 3D scene and alpha blending to render the image. In comparison to the NeRF, the 3DGS method exhibits superior rendering quality and speed. A number of previous studies have addressed the misrendering issues caused by viewpoint changes during 3DGS rendering. Mip-splattingYu et al. () and SA-GSSong et al. () address the frequency inconsistency of Gaussian primitives when the camera is zoom-in or zoom-out by employing different 3D or 2D filters. Other works enhance the rendering quality by modifying the form of Gaussian primitives. For example, 2DGSHuang et al. () compresses Gaussian primitives into 2D facets, enhancing the Gaussian geometry and facilitating superior normal vector rendering. 3d-HGSLi et al. () disassembles the Gaussian into a half-Gaussian, enhancing its scene representation. Furthermore, certain worksLu et al. (); Ren et al. (); Yan et al. (); Yang et al. () endeavor to restrict the use of GS to achieve more compact scene structures. Besides, some works have employed various deep learning modules (e.g., GS in the DarkYe et al. (), Deblur-GSChen & Liu ()) to address the issue of noisy images. Others have utilized GS to model large-scale streetscapes (e.g., Street GSYan et al. (), AutosplatKhan et al. ()). Moreover, some works employ the modeling of physical properties in 3DGS through the representation of BRDF functions, as exemplified by Relightable GSGao et al. () and 3D Gaussian Splatting with Deferred ReflectionYe et al. (). In this study, we also use 3DGS to model a 3D car and integrate it with physical attributes to model the vehicle under arbitrary lighting conditions and to enable light editing of the vehicle.

### 2.3 Vehicle Modelling and Datasets

To facilitate the development of autonomous driving and vehicle modeling, researchers have created a series of datasets. Real-world driving datasets like KITTIGeiger et al. (), Waymo, and nuScenesCaesar et al. (), offer comprehensive sensor data that are essential for capturing realistic driving scenarios. These datasets focus on tasks such as object detection, tracking, and trajectory prediction. However, their sparse viewpoints make them inadequate for vehicle reconstruction. Simulation datasets from platforms like CARLADosovitskiy et al. () allow researchers to create diverse and controlled driving environments; nevertheless, the vehicles in these datasets tend to be of low quality, rendering them unsuitable for realistic downstream applications. SRN-CarChang et al. () and ObjaverseDeitke et al. () compile 3D car models from various repositories and online resources, but their quality is insufficient to meet the demands of real vehicles for autonomous driving. MVMCZhang et al. (), collected from car advertisement websites, is too sparse to generate high-quality 3D car models. The dataset CarStudioLiu et al. () provides images from 206,403 car instances, while the 3DRealCar datasetSong et al. () includes multi-view images of 2,500 car instances derived from real vehicle data. However, users do not have access to additional attribute information, such as normals, metallic properties, and roughness. In this paper, we introduce a novel vehicle dataset named Carverse, which not only presents high-quality images from a diverse set of cars but also includes geometry and material information, facilitating a deeper understanding of vehicles. Tab. 1 shows statistic comparisons with previous datasets.

**Table 1: Statistic comparison with concurrent vehicle datasets.**
| Dataset | Instances | Views | Quality | Diversity | Normal | Material |
| --- | --- | --- | --- | --- | --- | --- |
| SRN-Car Chang et al. () | 2,151 | 250 | Low | Medium | ✗ | ✗ |
| MVMC Zhang et al. () | 576 | $\sim$10 | High | Low | ✗ | ✗ |
| Objvaverse Deitke et al. () | 511 | - | Medium | Low | ✗ | ✗ |
| CarStudio Liu et al. () | 206,403 | 1 | High | High | ✗ | ✗ |
| 3DRealCar Jiang et al. () | 2,500 | $\sim$200 | High | High | ✗ | ✗ |
| Carverse | 1,006 $\times$ 5 | 100 | High | High | ✔ | ✔ |

## 3 Method

In this section, we present our relightable car assets generation framework, shown in Fig. 3, which reconstructs a car’s geometry, materials, and illumination from images under unknown lighting. We first introduce a novel large-scale synthetic car dataset (Sec. 3.1). Then, we present relightable Gaussian primitives to represent 3D objects with material properties (Sec. 3.2) and propose a relightable 3D-GS generative model that infers from images and output the relightable Gaussians (Sec. 3.3). Finally, we show an autonomous driving simulation application in Sec. 3.4 to integrate the generated car into road scenes and render photorealistic images.

### 3.1 Carverse Dataset Construction

As a crucial component of autonomous driving simulation, the manual creation of vehicle models by artists is an exceptionally time-consuming process. In this paper, we present a data-driven pipeline for automatic vehicle generation, which necessitates a large-scale, high-quality vehicle dataset. Existing datasets are often limited by low quality and a restricted variety of vehicles. To overcome these challenges, we introduce Carverse, a high-quality synthetic dataset designed to support research in vehicle reconstruction and related tasks. Carverse is constructed using Blender’s physics engine Blender () and the ray-tracing Blender Cycles rendering engine, enabling the generation of photorealistic images with diverse materials and global illumination settings.

To develop a comprehensive and realistic dataset, we sourced over 1,000 high-quality 3D vehicle models and 3,000 distinct HDR maps from online repositories. The dataset includes a wide range of vehicle types, such as sedans, sports cars, vans, and trucks. The 3D vehicle models were divided into training and testing sets with 1,006 as the train set and 50 as the test set. To ensure dataset diversity, each model was subjected to random z-axis rotations and minor flipping along the XY plane prior to rendering. Additionally, HDR maps, vehicle colors, and materials were randomly varied to simulate different illumination and surface properties. Each vehicle in the training set was rendered 5 times with randomly assigned textures and lighting conditions, yielding a total of 5,030 samples across the 1,006 vehicles. For each sample, we captured ground truth data from 100 distinct camera poses, including monocular RGB renderings, camera pose information, normal maps, and material maps such as albedo, roughness, and metallic. In the test set, 50 vehicle samples were rendered, with one image used as input and 30 random views for benchmarking, along with a mesh file for geometry evaluation. Fig. 2
illustrates the dataset construction pipeline and showcases sample diversity.

Figure: Figure 2: The pipeline for constructing Carverse dataset.
Refer to caption: x2.png

### 3.2 Relightable 3D Gaussian

We aim to efficiently generate 3D objects with material properties from a single image, allowing seamless application to downstream tasks. Recent advancements in 3D object generation have leveraged transformer architectures for rapid feed-forward 3D generation, processing images as inputs and producing 3D representations such as 3D-GS. Specifically, in the representation of 3D-GS, each 3D Gaussian primitive $\rm G_{i}(x)$ is mathematically defined as:

$$ $\rm G_{i}(x)=e^{-\frac{1}{2}(x-\mu_{i})^{T}{\Sigma_{i}}^{-1}(x-\mu_{i})},$ (1) $$

$\rm x\in\mathbb{R}^{3}$ is a random 3D coordinate, $\rm\mu_{i}\in\mathbb{R}^{3}$ stands for the mean vector of the Gaussian, and $\rm\Sigma_{i}\in\mathbb{R}^{3\times 3}$ is the covariance matrix. To render the Gaussians into the image space, each pixel $\rm p$ is colored by $\rm\alpha$-blending $\rm K$ sorted Gaussians overlapping $\rm p$ as:

$$ $\rm C(p)=\sum_{i=1}^{K}c_{i}\alpha_{i}\prod_{j=1}^{i-1}\left(1-\alpha_{j} \right),$ (2) $$

where $\rm\alpha_{j}$ is derived by projecting the 3D Gaussian $\rm G_{i}$ onto pixel $\rm p$ and multiplying by its opacity, and $\rm c_{i}$ denotes the color of $\rm G_{i}$. Using a differentiable tile-based rasterizer Kerbl et al. (), all Gaussian attributes can be optimized in an end-to-end manner.

However, limited by the representation of 3D-GS, existing feed-forward generative models primarily focus on geometry and texture, overlooking material properties and lighting effects, which typically require manual adjustment before the generated objects can be applied to downstream applications. To address this, we extend the representation of 3D-GS by modeling a 3D object with global illumination and a set of relightable 3D Gaussians, where each Gaussian primitive is associated with a normal direction and BRDF parameters. This enriched representation facilitates the generating of material properties and supports object editing tasks such as relighting.

To model the shading color of objects under different lighting conditions, we use the physically-based rendering equation to compute the outgoing radiance $\rm L_{o}$ at an intersected surface point $\rm x$:

$$ $\displaystyle\rm L_{o}(x,\omega_{o})=\int_{\Omega^{+}}f_{r}(x,\omega_{i}, \omega_{o})L_{i}(x,\omega_{i})cos\theta d\omega_{i},$ (3) $$

where $\rm f_{r}$ is the BRDF function, $\Omega^{+}$ is the positive hemisphere determined by the surface normal n at point x, $\omega_{o}$ is the direction of the outgoing ray, $\omega_{i}$ represents all possible incident ray directions in $\Omega^{+}$, $\rm L_{i}$ is the incident radiance from direction $\rm\omega_{i}$, and $\theta$ is the angle between $\rm\omega_{i}$ and $\rm n$. By integrating Eq. 3 with 3D-GS, we present the representation of relightable Gaussian primitives as follows.

Normal modelling. Surface normals are critical in rendering equations to determine the shading of a surface point. Though we can compute the pseudo normal $\rm\widetilde{N}$ with the rendered depth map D of 3D-GS, it is not differentiable and is usually imprecise. To accurately model the geometry of 3D objects, we first enhance 3D-GS representation by integrating it with a learnable normal direction. Specifically, for each 3D Gaussian primitive $\rm G_{i}$, we introduce an additional 3D unit vector $\rm n_{i}\in\mathbb{S}^{2}$. Then we can obtain a rendered normal map N through $\rm\alpha$-blending as:

$$ $\rm N(p)=\sum_{i=1}^{K}n_{i}\alpha_{i}\prod_{j=1}^{i-1}\left(1-\alpha_{j} \right).$ (4) $$

We use the pseudo normal map $\rm\widetilde{N}$ as supervision to guide learning of the normal attributes, with a consistency loss formulated as:

$$ $\rm\mathcal{L}_{consistency}=\|N-\widetilde{N}\|_{2}^{2}.$ (5) $$

Material modelling. We employ a simplified Disney BRDF function $f_{r}$ Burley & Studios () to capture the effects of spatially varying materials on light reflection. The BRDF describes the relationship between incoming and outgoing light at a surface point, illustrating how material properties influence the reflection. Specifically, in the BRDF formulation $\rm f_{r}(\rm x,\omega_{i},\omega_{o};n,b,r,m)$, the variables are defined as follows: $\rm n$ represent the surface point’s normal, while $\rm b\in[0,1]^{3}$, $\rm r\in[0,1]$, and $\rm m\in[0,1]$ represent diffuse albedo, and roughness and metallic parameter respectively. For each Gaussian primitive $\rm G_{i}(x)$, we associate these parameters to model the material properties, therefore the variable for a relightable Gaussian includes $\rm G_{i}=\rm\{\mu_{i},\Sigma_{i},c_{i},\alpha_{i},n_{i},b_{i},r_{i},m_{i}\}$. These attributes can be rendered into corresponding attribute maps (e.g., albedo map B, roughness map R and metallic map M) using the differentiable tile-based rasterizer. Consequently, they can be optimized through a $\rm L_{2}$ loss $\rm L_{material}$ with the ground truth maps.

Illumination modelling. To enable the insertion of objects into scenes with various illumination, it is essential to utilize a flexible light source representation that adapts to different lighting and supports realistic rendering. In this work, we use a spherical harmonic illumination model to approximate the global illumination around a 3D object. Specifically, for each scene containing a car, we represent the lighting parameters using 3rd-order spherical harmonic coefficients, denoted as $\rm L\in\mathbb{R}^{16\times 3}$, the incident radiance $\rm L_{i}$ in Eq. 3 is defined as:

$$ $\rm{L}_{i}(x,\omega_{i})={L}_{i}(\omega_{i})=L\cdot SH(\omega_{i}),$ (6) $$

where $\rm SH(\cdot)$ represents the spherical harmonic basis functions.

Physically-based rendering. Given the material attributes, the shading color for each Gaussian primitive can be computed. Since the definite integral in Eq. 3 is analytically intractable, we approximate it using numerical integration. For each Gaussian primitive G, we generate $\rm M$ incident ray directions through Fibonacci sphere sampling based on the normal direction. Since the scene contains only a single object and the geometry of a vehicle can be approximated as convex, we assume full visibility of the object to the environment lighting, setting the visibility factor for these ray directions to 1. Consequently, the physically-based shading color $\rm\widetilde{c}_{i}$ of a Gaussian primitive $\rm G_{i}$ for outgoing direction $\omega_{o}$ can be formulated as:

$$ $\displaystyle\rm\widetilde{c}_{i}(\omega_{o})=\frac{1}{M}\sum_{j=1}^{M}f_{r}( \mu_{i},\omega_{j},\omega_{o};n_{i},b_{i},r_{i},m_{i})L_{i}(\omega_{j})\cos\theta,$ (7) $$

The physically-based rendered image $\rm\widetilde{C}$ is then obtained using the differentiable tile-based rasterizer.

Figure: Figure 3: Overall architecture of RGM. We first input a single image into the multi-view generation model to obtain multi-view consistent images of the car. These images and camera embeddings are then fed into the relightable 3D-GS generative model, which produces both global illumination parameters and relightable Gaussian representations. Through a physically-based rendering layer, we can obtain the material properties of the car and relight the car with a new illumination.
Refer to caption: x3.png

### 3.3 Relightable 3D-GS Generative Model

Given a single input image $\rm I$, we initially process it through a multi-view generation model Wonder3d Long et al. () finetuned on Carverse to produce images from six predefined viewpoints, covering the front, back, left, right, left-front, and right-front perspectives of the vehicle. With these multi-view images $\rm I_{m}$ and corresponding camera poses $\rm\pi_{m}$ as input, we then exploit the interpolation capabilities of the transformer architecture and introduce an asymmetric U-Net Transformer $\rm g_{\theta}$ as the Relightable 3D-GS Generative Model (RGM), to estimate the corresponding relightable GS and global illumination parameters. This can be mathematically expressed as:

$$ $\displaystyle\rm G,L$ $\displaystyle=\rm g_{\theta}(F),$ (8) $\displaystyle\rm C,\widetilde{C},P$ $\displaystyle=Renderer{\rm(G,L,\pi)},$ (9) $$

$\rm G=\{G_{0},G_{1},...G_{K}\}$ is a set of pixel-aligned relightable Gaussian primitives, $\rm L$ denotes the SH coefficients, and $\rm F=\{f_{0},f_{1},f_{2},...\}$ is the embeddings of input images and camera poses. $Renderer(\cdot)$ is the physically-based rendering function and differentiable rasterizer, $\rm P=\{N,B,R,M\}$ are the rendered material maps under camera pose $\pi$. Following LGM Tang et al. (), we utilize the Plücker ray embedding to densely encode the camera poses. The RGB values and ray embeddings are concatenated into a 9-channel feature map, which serves as the input to RGM:

$$ $\rm f_{i}=\{v_{i},o_{i}\times d_{i},d_{i}\},$ (10) $$

where $\rm f_{i}$ is the feature for pixel $\rm i$, $\rm v_{i}$ is the RGB value, $\rm d_{i}$ is the ray direction, and $\rm o_{i}$ is the ray origin.

The architecture of $\rm g_{\theta}$ is a hybrid model that integrates the hierarchical structure of U-Net for local feature extraction with self-attention mechanisms to capture global dependencies. It consists of three components: (1) Down Blocks to extract features from the input images while progressively reducing spatial resolution and increasing channel depth; (2) Mid Blocks to refine the features at the lowest resolution; and (3) Up Blocks to upsample the features to higher spatial resolutions, with skip connection that concatenate the upsampled features with corresponding features from the Down Blocks, allowing the network to retain fine-grained details. With the pixel-aligned features from the last U-Net layer, we then decode them through two prediction heads, producing spherical harmonics and relightable Gaussian parameters, respectively.

During training, we optimize RGM using a loss function that accounts for both rendered images and material properties. Denoting the ground-truth images as $\rm\hat{C}$, the total loss is formulated as:

$$ $\displaystyle\rm\mathcal{L}_{total}=\lambda_{1}\|C-\hat{C}\|_{2}^{2}+\lambda_{ 2}\|\widetilde{C}-\hat{C}\|_{2}^{2}+\lambda_{3}\mathcal{L}_{material}+\lambda_ {4}\mathcal{L}_{consistency}+\lambda_{5}\mathcal{L}_{smooth},$ (11) $$

where $\rm\mathcal{L}_{smooth}$ is a bilateral smooth loss for materials and $\lambda_{i}$ is the loss weight for each term.

### 3.4 Applications for Autonomous Driving Simulation

To demonstrate the practicality of RGM, we further present a 3D-GS-based pipeline for autonomous driving scene simulation that seamlessly integrates a vehicle into a dynamic road scene with photorealistic rendering. Given a set of posed images of the scene, we first employ a dynamic 3D-GS Chen et al. () to reconstruct the road scenes. Subsequently, we adopt DiffusionLight Phongthawee et al. (), a diffusion-based lighting estimation module, to generate the environment map from an image of the scene. Using an input image of the vehicle, RGM infers and generates the vehicle’s Relightable GS, which is subsequently relighted using the environment map. The relighted GS is then spatially aligned to place the vehicle on the road, with its trajectory defined. Finally, the relighted GS is integrated into the road scene’s dynamic 3D-GS, enabling the realistic rendering of vehicles within various road environments.

## 4 Experiments

Figure: Figure 4: Qualitative results of generated novel views, compared with TripoSR Tochilkin et al. (),InstantMesh Xu et al. () and Real3D Jiang et al. ().
Refer to caption: x4.png

**Table 2: Comparison with state-of-the-art methods on Carverse test set.**
| Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Chamfer Dist. $\downarrow$ | F-score $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| LGM Tang et al. () | 14.95 | 0.843 | 0.312 | 0.171 | 0.183 |
| TripoSR Tochilkin et al. () | 14.60 | 0.826 | 0.230 | 0.073 | 0.201 |
| InstantMesh Xu et al. () | 14.27 | 0.827 | 0.243 | 0.038 | 0.404 |
| Real3D Jiang et al. () | 14.57 | 0.823 | 0.224 | 0.079 | 0.165 |
| Ours | 20.16 | 0.871 | 0.094 | 0.031 | 0.512 |

### 4.1 Comparison with SOTA methods

We benchmark our approach on the Carverse dataset using 50 car samples for testing. Each test sample includes a single input image, along with a corresponding point cloud and 30 images from random views for evaluation. For the quantitative assessment of the generated images, we apply standard image quality metrics, including PSNR, SSIM, and LPIPS. To evaluate the accuracy of the reconstructed 3D shapes, we use Chamfer Distance (cm) and F-score (w/ threshold of 0.01m) as the evaluation metrics. Several state-of-the-art feed-forward image-to-3D methods Tochilkin et al. (); Xu et al. (); Tang et al. (); Jiang et al. () are selected for comparison. We register the generated meshes of these methods with ground truth mesh for rendered image evaluation. The quantitative results, presented in Tab. 2, show that our method significantly outperforms the others in generating car assets with accurate geometry and realistic textures from a single RGB image. Qualitative comparison examples are provided in Fig. 4, where our method demonstrates superior consistency and texture realism, largely due to the large-scale dataset that provides a robust foundation for generating high-quality 3D car assets. Moreover, RGM can generate the normal and material maps of cars as illustrated in Fig. 1, while other methods are limited to texture generation.

### 4.2 Ablation Studies

Supervision for Materials. We also conduct an ablation study to evaluate the impact of material supervision by removing the loss term $\rm\mathcal{L}_{material}$. The results, presented in Tab. 3, demonstrate a slight performance drop, but the model still achieves good results overall. This indicates that RGM can also be effectively trained on datasets lacking ground truth material information.

**Table 3: Ablation studies on material supervision.**
| Method | PSNR $\uparrow$ | SSIM $\uparrow$ | LPIPS $\downarrow$ | Chamfer Dist. $\downarrow$ | F-score $\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| w/o $\rm\mathcal{L}_{material}$ | 19.85 | 0.866 | 0.123 | 0.054 | 0.352 |
| Ours | 20.16 | 0.871 | 0.094 | 0.031 | 0.512 |

Figure: Figure 5: Visualization of gaussian points with or without relightable gaussians.
Refer to caption: x5.png

### 4.3 Applications

Figure: Figure 6: Qualitative results for applications.
Refer to caption: x6.png

Applications for Autonomous Driving Simulation. We present the results of the autonomous driving simulation detailed in Sec. 3.4. As illustrated in Fig. 6, the system successfully achieves photorealistic rendering, seamlessly inserting the given vehicle into the road scene and the appearance of the car matches the lighting naturally.

Real-world Applications. To further validate the effectiveness of RGM, we also applied it to in-the-wild images of real-world cars. The qualitative novel view images are given in Fig. 6. This shows the capability of RGM to transform a random car on the road into a 3D digital asset.

## 5 Conclusion

In conclusion, in this paper we successfully tackle the challenges in 3D vehicle generation, such as limited datasets and lack of modeling of material properties for relighting. By introducing the Carverse dataset, containing over 1,000 high-precision 3D vehicle models, we support robust learning and generalization. Additionally, our novel relightable 3D-GS generation framework named RGM enables the efficient creation of detailed 3D car models by modeling global illumination and representing 3D scenes through Gaussian primitives that capture geometry, appearance, and material properties. This approach allows for realistic relighting under different illumination conditions, advancing 3D asset generation for AR/VR applications and industrial design by improving both speed and quality in automatic 3D car modeling from a single image.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 3
presentation: 2
questions: Q1. You mention that RGM supports relighting through spherical harmonics. How flexible is the lighting adjustment when integrating the 3D models into new scenarios? For example a scene with much dimmer environmental lighting. So far I only saw bright daylight insertion in the paper. 

Q2. How does RGM handle complex lighting environments like dynamic or moving light sources?

Q3. In Table 1, terms like "high" and "low" are used to describe dataset quality and diversity. Can you clarify how these metrics are quantitatively defined? What specific attributes (maybe resolution, texture detail, polygon density) are used to distinguish between "high" and "low"?

Q4. When conducting comparison experiments with other SOTA models, were those models trained or fine-tuned on Carverse before testing? If not, and only RGM has prior knowledge of the distribution of the Carverse data, how do you address the potential bias in comparing RGM's performance to SOTA methods that were not trained on the Carverse dataset?
soundness: 1
strengths: The relightable 3DGS car model is interesting and worth exploring, The RGM method enables the creation of high-quality 3D car models from a single image, with realistic relighting features, addressing key shortcomings in existing 3D generation techniques.

The Carverse dataset offers a large-scale collection of 3D vehicle models, and what sets it apart is the inclusion of detailed material and normal information. This gives it a unique edge compared to older datasets, making it useful for robust training and improving the realism of generated 3D assets.

If the relighting capability works as well as claimed, this paper shows strong potential for practical applications, especially in fields like autonomous driving,  where realistic lighting is crucial.
summary: The main contribution of the paper is the introduction of Relightable 3D-GS Generative Model (RGM), a novel framework for automatically generating high-fidelity 3D car assets from a single image. RGM addresses the limitations of existing models by enabling relighting under various lighting conditions. It achieves this through the use of 3D Gaussian primitives integrated with BRDF material properties and global illumination modeling. Additionally, the authors introduce the Carverse dataset, a large-scale synthetic collection of over 1,000 detailed 3D vehicle models, to train and benchmark their approach.
weaknesses: For table 1, the use of terms like 'high' and 'low' for the quality and diversity metrics is very ambiguous and biased, especially when many of the datasets being compared are real-world datasets. From the images presented in the paper, I don't observe a significant quality improvement solely based on these visuals. However, I do recognize the added value that Carverse provides through its inclusion of normal and material information, which may not be available in the other datasets mentioned. That said, when comparing diversity and quality, datasets such as CarStudio visually appear much more diverse and of higher quality to me. Therefore, using 'high' to describe both Carverse and CarStudio can mislead readers into thinking their visual appearances in terms of diversity and quality is comparable, which doesn’t seem accurate. If we further consider the difference in instance numbers, CarStudio’s over 200k instances are much denser than the 1,000 instances introduced by Carverse.

The experimental section appears biased. If my understanding is correct, when the authors compared the RGM model’s ability to generate car models with other SOTA models, they used 50 car samples from their own Carverse dataset. However, only the RGM model was trained on the Carverse dataset, while the other models had no prior exposure to this dataset during training. This creates an inherent advantage for the RGM model, as it has a better understanding of the Carverse data distribution, which would inevitably lead to better quality metric scores compared to the other models.

The authors claim that the quality of existing synthetic car datasets is insufficient for modern real car rendering, but from visual inspection, I don’t see a significant quality improvement in Carverse compared to other synthetic datasets like Objaverse. In fact, in some instances, Objaverse’s car models even appear to have better visual quality. To truly demonstrate that Carverse is superior to previous datasets, a more convincing A/B test with identical conditions under RGM would be necessary.

I would also like to see more examples of scene insertion results to better assess how well the RGM model handles relighting. In Figure 6, both scenes shown are under bright daylight conditions. It would be more convincing to see examples in dimmer lighting environments to showcase how the relighting function performs under different lighting scenarios.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The results of both the generated novel view images and the final 3D-GS rendered results are of relatively low resolution. It is recommended that the authors could provide some high-resolution results or zoomed-in images of specific areas to better illustrate the effects in detailed regions.
2. It is unclear whether the compared methods were trained on the same dataset. The authors could clarify the training data used for each compared method in their experimental setup, and if different, discuss how this might impact the fairness of the comparisons.
soundness: 3
strengths: 1. Clear Writing: The paper is well-structured and clearly written, with logical flow and coherent narrative, making the motivation, methodology, and experiments easy to follow.
2. Contribution of Dataset Construction: The authors constructed a large-scale, high-quality synthetic dataset, Carverse, with over 1,000 3D vehicle models and extensive multi-view and material attribute data. This dataset serves as a valuable foundation for 3D car model generation and advances the field by supporting further research.
summary: This paper introduces RGM (Relightable 3D-GS Generative Model), a framework designed to generate high-fidelity, relightable 3D car models from a single image. The study includes the development of Carverse, a synthetic dataset with over 1,000 3D car models, providing robust training data. The RGM framework combines 3D Gaussian splatting with physically-based rendering, enabling realistic, material-consistent models adaptable to various lighting scenarios. Experimental results show that RGM surpasses previous methods in terms of image quality, geometric accuracy, and material consistency, with potential applications in autonomous driving simulation.
weaknesses: 1. Lack of Real Data Testing: The experiments were conducted mostly on synthetic data, the paper only presents two qualitative results on real data in section 4.3. The authors could conduct more experiments on real data, especially providing some quantitative results.
2. Insufficient Analysis on Viewpoint Consistency: The paper lacks an analysis of material consistency between generated multi-view images and the input image. Without ensuring consistency in material properties across views, the relightable 3D-GS model may have limited relevance, potentially restricting its applicability.
3. Moderate Novelty: The design of the relightable 3D Gaussian splatting model and the generative model shows limited innovation compared to existing methods. While performance improvements are achieved, the methodological design lacks significant originality.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: How to obtain the corresponding camera poses of the multi-view images?
Does this work need the camera pose of the input image and how to obtain it?
soundness: 3
strengths: This paper presents a novel approach for generating high-quality 3D car assets with material properties from a single image, demonstrating promising results.
summary: This paper presents a novel approach for generating high-quality 3D car assets with material properties from a single image, demonstrating promising results. However, there are a few aspects that could be improved for further advancement.
weaknesses: Dataset Generalization: The method heavily relies on the Carverse dataset, which may limit its applicability to other types of 3D objects. Exploring data augmentation techniques or constructing additional datasets for different object categories would enhance the model’s generalization capabilities.
Model Efficiency: The current model, based on U-Net Transformer, incurs high training costs. Investigating more lightweight architectures or efficient training strategies could improve the model’s practicality.
Detail Generation: While the method achieves impressive results for overall car structures, the generation of fine details remains limited. Incorporating techniques to better capture intricate features would be beneficial.
Real-time Rendering: The differentiable rasterizer employed in the model compromises real-time performance. Exploring more efficient rendering methods would enable the model to be utilized in real-time applications.
Model Interpretability: The black-box nature of the model hinders understanding its inner workings. Investigating methods to enhance model interpretability would provide valuable insights into the generation process and potential limitations.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: The result quality and the limited contribution should be clearly addressed.
soundness: 3
strengths: 1. The large-scale car dataset is a good contribution.
2. The paper is well organized and easy to read.
summary: The paper addresses the task of relightable object generation, overcoming the previous technical challenge of generating only Lambertian objects under fixed lighting conditions. The key insight and motivation behind the paper's approach is the construction of a large-scale synthetic car dataset, the representation of 3D objects using global illumination and relightable 3D Gaussian primitives, and the use of a feed-forward model to directly predict relightable Gaussian primitives and global illumination. The specific methodology involves taking a single image as input and employing a multi-view diffusion model to generate multi-view images. These multi-view images, along with their corresponding camera ray embeddings, are then fed into the feed-forward model to predict the relightable Gaussians and global illumination, enabling the generation of objects that can be relit under various lighting conditions.
weaknesses: 1. The insertion effect in Figure 6 of the paper is not satisfactory; the reconstructed car appears artificial when placed into the scene. The authors need to explain the reasons behind this. I suspect it is due to the lack of reflective texture on the car and the absence of shadow modeling. While I can accept the lack of shadow modeling, the absence of reflective texture implies inaccurate material prediction. Is this due to the poor quality of the dataset created in the paper?
2. The core contribution of the paper is the prediction of relightable objects, but the experiments lack validation of relighting, including the accuracy of materials and relighting. The paper also does not compare with methods that do relightable object generation, such as UniDream. Without these experiments, it is impossible to see the effectiveness of the paper's method. Moreover, the paper lacks relevant visualization results.
3. The paper states that the feed-forward model outputs pixel-aligned Gaussian primitives. Does this refer to Gaussian maps? This is not clear. If it is Gaussian maps, Figure 3 only shows the Gaussian maps from a single perspective, and the paper does not mention whether it is multi-view. If it is only a single perspective, how can it represent a 360-degree object?
4. In Figure 3, the feed-forward model is referred to as the "Relightable 3D-GS Generative Model", but based on the text description of the paper, I do not see that this GS prediction model is a generative model. Why is it called a generative model in Figure 3? Or is it just a typo?
5. The technical contribution of the paper is relatively weak, mainly adding the prediction of relightable Gaussian primitives to LGM. I believe this is not sufficient for the ICLR conference.