Title: X-Ray: A Sequential 3D Representation For Generation
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D generation

3D representation

3D reconstruciton

diffusion model
Abstract: We introduce X-Ray, a novel 3D sequential representation inspired by the penetrability of x-ray scans. X-Ray transforms a 3D object into a series of surface frames at different layers, making it suitable for generating 3D models from images. Our method utilizes ray casting from the camera center to capture geometric and textured details, including depth, normal, and color, across all intersected surfaces. This process efficiently condenses the whole 3D object into a multi-frame video format, motivating the utilize of a network architecture similar to those in video diffusion models. This design ensures an efficient 3D representation by focusing solely on surface information. Also, we propose a two-stage pipeline to generate 3D objects from X-Ray Diffusion Model and Upsampler. We demonstrate the practicality and adaptability of our X-Ray representation by synthesizing the complete visible and hidden surfaces of a 3D object from a single input image. Experimental results reveal the state-of-the-art superiority of our representation in enhancing the accuracy of 3D generation, paving the way for new 3D representation research and practical applications. 
Our project page is in \url{https://tau-yihouxiang.github.io/projects/X-Ray/X-Ray.html}.

X-Ray: A Sequential 3D Representation For Generation

Abstract

We introduce X-Ray, a novel 3D sequential representation inspired by the penetrability of x-ray scans. X-Ray transforms a 3D object into a series of surface frames at different layers, making it suitable for generating 3D models from images. Our method utilizes ray casting from the camera center to capture geometric and textured details, including depth, normal, and color, across all intersected surfaces. This process efficiently condenses the whole 3D object into a multiframe video format, motivating the utilization of a network architecture similar to those in video diffusion models. This design ensures an efficient 3D representation by focusing solely on surface information. Also, we propose a two-stage pipeline to generate 3D objects from X-Ray Diffusion Model and Upsampler. We

demonstrate the practicality and adaptability of our X-Ray representation by synthesizing the complete visible and hidden surfaces of a 3D object from a single input image. Experimental results reveal the state-of-the-art superiority of our representation in enhancing the accuracy of 3D generation, paving the way for new 3D representation research and practical applications. Our project page is in https://tau-yihouxiang.github.io/projects/X-Ray/X-Ray.html .

1 Introduction

General, accurate, and efficient 3D representations are three of the most critical requirements for 3D generation [23, 29, 32, 21]. The significance of this goal stems from the ever-expanding array of applications reliant on 3D technology, ranging from virtual reality and augmented reality to computeraided design and beyond. Previous approaches to 3D representation such as meshes, point clouds, voxels, Neural Radiance Fields (NeRF) [36, 45, 58, 24, 15] and 3D Gaussian Splatting [21] possess unique strengths respectively, but face challenges in concurrently satisfying the three requirements for 3D synthesis. Specifically, meshes are widely used in 3D modeling, while they are constrained by their topology when describing complex objects, which limits their generative capacity. Point clouds offer a more flexible capture of the object geometries but lack continuous and dense feature extraction [12, 42]. Voxels simplify spatial reasoning at the cost of significant rising memory complexity with increasing resolution. Neural representations, such as NeRF [36] and 3D Gaussian Splatting [21], offer an impressive leap in rendering photorealistic scenes. Nevertheless, the 3D object are predicted by multi-view images with a relatively long optimized period.

Recently, rendering-based 3D generative methods [26, 57, 48, 14, 53, 49, 18] have gained significant attention for their ability to achieve general and even efficient 3D generation by incorporating neural representations [36, 21] with Transformers [52] or Diffusion Models [43, 2]. However, these methods have a critical limitation: they cannot completely generate objects that include both visible and hidden surfaces. This limitation arises from the methodological design of rendering-based 3D generation, which relies on the 2D supervision of rendered images. As a result, these methods primarily focus on reconstructing the visible external surfaces of objects, while neglecting the internal hidden surfaces. This oversight leads to incomplete or unrealistic reconstructions, as shown in Fig. 1 (a).

In this paper, we propose the X-Ray representation to overcome this limitation of incomplete generation while maintaining the efficiency and generalization required for 3D generation. As illustrated in Fig. 1 (b), our X-Ray, inspired by x-ray imaging in the medical field, can see through the entire object. It efficiently captures and stores information about both visible and hidden surfaces. Consequently, the hidden interior of the object can be fully reconstructed.

Our X-Ray is designed to capture the shape (depth and normal) and appearance attributes (color) along all the sequentially intersected surfaces through ray casting. We transpose the collected slim grid voxels into a multi-frame surface representation, which significantly reduces the data footprint while preserving essential detail. Moreover, the compatibility of our X-Ray's data structure with sequential 3D representation in video formats, as illustrated in Fig. 2, opens novel pathways for leveraging video diffusion models in 3D generation. Specifically, by treating our X-Ray representation as sequences of frames, we first harness the power of the video diffusion model [2] and then utilize the video upsampler [7] to generate 3D objects from low to high resolution. As a result, our approach yields high-quality results while inheriting the advanced capabilities and efficiency of video processing.

We demonstrate the advantages of our X-Ray representation through comprehensive experiments, showcasing its superiority, especially completeness in 3D object generation. We train and evaluate our method in image-to-3D reconstruction task and pure 3D generation task. The experimental results reveal that the proposed X-Ray achieves a significant leap forward in the quality of 3D object generation, positioning it as a feasible solution to longstanding challenges in the field. The main contribution of the paper can be summarized as follows:

We present X-Ray, a novel 3D representation that encode the whole visible and hidden surfaces in to video format through ray casting algorithm for maintaining generalization, accuracy and efficiency.

We propose the generative model of our X-Ray via video diffusion model and video upsampling model, enabling low-to-high generation of 3D objects from single images.

𝑯

𝑫

𝑵

𝑪

𝑯

𝑫

𝑵

𝑪

𝑯

𝑫

𝑵

𝑪

𝑯

𝑫

𝑵

𝑪

i

10t.

18t.

182

a.

We showcase the state-of-the-art performance of our X-Ray in 3D generation quality, setting a new benchmark for Image-to-3D modeling.

2 Related Work

2.1 Representation for 3D Models

Handling 3D data is much more complex and resource-intensive than dealing with 1D ( e.g. text and voice) and 2D ( e.g. image) data. This complexity makes it imperative to find effective ways to organize, process, and infer 3D information. Traditional methods for representing 3D data include meshes, which are good for creating detailed visuals but hard to be generalized; point clouds, which are simple and useful for capturing real-world scenes but lack consistent and dense structure in 3D creation; Although, 3D Gaussian Splatting [21] smooths point cloud data into continuous surface but requires additional an initial point cloud as shape, making them less flexible for 3D synthesis; voxels which are excellent for detailed volumetric data but require much computing resources. Multi-Plane Images [35, 51] try to extent the depth concept to multi-layer with a fixed distance, but they can only describe the visible surface toward camera.

Recent advancements in 3D representation have primarily focused on point-level details and implicit functions, such as Occupancy [32], Signed Distance Fields (SDF) [55, 11], Triplanes [10, 16], and Neural representations [36, 21]. These methods have significantly enhanced modeling and rendering capabilities. Occupancy models map the location of any 3D point to its probability of being inside or outside an object, offering a probabilistic approach to shape definition. SDFs [55, 19] refine this concept by quantifying the nearest signed surface distance from any given point, improving the precision of surface representations. Triplanes [3] employ intersecting 2D planes to provide a more efficient route to 3D representation, albeit with some detail loss. NeRFs and Gaussian Splatting [21, 36] produce remarkably realistic renderings from a limited number of viewpoints but require extensive computational effort. Despite these advancements, implicit function-based models often face challenges in extracting full and high-resolution 3D features, hindering high-quality generalization. Empirically, representations that focus on surface is more efficient, and representations with grid representation are more easily to be generalized [37]. Consequently, capturing all surface attributes and organizing in a dense but lightweight data structure renders our X-Ray an accurate, efficient and generalized representation. Noted that our X-Ray is similar to Depth peeling method [9], which is designed for rendering transparent surfaces, while X-Ray transform any 3D object in video format. Besides, our main contribution is using video diffusion as generator to generate objects.

2.2 Generative Models for 3D Generation

Recent 3D generative models can be primarily categorized into two types: diffusion-based [30, 37, 38] and rendering-based [14, 13, 49, 41, 34, 54, 33, 47]. Diffusion-based models fall under direct 3D supervision, while rendering-based models belong to indirect 2D image supervision. Diffusion-based generative models have emerged as powerful tools for 3D generation, leveraging stochastic diffusion processes to gradually transition from noise to structured 3D objects. These models, such as DPM [30], DiT-3D [37], and Point-E [38], have demonstrated remarkable ability in generating high-quality 3D point clouds. They operate by iteratively refining a random noise distribution into a coherent structure that resembles the target 3D shape, capturing complex geometries and surface details with high fidelity. The strength of these models lies in their capacity to model the distribution of 3D points in a continuous space, allowing for the generation of 3D objects with nuanced variations and detailed textures. However, both point-based networks [28, 37, 17] and voxel-based networks [37] is limited by generating high-resolution objects. Another group of methods [41, 34, 54, 33, 47] adopt Score Distillation Sampling (SDS) as prior to train a NeRF [36] or 3D Gaussian Splatting [21] for 3D generation. However, it is not efficient for optimizing a number of different views over a short period.

On the other hand, rendering-based generative models focus on the visible aspect of 3D generation, transforming abstract 3D representations into detailed and photorealistic images or videos. Models such as LRM [14], Open-LRM [13], LGM [48], DMV3D [57], and TripoSR [49] employ advanced rendering techniques to achieve this. However, rendering-based models are optimized only for the visible surfaces of objects, making it difficult to synthesize the invisible or internal surfaces.

In response to these challenges, our approach utilizes a video diffusion model as the foundation for developing 3D X-Ray. This strategy benefits from the strengths of existing video diffusion models while innovatively addressing the limitations of rendering-based techniques, offering a more comprehensive solution for 3D generation that is sensitive to both visible and hidden parts of objects.

3 Our X-Ray Representation

In this section, we detail our X-Ray representation, which includes both encoding and decoding process to facilitate the conversion between 3D mesh formats and our X-Ray representation. Specifically, the encoding process converts a 3D mesh into our proposed X-Ray format, and the decoding process converts our X-Ray back into a 3D mesh.

3.1 Encoding

Given a 3D object under an arbitrary camera view, we apply the ray casting algorithm to encode a 3D object mesh into the proposed X-Ray representation. The ray casting algorithm plays a crucial role in both computer graphics and computational geometry, where it is used for scene rendering, visibility determination, and addressing geometric queries. Specifically, a ray is emitted from camera center into the environment and all the interactions of this ray with target 3D objects are captured sequentially. For each ray r ∈ R within the field of view that intersects with L sequential faces in the mesh, we record their 3D attributes which include depth (distance to camera center) D r = ( d 1 , d 2 , ..., d L ) ∈ R L × 1 , normal N r = ( n 1 , n 2 , ..., n L ) ∈ R L × 3 , and color C r = ( c 1 , c 2 , ..., c L ) ∈ R L × 3 . To indicate surface presence, we denote Hit H r = ( h 1 , h 2 , ..., h L ) ∈ R L × 1 to indicate whether there is a surface. Since L is usually very small (Sec. 5.3), we denote the efficient and thin grid voxels X ∈ R H × W × L × 8 as the object representation, where the ray X ij with image coordinate [ i, j ] can be represented as:

Note that X [ i, j, k ] = 0 when there is no surface for the k th layer at the image ray coordinate [ i, j ] . Examples of X-Ray encoding are shown in Fig. 2. Through the encoding process, we can transform any mesh into a sequential representation with varying lengths, same as a video with different numbers of frames. Finally, we transpose the voxels as X-Ray X ∈ R L × 8 × H × W , resembling a video format with L frames, each with a resolution of H × W and 8 feature channels.

𝑳×𝟖× 𝑯

𝟒

× 𝑾

𝟒

𝑳×𝟖×𝑯×𝑾

Upsampler

×𝟒

(b) X-Ray Upsampler

Screened

Poisson

Image

Embedding

𝑬

Spatial-Temporal 3D UNet

N

step

Mid

Up

Down

(a) X-Ray Diffusion Model

Noise

𝑳×𝟖× 𝑯

𝟒

× 𝑾

𝟒

𝒛: 𝟒×	 𝑯

𝟒

× 𝑾

𝟒

𝒛: 𝟒× 𝑯

𝟒

× 𝑾

𝟒

𝒆: 𝟏×𝟕𝟔𝟖

𝑳

𝑳

Mesh

3.2 Decoding

The decoding process converts the X-Ray representation back into a 3D mesh. To achieve this, we first convert the video format of X-Ray into a point cloud and subsequently apply the Screened Poisson algorithm [20] to transform the point cloud into a 3D mesh.

X-Ray → Point Cloud. Given an X-Ray, we first compute the 3D object's point cloud P r = { P x , P n , P c } for Ray r , including location P x , color P c , and normal P n defined by the equation:

r o and r d denote the origin and direction of the camera ray, respectively. Furthermore, D r , N r , C r and H r are the depth, surface normal, color and hit attributes of the ray defined in Eq. 1. Upon processing all camera rays, we obtain a comprehensive point cloud P = { P r } r ∈R representation that includes location, normal, and color attributes of the 3D object.

Point Cloud → Mesh. The Screened Poisson algorithm [20] for converting point clouds with color and normal into 3D colored meshes is a classic method that leverages the mathematical principles of the Poisson equation. The core idea involves solving a variation of the Poisson equation to interpolate a smooth surface that fits the input point cloud. The Poisson equation is a partial differential equation of the form: ∇ 2 ϕ = f , where ∇ 2 denotes the Laplace operator (which represents the divergence of the gradient of a function), ϕ is the potential field to be solved, and f is a scalar function representing the divergence of the vector field derived from the input point cloud. In the context of point cloud to 3D mesh conversion, the algorithm first employs the given normal to define a vector field that suggests the orientation of the surface at each point. The divergence of this vector field serves as the function f the Poisson equation.

Encoding-Decoding Intrinsic Error. The encoding-decoding process will introduce a intrinsic and minor reconstruction error that varies with the number of layer L and the frame resolution ( H,W ) . To explore this, we conduct an experiment in Sec. 5.3 aimed at analyzing these variables to identify their optimal values. Our goal is to achieve a balance where all pertinent information is preserved while maintaining a lightweight model.

4 X-Ray for 3D Generation

Our primary objective of introducing a new 3D representation model is to facilitate the generation of 3D objects from single images. The challenge lies in accurately predicting the characteristics that are not immediately visible on the first surface when only a single image is available. To overcome this challenge, we utilize diffusion and upsampling models for X-Ray synthesis. Given that our proposed X-Ray is in video format, we leverage advanced Video Diffusion models as our backbone. To exploit this structure for high-resolution X-Ray synthesis, we incorporate principles from advanced video

diffusion models as our foundational framework. Notable models in this domain include Stable Video Diffusion (SVD) [2], VideoFusion [31], and the state-of-the-art Sora. To efficiently train the diffusion model, we begin by training a low-resolution X-Ray diffusion model that generates X-Ray from a single image. Subsequently, we employ an upsampler to enhance these synthesized X-Rays to high resolution. This two-step approach ensures a more manageable and efficient training process, gradually improving the quality of the output.

Framework Overview. Fig. 3 presents an overview of our generative model using X-Ray representation ( cf. Sec. 3.1) for 3D generation. Our X-Ray diffusion model ( cf. Sec. 4.1) operates at the core of the framework, transforming random Gaussian noise into a low-resolution X-Ray representation conditioned by an input image. These low-resolution X-Rays are then enhanced to high resolution through the application of a 3D Spatial-Temporal Upsampler ( cf. Sec. 4.2). Finally, the high-resolution X-Rays are decoded into 3D meshes using a combination of point cloud transformation and the Screened Poisson algorithm ( cf. Sec. 3.2).

4.1 X-Ray Diffusion Model

Diffusion models [43] are generative models that transform a random noise distribution into a data distribution through a reverse process, counteracting a forward process that incrementally adds Gaussian noise to the data. The forward process is a Markov chain described by x t = √ α t x t -1 + √ 1 -α t ϵ , where x t represents the data at step t , α t controls the noise level, and ϵ ∼ N (0 , I ) is sampled noise. The reverse process, aimed at reconstructing the original data from noise, is modeled by a neural network predicting the noise added at each step or directly denoising the data, following

optimizing the network to minimize the difference between the original and reconstructed data, effectively learning to invert the noise addition process given by:

where t is uniformly sampled from the set {1, . . . , T }.

Diffusion Model for X-Ray. A prevalent technique in diffusion models is the utilization of latent spaces with a VQ-VAE [7] to perform the initial data transformation to compress the data. This method poses a significant challenge for our X-Ray representation as it requires the development of a VQ-VAE model from scratch due to the absence of a suitable off-the-shelf latent model for X-Ray, which would consequently increase our training burden. Another promising approach for efficiently training high-resolution generators is the cascaded synthesis pipeline. This method, exemplified by works such as Imagen [44], DeepFloyd IF [1], and Stable Cascaded [40], involves progressively training the diffusion model or upsampling network from lower to higher resolutions. Given our limited computing resources, we opted to implement this cascaded upsampling strategy. This technique facilitates a more gradual and controlled enhancement of X-Ray quality, providing a more flexible and efficient alternative to traditional latent space diffusion models.

Specifically, we use the Spatial-Temporal 3D U-Net network from Stable Video Diffusion [2] for our diffusion model to generate low-resolution X-Rays, with modifications to the input and output channels. As shown in Fig. 3, the input image is sent to the encoder E , producing an image latent z via image VAE [7] and an embedding e via a ViT [5]. z is concatenated with the X-Ray latent as input, and e interacts with the 3D U-Net through cross attention [2] to finally output the denoised latent. This model employs spatial-temporal attention mechanisms to alternately extract features from 2D frame spaces and 1D surface layer sequences, enhancing its ability to process and interpret the different layers of the X-Ray. This approach allows for nuanced handling of the temporal information inherent in sequential X-Ray data, crucial for achieving high-quality diffusion results.

4.2 X-Ray Upsampler

The X-Ray upsampler focuses on enhancing previously generated X-Rays to a higher resolution. We considered two potential methods: point cloud up-sampling and video up-sampling. Encoding low-resolution X-Rays into a point cloud with color and normal information is straightforward (see Sec 3.1). However, point cloud up-sampling often increases only the number of points without effectively enhancing attributes like texture and color due to its unstructured nature. To improve

efficiency and consistency, we adopted a video up-sampling approach using a spatial-temporal V AE decoder from Stable Video Diffusion (SVD) [2]. We concatenate the previous image latent z with the low-resolution X-Ray as input and output the high-resolution X-Ray. The model upsamples previously synthesized low-resolution X-Ray frames fourfold while preserving the original layer number L . It applies attention at both the 2D surface frame and 1D surface layer levels, enhancing frame resolution and overall quality. This makes the X-Ray diffusion model, followed by the upsampler, a more integrated and effective solution.

Loss . The loss function for the Upsampler differs notably from that of the diffusion model. While the diffusion model loss typically addresses volumetric or textural aspects, the Upsampler loss concentrates specifically on the surface area accuracy, reflecting the critical importance of maintaining high fidelity in the enhanced images. The loss function we use for the Upsampler is given by:

where X gt [ H gt ] represents the ground-truth high-resolution X-Ray at hit surface and X up [ H gt ] denotes the Upsampler's output at hit surface, and H gt , H up denotes the ground-truth and upsampled Hit, respectively. The loss is computed as the squared Euclidean distance between these two matrices, quantifying the pixel-wise discrepancy in surface details. This metric effectively ensures that the upsampling process preserves essential surface features, thereby optimizing the quality and utility of the resulting high-resolution X-Ray.

5 Experiments

5.1 Dataset and Implementation

Datasets. We train our X-Ray pipeline using a subset of the Objaverse dataset [4], which have been removed entries with missing textures and inadequate prompts as outlined in [48]. This subset consists of more than 60,000 3D objects. For each object, we select 8 random camera views, covering azimuth angles from -180 to 180 degrees and elevation angles from 0 to 45 degrees with camera distance to object center fixed at 1.2. The images are then rendered using Blender Software, and the corresponding X-Rays are generated through the ray casting algorithm provided by the trimesh library [50]. Through these processes, we create a dataset of approximately 480,000 paired images and X-Rays to train the generative model. For the evaluation datasets, we adopt two commonly adopted datasets: Google Scanned Objects [6] and OmniObject3D [56], to assess generative performance via single-view reconstruction tasks.

Metrics. Recent 3D generative models [26, 18, 27, 14, 49] lack unified reconstruction evaluation metrics due to the challenge of determining an object's size and orientation from a single image [46]. To ensure fair comparisons with state-of-the-art methods, we align all methods before evaluation. The predicted and ground-truth 3D objects will be normalized to a range of -0.5 to 0.5 along all three axes and face forward the same -z axis. We then align them using the Iterative Closest Points (ICP) algorithm and calculate Chamfer Distance (CD) in L 1 norm and F-Score (FS) at threshold 0.1 (FS@0.1) for reconstruction.

Implementation Details. Our X-Ray diffusion model is based on the Spatrial-Temporal 3D UNet architecture used in Stable Video Diffusion (SVD) [2], modified to synthesize 8 channels: 1 hit channel, 1 depth channel, 3 color channels, and 3 normal channels, compared to the original 4 channels. During training, we maintain a learning rate of 0.0001 using the AdamW optimizer. Since different X-Rays have varying numbers of layers, we pad or truncate them to a uniform 8 layers for efficient batching and training. Each layer's frame has dimensions of 64 × 64 . For the upsampler, each layer's output remains at 8 channels, but the resolution of each frame is increased to 256 × 256 to enhance detail and clarity in the upscaled X-Ray. The entire training pipeline is conducted on 8 NVIDIA A100 GPU servers for two weeks. During inference, the 3D generation process takes approximately 7 seconds: about 1 second for the diffusion model, 1 second for the upsampler, and 5 seconds for mesh decoding. As for GPU usage during inference, the GPU memory required is 4.8 GB for X-Ray diffusion model and 2.5 GB for X-Ray Upsampler.

1

2

3

4

5

6

7

8

9

10

11

12

Surface Layer /glyph1197umber (L)

2

4

6

8

Chamfer Distance (CD)

1e

2

6.55e-02

1.93e-02

9.60e-03

8.30e-03

7.79e-03

7.69e-03

7.64e-03

The Encoding-Decoding Intrinsic Error of Surface Layer /glyph1197umber (L)

(a) The intrinsic error of layer number ( L ), when H = W = 256

32 64

128

256

512

1024

Frame Height (H) / Width (W)

1

2

3

4

5

6

Chamfer Distance (CD)

1e

2

5.29e-02

2.78e-02

1.47e-02

7.79e-03

7.74e-03

7.70e-03

The Encoding-Decoding Intrinsic Error of Frame Height (H) / Width (W)

(b) The intrinsic error of frame height ( H ) or width ( W ) when L = 8

5.2 Efficient Comparison with Different 3D Representation

We compared the efficiency of different representations using 500 3D meshes from random selected models. The results showed that both point cloud and X-Ray were highly efficient, with lower memory, faster encoding & decoding times. However, the X-Ray had the advantage of being reorganizable as a video format for diffusion models, leading to better performance.

5.3 Analysis of Encoding-Decoding Intrinsic Error

Due to the finite number of layer L and resolution H,W of X-Ray, a slight intrinsic error is inevitable during the encoding of 3D meshes into X-Ray format and the subsequent decoding back into 3D meshes. To quantitatively assess this error, we conducted an experiment to evaluate intrinsic error via Chamfer Distance (CD) ( L 1 norm) between the original (ground-truth) mesh and the encodingdecoding mesh across various resolutions and layers. In the experiments, we set the frame resolution through a series of predefined values: 32 , 64 , 128 , 256 , 512 , and 1 , 024 and vary the number of layers from 1 to 12 . Fig. 4a indicates that the intrinsic error decreases as the number of layers in our X-Ray representation increases, becoming convergent after 8 layers. Similarity as illustrated in Fig. 4b, the intrinsic error decreases with increasing resolution and stabilizes after 256. Therefore, for a balance between accuracy and efficiency, we use a resolution 8 × 256 × 256 with H = W = 256 and L = 8 in our experiments. Compared with the dense volume achieving a 256 × 256 × 256 resolution for voxel-based methods, our X-Ray representation is significantly efficient for focusing only on surfaces and reducing the data volume by 96.88%.

One-2-3-4-5

OpenLRM

TripoSR

Ours

GT Mesh

OmniObject3D

GSO

ShapeNet

5.4 Quantitative Comparison

For image-to-3D mesh generation on the GSO [6] and OmniObject3D [56] datasets, we build a new benchmark that randomly selects 500 image-mesh pairs from each dataset. We re-ran the baselines using their released source code and used normalized Chamfer distance and F-score for a fair comparison. The results, summarized in Tab. 2, show that our X-Ray method achieves significant improvements over all previous rendering-based state-of-the-art methods [25, 18, 8, 13, 49] on both datasets, with a relative 33% improvement (0.084 → 0.056) in Chamfer distance and also a significant improvement in FS@0.1 (0.878 → 0.973, where the maximum is 1) on GSO dataset and similar performance on OmniObject3D dataset. This demonstrates the superiority of our approach over rendering-based methods.

5.5 Qualitative Comparison

A qualitative comparison effectively demonstrates the advantages of our proposed X-Ray generation method. Using the three datasets mentioned, we selected single images as input and generated 3D meshes without and with textures, as shown in Fig. 5. Our proposed method has three key advantages: 1. It can decompose shape and appearance to accurately reconstruct flat surfaces (Rows 1 and 3); 2. It can detect the sealing of containers (Rows 2 and 4); 3. It can generate the internal structure of objects within their outer surfaces (Rows 5 and 6). These obvious advantages highlight the our effectiveness.

5.6 Failure Cases.

The failure cases highlight the limitations of the current generative model based on X-Ray representation when the number of frame layers is very large. As illustrated in Fig. 6, given an input image containing a complex object, such as an exquisite hamburger, the number of frame layers L of the encoded or generated X-Ray tends to exceed the maximum length of 8. Consequently, any surface

behind layer 8 will be omitted, resulting in missing parts of the reconstructed mesh. The solution is to increase the value of L so that the X-Ray can represent more surfaces. However, this would also increase computing resource requirements. We will reconsider the sparsity of deeper surface frames and propose a more efficient generative model to overcome these failure cases.

Generated X-Ray

Generated Mesh

Ground-Truth Mesh

6 Conclusion

In this work, we introduced a novel X-Ray representation for 3D objects that encompasses both visible and hidden surfaces within the camera's field of view, unlike recent rendering-based methods that typically focus only on the visible surface. We demonstrated the effectiveness of the X-Ray approach in single-view 3D generation tasks. Our generative model shows that the underlying generator for X-Ray shares foundational similarities with existing video diffusion models, allowing us to leverage their advantages. Experimental results highlight the outstanding performance of our method.

Limitations. Our generator uses the Stable Video Diffusion (SVD) pipeline to produce high-quality X-Ray. However, the X-Rays generated consist of an uncertain number of sequential layers, with the posterior layers tending to be more and more sparse, which can be redundant. Additionally, the generated mesh lacks enough smoothness and has missing part when X-Ray is truncated. We plan to explore advanced network architectures, such as Large Language Model, that can better handle the complexities of X-Ray data, including layer sparsity and sequential format. Additionally, we aim to investigate further applications of the X-Ray representation to broaden its utility and impact in 3D modeling and beyond.

References

DeepFloyd Lab at StabilityAI. DeepFloyd IF: a novel state-of-the-art open-source text-to-image model with a high degree of photorealism and language understanding. https://www.deepfloyd.ai/ deepfloyd-if , 2023. Retrieved on 2023-11-08.

Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR , 2023.

Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In CVPR , 2022.

Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.08051 , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR , 2021.

Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas Barlow McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022 , 2022.

Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.

Zi-Xin Zou et. al. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09147 , 2023.

Cass W. Everitt. Interactive order-independent transparency. In NVIDIA , 2001.

Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojcic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In Advances In Neural Information Processing Systems , 2022.

Wenhang Ge, Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cong Chen. Ref-neus: Ambiguity-reduced neural implicit surface learning for multi-view reconstruction with reflection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4251-4260, 2023.

JunYoung Gwak, Christopher B Choy, and Silvio Savarese. Generative sparse detection networks for 3d single-shot object detection. In European conference on computer vision , 2020.

Zexin He and Tengfei Wang. Openlrm: Open-source large reconstruction models. https://github. com/3DTopia/OpenLRM , 2023.

Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: large reconstruction model for single image to 3d. CoRR , abs/2311.04400, 2023.

Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientnerf efficient neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12902-12911, 2022.

Tao Hu, Xiaogang Xu, Ruihang Chu, and Jiaya Jia. Trivol: Point cloud rendering via triple volumes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 20732-20741, 2023.

Tao Hu, Xiaogang Xu, Shu Liu, and Jiaya Jia. Point2pix: Photo-realistic point cloud rendering via neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8349-8358, 2023.

Zixuan Huang, Stefan Stojanov, Anh Thai, Varun Jampani, and James M Rehg. Zeroshape: Regressionbased zero-shot shape reconstruction. arXiv preprint arXiv:2312.14198 , 2023.

Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, and Chi-Wing Fu. Make-a-shape: a ten-million-scale 3d shape model. CoRR , 2024.

Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM Trans. Graph. , 2013.

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), 2023.

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.02643 , 2023.

Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, and Ying Shan. Advances in 3d generation: A survey. CoRR , abs/2401.17807, 2024.

Lingjie Liu, Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurIPS , 2020.

Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-23-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems , 36, 2024.

Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023.

Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453 , 2023.

Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel CNN for efficient 3d deep learning. In NeurIPS , 2019.

Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J. Black, and Bernhard Schölkopf. Ghost on the shell: An expressive representation of general 3d shapes. In ICLR , 2024.

Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In CVPR , 2021.

Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023.

Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR , 2019.

Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shapeguided generation of 3d shapes and textures. arXiv preprint arXiv:2211.07600 , 2022.

Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. arXiv preprint arXiv:2112.03221 , 2021.

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) , 2019.

Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

Shentong Mo, Enze Xie, Yue Wu, Junsong Chen, Matthias Nießner, and Zhenguo Li. Fast training of diffusion transformer with extreme masking for 3d point clouds generation. arXiv preprint arXiv: 2312.07231 , 2023.

Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. CoRR , abs/2212.08751, 2022.

William Peebles and Saining Xie. Scalable diffusion models with transformers. In ICCV , 2023.

Pablo Pernias, Dominic Rampas, Mats L. Richter, Christopher J. Pal, and Marc Aubreville. Wuerstchen: An efficient architecture for large-scale text-to-image diffusion models, 2023.

Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv , 2022.

Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J. Guibas. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In NeurIPS , 2017.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022 , 2022.

Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kamyar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS , 2022.

Sara Fridovich-Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR , 2022.

Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang, Eric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310.17994 , 2023.

Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653 , 2023.

Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.05054 , 2024.

Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang, , Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint arXiv:2403.02151 , 2024.

Trimesh. Trimesh [computer software]. https://github.com/mikedh/trimesh , 2019.

Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , 2020.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017.

Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: novel multi-view synthesis and 3d generation from a single image using latent video diffusion. CoRR , abs/2403.12008, 2024.

Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint arXiv:2212.00774 , 2022.

Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. NeurIPS , 2021.

Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Liang Pan Jiawei Ren, Wayne Wu, Lei Yang, Jiaqi Wang, Chen Qian, Dahua Lin, and Ziwei Liu. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In CVPR , 2023.

Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. DMV3D: denoising multi-view diffusion using 3d large reconstruction model. CoRR , abs/2311.09217, 2023.

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV , 2021.

A Appendix

The Appendix section contains detailed information on the network structure (including the X-Ray diffusion model and upsampler), ablation studies, additional experimental quantitative and qualitative results, failure cases, and key source code.

A.1 Network Details

X-Ray Diffusion Model As outlined in our study, we utilized the 3D Spatial-Temporal UNet as the foundational architecture for our diffusion model. For image-to-3D generation, the input includes 4-channel image latent and 8-channel noise X-Ray and the output has 8 channel to denoise XRay. Initially, we loaded all network parameters and fine-tuned the model on our training dataset. Despite these efforts, the final performance fell short of expectations. We hypothesized that this underperformance stemmed from the limited size of our training dataset, which was insufficient for such a parameter-rich model, as well as a substantial domain gap between the original video data and our X-ray data.

To address these issues, we reduced the network size to 10% of its original configuration and trained the model from scratch using a significantly larger batch size. This approach proved highly effective, as the final performance exceeded the current state-of-the-art methods by a considerable margin. Consequently, our findings suggest that a smaller, more focused network trained from scratch can be more effective than a larger pre-trained model when faced with limited and domain-specific datasets. This approach not only enhances performance but also highlights the importance of customizing the model size and training strategy to the specific characteristics of the data. Our results indicate that careful consideration of the network architecture and training regimen is crucial for optimizing performance, particularly in specialized applications such as X-ray generation. This study provides valuable insights into model adaptation and training strategies that can be applied to other domains facing similar challenges.

The following JSON file contains the configuration details for our X-Ray Spatial-Temporal UNet.

A.2 X-Ray Upsampler

The X-Ray diffusion model enables the generation of low-resolution 3D objects. To enhance resolution and improve performance, our X-Ray upsampler increases the frame resolution by a factor of 4. For image-to-3D generation, we concatenate the 4-channel image latent representation with the 8-channel low-resolution X-Ray, producing an 8-channel high-resolution X-Ray.

This process begins with the diffusion model creating a coarse, low-resolution 3D representation that captures the essential structure of the object. The upsampler then refines this representation, significantly improving the resolution and adding finer details that contribute to the realism of the 3D model. By combining the latent image features with the initial low-resolution X-Ray data, we ensure that the final high-resolution output retains the context and nuances of the original image while also incorporating the detailed structural information provided by the X-Ray data.

The following JSON file contains the configuration details for our X-Ray Upsampler.

A.3 Ablation Studies

The Effect of Diffusion Model

As described in Sec. A.1, we train the diffusion model with different configurations, including varying model sizes and how to conduct initialization. Specifically, we evaluate three models:

Finetuned Original UNet

Randomly Initialized Original UNet

Randomly Initialized UNet with 10% Parameters

The experimental evaluation results on the GSO [6] dataset are illustrated in Tab. 3. These results allow us to analyze the impact of different initialization and scaling strategies on the performance of the diffusion model. By comparing the outcomes, we can identify the trade-offs between model size, training time, and overall accuracy. We observed that finetuning the model did not introduce significant improvement because of the domain gap between video and X-Ray data. Also, it is not necessary to adopt a large diffusion model for Objeverse dataset [4] for significantly reducing the batch size. Thus, we adopt the randomly initialized UNet with 10% parameters as our diffusion

Synthesized X-Ray

Encoded Point Cloud

Decoded Mesh View 1

Decoded Mesh View 2

Decoded Mesh View 3

model. These findings provide insights into the optimal configuration for diffusion models in 3D related tasks.

The Effect of Hit H

The original surface attributes only contain depth D , normal N , and color C . We add an additional Hit H attribute to indicate whether there is a surface. In this ablation study, we demonstrate the necessity of including the Hit H attribute. We conduct training experiments with and without the Hit H attribute and evaluate on the GSO [6] dataset. The results, shown in Tab. 4, indicate that including the Hit H attribute can improve performance. The reason might be that the X-Ray is sparse and requires an indicator to balance the generative process. The UNet model and Upsampler with the Hit H attribute achieves better CD and FS@0.1 scores, demonstrating its importance in accurate 3D generation.

A.4 More Visualization

We shown more Visualization results of single-view image generation in Fig. 7. Also, we extend the task to generate 3D object from text prompt. Text-to-3D mesh generation can also be achieved via using text-to-image, object segmentation, and image-to-3D processes. We utilize established diffusion models that are already proficient in image synthesis from textual descriptions instead of

"a black and silver

power supply"

Synthesized Image

Segmented Image

Synthesized X-Ray

Encoded Point Cloud

Decoded Mesh

'green wine bottle"

"a polar bear"

"a rolled haystack"

developing a new text-conditioned diffusion model. One Model such as Stable Diffusion [43], Stable Cascaded [40], or DiT [39] are employed to generate images based on the input text. Following this, we apply an image segmentation tool, specifically the Segment Anything Model [22] to eliminate the background. This streamlined method avoids the complexities of training a new model from scratch instead of making use of sophisticated pre-trained models to handle the text-to-image translation, thereby simplifying the process of generating 3D meshes from textual inputs. The output results of Image-to-3D and Text-to-3D are illustrated in Fig. 8.

A.5 Key Source Code

Ray Casting: The source code of ray casting the mesh to get the point cloud.

Point Cloud to Mesh: The source code of transferring the predicted point cloud to mesh using Poisson Surface Reconstruction.

Normalized Metrics: The source code of normalize and align the predicted point cloud to the gt point cloud before evaluation.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper discusses the limitations of the work performed by the authors in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The theory assumptions in this paper have already been proved in previous works (Diffusion Model and Screened Poisson Algorithm).

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The training and evaluation source code is uploaded so that the reviewer can check the details and re-implement our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The training and evaluation source code is uploaded so that the reviewer can check the details and re-implement our results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The paper specify all the training and test details.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The paper adopts the mean metric values for 3 times testing for a fair comparison.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In the Experimental section Implementation section, the paper introduces the detailed implementation details, including the computing resources, training and evaluation details. Also, the source code is uploaded for checking.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper is sure to preserve anonymity.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: This paper contributes to fundamental 3D representation research and extends beyond just 3D generation tasks. In the future, it has the potential to enhance other research fields and benefit human society.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper cites the original paper that produced the code package or dataset, and follows the licenses for existing assets.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The paper introduce the details of the dataset to be released.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowd-sourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents X-Ray, a new 3D sequential representation inspired by the penetrating quality of X-ray scans. This technique converts a 3D object into surface frames at different layers, ideal for creating 3D models from images. Experimental findings show that the X-Ray approach outperforms existing methods in improving the precision of 3D generation, opening up new possibilities for research and real-world applications in 3D representation.\n\nSTRENGTHS:\n* The motivation is interesting. Exising methods indeed cannot completely generate objects that include both visible and hidden surfaces. While the proposed method can the hidden interior of the object can be fully reconstructed.\n* The compatibility of X-Ray data structures with sequential 3D representations in video formats opens up new opportunities for using video diffusion models in 3D generation.\n* Great experimental results. The paper outperforms competitors by achieving the best results.\n\nWEAKNESSES:\n* The author asserts that \"General, accurate, and efficient 3D representations are three crucial requirements for 3D generation.\" However, the citation is missing or the author should provide evidence to support the claim that 3D representations need to be general, accurate, and efficient.\n* The ablation study is insufficient. The author should conduct ablation on X-Ray frames, for example, by using video or image diffusion techniquesm or the impact of frame complexity.\n* The author claims to have synthesized a complete interior mesh in Figure 1, but I did not observe any other results besides cars or analysis in the experiments.\n* I think the method used is the X-ray edge method because the dataset quality does not allow for a complete X-ray scan. For instance, the 3D CAT data has empty spaces within.\n\nQUESTIONS:\n* The authors need to provide a clear explanation of X-ray and complete internal mesh, as some training data contain empty spaces within.\n* The authors should conduct additional ablation studies.\n\nLIMITATIONS:\nThe authors acknowledged limitations but did not address the potential negative societal impact of the proposed technology.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-08"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the problem of missing mesh interiors in image-to-3D generation. The proposed representation, X-Ray, adopts ray casting to encode both visible and hidden surfaces into a video format. With this representation, the authors enable single image-to-3D mesh generation, including the inside of the mesh, through a video diffusion model. Experiments demonstrate the effectiveness of the proposed method in 3D generation from a single image, both quantitatively and qualitatively.\n\nSTRENGTHS:\n(1) The task of single-view image to 3D mesh generation including interior surface of mesh is extremely challenging and well-motivated. As far as I know, this is the first paper that generates a mesh considering the interior surface.\n\n(2) The idea of using ray casting to scan the interior of the mesh and convert it into video frames is novel. I really like this approach.\n\n(3) The results show that the proposed method achieves SOTA performance on single image-to-3D generation and point cloud generation with a short inference time compared to existing baselines.\n\n(4) The writing is easy to understand and the design choices are well presented.\n\nWEAKNESSES:\n(1) Considering Computed Tomography, it is expected that the performance will significantly increase as the number of layer $L$ increases. However, the experiments analyzed in Section 5.2 show that performance improvements are very minimal when $L$ is larger than 8. If additional computing resources are available, it would be interesting to see the performance when the frame resolution is low, but $L$ is extremely high (not mandatory).\n\n(2) The entire training pipeline is quite long and memory-intensive. It was conducted on 8 NVIDIA A100 GPU servers for a week.\n\nQUESTIONS:\n(1) When training, it seems necessary to normalize the object so that the rays from a specific camera position can capture the entire object, making it easier to train. I'm curious whether this preprocessing step was applied to the Objaverse mesh. If this process was included, it would be beneficial to add this information to the paper.\n\n(2) How much GPU memory is required during inference?\n\nLIMITATIONS:\nThe authors mention the limitations related to the number of sequence layers and the missing parts when X-Ray is truncated, as discussed in Section 6 and Appendix A.5.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new representation that encodes a 3D mesh into the ray intersection points from a single view point.  The position, color, and surface normal at the intersection points are stored as the representation. Poisson reconstruction is used to recover the 3D mesh from the representation. A cascaded diffusion model is trained using the extracted representation from 60k objaverse meshes. Evaluation is conducted on three other datasets.\n\nSTRENGTHS:\nThe paper proposes a new representation, which is compact, efficient, and tailored to single image to 3D applications. It demonstrated better performance compared to other methods in the paper in the metrics. The visual quality of the results are also better.  3D generation is a very relevant topics and the paper will be interesting to many readers.\n\nWEAKNESSES:\n1. The paper does not discuss how to extend the proposed representation to include multiple viewpoints to provide better encoding/decoding quality. Even though the representation records the intersection points along a ray, the intersection points from front and top views would be very different.  \n\n2. One limitation of X-ray is that the limited number of intersection points (8 in the paper) makes it difficult to encode complex shapes like hairs and carefully designed meshes with all internal structures. For example, taking a real engine, the number of surfaces a ray can pass through would be far more than 8 (eg., the wires, the bores, shafts).  The paper mentions the limitation, however, it does not seem clear to me from the paper how the limitation can be addressed efficiently.  \n\n3. Another related limitation is the potentially nonuniform number of intersection points along different rays from the same viewpoint. For example, on a human head, the rays passing through the hair region would need a much higher number of intersections to avoid losing information than those through the face region.  The nonuniformity extends to across objects.  To model one object that has a large number of intersection points, it might need to increase L used for all objects in order to learn a diffusion model.  This property makes the presentation inefficient.  \n\n4. In order to generate all interior structures of a 3D object, we need meshes that are carefully designed with interior details.  This kind of dataset is difficult to get and if existed would increase L, making the method inefficient.\n\nQUESTIONS:\n1. The proposed representation is related to the pointmap used in DUSt3R (CVPR2024).  From my understanding, at a high level, it extends pointmaps to multiple points along a ray. To me, it is an interesting connection and can be mentioned in the paper. \n\n2. The core of the representation is finding the intersection points and creating the tensor representation. Currently the paper only mentions that it calls a function in trimesh.  I would suggest describing the operation in more detail -- for example, I do not know how the Hit is constructed. Does the method try to aggregate intersection points at similar depths at the same index along the L-dimension?  Or is Hit simply a padding mechanism?  \n\n3. In Sec 5.3 quantitative comparison, Table 1 and Table 2 show metrics that evaluate the quality of the geometry. How is the photometric quality of the proposed method compared to other methods, e.g., PSNR or SSIM on the conditioning viewpoint (and other novel views)?  \n\n4. How significant is the effect of the predicted normal, and is it fair to compare rendering-based methods like LRM using the extracted meshes in Table 1?  My understanding is the quality of the Poisson reconstruction can be affected heavily by the quality of the normal, which may be poorly modeled by volumetric density.  Should the paper also show the same metrics directly on the point clouds or depth maps?\n\nLIMITATIONS:\nThe paper mentions the limited number of intersection points. However, it does not seem clear how to address the limitation potentially without significantly increasing the number of L.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces X-Ray, a new 3D representation designed for efficient generation of 3D objects from single images. The key idea is to represent a 3D object as a sequence of 2D \"surface frames\" capturing hit/miss, depth, normal, and color information along rays cast from the camera viewpoint. This sequential representation lends itself well to generation using video diffusion models, enabling the synthesis of both visible and hidden surfaces. The authors propose a pipeline consisting of an X-Ray diffusion model to generate low-resolution surface frames from an input image, followed by an X-Ray upsampler to enhance resolution. They evaluate their method on single-view 3D reconstruction and unconditional 3D shape generation tasks, reporting quantitative results on standard benchmarks.\n\nSTRENGTHS:\n- The paper tackles the important challenge of reconstructing complete 3D models, including hidden surfaces, from single images. This is a significant limitation of current rendering-based approaches, and the authors' focus on this problem is well-motivated and timely.\n\n-  The X-Ray representation, while its novelty requires further substantiation, offers an intuitive way to encode 3D surface information in a sequential manner. By focusing solely on surface details rather than volumetric data, the representation has the potential to be more memory-efficient than voxel grids or dense point clouds, especially for objects with complex internal structures.\n\n- The authors smartly leverage recent advancements in video diffusion models for 3D generation. This is a promising direction, as video diffusion models have shown impressive capabilities in synthesizing high-quality and temporally coherent sequences of images. Adapting these models to the task of 3D generation through the X-Ray representation is a reasonable and potentially fruitful approach.\n\nWEAKNESSES:\n- The paper does not provide a convincing argument for the novelty of the X-Ray representation. While its sequential capture of surface information is intuitive, a thorough comparison to existing techniques is lacking. Specifically, the authors should clearly differentiate X-Ray from methods like depth peeling, multi-view depth images, multi-plane images (MPI), and notably, the PI3D representation (Liu et al., CVPR 2024), which also leverages diffusion models for 3D generation. \n  * Liu, Ying-Tian, Yuan-Chen Guo, Guan Luo, Heyi Sun, Wei Yin, and Song-Hai Zhang. \"Pi3d: Efficient text-to-3d generation with pseudo-image diffusion.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19915-19924. 2024.\n\n- The paper repeatedly emphasizes the efficiency of the X-Ray representation without providing concrete evidence or analysis. The authors should quantify their claims by comparing the memory footprint and computational costs of X-Ray to alternative representations like voxel grids, point clouds, and neural implicit representations (e.g., NeRFs) for objects of varying complexity.\n\n- While leveraging video diffusion models is promising, the paper does not clearly articulate how the specific properties of the X-Ray representation are exploited within the diffusion process beyond being a sequential data format. Do the hit/miss indicators or the ordered nature of surface frames influence the model architecture or training? Would similar performance be achieved with alternative sequential representations as input to the diffusion model?\n\n- The evaluation heavily relies on reconstruction metrics (CD, EMD), even when assessing a generative model. While these metrics are relevant for the single-view reconstruction task, they do not capture the generative capabilities of X-Ray. The authors should expand generative evaluation to diverse categories beyond ShapeNet Cars. The paper could assess generative quality by evaluating the diversity and realism of multiple shapes generated from the same input image.\n\n- The evaluation would be also strengthened by: (1) Including recent SDS-based single-view 3D generation methods as baselines; (2) Providing visualizations of generated shapes for the unconditional generation experiment; (3) Dedicating a section to analyze failure cases, visually showcasing problematic inputs and outputs.\n\nQUESTIONS:\n- Could the authors please elaborate on the key differences between the X-Ray representation and existing techniques like depth peeling, multi-view depth images, and multi-plane images (MPI)? Also please include the discussion with PI3D.\n\n- To support the claims of efficiency, could the authors provide a quantitative analysis of the memory footprint and computational cost (encoding, decoding, generation) of X-Ray compared to voxel grids, point clouds, or NeRFs? This analysis should consider objects of varying complexity and resolutions.\n\n- Could the authors please include generative metrics for the single-view 3D reconstruction experiments on GSO and OmniObject3D?\n\nLIMITATIONS:\n- The authors merely state limitations without explaining their causes, impact, or potential solutions. E.g., saying \"X-Ray frames become sparse\" is not enough. How does this sparsity affect generation? How can it be addressed?", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}