X-Ray: A Sequential 3D Representation For Generation
Authors: Tao Hu
Wenhang Ge
Yuyang Zhao
Gim Hee Lee
Venue: NeurIPS 2024 spotlight (2024)
Keywords: 3D generation
3D representation
3D reconstruciton
diffusion model
Primary area: machine_vision
TL;DR: We propose a novel 3D representation  that enhances 3D synthesis by capturing intricate object details, both inside and outside!

Abstract: We introduce X-Ray, a novel 3D sequential representation inspired by the penetrability of x-ray scans. X-Ray transforms a 3D object into a series of surface frames at different layers, making it suitable for generating 3D models from images. Our method utilizes ray casting from the camera center to capture geometric and textured details, including depth, normal, and color, across all intersected surfaces. This process efficiently condenses the whole 3D object into a multi-frame video format, motivating the utilize of a network architecture similar to those in video diffusion models. This design ensures an efficient 3D representation by focusing solely on surface information. Also, we propose a two-stage pipeline to generate 3D objects from X-Ray Diffusion Model and Upsampler. We demonstrate the practicality and adaptability of our X-Ray representation by synthesizing the complete visible and hidden surfaces of a 3D object from a single input image. Experimental results reveal the state-of-the-art superiority of our representation in enhancing the accuracy of 3D generation, paving the way for new 3D representation research and practical applications. 
Our project page is in \url{https://tau-yihouxiang.github.io/projects/X-Ray/X-Ray.html}.

X-Ray: A Sequential 3D Representation For Generation
Tao Hu
Wenhang Ge 2+ Yuyang Zhao 1*
Gim Hee Lee
Department of Computer Science, National University of Singapore
Hong Kong University of Science and Technology (Guangzhou)
taohu Giohu@nus.edu.sg
gimhee.lee @ nus.edu.sg
Sequential
Depth
Ruy Casting
BB
Sequential
Generated Mesh
Missing Interior
Generated Mesh
Complete Interior Mesh
(a) Rendering- Bassed Based Generation:
(b) X-Ray Generation (Ours): Outer and Inner Surfaces
Outer Surface
Figure 1: Comparison between the rendering- based 3D generation [49, 14] and our proposed X-Ray generation. The competitors focus on the visible outer surface in multiple camera views. In contrast, our model can sense both the visible and hidden surface single camera view and generate the outer and inner surfaces of objects. An example of missing mesh interior from rendering-basee 31) synthesis vs. complete mesh interior from our X-Ray generator are shown the bottom row.
Abstract
We introduce X-Ray, a novel 3D sequential representation inspired by the pen- etrability of x-ray scans. X-Ray transforms 3D object into series of surface frames at different layers, making it suitable for generating 3D models from im- ages. Our method utilizes ray casting from the camera center to capture geometric and textured details, including depth, normal, and color, across all intersected surfaces. This process efficiently condenses the whole 3D object into multi- frame video format, motivating the utilization of a network architecture similar those in video diffusion models. This design ensures efficient 3D represen- tation by focusing solely on surface information. Also, we propose two-stage pipeline generate 3D objects from X-Ray Diffusion Model and Upsampler: We
"Co-sccond authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

demonstrate the practicality and adaptability of our X-Ray representation by syn- thesizing the complete visible and hidden surfaces of a 3D object from a single input image. Experimental results reveal the tfate-of-the-an superiority of our representation in enhancing the accuracy of 3D generation, paving the way for new 3D representation research and practical applications. Our project page is in https: / //auu ttyyttauulyyaaa sacaauyyhouue yiiouxianng github. io wo/projectss/-Reae
Ray html..
Introduction
General, accurate, and efficient 3D representations are three of the most critical requirements for 3D generation [23, 29, 32, 21]. The significance of this goal stems from the ever-expanding array of applications reliant on 3D technology, ranging from virtual reality and augmented reality to computer- aided design and beyond. Previous approaches to 3D representation such meshes, point clouds, voxels, Neural Radiance Fields (NeRF) [36, 45, 58, 24, 15] and 3D Gaussian Splatting [21] possess unique strengths respectively, but face challenges in concurrently satisfying the three requirements for 3D synthesis. Specifically, meshes are widely used in 3D modeling. while they are constrained by their topology when describing complex objects, which limits their generative capacity. Point clouds offer a more flexible capture of the object geometries but lack continuous and dense feature extraction [12, 42]. Voxels simplify spatial reasoning at the cost of significant rising memory complexity with increasing resolution. Neural representations. such as NeRF [36] and 3D Gaussian Splatting [21], offer an impressive leap in rendering photorealistic scenes. Nevertheless, the 3D object are predicted by multi-view images with relatively long optimized period.
Recently, rendering-based 3D generative methods [26, 57, 48, 14, 53, 49, 18] have gained significant attention for their ability to achieve general and even efficient 3D generation by incorporating neural representations [36, 21] with Transformers [52] Diffusion Models [43, 2]. However, these methods have a critical limitation: they cannot completely generate objects that include both visible and hidden surfaces. This limitation arises from the methodological design of rendering-basee baase 3D generation, which relies on the 2D supervision of rendered images. As a result, these methods primarily focus on reconstructing the visible external surfaces of objects, while neglecting the internal hidden surfaces. This oversight leads to incomplete or unrealistic reconstructions, as shown in Fig. 1 (a).
In this paper, we propose the X- X-Ray representation to overcome this limitation of incomplete generation while maintaining the efficiency and generalization required for 3D generation. As illustrated in Fig. (b), our X-Ray, inspired by x-ray imaging in the medical field, can through the entire object. It efficiently captures and stores information about both visible and hidden surfaces. Consequently, the hidden interior of the object can be fully reconstructed.
Our X- X-Ray is designed to capture the shape (depth and normal) and appearance attributes (color) along all the sequentially intersected surfaces through ray casting. We transpose the collected slim grid voxels into a multi-frame surface representation, which significantly reduces the data footprint while preserving essential detail. Morcover, the compatibility of our X- X-Ray's 's data structure with sequential 3D representation in video formats, as illustrated in Fig. 2, opens novel pathways for leveraging video diffusion models in 3D generation. Specifically, by treating our X-Ray representation as sequences of frames, we first harness the power of the video diffusion model [2] and then utilize the video upsampler [7] to generate 3D objects from low high resolution. As result, our approach yields high-quality results while inheriting the advanced capabilities and efficiency of video processing.
We demonstrate the advantages of our X-Ray representation through comprehensive experiments, showcasing its superiority, especially completeness in 3D object generation. We train and evaluate method in image-to-30 3D reconstruction task and pure 3D generation task. The experimental results reveal that the proposed X-Ray achieves a significant leap forward in the quality of 3D object generation, positioning it as a feasible solution to longstanding challenges in the field. The main contribution of the paper can be summarized as follows:
1. We present X-Ray, a novel 3D representation that encode the whole visible and hidden surfaces in video format through ray casting algorithm for maintaining generalization, accuracy and efficiency.
2. We propose the generative model of our X-Ray via video diffusion model and video upsam- pling model, enabling low-to-high generation of 3D objects from single images.

---

H
H
Figure 2: Samples of our X-Ray 3D sequential representation. Given viewpoint, we capture the attributes multi-layer surface frames, including hit H, depth D, normal N, and color C, in video format. Noted that the number of frames an X-Ray varies depending on the complexity of the 3D) objects. The dotted yellow lines indicate the ray sequence direction.
3.
We showcase the state-of-the-art performance of our X-Ray in 3D generation quality, setting a new benchmark for Image-to-3D modeling.
Related Work
2.1 Representation for 3D Models
Handling 3D data is much more complex and resource-intensivy than dealing with ID (e.g. text and voice) and 2D (e.g. image) data. This complexity makes it imperative find effective ways to organize, process, and infer 3D information. Traditional methods for representing 3D data include meshes, which are good for creating detailed visuals but hard to be generalized; point clouds, which are simple and useful for capturing real-world scenes but lack consistent and dense structure in 3D creation; Although, 3D Gaussian Splatting [21] smooths point cloud data into continuous surface but requires additional an initial point cloud as shape, making them less flexible for 3D synthesis; voxels which are excellent for detailed volumetric data but require much computing resources. Multi-Plas Images [35, 51] try to extent the depth concept to multi-layer with a fixed distance, but they can only describe the visible surface toward camera.
Recent advancements in 3D representation have primarily focused on point-level details and implicit functions, such Occupancy [32], Signed Distance Fields (SDF) [55, 11], Triplanes [10, 16], and Neural representations [36, 21j. These methods have significantly enhanced modeling and rendering capabilities. Occupancy models map the location of any 3D point to its probability of being inside or outside an object, offering a probabilistic approach to shape definition. SDFs [55, 19] refine this concept by quantifying the nearest signed surface distance from any given point, improving the precision of surface representations. Triplanes [3] employ intersecting 21) planes to provide more efficient route to 3D representation, albeit with some detail loss. NeRFs and Gaussian Splatting [21, 36] produce remarkably realistic renderings from a limited number of viewpoints but require extensive computational effort. Despite these advancements, implicit function-base models often face challenges in extracting full and high-resolution 3D features, hindering high-quality generalization. Empirically, representations that focus on surface more efficient, and representations with grid representation are more easily to be generalized [37]. Consequently, capturing all surface attributes and organizing in a dense but lightweight data structure renders our X-Ray an accurate, efficient and generalized representation. Noted that our X-Ray is similar to Depth peeling method [9], which is designed for rendering transparent surfaces, while X-Ray transform any 3D object in video format. Besides, main contribution is using video diffusion generator to generate objects.

---

2.2 Generative Models for 3D Generation
Recent 3D generative models can be primarily categorized into two types: diffusion-based [30, 37, 38] and rendering-basee based [14, 13, 49, 41, 34, 54, 33, 47]. Diffusion-based models fall under direct 3D supervision, while rendering-based models belong to indirect 2D image supervision. Diffusion-basse generative models have emerged as powerful tools for 3D generation, leveraging stochastic diffusion processes to gradually transition from noise to structured 3D objects. These models, such as DPM [30], DiT-3D [37]. and Point-E EE [38], have demonstrated remarkable ability in generating high-quality 3D point clouds. They operate by iteratively refining a random noise distribution into a coherent structure that resembles the target 3D shape, capturing complex geometries and surface details with high fidelity. The strength of these models lies in their capacity to model the distribution of 3D points in continuous space, allowing for the generation of 3D objects with nuanced variations and detailed textures. However, both point-based networks [28, 37, 17] and voxel-based networks [37] is limited by generating high-resolution objects. Another group of methods [41, 34, 54, 33, 47] adopt Score Distillation Sampling (SDS) as prior to train NeRF [36] or 3D Gaussian Splatting [21] for 3D generation. However, is not efficient for optimizing number of different views over short period.
On the other hand, rendering based generative models focus on the visible aspect of 3D generation, transforming abstract 3D representations into detailed and photorealistic images videos. Models such as LRM [14], Open-LRM [13), LGM [48], DMV3D [57], and TripoSR [49] employ advanced rendering techniques to achieve this. However, rendering- based models optimized only for the visible surfaces objects, making it difficult to synthesize the invisible or internal surfaces.
In response to these challenges, our approach utilizes video diffusion model as the foundation for developing 3D X-Ray. This strategy benefits from the strengths of existing video diffusion models while innovatively addressing the limitations of rendering-based techniques, offering more comprehensive solution for 3D generation that is sensitive to both visible and hidden parts of objects.
Our X-Ray Representation
In this section, we detail our X-Ray representation, which includes both encoding and decoding pro- cess facilitate the conversion between 3D mesh formats and our X-Ray representation. Specifically, the encoding process converts a 3D mesh into our proposed X-Ray format, and the decoding process converts our X-Ray back into a 3D mesh
3.1 Encoding
Given 3D object under an arbitrary camera view, we apply the ray casting algorithm to encode a 3D object mesh into the proposed X-Ray representation. The ray casting algorithm plays crucial role in both computer graphics and computational geometry, where it is used for scene rendering, visibility determination, and addressing geometric queries. Specifically, a ray is emitted from camera center into the environment and all the interactions of this ray with target 3D objects are captured sequentially. For each ray r E R within the field of view that intersects with L sequential faces in the mesh we record their 3D attributes which include depth (distance camera center) D, = (d,, d2 ..., dz) R4x1, normal N, = (nj, n2 n2) E R and color C, = (C1, C2 .... CL) E R To indicate surface presence,
denote Hit H, (h1, h2. h2) E R to indicate whether there is surface.
Since L is usually very small (Sec. 5.3), we denote the efficient and thin grid voxels E If XW xWx.xx as the object representation, where the ray X.j with image coordinate [i, can be represented as:
x = C,) E
Note that X(i.j. k] = O when there is no surface for the th layer at the image ray coordinate [i Examples of X-Ray encoding are shown Fig. Through the encoding process, we can transform any mesh into sequential representation with varying lengths, same as a video with different numbers of frames. Finally, we transpose the voxels as X-Ray X Lxxxxxxxx XxW resembling video format with L frames, each with resolution of H x W and 8 feature channels.

---

step
W Extling "
Exii
M
Point Cloud
Spetial Temporal 3D UNet
Down
Mid
Up
X-Ray
Upsampler x
X- Ray
Screened
1:7768
Image
Embedding
24"7""
W.W.
Mel
Iaput Image
(a) X-Ray Diffusion Model
(b) X-Ray Upsampler
(c) Mesh Decoding
Figure 3: Overview of our proposed generative pipeline for the X-Ray 3D representation. There three main components: (a) The X-Ray diffusion model, which generates a low-resolution X-Ray from image input. (b) The upsampler, which enlarges the low-resolution X-Ray into 4x high resolution. (c) The mesh decoding model, which decodes the high-resolution X- X-Ray into a point cloud with color and normal, and then converts it into the final generated mesh.
3.2 Decoding
The decoding process converts the X-Ray representation back into 3D mesh. To achieve this, we first convert the video format of X-Ray into a point cloud and subsequently apply the Screened Poisson algorithm [20] to transform the point cloud into a 3D mesh.
X-Ray + Point Cloud. Given an X-Ray, we first compute the 3D object's point cloud P. (Px, P Pn. for Ray including location Px color P and normal Pn defined by the equation:
Px = D, rd. PA=N, P = Cr, when H. 11=1.
To and ra denote the origin and direction the camera ray, respectively. Furthermore, D,, Nr. C, and H, the depth, surface normal, color and hit attributes of the ray defined in Eq. Upon processing all camera rays, we obtain comprehensive point cloud P (Pr)vER representation that includes location, normal, and color attributes of the 3D object.
Point Cloud + Mesh. The Screened Poisson algorithm [20] for converting point clouds with color and normal into 3D colored meshes is classic method that leverages the mathematical principles of the Poisson equation. The core idca involves solving a variation of the Poisson equation to interpolate smooth surface that fits the input point cloud. The Poisson equation is partial differential equation of the form: V2Φ- = f, where 2 denotes the Laplace operator (which represents the divergence of the gradient of function), ø is the potential field to be solved, and is scalar function representing the divergence of the vector field derived from the input point cloud. In the context of point cloud 3D mesh conversion, the algorithm first employs the given normal to define a vector field that suggests the orientation of the surface at each point. The divergence of this vector field serves as the function f the Poisson equation.
Encoding Bacoding-Decoding Intrinsic Error. minor reconstruction error that varies with the number of layer L and the frame resolution (H, W). To explore this, we conduct an experiment in Sec. 5.3 aimed at analyzing these variables to identify their optimal values. Our goal is to achieve balance where all pertinent information is preserved while maintaining a lightweight model.
The secoding-decodding process will introduce a intrinsic and
X-Ray for 3D Generation
Our primary objective of introducing a new 3D representation model is to facilitate the generation of 3D objects from single images. The challenge lies in accurately predicting the characteristies that are not immediately visible on the first surface when only single image is available. To overcome this challenge, we utilize diffusion and upsampling models for X-Ray synthesis. Given that our proposed X-Ray in video format, we leverage advanced Video Diffusion models as our backbone. To exploit this structure for high-resolution X-Ray synthesis, incorporate principles from advanced video
5

---

diffusion models as our foundational framework. Notable models in this domain include Stable Video Diffusion (SVD) [2], VideoFusion [31], and the state-o--the-an Sora. To efficiently train the diffusion model, we begin by training low-resolution X-Ray diffusion model that generates X-Ray from a single image. Subsequently, we employ an upsampler to enhance these synthesized X-Rays to high resolution. This two-step approach ensures a more manageable and efficient training process, gradually improving the quality of the output.
Framework Overview. Fig. 3 presents an overview of our generative model using X-Ray rep- resentation (ef. Sec. 3.1) for 3D generation. Our X-Ray diffusion model (cf. Sec. 4.1) operates at the core of the framework, transforming random Gaussian noise into a low-resolution X X-Ray representation conditioned by an input image. These low-resolution X-Rays are then enhanced to high resolution through the application of a 3D Temporal Upsampler (cf. Sec. 4.2). Fi- nally, the high-resolution X-Rays are decoded into 3D) meshes using a combination of point cloud transformation and the Screened Poisson algorithm (ef. Sec. 3.2).
4.1 R-Ray Diffusion Model
Diffusion models [43] are generative models that transform a random noise distribution into a data distribution through reverse process, counteracting a forward process that incrementally adds Gaussian noise to the data. The forward process a Markov chain described by 2 = v /afT-11 + v1 -1-au a1, where zy represents the data at step t, a controls the noise level, and e ~ _V I) is sampled noise. The reverse process, aimed at reconstructing the original data from noise, is modeled by a neural network predicting the noise added at each step or directly denoising the data, following
بق c (2), with (x2,)) being the predicted noise. Training involves VT
Tt-1 It
optimizing the network to minimize the difference between the original and reconstructed data, effectively learning to invert the noise addition process given by:
Lam Er,.),(0,11.. [lle (zt,,))))) +)]
(3)
where is uniformly sampled from the set (l....,7 T).
Diffusion Model for X-Ray. A prevalent technique in diffusion models is the utilization of latent spaces with VQ-VAE [7] perform the initial data transformation to compress the data. This method poses a significant challenge for our X-Ray representation as it requires the development of a VQ-VAE model from scratch due to the absence of a suitable off-the-shelf latent model for X-Ray, which would consequently increase our training burden. Another promising approach for efficiently training high-resolution generators is the cascaded synthesis pipeline. This method, exemplified by works such as Imagen [44], DeepFloyd IF [1], and Stable Cascaded [40], involves progressively training the diffusion model or upsampling network from lower to higher resolutions. Given our limited computing resources, we opted to implement this cascaded upsampling strategy. This technique facilitates a more gradual and controlled enhancement of X- X-Ray quality, providing a more flexible and efficient alternative to traditional latent space diffusion models.
the Spatial-Temponte 3D U-Net network from Stable Video Diffusson [2] for our diffusion model to generate low-resolution X-Rays, with modifications to the input and output channels. As shown in Fig. 3, the input image is sent the encoder E, producing an image latent via image VAE [7] and an embedding e via a ViT [5]. is concatenated with the X-Ray latent as input, and interacts with the 3D U-Net through cross attention [2] to finally output the denoised latent. This model employs spatial-temporal attention mechanisms to alternately extract features from 2D frame spaces and ID surface layer sequences, enhancing its ability process and interpret the different layers of the X-Ray. This approach allows for nuanced handling of the temporal information inherent in sequential X- X-Ray data, crucial for achieving high-quality diffusion results.
Specifically,
4.2 X-Ray Upsampler
The X-Ray upsampler focuses on enhancing previously generated X- X-Rays to a higher resolution. We considered two potential methods: point cloud up-sampling and video up-sampling. Encoding low-resolution X-Rays into point cloud with color and normal information is straightforward (see See 3.1). However, point cloud up-sampling often increases only the number of points without effectively enhancing attributes like texture and color due to its unstructured nature. To improve

---

efficiency and consistency, we adopted a video up-sampling approach using a decoder from Stable Video Diffusion (SVD) [2]. We concatenate the previous image latent with the low-resolution X-Ray as input and output the high-resolution X-Ray. The model upsamples previously synthesized low-resolution X-Ray frames fourfold while preserving the original layer number I.. applies attention at both the 2D surface frame and 1D surface layer levels. enhancing frame resolution and overall quality. This makes the X-Ray diffusion model, followed by the upsampler, a more integrated and effective solution.
VAE
Loss. The loss function for the Upsampler differs notably from that of the diffusion model. While the diffusion model loss typically addresses volumetric or textural aspects. the Upsampler loss concentrates specifically on the surface area accuracy, reflecting the critical importance of maintaining high fidelity in the enhanced images. The loss function we use for the Upsampler is given by:
Lup = X (gg
X.p
H Hgt]l
H gt H
where X.e [gg] represents the ground-truth high-resolution X-Ray at hit surface and X Hgtt] denotes the Upsampler's output at hit surface, and Hgt. Hup denotes the ground-truth and upsampled Hit, respectively. The loss is computed the squared Euclidean distance between these two matrices, quantifying the pixel-wise discrepancy in surface details. This metric effectively ensures that the upsampling process preserves essential surface features, thereby optimizing the quality and utility of the resulting high-resolution X-Ray.
Experiments
5.1 Dataset and Implementation
Datasets. We train our X- X-Ray pipeline using a subset of the Objaverse dataset [4], which have been removed entries with missing textures and inadequate prompts as outlined in [48]. This subset consists of more than 60,000 3D objects. For each object, we select 8 random camera views, covering azimuth angles from -180 to 180 degrees and elevation angles from 0 to 45 degrees with camera distance to object center fixed at 1.2. The images are then rendered using Blender Software, and the corresponding X- X-Rays are generated through the ray casting algorithm provided by the trimesh library [50]. Through these processes, we create a dataset of approximately 480,000 paired images and X-Rays to train the generative model. For the evaluation datasets, we adopt two commonly adopted datasets: Google Scanned Objects [6] and OmniObject3D [56], to assess generative performance via single-view reconstruction tasks.
Metries. Recent 3D generative models [26, 18, 27, 14, 49] lack unified reconstruction evaluation metrics due to the challenge of determining an object's size and orientation from single image [46]. To ensure fair comparisons with state-of-the-an -he- methods, we align all methods before evaluation. The predicted and ground-truth 3D objects will be normalized three axes and face forward the same z axis. We then align them using the Iterative Closest Points (ICP) algorithm and calculate Chamfer Distance (CD) in L1 norm and F-Score (FS) at threshold 0.1 (FS @..1) 1) for reconstruction.
range of -0.5 0.5 along all
Implementation Details. Our X-Ray diffusion model is based on the Seaprial-Temppral 3D UNet architecture used in Stable Video Diffusion (SVD) [2], modified to synthesize 8 channels: 1 hit channel, depth channel, 3 color channels, and normal channels, compared to the original channels. During training, we maintain a learning rate of 0.0001 using the Adam optimizer. Since different X-Rays have varying numbers of layers, we pad or truncate them a uniform layers for efficient batching and training. Each layer's 's frame has dimensions of 64 x 64. For the upsampler, each layer's 's output remains 8 channels, but the resolution of each frame is increased to 256 x 256 to enhance detail and clarity in the upscaled X-Ray. The entire training pipeline is conducted on NVIDIA 100 GPU servers for two weeks. During inference, the 3D generation process takes approximately seconds: about 1 second for the diffusion model, 1 second for the upsampler, and 5 seconds for mesh decoding. As for GPU usage during inference, the GPU memory required 4.8 GB for X- X-Ray diffusion model and 2.5 GB for X- X-Ray Upsampler.
7

---

Table Comparison with other 3D representations Efficiency.
Multi- View Depths
MPI
Point Cloud
X-Ray
Metric
Grid
(8 views)
(8 planes)
(200,000 points)
(8 layers)
Memory (L)
67.09 MB
1.57 MB
1.57 MB
0.90 MB
0.62 MB
Encoding Method
Voxlization
Rendering
Slicing & Rendering
Sampling
Ray Casting
Encoding Time (U)
0,105 s
0.045 s Fusion & Poisson
0.049 s
0.013
0.016s
Decoding Method
Poisson
Poisson
Poisson
Poisson
Decoding Time (L)
~ s
110 1.1c-2
~
~ s
~ s
CD(L)
7.7c-3
8.9e-3
7.2e-3
7.8c-03
The Bacoding-Decoding Intrinsic Error Sf Layer Number (L)
Chamfer Distance (CD)
6.55.-02
1.93--02
9.60e-133
K.10--1
7.79c-8
7.44-48
Surface Layer Number (L)
(a) The intrinsic error of layer number (T.), when H W 256
The Encoding Decoding Intrinsic Error of Frame Height (IT) Width (W)
Chamfer Distance (CD)
5.290-12
2.78e412
1.47c-02
7.7%-4
7.74.43
7:01c-03
32
256
512
1604
Frame Height (II) Width (W)
(b) The intrinsic error of frame height (H) or width (W) when =
Figure 4: The encoding-decoding intrinsic error of different frame resolutions and number of layers.
5.2 Efficient Comparison with Different 3D Representation
We compared the efficiency of different representations using 500 3D meshes from random selected models. The results showed that both point cloud and X-Ray were highly efficient, with lower memory, faster encoding & decoding times. However, the X-Ray had the advantage of being reorganizable as a video format for diffusion models, leading to better performance.
5.3 Analysis
Intrinsic Error
Due to the finite number of layer and resolution H, W of X-Ray, a slight intrinsic error is inevitable during the encoding of 3D meshes into X-Ray format and the subsequent decoding back into 3D meshes. To quantitatively assess this error, we conducted an experiment to evaluate intrinsic error via Chamfer Distance (CD) (L1 norm) between the original 'ground-truth mesh and the encoding- decoding mesh across various resolutions and layers. In the experiments, we set the frame resolution through series of predefined values: 32, 64, 128, 256, 512, and 024 and vary the number of layers from to 12. Fig. 4a indicates that the intrinsic error decreases as the number of layers in our X- X-Ray representation increases, becoming convergent after layers. Similarity as illustrated in Fig. 4b, the intrinsic error decreases with increasing resolution and stabilizes after 256. Therefore, for a balance between accuracy and efficiency, we use a resolution 8 256 256 with H W 256 and in our experiments. Compared with the dense volume achieving a 256 x 256 x 256 resolution for voxel-based methods, our X- X-Ray representation significantly efficient for focusing only on surfaces and reducing the data volume by 96.8%%
8

---

Input Image One-2-3-4-5
OpenL.MM
TripoSR
Ours
GT Mesh
GSO
OmniOhjeet33
ShapeNet
Figure 5: Quantitative Comparison Image-to-31l Generation.
5.4 Quantitative Comparison
For image-to-3D mesh generation on the GSO [6] and OmniOhject3D [56] datasets, we build a new benchmark that randomly selects 500 image-mesh pairs from each dataset. We re-ran the baselines using their released source code and used normalized Chamfer distance and F-score for fair comparison. The results, summarized in Tab. show that our X-Ray method achieves significant improvements over all previous rendering-base datasets, with relative 33% improvement (0.084 0.056) Chamfer distance and also significant improvement in FS@0.1 (0.878 - 0.973, where the maximum is 1) on GSO dataset and similar performance on OmniObject3D dataset. This demonstrates the superiority of our approach over rendering-basee based methods.
state-of-the-ar methods [25, 18, 8, 13, 49] on both
Table 2: Quantitative reconstruction comparison the GSO [6] and OmniObject3D [56] datasets
Datasets
Metries
One-2-3-45 [25]
ZeroShape
TGS
OpenLRMI [13]
TripoSK
X-Ray (Ours)
CD ]
0.175
0.136
0.096
0,143
0.084
0.056
GSO [6]
FS@0.1
0.465
0.627
0.803
0.621
0.878
OmniObject3D [56]
CD J
0,187
0.138
0.091
0.148
0.080
0.054
FS @50.1
0.490
0.619
0.822
0.664
0.892
0.972
5.5
Qualitative Comparison
qualitative comparison effectively demonstrates the advantages of our proposed X-Ray generation method. Using the three datasets mentioned, selected single images input and generated 3D meshes without and with textures, as shown Fig. 5. Our proposed method has three key advantages: 1. It can decompose shape and appearance to accurately reconstruct flat surfaces (Rows 1 and 3); 2. It can detect the sealing of containers (Rows 2 and 4); It can generate the internal structure objects within their outer surfaces (Rows 5 and 6). These obvious advantages highlight the our effectiveness.
5.6 Failure Cases.
The failure cases highlight the limitations of the current generative model based on X-Ray representa- tion when the number of frame layers is very large. As illustrated in Fig. 6, given an input image containing a complex object, such as an exquisite hamburger, the number of frame layers I. of the encoded generated X-Ray tends to exceed the maximum length of 8. Consequently, any surface

---

behind layer 8 will be omitted, resulting in missing parts of the reconstructed mesh. The solution is increase the value of that the X-Ray can represent more surfaces. However, this would also increase computing resource requirements. We will reconsider the sparsity of deeper surface frames and propose more efficient generative model to overcome these failure cases.
Iaput Image
Generated X-Ray
Generated Mesh
Ground-Trath Mesh
Figure 6: Failure cases. The generated meshes will miss behind parts because of the limited number of frame layers.
6 Conclusion
In this work, we introduced a novel X-Ray representation for 3D objects that encompasses both visible and hidden surfaces within the camera's field of view, unlike recent rendering- based methods that typically focus only on the visible surface. We demonstrated the effectiveness of the X-Ray approach in single-view 3D generation tasks. Our generative model shows that the underlying generator for X-Ray shares foundational similarities with existing video diffusion models, allowing us to leverage their advantages. Experimental results highlight the outstanding performance of method.
Limitations. Our generator uses the Stable Video Diffusion (SVD) pipeline to produce high-quality
X- Ray. However, the X-Rays generated consist of an uncertain number of sequential layers, with the posterior layers tending to more and more sparse, which can redundant. Additionally, the generated mesh lacks enough smoothness and has missing part when X-Ray is truncated. We plan to explore advanced network architectures, such as Large Language Model, that can better handle the complexities of X-Ray data, including layer sparsity and sequential format. Additionally, aim to investigate further applications of the X-Ray representation to broaden its utility and impact in 3D modeling and beyond.
Acknowledgement
This research project is supported by the National Research Foundation (NRF) Singapore, under its NRF-Investigatorshiip Programme (Award ID. NRF-NRFI09-000S),
References
[1] DeepFloyd Lab at StabilityAI. DeepFloyd IF; novel state-of-the-an open-source text-to text-to-image model with a high degree of photorealism and language understanding. btttps://ww/ deepfloyd ai/ deepfloyd-it if, 2023. Retrieved 2023-11-08
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, Varun Jampani, and Robin Rombach. Stable video diffusion: Scaling latent video diffusion models to large datasets. CoRR, 2023.
[3] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-ave aware 3D generative adversarial networks. In CVPR, 2022.
10

---

[4] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. arXiv preprint arXiv:2212.00051, 2022.
[5] Alexey Dosovitskiy. Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakoh Uszkoreit, and Neil Houlshy. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
[6] Laura Downs, Anthony Francis. Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas Barlow McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation, ICRA 2022, Philadelphia, PA, USA, May 23-27, 2022, 2022.
[7] Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution image synthesis, 2020.
[8] Zi-Xin Zou et. al. Triplane meets gaussian splatting: Fast and generalizable single- view 3d reconstruction with transformers. arXiv preprint arXiv:2312.09147. 2023.
[9] Cass W. Everitt. Interactive order-independent transparency. In NVIDIA, 2001.
[10] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, Zan Gojeic, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In Advances In Neural Information Processing Systems, 2022.
[11] Wenhang Ge. Tao Hu, Haoyu Zhao, Shu Liu, and Ying-Cou Cong Chen. Ref-neus: Ambiguity-rddused neural implicit surface learning for multi-view reconstruction with reflection. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4251 12560, 2023.
[12] Jun Young Gwak, Christopher B Choy, and Silvio Savarese. Generative sparse detection networks for 3d single-shot object detection. In European conference on computer vision, 2020.
[13] Zexin He and Tengfei Wang. Openlmm: Open-source large reconstruction models. https: //gthbbb.
com/ Han//DTopiaa//o /OpenLRM, 2023.
[14] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. LRM: large reconstruction model for single image to 3d. CoKR, has/2311.04400 2023.
[15] Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientnerf efficient neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 12902 12911, 2022.
[16] Tao Hu, Xiaogang Xu, Ruihang Chu, and Jiaya Jia. Trivol: Point cloud rendering via triple volumes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20732-20741, 2023.
[17] Tao Hu, Xiaogang Xu, Shu Liu, and Jiaya Jia. Point2pix: Photo-realistic point cloud rendering via neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). pages 8349 8358, 2023.
[18] Zixuan Huang, Stefan Stojanov, Anh Thai, Varun Jampani, and James M Rehg. Zeroshape: Regression- based zero-shot shape reconstruction. arXiv preprint arXiv:2312. 2023.
[19] Ka-Hei Hui, Aditya Sanghi, Arianna Rampini, Kamal Rahimi Malekshan, Zhengzhe Liu, Hooman Shayani, and Chi Wing Fu. Make-a-shape: ten-million-scale 3d shape model. CoRR, 2024.
[20] Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM Trans. Gruph., 2013.
[21] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
[22] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg. Wan-Yen Men Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.00 026433 2023.
[23] Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan- Cao, and Ying Shan. Advances in 3d generation: A survey. CoRR, abs/2401.17807,, 2024.
11

---

[24] Lingjie Liu. Jiatao Gu, Kyaw Zaw Lin, Tat-Seng Chua, and Christian Theobalt. Neural sparse voxel fields. NeurIPS, 2020.
[25] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T. Zexiang Xu, and Hao Su. One-2- 3-45: Any single image to 3d mesh 45 seconds without per-shape optimization. Advances in Neural Information Processing Systems, 36, 2024.
[26] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3: Zero-shot one image to 3d object, 2023.
[27] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingje Liu, Taku Komura, and Wenping Wang. Syncdreamer: Generating multiview-consister images from a single-view image. arXiv preprint w091v:3399.33533, 2023.
[28] Zhijian Liu, Haotian Tang, Yujun Lin, and Song Han. Point-voxel CNN for efficient 3d deep learning. In NeurIPS, 2019.
[29] Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J. Black, and Bernhard Schölkopf. Ghost on the shell: An expressive representation of general 3d shapes. ICLRR 2024.
[30] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. CVPR, 2021.
[31] Zhengxiong Luo, Dayou Chen, Yingya Zhang. Yan Huang, Liang Wang. Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[32] Lars Mescheder, Michael Occhsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In CVPR, 2019.
[33] Gal Metzer, Flad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape- guided generation of 3d shapes and textures. arXiv preprint arXi::2211.066A0 2022.
[34] Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. arXiv preprint arXiv:2112.032211 2021.
[35] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Caiz-, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019.
[36] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren
Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[37] Shentong Mo, Enze Xie, Yue Wu, Junsong Chen, Matthias Nießner, and Zhenguo Li. Fast training of diffusion transformer with extreme masking for 3d point clouds generation. arXiv preprint arXiv: 2312.07231. 2023.
[38] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-e: A system for generating 3d point clouds from complex prompts. CoRR, abs/2212.08751, 2022.
[39] William Pecbles and Saining Xic. Scalable diffusion models with transformers. In ICCV, 2023.
[40] Pablo Pernias, Dominic Rampas, Mats L. Richter, Christopher J. Pal, and Mare Aubreville. Wuerstchen: An efficient architecture for large-scale text-to-image diffusion models, 2023.
[41] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv, 2022.
[42] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Lconidas J. Guibas, Pointnet++: Deep hierarchical feature learning point sets in a metric space. In NeurlPS, 2017.
[43] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022, 2022.
[44] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L. Denton, Seyed Kam- yar Seyed Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, Jonathan Ho, David J. Fleet, and Mohammad Norouzi. Photorealistic text- to-image diffusion models with deep language understanding. In NeurIPS, 2022.
12

---

[45] Sara Fridovich-Kea Keil and Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022.
[46] Kyle Sargent, Zizhang Li, Tanmay Shah, Charles Herrmann, Hong-Xing Yu, Yunzhi Zhang. Fric Ryan Chan, Dmitry Lagun, Li Fei-Fei, Deqing Sun, and Jiajun Wu. ZeroNVS: Zero-shot 360-degree view synthesis from a single real image. arXiv preprint arXiv:2310. 2023.
[47] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaussian splatting for efficient content creation. arXiv preprint arXiv:2309. 16653, 2023.
[48] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng. and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint arXiv:2402.0503: 2024.
[49] Dmitry Tochilkin. David Pankratz, Zexiang Liu, Zixuan Huang, , Adam Letts, Yangguang Ding Liang, Christian Laforte. Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction from a single image. arXiv preprint 0031:1240, 2024.
[50] Trimesh. Trimesh [computer software]. https://github. com/mikedi trimesh, 2019.
[51] Richard Tucker and Noah Snavely. Single-view view synthesis with multiplane images. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2020.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E. ukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrlPS, 2017.
[53] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts. David Pankratz. Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: novel multi-view synthesis and 3d generation from a single image using latent video diffusion. CoRR, abs/2403. 12008, 2024.
[54] Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. arXiv preprint 4rXiv:2212.00774, 2022.
[55] Peng Wang. Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for nulti-view reconstruction. Neur/PS, 2021.
[56] Tong Wu, Jiarui Zhang, Xiao Fu, Yuxin Wang, Liang Pan Jiawei Ren. Wayne Wu, Lei Yang, Jiaqi Wang. Chen Qian, Dahua Lin, and Ziwei Liu. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In CVPR, 2023.
[57] Yinghao Xu, Hao Tan, Fujun Luan. Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gor- don Wetzstein. Zexiang Xu, and Kai Zhang. DMV3D: denoising multi-view diffusion using 3d large reconstruction model. CoRR,
2023.
[S8] Alex Yu, Ruilong Li, Mathew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. PlenOctrees for real-time rendering of neural radiance fields. In ICCV, 2021.
13

---

A
Appendix
The Appendix section contains detailed information on the network structure (including the X-Ray diffusion model and upsampler), ablation studies, additional experimental quantitative and qualitative results, failure cases, and key source code.
A.1 Network Details
X-Ray Diffusion Model As outlined in our study, utilized the 3D Spatial-Temporal UNet the foundational architecture for our diffusion model. For image-to-3D generation, the input includes 4-channel image latent and 8-channel noise X- X-Ray and the output has 8 channel to denoise X- Ray. Initially, we loaded all network parameters and fine-tuned the model on our training dataset. Despite these efforts, the final performance fell short of expectations. We hypothesized that this underperformance stemmed from the limited size of our training dataset, which was insufficient for such parameter-rich model, well as substantial domain gap between the original video data and our X-ray data.
To address these issues, we reduced the network size to 10% of its original configuration and trained model from scratch using a significantly larger batch size. This approach proved highly effective, the final performance exceeded the current state-of- state-of-the-arr the-art methods by considerable margin. Consequently, our findings suggest that a smaller, more focused network trained from scratch can be more effective than larger pre-trained model when faced with limited and domain-specifi datasets. This approach not only enhances performance but also highlights the importance of customizing the model size and training strategy the specific characteristics of the data. Our results indicate that careful consideration the network architecture and training regimen is crucial for optimizing performance, particularly in specialized applications such as X-ray generation. This study provides valuable insights into model adaptation and training strategies that can be applied to other domains facing similar challenges.
The following JSON file contains the configuration details for our X-Ray Spatial-Tempont UNet.
__class_namee
aeciiiinnttmmmemmeeeeee dim": 256,
64,
128, 256, 256
],
10
" "cross__ttennion_ddi" 1024, "down_bbock_typees":
11
12
"CresattaDonBBcccspttttppppap
15
*DornBlockkpattiofemppaall
16
},
17
"in_channele 12,
Clatent_channelll 8,
"layers_per_bloc" 1,
20
"nun_attention_leads
21
4,
22
8,
16,
24
25
],
26
"num_frames":
"out_channels":
768,
29
"sample_size"; 64,
30
"transtorer__ayessppp_. 1,
hxp_block_types""
"IpBlockSpatioTeppraa""
14

---

"Grossttttlgllocssssiitpppppa
35
36
37
A.2
X-Ray Upsampler
The X-Ray diffusion model enables the generation of low-resolution 3D objects. To enhance resolution and improve performance, our X-Ray upsampler increases the frame resolution by factor of 4. For image-to-3D generation, we concatenate the 4-channel image latent representation with the 8-channel low-resolution X-Ray, producing 8-channel high-resolution X-Ray.
This process begins with the diffusion model creating a coarse, low-resolution 3D representation that captures the essential structure of the object. The upsampler then refines this representation. significantly improving the resolution and adding finer details that contribute to the realism of the 3D model. By combining the latent image features with the initial low-resolution X-Ray data, ensure that the final high-resolution output retains the context and nuances of the original image while also incorporating the detailed structural information provided by the X- X-Ray data.
The following JSON file contains the configuration details for our X-Ray Upsampler.
128, 256,
512
"down_blockktypes":
"DonEnnoderBlock2""
10
"DomEncolerBlock2"""
"DownEacoderBloc220
12 ]
13
"force_upcast" false,
14
"in_ "in_channels": 4,
15
"Latent_channels" 12,
'layers_per_block": 2,
17
"out_channels"; 8,
18 "sample_aize" 768,
19
accalingffcctor 1.0
20
}
A.3 Ablation Studies
The Effect of Diffusion Model
As described in Sec. A.I, train the diffusion model with different configurations, including varying model sizes and how to conduct initialization. Specifically, we evaluate three models:
1. Finetuned Original UNet
2. Randomly Initialized Original UNet
3. Randomly Initialized UNet with 10% Parameters
The experimental evaluation results on the GSO [6] dataset are illustrated in Tab. 3. These results allow us analyze the impact of different initialization and scaling strategies on the performance of the diffusion model. By comparing the outcomes, we can identify the trade-off's between model size, training time, and overall accuracy. We observed that finetuning the model did not introduce significant improvement because of the domain gap between video and X-Ray data. Also, it is not necessary to adopt large diffusion model for Objeverse dataset [4] for significantly reducing the batch size. Thus, adopt the randomly initialized UNet with 10% parameters
diffusion
15

---

Inpet
Spadhrstodd X-Ray
Encoded Claud
Meck
Figure 7: Visualization of Image-to-3D Generation from X-Ray.
model. These findings provide insights into the optimal configuration for diffusion models in 3D related tasks.
Table 3: Quantitative reconstruction comparison in different diffusion model configurations
Model Contigunstoms Finetuned Original UNE
CD
FS200
Traiiig Time (days)
Inference Time (secodds)
Barch Size (GB)
L095
0.12
6.1
Randomlly Initilized Driginal UNe
0,99
0306
6.1
Randomly Inisalized UNet with 10% Parameters
0,056
0,973
7
24
0.6
The Effect of Hit H
The original surface attributes only contain depth D, normal N, and color We add an additional Hit H attribute to indicate whether there is surface. In this ablation study, we demonstrate the necessity of including the Hit H attribute. We conduct training experiments with and without the Hit H attribute and evaluate the GSO [6] dataset. The results, shown in Tab, 4, indicate that including the Hit H attribute can improve performance. The reason might be that the X-Ray is sparse and requires indicator balance the generative process. The UNet model and Upsampler with the Hit H attribute achieves better CD and FS@0. @0.1 scores, demonstrating its importance in accurate 3D generation.
Table 4: Quantitative evaluation of the effect of Hit H attribute on the GSO [6]
dataset
Diffusion Model
CD J.
FS@0.1 t
Upsampler
CD
FS@0.1
wo. w/ Hit H
0.074
0.901
wo. Hit 0.060
0.956
0.068
0.934
w/ Hit H
0.056
0.973
A.4 More Visualization
We shown more Visualization results single-view image generation in Fig. Also, we extend the task to generate 3D object from text prompt. Text-to-3D mesh generation can also be achieved via using text-to-image, object segmentation, and image-to-3D processes. We utilize established diffusion models that are already proficient in image synthesis from textual descriptions instead of
16

---

Synthesized Imagee
Segmented Image
Sputhesized X-Rayy
Encoded Point Cloud
Decoded Mesh
black and silver power supply"
"greere wilne bottle"
"a polor beur
"a rolled haysstcck
Figure Visualization of Text-to-3D Generation from X-Ray.
developing a new text-conditioned diffusion model. One Model such as Stable Diffusion [43]. Stable Cascaded [40], DiT [39] are employed generate images based the input text. Following this, we apply an image segmentation tool, specifically the Segment Anything Model [22] to climinate the background. This streamlined method avoids the complexities of training a new model from scratch instead of making use of sophisticated pre-trained models to handle the text-to-image translation, thereby simplifying the process generating 3D meshes from textual inputs. The output results of Image-to-3D and Text-to-3D are illustrated in Fig. 8.
A.5
Key Source Code
Ray Casting: The source code of ray casting the mesh get the point cloud.
The source code ray casting the mesh to get the point cloud.
unn
from trimesh. ray ray_pyembree import RayMeshInterseetor
def ray cay__cas_meeh (mesh, rays_origins, ray_ diyeetectti::):
intersector = RayMeshinteraector (mesh)
index_triangle index_ray, point_cloud = intersector. intersects_id(
10
ray eey_orrigan--rays.origl
11
ray soy.diieetionsrrsyy.drrcttoo.s multiple.hite=Tin return_ Isturn_locations-Truu return index_triangies index_ray, point_cloud
12
X-Ray to Point Cloud: The source code of transfering X-Ray Point Clouds with normals and colors.
The source code of transfering X- X-Ray to Point Clouds with normals and colors.
17

---

import numpy as np import open3d as o3d
def get_rays (directions, c2w):
Rotate ray directions from coordinate the world
coordinatee
10
rays_d directions a c2wl:3, :3].T # (H, W, 3) rays_d rays_d / (np linalg norm (rays_ss axis--1, keepdims-True) 1e-8)
12
13
The origin of all rays is the camera origin world coordinate rays_o np. broadcast_to (c2u [:3, 3], rays_d d. shape)
14
(H, W, 3)
return rays_o, rays_d
17
18
def X_Ray taay_to_Piint_Coe Cloud (XRay)
19
Converts X R-yay data to point cloud with normals and colors.
22
XDepthe, XNormale, XColors, XHits I:ay[:, 0:11, I:, 1:4], XRay [:, 4:7], I: 7:8]
comera_angle_t 0. 0.8575660450553894
25
image_width XDepths shape [-1]
26
image_height XDepths shape [-2]
27
fx 0.5 image_width up. tan (0.5 camera_
29
rays_acreen_ up mg.igrid [o: imgee 0:image_witth reshape
30
2, image_height image_vidth) # [h, w, 2]
grid rays_screen_coords reshape amigg_heiigh. width, 2)
33
34
cx = height 2.0
width 2.0
35
i, j grid..... 11, gridl.... 0]
38
39
directions = mp stack ([(i cx) / fx, -(j cy) / fx, -up
ones like (i)], -1) # (H, W, 3)
41
n eye astype (np float32)
42
43
rays origins,, ray_directions get_ net_rays (directions. c2x) rays_origins rays_origins [None] repeat (XDepths shape [0] 0) ray ray_directiona ray. ray.directions [None] repeat (XDeptha.. shape [0], 0)
46
47
XDepths XDepths. Depttss.transpose (0, 2, 3, 1)
XNormals = XNormals transpose (0, 2, 3, 1) 48
IColors IColors. transpose (0, 2, 3, 1)
51
rays_ origins vaya_origins [XHits]
52
ray direccioonnns = ray_directions [XHits]
XDepths XDepths [XHits] [XHits]
normals XNormals[XHite [XHits]]
55
colors IColors [XHits]
56
xyz rays_origins ray_directions XDepths
57
convert to open3d point cloud xyz xyz byz.rem reshape normals normals. reshape (-1, 3)
60
61
colors colors. (-1, 3)
18

---

pcd = o3d. geometry. PointCloud
pcd. points o3d. utility. Vector3dWector (zyz) (xyz)
65
ped normals o3d. utility. Vector3dVector (normals)
66
ped. colors o3d. utility. Vector3dVector (colorss)
67
return ped
Point Cloud to Mesh: The source code of transferring the predicted point cloud to mesh using Poisson Surface Reconstruction.
The source code of transferring the predicted point cloud to mesh using Poisson Surface Reconstruction.
import open3d o3d
Load point cloud
pcd o3d. io read_ peadd_ponttclouu ("path_to_ cow.__ooiittcloueeeee ply")
def poisson surfacurfucooo
12
13
Converts a point cloud to a mesh using Poisson Surface
Reconstruction
15
Ensure the point cloud has normals
16
not pcd. () pcd.estimate_normal..
17
Perform Screened Poisson Surface Reconstruction mesh, densities o3d. gedgeteeyrr.... TriangleMesh. create_ ltto_/froooooottllleeeee clodd_poissoppoo ped, depth=9, width=0, acale=1.1, finear_ii==aale
20
21
24
Optionally crop the mesh using the density values remove low density areas
# You can adjust the threshold based your requirements vertices_to_rrment 0.01 mesh.
27
densities density. baatessbb_mmsskeeee mask (verticcs
28
29
Assign colors to the meah
meah. vertex,colors utdlutylity Vector3dVector (pcd. colors) return mesh
32
The source code of evaluating the predicted mesh using Chamfer Distance and F-Score.
import numpy as np from ocipy.spatian spatial import cKDTree
def chanfer_distaneee fnd. (P. q, chreshold=0.1):
***
10
Calculates the Chanfer Distance and F-Score between two point clouds.
kdtree_P cKDTree(P) kdtree_ cKDTree
13
14
15
dist_P_to_p kdtree_P query (Q)
List_Q_to_P, kdtree_0 query
querry (P)
19

---

The source code normalize and align the predicted point cloud to chamfer_dist np. (dist_P_to_Q) mp. (dist_Q_to_F)
19
20
precision np (dist_ Poto_ threshold)
recall np mean mean mean threshold)
precision recall 0:
24
f_score (precision recall) / (precision recall) elae:
25
26
f_score 0.0
28
return Chamfer_dist, f_score
Normalized Metrics: The source code normalize and align the predicted point cloud to the gt point cloud before evaluation.
the
gt point cloud before evaluation. unn
import numpy as np
import open3d as o3d
def ray_ cayt_aasmmmell (mesh, rays_origins, ray_directions):
10
Performs ray casting on mesh to obtain point cloud.
11
12
intersector Raylesslntersscto(eeell
index_triangles index_ray, point_cloud intersector.
intersects_id(
14
ray ray_origgieerrassoooiiinn,,
15
ray a.g.diirccioonerraa..ddiccctaaa
multiple.hits=Trree
retura.locatioos-ITree)
18
return trdex_triangles, index_ray, point_cloud
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification:
The paper discusses the limitations of the work performed by the authors in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [NA]
Justification: The theory assumptions this paper have already been proved in previous works (Diffusion Model and Screened Poisson Algorithm).
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The training and evaluation source code is uploaded so that the reviewer can check the details and re-implement our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

reproducible, they Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: The training and evaluation source code is uploaded so that the reviewer can check the details and re-implement our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / httppsscccppbccccccceecce ceroolccgguideee CodeeSbbm ssi aasonnooicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The paper specify all the training and details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
The paper adopts the mean metric values for 3 times testing for a fair compari-
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In the Experimental section Implementation section, the paper introduces the detailed implementation details, including the computing resources, training and evaluation details. Also, the source code is uploaded for checking.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https / http///eurrrp.peeee//ee ctppp//lcc/tiiccccueeeee.....
Answer: [Yes]
Justification:
The paper is sure preserve anonymity.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification: This paper contributes to fundamental 3D representation research and extends beyond just 3D generation tasks. In the future, it has the potential to enhance other research fields and benefit human society.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The paper cites the original paper that produced the code package or dataset, and follows the licenses for existing assets.
Guidelines:
existing assets. The answer NA means that the paper does
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The paper introduce the details of the dataset to be released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowd-sourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "pDM2QR5Txi", "reviewer": "Reviewer_xHzv", "summary": "This paper introduces X-Ray, a new 3D representation designed for efficient generation of 3D objects from single images. The key idea is to represent a 3D object as a sequence of 2D \"surface frames\" capturing hit/miss, depth, normal, and color information along rays cast from the camera viewpoint. This sequential representation lends itself well to generation using video diffusion models, enabling the synthesis of both visible and hidden surfaces. The authors propose a pipeline consisting of an X-Ray diffusion model to generate low-resolution surface frames from an input image, followed by an X-Ray upsampler to enhance resolution. They evaluate their method on single-view 3D reconstruction and unconditional 3D shape generation tasks, reporting quantitative results on standard benchmarks.", "questions": "- Could the authors please elaborate on the key differences between the X-Ray representation and existing techniques like depth peeling, multi-view depth images, and multi-plane images (MPI)? Also please include the discussion with PI3D.\n\n- To support the claims of efficiency, could the authors provide a quantitative analysis of the memory footprint and computational cost (encoding, decoding, generation) of X-Ray compared to voxel grids, point clouds, or NeRFs? This analysis should consider objects of varying complexity and resolutions.\n\n- Could the authors please include generative metrics for the single-view 3D reconstruction experiments on GSO and OmniObject3D?", "limitations": "- The authors merely state limitations without explaining their causes, impact, or potential solutions. E.g., saying \"X-Ray frames become sparse\" is not enough. How does this sparsity affect generation? How can it be addressed?", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper tackles the important challenge of reconstructing complete 3D models, including hidden surfaces, from single images. This is a significant limitation of current rendering-based approaches, and the authors' focus on this problem is well-motivated and timely.\n\n-  The X-Ray representation, while its novelty requires further substantiation, offers an intuitive way to encode 3D surface information in a sequential manner. By focusing solely on surface details rather than volumetric data, the representation has the potential to be more memory-efficient than voxel grids or dense point clouds, especially for objects with complex internal structures.\n\n- The authors smartly leverage recent advancements in video diffusion models for 3D generation. This is a promising direction, as video diffusion models have shown impressive capabilities in synthesizing high-quality and temporally coherent sequences of images. Adapting these models to the task of 3D generation through the X-Ray representation is a reasonable and potentially fruitful approach.", "weaknesses": "- The paper does not provide a convincing argument for the novelty of the X-Ray representation. While its sequential capture of surface information is intuitive, a thorough comparison to existing techniques is lacking. Specifically, the authors should clearly differentiate X-Ray from methods like depth peeling, multi-view depth images, multi-plane images (MPI), and notably, the PI3D representation (Liu et al., CVPR 2024), which also leverages diffusion models for 3D generation. \n  * Liu, Ying-Tian, Yuan-Chen Guo, Guan Luo, Heyi Sun, Wei Yin, and Song-Hai Zhang. \"Pi3d: Efficient text-to-3d generation with pseudo-image diffusion.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19915-19924. 2024.\n\n- The paper repeatedly emphasizes the efficiency of the X-Ray representation without providing concrete evidence or analysis. The authors should quantify their claims by comparing the memory footprint and computational costs of X-Ray to alternative representations like voxel grids, point clouds, and neural implicit representations (e.g., NeRFs) for objects of varying complexity.\n\n- While leveraging video diffusion models is promising, the paper does not clearly articulate how the specific properties of the X-Ray representation are exploited within the diffusion process beyond being a sequential data format. Do the hit/miss indicators or the ordered nature of surface frames influence the model architecture or training? Would similar performance be achieved with alternative sequential representations as input to the diffusion model?\n\n- The evaluation heavily relies on reconstruction metrics (CD, EMD), even when assessing a generative model. While these metrics are relevant for the single-view reconstruction task, they do not capture the generative capabilities of X-Ray. The authors should expand generative evaluation to diverse categories beyond ShapeNet Cars. The paper could assess generative quality by evaluating the diversity and realism of multiple shapes generated from the same input image.\n\n- The evaluation would be also strengthened by: (1) Including recent SDS-based single-view 3D generation methods as baselines; (2) Providing visualizations of generated shapes for the unconditional generation experiment; (3) Dedicating a section to analyze failure cases, visually showcasing problematic inputs and outputs.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer xHzv\n### Question 1: Comparison to Existing Techniques\nA thorough comparison to existing techniques like depth peeling, multi-view depth images, MPI, and the PI3D.\n### Response 1\n* We discussed multi-view images and MPI in the related work. Furthermore, Multi-view depth cannot sense the inside surface and might records the redundancy surfaces into nearby views. MPI divides the object into planes with fixed distances, while our X-Ray stores a dynamic number of surfaces. Refer to the following Table 1 for efficient comparison.\n* We apologize for not being familiar with depth peeling before submission. Our X-Ray and depth peeling serve different purposes. Depth peeling is mainly for rendering transparent surfaces, while X-Ray transform any 3D object in video format. Besides, our main contribution is using video diffusion as generator to generate objects.\n* The PI3D is an interesting approach that uses diffusion models for 3D generation. Since we do not have access to source code, to ensure a fair comparison, we will reimplement the PI3D method and include it in the revised paper.\n\n### Question 2: Efficiency of the X-Ray\nThe paper should provide more evidence and analysis to support its claims about the efficiency of the X-Ray. \n### Response 2\nWe compared the efficiency of different representations using 500 3D meshes from ShapeNet dataset. The results showed that both point cloud and X-Ray were highly efficient, with lower memory, faster encoding & decoding times. However, the X-Ray had the advantage of being reorganizable as a video format for diffusion models, leading to better performance.\n|Method|Memory (&darr;)|Encoding Method|Encoding Time (&darr;)|Decoding Method|Decoding Time (&darr;) |Reconstruction Error(CD)(&darr;)|Generation Metric (Cov-EMD) (&uarr;)|\n|-|-|-|-|-|-|-|-|\n|3D Grid|67.09 MB|Voxlization|0.105 s|Poisson|~5 s|7.7e-3|3D-DiT[36] &rarr; 56.38|\n|Multi-View Depths (8 views)|1.57 MB|Rendering via Blender|0.045 s|Fusion & Poisson|~10 s|1.1e-2|-|\n|MPI (8 planes)|1.57 MB|Slicing & Rendering via Blender|0.049 s|Poisson|~5 s|8.9e-3|-|\n|Point Cloud (200,000 points)|__0.90 MB__|Surface Sampling|__0.013__ s|Poisson|~5 s|__7.2e-3__|LION[59] &rarr; 56.53|\n|\n|X-Ray (8 layers)|__0.82 MB__|Ray Casting|__0.016 s__|Poisson|~5 s|7.8e-03|Ours &rarr; __60.27__|\n|\n\nTable 1. Comparison with other 3D representations in Efficiency.\n\n### Question 3: Ablation Studies\nThe paper should explain how the unique properties of the X-Ray are utilized in the diffusion process and whether other sequential representations could achieve similar performance.\n### Response 3\nIn section A.3 of the Appendix, we have conducted ablation studies to examine various aspects of the X-Ray model, including the impact of the hit/miss indicators. After exploring different formats, we transitioned to an off-the-shelf Stable Video Diffusion method, which proved to be more effective in addressing the sequential generation problem in 3D reconstruction.\n\n### Question 4: Evaluation Metrics\nThe evaluation should include a broader range of categories to assess the generative capabilities of X-Ray. Could the authors please include generative metrics for the single-view 3D reconstruction experiments on GSO and OmniObject3D?\n### Response 4\nWe chose to focus on cars as they have complex internal and external 3D surfaces, making them an ideal category to showcase the benefits of our method. Objaverse has over 1000 categories. Previous state-of-the-art methods only used reconstruction metrics for evaluation. Thank you for reminding us. We included generative metrics for the single-view 3D reconstruction experiments on GSO in the table below. Since our method can sense the inner surfaces, we have achieved overwhelming superiority. Any additional metrics will be included in the revised paper.\n\n|Method| 1-NNA (EMD) &darr; |Cov (EMD) &uarr;|\n|-|-|-|\n|One-2-3-45[24]|64.37|25.85|\n|OpenLRM[13]|59.14|38.31|\n|TripoSR[48]|57.25|40.69|\n|X-Ray|__52.42__|__48.27__|\n|\nTable 2. Generative metrics on GSO Dataset.\n\n### Question 5: Additional Suggestions\nThe evaluation would be also strengthened by: (1) Including recent SDS-based single-view 3D generation methods as baselines; (2) Providing visualizations of generated shapes for the unconditional generation experiment; (3) Dedicating a section to analyze failure cases, visually showcasing problematic inputs and outputs.\n### Response 5\n1. SDS-based methods like Zero-1-to-3 and DreamGaussian rely on diffusion loss to optimize NeRF or 3D Gaussian splatting. However, these methods are time-consuming, taking several minutes to generate a single 3D mesh from an image. This makes them impractical for evaluation on large datasets like GSO and OmniObject3D. They all did not report their 3D reconstruction metrics. In contrast, all rendering-based methods and our X-Ray can generate 3D meshes from images in just a few seconds. Additionally, SDS-based methods also struggle to generate inner surfaces, limiting their performance on datasets of GSO and OmniObject3D.\n\n2. We have add more visualizations in the Appendix (Sec A.4). Specifically, Figures 6 and 7 showcase the arbitary 3D objects generated by our proposed method. We appreciate your feedback and are planning to further enhance the performance on the 3D CAD model dataset (abundant inside and outside meshes) in the revised paper.\n\n3. We have included a dedicated section in the Appendix (Sec A.5) and Figure 8 to analyze failure cases, and we will enhance this section in the revised paper.\n\n### Question 6: Solution to Limitation\nHow does this sparsity affect generation? How can it be addressed?\n### Response 6\nTo further overcome the sparse related limitation, an autoregressive Large-Language Model (LLM) can be employed as the generator, allowing for each ray to generate varying numbers of surfaces, ranging from 0 to 100+, depending on the actual surface layers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FVlS0KDg3e", "reviewer": "Reviewer_EY92", "summary": "The paper proposes a new representation that encodes a 3D mesh into the ray intersection points from a single view point.  The position, color, and surface normal at the intersection points are stored as the representation. Poisson reconstruction is used to recover the 3D mesh from the representation. A cascaded diffusion model is trained using the extracted representation from 60k objaverse meshes. Evaluation is conducted on three other datasets.", "questions": "1. The proposed representation is related to the pointmap used in DUSt3R (CVPR2024).  From my understanding, at a high level, it extends pointmaps to multiple points along a ray. To me, it is an interesting connection and can be mentioned in the paper. \n\n2. The core of the representation is finding the intersection points and creating the tensor representation. Currently the paper only mentions that it calls a function in trimesh.  I would suggest describing the operation in more detail -- for example, I do not know how the Hit is constructed. Does the method try to aggregate intersection points at similar depths at the same index along the L-dimension?  Or is Hit simply a padding mechanism?  \n\n3. In Sec 5.3 quantitative comparison, Table 1 and Table 2 show metrics that evaluate the quality of the geometry. How is the photometric quality of the proposed method compared to other methods, e.g., PSNR or SSIM on the conditioning viewpoint (and other novel views)?  \n\n4. How significant is the effect of the predicted normal, and is it fair to compare rendering-based methods like LRM using the extracted meshes in Table 1?  My understanding is the quality of the Poisson reconstruction can be affected heavily by the quality of the normal, which may be poorly modeled by volumetric density.  Should the paper also show the same metrics directly on the point clouds or depth maps?", "limitations": "The paper mentions the limited number of intersection points. However, it does not seem clear how to address the limitation potentially without significantly increasing the number of L.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a new representation, which is compact, efficient, and tailored to single image to 3D applications. It demonstrated better performance compared to other methods in the paper in the metrics. The visual quality of the results are also better.  3D generation is a very relevant topics and the paper will be interesting to many readers.", "weaknesses": "1. The paper does not discuss how to extend the proposed representation to include multiple viewpoints to provide better encoding/decoding quality. Even though the representation records the intersection points along a ray, the intersection points from front and top views would be very different.  \n\n2. One limitation of X-ray is that the limited number of intersection points (8 in the paper) makes it difficult to encode complex shapes like hairs and carefully designed meshes with all internal structures. For example, taking a real engine, the number of surfaces a ray can pass through would be far more than 8 (eg., the wires, the bores, shafts).  The paper mentions the limitation, however, it does not seem clear to me from the paper how the limitation can be addressed efficiently.  \n\n3. Another related limitation is the potentially nonuniform number of intersection points along different rays from the same viewpoint. For example, on a human head, the rays passing through the hair region would need a much higher number of intersections to avoid losing information than those through the face region.  The nonuniformity extends to across objects.  To model one object that has a large number of intersection points, it might need to increase L used for all objects in order to learn a diffusion model.  This property makes the presentation inefficient.  \n\n4. In order to generate all interior structures of a 3D object, we need meshes that are carefully designed with interior details.  This kind of dataset is difficult to get and if existed would increase L, making the method inefficient.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer EY92\n### Question 1: Multiple viewpoints Extension.\nHow to extend the proposed representation to include multiple viewpoints to provide better encoding/decoding quality. \n### Response 1\nwe aim to show the advantage of the X-Ray by providing a simple baseline for single-view 3D reconstruction. While extending this to multiple viewpoints is interesting, it might not be necessary for this task. To address the reviewer’s concern, we encoded 1000 randomly selected 3D meshes into X-Ray from different viewpoints, like front and top views, and then decoded them back into 3D meshes. The standard deviation of reconstruction error from different views is negligible (CD < 1e-3). This demonstrates the robustness of the X-Ray to multiple viewpoints and its ability to reconstruct 3D meshes from any viewpoint. We will include this discussion in the revised paper.\n\n### Question 2: Limitation of finite and nonuniform intersection points\nOne limitation of X-ray is that the limited number of intersection points makes it difficult to encode complex shapes like hairs. Another limitation is nonuniform number of intersection points makes the presentation inefficient.\n### Response 2\n* In the experimental results (Figure 4 (a)/(b)), using 8 or even 6 frames is sufficient to cover most of the surfaces, as most 3D models are not highly detailed. However, for complex shapes such as hairs, this limitation can be addressed by increasing the number of frames, to encode more intricate details. \n* To further overcome the nonuniform number of surface layers limitation, autoregressive Large-Language Model (LLM) can be employed as the generator to handle surfaces with a dynamic number of layers. This approach allows for rays to generate varying numbers of surfaces, ranging from 0 to 100+, depending on the actual surface layers for each ray. \n\n### Question 3: Dataset\nCarefully designed dataset is difficult to get and if existed would increase L making the method inefficient.\n### Response 3\nObtaining a dataset with interior details is a challenge, as mentioned by the reviewer. However, we are optimistic about the increasing availability of fine-grained 3D datasets. With advancements in 3D modeling, more detailed 3D models will likely be included in future datasets. Additionally, we have access to indoor scenes and CAD models, which contain rich interior details that can be effectively learned by the X-Ray. This allows us to synthesize complete scene-level 3D rooms or buildings with intricate interior details. As described above, LLM is our next version of generator to handle surfaces with a dynamic number of layers, which will be more efficient in generating complex shapes.\n\n### Questions 4: Related Work: DUSt3R\nIt is a interesting to connect and mention DUSt3R (CVPR2024).\n### Response 4\nWe will mention the connection to the work of DUSt3R in our revised paper as it has made significant contributions to multi-view depth and pose estimation.\n\n### Question 5: Representation Operation\nDescribe the representation operation in more detail.\n### Response 5\nWe have described the process of obtaining X-Ray from a 3D mesh in the Appendix PDF and source code. The \"hit\" value is determined by checking if the depth is greater than zero. We first identify the intersected mesh face index and then query its properties as X-Ray. Below is the pseudocode we provided:\n\n```python\nfrom trimesh.ray.ray_pyembree import RayMeshIntersector\n\ndef ray_cast_mesh(mesh, ro, rd):\n    intersector = RayMeshIntersector(mesh)\n    index_faces, _, _ = intersector.intersects_id(\n        ray_origins=ro, rd=rd,  multiple_hits=True)\n    return index_faces\n\ndef Mesh2XRay(mesh, Rt, K):\n    # get camera center and ray direction\n    ros, rds = get_camera_ray(Rt, K) \n\n    XRay = []\n    for ro, rd in zip(ros, rds):\n        index_faces, = ray_cast_mesh(mesh, ro, rd)\n        depth, normal, color = extract_features(mesh, index_faces)\n        # calc hit\n        hit = depth > 0\n        xray = torch.cat([hit, depth, normal, color], dim=-1)\n        XRay += [xray]\n    XRay = torch.stack(XRay) # (H, W, L, 8)\n    return XRay\n```\n\n### Question 6: Photo-metric Quality\nHow is the photometric quality of the proposed method compared to other methods, e.g., PSNR or SSIM on the conditioning viewpoint (and other novel views)?\n### Response 6\nNovel view synthesis and single-view reconstruction are two distinct tasks. Evaluating the quality of 3D reconstruction based on photometric metrics may not be appropriate. For example, methods like NeRF and 3DGS can achieve high scores in novel-view synthesis at the photometric level, however, their extracted 3D shapes and textures often suffer from really poor quality. Ours focus is on accurately reconstructing the 3D ground-truth shape as the evaluation. We appreciate the reviewer's observation regarding photometric quality. We will predict additional 3D Gaussian Splatting parameters to synthesize high-quality novel view images in the revised paper.\n\n### Question 7: Normal Prediction\nHow significant is the effective of the predicted normal?\n### Response 7\nPoint normals can be estimated from the point cloud, but determining their orientation (inside or outside the surface) is challenging. To solve this, we generate additional point cloud normals for consistent orientation. We tested the importance of normal prediction for rendering methods like LRM and our method. By fine-tuning OpenLRM with normal supervision, we found that adding ground-truth normals did not significantly improve its reconstruction metric and is still not comparible with ours. However, normal prediction is crucial for our method due to Poisson surface reconstruction. This ablation study will be included in the revised paper.\n\n| Method       | Chanfer Distance &darr; |\n|--------------|----------|\n| OpenLRM w/o normals | 0.143     | \n| OpenLRM w/ normals | 0.138 (+3.5%)     |\n|\n| X-Ray w/o normals  | 0.067    |\n| X-Ray w/ normals  | 0.056  (+16.4%)   |\n|", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KA4VBx6jbA", "reviewer": "Reviewer_wtWj", "summary": "The paper addresses the problem of missing mesh interiors in image-to-3D generation. The proposed representation, X-Ray, adopts ray casting to encode both visible and hidden surfaces into a video format. With this representation, the authors enable single image-to-3D mesh generation, including the inside of the mesh, through a video diffusion model. Experiments demonstrate the effectiveness of the proposed method in 3D generation from a single image, both quantitatively and qualitatively.", "questions": "(1) When training, it seems necessary to normalize the object so that the rays from a specific camera position can capture the entire object, making it easier to train. I'm curious whether this preprocessing step was applied to the Objaverse mesh. If this process was included, it would be beneficial to add this information to the paper.\n\n(2) How much GPU memory is required during inference?", "limitations": "The authors mention the limitations related to the number of sequence layers and the missing parts when X-Ray is truncated, as discussed in Section 6 and Appendix A.5.", "rating": 9, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "(1) The task of single-view image to 3D mesh generation including interior surface of mesh is extremely challenging and well-motivated. As far as I know, this is the first paper that generates a mesh considering the interior surface.\n\n(2) The idea of using ray casting to scan the interior of the mesh and convert it into video frames is novel. I really like this approach.\n\n(3) The results show that the proposed method achieves SOTA performance on single image-to-3D generation and point cloud generation with a short inference time compared to existing baselines.\n\n(4) The writing is easy to understand and the design choices are well presented.", "weaknesses": "(1) Considering Computed Tomography, it is expected that the performance will significantly increase as the number of layer $L$ increases. However, the experiments analyzed in Section 5.2 show that performance improvements are very minimal when $L$ is larger than 8. If additional computing resources are available, it would be interesting to see the performance when the frame resolution is low, but $L$ is extremely high (not mandatory).\n\n(2) The entire training pipeline is quite long and memory-intensive. It was conducted on 8 NVIDIA A100 GPU servers for a week.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer wtWj\nWe appreciate the reviewer's highly positive rating to our paper! We address the reviewer's further concerns and questions in the following responses.\n### Questions 1: Performance of Computed Tomography\nConsidering Computed Tomography, it is expected that the performance will significantly increase as the number of layer L increases. However, the experiments analyzed in Section 5.2 show t performance improvements are very minimal when L is larger than 8. If additional computing resources are available, it would be interesting to see the performance when the frame resolution is but L is extremely high (not mandatory).\n\n### Response 1\nWe understand reviewer's concern about the performance when using all frames without omitting any surface. Indeed, conducting experiments with extremely high layers using Diffusion Models is not feasible for GPU memory limitation. However, this issue can be resolved via autoregressive Large-Large Model (LLM) to generate varying numbers of surfaces for each ray, ranging from 0 to even 100+, depending on the actual surface layers for each ray. With LLM, we can flexiblely model both indoor and outdoor scenes using X-Ray. We are going to release the excited research work in the future!\n\n\n### Questions 2: Training Resources\nThe entire training pipeline is quite long and memory-intensive. It was conducted on 8 NVIDIA A100 GPU servers for a week.\n\n### Response 2\nWe appreciate the reviewer's concern about the training pipeline. The training pipeline is relative long and memory-intensive, which is a common issue in training large-scale 3D models. For example, previous state-of-the-art method, Open-LRM, trained their model on 64 NVIDIA V100 GPUs for 5-6 days. Given our limited GPU resources, we made efforts to optimize and minimize the computational requirements as much as possible.\n\n### Questions 3: Normalization\nWhen training, it seems necessary to normalize the object so that the rays from a specific camera position can capture the entire object, making it easier to train. I'm curious whether this preprocessing step was applied to the Objaverse mesh. If this process was included, it would be beneficial to add this information to the paper.\n\n### Response 3\nThe reviewer is correct. We did apply a preprocessing step to normalize the object, ensuring that the rays from a specific camera position can capture the entire object. The normalization code for arbitrary 3D objects is as follows:\n\n```python\ndef normalize_object():\n    bbox_min, bbox_max = scene_bbox()\n    scale = 1 / max(bbox_max - bbox_min)\n    for obj in scene_root_objects():\n        obj.scale = obj.scale * scale\n    # Apply scale to matrix_world.\n    bpy.context.view_layer.update()\n    bbox_min, bbox_max = scene_bbox()\n    offset = -(bbox_min + bbox_max) / 2\n    for obj in scene_root_objects():\n        obj.matrix_world.translation += offset\n    bpy.ops.object.select_all(action=\"DESELECT\")\n```\n\n\n### Questions 4: GPU Memory\nHow much GPU memory is required during inference?\n\n### Response 4\nDuring inference, the GPU memory required is 4.8 GB for X-Ray diffusion model and 2.5 GB for X-Ray Upsampler. Thank you for the question, we will include this information in the revised paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nBI6ojIieZ", "reviewer": "Reviewer_5c6P", "summary": "The paper presents X-Ray, a new 3D sequential representation inspired by the penetrating quality of X-ray scans. This technique converts a 3D object into surface frames at different layers, ideal for creating 3D models from images. Experimental findings show that the X-Ray approach outperforms existing methods in improving the precision of 3D generation, opening up new possibilities for research and real-world applications in 3D representation.", "questions": "* The authors need to provide a clear explanation of X-ray and complete internal mesh, as some training data contain empty spaces within.\n* The authors should conduct additional ablation studies.", "limitations": "The authors acknowledged limitations but did not address the potential negative societal impact of the proposed technology.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The motivation is interesting. Exising methods indeed cannot completely generate objects that include both visible and hidden surfaces. While the proposed method can the hidden interior of the object can be fully reconstructed.\n* The compatibility of X-Ray data structures with sequential 3D representations in video formats opens up new opportunities for using video diffusion models in 3D generation.\n* Great experimental results. The paper outperforms competitors by achieving the best results.", "weaknesses": "* The author asserts that \"General, accurate, and efficient 3D representations are three crucial requirements for 3D generation.\" However, the citation is missing or the author should provide evidence to support the claim that 3D representations need to be general, accurate, and efficient.\n* The ablation study is insufficient. The author should conduct ablation on X-Ray frames, for example, by using video or image diffusion techniquesm or the impact of frame complexity.\n* The author claims to have synthesized a complete interior mesh in Figure 1, but I did not observe any other results besides cars or analysis in the experiments.\n* I think the method used is the X-ray edge method because the dataset quality does not allow for a complete X-ray scan. For instance, the 3D CAT data has empty spaces within.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# To Reviewer 5c6P\n### Question 1: General, accurate, and efficient 3D representations\nThe author asserts that \"General, accurate, and efficient 3D representations are three crucial requirements for 3D generation.\" However, the citation is missing or the author should provide evidence to support the claim that 3D representations need to be general, accurate, and efficient.\n\n### Response 1\nWe are grateful for the reviewer's feedback for our missing of citation. The statement \"General, accurate, and efficient 3D representations are three crucial requirements for 3D generation\" is based on the common understanding in the 3D generation community, and cited from [1].\n* General: A 3D representation should be able to represent a wide range of 3D objects, scenes, and shapes [2]. For example, voxel grids are general representations to any 3D shape, but they are computationally expensive.\n* Accurate: A 3D representation should accurately capture the geometry, appearance, and structure of 3D objects. For example, point clouds are accurate representations that capture the exact 3D points of an object, but they lack connectivity information [3].\n* Effcient: A 3D representation should be efficient in terms of memory usage, computational cost, and training time. For example, 3DGS [4] are efficient representations that can generate high-quality 3D shapes with low memory and computational cost.\n\n### Question 2: Synthesizing a complete interior mesh\nThe author claims to have synthesized a complete interior mesh in Figure 1, but I did not observe any other results besides cars or analysis in the experiments.\n### Response 2\nWe appologize for missing a detailed description to the concept of inside surface. More results in \"Figure 5: Quantitative Comparison in Image-to-3D Generation\" have illustrated the effectiveness of our method in synthesizing a __complete interior mesh__. Here are the explaination: \n* In the GSO dataset shown in Figure 5, our method is able to capture the inside surface of the shoe, while other methods fail to detect it and hide the welt;\n* Also, the OmniObject3D dataset in Figure 5 showcases our method's ability to capture the inside surface of objects such as cupboards and bowls. Other methods fail to accurately predict the inside surface under the input view.\n\n### Question 3: Complete Internel Mesh Dataset.\nThe authors need to provide a clear explanation of X-ray and complete internal mesh, as some training data contain empty spaces within.\n### Response 3\nFor datasets obtained through 3D scanning using multi-view images or depth sensors, it is true that the inside of the objects appears empty. However, the Objaverse dataset contains a variety of 3D models with and without interior meshes, which is why we conducted experiments on this dataset. Our X-Ray method outperforms the state-of-the-art method TripoSR by a large margin on this dataset. We acknowledge the reviewer's concern regarding the difficulty in obtaining datasets with interior details. However, as 3D modeling techniques advance, we anticipate that future datasets will include more detailed 3D models contains the inside mesh. Furthermore, we have found that indoor scenes and 3D CAD models are now accessible for training, providing abundant interior details that can be effectively learned by the X-Ray. As a result, we are able to synthesize complete scene-level 3D rooms or buildings with intricate interior details. We will include the results of these datasets in the revised paper.\n\n### Question 4: Ablation studies\nThe ablation study is insuffcient. The author should conduct ablation on X-Ray frames, for example, by using video or image diffusion techniquesm or the impact of frame complexity.\n### Response 4\n* We appreciate the reviewer's suggestion for additional ablation studies. Due to limited computational resources, we were unable to conduct extensive ablation studies on arbitary X-Ray frame. However, previously, we first selected 16 X-Ray frames to represent each 3D object and then found that using only 8 frames achieved very close performance.\n* Additionally, we conducted an experiment to analyze the Encoding-Decoding Intrinsic Error, as shown in Figure 4 of paper. This is why we chose to use 8 frames in our experiments. The previous experimental results are as follows:\n\n    |Method|CD(L1) &darr;|FS@0.1 &uarr;|\n    |-|-|-|\n    |X-Ray w/ 16 frames|0.053|0.982|\n    |X-Ray w/ 8 frames|0.056|0.973|\n    |\n\n* Besides, we attempted to use the image diffusion model as our backbone for the lightweight computation. However, it did not produce satisfactory results, which is because the image diffusion model only considers the 2D spatial relation and cannot effectively capture the relationship between different frames. Consequently, we made the decision to switch to an off-the-shelf Stable Video Diffusion method for generating X-Ray frames. This method is better suited for addressing the sequential generation problem in 3D reconstruction. The previous experimental results are as follows:\n\n    |Backbone|CD(L1) &darr;|FS@0.1 &uarr;|\n    |-|-|-|\n    |Stable Diffusion|0.114|0.651|\n    |Stable Diffusion + Frame Attention|0.062|0.936|\n    |Stable Video Diffusion |__0.056__|__0.973__|\n    |\n\n### Reference\n[1] Xiaoyu Li, Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, Ying Shan. \"Advances in 3D Generation: A Survey\". arXiv:2401.17807. \n\n[2] Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J. Black, Bernhard Schölkopf. \"Ghost on the Shell: An Expressive Representation of General 3D Shapes\". ICLR2014.\n\n[3] Mescheder, Lars and Oechsle, Michael and Niemeyer, Michael and Nowozin, Sebastian and Geiger, Andreas. \"Occupancy Networks: Learning 3D Reconstruction in Function Space\". CVPR2019.\n\n[4] Kerbl, Bernhard and Kopanas, Georgios and Leimk{\\\"u}hler, Thomas and Drettakis, George. \"3D Gaussian Splatting for Real-Time Radiance Field Rendering\". ACM Transactions on Graphics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
### Responses to All Reviewers:
We would like to express our sincere gratitude for the valuable feedback provided by the reviewers. It is truly encouraging to see that our X-Ray work has been given positive evaluation by all the reviewers. We appreciate the comments and constructive suggestions given by the reviewers. 
Although more visualizations and ablation studies have been included in the Appendix of the submitted paper, we apologize that there are some confusions caused by unclear expressions, potentially inadequate comparisons with existing methods, and a lack of detailed explanations in the paper. In response, we have diligently addressed all the concerns and suggestions raised by the reviewers in the corresponding rebuttals.

Meta-review: The paper received all positive recommendations. The area chair agreed with this recommendation.
Decision: Accept (spotlight)