X-Ray: A Sequential 3D Representation For Generation

Authors: tao  hu
wenhang  ge
yuyang  zhao
gim hee lee
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-08 11:21:41

Abstract: We introduce X-Ray, a novel 3D sequential representation inspired by the penetrability of x-ray scans. X-Ray transforms a 3D object into a series of surface frames at different layers, making it suitable for generating 3D models from images. Our method utilizes ray casting from the camera center to capture geometric and textured details, including depth, normal, and color, across all intersected surfaces. This process efficiently condenses the whole 3D object into a multi-frame video format, motivating the utilize of a network architecture similar to those in video diffusion models. This design ensures an efficient 3D representation by focusing solely on surface information. Also, we propose a two-stage pipeline to generate 3D objects from X-Ray Diffusion Model and Upsampler. We demonstrate the practicality and adaptability of our X-Ray representation by synthesizing the complete visible and hidden surfaces of a 3D object from a single input image. Experimental results reveal the state-of-the-art superiority of our representation in enhancing the accuracy of 3D generation, paving the way for new 3D representation research and practical applications. 
Our project page is in \url{https://tau-yihouxiang.github.io/projects/X-Ray/X-Ray.html}.

Main paper: 
 introduction: 
 General, accurate, and efficient 3D representations are three of the most critical requirements for 3D generation [23, 29, 32, 21] . The significance of this goal stems from the ever-expanding array of applications reliant on 3D technology, ranging from virtual reality and augmented reality to computeraided design and beyond. Previous approaches to 3D representation such as meshes, point clouds, voxels, Neural Radiance Fields (NeRF) [36, 45, 58, 24, 15] and 3D Gaussian Splatting [21] possess unique strengths respectively, but face challenges in concurrently satisfying the three requirements for 3D synthesis. Specifically, meshes are widely used in 3D modeling, while they are constrained by their topology when describing complex objects, which limits their generative capacity. Point clouds offer a more flexible capture of the object geometries but lack continuous and dense feature extraction [12, 42] . Voxels simplify spatial reasoning at the cost of significant rising memory complexity with increasing resolution. Neural representations, such as NeRF [36] and 3D Gaussian Splatting [21] , offer an impressive leap in rendering photorealistic scenes. Nevertheless, the 3D object are predicted by multi-view images with a relatively long optimized period.Recently, rendering-based 3D generative methods [26, 57, 48, 14, 53, 49, 18] have gained significant attention for their ability to achieve general and even efficient 3D generation by incorporating neural representations [36, 21] with Transformers [52] or Diffusion Models [43, 2] . However, these methods have a critical limitation: they cannot completely generate objects that include both visible and hidden surfaces. This limitation arises from the methodological design of rendering-based 3D generation, which relies on the 2D supervision of rendered images. As a result, these methods primarily focus on reconstructing the visible external surfaces of objects, while neglecting the internal hidden surfaces. This oversight leads to incomplete or unrealistic reconstructions, as shown in Fig. 1 (a) .In this paper, we propose the X-Ray representation to overcome this limitation of incomplete generation while maintaining the efficiency and generalization required for 3D generation. As illustrated in Fig. 1 (b), our X-Ray, inspired by x-ray imaging in the medical field, can see through the entire object. It efficiently captures and stores information about both visible and hidden surfaces. Consequently, the hidden interior of the object can be fully reconstructed.Our X-Ray is designed to capture the shape (depth and normal) and appearance attributes (color) along all the sequentially intersected surfaces through ray casting. We transpose the collected slim grid voxels into a multi-frame surface representation, which significantly reduces the data footprint while preserving essential detail. Moreover, the compatibility of our X-Ray's data structure with sequential 3D representation in video formats, as illustrated in Fig. 2 , opens novel pathways for leveraging video diffusion models in 3D generation. Specifically, by treating our X-Ray representation as sequences of frames, we first harness the power of the video diffusion model [2] and then utilize the video upsampler [7] to generate 3D objects from low to high resolution. As a result, our approach yields high-quality results while inheriting the advanced capabilities and efficiency of video processing.We demonstrate the advantages of our X-Ray representation through comprehensive experiments, showcasing its superiority, especially completeness in 3D object generation. We train and evaluate our method in image-to-3D reconstruction task and pure 3D generation task. The experimental results reveal that the proposed X-Ray achieves a significant leap forward in the quality of 3D object generation, positioning it as a feasible solution to longstanding challenges in the field. The main contribution of the paper can be summarized as follows: 3. We showcase the state-of-the-art performance of our X-Ray in 3D generation quality, setting a new benchmark for Image-to-3D modeling.2 Related Work 
representation for 3d models: 
 Handling 3D data is much more complex and resource-intensive than dealing with 1D (e.g. text and voice) and 2D (e.g. image) data. This complexity makes it imperative to find effective ways to organize, process, and infer 3D information. Traditional methods for representing 3D data include meshes, which are good for creating detailed visuals but hard to be generalized; point clouds, which are simple and useful for capturing real-world scenes but lack consistent and dense structure in 3D creation; Although, 3D Gaussian Splatting [21] smooths point cloud data into continuous surface but requires additional an initial point cloud as shape, making them less flexible for 3D synthesis; voxels which are excellent for detailed volumetric data but require much computing resources. Multi-Plane Images [35, 51] try to extent the depth concept to multi-layer with a fixed distance, but they can only describe the visible surface toward camera.Recent advancements in 3D representation have primarily focused on point-level details and implicit functions, such as Occupancy [32] , Signed Distance Fields (SDF) [55, 11] , Triplanes [10, 16] , and Neural representations [36, 21] . These methods have significantly enhanced modeling and rendering capabilities. Occupancy models map the location of any 3D point to its probability of being inside or outside an object, offering a probabilistic approach to shape definition. SDFs [55, 19] refine this concept by quantifying the nearest signed surface distance from any given point, improving the precision of surface representations. Triplanes [3] employ intersecting 2D planes to provide a more efficient route to 3D representation, albeit with some detail loss. NeRFs and Gaussian Splatting [21, 36] produce remarkably realistic renderings from a limited number of viewpoints but require extensive computational effort. Despite these advancements, implicit function-based models often face challenges in extracting full and high-resolution 3D features, hindering high-quality generalization. Empirically, representations that focus on surface is more efficient, and representations with grid representation are more easily to be generalized [37] . Consequently, capturing all surface attributes and organizing in a dense but lightweight data structure renders our X-Ray an accurate, efficient and generalized representation. Noted that our X-Ray is similar to Depth peeling method [9] , which is designed for rendering transparent surfaces, while X-Ray transform any 3D object in video format. Besides, our main contribution is using video diffusion as generator to generate objects.Recent 3D generative models can be primarily categorized into two types: diffusion-based [30, 37, 38] and rendering-based [14, 13, 49, 41, 34, 54, 33, 47] . Diffusion-based models fall under direct 3D supervision, while rendering-based models belong to indirect 2D image supervision. Diffusion-based generative models have emerged as powerful tools for 3D generation, leveraging stochastic diffusion processes to gradually transition from noise to structured 3D objects. These models, such as DPM [30] , DiT-3D [37] , and Point-E [38] , have demonstrated remarkable ability in generating high-quality 3D point clouds. They operate by iteratively refining a random noise distribution into a coherent structure that resembles the target 3D shape, capturing complex geometries and surface details with high fidelity. The strength of these models lies in their capacity to model the distribution of 3D points in a continuous space, allowing for the generation of 3D objects with nuanced variations and detailed textures. However, both point-based networks [28, 37, 17] and voxel-based networks [37] is limited by generating high-resolution objects. Another group of methods [41, 34, 54, 33, 47] adopt Score Distillation Sampling (SDS) as prior to train a NeRF [36] or 3D Gaussian Splatting [21] for 3D generation. However, it is not efficient for optimizing a number of different views over a short period.On the other hand, rendering-based generative models focus on the visible aspect of 3D generation, transforming abstract 3D representations into detailed and photorealistic images or videos. Models such as LRM [14] , Open-LRM [13] ,LGM [48] , DMV3D [57] , and TripoSR [49] employ advanced rendering techniques to achieve this. However, rendering-based models are optimized only for the visible surfaces of objects, making it difficult to synthesize the invisible or internal surfaces.In response to these challenges, our approach utilizes a video diffusion model as the foundation for developing 3D X-Ray. This strategy benefits from the strengths of existing video diffusion models while innovatively addressing the limitations of rendering-based techniques, offering a more comprehensive solution for 3D generation that is sensitive to both visible and hidden parts of objects. 
our x-ray representation: 
 In this section, we detail our X-Ray representation, which includes both encoding and decoding process to facilitate the conversion between 3D mesh formats and our X-Ray representation. Specifically, the encoding process converts a 3D mesh into our proposed X-Ray format, and the decoding process converts our X-Ray back into a 3D mesh.Given a 3D object under an arbitrary camera view, we apply the ray casting algorithm to encode a 3D object mesh into the proposed X-Ray representation. The ray casting algorithm plays a crucial role in both computer graphics and computational geometry, where it is used for scene rendering, visibility determination, and addressing geometric queries. Specifically, a ray is emitted from camera center into the environment and all the interactions of this ray with target 3D objects are captured sequentially. For each ray r ∈ R within the field of view that intersects with L sequential faces in the mesh, we record their 3D attributes which include depth (distance to camera center)D r = (d 1 , d 2 , ..., d L ) ∈ R L×1 , normal N r = (n 1 , n 2 , ..., n L ) ∈ R L×3 , and color C r = (c 1 , c 2 , ..., c L ) ∈ R L×3 .To indicate surface presence, we denote Hit H r = (h 1 , h 2 , ..., h L ) ∈ R L×1 to indicate whether there is a surface. Since L is usually very small (Sec. 5.3), we denote the efficient and thin grid voxels X ∈ R H×W ×L×8 as the object representation, where the ray X ij with image coordinate [i, j] can be represented as:EQUATIONNote that X[i, j, k] = 0 when there is no surface for the k th layer at the image ray coordinate [i, j].Examples of X-Ray encoding are shown in Fig. 2 . Through the encoding process, we can transform any mesh into a sequential representation with varying lengths, same as a video with different numbers of frames. Finally, we transpose the voxels as X-Ray X ∈ R L×8×H×W , resembling a video format with L frames, each with a resolution of H × W and 8 feature channels. The decoding process converts the X-Ray representation back into a 3D mesh. To achieve this, we first convert the video format of X-Ray into a point cloud and subsequently apply the Screened Poisson algorithm [20] to transform the point cloud into a 3D mesh.X-Ray → Point Cloud. Given an X-Ray, we first compute the 3D object's point cloud P r = {P x , P n , P c } for Ray r, including location P x , color P c , and normal P n defined by the equation:EQUATIONr o and r d denote the origin and direction of the camera ray, respectively. Furthermore, D r , N r , C r and H r are the depth, surface normal, color and hit attributes of the ray defined in Eq. 1. Upon processing all camera rays, we obtain a comprehensive point cloud P = {P r } r∈R representation that includes location, normal, and color attributes of the 3D object.Point Cloud → Mesh. The Screened Poisson algorithm [20] for converting point clouds with color and normal into 3D colored meshes is a classic method that leverages the mathematical principles of the Poisson equation. The core idea involves solving a variation of the Poisson equation to interpolate a smooth surface that fits the input point cloud. The Poisson equation is a partial differential equation of the form: ∇ 2 ϕ = f , where ∇ 2 denotes the Laplace operator (which represents the divergence of the gradient of a function), ϕ is the potential field to be solved, and f is a scalar function representing the divergence of the vector field derived from the input point cloud. In the context of point cloud to 3D mesh conversion, the algorithm first employs the given normal to define a vector field that suggests the orientation of the surface at each point. The divergence of this vector field serves as the function f the Poisson equation.Encoding-Decoding Intrinsic Error. The encoding-decoding process will introduce a intrinsic and minor reconstruction error that varies with the number of layer L and the frame resolution (H, W ).To explore this, we conduct an experiment in Sec. 5.3 aimed at analyzing these variables to identify their optimal values. Our goal is to achieve a balance where all pertinent information is preserved while maintaining a lightweight model.The experimental evaluation results on the GSO [6] dataset are illustrated in Tab. 3. These results allow us to analyze the impact of different initialization and scaling strategies on the performance of the diffusion model. By comparing the outcomes, we can identify the trade-offs between model size, training time, and overall accuracy. We observed that finetuning the model did not introduce significant improvement because of the domain gap between video and X-Ray data. Also, it is not necessary to adopt a large diffusion model for Objeverse dataset [4] for significantly reducing the batch size. Thus, we adopt the randomly initialized UNet with 10% parameters as our diffusion 
x-ray for 3d generation: 
 Our primary objective of introducing a new 3D representation model is to facilitate the generation of 3D objects from single images. The challenge lies in accurately predicting the characteristics that are not immediately visible on the first surface when only a single image is available. To overcome this challenge, we utilize diffusion and upsampling models for X-Ray synthesis. Given that our proposed X-Ray is in video format, we leverage advanced Video Diffusion models as our backbone. To exploit this structure for high-resolution X-Ray synthesis, we incorporate principles from advanced video diffusion models as our foundational framework. Notable models in this domain include Stable Video Diffusion (SVD) [2] , VideoFusion [31] , and the state-of-the-art Sora. To efficiently train the diffusion model, we begin by training a low-resolution X-Ray diffusion model that generates X-Ray from a single image. Subsequently, we employ an upsampler to enhance these synthesized X-Rays to high resolution. This two-step approach ensures a more manageable and efficient training process, gradually improving the quality of the output.Framework Overview. Fig. 3 presents an overview of our generative model using X-Ray representation (cf. Sec. 3.1) for 3D generation. Our X-Ray diffusion model (cf. Sec. 4.1) operates at the core of the framework, transforming random Gaussian noise into a low-resolution X-Ray representation conditioned by an input image. These low-resolution X-Rays are then enhanced to high resolution through the application of a 3D Spatial-Temporal Upsampler (cf. Sec. 4.2). Finally, the high-resolution X-Rays are decoded into 3D meshes using a combination of point cloud transformation and the Screened Poisson algorithm (cf. Sec. 3.2).Diffusion models [43] are generative models that transform a random noise distribution into a data distribution through a reverse process, counteracting a forward process that incrementally adds Gaussian noise to the data. The forward process is a Markov chain described byx t = √ α t x t-1 + √ 1 -α t ϵ, where x t represents the data at step t, α t controls the noise level, and ϵ ∼ N (0, I) is sampled noise. The reverse process, aimed at reconstructing the original data from noise, is modeled by a neural network predicting the noise added at each step or directly denoising the data, followingx t-1 = 1 √ αt x t -1-αt √ 1-α 2 t ϵ θ (x t , t), with ϵ θ (x t , t) being the predicted noise. Training involves optimizing the network to minimize the difference between the original and reconstructed data, effectively learning to invert the noise addition process given by:EQUATION)where t is uniformly sampled from the set {1, . . . , T }.Diffusion Model for X-Ray. A prevalent technique in diffusion models is the utilization of latent spaces with a VQ-VAE [7] to perform the initial data transformation to compress the data. This method poses a significant challenge for our X-Ray representation as it requires the development of a VQ-VAE model from scratch due to the absence of a suitable off-the-shelf latent model for X-Ray, which would consequently increase our training burden. Another promising approach for efficiently training high-resolution generators is the cascaded synthesis pipeline. This method, exemplified by works such as Imagen [44] , DeepFloyd IF [1] , and Stable Cascaded [40] , involves progressively training the diffusion model or upsampling network from lower to higher resolutions. Given our limited computing resources, we opted to implement this cascaded upsampling strategy. This technique facilitates a more gradual and controlled enhancement of X-Ray quality, providing a more flexible and efficient alternative to traditional latent space diffusion models.Specifically, we use the Spatial-Temporal 3D U-Net network from Stable Video Diffusion [2] for our diffusion model to generate low-resolution X-Rays, with modifications to the input and output channels. As shown in Fig. 3 , the input image is sent to the encoder E, producing an image latent z via image VAE [7] and an embedding e via a ViT [5] . z is concatenated with the X-Ray latent as input, and e interacts with the 3D U-Net through cross attention [2] to finally output the denoised latent. This model employs spatial-temporal attention mechanisms to alternately extract features from 2D frame spaces and 1D surface layer sequences, enhancing its ability to process and interpret the different layers of the X-Ray. This approach allows for nuanced handling of the temporal information inherent in sequential X-Ray data, crucial for achieving high-quality diffusion results.The X-Ray upsampler focuses on enhancing previously generated X-Rays to a higher resolution. We considered two potential methods: point cloud up-sampling and video up-sampling. Encoding low-resolution X-Rays into a point cloud with color and normal information is straightforward (see Sec 3.1). However, point cloud up-sampling often increases only the number of points without effectively enhancing attributes like texture and color due to its unstructured nature. To improve efficiency and consistency, we adopted a video up-sampling approach using a spatial-temporal VAE decoder from Stable Video Diffusion (SVD) [2] . We concatenate the previous image latent z with the low-resolution X-Ray as input and output the high-resolution X-Ray. The model upsamples previously synthesized low-resolution X-Ray frames fourfold while preserving the original layer number L. It applies attention at both the 2D surface frame and 1D surface layer levels, enhancing frame resolution and overall quality. This makes the X-Ray diffusion model, followed by the upsampler, a more integrated and effective solution.Loss. The loss function for the Upsampler differs notably from that of the diffusion model. While the diffusion model loss typically addresses volumetric or textural aspects, the Upsampler loss concentrates specifically on the surface area accuracy, reflecting the critical importance of maintaining high fidelity in the enhanced images. The loss function we use for the Upsampler is given by:EQUATIONwhere X gt [H gt ] represents the ground-truth high-resolution X-Ray at hit surface and X up [H gt ] denotes the Upsampler's output at hit surface, and H gt , H up denotes the ground-truth and upsampled Hit, respectively. The loss is computed as the squared Euclidean distance between these two matrices, quantifying the pixel-wise discrepancy in surface details. This metric effectively ensures that the upsampling process preserves essential surface features, thereby optimizing the quality and utility of the resulting high-resolution X-Ray. 
dataset and implementation: 
 Datasets. We train our X-Ray pipeline using a subset of the Objaverse dataset [4] , which have been removed entries with missing textures and inadequate prompts as outlined in [48] . This subset consists of more than 60,000 3D objects. For each object, we select 8 random camera views, covering azimuth angles from -180 to 180 degrees and elevation angles from 0 to 45 degrees with camera distance to object center fixed at 1.2. The images are then rendered using Blender Software, and the corresponding X-Rays are generated through the ray casting algorithm provided by the trimesh library [50] . Through these processes, we create a dataset of approximately 480,000 paired images and X-Rays to train the generative model. For the evaluation datasets, we adopt two commonly adopted datasets: Google Scanned Objects [6] and OmniObject3D [56] , to assess generative performance via single-view reconstruction tasks.Metrics. Recent 3D generative models [26, 18, 27, 14, 49] lack unified reconstruction evaluation metrics due to the challenge of determining an object's size and orientation from a single image [46] .To ensure fair comparisons with state-of-the-art methods, we align all methods before evaluation. The predicted and ground-truth 3D objects will be normalized to a range of -0.5 to 0.5 along all three axes and face forward the same -z axis. We then align them using the Iterative Closest Points (ICP) algorithm and calculate Chamfer Distance (CD) in L1 norm and F-Score (FS) at threshold 0.1 (FS@0.1) for reconstruction.Implementation Details. Our X-Ray diffusion model is based on the Spatrial-Temporal 3D UNet architecture used in Stable Video Diffusion (SVD) [2] , modified to synthesize 8 channels: 1 hit channel, 1 depth channel, 3 color channels, and 3 normal channels, compared to the original 4 channels. During training, we maintain a learning rate of 0.0001 using the AdamW optimizer. Since different X-Rays have varying numbers of layers, we pad or truncate them to a uniform 8 layers for efficient batching and training. Each layer's frame has dimensions of 64 × 64. For the upsampler, each layer's output remains at 8 channels, but the resolution of each frame is increased to 256 × 256 to enhance detail and clarity in the upscaled X-Ray. The entire training pipeline is conducted on 8 NVIDIA A100 GPU servers for two weeks. During inference, the 3D generation process takes approximately 7 seconds: about 1 second for the diffusion model, 1 second for the upsampler, and 5 seconds for mesh decoding. As for GPU usage during inference, the GPU memory required is 4.8 GB for X-Ray diffusion model and 2.5 GB for X-Ray Upsampler. Surface Layer umber (L) We compared the efficiency of different representations using 500 3D meshes from random selected models. The results showed that both point cloud and X-Ray were highly efficient, with lower memory, faster encoding & decoding times. However, the X-Ray had the advantage of being reorganizable as a video format for diffusion models, leading to better performance.Due to the finite number of layer L and resolution H, W of X-Ray, a slight intrinsic error is inevitable during the encoding of 3D meshes into X-Ray format and the subsequent decoding back into 3D meshes. To quantitatively assess this error, we conducted an experiment to evaluate intrinsic error via Chamfer Distance (CD) (L1 norm) between the original (ground-truth) mesh and the encodingdecoding mesh across various resolutions and layers. In the experiments, we set the frame resolution through a series of predefined values: 32, 64, 128, 256, 512, and 1, 024 and vary the number of layers from 1 to 12. Fig. 4a indicates that the intrinsic error decreases as the number of layers in our X-Ray representation increases, becoming convergent after 8 layers. Similarity as illustrated in Fig. 4b , the intrinsic error decreases with increasing resolution and stabilizes after 256. Therefore, for a balance between accuracy and efficiency, we use a resolution For image-to-3D mesh generation on the GSO [6] and OmniObject3D [56] datasets, we build a new benchmark that randomly selects 500 image-mesh pairs from each dataset. We re-ran the baselines using their released source code and used normalized Chamfer distance and F-score for a fair comparison. The results, summarized in Tab. 2, show that our X-Ray method achieves significant improvements over all previous rendering-based state-of-the-art methods [25, 18, 8, 13, 49] on both datasets, with a relative 33% improvement (0.084 → 0.056) in Chamfer distance and also a significant improvement in FS@0.1 (0.878 → 0.973, where the maximum is 1) on GSO dataset and similar performance on OmniObject3D dataset. This demonstrates the superiority of our approach over rendering-based methods. A qualitative comparison effectively demonstrates the advantages of our proposed X-Ray generation method. Using the three datasets mentioned, we selected single images as input and generated 3D meshes without and with textures, as shown in Fig. 5 . Our proposed method has three key advantages: 1. It can decompose shape and appearance to accurately reconstruct flat surfaces (Rows 1 and 3); 2. It can detect the sealing of containers (Rows 2 and 4); 3. It can generate the internal structure of objects within their outer surfaces (Rows 5 and 6). These obvious advantages highlight the our effectiveness.The failure cases highlight the limitations of the current generative model based on X-Ray representation when the number of frame layers is very large. As illustrated in Fig. 6 , given an input image containing a complex object, such as an exquisite hamburger, the number of frame layers L of the encoded or generated X-Ray tends to exceed the maximum length of 8. Consequently, any surface behind layer 8 will be omitted, resulting in missing parts of the reconstructed mesh. The solution is to increase the value of L so that the X-Ray can represent more surfaces. However, this would also increase computing resource requirements. We will reconsider the sparsity of deeper surface frames and propose a more efficient generative model to overcome these failure cases.Ground-Truth Mesh  
conclusion: 
 In this work, we introduced a novel X-Ray representation for 3D objects that encompasses both visible and hidden surfaces within the camera's field of view, unlike recent rendering-based methods that typically focus only on the visible surface. We demonstrated the effectiveness of the X-Ray approach in single-view 3D generation tasks. Our generative model shows that the underlying generator for X-Ray shares foundational similarities with existing video diffusion models, allowing us to leverage their advantages. Experimental results highlight the outstanding performance of our method.Limitations. Our generator uses the Stable Video Diffusion (SVD) pipeline to produce high-quality X-Ray. However, the X-Rays generated consist of an uncertain number of sequential layers, with the posterior layers tending to be more and more sparse, which can be redundant. Additionally, the generated mesh lacks enough smoothness and has missing part when X-Ray is truncated. We plan to explore advanced network architectures, such as Large Language Model, that can better handle the complexities of X-Ray data, including layer sparsity and sequential format. Additionally, we aim to investigate further applications of the X-Ray representation to broaden its utility and impact in 3D modeling and beyond. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: This paper contributes to fundamental 3D representation research and extends beyond just 3D generation tasks. In the future, it has the potential to enhance other research fields and benefit human society.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: The paper cites the original paper that produced the code package or dataset, and follows the licenses for existing assets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes]Justification: The paper introduce the details of the dataset to be released.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA]Justification: The paper does not involve crowd-sourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA]Justification: The paper does not involve crowdsourcing nor research with human subjects.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The Appendix section contains detailed information on the network structure (including the X-Ray diffusion model and upsampler), ablation studies, additional experimental quantitative and qualitative results, failure cases, and key source code.A.1 Network Details X-Ray Diffusion Model As outlined in our study, we utilized the 3D Spatial-Temporal UNet as the foundational architecture for our diffusion model. For image-to-3D generation, the input includes 4-channel image latent and 8-channel noise X-Ray and the output has 8 channel to denoise X-Ray. Initially, we loaded all network parameters and fine-tuned the model on our training dataset. Despite these efforts, the final performance fell short of expectations. We hypothesized that this underperformance stemmed from the limited size of our training dataset, which was insufficient for such a parameter-rich model, as well as a substantial domain gap between the original video data and our X-ray data.To address these issues, we reduced the network size to 10% of its original configuration and trained the model from scratch using a significantly larger batch size. This approach proved highly effective, as the final performance exceeded the current state-of-the-art methods by a considerable margin. Consequently, our findings suggest that a smaller, more focused network trained from scratch can be more effective than a larger pre-trained model when faced with limited and domain-specific datasets. This approach not only enhances performance but also highlights the importance of customizing the model size and training strategy to the specific characteristics of the data. Our results indicate that careful consideration of the network architecture and training regimen is crucial for optimizing performance, particularly in specialized applications such as X-ray generation. This study provides valuable insights into model adaptation and training strategies that can be applied to other domains facing similar challenges.The following JSON file contains the configuration details for our X-Ray Spatial-Temporal UNet. The X-Ray diffusion model enables the generation of low-resolution 3D objects. To enhance resolution and improve performance, our X-Ray upsampler increases the frame resolution by a factor of 4. For image-to-3D generation, we concatenate the 4-channel image latent representation with the 8-channel low-resolution X-Ray, producing an 8-channel high-resolution X-Ray. This process begins with the diffusion model creating a coarse, low-resolution 3D representation that captures the essential structure of the object. The upsampler then refines this representation, significantly improving the resolution and adding finer details that contribute to the realism of the 3D model. By combining the latent image features with the initial low-resolution X-Ray data, we ensure that the final high-resolution output retains the context and nuances of the original image while also incorporating the detailed structural information provided by the X-Ray data.The following JSON file contains the configuration details for our X-Ray Upsampler. As described in Sec. A.1, we train the diffusion model with different configurations, including varying model sizes and how to conduct initialization. Specifically, we evaluate three models:1. Finetuned Original UNet 2. Randomly Initialized Original UNetFigure 7 : Visualization of Image-to-3D Generation from X-Ray.model. These findings provide insights into the optimal configuration for diffusion models in 3D related tasks. We shown more Visualization results of single-view image generation in Fig. 7 . Also, we extend the task to generate 3D object from text prompt. Text-to-3D mesh generation can also be achieved via using text-to-image, object segmentation, and image-to-3D processes. We utilize established diffusion models that are already proficient in image synthesis from textual descriptions instead of developing a new text-conditioned diffusion model. One Model such as Stable Diffusion [43] , Stable Cascaded [40] , or DiT [39] are employed to generate images based on the input text. Following this, we apply an image segmentation tool, specifically the Segment Anything Model [22] to eliminate the background. This streamlined method avoids the complexities of training a new model from scratch instead of making use of sophisticated pre-trained models to handle the text-to-image translation, thereby simplifying the process of generating 3D meshes from textual inputs. The output results of Image-to-3D and Text-to-3D are illustrated in Fig. 8 .Ray Casting: The source code of ray casting the mesh to get the point cloud.1 """ 2 The source code of ray casting the mesh to get the point cloud .3 """ 4 5 from trimesh . ray . ray_pyembree import RayMeshIntersector 3 """ 4 5 import numpy as np 6 import open3d as o3d Point Cloud to Mesh: The source code of transferring the predicted point cloud to mesh using Poisson Surface Reconstruction.1 """ 2 The source code of transferring the predicted point cloud to mesh using 3 Poisson Surface Reconstruction . 4 """ Normalized Metrics: The source code of normalize and align the predicted point cloud to the gt point cloud before evaluation.1 """ 2 The source code of normalize and align the predicted point cloud to the 3 gt point cloud before evaluation . 4 """ 5 import numpy as np 6 import open3d as o3d Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: The training and evaluation source code is uploaded so that the reviewer can check the details and re-implement our results. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: In the Experimental section Implementation section, the paper introduces the detailed implementation details, including the computing resources, training and evaluation details. Also, the source code is uploaded for checking.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The paper is sure to preserve anonymity.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).We present X-Ray, a novel 3D representation that encode the whole visible and hidden surfaces in to video format through ray casting algorithm for maintaining generalization, accuracy and efficiency.We propose the generative model of our X-Ray via video diffusion model and video upsampling model, enabling low-to-high generation of 3D objects from single images."""The source code of transfering X -Ray to Point Clouds with normals and colors . 


Decision: The paper received all positive recommendations. The area chair agreed with this recommendation.

Reviews: review: paper_summary: paper_summary: This paper introduces X-Ray, a new 3D representation designed for efficient generation of 3D objects from single images. The key idea is to represent a 3D object as a sequence of 2D "surface frames" capturing hit/miss, depth, normal, and color information along rays cast from the camera viewpoint. This sequential representation lends itself well to generation using video diffusion models, enabling the synthesis of both visible and hidden surfaces. The authors propose a pipeline consisting of an X-Ray diffusion model to generate low-resolution surface frames from an input image, followed by an X-Ray upsampler to enhance resolution. They evaluate their method on single-view 3D reconstruction and unconditional 3D shape generation tasks, reporting quantitative results on standard benchmarks.
strength_weakness: strength_weakness: - The paper does not provide a convincing argument for the novelty of the X-Ray representation. While its sequential capture of surface information is intuitive, a thorough comparison to existing techniques is lacking. Specifically, the authors should clearly differentiate X-Ray from methods like depth peeling, multi-view depth images, multi-plane images (MPI), and notably, the PI3D representation (Liu et al., CVPR 2024), which also leverages diffusion models for 3D generation. 
  * Liu, Ying-Tian, Yuan-Chen Guo, Guan Luo, Heyi Sun, Wei Yin, and Song-Hai Zhang. "Pi3d: Efficient text-to-3d generation with pseudo-image diffusion." In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19915-19924. 2024.

- The paper repeatedly emphasizes the efficiency of the X-Ray representation without providing concrete evidence or analysis. The authors should quantify their claims by comparing the memory footprint and computational costs of X-Ray to alternative representations like voxel grids, point clouds, and neural implicit representations (e.g., NeRFs) for objects of varying complexity.

- While leveraging video diffusion models is promising, the paper does not clearly articulate how the specific properties of the X-Ray representation are exploited within the diffusion process beyond being a sequential data format. Do the hit/miss indicators or the ordered nature of surface frames influence the model architecture or training? Would similar performance be achieved with alternative sequential representations as input to the diffusion model?

- The evaluation heavily relies on reconstruction metrics (CD, EMD), even when assessing a generative model. While these metrics are relevant for the single-view reconstruction task, they do not capture the generative capabilities of X-Ray. The authors should expand generative evaluation to diverse categories beyond ShapeNet Cars. The paper could assess generative quality by evaluating the diversity and realism of multiple shapes generated from the same input image.

- The evaluation would be also strengthened by: (1) Including recent SDS-based single-view 3D generation methods as baselines; (2) Providing visualizations of generated shapes for the unconditional generation experiment; (3) Dedicating a section to analyze failure cases, visually showcasing problematic inputs and outputs.
questions: questions: - Could the authors please elaborate on the key differences between the X-Ray representation and existing techniques like depth peeling, multi-view depth images, and multi-plane images (MPI)? Also please include the discussion with PI3D.

- To support the claims of efficiency, could the authors provide a quantitative analysis of the memory footprint and computational cost (encoding, decoding, generation) of X-Ray compared to voxel grids, point clouds, or NeRFs? This analysis should consider objects of varying complexity and resolutions.

- Could the authors please include generative metrics for the single-view 3D reconstruction experiments on GSO and OmniObject3D?
limitations: limitations: - The authors merely state limitations without explaining their causes, impact, or potential solutions. E.g., saying "X-Ray frames become sparse" is not enough. How does this sparsity affect generation? How can it be addressed?
paper_summary: paper_summary: The paper proposes a new representation that encodes a 3D mesh into the ray intersection points from a single view point.  The position, color, and surface normal at the intersection points are stored as the representation. Poisson reconstruction is used to recover the 3D mesh from the representation. A cascaded diffusion model is trained using the extracted representation from 60k objaverse meshes. Evaluation is conducted on three other datasets.
strength_weakness: strength_weakness: 1. The paper does not discuss how to extend the proposed representation to include multiple viewpoints to provide better encoding/decoding quality. Even though the representation records the intersection points along a ray, the intersection points from front and top views would be very different.  

2. One limitation of X-ray is that the limited number of intersection points (8 in the paper) makes it difficult to encode complex shapes like hairs and carefully designed meshes with all internal structures. For example, taking a real engine, the number of surfaces a ray can pass through would be far more than 8 (eg., the wires, the bores, shafts).  The paper mentions the limitation, however, it does not seem clear to me from the paper how the limitation can be addressed efficiently.  

3. Another related limitation is the potentially nonuniform number of intersection points along different rays from the same viewpoint. For example, on a human head, the rays passing through the hair region would need a much higher number of intersections to avoid losing information than those through the face region.  The nonuniformity extends to across objects.  To model one object that has a large number of intersection points, it might need to increase L used for all objects in order to learn a diffusion model.  This property makes the presentation inefficient.  

4. In order to generate all interior structures of a 3D object, we need meshes that are carefully designed with interior details.  This kind of dataset is difficult to get and if existed would increase L, making the method inefficient.
questions: questions: 1. The proposed representation is related to the pointmap used in DUSt3R (CVPR2024).  From my understanding, at a high level, it extends pointmaps to multiple points along a ray. To me, it is an interesting connection and can be mentioned in the paper. 

2. The core of the representation is finding the intersection points and creating the tensor representation. Currently the paper only mentions that it calls a function in trimesh.  I would suggest describing the operation in more detail -- for example, I do not know how the Hit is constructed. Does the method try to aggregate intersection points at similar depths at the same index along the L-dimension?  Or is Hit simply a padding mechanism?  

3. In Sec 5.3 quantitative comparison, Table 1 and Table 2 show metrics that evaluate the quality of the geometry. How is the photometric quality of the proposed method compared to other methods, e.g., PSNR or SSIM on the conditioning viewpoint (and other novel views)?  

4. How significant is the effect of the predicted normal, and is it fair to compare rendering-based methods like LRM using the extracted meshes in Table 1?  My understanding is the quality of the Poisson reconstruction can be affected heavily by the quality of the normal, which may be poorly modeled by volumetric density.  Should the paper also show the same metrics directly on the point clouds or depth maps?
limitations: limitations: The paper mentions the limited number of intersection points. However, it does not seem clear how to address the limitation potentially without significantly increasing the number of L.
paper_summary: paper_summary: The paper addresses the problem of missing mesh interiors in image-to-3D generation. The proposed representation, X-Ray, adopts ray casting to encode both visible and hidden surfaces into a video format. With this representation, the authors enable single image-to-3D mesh generation, including the inside of the mesh, through a video diffusion model. Experiments demonstrate the effectiveness of the proposed method in 3D generation from a single image, both quantitatively and qualitatively.
strength_weakness: strength_weakness: (1) Considering Computed Tomography, it is expected that the performance will significantly increase as the number of layer $L$ increases. However, the experiments analyzed in Section 5.2 show that performance improvements are very minimal when $L$ is larger than 8. If additional computing resources are available, it would be interesting to see the performance when the frame resolution is low, but $L$ is extremely high (not mandatory).

(2) The entire training pipeline is quite long and memory-intensive. It was conducted on 8 NVIDIA A100 GPU servers for a week.
questions: questions: (1) When training, it seems necessary to normalize the object so that the rays from a specific camera position can capture the entire object, making it easier to train. I'm curious whether this preprocessing step was applied to the Objaverse mesh. If this process was included, it would be beneficial to add this information to the paper.

(2) How much GPU memory is required during inference?
limitations: limitations: The authors mention the limitations related to the number of sequence layers and the missing parts when X-Ray is truncated, as discussed in Section 6 and Appendix A.5.
paper_summary: paper_summary: The paper presents X-Ray, a new 3D sequential representation inspired by the penetrating quality of X-ray scans. This technique converts a 3D object into surface frames at different layers, ideal for creating 3D models from images. Experimental findings show that the X-Ray approach outperforms existing methods in improving the precision of 3D generation, opening up new possibilities for research and real-world applications in 3D representation.
strength_weakness: strength_weakness: * The author asserts that "General, accurate, and efficient 3D representations are three crucial requirements for 3D generation." However, the citation is missing or the author should provide evidence to support the claim that 3D representations need to be general, accurate, and efficient.
* The ablation study is insufficient. The author should conduct ablation on X-Ray frames, for example, by using video or image diffusion techniquesm or the impact of frame complexity.
* The author claims to have synthesized a complete interior mesh in Figure 1, but I did not observe any other results besides cars or analysis in the experiments.
* I think the method used is the X-ray edge method because the dataset quality does not allow for a complete X-ray scan. For instance, the 3D CAT data has empty spaces within.
questions: questions: * The authors need to provide a clear explanation of X-ray and complete internal mesh, as some training data contain empty spaces within.
* The authors should conduct additional ablation studies.
limitations: limitations: The authors acknowledged limitations but did not address the potential negative societal impact of the proposed technology.

Comments: title: Response to rebuttal
Response to Reviewer EY92
comment: Thank you once again for recognizing the value of our work and maintaining the rating of 9, which is highly encouraging for us and our future efforts to advance X-Ray representation in the 3D domain! We have carefully considered the reviewers’ comments and will add an experimental section on the efficiency of the X-Ray representation to enhance clarity. As mentioned in our response to Reviewer xHzv, we compare it with other 3D representations, such as Voxels, point clouds, and multi-view depths, to make it easier for readers to grasp its efficiency.
Thanks for the authors' response. I have read it as well as other reviews.

I believe this paper presents a new problem (single image to mesh generation including interior surface) that the academic community has not yet solved, and I consider it the first paper to suggest a new direction for research. Additionally, the proposed method (using ray casting + video diffusion model) is novel and has been proven through experiments to be superior to existing methods, which I believe can serve as a good reference for future research.

Regarding the concern about the number of $L$, which I and other reviewers were worried about, the authors claimed to address it through an autoregressive Large-Language Model (LLM), but this part is not explained in detail, so it's hard to understand. However, given the current limitations in computing resources, the performance is sufficiently guaranteed even with $L=8$, to the extent that it surpasses other baselines, so I don’t think this is a major issue.

In the revised version, it would be good to make revisions to address the concerns of the other reviewers, and particularly, I strongly recommend adding a section on the efficiency of the X-Ray representation (Rebuttal to Reviewer xHzv, Response 2). Since a new representation has been proposed, if a comparison with other representations is organized in a table, readers will be able to easily grasp the efficiency of the X-Ray representation at a glance.

I appreciate the additional experiments conducted for the rebuttal, and for the reasons mentioned above, I will maintain my previous rating of 9.
### We sincerely appreciate the reviewer's comments and the subsequent discussion. We will do our best to address the concerns raised.

### 1. Does it mean that it is difficult to extend x-ray to few-view to 3d use cases?
* Although single-view X-Ray can solve most cases as a simple baseline, the reviewer suggests further strengthening the performance by introducing a multi-view solution. 

* Similar to multi-view depth representation, it is easy to extend X-Ray to a multi-view style by casting rays under multiple cameras then concatenating these frames as video. There will be $N \times L$ frames in the multi-view X-Rays, where $N$ is the number of views and $L$ is the number of surface layers in each view. We can then generate the multi-view X-Rays from a single image via the same video diffusion model.

* We greatly appreciate the reviewer's suggestion and will focus on validating how multi-view X-Rays outperform single-view X-Ray generation in our subsequent research work.


### 2. PSNR value on the given single view is still an informative and meaningful metric even for single view to 3d task
* The reviewer is very expert in novel view synthesis and 3D reconstruction and generation research. We thank the reviewer's comment that the PSNR value is an informative and meaningful metric for the single-view to 3D task.

* For evaluation, the deeper question is which should be considered the ground truth for photometric performance: albedo (original surface color) or image (rendered color under lighting). We predict the albedo before rendering, while rendering-based methods predict the final image color after rendering. These are two different colors. Due to this ambiguity, we did not report these results. However, in our previous experiment using albedo as the ground truth, our X-Ray achieved 21.3 PSNR, whereas the state-of-the-art TripoSR only achieved 17.6 PSNR on the GSO dataset.
  
* We understand that the reviewer might be referring to the performance of evaluating rendered colors, similar to other rendering-based methods. Our solution is to predict the 3DGS parameters for each surface of X-Ray and render the image to compare with the ground truth. As described in Response 1, as the first work, we aim to provide a simple baseline X-Ray representation and avoid involving more technical complexities. We will report both photometric performance results in the revised paper. Similar to the normalized shape evaluation in this paper, we hope the photometric evaluation under both albedo and image can provide a new benchmark for the community.
I thank the authors for the reply. From Response 1, does it mean that it is difficult to extend x-ray to few-view to 3d use cases? Though x-ray can represent simple meshes well, the constraint of single-view only seems an important characteristic of the proposed method that should be discussed in the paper. For example, few-view (eg, 2-4) to 3d is also a frequently studied problem and common in practice.

For Response 6, I understand and agree with the description about the difference between single-view to 3d and multiview reconstruction. However, the PSNR value on the given single view is still an informative and meaningful metric even for single view to 3d task.