LAM3D: Large Image-Point Clouds Alignment Model for 3D Reconstruction from Single Image

Authors: ruikai  cui
xibin  song
weixuan  sun
senbo  wang
weizhe  liu
shenzhou  chen
taizhang  shang
yang  li
nick  barnes
hongdong  li
pan  ji
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 14:58:54

Abstract: Large Reconstruction Models have made significant strides in the realm of automated 3D content generation from single or multiple input images. Despite their success, these models often produce 3D meshes with geometric inaccuracies, stemming from the inherent challenges of deducing 3D shapes solely from image data. In this work, we introduce a novel framework, the Large Image and Point Cloud Alignment Model (LAM3D), which utilizes 3D point cloud data to enhance the fidelity of generated 3D meshes. Our methodology begins with the development of a point-cloud-based network that effectively generates precise and meaningful latent tri-planes, laying the groundwork for accurate 3D mesh reconstruction. Building upon this, our Image-Point-Cloud Feature Alignment technique processes a single input image, aligning to the latent tri-planes to imbue image features with robust 3D information. This process not only enriches the image features but also facilitates the production of high-fidelity 3D meshes without the need for multi-view input, significantly reducing geometric distortions. Our approach achieves state-of-the-art high-fidelity 3D mesh reconstruction from a single image in just 6 seconds, and experiments on various datasets demonstrate its effectiveness.

Main paper: 
 introduction: 
 High-quality 3D mesh creation has drawn increasing attentions for its great potential in numerous fields, such as video games, virtual reality, and films. Traditionally, 3D assets are usually created manually by expert artists or alternatively reconstructed from multi-view 2D images, which are time-consuming. Recent proposed Large Reconstruction Models (LRMs) [15, 62, 48, 52 ] from a single view or few-shot images bring significant advancements for high-fidelity 3D content creation.Single view based LRMs [15, 13] take a single-view image as input and use transformers to directly regress tri-plane-based [3] neural radiance fields (NeRF) [31] for 3D content creation. However, as shown in Fig. 1 (b) , due to ambiguity of 3D geometry from a single image, single-view-based LRMs commonly suffer from geometric distortions, especially for the unseen areas of the input view. Inspired by Hong et al. [15] , LRMs with multi-view inputs [62, 48, 52, 20] are proposed, where they typically first feed the single-view image into multi-view image diffusion models [51, 43, 28] for few-shot multi-view images generation. Then multi-view tri-plane features are generated and converted into 3D representations (NeRF [31] , SDFs [34] , etc.) for 3D mesh creation. However, as shown in Fig. 1 (c ), the generated multi-view images lack guaranteed multi-view consistency, which can lead to geometric distortions.To relieve these problems, we propose to introduce a 3D point cloud prior for single image based 3D mesh reconstruction with Large Image-Point-Cloud Alignment Model (LAM3D). LAM3D aims to [63] , (b) LRM [13] , (c) CRM [54] , and (d) Ours (LAM3D).transfer single image features into point cloud features, enriching the image features with accurate and effective 3D information for high-fidelity 3D mesh reconstruction. The motivation behind this choice is obvious: 3D point cloud contains effective structure prior of 3D geometries. Meanwhile, approaches [63, 64] of modality feature alignment have been proposed, aiming to learn unified representations of language, images, and point clouds for 3D understanding. Nevertheless, as shown in Fig. 1 (a), these techniques fall short when they use the aligned image features to replace point clouds features for 3D mesh creation. This is largely due to the exclusive usage of contrastive loss [63, 64] and the aligned feature size is commonly with (512×1) [63] and (1024×1) [64] , which is inadequate to fully capture the rich diversity of 3D shapes.In this work, we extend the image and point cloud alignment for 3D mesh creation, and present the Large Image-Point-Cloud Alignment Model (LAM3D) which utilize the prior of 3D point cloud for high-fidelity feature alignment. We start by designing an effective point cloud-based network with hierarchical tri-planes [3] for 3D mesh reconstruction. Following this, taking single image as input, image features are extracted using DINO [2] , whilst LAM3D is employed to transpose these image features into tri-planes obtained via the point-cloud-based network. The tri-plane representation [3] allows for the tuning of image features with precise and effective 3D information, which is essential for high-fidelity 3D mesh reconstruction. Meanwhile, to better retain 3D structure information and reduce the interference of each plane (XY, XZ, YZ) in tri-plane features, we enlist the use of independent diffusion operations to transfer image features to each respective plane. Note that the transferred features are with size of 3×C×M×N ((3×2×32×32) in our experiments), which is larger than previous representation size of (512×1) [63] and (1024×1) [64] , thus contains more information for better 3D mesh creation. As shown in Fig. 1 (d), more reasonable geometric mesh can be reconstructed by our approach.The main contributions of our work are as follows:• We propose an effective Large Image-Point-Clouds Alignment Model (LAM3D) for 3D mesh reconstruction from a single image. By utilizing point cloud priors, LAM3D delivers precise image tri-plane feature transformation, thereby enhancing the quality and accuracy of 3D mesh creation.• To preserve 3D structural information more effectively, we employ independent diffusion processes that transmit image features to each respective tri-plane (XY, XZ, and YZ) for accurate 3D mesh reconstruction.• Our method significantly relieves geometric distortions caused by ambiguities of 3D shape inferred from the image modality, and experiments on various of dataset demonstrate the effectiveness of our approach. 
related works: 
 Single Image to 3D Reconstruction: Recently, LRM [15] was the first to demonstrate that a regression model can be trained to robustly predict tri-plane-based NeRF from a single-view image.Inspired by LRM, numerous large models [25, 26, 36, 54, 69, 53, 50] for 3D reconstruction have since been proposed. Among these, Instant3D [20] trains a text-to-few-shot multi-view image diffusion model, as well as a multi-view LRM, to enable fast and diverse text-to-3D generation. Subsequent works have extended LRM by incorporating pose prediction from multi-view images [52] , combining it with diffusion [62] , and specializing on human data [56] . Tang et al. [48] introduced the Large Multi-View Gaussian Model to generate high-resolution 3D models from text prompts or singleview images. Wang et al. [54] presented a single-image-to-3D generative model that generates six orthographic images, utilizing Flexicubes [42] as a 3D representation. Concurrently, Wu et al. [58] proposed Direct3D, a method that differs from ours primarily in its approach to feature extraction from point clouds. Direct3D utilizes learnable tokens for feature extraction, whereas our method introduces a projection module to explicitly convert point cloud features into a tri-plane structure.Generative Modeling of 3D shapes: In recent years, a multitude of studies have been dedicated to the generation of 3D shapes. Existing 3D generative models have been developed using a variety of frameworks, including generative adversarial networks (GANs) [57, 1, 67] , variational autoencoders (VAEs) [47, 11, 17, 32, 12] , normalizing flows [65, 32] , autoregressive models [45, 32] , energy-based models [61, 8] , and more recently, denoising diffusion probabilistic models (DDPMs) [30, 68, 27, 49, 24, 4, 44, 60] . Luo et al. [30] pioneers the application of DDPMs for modeling the distribution of raw point clouds. Fellowing Luo et al, several works [49, 24, 4] explore generative modeling of 3D shapes in latent space to reduce computational complexity and enhance generation quality.For instance, LION [49] proposed an effective latent-space diffusion model to generate novel point clouds, while 3DQD [24] , SDFusion [4] utilize DDPM to model Signed Distance Function (SDF) in the latent space of an autoencoder for 3D shape generation. Concurrently, other research efforts have investigated DDPMs for 3D shape generation with other representations, such as mesh [27] , occupancy grid [44] , and neural radiance fields [33] .Feature Alignment / Multimodal 3D: Multimodal approaches are mainly about image-text and image-text-point-clouds modalities. Several works [21, 23, 46] propose to learn interactions between image regions and caption words using transformer-based architectures, which show great predictive capability despite being costly to train. Meanwhile, CLIP [38] uses image and text encoders to output a single image/text representation for each image-text pair, and then aligns the representations to a unified latent space. Built upon CLIP [38] , ULIP [63] and its successor, ULIP2 [64] , aim to learn a unified representation that integrates images, text, and 3D point clouds. These methods establish a 3D representation space aligned with the CLIP latent space by leveraging synthesized 3D-text-image triplets. The most closely related work to ours is Michelangelo [66] , which aligns point cloud features with frozen CLIP image/text embeddings through an autoencoder. Unlike Michelangelo, which employs contrastive loss to learn an aligned latent space, our approach uses diffusion to align 3D and 2D modalities. This results in a more continuous latent space, beneficial for 3D reconstruction. 
method: 
 In this section, we present the detailed design of our approach. The full training process consists of two stages: Point Cloud Compression (Sec. 3.1) and Image-Point-Cloud Alignment (Sec. 3.2).As shown in Fig. 2 , we encode point clouds into a tri-plane representation and then decode SDF values from the reconstructed tri-plane to extract 3D meshes via marching cubes [29] . The point cloud compression consists of Initial Point Feature Extraction and Tri-plane Compression. Specifically, we propose a point-cloud-based 3D reconstruction network to convert point cloud into a latent tri-plane, which can retain rich 3D information. In the tri-plane compression module, the latent tri-plane is up-sampled to reconstruct tri-planes followed by an MLP to compute SDF values. As shown in Fig. 2 , hierarchical tri-plane are used, including latent tri-planes and reconstructed tri-planes, where the latent tri-plane is designed for better image-point-cloud feature alignment and the reconstructed tri-plane is for better mesh reconstruction.To compress a point cloud into latent tri-planes, we employ a transformer-based approach to learn long-range interactions between points. We first use Furthest Point Sampling (FPS) to downsample the input point cloud to n center points. Then, we gather the K-nearest neighbors of the n center points together as a small point cloud and use a shallow PointNet [35] to embed each small point cloud as an embedding. By doing so, we reduce the sequence length of the point cloud, and then we can use a transformer to process these embeddings.After the transformer step, we need to collect the embeddings to form the three planes of the tri-plane structure. To this end, we use a projection operation. In specific, each embedding feature is associated with a knn center. For each plane (XY, XZ, Y Z), we diminish one axis of the center points and voxelize the center points into a plane of size R C×R×R where C is the channel dimension that is the same with the embedding dimension, and R is the plane resolution. We use average operation to aggregate embeddings associated with center points that land into the same voxel. As shown in Fig. 2 , the point cloud is converted into a initial tri-planes that faithfully preserves 3D information.The initial tri-planes is a sparse tensor as many of its grids are empty if there is no center points land there. To obtain a more compact and expressive latent tri-plane representation, we adopt an plane autoencoder to further compress initial tri-planes to latent tri-planes.Plane Encoder We find that directly aligning image features with initial tri-planes leads to two drawbacks: (1) Aligning in high-resolution requires high computational complexity; (2) Empty grids and overwhelmed details in high-resolution tri-planes substantially increase training difficulty, leading to noisy results. To further compress the initial tri-planes into a more compact latent, we opt for a transformer-based plane encoder. It consists of a convolution layer to down-sample the initial tri-planes followed by successive transformer blocks to achieve inter-plane correlations. In each transformer block, we follow Cui et al. [6] to concatenate tri-planes into one sequence and adopt linear attention [37] to process the sequence to model inter-plane correlations. In our settings, the extracted initial tri-plane is down-sampled by 4 times to obtain a point cloud latent tri-plane t ∈ R (3×2×32×32) .Plane Decoder We introduce a hierarchical scheme to decode the latent tri-plane into a high-resolution reconstructed tri-plane, as depicted in Fig. 2 , and for predicting the Signed Distance Function (SDF). Initially, the latent tri-plane is processed through a standard decoder that mirrors the encoder. This decoder comprises successive transformer blocks coupled with a convolutional layer, which augments the latent tri-plane by a factor of four.Plane Refiner After Plane Decoder, we adopt an intra-plane refinement module to further enhance the reconstruction quality. In latent tri-plane generation, the original point clouds are condensed to form a compact tri-plane latent, the compression inaccuracies are unavoidable. Therefore, we propose an asymmetric decoding of the tri-plane, which is accomplished by individually refining each plane with a plane refiner module subsequent to the conventional decoder. Specifically, the plane refiners are three small-scale UNet [41] models to refine each plane separately. We find that refining each plane individually extensively improves mesh quality while retains inter-plane correlations as we demonstrated in Appendices.Tab. 5. Finally, we can query SDF values of any spatial position using the SDF-MLP by interpolating features on the reconstructed tri-plane and extract 3D meshes using Marching Cubes [29] .The training objective for point cloud compression comprises a KL penalty and a geometry loss. To align image features onto a 3D-aware latent space, we attempt to preserve 3D prior to the greatest extent, even in the most compact compressed tri-plane latent. However, we discovered that the compressed tri-plane latent fails to maintain the 3D prior, leading to artifacts in the mesh outcomes if we merely employ an arbitrary encoder to compress the initial tri-plane without explicit constraints. Hence, we propose to concurrently supervise the latent tri-plane t and reconstructed tri-plane P using the ground truth SDF and two distinct SDF MLPs (Φ t and Φ P ). To this end, we combine the geometric SDF loss and latent regularization to train the compression module in an end-to-end manner. Formally, to allow the predicted tri-plane recover SDF values, we use the geometry loss:EQUATIONThese two terms are defined as:EQUATIONEQUATIONwhere Ω 0 and Ω are sets of query points sampled from object surface and off-surface points sampled from [-1, 1] 3 . The grouth truth SDF values and surface normal are denoted as d p and n p , respectively.The gradient ∇ p Φ P (p) = ∂Φ P (p) ∂X , ∂Φ P (p), ∂Φ P (p)represents the direction of the steepest change in SDF. It can be computed using finite difference, e.g., the partial derivative for the X-axis component reads ∂Φ(p) ∂X = Φ P (p+(δ,0,0))-Φ P (p-(δ,0,0))2δwhere δ is the step size. Similarly, the latent sdf loss L lsdf is defined as:EQUATIONWe use the corresponding MLP to predict a SDF value from either latent tri-plane t or the reconstructed tri-plane P . Specifically, given a query position p, we project it to each of the planes and retrieve the feature vectors F xy , F xz , F yz via bilinear interpolation. Then, the signed distance can be predicted as:Φ(p) = MLP(F xy ⊕ F xz ⊕ F yz ), where ⊕ denotes element-wise summation.Overall, the training objective is defined as fellow:EQUATIONwhere L KL is a KL penalty which leans towards a standard normal distribution on the learned latent tri-plane, akin to a Variational Autoencoder [19] .We align image and point cloud features within the same feature space. Since the point clouds have been compressed into latent tri-planes with 3D-aware point cloud priors in stage 1, we can project the image features onto this 3D-aware latent. We experiment with two approaches to accomplish this goal. First, following reconstruction methods such as LRM [15] , we can utilize a transformer-based structure to align the image features with the point cloud tri-planes, employing a distance metric like L1 or L2 loss. Alternatively, we can employ a probabilistic diffusion approach, wherein we initiate from random noise and denoise it, guided by the input image targeting the point cloud domain.Our experiments reveal that the transformer-based model yields unstable results. This instability arises because 3D reconstruction from a single image is an ill-posed problem, as the image lacks information about the occluded object parts. The 3D structural information in an image is not as rich as that in point clouds. Consequently, if we simply adopt a deterministic approach, the alignment result tends to be the average of all possible outcomes for regions that are not sufficiently observed [15] . Thus, we choose a diffusion model as a probabilistic alignment approach. Intuitively, the diffusion model can leverage its prior to envision unobserved regions and generate clear predictions for these areas, as demonstrated in previous 2D prior works [55, 22] . We provide more analysis in Appendices. D.2.As shown in Fig. 2 , first, we use a pre-trained DINO [2] image encoder to convert images to a S × D dim feature z img where S is the squence length and D is the feature dimension. Subsequently, the image features serve as conditioning information in the diffusion process, mapping the image feature from the image modality to the point cloud modality.Unlike previous methods [5, 6, 12] that depend on a single diffusion model to map an arbitrary 1D ordering vector, we employ three parallel UNets to map the image feature to each of the three latent planes, separately. Since each tri-plane contains a 2D-image-like feature, as demonstrated in Appendices, Fig. 6 . Prior methods [12, 6] use a single UNet to denoise concatenated planes. However, the UNet is designed without 3D-aware prior, meaning that there will be adjacent pixels without a direct relation due to the convolution operation being a local operation, leading to artifacts.We utilize a diffusion model to align the image and the point cloud features. Diffusion Models [14, 40] are designed to learn a data distribution q(z 0 ) by progressively denoising a normally distributed variable. The learning process is equivalent to performing the reverse operation of a fixed Markov Chain with a length of T , which transforms latent z 0 into purely Gaussian noise z T ∼ N (0, I) over T time steps. The forward step in this process is defined as:EQUATIONwhere noisy variable z t is derived by scaling the previous noise sample z t-1 with √ 1 -β t and adding Gaussian noise following a variance schedule β 1 , β 2 , . . . , β T .We train three parallel diffusion models on each of the three planes by learning to reverse the above diffusion process. To achieve this, we adopt the approach proposed by Aditya et al. [39] , wherein we use three neural networks that takes z t to directly predict z 0 corresponding to XY, XZ, and YZ planes. Specifically, given a uniformly sampled time step t from the set {1, ..., T }, we generate z t by sampling noise from the input latent tri-plane z 0 . A time-conditioned denoising autoencoder consisted of three parallel UNets [40] , denoted by Ψ, is employed to reconstruct z 0 from z t given the alignment source, i.e., the image latent feature z img . The objective of the alignment is given by:EQUATIONwhere γ(•) represents a positional embedding and || • || 2 denotes the mean squared error (MSE) loss. 
experiments: 
 Dataset: Following previous works [54, 15] , our model is trained on a subset of Objaverse [9] with 140k filtered 3D assets, and evaluated on unseen Objaverse objects and Google Scanned Objects [10] . The Objaverse dataset contains over 800k 3D assets. It was filtered to remove objects with thin faces and repeated/similar buildings, resulting in a final training set of 140k objects. More details can be found in Appendices. B.Baselines: To assess the effectiveness of our approach, we compare our method with state-ofthe-art approaches, including One-2-3-45 [25] , LRM [15] , SyncDreamer [26] , Wonder3D [28] , Magic123 [36] , TGS [69] , LRM [48] and CRM [54] .During training, we employ a two-stage training pipeline. In stage 1, a point cloud compressor is trained, comprising a transformer-based point feature extractor and a tri-plane autoencoder. Given The inference stage requires only a single-view image of an object. The process starts by encoding this image into a feature representation using the DINO image encoder. This image feature is then aligned to a latent tri-plane representation through three independent diffusion UNets. The latent tri-plane is subsequently upsampled to a high-resolution tri-plane using the plane decoder and refiner from the first training stage. We then employ the SDF MLP to decode the signed distance for any query position on the tri-plane, which allows us to use the Marching Cubes algorithm to reconstruct a mesh from the tri-plane.Qualitative Results: Fig. 3 presents a qualitative comparisons between our approach and other state-of-the-art approaches including One-2-3-45 [25] , LRM [15] and CRM [54] . Since LRM is not open-sourced, we use OpenLRM [13] , an open-sourced implementation of LRM for comparisons. For the other baselines, we use their official codes and model weights. As shown in Fig. 3 . While impressive results can be obtained with these methods, geometric distortions often occur due to ambiguities caused by single image or multi-view inconsistency. Additionally, these approach commonly fail for unseen areas of the input image, such as the inside of a cup. In contrast, our method leverages point cloud priors to reconstruct 3D meshes with better geometry and more details than all other baselines. Moreover, our model does not rely on multi-view consistency, unlike One-2-3-45, LRM, and CRM, thanks to the point cloud prior. This results in significantly reduced geometric distortions, as seen in the box, sofa and bowl examples in Fig. 3 .Quantitative Results: following previous studies [54, 48, 15] , we use the Google Scanned Objects (GSO) dataset [10] to evaluate the effectiveness of our method. In line with CRM [54] and One-2-3-45 [25] , we use Chamfer Distance (CD) [59, 7] , Volume IoU and F-Score with a threshold of 0.05 to evaluate the mesh geometry. The results are shown in Tab. 1. It can be seen that our method outperforms all of the baselines on all the evaluation metrics. This demonstrates the effectiveness of our method for 3D reconstruction. Notably, our model generates mesh within only 6 seconds on an NVIDIA V100 GPU. For our ablation study, we use a small subset of the Objaverse dataset containing 3000 game assets (weapons, gear, etc.) for training, and 216 objects for evaluation.Effectiveness of latent tri-plane representation: In our method, we compress the point cloud as a latent tri-plane representation, which is then used for image-point-cloud feature alignment. In this study, we evaluate the effectiveness of this proposed latent tri-plane representation.We compare three different variants in our ablation study: a) latent tri-plane w/ L lsdf : our introduced hierarchical decoding strategy, constraining latent tri-plane with a SDF loss; b) latent tri-plane w/o L lsdf , latent tri-plane without the SDF loss; c) vector, a vector with dim (6114×1) (same size with (3×2×32×32)) is used to represent the point cloud latent. We train the stage 1 compression models for point cloud-based reconstruction. As shown in Tab. 2, simply using a latent vector representation without spatial structure does not perform well with a performance of 4.81. However, converting the vector representation to tri-plane improves the performance to 1.95. Moreover, tri-plane with latent SDF loss (Ours in Tab. 2) can further refine the final results to 1.81. This emphasizes the importance of retaining 3D-aware structure in the point cloud latent representation and the necessarity of introducing the latent SDF loss L lsdf .In Fig. 4 , we provide illustrations of different sizes and representations. We observe that a vector size of 512 performs poorly, which is inadequate to represent the 3D shapes. A vector with dimensions of 6114 × 1 can recover the structure of a 3D mesh, but details such as the trigger are missing. In Fig. 4 (d) , we show that tri-plane structure can improve the visual performance with better trigger. In Fig. 4 (e), further refined results with better details can be obtained, validating the effectiveness of the latent SDF loss. Image-point-cloud alignment model vs. ULIP: In order to validate the effectiveness of our proposed image-point-cloud alignment model, we compare our method with ULIP [63] , which is designed for point cloud feature alignment with text and images.ULIP-based image reconstruction: ULIP uses a pre-trained CLIP and point cloud encoder to encode text, images, and point clouds to a feature of size (512×1). Contrastive loss is utilized for aligning the features of point clouds, text, and images. To evaluate the performance of ULIP for 3D reconstruction, we design a decoder network that decodes the (512×1)-dimension point cloud feature vector to a 3D mesh. We then use the decoder network to directly replace the point cloud features with the aligned image feature obtained by ULIP for 3D mesh reconstruction. As shown in Tab. 2, results obtained by ULIP [63] are unstable, leading to worse performance in charmer distance, which proves that the scheme of feature alignment used in ULIP [63] works poor for 3D reconstruction. On contrary, our method can obtain better results.To better retain 3D structure information and reduce the interference of each plane (XY, XZ, YZ) in latent tri-planes, we enlist the use of independent parallel diffusion operations to transfer image features to each respective plane. We valid the effectiveness of using three parallel diffusion operations to do the feature alignment instead of using a single UNet in Tab. 3. Parallel UNet variant has smaller CD error, and we show qualitative comparisons in Fig. 5 . We can observe that, thanks to the parallel UNet design, our model can focuses on aligning image feature to a specific plane and thus presents better visual quality. For example, the surface of the ice cream bar is broken in the reconstruction of the single UNet variant, while ours can maintains the continuity of the surface. Notably, to compare equally, we reduce the parameters of parallel diffusion to guarantee the same parameter size of parallel diffusion and single diffusion.We provide more ablation studies as well as more visualization results in the appendix.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We provide a detailed description of the model and experimental settings in our paper, ensuring that readers have the necessary information to reproduce the main experimental results. Additionally, we plan to release the code to further enhance reproducibility and facilitate verification of our results. 
conclusions: 
 In this work, we present the Large Image and Point Clouds Alignment Model (LAM3D), which harnesses the prior of 3D point clouds for high-fidelity 3D mesh creation. We design an effective point-cloud-based network to compress point cloud into compact and meaningful latent tri-plane. Subsequently, taking a single image as input, Image-Point-Clouds Feature Alignment is designed to transfer image features to latent tri-planes, enriching the image features with accurate and effective 3D information for high-fidelity 3D mesh reconstruction. We believe that LAM3D has potential to improve 3D content creation and assist the workflow of digital artists.Limitations and Future Work: The main goal of the proposed LAM3D is geometry reconstruction but texture reconstruction is not included, so that our method cannot achieve textured mesh reconstruction. We will further extend LAM3D to geometric and texture reconstruction in the future.Broader Impact: We hold the view that LAM3D has the potential to enhance 3D content creation and aid digital artists in their workflow. LAM3D was conceived with these applications in mind, and we hope it can evolve into a useful tool that reduce the labour cost in 3D asset production. While we do not foresee any immediate harmful uses for LAM3D, we believe it is crucial for users to exercise caution to minimize impacts, considering that the alignment framework can also be employed for harmful intents.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: While we currently do not provide open access to the data and code, we plan to release the code along with sufficient instructions to reproduce the main experimental results after the paper has been accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: We have provided all the necessary details regarding the training and testing process, including data splits, hyperparameters, the rationale for their selection, and the type of optimizer used.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Our paper does not report error bars or other information about the statistical significance of the experiments. However, we follow common practices in our research area and have comprehensively evaluated our method to demonstrate its effectiveness, which ensures that our results are reliable and meaningful within the context of our work.Guidelines:• The answer NA means that the paper does not include experiments. Justification: We provided sufficient information on the computer resources needed to reproduce the experiments in the implementation details section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [Yes] Justification: Our method presents a low risk for misuse or dual-use. Additionally, we have taken precautions to ensure responsible data handling by filtering our training data to avoid unsafe content. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Diffusion Models are probabilistic models designed to learn a data distribution z 0 ∼ q(z 0 ) by gradually denoising a normally distributed variable. This process corresponds to learning the reverse operation of a fixed Markov Chain with a length of T . The inference process works by sampling a random noise z T and gradually denoising it until it reaches a meaningful latent z 0 . Denoising diffusion probabilistic model (DDPM) [14] defines a diffusion process that transforms latent z 0 to white Gaussian noise z T ∼ N (0, I) in T time steps. Each step in the forward direction is given by:EQUATIONEQUATIONThe noisy latent z t is obtained by scaling the previous noise sample z t-1 with √ 1 -β t and adding Gaussian noise with variance β t at timestep t. During training, DDPM reverses the diffusion process, which is modeled by a neural network Ψ that predicts the parameters µ Ψ (z t , t) and Σ Ψ (z t , t) of a Gaussian distribution.pEQUATIONWith α t := 1 -β t and ᾱ := t s=0 α s , we can write the marginal distribution:EQUATIONEQUATIONwhere ϵ ∼ N (0, I). Using Bayes' theorem, we can calculate q(z t-1 |z t , z 0 ) in terms of βt and μt (z t , z 0 ) which are defined as follows:EQUATIONq(z t-1 |z t , z 0 ) = N (z t-1 ; μt (z t , z 0 ), βt I)Instead of predicting the added noise as in the original DDPM, in this paper, we predict z 0 directly with a neural network Ψ following Aditya et al. [39] . The prediction could be used in Eq. 14 to produce µ Ψ (z t , t). Specifically, with a uniformly sampled time step t from {1, ..., T }, we sample noise to obtain z t from input latent vector z 0 . A time-conditioned denoising auto-encoder Ψ learns to reconstruct z 0 from z t , guided by the image feature z img . The objective of latent tri-plane diffusion readsEQUATIONwhere γ(•) is a positional encoding function and ∥ • ∥ 2 denotes the Mean Squared Error loss.The Objaverse contains over 800k 3D assets. However, there are many low-quality 3D assets that are not suitable for training a 3D reconstruction, such as object with only thin faces. We filter those object by removing object with only thin faces, repeated and similar buildings and senses. Finally, we obtain 140k object for training.For the first stage compression training, we sample data from the object surface. Given a 3D object mesh, we first pre-process it to ensure it forms a watertight shape using the method proposed by Huang et al. [16] . Then, we normalize the shape within the [-1, 1] 3 box. Subsequently, we randomly sample on-surface points from the shape surface and uniformly sample off-surface points with ground truth SDF values in the [-1, 1] 3 space. In addition to the signed distance supervision, we leverage normal directions as an extra guidance to achieve detailed surface modeling. Consequently, we also sample the normal vector for each on-surface point.For the alignment stage training, we render each object from 48 fixed camera position equally distributed on a sphere covering the object. However, we do not use all 48 rendered images for the We provide a detailed formulation of metrics that are not explicitly defined in the main manuscript to enhance understanding of our experimental outcomes Chamfer Distance (CD): This metric measures the average distance between two point sets, usually the ground truth and the predicted 3D reconstruction. Given two point sets A and B, the Chamfer Distance is defined as:EQUATIONwhere |A|and |B| denote the number of points in sets A and B, respectively, and ||a -b|| 2 represents the squared Euclidean distance between points a and b. Note that CD is a symmetric and non-negative distance measure.To compute the CD for 3D reconstruction evaluation, we first sample 2048 points from the surfaces of the predicted mesh and the ground truth mesh. Then, we calculate the Chamfer distance between the point sets, as defined above.F-Score: This metric is a measure of the trade-off between precision and recall, which are commonly used metrics for evaluating the quality of a 3D reconstruction. Given a threshold value τ , the F-Score is defined as:EQUATIONwhere precision(τ ) and recall(τ ) are the precision and recall values at the given threshold τ , respectively. In our experiment, we set the threshold to 0.05 w.r.t. the Chamfer distance.Volume Intersection over Union (IoU): Volume Intersection over Union (IoU) is another metric used for evaluating the quality of 3D reconstructions. It measures the ratio of the volume of the intersection between the predicted mesh and the ground truth mesh to the volume of their union. The IoU is defined as:EQUATION)To compute the IoU for 3D reconstruction evaluation, we first convert the predicted mesh and the ground truth mesh into Signed Distance Functions (SDFs) with a voxel resolution of 128 × 128 × 128.All negative voxels in the SDFs represent the interior of the object. Then, we calculate the volume of the intersection and the volume of the union between the two SDFs. Finally, we compute the IoU using the formula above.We conduct experiments to study the effect of latent tri-plane resolution and feature dimension. We design four compared variants of our full model. There are two aspects that needs to be considered, i.e. Figure 7 : Alignment approach ablation study. We evaluate the single view reconstruction capability of deterministic vs. probabilistic approaches. Green objects are constructed from our probabilistic approach and gray samples are from a deterministic approach. We also present the reference image.the spatial resolution and the feature dimension of each plane. We vary these parameters and evaluate the point cloud to mesh reconstruction capability in terms of CD. The result is presented in Tab. 4.From which we can observe that setting the spatial resolution and feature dimension as (32 × 32) and 2 are the best choice. We can also notice that, when we simply introduce a tri-plane structure, the reconstruction quality improved by 0.64 in terms of CD (×10 4 ) despite the number of parameters in the 2 × 8 × 8 setting is significantly lower than it in the vector setting presented in Tab. 2. This improvement is brought by the tri-plane structure introduced 3D awareness to the latent structure, which benefits the decoding stage. Another founding is that improving the spatial resolution brings improvements to the reconstruction quality. However, the computation complexity will increase quadraticly as we increase the spatial resolution, therefore, increase the resolution beyond 32 × 32 becomes intractable in our experiment. Furthermore, we do ablation study on the feature dimension. We notice that increasing the feature dimension is not beneficial anymore, a potential reason is that increasing the feature dimension introduces too many parameters which leads to a latent space that is not compact enough to be continues. Therefore, it performs poorly when generalizing to novel objects. Also, decreasing the feature dimension to 1 leads to a worse performance as there is not adequate parameters to preserve the geometry information of the encoded object.In terms of the modality alignment strategy, a navie way to address this is using a projection network that directly project an image feature to the point cloud feature modality. However, since this is an ill-pose problem and a single-view image can corresponding to many possible valid 3D shapes due to occlusion. A deterministic approach tends to produce blurry predictions [15] . We demonstrate such an effect by designing a deterministic Image-Point-Cloud alignment network. To this end, we follow LRM [15] and use a 12-layer transformer to directly align image feature to point cloud feature modality and train on the 140k Objavese dataset. Fig. 7 shows the difference between our probabilistic alignment approach compared with a deterministic method. From which we can observe that the reconstruction with probabilistic process involved produces more clear and sharp details than a deterministic approach. To evaluate the effectiveness of our plane refiner module, we design a comparison about our full model with plane refiner and a variant without plane refiner. The result is shown in Tab. 5, and we also present a qualitative result in Fig. 8 . We can observe that the plane refiner module improves the capability of our compression model, allowing it to recover more fine-grained details.We present more visualization for 3D reconstruction by aligning image and point cloud modality using our proposed method in Fig. 9 Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA] Justification: Our paper does not include theoretical results, and therefore, this question is not applicable to our work.• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification: Our research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: This paper discussed both potential positive societal impacts and negative societal impacts of the work performed Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 


Decision: This paper was reviewed by four experts in the field.  Based on the reviewers' feedback, the decision is to recommend the paper for acceptance.  The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.   In particular, it is strongly recommended to release the code as promised in "5. Open access to data and code" field.

Reviews: review: paper_summary: paper_summary: The paper proposes a two-stage 3D reconstruction method that first uses a transformer-based 3D point cloud feature extractor to initialize hierarchical latent triplanes (XY, XZ, YZ) and reconstructs the 3D mech. Next, it presents an image-point cloud feature alignment approach that leverages initial latent triplanes to align them with the image-based features using independent plane diffusion models to produce better detail and reduced geometric distortion reconstructions.

The approach achieves state-of-the-art high-fidelity 3D mesh reconstructions from a single image in just 6 seconds, and experiments on various datasets demonstrate its effectiveness.
strength_weakness: strength_weakness: •	Comparison of the model capacity with the current SOTA methods might be added as the proposed methods seems heavy that uses three diffusion, three UNet and two transformer-based architecture.
•	Also, the paper might want to compare the inference time with the existing methods as it specifies that it only requires 6 secs for inference.
•	Details of the module and inference stage missing (see section correctness).
•	The method is evaluated using only a single dataset. 
•	Lack of reproducibility

The paper is not well-written. The figure is supposed to give an overview of the pipeline, however, if just reading the figure it is not clear what the input is. The paper refers to input point clouds multiple times, which makes it furthermore confusing.  Or maybe the point clouds are also part of the input as it is referred to as prior point clouds in the paper. If so the performance gain might also be due to the additional multi-modality inputs which is not a fair comparison with the other baseline methods.

Typo:

•	Line 622: We follow LRM [14] ‘and uses’ a 12-layer transformer to directly align image feature to point cloud seems mistaken.
questions: questions: The paper needs to be improved in the presentation and make the flow chart more clear for example, what is the input?

•	Missing details of the Point Net and Transformer block utilized in the point feature extraction stage, plane encoder in triplane compression, UNet utilized for the plane refiner process, and diffusion models in image-point cloud alignment. 
•	The inference process is unclear, specifically lines 249-251, and might need more discussion.
•	The paper misses mentioning the concept and gaps of the current compared methods called One-2-3-45 [24], SyncDreamer [25], TGS [63], CRM [51], and Magic123 [35] in the related work.
limitations: limitations: The authors adequately addressed the limitations.

Overall, the idea of aligning the 3D point features induced triplane representation and image-based features utilizing independent plane diffusion models seems interesting and can produce less geometric distortion reconstruction. However, the proposed method is evaluated only on a single dataset and misses the related work, mentioned module details, and model capacity comparison that must be addressed.
paper_summary: paper_summary: This paper introduces the Large Image and Point Cloud Alignment Model (LAM3D), a novel framework that enhances 3D mesh reconstruction from single images by utilizing point cloud data. LAM3D integrates a point-cloud-based network for generating precise latent tri-planes, followed by an Image-Point-Cloud Feature Alignment technique that enriches image features with robust 3D information. This approach allows for the production of high-fidelity 3D meshes while mitigating geometric distortions commonly associated with single-image inputs. The method demonstrates its effectiveness across various datasets, achieving state-of-the-art performance in high-fidelity 3D mesh reconstruction.
strength_weakness: strength_weakness: 1. The LAM3D framework's omission of texture mapping is a significant limitation, particularly when compared against methods like One-2-3-45 and CRM that integrate both geometric and texture reconstructions. This absence not only restricts the visual and practical applicability of the generated meshes but also raises concerns about the fairness of comparisons made in the paper. The inclusion of texture details is pivotal for realistic 3D reconstructions, and its absence in LAM3D suggests a crucial area for potential improvement. Furthermore, the comparisons with models that handle both geometry and texture may not provide a balanced view. It's more appropriate to compare LAM3D against models such as "Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction" (CVPR, 2024) and "D^2IM-Net: Learning Detail Disentangled Implicit Fields from Single Images" (CVPR, 2021), which are more aligned in terms of focusing primarily on geometric details.
2. LAM3D requires point cloud data during training, its applicability is notably restricted to specific categories or types of objects. This limitation indicates a constrained generalization ability, which may impede its deployment in diverse real-world scenarios that demand robust 3D reconstruction capabilities across varied object environments.
3.  The use of independent diffusion processes for each tri-plane is a novel approach, but the paper does not provide a comparative analysis of this method against more integrated approaches.
questions: questions: 1. Can you provide more details on how LAM3D handles various object categories, especially those not directly represented in the training datasets?
2. Could the authors elaborate on how LAM3D might be adapted or extended to include texture reconstruction?
limitations: limitations: While LAM3D effectively enhances mesh reconstruction quality by leveraging 3D point cloud data, its current implementation omits texture mapping. This omission not only limits the visual realism and practical utility of the generated 3D meshes but also makes direct comparisons with methods like One-2-3-45 and CRM, which include textural details, somewhat unfair.
paper_summary: paper_summary: This paper introduces a new 3D generation framework, LAM3D, which uses point cloud data and image-point-cloud feature alignment method to improve the geometry of the generation results. Authors use triplanes as representation, combining image feature as well as point cloud feature. For point cloud compression, they use point cloud transformer to encode the point cloud to triplane. For better generation, they encode the point cloud triplane to latent triplane, and trained three diffusion model to generate latent triplane. The whole method can generate untextured mesh within 6 seconds, showing better geometry results comparing baselines.
strength_weakness: strength_weakness: 1. There are already some work on Shape-Image-Aligned generation work such as 3DGen[1] and Michelangelo[2], authors should compare with these work and point out what are the advantages of latent triplane and three parallel diffusion model.
2. Parallel diffusion design may lose some 3D information. 

[1] Gupta, Anchit, et al. "3dgen: Triplane latent diffusion for textured mesh generation." arXiv preprint arXiv:2303.05371 (2023).

[2] Zhao, Zibo, et al. "Michelangelo: Conditional 3d shape generation based on shape-image-text aligned latent representation." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: 1. For the parallel diffusion part, why not try using one diffusion to output a tensor of size 2x32x96, then resize it to 3x2x32x32 (similar to the design in CRM)? What are the differences, and what are the advantages of parallel diffusion?
2. You could show more results on real-world data, such as MvImgNet[1] or photos taken by phones.
3. Some works based on DiT[2], like Sora and the concurrent work Direct3D[3], have shown great success. Have you tried using other diffusion models?
4. If LAM3D is open-sourced, it could greatly benefit the community. What are your plans for open-sourcing it?

[1] Yu, Xianggang, et al. "Mvimgnet: A large-scale dataset of multi-view images." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[2] Peebles, William, and Saining Xie. "Scalable diffusion models with transformers." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Wu, Shuang, et al. "Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer." arXiv preprint arXiv:2405.14832 (2024).
limitations: limitations: Yes.
paper_summary: paper_summary: The paper addresses the problems of multi-view consistency and geometric detail in image-to-3D generation. The proposed method, LAM3D, adopts a two-stage approach for training. In the first stage, the authors train a plane encoder and decoder to compress point clouds into a latent tri-plane representation. In the second stage, to align image and point cloud features within the same feature space, they train the diffusion model. During inference, the image is converted into a latent tri-plane by passing through the trained U-Nets from the second stage, and the features are decoded by the plane decoder trained in the first stage. A final mesh is extracted using the marching cubes algorithm from the reconstructed tri-plane. Experiments demonstrate the effectiveness of the proposed method in 3D generation from a single image, both quantitatively and qualitatively. Additionally, the authors provide an extensive ablation study on different design components.
strength_weakness: strength_weakness: (1) As mentioned in the limitation, LAM3D cannot generate a textured mesh, while baselines (One-2-3-45, LRM, and CRM) can. In generating 3D content, the accurate geometry of the mesh is important, but the corresponding texture is also a crucial component. It is unfortunate that this aspect cannot be generated. Although it would take more time, generating multi-view images based on the input image using a multi-view diffusion model like Zero-1-to-3 and then unprojecting them to the generated mesh could be a solution.

(2) Recently, many papers related to single image-to-3D generation have been published, and there are many papers like [1, 2] that were published 2 months before the NeurIPS 2024 submission deadline. Among these, I guess Michelangelo [1] has the best quality in geometry generation, so comparing this paper would make the argument more compelling.

[1] Zhao et al., Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation, NeurIPS 2023

[2] Vikram Voleti et al., SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion, arXiv 2024
questions: questions: (1) A minor question: does collecting points using KNN and then using the embedding created by PointNet improve performance compared to directly inputting the points extracted through FPS into the transformer?
limitations: limitations: The authors mention that the limitation of the current pipeline cannot generate texture, as discussed in Section 5.

Comments: title: Response to rebuttal
comment: Thank you for your feedback! As promised, we will refine the implementation details and pipeline figure, as well as any other areas that need improvement. We appreciate your support and consideration.
Thank you for your feedback! We acknowledge that our method may present some mismatches between the reconstruction and the reference image in certain details. This occurs because our approach uses a diffusion process to align image features with latent tri-plane features, which introduces some statistical variation and can lead to discrepancies in finer details. However, our method effectively addresses the geometry distortion problem and accurately reconstructs the overall structure from the reference image. We appreciate your insights and thank you again for your valuable comments.
Thank you for the response. I have read it as well as other reviews.

The authors have promised to address the missing implementation details and pipeline figure in a future revision, so there doesn’t seem to be any further cause for concern. Therefore, I will maintain my original rating of 7.
Thank you for your response. But I found that the mesh generated by your method does not resemble the original image(such as in your rebuttal file Figure 3 Line 3, the bottle). I have no more questions and I will keep my rating. Thank you for your work!