LAM3D: Large Image-Point Clouds Alignment Model for 3D Reconstruction from Single Image
Authors: Ruikai Cui
Xibin Song
Weixuan Sun
Steve Wang
Weizhe Liu
Shenzhou Chen
Taizhang Shang
YANG LI
Nick Barnes
Hongdong Li
Pan Ji
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Representation
3D Reconstruction
Feature Alignment
Primary area: generative_models

Abstract: Large Reconstruction Models have made significant strides in the realm of automated 3D content generation from single or multiple input images. Despite their success, these models often produce 3D meshes with geometric inaccuracies, stemming from the inherent challenges of deducing 3D shapes solely from image data. In this work, we introduce a novel framework, the Large Image and Point Cloud Alignment Model (LAM3D), which utilizes 3D point cloud data to enhance the fidelity of generated 3D meshes. Our methodology begins with the development of a point-cloud-based network that effectively generates precise and meaningful latent tri-planes, laying the groundwork for accurate 3D mesh reconstruction. Building upon this, our Image-Point-Cloud Feature Alignment technique processes a single input image, aligning to the latent tri-planes to imbue image features with robust 3D information. This process not only enriches the image features but also facilitates the production of high-fidelity 3D meshes without the need for multi-view input, significantly reducing geometric distortions. Our approach achieves state-of-the-art high-fidelity 3D mesh reconstruction from a single image in just 6 seconds, and experiments on various datasets demonstrate its effectiveness.

LAM3D: Large Image-Point-Cloud Alignment Model for 3D Reconstruction from Single Image
Ruikai Cui
Xibin Song?! 2
Weixuan Sun
Senbo Wang?
Weizhe Liu?
Shenzhou Chen
Taizhang Shang"
Yang Li
Nick Barnes
Hongdong Li"
Pan Ji
Australian National
University
Tencent XR Vision Labs
Abstract
Large Reconstruction Models have made significant strides in the realm auto- mated 3D content generation from single multiple input images. Despite their success, these models often produce 3D meshes with geometric inaccuracies, stem- ming from the inherent challenges of deducing 3D shapes solely from image data. In this work, we introduce a novel framework, the Large Image and Point Cloud Alignment Model (LAM3D), which utilizes 3D point cloud data enhance the fi- delity of generated 3D meshes. Our methodology begins with the development of a sit--cloud-bassee network that effectively generates precise and meaningful latent tri-planes, laying the groundwork for accurate 3D mesh reconstruction. Building upon this, our Image-Porne Paee-PPoonntoooieeee Feature Alignment technique processes single input image, aligning to the latent tri-planes to imbue image features with robust 3D information. This process not only enriches the image features but also facilitates the production of high-fidelity 3D meshes without the need for multi-view input, significantly reducing geometric distortions. Our approach achieves state-of-the-ard high-fidelity 3D mesh reconstruction from single image in just 6 seconds, and experiments on various datasets demonstrate its effectiveness.
Introduction
High-quality 3D mesh creation has drawn increasing attentions for its great potential in numerous fields, such video games, virtual reality, and films. Traditionally. 3D assets are usually created manually by expert artists or alternatively reconstructed from multi-view 2D images, which are
time-consuming. Recent proposed Large Reconstruction Models (LRMs) LS 62
from single view or few-shot images bring significant advancements for high-fidelit 3D content creation.
4% 52
Single view based LRMs [35 take single-view image input and transformers directly regress tri-plane-based based 3 neural radiance fields (NeRF) [31] for 3D content creation. However, shown in Fig. (b), due to ambiguity of 3D geometry from single image, single-view-based LRMs commonly suffer from geometric distortions, especially for the unseen areas of the input view. are proposed, where they typically first feed the single-vier image into multi-view image diffusion models [5]] 43 (28
Inspired by Hong er al. [15], LRMs with multi-view inputs 62
48 52 20
for sho- multi-view images generation. Then multi-view tri-plane features are generated and converted into 3D representations (NeRF 31], SDFs 34,, etc.) for 3D mesh creation. However, as shown in (c). the generated multi-view images lack guaranteed multi-view consistency, which can lead to geometric distortions.
To relieve these problems, propose to introduce 3D point cloud prior for single image based 3D mesh reconstruction with Large Siagg--Piit-CCe Alignment Model (LAM3D). LAM3D aims to
'The contribution of Ruikai Cui was made during an internship at Tencent XR Vision Labs. Corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Reference Image
(a) ULIP
(b) LRM
(c) CRM
(d) Ours
Figure 1: An example of single-ing single--mggeeeee reconstruction from ofte-of-thheart methods: (a) ULIP 63
(b) LRM [3]. (c) CRM [54], and (d) Ours (LAM3D).
transfer single image features into point cloud features, enriching the image features with accurate and effective 3D information for high-fidelity 3D) mesh reconstruction. The motivation behind this choice is obvious: 3D point cloud contains effective structure prior of 3D geometries. Mcanwhile, approaches 663 (64] of modality feature alignment have been proposed, aiming to learn unified representations of language, images, and point clouds for 3D understanding. Nevertheless, as shown in Fig. (a), these techniques fall short when they use the aligned image features to replace point clouds features for 3D mesh creation. This is largely due to the exclusive usage of contrastive loss [63 644 and the aligned feature size commonly with (512x (2Xx))(6) and (1024 x 1)44 which is inadequate fully capture the rich diversity of 3D shapes.
In this work, we extend the image and point cloud alignment for 3D mesh creation, and present the Large Maage-Point-Clluue Alignment Model (LAM3D) which utilize the prior of 3D point cloud for high-fidelity feature alignment. We start by designing an effective point cloud-based network with hierarchical tri-planes 3 for 3D mesh reconstruction. Following this, taking single image as input, image features are extracted using DINO (2). whilst LAM3D is employed to transpose these image features into tri-planes obtained via the point-cloud-based network. The tri-plane representation 3 allows for the tuning of image features with precise and effective 3D information, which essential for high-fidelity 3D mesh reconstruction. Mcanwhile, better retain 3D structure information and reduce the interference each plane (XY, XZ. YZ) in tri tri-plane features, we enlist the use of independent diffusion operations transfer image features to each respective plane. Note that the transferred features are with size of 3xCxM MCCxxX (3xx2x x222xx3xxx x2x32) in our experiments), which 63 and (1024x] 6644, thus contains more information for better 3D mesh creation. As shown in Fig. (),, more reasonable geometric mesh can be reconstructed by our approach.
larger than previous representation size of (512x1)
The main contributions our work are as follows:
We propose effective Large Mmage-Point-Cloute Alignment Model (LAM3D) for 3D mesh reconstruction from single image. By utilizing point cloud priors, LAM3D delivers precise image tri-plane feature transformation, thereby enhancing the quality and accuracy of 3D mesh creation.
To preserve 3D structural information more effectively, we employ independent diffusion processes that transmit image features to each respective tri tri-plane (XY, XZ, and YZ) for accurate 3D mesh reconstruction.
Our method significantly relieves geometric distortions caused by ambiguities of 3D shape inferred from the image modality, and experiments various of dataset demonstrate the effectiveness of our approach.
Related Works
Single Image 3D Reconstruction: Recently, LRM 15 was the first to demonstrate that regression model can be trained to robustly predict tri- planelan--ie based NeRF from a single-view image. Inspired by LRM, numerous large models 25 26 36 54 69
50 for 3D reconstruction have since been proposed. Among these, Instant3D 20 trains a text- toxtt-to-fewshheee sho-shot multi-vies image diffusion model, well multi-view LRM, to enable fast and diverse text-to-3D generation. Subsequent works have extended LRM by incorporating pose prediction from multi-view view images (52), combining with diffusion (62] and specializing on human data [56]. Tang et al. [8] introduced the Large Multi- View Gaussian Model to generate high-resolution 3D models from text prompts single-
53
view images. Wang al. [54) presented a single-image-to-311 generative model that generates six orthographic images, utilizing Flexicubes 42 3D representation. Concurrently, Wu et al. [58]

---

proposed Direct3D, a method that differs from ours primarily in its approach to feature extraction from point clouds. Direct3D utilizes learnable tokens for feature extraction, whereas our method introduces projection module to explicitly convert point cloud features into tri-plane structure.
Generative Modeling of 3D shapes: In recent years, multitude of studies have been dedicated to the generation of 3D shapes. Existing 3D generative models have been developed using variety of frameworks, including generative adversarial networks (GANs) (57 677, variational autoencoders
(VAEs) 47
32 [2], normalizing flows 665 32], autoregressive models 45 32]. energy-basce
models 49
and more recently, denoising diffusion probabilistic models (DDPMs) [0 68 27
24
60]. Luo et al. 30 pioneers the application of DDPMs for modeling the distribution
of raw point clouds. Fellowing Luo al. several works 49 24 4 explore generative modeling of 3D shapes in latent space to reduce computational complexity and enhance generation quality. For instance, LION clouds, while 3DQD [24], SDFusion [4] utilize DDPM to model Signed Distance Function (SDF) in the latent space of an autoencoder for 31D shape generation. Concurrently, other research efforts have investigated DDPMs for 3D shape generation with other representations, such mesh [27). occupancy grid 44], and neural radiance fields [33].
49 proposed an effective latent-space diffusion model to generate novel point
Feature Alignment/ Multimodal 3D: Multimodal approaches are mainly about image-text and image-text wereeeeexx---i----eee point-clouds modalities. Several works [2] 23 46 propose to lear interactions between image regions and caption words using transformer- based architectures, which show great predictive capability despite being costly to train. Meanwhile, CLIP
38 uses image and text encoders to output single image/text representation for each image-text pair, and then aligns the representations to unified latent space. Built upon CLIP 38 ULIP 63 and its successor, ULIP2 [64], aim to learn a unified representation that integrates images, text, and 3D point clouds. These methods establish a 3D representation space aligned with the CLIP latent space by leveraging synthesized 3D-text-image triplets. The most closely related work to ours is Michelangelo (66), which aligns point cloud features with frozen CLIP image/text embeddings through an autoencoder. Unlike Michelangelo, which employs contrastive loss to learn an aligned latent space, our approach uses diffusion to align 3D and 2D modalities. This results in a more continuous latent space, beneficial for 3D reconstruction.
3 Method
In this section, we present the detailed design of our approach. The full training process consists of two stages: Point Cloud Compression (Sec. 3.1 and Image-Point Poinge--onn--Clou Alignment (Sec. 3.2)..
3.1 Stage Point Cloud Compression
As shown in Fig. 2 we encode point clouds into tri-plane representation and then decode SDF values from the reconstructed tri-plane to extract 3D meshes via marching cubes (29). The point cloud compression consists of Initial Point Feature Extraction and Tri-plane Compression. Specifically, we propose a point-cloud- based 3D reconstruction network to convert point cloud into a latent tri-plane, which can retain rich 3D information. In the tri-plane compression module, the latent tri-plane is up-sampled to reconstruct tri- tri-planes followed by an MLP to compute SDF values. As shown in Fig. [ hierarchical tri plane are used, including latent tri-planes and reconstructed tri-planes, where the latent tri tri-p is designed for better image-point-eee poine---ot--loud feature alignment and the reconstructed tri-plane is for better mesh reconstruction.
3.1.1 Initial Point Feature Extraction
To compress point cloud into latent tri-planes, employ wansformer-basee approach to learn long-range interactions between points. We first use Furthest Point Sampling (FPS) to downsample the input point cloud to center points. Then, we gather the K-nearest neighbors of the center points together as a small point cloud and use a shallow PointNet [35] to embed each small point cloud an embedding. By doing so, reduce the sequence length of the point cloud, and then can use a transformer to process these embeddings.
After the transformer step, we need to collect the embeddings to form the three planes of the tri-plane structure. To this end, we use a projection operation. specific, each embedding feature associated with a knn center. For each plane (XY,XZ) YY), we diminish axis of the center points and
3

---

Stage 1: Point Cloud Compression
Point Feature Extraction
Initial
Tri-plane Compression
Tri-planc
Reconstructed
Projection
Tri-plane
Latent
Tri-plane
Plane Refiner
FPS
Center Points
Point Cloud
Encoder
Plane
Plane Decoder
Transformer
Embed
Sample Points
Tri-plane Interpolation
Stage Image-Point Cloud Alignment
Initial Noise
Marching Cubes
(O, 1)
SDF
UNets
MLP
DINO $
Mesh
Image l'eature
Imrge
Reconstructed Mesh
Figure 2: Overview of our method. Our method contains two training stage. Stage 1: we train an encoder-decoder structure take point clouds input and compress representation; Stage 2: we employ diffusion to align image modality to latent tri-planes obtained in stage 1. The diffusion step takes an initial noise and an image feature from freezed DINO feature encoder and progressively align the image feature to the latent tri tri-plane. Inference: To reconstruct 3D mesh from a single-view image, we use the alignment step, following the decoder (Plane Decoder, Plane Refiner) from the compression step, to predict tri-plane. Then, we can use algorithms like marching cubes to extract 3D meshes from the reconstructed tri-plane.
a latent tri-plane
voxelize the center points into plane of size IR CxxRR where C' is the channel dimension that is the same with the embedding dimension, and R is the plane resolution. We use average operation to aggregate embeddings associated with center points that land into the same voxel. As shown in Fig. 2 the point cloud is converted into initial tri-planes that faithfully preserves 3D information.
3.1.2 Tri-plane Compression
The initial tri- tri-planes is sparse tensor as many of its grids are empty if there is no center points land there. To obtain a more compact and expressive latent tri-plane representation, we adopt an plane autoencoder to further compress initial tri-planes to latent tri-planes.
Plane Encoder We find that directly aligning image features with initial tri-planes leads two drawbacks: (1) Aligning in high-resolution requires high computational complexity; (2) Empty grids and overwhelmed details in high-resolution tri-planes substantially increase training difficulty, leading to noisy results. To further compress the initial tri-planes into a more compact latent, we opt for a transformer-basee plane encoder. It consists of convolution layer down-sample the initial tri-planes followed by successive transformer blocks to achieve inter-plane correlations. In each transformer block, we follow Cui et al. (6] concatenate tri- tri-planes into one sequence and adopt linear attention [37] to process the sequence to model inter-plane correlations. In our settings, the extracted initial tri-plane down- wwwn-sdmpled by 4 times to obtain a point cloud latent tri-plane
t (3x2x32x32).
Plane Decoder We introduce hierarchical scheme to decode the latent tri-plane into a high-resolution reconstructed tri-plane, as depicted in Fig. 2 and for predicting the Signed Distance Function (SDF). Initially, the latent tri-plane is processed through a standard decoder that mirrors the encoder. This decoder comprises successive transformer blocks coupled with convolutional layer, which augments the latent trinp by factor of four.
Plane Refiner After Plane Decoder, we adopt intra-plane refinement module to further enhance the reconstruction quality. latent tri-plane generation, the original point clouds are condensed to form a compact tri-plane latent, the compression inaccuracies unavoidable. Therefore, propose

---

an asymmetric decoding of the tri tri-plane, which is accomplished by individually refining each plane with plane refiner module subsequent to the conventional decoder. Specifically, the plane refiners are three small-scale UNet [41] models to refine each plane separately. We find that refining each plane individually extensively improves mesh quality while retains inter-plane correlations as we demonstrated in Appendices. Tab. 5 Finally, we can query SDF values of any spatial position using the SDF-MLP by interpolating features on the reconstructed tri-plane and extract 3D meshes using
Marching Cubes [29]
3.1.3 Compression Loss Function
The training objective for point cloud compression comprises a KL penalty and a geometry loss. To align image features onto 3D-aware latent space, we attempt to preserve 3D prior to the greatest extent, even in the most compact compressed tri-plane latent. However, we discovered that the compressed tri-plane latent fails to maintain the 3D prior, leading to artifacts in the mesh outcomes if we merely employ an arbitrary encoder to compress the initial tri-plane without explicit constraints. Hence, we propose to concurenttyy supervise the latent tri- tri-plane and reconstructed tri tri-plane P using the ground truth SDF and two distinct SDF MLPs (L, and p)). To this end, we combine the geometric SDF loss and latent regularization train the compression module in an end- end-to-end manner. Formally, to allow the predicted tri- tri-plane recover SDF values, we use the geometry loss:
Egen Ldds + Lnoomal.
These two terms are defined as:
sddf
A Σ |1 + A2
Σ PER
[1Dr(p) dplls
PEll
Lnormal = Aa
Σ
/Vv - -ppl:
PEAl
where 2o and Q are sets of query points sampled from object surface and off-surface points sampled from 1J. The grouth truth SDF values and surface normal denoted as dp and ny. respectively, The gradient ppp)
(0 @bp(p) (bp(p) represents the direction of the steepest change in SDF It can be computed using finite difference, egg., the partial derivative for the X-axis component reads (((()) (=(00()0.0)) 1pppppp(600p0 where ö is the step size. Similarly, the latent sdf loss Øx
Lisay is defined as:
Liddf d Σ [|0,(D)1 + As Σ (1) dpll.
PEñLo
PEl
We use the corresponding MLP predict SDF value from either latent tri-plane or the reconstructed tri-plane P. Specifically, given a query position p. we project it to each of the planes and retrieve the feature vectors Fzy. Fzs: Fyz Faz via bilinear interpolation. Then, the signed distance can be predicted as: (pp) = MLP(Fag my ① Faz @ Fys), where @ denotes element-wise summation.
Overall, the training objective is defined as fellow:
Leomp Lyea + Lisdf + LKL,
(5)
where LK is a KL penalty which leans towards a standard normal distribution on the learned latent tri-plane, akin to a Variational Autoencoder [19
3.2 Stage 2: Image-Point-CUn CCoud Alignment
We align image and point cloud features within the same feature space. Since the point clouds have been compressed into latent tri-planes with 3D-aware point cloud priors in stage 1, we can project the image features onto this 3D-aware latent. We experiment with two approaches to accomplish this goal. First, following reconstruction methods such as LRM [15), we can utilize a transformer-basee structure align the image features with the point cloud tri- tri-planes, employing a distance metric like L1 or L2 loss. Alternatively, we can employ a probabilistic diffusion approach, wherein we initiate from random noise and denoise it, guided by the input image targeting the point cloud domain.
Our experiments reveal that the transformer-hase hased model yields unstable results. This instability arises because 3D reconstruction from single image is an ill-posed problem, the image lacks information
5

---

about the occluded object parts. The 3D structural information in an image is not as rich as that in point clouds. Consequently, if we simply adopt a deterministic approach, the alignment result tends to be the average of all possible outcomes for regions that are not sufficiently observed [15]. Thus, we choose diffusion model can leverage its prior to envision unohserved regions and generate clear predictions for these areas, as demonstrated in previous 2D prior works [55 [2].
probabilistic alignment approach. Intuitively, the diffusion model
We provide more analysis Appendices, D.2
3.2.1 Alignment Model Structure
As shown in Fig. first, we use pre-trained DINO [2) image encoder to convert images to a S x D dim feature Zimg where S is the squence length and D is the feature dimension. Subsequently, the image features serve conditioning information in the diffusion process, mapping the image feature from the image modality to the point cloud modality.
that depend on a single diffusion model to map an arbitrary ID ordering vector, we employ three parallel UNets to map the image feature each of the three latent planes, separately. Since each tri-plane contains a 2D-image-lik feature, as demonstrated in Appendices, Fig. Prior methods [22 o use a single UNet denoise concatenated planes. However, the UNet is designed without 3wa-a prior, meaning that there will be adjacent pixels without direct relation due to the convolution operation being local operation, leading to artifacts.
Unlike previous methods [ 6 [2]
3.2.2 Alignment Loss Function
We utilize diffusion model to align the image and the point cloud features. Diffusion Models [4] 40
are designed learn a data distribution ((2g) by progressively denoising a normally distributed variable. The learning process is equivalent to performing the reverse operation of a fixed Markov
Chain with a length of T, which transforms latent 2n into purely Gaussian noise 2T NV I) over T time steps. The forward step in this process is defined as:
q 21-1 = (zzi v1 1,
(6)
where noisy variable z, derived by scaling the previous noise sample zt-1 with v - Be and adding Gaussian noise following variance schedule $1, ,7.
We train three parallel diffusion models on each of the three planes by learning to reverse the above diffusion process. To achieve this, we adopt the approach proposed by Aditya et al. [39], wherein we use three neural networks that takes 2, to directly predict 2g corresponding planes. Specifically, given uniformly sampled time step f from the set {1, T), we generate 2; by sampling noise from the input latent tri-plane Zu- A cone-conditioned" denoising autoencoder consisted of three parallel UNets [40], denoted by V, is employed to reconstruct 20 from = given the alignment source, i.e., the image latent feature Zimg. The objective of the alignment is given by:
XY, XZ, and YZ
Lalign [ Zimg: (t)) =oll2;
where 7-) represents positional embedding and || 112 denotes the mean squared error (MSE) loss.
4 Experiments
Following previous works [54] 15), our model is trained on subset of Objaverse 9 with 140k filtered 3D assets, and evaluated unseen Objaverse objects and Google Scanned Objects 10] The Objaverse dataset contains over 800k 3D assets. It was filtered to remove objects with thin faces and repelaed//imiilar buildings, resulting in final training set of 140k objects. More details can be
Dataset:
found in Appendices. B
Baselines: To assess the effectiveness of approach, we compare method with state-of-
the-art approaches. including One-2-3-45 25) LRM [1], SyncDreamer (66. Wonder3I) (8).
Magic 123 (36]. TGS 669. LRM 48 and CRM (54
4.1
Implementation Details
During training, we employ a two-stage training pipeline. In stage 1, a point cloud compressor trained, comprising transformer-basee point feature extractor and tri tri-plane autoencoder. Given

---

Input
Image
One-2-3-45
LRM
CRM
Ours
Input Image
One-2-3-45
LRM
CRM
Ours
Figure 3: Rendered images of shapes reconstructed by various methods from single images. The upper samples are from Objaverse and the lowers are from Google Scanned Objects.
point cloud, we sample 8192 points to feed them into the point feature extractor followed by KNN-based sampler obtain 512 point features. Subsequently. the point features are projected onto three orthogonal planes to generate tri-plane with a 128 x 128 spatial resolution. Then we compress the tri-plane into a latent tri-plane size x3x2x32x33 which has a feature dimension components including the point feature extractor, tri-plane autoencoder, and two SDF MIPs, are trained in an end-to-end manner. We utilize 32 NVIDIA V V100 GPUs to train for 20 epochs with batch size of 25 and Adam [18] optimizer, which takes approximate days.
All
In stage the compressed latent tri-planes are utilized as supervision signals train our alignment network. For an input image of size (112x512). we use pre-trained DINO ( to extract image feature of size 1025 x 768, where 768 is the feature dimension and 1025 is 32 x image patches plus global image feature. The image features serve as conditioning signal through cross- cross-attention, guiding the diffusion process of the three UNets to generate three planes. During the alignment training, only the UNets are trained, while the DINO remains fixed. The training cost for the alignment relatively low since we can pre-encode all point clouds as well as all the source images. We employ 8 V V100 GPUs train the second stage model for 2 days with batch size of 64.

---

Table 1: Quantitative comparisons for the geometry quality between our method and basclines.
Method
Chamfer Dist.
Vol. IoU* F-Sco.(%)
One-2-3-45
25
0.0172
0.4463
72.19
SyneDreamer
26
0,0140
0.3900
75.74
Wonder3D
28
0,0186
0.43998
76.75
Magic123 36
0.0188
0.3714
60.66
TGS [69
0.0172
0,2982
65.17
Open RM
15
13
0,0168
0.3774
63.22
LGM 48
0,0117
0.4685
68.69
CRM
54
0.0094
0.6131
79.38
Ours
0.0083
0.6235
85.40
(a) Reference (b) vector-512 (c) vector-6114 (d) w/o Lisaf
(e) Full model
Figure 4: Qualitative comparisons of different latent representations.
The inference stage requires only a single-view image of an object. The process starts by encoding this image into feature representation using the DINO image encoder. This image feature is then aligned to a latent tri-plane representation through three independent diffusion UNets. The latent tri-plane is subsequently upsampled to high-resolutee resolution tri-plane using the plane decoder and refiner from the first training stage. We then employ the SDF MLP to decode the signed distance for any query position the tri-plane, which allows us the Marching Cubes algorithm to reconstruct mesh from the tri-plane.
4.2 Comparisons with state-of-the-arts
Qualitative Results: Fig. presents qualitative comparisons between our approach and other state-of-the-art approaches including One-2-3-45 (25), LRM [15] and CRM [54]. Since LRM is not open-sourced, we use Rp 13] an open-sourced implementation of LRM for comparisons. For the other baselines, we use their official codes and model weights. As shown in Fig. While impressive results can be obtained with these methods, geometric distortions often occur due to ambiguities caused by single image or multi-view inconsistency. Additionally, these approach commonly fail for unseen areas of the input image, such as the inside of a cup. contrast, our method leverages point cloud priors to reconstruct 3D meshes with better geometry and more details than all other baselines. Moreover, our model does not rely on multi-view consistency, unlike One-2- 3-45, LRM, and CRM, thanks the point cloud prior. This results in significantly reduced geometric distortions, as seen in the box, sofa and bowl examples in Fig.
3
we use the Google Scanned Objects (GSO) dataset [10 evaluate the effectiveness of our method. In line with CRM (54] and One- 2-3-45 [25]. we use Chamfer Distance (CD) [59 7, Volume IoU and F-Score with a threshold of 0.05 to evaluate the mesh geometry. The results are shown Tab. [ can be seen that our method outperforms all of the baselines on all the evaluation metrics. This demonstrates the effectiveness of our method for 3D reconstruction. Notably, our model generates mesh within only seconds on an NVIDIA V100 GPU.
Quantitative Results: following previous studies 54
48
15

---

Reference image
Parallel Diffusion
Single Diffusion
Reference image
Parallel Diffusion
Single
Diffusion
Figure 5: Green objects are generated from parallel UNets and gray samples are from single UNet.
4.3
Ablation Study
For our ablation study, we use a small subset of the Objaverse dataset containing 3000 game assets (weapons, gear, etc.) for training, and 216 objects for evaluation.
Effectiveness latent tri-plane representation: In our method, we compress the point cloud as latent tri ti-plane representation, which is then used for maage-point-coud feature alignment. In this study, we evaluate the effectiveness of this proposed latent tri-plane representation.
We compare three different variants in our ablation study: a) latent tri- rl-plane w/ Lisay our introduced hierarchical decoding strategy, constraining latent tri-plane with a SDF loss; b) latent tri tri-plane w/o Lisat, latent tri-plane without the SDF loss; c) vector, vector with dim (61 (6114x1) (same size with (3 xx2x32x32)) used to represent the point cloud latent. We train the stage compression models for point cloud-based reconstruction. As shown in Tab. 2 simply using latent vector representation without spatial structure does not perform well with performance of 4.81. However, converting the vector representation to tri-plane improves the performance to 1.95. Moreover, tri-plane with latent SDF loss (Ours in Tab. 2 can further refine the final results to 1.81. This emphasizes the importance of retaining awae structure in the point cloud latent representation and the necessarity of introducing the latent SDF loss Ciedf-
In Fig. 4 we provide illustrations of different sizes and representations. We observe that vector size of 512 performs poorly, which is inadequate to represent the 3D shapes. A vector with dimensions of 6114 x 1 can recover the structure of 3D mesh, but details such as the trigger are missing. In Fig. 4 (d), we show that tri-plane structure can improve the visual performance with better trigger. In Fig. 4 (e), further refined results with better details can be obtained, validating the effectiveness of the latent SDF loss.
Table 2: Comparisons of different representations.
Table 3: Quantitative comparison between
Method
CD(x 10')1
parallel diffusion UNet and single UNet.
Single Diffusion
Parallel Diffusion
UL.IP 63
35.23
Method
vector (61 14x1)
4.81
latent tri-plane w/o Li.df
1.95
CD(X10*)
3.92
2.59
latent tri-plane w/Crsay (Ours)
1.81
Image-point-cloud alignment model vs. ULIP: In order to validate the effectiveness of our proposed image ponge--oott-cloud alignment model, we compare our method with UL.IP 63,, which is designed for point cloud feature alignment with text and images.
ULIP-based image reconstruction: ULIP uses pre-trained CLIP and point cloud encoder to encode text, images, and point clouds to feature of size (512x 1). Contrastive loss is utilized for aligning the features of point clouds, text, and images. To evaluate the performance of ULIP for 3D reconstruction. we design decoder network that decodes the (512x 1) --dimension point cloud feature vector a 3D mesh. We then use the decoder network to directly replace the point cloud features with the aligned image feature obtained by ULIP for mesh reconstruction. As shown in Tab. 2 results obtained by ULIPP (63 are unstable, leading to worse performance in charmer distance, which proves that the scheme feature alignment used in ULIP 63 works poor for 3D reconstruction. On contrary, our method can obtain better results.

---

Parallel Diffusion vs. Single Diffusion: To better retain 3D structure information and reduce the interference of each plane (XY, XZ, YZ) in latent tri-planes, we enlist the use of independent parallel diffusion operations to transfer image features to each respective plane. We valid the effectiveness of using three parallel diffusion operations to do the feature alignment instead of using single UNet in Tab. 3 Parallel UNet variant has smaller CD error, and we show qualitative comparisons in Fig. [ We can observe that, thanks to the parallel UNet design, our model can focuses on aligning image feature to specific plane and thus presents better visual quality. For example, the surface of the ice the single UNet variant, while ours can maintains the continuity of the surface. Notably, to compare equally, we reduce the parameters of parallel diffusion to guarantee the same parameter size of parallel diffusion and single diffusion.
cream bar is broken the reconstruction
We provide more ablation studies as well as more visualization results in the appendix.
5 Conclusions
In this work, present the Large Image and Point Clouds Alignment Model (LAM3D), which harnesses the prior of 3D point clouds for high-fidelity 3D mesh creation. We design an effective honnt-loud-based network to compress point cloud into compact and meaningful latent tri-plane. Subsequently, taking a single image as input, Image-Point-Cloude Feature Alignment is designed to transfer image features to latent tri-planes, enriching the image features with accurate and effective 3D information for high-fidelity 3D mesh reconstruction. We believe that LAM3D has potential to improve 3D content creation and assist the workflow of digital artists.
Limitations and Future Work: The main goal of the proposed LAM3D is geometry reconstruction but texture reconstruction is included, so that method cannot achieve textured mesh reconstruction. We will further extend LAM3D to geometric and texture reconstruction in the future.
Broader Impact: We hold the view that LAM3D has the potential to enhance 3D content creation and aid digital artists in their workflow. LAM3D was conceived with these applications in mind, and we hope can evolve into a useful tool that reduce the labour cost in 3D asset production. While do not foresee any immediate harmful uses for LAM3D, we believe it is crucial for users to exercise caution to minimize impacts, considering that the alignment framework can also be employed for harmful intents.
References
[1] Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning repre- sentations and generative models for 3d point clouds. In International conference on machine learning, pages 40-49. PMLR, 2018
[2] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660 2021.
[3] Eric R Chan, Connor Z Lin, Matthew A Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, et Efficient geometry- aware 3d generative adversarial networks. In Proceedingy of the IEEE/CVF conference on computer vision and pattern recognition, pages 16123-16133 2022.
[4] Yen-Chi Cheng, Hsin-Ying Ying Lee, Sergey Tulyakov, Alexander G Schwing, and Liang- Lan Gui. Sdfusion: Multimodal 3d shape completion, reconstruction, and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4456-4465, 2023.
[5] Gene Chou, Yuval Bahat, and Felix Heide. Diffusson sdf: Conditional generative modeling of signed distance functions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2262-2272, 2023.
[6] Ruikai Cui, Weizhe Liu, Weixuan Sun, Senbo Wang, Taizhang Shang, Yang Li, Xibin Song, Han Yan, Zhennan Wu, Shenzhou Chen, et al. Neusdfusion: A spatial-aware generative model for 3d shape completion, reconstruction, and generation. arXiv preprint 2403v:2403.188 18241, 2024.
10

---

[7] Ruikai Cui, Shi Qiu, Saeed Anwar Jiawei Liu, Chaoyue Xing, Jing Zhang, and Nick Barnes. P2c: Self- Supe-supe point cloud completion from single partial clouds. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 14351-14360 2023.
[8] Ruikai Cui, Shi Qiu, Saeed Anwar, Jing Zhang, and Nick Barnes. Energy-based residual latent transport for unsupervised point cloud completion. arXiv preprint arXiv:2211.0663 2022.
[9] Matt Deitke, Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13142- 11142-131533 13153, 2023.
[10] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Rey- mann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553-2560 IEEE, 2022.
[11] Lin Gao, Tong Wu, Yu-Jie Yuan, Ming Lin, Yu-Kun Lai, and Hao Zhang. Tm-net: Deep generative networks for textured meshes. ACM Transactions on Graphics (TOG), 10(6):1-15, 2021.
[12] Anchit Gupta, Wenhan Xiong, Yixin Nie, Ian Jones, and Barlas Oguz. 3dgen: Triplane latent diffusion for textured mesh generation. arXiv preprint arXiv:2303.05371, 2023.
[13] Zexin He and Tengfei Wang. OpenIrm: Open-source large reconstruction models. https //github. hamm//DDopaa//peniit 2023.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in neural information processing systems, pages 6840-6851, 2020.
[15] Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. arXiv preprint / 1.33/1..000 .04400, 2023.
[16] Jingwei Huang, Hao Su, and Leonidas Guibas. Robust watertight manifold surface generation method for shapenet models. arXiv preprint arXiv:1802.016986 2018
[17] Jinwoo Kim, Jaehoon Yoo, Juho Lee, and Seunghoon Hong. Setvae: Learning hierarchical composition for generative modeling of set-structured data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15059-15068, 2021.
[18] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[19] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proceedings of International Conference on Learning Representations, 2014.
[20] Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311. 1..6214. 2023.
[21] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694 9705, 2021.
[22] Tianhong Li, Huiwen Chang, Shlok Mishra. Han Zhang, Dina Katabi, and Dilip Krishnan. Mage: Masked generative encoder to unify representation learning and image synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2142-2152, 2023.
[23] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for lasion-language tasks. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK. August 23-28, 2020, Proceedings, Part XXX 16, pages 121-137. Springer, 2020.
11

---

[24] Yuhan Li, Yishun Dou, Xuanhong Chen, Bingbing Ni, Yilin Sun, Yutian Liu, and Fuzhen Wang. Generalized deep 3d shape prior via part-discretized diffusion process. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16784-16794, 2023.
[25] Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. In Advances in neural information processing systems, 2023.
[26] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. Synedreamer: Generating multiview-consisere images from Sinle-view image. arXiv preprint arXiv:2309.034555 2023.
[27] Zhen Liu, Yao Feng. Michael Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu. Meshdiffusion: Score-based generative 3d mesh modeling. In The Eleventh International Conference on Learning Representations, 2022.
[28] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.1500 2023.
[29] William E. Lorensen and Harvey E. Cline. Marching cubes: A high resolution 3d surface construction algorithm. In Maureen C. Stone, editor, Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPHI 1987, Anaheim, California, USA, July 27-31, 1987, pages 163-169, ACM, 1987.
[30] Shitong Luo and Wei Hu. Diffusion probabilistic models for 3d point cloud generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2837-2845, 2021.
[31] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
[32] Paritosh Mittal, Yen-Chi Cheng. Maneesh Singh, and Shubham Tulsiani. Autosdf: Shape priors for 3d completion, reconstruction and generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 306-315, 2022.
[33] Norman Müllrr, Yawar Siddiqui, Lorenzo Porzi, Samuel Rota Bulo, Peter Kontschieder, and Matthias Nießner. Diffrf: Rendering-guide guided 3d radiance field diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4328 4338,, 2023.
[34] Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165-174, 2019.
[35] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
[36] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, et al. Magic One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint 2rXiv:2306.17443 2023.
[37] Zhen Qin, Weixuan Sun, Hui Deng, Dongxu Li, Yunshen Wei, Baohong Lv, Junjie Yan, Lingpeng Kong, and Yiran Zhong, cosformer: Rethinking softmax in attention. arXiv preprint arXiv:2202.08791 2022.
[38] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PMLR, 2021.
12

---

[39] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 00X1v22204.06/25, 1(2):3 2022.
[40] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Pruceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 10684-10695, 2022.
[41] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. Medical image computing and computer-assisted intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234-241. Springer, 2015.
[42] Tianchang Shen, Jacob Munkberg, Jon Hasselgren, Kangxue Yin, Zian Wang, Wenzheng Chen, Zan Gojeic, Sanja Fidler, Nicholas Sharp, and Jun Gao. Flexible isosurface extraction for gradient-based mesh optimization. ACM Transactions on Graphics (TOG), 42(4):1-16, 2023.
[43] Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zero +23++; a single image to consistent multi-view diffusion base model. arXiv preprint arXiv: 15110, 2023.
[44] J Ryan Shue, Eric Ryan Chan, Ryan Po, Zachary Ankner, Jiajun Wu, and Gordon Wetzstein. 3d neural field generation using triplane diffusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 20875-20888 2023.
[45] Yongbin Sun, Yue Wang. Ziwei Liu, Joshua Siegel, and Sanjay Sarma. Pointgrow: Autore- gressively learned point cloud generation with self-attention. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 61-70, 2020.
[46] Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. arXiv preprint arXiv: 1rXiv:1908.074400 077990 2019.
[47] Qingyang Tan, Lin Gao, Yu-Kun Lai, and Shihong Xia. Variational autoencoders for deforming 3d mesh models. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5841-5850, 2018
[48] Jaaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. ggm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint 40X1v:2402.05054, 2024.
[49] Arash Vahdat, Francis Williams, Zan Gojeic, Or Litany, Sanja Fidler, Karsten Kreis, al. Lion:
Latent point diffusion models for 3d shape generation. Advances in Neural Information Processing Systems, 35:10021-10039 2022.
[50] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts, David Pankratz, Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. Sv3d: Novel multi-view synthesis and 3d generation from single image using latent video diffusion. In European Conference on Computer Vision, pages 439-457. Springer, 2025.
[51] Peng Wang and Yichun Shi. Imagedream: Image-prone prompt multi-view diffusion for 3d generation. arXiv preprint arXiv:2312.02200 2023.
[52] Peng Wang. Hao Tan, Sai Bi, Yinghao Xu, Fujun Luan, Kalyan Sunkavalli, Wenping Wang, Zexiang Xu, and Kai Zhang. Pf-lrm: Pose-fre free large reconstruction model for joint pose and shape prediction. arXiv preprint arXiv:231 120244. 2023.
[53] Yizhi Wang, Wallace P. Lira, Wengi Wang, Ali Mahdavi-Amiri, and Hao Zhang, Slice3d: Multi-slice, occlusion-revealing single view 3d reconstruction. In IEFF/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pages 9881-9891. IEEE, 2024.
[54] Zhengyi Wang, Yikai Wang, Yifei Chen, Chendong Xiang, Shuo Chen, Dajiang Yu, Chongxuan Li, Hang Su, and Jun Zhu. Crm: Single image to 3d textured mesh with convolutional reconstruction model. arXiv preprint arX5::4433.00034, 2024.
13

---

[55] Chen Wei, Karttikeya Mangalam, Po-Yao Yao Huang, Yanghao Li, Haoqi Fan, Hu Xu, Huiyu Wang, Cihang Xie, Alan Yuille, and Christoph Feichtenhofer. Diffusion models as masked autoencoders. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 16284-16294, 2023.
[56] Zhenzhen Weng. Jingyuan Liu, Hao Tan, Zhan Xu, Yang Zhou, Serena Yeung-Levy, and Jimei Yang. Single-view 3d human digitalization with large reconstruction models. arXiv preprint a4Xi 2024.
[57] Jiajun Wu, Chengkai Zhang, Tianfan Xue, Bill Freeman, and Josh Tenenbaum. Learning a probabilistic latent space of object shapes via 3d generative-adveerarial modeling. Advances in neural information processing systems, 29, 2016.
[58] Shuang Wu, Youtian Lin, Feihu Zhang, Yifei Zeng, Jingxi Xu, Philip Torr, Xun Cao, and Yao Yao. Direct3d: Scalable image-to-3d generation via 3d latent diffusion transformer. CoRR, abs/2405. 14832, 2024.
[59] Tong Wu, Liang Pan, Junzhe Zhang, Tai Wang, Ziwei Liu, and Dahua Lin. Density-aware chamfer distance as a comprehensive metric for point cloud completion. arXiv preprint ftf::2111.127022. 2021.
[60] Zhennan Wu, Yang Li, Han Yan, Taizhang Shang, Weixuan Sun, Senbo Wang, Ruikai Cui, Weizhe Liu, Hiroyuki Sato, Hongdong Li, et al. Blockfusion: Expandable 3d scene generation using latent tri- tri-plane extrapolation. arXiv preprint arXiv:2401. 17053, 2024.
[61] Jianwen Xie, Yifei Xu, Zilong Zheng, Song-Chun Chun Zhu, and Ying Nian Wu. Generative pointnet: Deep mnergy-based learning unordered point sets for 3d generation, reconstruction and classification. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 14976-14985, 2021.
[62] Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model, 2023.
[63] Le Xue, Mingfei Gao, Chen Xing, Roberto Martín Marttn, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip: Learnnng unified representation of language, images, and point clouds for 3d understanding. In Proceedings of the /EEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1179-1189, June 2023.
[64] Le Xue, Ning Yu, Shu Zhang, Junnan Li, Roberto Martín-Martín, Jiajun Wu, Caiming Xiong, Ran Xu, Juan Carlos Niebles, and Silvio Savarese. Ulip-2: Towards scalable multimodal pre-training for 3d understanding. arXiv preprint arXiv:2305.002275 2023.
[65] Guandao Yang. Xun Huang, Zekun Hao, Ming- Yu Liu, Serge Belongie, and Bharath Hariharan. Pointflow: 3d point cloud generation with continuous normalizing flows. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4541-4550, 2019.
[66] Zibo Zhao, Wen Liu, Xin Chen, Xianfang Zeng. Rui Wang. Pei Cheng, Bin Fu, Tao Chen, Gang Yu, and Shenghua Gao. Michelangelo: Conditional 3d shape generation based on shape-image- text aligned latent representation.
Alice Oh, Tristan Naumann, Amir Globerson. Kate Saenko, Moritz Hardt, and Sergey Levine, editors, Advances in Neural Information Processing Systems 36: Annual Conference on Neural Information Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 16, 2023, 2023.
[67] Xinyang Zheng, Yang Liu, Pengshuai Wang, and Xin Tong. Sdf-stylegan: Implicit sdf-based stylegan for 3d shape generation. In Computer Graphics Forum, volume 41, pages 52-63, Wiley Online Library, 2022.
[68] Linqi Zhou, Yilun Du and Jiajun Wu. 3d shape generation and completion through voxet diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5826-5835, 2021.
[69] Zi-Xin Zou, Zhipeng Yu, Yuan-Chen Guo, Yangguang Li, Ding Liang, Yan-Pei Cao, and Song-Hai Zhang. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. arXiv preprint 2xXiv:2312.091441 2023,
14

---

A Diffusion Models Formulation
Diffusion Models are probabilistic models designed learn data distribution 20 gradually denoising a normally distributed variable. This process corresponds to learning the reverse operation of a fixed Markov Chain with length of T. The inference process works by sampling random noise 2y and gradually denoising it until it reaches meaningful latent 2o. Denoising diffusion probabilistic model (DDPM) [14] defines a diffusion process that transforms latent 2p to white Gaussian noise 2T ~ No,, 1) in T time steps. Each step in the forward direction is given by:
((20) by
T
q ... oT|20)
II (=c((20-1)
(8)
[=1
g( = Nzzi /1 1,
The noisy latent 2, is obtained by scaling the previous noise sample 2 with v Bg and adding Gaussian noise with variance 3e at timestep During training, DDPM reverses the diffusion process, which is modeled by a neural network V that predicts the parameters jou (zz, t) and Ev (z,, () of a Gaussian distribution.
pv po(z=1122) = (2 (ze, t), Eg (2x, t))
(10)
With ag == 1 - Bg and ā = IT== C'.. we can write the marginal distribution:
q = (2: (1 az)
(11)
2 = 0, + v1 āje
(12)
where e No0 1). Using Bayes' theorem, we calculate q(21- [ee 20) in terms of B, and (z 20) which are defined as follows:
3. 1 0x-1
Bt.
(13)
R4 (2, 20) = V04-18
Vas (1 Ōz-1)
20 +
(14)
1 1-ai
1
20) = =1(22-1;; 20), B,1)
(15)
Instead of predicting the added noise as in the original DDPM, in this paper, we predict 2g directly with a neural network V following Aditya et al. (39]. The prediction could be used in Eq. 14 to produce Mig (2,, t). Specifically, with uniformly sampled time step from (1,..., T), we sample noise to obtain žı from input latent vector sp. A time-conditioned denoising auto-encoder v learns to reconstruct 2g from ãt, guided by the image feature Zimg- The objective of latent tri-plane diffusion reads
Ciam = [2 Zimg. (()) - 20 2|
(16)
where 7(-) is a positional encoding function and || |12 denotes the Mean Squared Error loss.
B Dataset Preparation
The Objaverse contains over 800k 3D assets. However, there many low-quality 3D assets that are not suitable for training a 3D reconstruction, such as object with only thin faces. We filter those object by removing object with only thin faces, repeated and similar buildings and senses. Finally, we obtaín 140k object for training.
For the first stage compression training, sample data from the object surface. Given 3D object mesh we first pre-process it to ensure it forms a watertight shape using the method proposed by Huang et al. [16]. Then, we normalize the shape within the [-1, 1 box. Subsequently, we randomly sample on-surface points from the shape surface and uniformly sample off-surface points with ground truth SDF values in the 1, 1]3 space. In addition to the signed distance supervision, leverage normal directions as an extra guidance to achieve detailed surface modeling. Consequently, we also sample the normal vector for each on-surface point.
For the alignment stage training, we render each object from 48 fixed camera position equally distributed on a sphere covering the object. However, do not all 48 rendered images for the
15

---

Figure 6: Visualization of latent tri tri-plane by converting the value density.
second stage training. Instead, count the area of observed region and only select the image with the largest observed area to align its feature to the point cloud modality at the second stage. This operation aims to reduce the ambiguity caused by symmetry of objects. By doing this operation, also have the benefit that camera parameters are no longer needed to do the inference, the second stage model will determine this by itself in and end-to-end manner.
C Evaluation Metrics
We provide detailed formulation of metrics that are not explicitly defined in the main manuscript to enhance understanding of our experimental outcomes
Chamfer Distance (CD): This metric measures the average distance between two point sets, usually the ground truth and the predicted 3D reconstruction. Given two point sets A and B, the Chamfer Distance defined as:
CD(A,B)
= A
Σ aеΛ
min a b112 +
BEB
B
min a -112, aEA
(17)
BEB
where A A|and|||/ B denote the number of points in sets and B, respectively, and ||a - bl12 represents the squared Euclidean distance between points and b. Note that CD is symmetric and non-negative distance measure.
To compute the CD for 3D reconstruction evaluation, we first sample 2048 points from the surfaces of the predicted mesh and the ground truth mesh. Then, we calculate the Chamfer distance between the point sets, as defined above.
F-Score: This metric is measure of the trade-off between precision and recall, which commonly used metrics for evaluating the quality of a 3D reconstruction. Given a threshold value т, the F-Score defined as:
F(r) precision(T) recall(T)
(18)
precision(? (T) + recall(7)"
where precision(T) and recall(T) are the precision and recall values at the given threshold respec- tively. our experiment, we the threshold 0.05 w.r.t. the Chamfer distance.
Volume Intersection over Union (IoU): Volume Intersection over Union (IoU) is another metric used for evaluating the quality of 3D reconstructions. It measures the ratio of the volume of the intersection between the predicted mesh and the ground truth mesh to the volume of their union. The IoU is defined as:
IoU volume of intersection
(19)
volume of union
To compute the IoU for 3D reconstruction evaluation, we first convert the predicted mesh and ground truth mesh into Signed Distance Functions (SDFs) with voxel resolution of 128 128 128. All negative voxels in the SDFs represent the interior of the object. Then, we calculate the volume of the intersection and the volume of the union between the two SDFs. Finally, compute the IoU using the formula above.
D
More analysis
D.1 Effect of latent tri-plane resolution and feature dimension
We conduct experiments to study the effect of latent tri-plane resolution and feature dimension. We design four compared variants of our full model. There are two aspects that needs to be considered, i.e.
16

---

Table 4: Quantitative comparison for the geometry quality between different spatial resolution and feature dimension of latent tri- tri-plane on point cloud 3D mesh.
Spatial Reso.
Feature Dim.
#parameters
CD(x10")1
x 8
2
384
4.15
16 16
2
1536
2.12
32 x 32
1
3072
1.87
32 32
2
6144
1.81
32 x 32
4
12288
2.63
Reference image
Reference image
Probabilistic
Deterministic
Probabilistic
Deterministic
Figure 7: Alignment approach ablation study. We evaluate the single view reconstruction capability of deterministic vs. probabilistic approaches. Green objects are constructed from our probabilistic approach and gray samples are from a deterministic approach. We also present the reference image.
the spatial resolution and the feature dimension of each plane. We vary these parameters and evaluate the point cloud to mesh reconstruction capability terms of CD. The result presented in Tab. [ From which and are the best choice. We can also notice that, when we simply introduce a tri-plane structure, the reconstruction quality improved by 0.64 in terms of CD (x 10 despite the number of parameters in the x 8 x setting is significantly lower than it in the vector setting presented in Tab. 2 This improvement is brought by the tri-plane structure introduced 3D awareness the latent structure, which benefits the decoding stage. Another founding is that improving the spatial resolution brings improvements the reconstruction quality. However, the computation complexity will increase quadraticly as we increase the spatial resolution, therefore, increase the resolution beyond 32 x 32 becomes intractable in our experiment. Furthermore, we do ablation study on the feature dimension. We notice that increasing the feature dimension is not beneficial anymore, potential reason is that increasing the feature dimension introduces too many parameters which leads to a latent space that is not compact enough to be continues. Therefore, it performs poorly when generalizing to novel objects. Also, decreasing the feature dimension leads to a worse performance as there is not adequate parameters to preserve the geometry information of the encoded object.
observe that setting the spatial resolution and feature dimension (32 32)
D.2 Effect of using probabilistic alignment rather than deterministic
In terms of the modality alignment strategy, navic way to address this using a projection network that directly project an image feature to the point cloud feature modality. However, since this is an ill-pose problem and a single-view image can corresponding to many possible valid 3D shapes due to occlusion. A deterministic approach tends to produce blurry predictions [15]. We demonstrate such an effect by designing a deterministic Image-Point Paiee-PPonntClllci we follow LRM [I5] and use a 12- 12-layer transformer to directly align image feature to point cloud feature modality and train on the 140k Objavese dataset. Fig. 7 shows difference between our probabilistic alignment approach compared with a deterministic method. From which we can observe that the reconstruction with probabilistic process involved produces more clear and sharp details than deterministic approach.
aligmment network. To this end,
17

---

Table 5: Quantitative comparison for the geometry quality between our full model w/ or w/o the plane refiner on point cloud to 3D mesh.
Method
w/o Refiner
w/ Refiner
CD(X10'N
2.37
1.81
Reference image
refiner
refiner
We evaluate the single view reconstruction capability of our model w/ and w/o the plane refiner. Green objects are constructed from our model with the plane refinr and gray samples are from a the model without the plane refiner. We also present the reference image.
Figure 8:
D.3 Effect of plane refiner
To evaluate the effectiveness of our plane refiner module, design comparison about our full model with plane refiner and a variant without plane refiner. The result is shown in Tab. [ and we also present a qualitative result in Fig. 8 We can observe that the plane refiner module improves the capability of our compression model, allowing it to recover more fine-grained details.
E
More Visualization
We present more visualization for 3D reconstruction by aligning image and point cloud modality using our proposed method in Fig. gla Fig T0
18

---

Aggrayation
Reference Image
Our Recon.
Reference Image
Our Recon.
Reference Image
Our Recon.
Figure 9: Rendered images of shapes reconstructed by our LAM3D from single images the GSO dataset. For each tuple of samples, the left image the reference image and the right image the reconstructed geometry.
19

---

MOKUDAA
mo8 B998 Minm 10000
Reference Our Image
Reference Image
Our Recon.
Reference
Our Recon.
Recon.
Image
Figure 10: Rendered images of shapes reconstructed by our LAM3D from single images on the Objaverse dataset. For each tuple of samples, the left image is the reference image and the right image is the reconstructed geometry.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly stated the claims made in the abstract and introduction, accurately reflecting the paper's contributions and scope, including the contributions made in the paper and important assumptions and limitations.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations of our work in the paper, which include the absence of texture recovery and the occasional presence of local geometries in the reconstructed shape that are misaligned with the reference image.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
21

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results, and therefore, this question is not applicable to our work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide a detailed description of the model and experimental settings in our paper, ensuring that readers have the necessary information to reproduce the main exper- imental results. Additionally, we plan to release the code to further enhance reproducibility and facilitate verification of our results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
22

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: While we currently do provide open access the data and code, we plan to release the code along with sufficient instructions to reproduce the main experimental results after the paper has been accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/ //iips.c//pple wttpccpppppppppeeeeeeeeeeeeeeeee
httpa///gggieee//deeebbiiissssss Coodeeooodssssiiieccca sonooooocsss for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: We have provided all the necessary details regarding the training and testing process, including data splits, hyperparameters, the rationale for their selection, and the type optimizer used.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
23

---

Justification: Our paper does not report error bars or other information about the statistical significance of the experiments. However, we follow common practices in our research area and have conprehensively evaluated our method to demonstrate its effectiveness, which ensures that our results are reliable and meaningful within the context of our work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provided sufficient information on the computer resources needed to reproduce the experiments in the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics /tteehhtggs////prppp../////////
Answer: [Yes]
Justification: Our research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
24

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: This paper discussed both potential positive societal impacts and negative socictal impacts of the work performed
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our method presents low risk for misuse dual-use. Additionally, we have taken precautions to ensure responsible data handling by filtering our training data to avoid unsafe content.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
25

---

Answer: [Yes]
Justification: We have ensured that all the assets used in our research, including code, data, and models, are properly credited. The licenses and terms of for these assets have been explicitly mentioned and are adhered to in our work, demonstrating our respect for the rights and contributions of the original creators.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcode com/coo//oaoooooo datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re re-packagdd, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: We did not submit any new assets at the time of submission. However, we plan to release well-documented code after the paper's acceptance.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Our research does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
26

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our research does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "K29NbIKXBW", "reviewer": "Reviewer_tycK", "summary": "The paper proposes a two-stage 3D reconstruction method that first uses a transformer-based 3D point cloud feature extractor to initialize hierarchical latent triplanes (XY, XZ, YZ) and reconstructs the 3D mech. Next, it presents an image-point cloud feature alignment approach that leverages initial latent triplanes to align them with the image-based features using independent plane diffusion models to produce better detail and reduced geometric distortion reconstructions.\n\nThe approach achieves state-of-the-art high-fidelity 3D mesh reconstructions from a single image in just 6 seconds, and experiments on various datasets demonstrate its effectiveness.", "questions": "The paper needs to be improved in the presentation and make the flow chart more clear for example, what is the input?\n\n•\tMissing details of the Point Net and Transformer block utilized in the point feature extraction stage, plane encoder in triplane compression, UNet utilized for the plane refiner process, and diffusion models in image-point cloud alignment. \n•\tThe inference process is unclear, specifically lines 249-251, and might need more discussion.\n•\tThe paper misses mentioning the concept and gaps of the current compared methods called One-2-3-45 [24], SyncDreamer [25], TGS [63], CRM [51], and Magic123 [35] in the related work.", "limitations": "The authors adequately addressed the limitations.\n\nOverall, the idea of aligning the 3D point features induced triplane representation and image-based features utilizing independent plane diffusion models seems interesting and can produce less geometric distortion reconstruction. However, the proposed method is evaluated only on a single dataset and misses the related work, mentioned module details, and model capacity comparison that must be addressed.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tThe concept of utilizing the proposed image-point cloud feature alignment approach using an independent plane diffusion model seems distinctive. \n•\tThe proposed method of 3D reconstructions shows reduced geometric distortion reconstructions compared to the currently mentioned methods.\n•\tAchieves SOTA results on the mentioned 3D object-based dataset.", "weaknesses": "•\tComparison of the model capacity with the current SOTA methods might be added as the proposed methods seems heavy that uses three diffusion, three UNet and two transformer-based architecture.\n•\tAlso, the paper might want to compare the inference time with the existing methods as it specifies that it only requires 6 secs for inference.\n•\tDetails of the module and inference stage missing (see section correctness).\n•\tThe method is evaluated using only a single dataset. \n•\tLack of reproducibility\n\nThe paper is not well-written. The figure is supposed to give an overview of the pipeline, however, if just reading the figure it is not clear what the input is. The paper refers to input point clouds multiple times, which makes it furthermore confusing.  Or maybe the point clouds are also part of the input as it is referred to as prior point clouds in the paper. If so the performance gain might also be due to the additional multi-modality inputs which is not a fair comparison with the other baseline methods.\n\nTypo:\n\n•\tLine 622: We follow LRM [14] ‘and uses’ a 12-layer transformer to directly align image feature to point cloud seems mistaken.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions and suggestions. We have provided our responses below:\n\n***Q1. Model Capacity***\n\nAs shown in Tab. 1 of the global response, we provide parameter size comparisons between our model and SOTA methods. It is worth noting that recent large reconstruction models are designed with high capacity to leverage large-scale training data, enhancing their generalizability in reconstructing novel objects. Our method aligns with this design philosophy.\n\n***Q2. Inference Time***\n\nAs shown in Tab. 1 of the global response, our method achieves better inference time. Specifically, our time breakdown is as follows: approximately 0.02 seconds for encoding an image with DINO, 4.52 seconds to align the image feature to latent tri-planes using 50 diffusion steps, 0.24 seconds to upsample the latent tri-plane to the raw tri-plane space with the plane decoder and refiner, and 1.05 seconds for mesh extraction using Marching Cubes.\n\n***Q3. Missing Details***\n\nWe have described the design of each module in the method section. We mainly focus on the conceptual aspects in our paper, and we acknowledge that the implementation details are somewhat limited due to page limitations. The inference pipeline is outlined in the caption of Fig. 2 and in Section 4.1, lines 249 to 251. We will provide additional explanations to clarify any potential misunderstandings.\n\n***Q4. Single Dataset Evaluation***\n\nOur evaluation protocol was precisely adopted from CRM (ECCV 2024) [51]. A common practice in large reconstruction models is to train on the Objaverse dataset and evaluate on the Google Scanned Objects (GSO) dataset. This approach was also used in TGS (CVPR 2024) [63] and One-2-3-45 (NeurIPS 2023) [24]. In response to the reviewer's feedback, we have included additional evaluation results in Fig. 3 of the global response. These results encompass reconstruction outcomes for images from ImageNet, MVImageNet, AI generated images, and images collected from the Internet.\n\n***Q5. Clarity***\n\nWe appreciate the reviewer highlighting areas for improvement. We acknowledge the concerns regarding the clarity of the figure and the input description in our manuscript. To clarify, our pipeline consists of two stages during training: point clouds are used as input for the first stage, while images are used for the second stage. During inference, only images are required as input. Thus, the performance gains achieved are attributed to the effectiveness of our proposed modules and strategies, rather than the use of additional input modalities. We will revise the figure and manuscript to clearly distinguish between the training and inference phases, ensuring that the role of each input is clearly communicated.\n\n***Q6. Presentation***\n\nWe appreciate the reviewer’s feedback on the presentation and clarity of the flow chart. We recognize that the current chart may not clearly indicate the inputs. We will revise the flow chart to explicitly show that point clouds are used in the first training stage, images in the second training stage, and only images are needed during inference. We will ensure that the inputs are clearly highlighted at each stage of the pipeline to improve overall clarity.\n\n***Q7. Lack of Details and Reproducibility***\n\nIn our manuscript, we focused on the conceptual aspects and design choices of each module to provide a clear understanding of the overall framework and its motivations. We acknowledge that this approach did not cover the implementation details of each component in depth. To address this, we will include additional implementation details in the appendix. Additionally, we are actively seeking institutional approval to release code and pre-trained models to the public for easy reproduction of our results by the community.\n\n***Q8. Inference Process***\n\nWe apologize for any confusion caused by the description of the inference process in lines 249-251. To clarify, our model requires only a single-view image of an object during inference. The process starts by encoding this image into a feature representation using the DINO image encoder. This image feature is then aligned to a latent tri-plane representation through three independent diffusion UNets, where the latent tri-plane represents a compressed tri-plane shape in the latent space. The latent tri-plane is subsequently upsampled to a high-resolution tri-plane using the plane decoder and refiner from the first training stage. We then employ the SDF MLP to decode the signed distance for any query position on the tri-plane, as described in lines 173-175, which allows us to use the Marching Cubes algorithm to reconstruct a mesh from the tri-plane. We will revise the manuscript to include a more detailed explanation of this inference process to enhance clarity.\n\n***Q9. Concept and Gaps***\n\nWe appreciate the feedback. We will expand the related work to include a comprehensive review of [24,25,63,51,35]. Specifically, we will highlight that these previous methods primarily employ a rendering-based loss for supervision, which overlooks the use of available explicit 3D priors. This absence of explicit 3D priors often results in 3D geometric distortions, examples of which are demonstrated in Fig. 1(b) for LRM [12] and Fig. 1(c) for CRM [51]. In contrast, our method leverages 3D point clouds as explicit guidance during the training stage. We explore the approach of achieving accurate single-image 3D reconstruction by aligning image features with explicit 3D features, marking a novel advancement in the domain.\n\n***Q10. Typo***\n\nWe apologize for the oversight. This typo will be corrected in the revised manuscript.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MvZULKcltb", "reviewer": "Reviewer_XUip", "summary": "This paper introduces the Large Image and Point Cloud Alignment Model (LAM3D), a novel framework that enhances 3D mesh reconstruction from single images by utilizing point cloud data. LAM3D integrates a point-cloud-based network for generating precise latent tri-planes, followed by an Image-Point-Cloud Feature Alignment technique that enriches image features with robust 3D information. This approach allows for the production of high-fidelity 3D meshes while mitigating geometric distortions commonly associated with single-image inputs. The method demonstrates its effectiveness across various datasets, achieving state-of-the-art performance in high-fidelity 3D mesh reconstruction.", "questions": "1. Can you provide more details on how LAM3D handles various object categories, especially those not directly represented in the training datasets?\n2. Could the authors elaborate on how LAM3D might be adapted or extended to include texture reconstruction?", "limitations": "While LAM3D effectively enhances mesh reconstruction quality by leveraging 3D point cloud data, its current implementation omits texture mapping. This omission not only limits the visual realism and practical utility of the generated 3D meshes but also makes direct comparisons with methods like One-2-3-45 and CRM, which include textural details, somewhat unfair.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper presents a method combining point cloud data with image features to reconstruct 3D meshes, which is a promissing solution to the problem of geometric inaccuracies in single-image 3D reconstructions.\n2. The experimental results are robust, showing clear improvements over existing methods.", "weaknesses": "1. The LAM3D framework's omission of texture mapping is a significant limitation, particularly when compared against methods like One-2-3-45 and CRM that integrate both geometric and texture reconstructions. This absence not only restricts the visual and practical applicability of the generated meshes but also raises concerns about the fairness of comparisons made in the paper. The inclusion of texture details is pivotal for realistic 3D reconstructions, and its absence in LAM3D suggests a crucial area for potential improvement. Furthermore, the comparisons with models that handle both geometry and texture may not provide a balanced view. It's more appropriate to compare LAM3D against models such as \"Slice3D: Multi-Slice, Occlusion-Revealing, Single View 3D Reconstruction\" (CVPR, 2024) and \"D^2IM-Net: Learning Detail Disentangled Implicit Fields from Single Images\" (CVPR, 2021), which are more aligned in terms of focusing primarily on geometric details.\n2. LAM3D requires point cloud data during training, its applicability is notably restricted to specific categories or types of objects. This limitation indicates a constrained generalization ability, which may impede its deployment in diverse real-world scenarios that demand robust 3D reconstruction capabilities across varied object environments.\n3.  The use of independent diffusion processes for each tri-plane is a novel approach, but the paper does not provide a comparative analysis of this method against more integrated approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions and suggestions. We have provided our responses below:\n\n***Q1. Omission of Texture and Texture Extension***\n\nWe appreciate the reviewer's observation regarding the omission of texture mapping. Our research primarily focused on addressing the limitations of previous methods that rely solely on volumetric rendering for training large-scale 3D reconstruction models, often neglecting the integration of explicit 3D prior knowledge. By aligning image domain features with the 3D domain, our approach significantly improves geometry reconstruction capabilities.\n\nWhile texture mapping is left for future work, our method can be easily extended to reconstruct both texture and geometry. This would involve modifying the first stage of training by adjusting the autoencoder to accept a colored point cloud, then introducing an additional MLP to decode the interpolated tri-plane features and regress an RGB vector. In this extended approach, the model would decode both the signed distance and the color at corresponding positions. The inference pipeline remains unchanged, as we can use Marching Cubes to extract both geometry and texture. Another potential approach, as suggested by Reviewer *vD6Z*, involves unprojecting multi-view images generated from models like zero-1-to-3.\n\n***Q2. Evaluation Fairness***\n\nWe appreciate the reviewer’s concern regarding fairness. We believe our comparison is fair because our primary aim is to address the geometry distortion problem caused by the lack of explicit 3D prior knowledge in large reconstruction models. Our method tackles a fundamental aspect of 3D modeling that is crucial for both textured and non-textured models. Additionally, we have included a comparison with the Slice3D model in our global response (Fig. 2 and Fig. 3) to ensure our evaluation covers methods with a similar geometric focus. While D$^2$IM-Net is a pioneering work in disentangling detail in implicit fields, it was trained on ShapeNet, a smaller and less diverse dataset than the large-scale Objaverse dataset we used, making a direct performance comparison challenging. We will discuss both Slice3D and D$^2$IM-Net in the related work section.\n\n***Q3. Constrained Generalization Ability***\n\nWe appreciate the reviewer's observation regarding the requirement for point cloud data during training. Unlike previous rendering-based reconstruction models like LRM and CRM that utilize both 3D assets and multi-view images for training, our approach is constrained to train solely on 3D assets. Despite this, our method demonstrates strong generalization capabilities. While LRM uses 730k 3D assets and 220k videos and CRM employs 376k 3D objects for training, our model is trained on a smaller, category-agnostic dataset of 140k 3D assets. Nonetheless, we achieve state-of-the-art performance on the GSO dataset, which contains over 1,000 3D-scanned real-world items not included in our training data. This highlights the effectiveness of our method in generalizing across varied objects. We argue that although rendering-based methods benefit from larger datasets, their effectiveness is often limited by the lack of explicit 3D knowledge, which can reduce data efficiency. In contrast, our approach, with direct access to 3D representations, enables more efficient learning and improves our ability to generalize across diverse object categories.\n\n***Q4. Diffusion***\n\nOur motivation for using independent diffusion UNets is outlined in lines 51-53 of the paper, and we have experimentally demonstrated that the design leads to improved alignment quality (Tab. 3 and Fig. 5 of our paper). The UNet-based diffusion process is inherently designed to process 2D planar data, while tri-plane are composed of three 2D orthographic projections of a 3D object. Previous methods, such as 3DGen [11] and NFD [43], either roll out the tri-plane as a continuous plane or concatenate channels based on spatial positions, both of which have significant drawbacks. The roll-out method involves convolution operations that overlap between planes, yet the values at the edges of adjacent planes are not spatially continuous, leading to interference. Conversely, while channel concatenation aligns features at the same spatial location, there is no explicit relationship between them, and applying the same convolution kernel to process them results in interference with the regression of each plane. These issues and their impact on convolution operations are illustrated in Fig. 1 of the global response. \n\nTo address these problems, we leverage the tri-plane structure’s property of being three orthographic representations of an object and employed independent diffusion processes for each plane to effectively align image features with planar features. Our ablation study, detailed in lines 305 to 314 of our paper, further supports the effectiveness of this approach. We will also include this additional analysis of the independent diffusion method in the revised paper.\n\n***Q5. Various Categories***\n\nUnlike methods trained on ShapeNet in a category-specific manner with a limited number of object categories, our approach is category-agnostic and does not rely on predefined categories during training. This strategy enables our model to handle a broader range of objects by focusing on general geometric features rather than specific types. Similar to other large reconstruction models, our training data includes a wide variety of object shapes and structures. To validate our model's generalization capability, we conducted evaluations on the GSO dataset, which is composed of objects not seen during training, as shown in Fig. 3 of our paper. Our model not only successfully handled these unseen objects but also outperformed existing methods, demonstrating robustness across different categories. We also provide supplementary evaluations on data from various sources in Fig. 3 of the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4SXOi3mteP", "reviewer": "Reviewer_1gWL", "summary": "This paper introduces a new 3D generation framework, LAM3D, which uses point cloud data and image-point-cloud feature alignment method to improve the geometry of the generation results. Authors use triplanes as representation, combining image feature as well as point cloud feature. For point cloud compression, they use point cloud transformer to encode the point cloud to triplane. For better generation, they encode the point cloud triplane to latent triplane, and trained three diffusion model to generate latent triplane. The whole method can generate untextured mesh within 6 seconds, showing better geometry results comparing baselines.", "questions": "1. For the parallel diffusion part, why not try using one diffusion to output a tensor of size 2x32x96, then resize it to 3x2x32x32 (similar to the design in CRM)? What are the differences, and what are the advantages of parallel diffusion?\n2. You could show more results on real-world data, such as MvImgNet[1] or photos taken by phones.\n3. Some works based on DiT[2], like Sora and the concurrent work Direct3D[3], have shown great success. Have you tried using other diffusion models?\n4. If LAM3D is open-sourced, it could greatly benefit the community. What are your plans for open-sourcing it?\n\n[1] Yu, Xianggang, et al. \"Mvimgnet: A large-scale dataset of multi-view images.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.\n\n[2] Peebles, William, and Saining Xie. \"Scalable diffusion models with transformers.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Wu, Shuang, et al. \"Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer.\" arXiv preprint arXiv:2405.14832 (2024).", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written with good soundness. Logic is clear and motivation is reasonable. \n2. Qualitative and quantitative experiment are carried out in detail and are executed with precision. Ablation study was done with a high degree of accuracy, confirming the significance of parallel diffusion and latent triplane.\n3. To handle the difficulty of image triplane and point cloud triplane alignment, LAM3D use a plain encoder to encode the initial triplane to latent triplane, this may give the community another insight in 3D representation and 2D image alignment.", "weaknesses": "1. There are already some work on Shape-Image-Aligned generation work such as 3DGen[1] and Michelangelo[2], authors should compare with these work and point out what are the advantages of latent triplane and three parallel diffusion model.\n2. Parallel diffusion design may lose some 3D information. \n\n[1] Gupta, Anchit, et al. \"3dgen: Triplane latent diffusion for textured mesh generation.\" arXiv preprint arXiv:2303.05371 (2023).\n\n[2] Zhao, Zibo, et al. \"Michelangelo: Conditional 3d shape generation based on shape-image-text aligned latent representation.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions and suggestions. We have provided our responses below:\n\n***Q1. Comparison***\n\nWe thank the reviewer for highlighting these related works. Shape-Image-Alignment is not a novel concept, and has been explored in works like Michelangelo and ULIP [58]. The essence of both Michelangelo and ULIP is training a 3D autoencoder with a CLIP-aligned latent space. However, the experiments of Michelangelo were limited to the ShapeNet scale. To examine the effectiveness of this approach on a larger scale, we built a baseline model using the 3D encoder of ULIP, as described in lines 293 to 304, and found notable limitations when scaling to the Objaverse dataset with 140k category-agnostic objects. Firstly, the contrastive loss used in Michelangelo and ULIP enhances linear separability in the latent space, which is beneficial for discriminative tasks. However, this approach is problematic for 3D reconstruction, as it requires a continuous latent space to capture the morphable nature of 3D objects. While Michelangelo succeeded within the relatively small and dense ShapeNet dataset, this alignment approach failed to generalize at the Objaverse scale, leading to poor 3D reconstruction quality as demonstrated in Fig. 1 and Tab. 2. Secondly, the constraint of encoding 3D shapes into a vector feature representation of size $\\mathbb{R}^{512}$ to align with pre-trained CLIP features proved inadequate for capturing the rich diversity of 3D geometries, as seen in our comparisons in Fig. 4. In contrast, our diffusion-based alignment strategy fosters a continuous latent space, while our tri-plane latent representation effectively preserves spatial information that is often lost in vector-based latent representations, thereby enabling more accurate and scalable 3D reconstructions. Regarding the 3DGen method, it treats the image as a condition for the diffusion model and uses a CLIP image encoder to produce a vector-shaped feature for the diffusion process. However, as we discussed earlier, this approach can lead to a loss of spatial information. In contrast, our method utilizes DINO as the image encoder, generating per-patch encodings of size $\\mathbb{R}^{1025 \\times 768}$, which allows the subsequent diffusion alignment module to transform these into a tri-plane representation, preserving crucial spatial information. Additionally, our diffusion model differs fundamentally from 3DGen’s approach, which we discuss further in the response to Q2.\n\n***Q2. Parallel Diffusion***\n\nWe appreciate the reviewer's concern about the potential loss of 3D information. Previous methods often handle tri-plane structures through either roll-out operations, as seen in 3DGen, or channel concatenation, as discussed in response to Q3. However, both approaches have significant drawbacks. The roll-out method involves convolution operations that span adjacent planes, but the values at the edges of these planes are not spatially continuous, leading to interference and inaccuracies during the learning process. Similarly, channel concatenation aligns features at the same spatial location across planes, but without an explicit relationship between them, applying a single convolution kernel can cause interference in the regression of each plane. To address these issues, our method leverages the planar properties of the tri-plane structure by processing each plane independently. This approach effectively resolves inter-plane interference and results in better reconstruction outcomes. Furthermore, the three planes can be viewed as orthographic projections of the 3D object. We explicitly decouple features across planes by introducing a latent tri-plane loss, which preserves the tri-plane structure in the latent space, as shown in Fig. 6 of the appendix. Although our three independent diffusion models may lack direct inter-plane interaction, this design diminishes inter-plane interference and ensures that 3D information is not lost, as the plane features are decoupled and maintained through the latent tri-plane loss during the compression stage.\n\n***Q3. Channel Concatenation and Parallel Diffusion***\n\nAs explained in our response to Q2, using channel concatenation results in the regression of values based on all three planes at the same spatial location, despite there being no explicit relationship between them. This lack of spatial relationship leads to interference during the learning process. To address this, we constructed a baseline model with a single diffusion network that concatenates the planes channel-wise. Our experimental results, presented in Fig. 5 and Tab. 3, demonstrate that the parallel diffusion approach outperforms this single UNet variant, offering better reconstruction quality and visual effects. The parallel diffusion design mitigates the inter-plane interference, leading to more accurate and effective 3D reconstructions.\n\n***Q4. More Evaluation***\n\nWe appreciate the reviewer's suggestion to include results on real-world data. We evaluated our method on the Google Scanned Objects dataset, which includes a diverse range of real-world household objects, and our approach outperforms previous methods as shown in Tab. 1. In response to the reviewer's comment, we have now included additional comparisons on more diverse sources of images in Fig. 3 of the global response.\n\n***Q5. DiT***\n\nDue to limited computational resources during the development stage, we focused on validating our approach with a parallel UNet design, which delivered state-of-the-art results. However, DiT is indeed a promising direction, and we plan to explore it in future work.\n\n***Q6. Open Source***\n\nWe appreciate the reviewer's interest in open-sourcing of LAM3D. We are actively seeking institutional approval to release both the training and inference codebase to the public. Additionally, we plan to provide an online demo to facilitate quick and easy access to our model for the community.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3LFfV2nt5z", "reviewer": "Reviewer_vD6Z", "summary": "The paper addresses the problems of multi-view consistency and geometric detail in image-to-3D generation. The proposed method, LAM3D, adopts a two-stage approach for training. In the first stage, the authors train a plane encoder and decoder to compress point clouds into a latent tri-plane representation. In the second stage, to align image and point cloud features within the same feature space, they train the diffusion model. During inference, the image is converted into a latent tri-plane by passing through the trained U-Nets from the second stage, and the features are decoded by the plane decoder trained in the first stage. A final mesh is extracted using the marching cubes algorithm from the reconstructed tri-plane. Experiments demonstrate the effectiveness of the proposed method in 3D generation from a single image, both quantitatively and qualitatively. Additionally, the authors provide an extensive ablation study on different design components.", "questions": "(1) A minor question: does collecting points using KNN and then using the embedding created by PointNet improve performance compared to directly inputting the points extracted through FPS into the transformer?", "limitations": "The authors mention that the limitation of the current pipeline cannot generate texture, as discussed in Section 5.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "(1) The task of single-view image to 3D generation is extremely challenging and well-motivated.\n\n(2) The proposed method addresses the limitations of existing baselines, which only use images for training and fail to effectively reflect multi-view consistency and geometric detail. To solve the problems, the authors use point clouds as a 3D prior and converting them into tri-planes to facilitate training. Additionally, the method is carefully designed to align the images and tri-planes within the same feature space.\n\n(3) The results show that the proposed method achieves SOTA performance on single image-to-3D generation with a short inference time compared to existing baselines.\n\n(4) A numerous ablation studies on design choice is conducted to achieve optimal performance.", "weaknesses": "(1) As mentioned in the limitation, LAM3D cannot generate a textured mesh, while baselines (One-2-3-45, LRM, and CRM) can. In generating 3D content, the accurate geometry of the mesh is important, but the corresponding texture is also a crucial component. It is unfortunate that this aspect cannot be generated. Although it would take more time, generating multi-view images based on the input image using a multi-view diffusion model like Zero-1-to-3 and then unprojecting them to the generated mesh could be a solution.\n\n(2) Recently, many papers related to single image-to-3D generation have been published, and there are many papers like [1, 2] that were published 2 months before the NeurIPS 2024 submission deadline. Among these, I guess Michelangelo [1] has the best quality in geometry generation, so comparing this paper would make the argument more compelling.\n\n[1] Zhao et al., Michelangelo: Conditional 3D Shape Generation based on Shape-Image-Text Aligned Latent Representation, NeurIPS 2023\n\n[2] Vikram Voleti et al., SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion, arXiv 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful questions and suggestions. We have provided our responses below:\n\n***Q1. Texture***: \n\nWe acknowledge the reviewer's concern about LAM3D's current limitation in generating textured meshes, as mentioned in the paper. While our primary focus was on achieving accurate geometry reconstruction, we fully agree that texture is a crucial component in generating realistic 3D content. Our approach was motivated by the need to resolve geometric distortion issues caused by the lack of explicit 3D priors in recent large reconstruction models. We believe that recovering high-quality geometries is foundational for both textured and non-textured 3D reconstruction. We appreciate the suggestion to use a multi-view diffusion model like Zero-1-to-3 to generate textures, and we recognize this as a promising direction to effectively complement LAM3D’s geometry output. Furthermore, we are committed to exploring such solutions in future work to enhance the capabilities of LAM3D.\n\n\n***Q2. More Comparison***\n\nWe appreciate the reviewer highlighting these recent papers, particularly Michelangelo, which also adopts an alignment-based approach for 3D reconstruction. As discussed in response to Q1 of Reviewer *1gWL* and in our main paper, Michelangelo's alignment strategy has two key limitations: (1) the contrastive loss used in Michelangelo enhances linear separability in the latent space, which is beneficial for discriminative tasks but problematic for 3D reconstruction that requires a continuous latent space to capture the morphable nature of 3D objects; (2) the choice of a vector-shaped latent representation falls short in preserving the spatial information necessary for accurate 3D reconstructions. We address these issues with our diffusion-based alignment strategy, which promotes continuity in the latent space, and a tri-plane structured latent representation that retains spatial information. To validate our approach, we constructed a baseline model based on ULIP [58], a framework similar to Michelangelo that unifies 3D, image, and text modalities. Our experiments demonstrated that while Michelangelo's strategy performs well at the ShapeNet scale, it fails to generalize effectively to the large-scale and category-agnostic Objaverse dataset. These results are detailed in Figures 1, 4, and Tab. 2 of our main paper. Additionally, we recognize the promising direction presented by SV3D, which employs large-scale video diffusion models for 3D reconstruction, and we will discuss this in the revised related works section.\n\n***Q3. KNN+PointNet v.s. FPS+Transformer***\n\nThe method mentioned by the reviewer, using KNN for point collection followed by a PointNet encoder, was initially our preferred approach due to its simplicity and efficiency during the algorithm development stage. However, our experiments revealed that the KNN+PointNet combination struggled to fully capture the input geometry, often resulting in oversmoothed reconstructions and low-quality details. This limitation likely arises because the projection operation aggregates information from a broader range of spatial locations, whereas KNN+PointNet is constrained to local operations. This finding motivated our shift to using a Transformer-based approach, which better integrates long-range interactions between local patches, and our experiments demonstrated the effectiveness of this method in improving reconstruction quality.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their thoughtful comments and the time invested in reviewing our work. We take all suggestions seriously and are committed to carefully revising the paper based on your feedback.

In the attached PDF, we have included one table and three figures for your reference:

+ **Table 1**: Comparison of model capacity and inference time.

+ **Figure 1**: Illustration of unintended convolution behavior in different methods for processing tri-plane structured data.

+ **Figure 2**: Qualitative comparisons of our method and Slice3D on the Objaverse and GSO datasets.

+ **Figure 3**: Shapes reconstructed by our LAM3D from single images, compared with state-of-the-art methods.

We respond to reviewers' specific concerns individually below.

Meta-review: This paper was reviewed by four experts in the field.  Based on the reviewers' feedback, the decision is to recommend the paper for acceptance.  The reviewers did raise some valuable concerns that should be addressed in the final camera-ready version of the paper. The authors are encouraged to make the necessary changes to the best of their ability.   In particular, it is strongly recommended to release the code as promised in "5. Open access to data and code" field.
Decision: Accept (poster)