Title: Direct3D: Scalable Image-to-3D Generation via 3D Latent Diffusion Transformer
Venue: NeurIPS 2024
Accepted: yes
Keywords: 3D Generation

Diffsion Model
Abstract: Generating high-quality 3D assets from text and images has long been challenging, primarily due to the absence of scalable 3D representations capable of capturing intricate geometry distributions. In this work, we introduce Direct3D, a native 3D generative model scalable to in-the-wild input images, without requiring a multi-view diffusion model or SDS optimization. Our approach comprises two primary components: a Direct 3D Variational Auto-Encoder (D3D-VAE) and a Direct 3D Diffusion Transformer (D3D-DiT). D3D-VAE efficiently encodes high-resolution 3D shapes into a compact and continuous latent triplane space. Notably, our method directly supervises the decoded geometry using a semi-continuous surface sampling strategy, diverging from previous methods relying on rendered images as supervision signals. D3D-DiT models the distribution of encoded 3D latents and is specifically designed to fuse positional information from the three feature maps of the triplane latent, enabling a native 3D generative model scalable to large-scale 3D datasets. Additionally, we introduce an innovative image-to-3D generation pipeline incorporating semantic and pixel-level image conditions, allowing the model to produce 3D shapes consistent with the provided conditional image input. Extensive experiments demonstrate the superiority of our large-scale pre-trained Direct3D over previous image-to-3D approaches, achieving significantly better generation quality and generalization ability, thus establishing a new state-of-the-art for 3D content creation. Project page: https://www.neural4d.com/research/direct3d.


Peer reviews:
comments: SUMMARY:
An image-conditioned 3D generation method with two parts: 1. 3D triplane VAE taking point cloud as input and generate the semi-continuous occulancy, which includes a point2latent encoder and a latent2triplane decoder 2. A  direct 3D transformer diffusion model conditioned on single image and generate tokens of the latent, with two input heads: DINO and CLIP. A wide range of evaluation with input from GSO object rendering to 2D diffusion generated images.

STRENGTHS:
1. Well-written maniscript explains motivations and methods clearly.
2. A novel 3D diffusion transformer to generate triplane token, then decode the semi-continuous occupancy.
3. Good qualitative results given image rendered from GSO dataset or text conditioned image diffusion model.

WEAKNESSES:
1. Lack of analysis of your data's contribution on method performance. I'm curious whether your good quality is because of 500K internal data since most baselines are only trained on objaverse. It would be great if I can see the result of baselines trained on your data (If this is too expensive, maybe training your method only trained on objaverse small subset is easier).

2. Lack of more quantitative results: The only quantitative result is the user study, which may be not enough. It's natural to comare with other baselines on chamfer distance, PSNR or SSIM of depth and normal maps from some views, and also compare on a larger set (not as few what you mentioned as 30 GSO objects)

QUESTIONS:
See the weakness.

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an image-to-3D generation model without using SDS optimization. It includes a 3D-VAE that encodes 3D shapes into latent space and a 3D DiT that models the latent distributions.\n\nSTRENGTHS:\n- Single-step 3D generation is a relatively unexplored area.\n- Exploring the DiT model in 3D tasks is interesting.\n\nWEAKNESSES:\n- Lack of novelty. Specifically, 1) this paper mainly uses a 3D-VAE and a 3D-DiT to encode a 3D point cloud into latent space and model the distribution, which has been explored in previous methods like Point-E/Shap-E. This pipeline is widely used in image diffusions, and this paper extends it to 3D, which does not seem novel to me. 2) One novelty the authors mention is the semi-continuous occupancy used for supervision, which, compared with standard 3D supervision, only adds a continuous value between [0,1] when it is close to the boundary. 3) The semantic-level 3D-specific image conditioning is provided by DINO.\n\n- What is the dataset used for training? It seems that a private dataset is used for training, so I am not sure if the examples shown in the paper were seen during training. In addition, since the paper emphasizes that they can use in-the-wild input images for 3D generation, results on in-the-wild input images would be good to show.\n\n- There is a paper in CVPR 2024 named \"DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data\" [1]. They are also doing single-step 3D generation without SDS. This submission uses the same model name as the CVPR 2024 paper. This is very confusing. I think the authors should consider changing the method name. For me, Direct3D or D3D are both not good names. In addition, considering this submission and the CVPR paper are doing the same thing, the authors may also consider discussing the differences.\n\n[1] Liu, Qihao, et al. \"DIRECT-3D: Learning Direct Text-to-3D Generation on Massive Noisy 3D Data.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\nPlease see the questions in the weaknesses section. I will be willing to raise my rating if these questions are resolved.\n\nLIMITATIONS:\nYes, it is discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary: This manuscript focuses on the task of 3D generation, including image-to-3D and text-to-3D tasks. The framework is structured in two stages. In the first stage, a geometry VAE is trained using 3D ground-truth occupancy as supervision. Triplane representations are leveraged as the explicit 3D format, which are acquired through a Transformer processing dense point clouds. A decoder is then used to map the interpolated features into occupancy fields. In the second stage, conditioned images are encoded using CLIP and DINO to provide pixel-level and semantic-level conditions, respectively. DIT blocks are utilized to denoise the latent space of the geometry VAE. Extensive experiments validate the effectiveness of the proposed framework, demonstrating promising capabilities in conditional 3D generation.\n\nSTRENGTHS:\n1.The proposed Semi-continuous Surface Sampling technique is technically sound.\n2.The DIT model is innovatively applied for the first time in conditional 3D generation.\n3.The manuscript is well-organized and clearly written, facilitating easy comprehension of the complex concepts discussed.\n4.The overall framework exhibits promising capabilities in conditional 3D generation, suggesting strong potential for future applications and research.\n\nWEAKNESSES:\n1.The manuscript lacks a robust quantitative evaluation metric for the geometry VAE and conditional generation, particularly in comparison with state-of-the-art models and within its own ablation study. For instance, while the effectiveness of the explicit triplane latent representation and the semi-continuous surface sampling strategy is demonstrated, it is shown only through the visualization of three or four examples without any quantitative metrics. However, these results could be quantitatively evaluated using the Chamfer Distance by comparing the reconstructed mesh with the ground truth (GT) mesh, both of which are readily accessible. Additionally, as other methods also utilize 3D GT for supervision, a quantitative comparison using the Chamfer Distance should be provided to establish the proposed method's competitiveness and efficacy more definitively. \n2. The manuscript currently lacks sufficient detail in the \"Implementation Details\" section, particularly regarding the near-surface sampling strategy. For the sake of reproducibility, it is crucial to provide more information:\n\n+Sampling Strategy: What specific strategy is employed for near-surface sampling?\n+Is the loss weight of BCE loss the same across near-surface points and uniformly distributed points?\n+How many points are sampled per iteration?\n\nQUESTIONS:\nI recommend a detailed discussion of how pixel-level and semantic-level conditions are implemented, particularly since both conditions are introduced at the token-level. The distinction between the two is crucial for understanding the underlying mechanisms of the model. Specifically, the manuscript mentions that the CLIP token is integrated using AdaLayerNorm, while the DINO token is incorporated through self and cross-attention. It would be beneficial for the manuscript to elaborate on why these specific implementations represent pixel-level and semantic-level conditions, respectively. Such a clarification will not only enhance the clarity of the manuscript but also substantiate the design choices made in the architecture, providing readers with a deeper understanding of how each contributes to the model's performance.\n\n2.I would like to see a detailed discussion regarding the differences between the vector set representation used in 3D Shape2VecSet and the triplane representation employed as the latent representation within the framework. This discussion should include an analysis of how each representation impacts the training of the geometry VAE and the conditional generation process. Such a comparative analysis would not only clarify the choice of representation in the current framework but also provide valuable insights for future research in 3D object generation. This discussion could significantly strengthen the manuscript by providing a clearer justification for the architectural decisions and their implications on the model's performance.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents Direct3D, a new approach for scalable image-to-3D generation using a 3D Latent Diffusion Transformer. This method enables the generation of high-quality 3D assets from text and images without the need for complex optimization techniques. Direct3D introduces a native 3D generative model that efficiently produces 3D shapes consistent with conditional image input. By incorporating semantic-level and pixel-level image conditions, Direct3D demonstrates the efficacy of its approach through detailed experimental results and discussions on limitations and theoretical considerations.\n\nSTRENGTHS:\n- Direct3D can handle in-the-wild input images without the need for multi-view diffusion models or complex optimization techniques.\n\nDirect3D surpasses previous image-to-3D approaches in terms of generation quality and generalization ability. \n\n- Direct3D comprises two primary components - a Direct 3D Variational Auto-Encoder (D3D-VAE) and a Direct 3D Diffusion Transformer (D3D-DiT). These components work together to encode high-resolution 3D shapes into a compact latent space and model the distribution of encoded 3D latents, respectively.\n\nWEAKNESSES:\n- The model is trained on 500k internal data, which is not fair to compare with other methods. It's good to see quality improvement and very interesting pipeline. But as an academic publication, it's not acceptable to me, as a reviewer, that we dont know it enough whether the improvement comes from algorithm or newly added dataset. It's just not scientific.\n\n- There is very limited qualitative evaluation conducted to demonstrate the effectiveness, E.g., CLIP-score, LPIPS, FID.\n\n- Eq (2) lacks input of $L_{BCE} $  and $L_{KL}$, which cause trouble in understanding the pipeline.\n\n- typo in figure 2, \"Nosiy Latent\"\n\n- Better to denote the shape of noisy latent and the output of DiT in the figure 2 to help reader better understand it.\n\nQUESTIONS:\nWhat is the reconstruction error of the proposed 3D-VAE?\n\nLIMITATIONS:\nMy biggest concern comes from the usage of private data. While the algorithm looks novel and interesting (and I personally like it very much), I can not recommend acceptance to a paper without a clear justification of where the improvement comes from. I'd love to change my mind if the author is willing to 1) train Direct3D without internal data and put it on the main manuscript while leave the internal data version to supplementary; or 2) open source the dataset (including category of dataset, its sampled visual examples, and the way of collection). I guess 1) is easier to achive.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGenerating high-quality 3D assets from text and images has been difficult due to the lack of scalable 3D representations. This paper, Direct3D, addresses this by introducing a native 3D generative model that scales to real-world input images without needing multiview diffusion models or SDS optimization. The method includes two main components:\n\nDirect 3D Variational Auto-Encoder (D3D-VAE): Encodes high-resolution 3D shapes into a compact latent triplane space, supervising the decoded geometry using a semi-continuous surface sampling strategy instead of relying on rendered images.\n\nDirect 3D Diffusion Transformer (D3D-DiT): Models the distribution of encoded 3D latents and fuses positional information from the triplane latent's three feature maps, creating a scalable native 3D generative model for large datasets.\n\nThe approach also introduces an innovative image-to-3D generation pipeline that incorporates semantic and pixel-level conditions from images, enabling the generation of 3D shapes consistent with the provided images. Experiments show that Direct3D outperforms previous methods in generation quality and generalization, setting a new benchmark for 3D content creation.\n\nSTRENGTHS:\n1. The origin, generative-based 3D AIGC pipeline is absolutely missing. Though LRM-based methods have achieved great success in scaling up 3D models, it is still reconstruction-based and heavily relying on the first-stage MV-Diffusion for high-quality 3D objects synthesis. I really like the pipeline proposed here, and believe it is a firm step towards a converged 3D AIGC framework.\n\n2. The performance is good, the generated mesh is clean, and the user study shows the proposed method is better against Shape-E, InstantMesh etc.\n\n3. This pipeline shows that scaling up 3D DiT is also a feasible way towards high-quality AIGC when data is enough.\n\nWEAKNESSES:\nThough performance is good, I still have some concerns regarding the proposed method:\n\n1. This idea is not new. Shape-E /Meta 3DGen first verified the VAE + LDM pipeline, and recently, LN3Diff (ECCV 24') also verifies this pipeline on Objaverse with tri-plane latent space. More difference / discussions with the previous methods shall be discussed, e.g., compared with LN3Diff, any more differences beyond \"combines the advantages of explicit 3D latent representation and direct 3D supervision to achieve high-quality VAE reconstruction\". Ln3diff also has an explicit latent tri-plane latent space and direct depth supervision.\n2. The proposed VAE requires high-quality 3D point clouds and mesh as the ground truth, and can only model geometry information. Compared against Ln3Diff and CLAY which can also model texture, this is definitely a drawback. Another problem of the proposed design is that Direct3D cannot benchmark its performance on the GSO dataset with PSNR/LPIPS, which is a canonical evaluation pipeline for existing 3D AIGC method.\n3. No thorough quantitative comparison against the existing methods, just user study. MMD/COV are also reasonable metrics if no texture is available here.\n4. The proposed DiT pipeline cannot be seen as a contribution, since it shares very similar design with DiT-3D / PixArt.\n\nQUESTIONS:\n1. What data is Direct3D being trained on? I do wonder whether the comparison is fair, since the paper did not elaborate on the details of data filtering pipeline / dataset size. If Direct3D wants to show its advantages against existing methods, it should be trained on the same dataset (e.g., Objaverse) and do thorough comparison.\n\n2. Whether you can include comparison with other 3D origin generative-based methods, e.g., LN3Diff and Crafsman? Since this line of work is very sparse, I think a more thorough comparison is well required.\n\n2. Whether the trained models will be released?\n\nLIMITATIONS:\nOverall, though the paper shows great scalability and astounding performance over 3D AIGC using a DiT pipeline, I still find many concerns regarding this paper. I hope the reviewer can address my concerns during the rebuttal stage, and I am very happy to increase my rating.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-29"}