Hallo3D: Multi-Modal Hallucination Detection and Mitigation for Consistent 3D Content Generation

Authors: hongbo  wang
jie  cao
jin  liu
xiaoqiang  zhou
huaibo  huang
ran  he
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 16:13:51

Abstract: Recent advancements in 3D content generation have been significant, primarily due to the visual priors provided by pretrained diffusion models. However, large 2D visual models exhibit spatial perception hallucinations, leading to multi-view inconsistency in 3D content generated through Score Distillation Sampling (SDS). This phenomenon, characterized by overfitting to specific views, is referred to as the "Janus Problem". In this work, we investigate the hallucination issues of pretrained models and find that large multimodal models without geometric constraints possess the capability to infer geometric structures, which can be utilized to mitigate multi-view inconsistency. Building on this, we propose a novel tuning-free method. We represent the multimodal inconsistency query information to detect specific hallucinations in 3D content, using this as an enhanced prompt to re-consist the 2D renderings of 3D and jointly optimize the structure and appearance across different views. Our approach does not require 3D training data and can be implemented plug-and-play within existing frameworks. Extensive experiments demonstrate that our method significantly improves the consistency of 3D content generation and specifically mitigates hallucinations caused by pretrained large models, achieving state-of-the-art performance compared to other optimization methods.

Main paper: 
 introduction: 
 Recent studies on 3D content generation have made significant progress, emerging as a central research focus in computer vision and computer graphics. The approaches for 3D content generation can be categorized into two primary categories: those based on 2D priors and those based on 3D priors. The strategies utilizing 2D priors typically learn 3D representations by approximating the probability distribution of 2D rendered images relative to a pre-trained diffusion model. This approximation is achieved during the visual guidance phase through a sophisticated optimization technique known as Score Distillation Sampling (SDS) [40] .However, methods based on 2D priors often suffer from overfitting to specific viewpoints of rendered images, resulting in generated 3D content that deviates from the expected distribution [2] . This overfitting leads to spatial perception inaccuracies, such as the Janus problem, where the generated objects display implausible duplications of features like faces or limbs, as depicted in Fig. 1 . An intuitive would be to learn priors from high-quality 3D data [19, 36, 50, 65, 62, 29, 18] . However, the limited availability and the often sparse supervision of 3D data pose significant challenges to maintaining view consistency and enhancing the generalizability of the generated content [23, 21] . Moreover, due to 3D generation tasks spanning a diverse array of domains, the scalability of data-driven models remains markedly constrained, limiting their applicability across a comprehensive spectrum of potential uses. The first two rows show overfitting with repeated frontal views, while the third row, using Hallo3D, achieves more consistent results. This highlights the issue and clarifies the expected outcome.To mitigate the hallucination problem and ensure view-consistent generation, we leverage the large multi-modal models to infer and adjust the geometric structures of the generated content. These models can recognize spatial relationships and evaluate the structural consistency of visual contexts by interpreting 3D elements such as lighting and proportion from 2D renderings. Building on this observation, we propose a novel generation-detection-correction paradigm.In this paradigm, we utilize multi-modal models to refine rendered images, ensuring visually coherent results. Our strategy improves cross-view consistency without relying on the prompt provided for diffusion guidance, thereby bridging text-driven and image-driven methods.In light of the findings above, we present Hallo3D, a novel, tuning-free approach that significantly enhances the multi-view consistency of 3D generation and is applicable across various generation methods. Our approach comprises three core techniques. Multi-Modal Hallucination Detection: This technique detects concrete hallucinations in renderings by leveraging large multi-modal models to represent inconsistency query information. Prompt-Enhanced Reconsistency: Utilizing the detection results from Multi-Modal Hallucination Detection as an enhanced negative prompt to precisely eliminate inconsistent artifacts in the renderings. Multi-view Appearance Alignment: This technique ensures uniform color and texture in renderings from different viewpoints by controlling attention in the diffusion de-noising process. By integrating these techniques, Hallo3D effectively addresses the challenges of maintaining consistency in 3D generation, providing a robust solution applicable to various generative methods.Experimental results demonstrate that our method exhibits a significant advantage in multi-view consistency compared to baseline models and can be robustly applied to various 3D generation tasks.Our contributions can be summarized as follows:• We propose Hallo3D, a novel tuning-free method that significantly enhances the multi-view consistency of 3D content generation and can be widely applied across various 3D generation paradigms, achieving outstanding experimental results.• We demonstrate that large multi-modal models, unconstrained by geometry, can infer geometric structures and be utilized to detect and mitigate hallucinations in 3D generation.• We introduce an optimization strategy that aligns the structures and surfaces of 3D content across views and addresses artifacts and hallucination through enhanced prompts. 
related work: 
 Text-to-3D Generation. The evolution of diffusion models has markedly enhanced text-to-3D generation. DreamFusion [40] initiated text-guided 3D modeling by using visual priors from 2D diffusion models to train 3D architectures, incorporating MipNeRF 360 [3] and Imagen [47] . While NeRF-based methods [33, 32, 48, 66, 59, 22, 12, 24, 10] handle complex lighting well, they are slower due to continuous parameter updates. In contrast, 3D Gaussian Splatting (3DGS) methods [20, 56, 5, 63, 39] have improved rendering speeds, showing their efficacy in complex scenarios.Image-to-3D Generation. Images from specific views of a 3D model demonstrate improved visual consistency for 3D generation tasks. Image-based methods [53, 11, 30, 54, 1, 41] typically surpass text-based approaches by leveraging viewpoint-specific ground truth. 3D-aware image generation techniques [61, 7] utilize neural networks to enhance rendering beyond the primary viewpoint, although training data scarcity [23, 21] remains a challenge. Recent strides in integrating 3D visual data into 2D diffusion models [19, 36] have notably enhanced image-based 3D generation, reducing perceptual errors and improving generative quality.Methods for Enhancing Multi-view Consistency. The primary challenge in enhancing 3D generation consistency is addressing hallucinations from 2D pre-trained diffusion models. A common strategy involves integrating additional 3D information into the diffusion process via fine-tuning [50, 65, 62, 29, 18] . This includes training models to handle consistent 3D subjects [46, 44] , transparent backgrounds [64] , and diverse viewpoints [49] . Recent advancements have also adjusted prompt embeddings to enhance viewpoint accuracy [17, 2] , although this remains limited for non-orthogonal views. Alternatives include using geometric methods [25] or treating 3D generation analogously to video generation [57] , though these tend to be framework-specific. In contrast, our method effectively optimizes arbitrary viewpoints, making it versatile across different 3D frameworks. 
preliminaries: 
 Diffusion Models. The diffusion model [15] has a fozrward diffusion process with diffusion steps from 0 to T , which degrades the original sample x 0 into pure noise x T ,EQUATION)where t is the noise injection level, and α := (α 1 , . . . , α T ) ∈ R T ≥0 are hyper-parameters to determine noise scales at T diffusion steps, and the reverse diffusion process is used during inference to generate x 0 from x t . In text-guided diffusion models [52] , the model is conditioned on text prompts P , which are converted into text embeddings via a text encoder such as CLIP [43] . The diffusion model ϵ ϕ is trained using the MSE loss between the predicted noise εϕ and the actual noise ϵ,L(ϕ) = E t∼U (1,T ),ϵ∼N (0,I) ∥ϵ -ϵ ϕ (x t , t, P )∥ 2 2, where U (1, T ) represents a uniform distribution over the set {1, • • • , T }, and N (µ, Σ) represents a multivariate Gaussian distribution with mean µ and covariance Σ. To enhance the alignment between text and images, Classifier-free guidance (CFG) [16] guides the generation of samples usingEQUATION)where ∅ is a special null text prompt representing the unconditional case, and s > 0 is the guidance scale. Increasing the guidance scale improves text-image alignment but reduces diversity. In practice, the ∅ text prompt is replaced with a negative prompt P -consisting of negative descriptions [9] to avoid undesired content in the generated samples.Score Distillation Sampling. Score Distillation Sampling (SDS) employs vision priors from pre-trained 2D diffusion models to supervise 3D models, establishing it as a foundational learning method in the domain of 3D generation, initially proposed by DreamFusion [40] . A 3D representation model, parameterized by θ, and the pre-trained diffusion model ϵ ϕ , together enable the rendering of an image x = g(θ, c) from the 3D content, where g(•) is a differentiable generator to render x and c is the camera pose. To ensure that x consistently exhibits high quality from any view and to align the probability of x with p(ϕ), SDS introduces a score estimation function εϕ (x t ; t, P ). This function We jointly optimize our model using L SDS and L CG . For L SDS , we identify a focal view from multi-view renderings based on the camera pose, utilizing it as the keys (K) and values (V) to align all the four images using attention. This process harmonizes the appearance and feeds the output into the 2D Diffusion on the left, which plays a crucial role in refining the noise prediction. For L CG , we query hallucinations and inconsistencies in the rendering using an LMM and apply the results, outputted as enhanced negative prompt, to the following image optimization process to re-consistent a high-quality image. We calculate the L CG based on the differences between the two images, thereby enhancing the consistency of the 3D content.predicts the noise εϕ based on the text condition P and the noisy image x t , into which Gaussian noise ϵ has been injected. Furthermore, by calculating the discrepancy between εϕ and ϵ, the score function identifies the gradient direction for updating parameter θ, thereby enhancing the training of the 3D model. The specific computation of the gradient is as follows:EQUATIONwhere w(t) is a weighting function.Building upon our understanding of the SDS as discussed in Sec. 3.1, we further explored the impact of SDS on the consistency of appearances in 3D generation. We observed that SDS processes images from only one view at a time, which contradicts our intuition that enhancing 3D multi-view consistency should involve the simultaneous processing of multiple viewpoint images. Experimental results demonstrate that this approach led to a lack of interaction between images from different views during the training process, resulting in the loss of some surface information during the noise prediction process, as shown in Fig. 7 in the ablation study.To circumvent the limitations of SDS, which typically favors image generation from a single view, we propose a "Multi-view Appearance Alignment" strategy. This approach introduces a consistent denoising method εϕ (•) that incorporates an attention mechanism AAttn(•), enabling the rendering of multiple images from random views and providing a broader perspective compared to techniques focused primarily on single-view image generation [40, 58] .Specifically, inspired by recent advancements in diffusion models [27, 14, 4, 55, 35] , which suggest that query features within attention spaces primarily shape image structure and layout, while key and value features influence texture, our method leverages this insight. As illustrated in the top right corner of Fig. 3 , we select a focal view i based on the camera pose, using the image from this viewpoint to provide the key and value features in the attention module. These are used to compute query features across all views, ensuring alignment of appearances. The attention is defined by theThe extent of the blur on the background dancers appear slightly exaggerated and multiple subjects appear in the image, suspected to be ghostly figures.This image was rendered from the 3D generated by AI, and you can look at the spatial structure of the object in the picture to see if there are any inconsistencies with body shape, perspective, surface texture, color, and so on.You are a master of graphic restoration. Based on your analysis above, summarize the 3D inconsistency problems as negative prompts for restoring the image I provided. Three references are as bellow: Negative Prompt: "multi-head, unnatural lighting, smooth appearance, distorted color, long neck"The whole photo, especially the dancer's face, is blurry. Subjects in the picture are confusing and there may be structural inconsistencies.Negative Prompt: "ghost image, incongruous perspective, obscured body positioning, indistinct fingers and feet due to excessive blur, imprecise shadowing, lack of texture detail, absence of volumetric depth in clothing"Negative Prompt: "blurry background, duplicated feature, unsharp edges, lack of detail clarity, artificial object"Figure 4 : A multi-modal case study for evaluating the capabilities of LMMs in 3D generation tasks. The first round of dialogue demonstrates that LMMs can infer structural consistency from 3D rendered images, while the second round shows that LMMs can respond in specific formats, allowing us to subsequently identify the negative prompts output using regular expressions.following formula:EQUATIONwhere AAttn(•) is the appearance attention, with K i and V i as the key and value features corresponding to the image rendered from the focal view i, and Q as the query feature from all views. The key and value are derived from the focal view, while each of the four views calculates a distinct query. In the denoising strategy εϕ (•), this attention mechanism functions as cross-attention, aligning features from all views with the focal view to ensure consistent appearances. This process occurs within the U-Net network [45] in εϕ (•), prior to the cross attention with the prompt.As shown in Fig. 4 , the rendering image in the top right corner of the figure exhibits significant inconsistencies, due to the limitations of 2D pre-trained models in comprehending spatial concepts. This often leads to hallucinations and overfitting to specific viewpoints. However, we believe that Large Multi-modal Models (LMMs) have the capability to reason about and mitigate these hallucinations. To demonstrate this, we designed a two-phase inquiry involving LMMs, specifically using high-performing GPT-4V [38] and LLaVA [26] as examples. The dialogue depicted in the figure indicates that although LMMs were not explicitly trained with geometric constraints, they could identify inconsistencies in the 3D renderings and categorize them as negative prompts. Additionally, LMMs can standardize their output format based on a one-shot reference, making it easier for us to extract negative prompts.Specifically, in our model, we input one 2D rendered image alongside 3D-aware inquiry prompts, denoted as P I , into the multi-modal large modal to assist in automatically identifying inconsistencies present during the 3D generation process. To further mitigate hallucinations and correct inconsistencies, we have standardized the output format of the LMM, enabling it to accurately generate negative prompts based on the provided shots. These negative prompts can then be used to rectify distorted images in subsequent steps. Given their effectiveness in purposefully addressing inconsistencies, we refer to them as "Enhanced Negative Prompts". We formalize this process as follows:EQUATIONwhere D ψ is the LMM parameterized by ψ, and P - E is the enhanced negative prompt.With the enhanced negative prompt P - E introduced in Sec. 3.3, a straightforward method to refine 2D renderings involves employing image editing algorithms to address inconsistencies. However, existing approaches predominantly focus on adjustments to the null prompt [34] or the modification of positive prompts [14] , which are generally ineffectual for altering the geometric structures in 2D images derived from 3D models. To address this limitation, we introduce a novel module for achieving re-consistency in 2D renderings, termed "Prompt-Enhanced Re-consistency," which leverages P - E to effectively refine the geometric fidelity of the rendered images.We regenerate the 2D rendered image x 0 under the guidance of P - E . Specifically, to preserve the original semantic information of x 0 , we employ Denoising Diffusion Implicit Models (DDIM) [51] to invert theimage x 0 to its noisy representation x T . Subsequently, we apply DDIM sampling to generate the consistent versions of the image, denoted as x0 , from x T as follows:EQUATIONwhere εϕ (x t , t, P + , P - E ) is the denoising strategy incorporated the attention mechanism Attn(•) in Sec. 3.2, and is adjusted by Classifier-Free Guidance (CFG) [16] , with the null text prompt replaced by the enhanced negative prompt P - E . This approach ensures that the regenerated image retains its core semantic integrity while improving its multi-view consistency. After completing T iterations as delineated by Eq. 1, we successfully achieve the re-consistent image x0 , effectively reconciling the image consistency with its original semantic information.Finally, we train the 3D model θ using the MSE loss L CG between x 0 and x0 in the image space:EQUATIONIt is worth noting that we apply Prompt-Enhanced Reconsistency only when the rendered image exhibit complete semantic structure. Our detector, D ψ , assesses the semantic completeness of the image. If the semantic structure is deemed incomplete or unclear, D ψ returns None, precluding further processing. This ensures that enhancements are only applied to images that are adequately prepared. The dependency of our enhancement process on the state of semantic completeness directly influences the formulation of the final training loss for the 3D model, as detailed below:EQUATION)where w is set to balance the magnitude of L SDS and L CG . By incorporating L CG , which is only applied when D ψ confirms the semantic readiness of the image, we ensure that our model focuses on enhancing well-formed images. This selective application of L CG prevents further exacerbating the quality of images already of poor quality. Simultaneously, it avoids misallocating resources to images that do not benefit from the intended enhancements, thereby improving the efficiency and effectiveness of our training process. For more implementation details, see the Appendix.A.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Justification: Without theory assumptions and proofs.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
expremients: 
 In this section, we comprehensively evaluate Hallo3D's performance within two categories of 3D generation frameworks: text-to-3D and image-to-3D. We present comparative results with other baseline models to highlight its capabilities. Additionally, to further substantiate the effectiveness of Hallo3D in enhancing multi-view consistency in 3D generation, we have conducted an extensive user study. Finally, we designed ablation experiments to validate the necessity of the framework's design.Baselines. We evaluated our method against several established baselines, demonstrating strong performance across diverse frameworks. These include text-to-3D models like GaussianDreamer [63] , Score Jacobian Chain (SJC) [58] , DreamFusion-IF [40] , and Magic3D [24] , as well as imageto-3D models such as DreamGaussian [53] and Zero-1-to-3 [28] , an extension of DreamFusion.We also included methods based on NeRF [33] and 3DGS [20] for a comprehensive comparison. Identical parameter configurations and seed values were maintained for fair comparison, using default hyperparameters from the baselines' open-source implementations. We employed the Threestudio library [13] for SJC and Magic3D, and the official codebases for the other methods. Additionally, we conducted experiments to evaluate the time consumption of Hallo3D, detailed in the Appendix.B. Metrics. The field of 3D generation struggles with the absence of ground truth, complicating the development of a unified evaluation metric. To address multi-view consistency, we reviewed existing evaluation methods and identified 3D inconsistencies using CLIP-Score [43] . We generated 80 unique 3D prompts using ChatGPT [37] and arranged 16 cameras in a 360-degree configuration around the z-axis. The average CLIP-Score across all views measured consistency.Zero-1-to-3 Hallo3DZero-1-to-3 Hallo3D To facilitate a more direct comparison, we rendered both Hallo3D and the baseline models from two complementary angles and magnified specific details. In our qualitative evaluation for text-driven 3D content generation, we randomly selected three prompts from a dataset of 80 and used two high-definition images from Google Images for imagedriven 3D generation. The results, shown in Fig. 5 and Fig. 6 , reveal significant enhancements in multi-view consistency. Baseline models often produced flawed figures, such as headless "flamingos" or "dogs" with multiple heads and ears. In contrast, our models achieved more realistic and consistent outputs, confirming the effectiveness of our approach. The 360-degree visualization is shown in Appendix.C.Computational results. Following [63, 42, 53] , we evaluated the CLIP-Score to assess the quality and consistency of 3D generated contents, as presented in Tab.1. The results indicate that our approach outperforms all baseline models, confirming the effectiveness of our method. It should be noted that the existence of a ground truth corresponding to the front view in image-driven 3D generation generally leads to higher generation quality.Consequently, for image-3D tasks, we adhered to the experimental setup outlined in [31, 60] . Specifically, we selected 60 objects from the GSO [8] and Objaverse [6] datasets, replacing overly simple objects to ensure a more robust evaluation. These objects were rendered in frontal views at a resolution of 256x256. To comprehensively assess performance, we utilized Chamfer Distance (CD) and Volume IoU (Vol. IoU) for evaluating geometric accuracy, along with PSNR, SSIM, and LPIPS for measuring visual quality. As presented in Tab.4, the experimental results clearly indicate that our method surpasses the baseline across all metrics, achieving significant improvements in both geometry and texture quality. This further substantiates the broad applicability of our approach, demonstrating its capacity to enhance both text-to-3D and image-to-3D tasks. User study. In our user study, we recruited 58 volunteers with expertise in artificial intelligence to participate in our experiment. To comprehensively assess the quality discrepancies among various generated 3D models, we developed an extensive scale for the volunteers to fill out. Specifically, we generated 120-frame videos for each 3D model, totaling 32 video sets. Our comparative approach involved evaluating each model independently on three criteria: "Multi-view Consistency," "Overall Quality," and "Alignment with Prompt," with ratings on a scale from 1 to 10. The findings were summarized by compiling the average scores, and can be seen in Tab.2 and Tab.3.We conducted ablation experiments on the three Hallo3D modules, as shown in Fig. 7 . Starting from the complete model, we independently removed each module and assessed their effects. Notably, in the "w/o C" setting, the output of LMM, P - E is applied to L SDS calculations to demonstrate the necessity of L CG . Additionally, we conducted an ablation on "w/o C & P - E ", where P - E is not applied anywhere in the "w/o C" setting, to further highlight the effectiveness of the module.In Fig. 7 , we focus on how module A primarily affects color and texture, while module B and module C enhance cross-view consistency. Specifically,• In Row 2, w/o A: The lion appears significantly darker than in Row 1, and the deer exhibits a blurry halo accompanied by an unnatural color shift.• In Row 3, w/o B: The lion's head is noticeably deformed in both the first and third columns, while the deer entirely loses its head.• Row 4, w/o C: The "second face" appears on the lion's back in the third column, and the right deer's image shows a clear Janus Problem, with multiple legs and a distorted body visible in the second and fourth columns. In our experiments, we observed that the loss function L CG , which is computed on a per-pixel basis, typically exhibits a larger magnitude in comparison to L SDS . To achieve a balanced scale between these losses, we assigned a weight of w = 0.1 to L CG in Eq.8. It is important to note that this adjustment in weight does not diminish the importance of L CG in any way. Specifically, as shown in Fig. 8 , even after scaling L CG by the factor w, it maintains a larger magnitude compared to L SDS . This demonstrates that L CG provides ample guidance for 3D generation, ensuring effective optimization throughout the process.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: We describe it in Section 4. 
conclusion: 
 In this paper, we introduce Hallo3D, a novel approach designed to enhance 3D content generation through both text-driven and image-driven methods. We demonstrate the capability of large multimodal models to infer geometric structures and detect hallucination arising from 2D diffusion models. By combining LMM with diffusion models, we achieve re-consistent 2D images applicable in the 3D domain. Extensive experimental evidence substantiates that our method significantly improves consistency and mitigates hallucinations in 3D content generation. Additionally, we thank Haoyang Tong from the University of Chinese Academy of Sciences for his contributions to this work.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [No] Justification: We are currently organizing the code and plan to release it as open-source in the future.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: We describe it in Section 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Our main results are based on user study.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: We describe it in Section A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: None. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All existing assets such as code, data, and models used in our research are properly credited to their respective creators or owners. We have explicitly mentioned the licenses and terms of use for each asset in our documentation. Additionally, we have ensured that all terms of use are strictly adhered to, including obtaining necessary permissions for assets with restrictive licenses. This practice supports ethical research standards and ensures legal compliance in the use and distribution of third-party resources.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [No] Justification: We are currently organizing the code and plan to release it as open-source in the future.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Selection of focal view. We use Fovy, the camera's vertical view, as our selection standard. The first view with Fovy exceeding 120% of each baseline's default becomes our focal view, enabling a broader shot capturing more object details.Details of LMM setup. We used different P I than in Fig. 4 . The primary purpose of Fig. 4 is to use a case study to demonstrate how LMMs can infer structural consistency and respond in specific formats.To highlight this capability, we employed two dialogues. In practice, we used a single interaction to query the LMM to achieve faster runtime. The specific setting is as follows."You are a master of 3D generation, and please refer to the 'Prompt' and 'Negative Prompt' below to identify the inconsistency in the image I provided you with, with body shape, perspective, texture, and so on. Reference: 'Prompt': '3d render of xx, front view, standing, high quality, 4K', 'Negative Prompt': 'multi-head, unnatural lighting, smooth appearance, distorted color, long neck, two-nosed, extra limbs' ".For LMM, we chose the locally deployed LLaVA [26] , using the version llava-v1.6-34b.General prompt. Our method acts as a universal enhancement for 3D generation, considering the common use of general negative prompts in baseline methods [63, 58, 40, 53, 24, 28] . The specific setting is as follows."unnatural colors, poor lighting, low quality, artifacts, smooth texture".We recorded the runtime using two baselines: GaussianDreamer [63] based on 3DGS with fewer iterations and faster speed, and DreamFusion [40] based on NeRF with more iterations and slower speed, on NVIDIA V100.To optimize the process, we begin calculating L CG later in the training and only every 4 iterations in our experiments. This approach is consistent with the statement in Sec.3.4 that "this module only works when the rendered images exhibit complete semantic structures." The rationale is twofold: first, during the early stages of training, the 3D assets are relatively disorganized and lack clear semantic structures, making it challenging for LMMs to reason accurately. Therefore, we delay the introduction of L CG . Second, we empirically found that calculating L CG every 4 iterations does not affect performance, allowing us to reduce training time. The results are presented in the Tab.6.Notably, since our method includes the "Multi-View Appearance Alignment" module, which requires attention calculations across four differently angled rendered images, we set the batch size to 4 for all baselines. To ensure a fair comparison, we reduced the number of iterations to 1/4 of the original. For example, DreamFusion originally trained for 10,000 iterations, and we adjusted it to 2,500 for optimization. GaussianDreamer(iteration=1200) already uses batch=4, so we matched its iteration count at 1,200.The experimental results indicate that while our method introduces some additional time overhead, this is fully justified by the significant improvements in performance and quality, especially considering the challenging nature of addressing the Janus Problem.To further underscore the necessity of incorporating L CG , we plotted the curve of L CG over the course of iterations. In conjunction with this, we also plotted the CLIP-Score for both the complete model and an ablated version that omits L CG . As illustrated in Fig. 8 , it is evident that L CG steadily decreases with increasing iterations, contributing to a marked improvement in the CLIP-Score. In Figure 10 : 360-degree visualization results in Fig. 5 (1).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. 


Decision: In the initial reviews, all reviewers acknowledged the innovative introduction of the LMMs, the significant performance improvements, and the solid comparative experiments. However, some reviewers pointed out the missing quantitative ablation experiment and analysis, and confused term such as consistency in the paper. During the rebuttal phase, the authors addressed these concerns with additional experiments, leading to positive feedback from all reviewers. AC has also carefully reviewed the paper, the reviews, and the rebuttal, and concurs with the reviewers' opinions. Therefore, AC recommends acceptance. It would be highly beneficial to include the additional results from the rebuttal phase in the final version.

Reviews: review: paper_summary: paper_summary: This paper aims to alleviate the multi-Janus problem in SDS-based 3D generation tasks. Inspired by the spatial structure inference capability of large multimodality models (LMMs), they propose a novel automatic negative prompt strategy. Specially, they input rendered images and 3D-aware inquiry prompts to LMM to obtain negative prompts. To keep the semantic consistency, they regenerate rendered image guided by a negative prompt and calculate regularization loss between the originally rendered image and the regenerated one.
strength_weakness: strength_weakness: - Lack of detaild of implementation and ablation study. I doubt the reproduction of this work.
    - For Multi-modal Hallucination Detection, is P_I  always the same as the prompts in Figure 4? All rendered images are inputted to LMM?   Which LMM is used in the final experiments, given GPT-4V and LLaVA in Figure 4? 
    - For Prompt-Enhanced Re-consistency, how to balance loss_SDS and loss_CG? Can you provide the loss curve of them? The author mentions that this module only works when the rendered images exhibit complete semantic structures. So what is the proportion in the training does it work on average? Why Dψ can produce "None"? I cannot find the corresponding instruction in Figure4's prompts.
    - For the Ablation Study, does w/o B mean that there are no adaptive negative prompts and a general negative prompt is used in the Prompt-Enhanced Re-consistency module to calculate L_CG? So what is the general negative prompt in w/o B ablation study? In w/o C study, does this mean that the generated negative prompt is directly used in equation (3) and no L_CG?

- Lack of some necessary comparison.
    - Please show the training time compared with multiple baselines. In my aspect, inference LMM for each iteration may introduce more training time.
    - The ablation study and analysis are not complete. It's interesting to show that replacing P_E^− with a general negative prompt. And why not introduce L_CG all the time? What is the impact of P_I for Hallucination Detection?
    - Magic3D is very similar with DreamFusion-IF except for the additional finetune phrase. Actually, I think finetune stage has less impact for geometric structures. I suggest authors replace Magic3D with ProlificDreamer. Prolificdreamer has a serious Janus problem but it produces clearer texture which is beneficial for Multi-modal Hallucination Detection.
    - Lack of comparison with mentioned prompt engineering methods: "prep-neg" and "debiasing scores and prompts".
questions: questions: - How to select a focal view in the Multi-view Appearance Alignment module is unclear. Is the focal view chosen randomly? Or the front view as a focal view always be chosen for each iteration? 
- Why the shown examples are low quality? Can the proposed method work for high-quality 3D generation?
- Why only provide one 360-degree example? Why not consider providing complete results in an appendix?
limitations: limitations: Based on the provided examples, it seems like this work just works for very low-quality 3D generation and simple prompts.

---------------------

This work presents the integration of VLM into text-to-3D generation, framing the Janus problem as hallucination detection. It automatically produces adaptive negative prompts via VLM. While the effect is constrained, this approach, devoid of training and 3D priors, offers insights into leveraging LLM in 3D generation. The primary constraint lies in the additional time overhead.
paper_summary: paper_summary: Hallo3D presents a tuning-free method, empowering 3D content generation frameworks via multi-modal LLM. The paper aims to solve the hallucination and inconsistent problems in SDS-based 3D content generation pipelines. With the design of multi-view appearance alignment and enhanced negative prompts, Hallo3D is able to improve the quality of generated 3D objects. The effectiveness of the proposed method was demonstrated through experimental verification using multiple frameworks.
strength_weakness: strength_weakness: 1. The method description is not clear enough.
- In lines 148-150, the paper claims defining a new denoising strategy, aiming to enhance the appearance consistency. But there's no further details about how to apply the new denoising strategy with appearance attention.
- Lines 184-186 provide some description, but lack a detailed explanation of how to utilize appearance attention, which is an important contribution claimed by the paper.
2. The ablation study lacks quantitative analysis.
- Figure 7 presents qualitative results; however, the lack of quantitative analysis makes these results less convincing.
3. Lack of enough quantitative analysis showcasing performance improvements on image-to-3d frameworks.
- Image-to-3D models can use images rendered from original 3D objects as input and ground truth, and then use the metrics of 2D images for measurement, such as SSIM, PSNR, etc. The GSO dataset or Objaverse dataset can be used.
4. The impact of methods on training time is not introduced. 
- The paper refine 3D contents generated by baseline models, but the impact of training time is not addressed.
questions: questions: 1. What does consistency refer to in the paper? The SDS-based method is optimized to generate a complete 3D content, not like directly using 2D diffusion models to generate multi-views. Does inconsistency refer to Janus problem?
2. Why need an additional $L_{cg}$ loss? Why not add negative prompts directly to the 2D diffusion models in the top left of Figure 3 for $L_{SDS}$ loss?
3. Is it necessary to first use the baseline methods to obtain a raw 3D content before cooperating with Multi-modal LLM for subsequent operations.
limitations: limitations: The paper has addressed the potential negative societal impact: the potential misuse of advanced 3D generation technologies could undermine social trust and compromise information integrity. And there are no other obvious potential social impacts.
paper_summary: paper_summary: The authors point out that generating 3D content with Score Distillation Sampling leads to multi-view inconsistencies. To address this challenge, they propose a novel, tuning-free method called Hallo3D. Specifically, they utilize large multi-modal models (e.g., GPT-4V) to detect and correct these inconsistencies during optimization. Notably, their method can be implemented in a plug-and-play manner with several text-to-3D generation baselines, achieving satisfying visual results.
strength_weakness: strength_weakness: The overall quality of the paper is quite good, but some problems still exist:

1.	In Figure 2, the "Illustration of the Janus problem" seems to understate the performance of current baselines. In the first two rows of the figure, eight faces of the dog are visible around the sculpture, which does not align with the experimental results from methods like DreamFusion. This is acceptable as an illustration, but it would be better to be more rigorous.

2.	Equation 6 in line 183 does not seem to be derivable from Equation 12 in the original DDIM paper. Please check this carefully.

3.	The ablation study section lacks clarity. Rows 2-4 do not show a clear visual difference, and the text in Section 4.4 does not provide an easy-to-follow illustration.
questions: questions: 1.	Section 3.4 is not clear enough. From lines 180-183, the authors seem to use the “DDIM Inversion” technique to regenerate an image without the so-called hallucination. However, generating such an image with CFG without damaging the original structure of the image is not an easy task (see Null-text Inversion). More illustration of this part should be provided.

2.	From Figures 3 and 4, it appears that you only feed a single image into the LMM for hallucination detection. However, a less serious multi-view inconsistency (e.g., a rabbit with three eyes) can only be observed from certain view directions. How can the effectiveness of Hallo3D be proven in this context?

3.	The generation-detection-correction paradigm introduced in this paper appears to require frequent use of the LMM. Given that generating 3D content with SDS optimization typically requires around 5,000-10,000 iterations, what is the overall time cost of your algorithm?

I’m willing to give this paper a higher rating if these three questions are well addressed!
limitations: limitations: Yes, the authors discuss the limitations of the proposed Hallo3D clearly at the end of the "Conclusion and Discussion" section.

Comments: title: Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
Official Comment by Authors
comment
Official Comment by Authors
comment: Thank you for your recognition and encouragement! We will take your suggestions into account and improve our paper in the further version.
Thanks for the response! I don't have further questions and I raise my grade to 5.
We sincerely hope our response has addressed your concerns, and we genuinely look forward to further communication with you!
We sincerely appreciate your thoughtful feedback on our rebuttal and the opportunity to further clarify our points. Thank you for your continued engagement with our work.


**Q1: Upon reviewing the quantitative results of the ablation study, it appears that the addition of self-attention did not result in significant changes. What might be the reason for this?**

CLIP-Score is widely used to estimate 3D consistency [1, 2, 3]. In line with these approaches, our ablation study reports the average CLIP-Score, calculated from the CLIP-Scores between 3D rendered images from different viewpoints and the prompt. However, due to the complexity of 3D generation, no single metric can fully capture consistency [2]. Consequently, the CLIP-Score may not completely account for appearance-related issues, such as color discrepancies, texture variations, or blurry halos across different viewpoints. While the images may still align with the prompt, these subtle appearance details may not be fully reflected by the CLIP-Score. For instance, differences between B/16 and L/14 highlight the role of A, but this difference is less noticeable in B/32.

To address this, we drew on the metric design from [4] by calculating the LPIPS between adjacent viewpoints and averaging these values across all viewpoints to compute the A-LPIPS. LPIPS offers a perceptual similarity measure that is closer to human vision, and we believe this metric better reflects the impact of the "Multi-view Appearance Alignment" module in our work. The experimental results are as follows:

| Metrics           | Hallo3D | w/o A  | w/o B  | w/o C  | w/o C & $P ^ {-} _ E$ | Baseline |
| ----------------- | :-----: | :----: | :----: | :----: | :-------------------: | :------: |
| A-LPIPS$\uparrow$ | 0.1863  | 0.1709 | 0.1582 | 0.1479 |        0.1382         |  0.1237  |

It can be seen that module A plays a crucial role in Hallo3D, as its absence leads to a noticeable performance drop. This is also illustrated in Fig. 7. In the left part, the lion in the second row appears dim in the first column but much brighter in the last four columns, indicating some inconsistencies in lighting. On the right, the deer in the second row shows a blurry halo, and the third column displays more saturated colors than the fourth, suggesting some distortion.

As discussed, the "Multi-view Appearance Alignment" module effectively enhances 3D consistency. In future work, we are open to collaborating with the research community to develop better metrics for evaluating 3D consistency and improving 3D generation methods.

**Q2: Could you analyze why incorporating the output of the LMM into the training of the SDS Loss is less effective than handling the CG Loss and SDS Loss separately?**

$\mathcal{L} _ {\rm{CG}}$ involves multiple denoising steps, which increases its ability to correct inconsistencies, making it more effective in addressing issues identified by $P_E^-$. In contrast, $\mathcal{L} _ {\rm{SDS}}$ focuses on overall generation quality, using single-step denoising to fit the diffusion model’s distribution. Therefore, we incorporate $P _ E ^ -$ into the calculation of $\mathcal{L} _ {\rm{CG}}$ to more effectively enhance 3D consistency.




[1] Yi T, et al. Gaussiandreamer: Fast generation from text to 3d gaussian splatting with point cloud priors. In *CVPR*, 2023.

[2] Liu F, et al. Sherpa3d: Boosting high-fidelity text-to-3d generation via coarse 3d prior. In *CVPR*, 2024.

[3] Tang J, et al. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. In *ICLR*, 2023.

[4] Susung Hong, et al. Debiasing Scores and Prompts of 2D Diffusion for View-consistent Text-to-3D Generation. In *NeurIPS*, 2023.
Thank you to the author for the response. I have two additional questions:

1. Upon reviewing the quantitative results of the ablation study, it appears that the addition of self-attention did not result in significant changes. What might be the reason for this?

2. Could you analyze why incorporating the output of the LMM into the training of the SDS Loss is less effective than handling the CG Loss and SDS Loss separately?
Thank you for recognizing our work! We’re pleased to have addressed most of your concerns, and we will incorporate your suggestions in future versions.
I think the authors have addressed most of my concerns, and proved the feasibility of their work. Thus I decide to raise my grade from 5 (borderline accept) to 6 (weak accept).
Thank you for taking the time to read our rebuttal and for providing your thoughtful response! We appreciate the opportunity to address these further concerns and provide additional clarification.

**Q1: Time Consumption**.

**Q1.1: About the substantial time consumption.**

A1: Thank you for your suggestion. We will clarify the "Original Time" and the corresponding training iterations in future versions. While our method does introduce additional time overhead, we believe it is within an acceptable range given the improvements in performance and quality it provides, especially considering the challenging nature of addressing the Janus Problem. 

**Q1.2: About the training iterations.**

Since our method includes the "Multi-View Appearance Alignment" module, which requires attention calculations across **four** differently angled rendered images, we set the batch size to 4 for all baselines. To ensure a fair comparison, we reduced the number of iterations to 1/4 of the original. For example, DreamFusion originally trained for 10,000 iterations, and we adjusted it to 2,500 for optimization. GaussianDreamer(iteration=1200) already uses batch=4, so we matched its iteration count at 1,200.

Additionally, as shown in Fig. 4, GaussianDreamer has already converged at 1,200 iterations, indicating sufficient training. Empirically, we also observed that further increasing the number of iterations did not improve the consistency of 3D generation.
 
**Q2: Confusion in Fig.2**.

A2: Thank you for your careful observation. Completely resolving the Janus Problem remains a significant challenge in the field. Our method specifically aims to enhance view consistency in 3D generation by addressing the hallucination issues commonly found in large models. As demonstrated by both quantitative and qualitative experiments, while there may still be a small presence of inconsistencies, our approach effectively mitigates these issues.

As noted in W2.3, ProlificDreamer produces impressive 3D quality but also exhibits a pronounced Janus Problem. In our experiments using ProlificDreamer as a baseline, we ensured a genuine evaluation of our method's performance by not cherry-picking any results. Despite this, as seen in Fig.2, our method still shows significant improvements over the baseline, greatly enhancing consistency.

Our method has demonstrated improvements in 3D generation consistency across various baselines (both text-based and image-based). Moving forward, we will treat the Janus Problem as a key research direction and look forward to contributing further to 3D generation alongside our fellow researchers.

Due to the limitations of the rebuttal format, we regret that we are unable to further modify the PDF to provide a more typical prompt visualization. However, we will modify the "cottage" prompt to better present a typical Janus prompt setup in final version.

---

Thank you once again for your valuable suggestions. We hope our response has addressed your concerns and would be happy to continue the discussion with you!
Thanks for the detailed implementation supplement provided in response, addressing concerns regarding reproducibility. Kindly incorporate these details into the updated Appendix. And supplementary experiments showcasing Hallo3D's superior CLIP Score and the loss curve trends are beneficial for understanding the effectiveness of this work.

**Questions Raised in the Rebuttal**

**Q1: Time Consumption**

Please clarify that "Original Time" corresponds to the baseline method with 1200/2500 iterations. Concern arises over the substantial time consumption introduced by Hallo3D, particularly concerning the potential increase with high-resolution generation. Additionally, it is queried whether all results are produced with training solely 1200/2500 iterations for Gaussian and NeRF representations, a difference from conventional implementation in 3D generation. 
Insufficient training may impact the quality of results in the paper. Addressing how to achieve high-quality results within a reasonable time is important.

**Confusion in Fig.2**

Please explain the suspected dual-headed appearance observed in the second render image of Hallo3D in rebuttal Fig.2. Furthermore, for enhanced difference, consider changing the prompt in rebuttal Fig. 3 in the updated version, as prompts like "cottage" typically do not exhibit Janus issues in 3D generation.