Hallo3D: Multi-Modal Hallucination Detection and Mitigation for Consistent 3D Content Generation
Authors: Hongbo Wang
Jie Cao
Jin Liu
Xiaoqiang Zhou
Huaibo Huang
Ran He
Venue: NeurIPS 2024 poster (2024)
Keywords: 3D Generation
Hallucination Detection and Mitigation
Multi-modal Inference
View Alignment
Primary area: generative_models
TL;DR: We address the 'Janus Problem' in 3D content generation with a novel method that leverages multimodal models to enhance multi-view consistency without requiring 3D training data.

Abstract: Recent advancements in 3D content generation have been significant, primarily due to the visual priors provided by pretrained diffusion models. However, large 2D visual models exhibit spatial perception hallucinations, leading to multi-view inconsistency in 3D content generated through Score Distillation Sampling (SDS). This phenomenon, characterized by overfitting to specific views, is referred to as the "Janus Problem". In this work, we investigate the hallucination issues of pretrained models and find that large multimodal models without geometric constraints possess the capability to infer geometric structures, which can be utilized to mitigate multi-view inconsistency. Building on this, we propose a novel tuning-free method. We represent the multimodal inconsistency query information to detect specific hallucinations in 3D content, using this as an enhanced prompt to re-consist the 2D renderings of 3D and jointly optimize the structure and appearance across different views. Our approach does not require 3D training data and can be implemented plug-and-play within existing frameworks. Extensive experiments demonstrate that our method significantly improves the consistency of 3D content generation and specifically mitigates hallucinations caused by pretrained large models, achieving state-of-the-art performance compared to other optimization methods.

Hallo3D: Multi-Modal Hallucination Detection and Mitigation for Consistent 3D Content Generation
Hongbo Wang 1.
Jie Cao Jin Liu!,3 Xiaoqiang Zhou 1.4 Huaïbo Huang 4.2+ Ran Hel
MAIS & NLPR, Institute of Automation, Chinese Academy of Sciences, Beijing, China "cchoo of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China
"School of Information Science and Technology, ShanghaïTech University, Shanghai, China *University of Science and Technology of China, Hefei, China
wanghongbo20240ia.ee ac. jie. caodcripac. ia. ac. liuj in2@shanghaitech. edu. en xq525kmail ustc. wstc..du..c. en, huaibo. hhwaggccriaaccccc ia.ac. ac..ac.com en, rhe@nlpr. aa.ac.cm
Abstract
Recent advances in pretrained 2D diffusion models have significantly improved visual prior guidance for 3D content generation. However, this process often lacks geometric constraints, leading spatial perception hallucinations and multi-view inconsistencies. To address this, we introduce Hallo3D, a tuning- tuning-free method for 3D content generation that leverages the geometric perception capabilities of large multi-modal models to detect and mitigate these hallucinations. Our approach follows
paradigm, using multi-modal inconsis- tencies as query information to guide the detection of hallucinations and formulate enhanced negative prompts that ensure consistent renderings. Additionally, we propose denoising strategy that employs attention mechanisms to maintain con- sistency color and texture across multiple views during visual guidance. Our method is data-independent, easily integrates with existing 3D content generation frameworks, and supports both text-driven and image-driven approaches. Extensive experiments demonstrate that method significantly improves the consistency and quality of generated 3D content, particularly in mitigating hallucinations common with 2D pretrained models.
Introduction
Recent studies on 3D content generation have made significant progress, emerging as a central research focus in computer vision and computer graphics. The approaches for 3D content generation can be categorized into two primary categories: those based on 2D priors and those based on 3D priors. The strategies utilizing 2D priors typically learn 3D representations by approximating the probability distribution of 2D rendered images relative to a pre- pre-trained diffusion model. This approximation is achieved during the visual guidance phase through sophisticated optimization technique known as Score Distillation Sampling (SDS) 40
However, methods based on 2D priors often suffer from overfitting to specific viewpoints of rendered images, resulting in generated content that deviates from the expected distribution (2) This over- fitting leads to spatial perception inaccuracies, such as the Janus problem, where the generated objects display implausible duplications of features like faces or limbs, as depicted in Fig | An intuitive
would be to learn priors from high-quality 3D data [1 36 50 65 62 29 [8] However, the limited availability and the often sparse supervision of 3D data pose significant challenges to maintaining view consistency and enhancing the generalizability of the generated content (23 21]. Moreover, due to 3D
Huaiio Huang is the corresponding author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Baselines
Hallo3D
Baselines
Hallo3D
prompt: ballerina in futu is practicing dancing.
prompt: A graceful gazelle is sprinting.
prompt: A standing knight in full armor.
prompt: A spiderman model with his arms open.
Figure 1: 3D Content Generation Results between Hallo3D (ours) and Baseline Model. Hallo3D can effectively solve the "Janus" problem and improve the multi-view consistency of the 3D generation.
generation tasks spanning diverse array of domains, the scalability of data-driven models remains markedly constrained, limiting their applicability across a comprehensive spectrum potential uses.
To mitigate the hallucination problem and en- sure view-consistent generation, we leverage the large multi-modal models infer and adjust the geometric structures of the generated con- tent. These models can recognize spatial rela- tionships and evaluate the structural consistency of visual contexts by interpreting 3D elements such lighting and proportion from 2D render- ings. Building on this observation, we propose novel -errattionnettcttoon- In this paradigm, we utilize multi molall models to refine rendered images, ensuring visually co- herent results. Our strategy improves cross-view consistency without relying the prompt pro- vided for diffusion guidance, thereby bridging text-driven and image-driven methods.
Baselines
Hallo3D (ours)
paradigm.
360
Illustration of the Janus problem. The first two rows show overfitting with repeated frontal views, while the third row, using Hallo3D, achieves more consistent results. This highlights the issue and clarifies the expected outcome.
Figure 2:
In light of the findings above, we present
Hallo3D, a novel, tuning-free approach that significantly enhances the multi-view consistency of 3D generation and is applicable across various generation methods. Our approach comprises three techniques. Multi-Modal Hallucination Detection: This technique detects concrete hal- lucinations in renderings by leveraging large multi-modal models to represent inconsistency query information.
Enhanced Reconsistency: Utilizing the detection results from Multi-Modal Hallucination Detection as an enhanced negative prompt to precisely eliminate inconsistent artifacts in the renderings. Multi-view Appearance Alignment: This technique ensures uniform color and texture renderings from different viewpoints by controlling attention in the diffusion de-noising process. By integrating these techniques, Hallo3D effectively addresses the challenges of maintaining consistency 3D generation, providing a robust solution applicable to various generative methods. Experimental results demonstrate that method exhibits a significant advantage in multi-view consistency compared to baseline models and can be robustly applied to various 3D generation tasks.
Our contributions can be summarized as follows:
We propose Hallo3D, novel tuning- free method that significantly enhances the multi-view consistency of 3D content generation and can be widely applied across various 3D generation paradigms, achieving outstanding experimental results.
We demonstrate that large multi-modal models, unconstrained by geometry, infer geometric structures and be utilized to detect and mitigate hallucinations 3D generation.
We introduce an optimization strategy that aligns the structures and surfaces of 3D content across views and addresses artifacts and hallucination through enhanced prompts.

---

2 Related Work
Text-to-3D Generation. The evolution of diffusion models has markedly enhanced text-to-3D generation. DreamFusion (40) initiated text-guided 3D modeling by using visual priors from 2D diffusion models to train 3D architectures, incorporating MipNeRF 360 B and Imagen [47]. While
NeRF-based methods [33 32
10] handle complex lighting well, they are slower due to continuous parameter updates. In contrast, 3D Gaussian Splatting (3DGS) methods
48
66
59
22
12
24
20 56 5
63 39
have improved rendering speeds, showing their efficacy in complex scenarios.
Image-to-3D Generation. Images from specific views of 3D model demonstrate improved visual consistency for 3D generation tasks. Image-based methods [53 [I 30 14 4T] typically surpass text-based approaches by leveraging viewpoint-specifie ground truth. 3D-aware image generation techniques [6]] [7] utilize neural networks to enhance rendering beyond the primary viewpoint, although training data scareity [2 remains a challenge. Recent strides in integrating 3D visual have notably enhanced image-based 3D generation, reducing perceptual errors and improving generative quality.
data into 2D diffusion models 19 36
Methods for Enhancing Multi-view Consistency. The primary challenge in enhancing 3D gen- eration consistency is addressing hallucinations from 2D pre-trained diffusion models. A common strategy involves integrating additional 3D information into the diffusion process via fine-tuning 50
This includes training models to handle consistent 3D subjects [46 44], transpar- ent backgrounds [64). and diverse viewpoints [49]. Recent advancements have also adjusted prompt
65 62 29
embeddings to enhance viewpoint accuracy 17 although this remains limited for non-orthogonal
views. Alternatives include using geometric methods [25] or treating 3D generation analogously to video generation [57], though these tend to be framework-speciff In contrast, our method effectively optimizes arbitrary viewpoints, making it versatile across different 3D frameworks.
3 Methodology
3.1 Preliminaries
Diffusion Models. The diffusion model 15 has fozrward diffusion process with diffusion steps from to T. which degrades the original sample Xo into pure noise X7,
X = Va,Xo + VI O4C,
E E~ (0,11,
(I)
where is the noise injection level, and a = (a1...., (T) E TR are hyper-parametert to determine noise scales at T diffusion steps, and the reverse diffusion process is used during inference to generate Xo from x,- In text- -oxt-g diffusion models [52]. the model is conditioned on text prompts P. which are converted into text embeddings via a text encoder such as CLIP (3). The diffusion model eg is trained using the MSE loss between the predicted noise Eg and the actual noise e,
( EL.... || Ex, t, P)
where (1, T) represents a uniform distribution over the set (1, ... T), and Mir, Σ) represents a multivariate Gaussian distribution with mean μ and covariance Σ. To enhance the alignment between text and images, Classifier- free guidance (CFG) [6) guides the generation of samples using
(x,, L, P,O) E (x,, 1, 0) + (Eg(X,, t, P) E (xr, 1,xx,,,90 0))
where 0 is special null text prompt representing the unconditional case, and 0 is the guidance scale. Increasing the guidance scale improves text-image alignment but reduces diversity. In practice, the 0 text prompt is replaced with a negative prompt P- consisting of negative descriptions [9] to avoid undesired content in the generated samples.
Score Distillation Sampling. Score Distillation Sampling (SDS) employs vision priors from
pre-trained 2D diffusion models to supervise 3D models, establishing it as a foundational learning method in the domain of 3D generation, initially proposed by DreamFusion 40). A 3D representation model, parameterized by 0, and the pre-trained diffusion model cs. together enable the rendering of an image x = g(0, c) from the 3D content, where g(-) is a differentiable generator to render and the camera pose. To ensure that x consistently exhibits high quality from any view and to align the probability of x with p(o), SDS introduces score estimation function és (xx; t, P). This function
3

---

Csns
k
Multi- View Appearance Alignment
2D Diffusion Models
3D Content Multi-view Renderings
cartoon
Attention
Cog
"Are there any Inconsistency?"
Pos: "A cartoon Fox
Enhanced Negative Prompt
Input Image
Input Inquires
Hollo2D
20 Diffusion Models
Abnormal ears, Distorted Perspective"
Input Image
Output Image
LMM
Output Analysis
Neg: Abnormal ears...
Multi-Modal Hallucination Detection
Prompt-Enhanee Enhanced Re- Consistency
Figure 3: Illustration of our pipeline. We jointly optimize our model using LSDS and Log. For CSDS, we identify focal view from multi-view renderings based on the camera pose, utilizing it as the keys (K) and values (V) to align all the four images using attention. This process harmonizes the appearance and feeds the output into the 2D Diffusion on the left, which plays a crucial role in refining the noise prediction. For LCG, we query hallucinations and inconsistencies in the rendering using an LMM and apply the results, outputted as enhanced negative prompt, to the following image optimization process to re-consistent high-quality image. We calculate the Lcg based the differences between the two images, thereby enhancing the consistency the 3D content.
predicts the noise éø based on the text condition P and the noisy image x into which Gaussian noise has been injected. Furthermore, by calculating the discrepancy between éG and the score function identifies the gradient direction for updating parameter 0, thereby enhancing the training of the 3D model. The specific computation of the gradient is as follows:
0x
VoLSDs x g(0)) & Er,c w((t) (E,( P.t)
(3)
00
where wtt is weighting function.
3.2 Multi-view Appearance Alignment
Building upon our understanding of the SDS as discussed in Sec. 3.1 we further explored the impact of SDS the consistency of appearances in 3D generation. We observed that SDS processes images from only one view at time, which contradicts our intuition that enhancing 3D multi-view consistency should involve the simultaneous processing of multiple viewpoint images. Experimental results demonstrate that this approach led to a lack of interaction between images from different views during the training process, resulting in the loss of some surface information during the noise prediction process, as shown in Fig. 7 in the ablation study.
To circumvent the limitations of SDS, which typically favors image generation from single view, we propose a "Multi-view Appearance Alignment" strategy. This approach introduces a consistent denoising method Eg(-) that incorporates attention mechanism A-ttn(-), enabling the rendering of multiple images from random views and providing a broader perspective compared to techniques
focused primarily single-view image generation 40 58).
which suggest that query features within attention spaces primarily shape image structure and layout, while key and value features influence texture, our method leverages this insight. As illustrated in the top right corner of Fig. 3 we select focal view based on the camera pose, using the image from this viewpoint to provide the key and value features the attention module. These are used to compute query features across all views, ensuring alignment of appearances. The attention is defined by the
Specifically, inspired by recent advancements in diffusion models 27
55 351

---

Demonstration of the Spatial Structure inference Capabilities Large Multi- modal Models
This image was rendered from the 3D generated by Al, and you can look the spatial structure of the object the picture to see there are any incomsistencies with body shape, perspective, surface texture, color, sa on.
The extent of the blur on the background dancers appear slightly exaggerated and multiple subjects appear in the image, suspected to be ghastly figures.
GPT-VV
The whole photo, especially the dancer's face, blurry. Subjects the picture confusing and there may be structural inconsistencies.
LLaVA
You are master of graphic restoration, Based on your analysis above, summarize the 3D inconsistency problems as negative prompts restoring image provided. Three references bellow:
Negative Prompt: "multi-head, unnatural lighting, smooth appearance, distorted color, long neck'
"ghost image, incongrupus perspective, obscured body positioning, indistinct fingers and feet due to excessive blur, imprecise shadowing. lack of texture detail, absence of volumetric depth in clothing"
Negative Prompt:
Negative Prompt: "blurry background, duplicated feature, unsharp edges, lack of detail clarity, artificial object"
A multi-modal case study for evaluating the capabilities of LMMs in 3D generation tasks. The first round of dialogue demonstrates that LMMs can infer structural consistency from 3D rendered images, while the second round shows that LMMs can respond in specific formats, allowing us subsequently identify the negative prompts output using regular expressions.
Figure 4:
following formula:
QK?
AAttn(Q, Ki, Vt) Softmax
V1
Vd
where AAtt(-)) is the appearance attention, with K, and Vi as the key and value features correspond- ing to the image rendered from the focal view i, and Q as the query feature from all views. The key and value are derived from the focal view, while each of the four views calculates distinct query. In the denoising strategy Eo (-), this attention mechanism functions cross-attention, aligning features from all views with the focal view to ensure consistent appearances. This process occurs within the U-Net network 45 in E (-). prior to the cross attention with the prompt.
3.3 Multi-modal Hallucination Detection
As shown in Fig. the rendering image in the top right corner of the figure exhibits significant inconsistencies, due to the limitations of 2D pre-trained models in comprehending spatial concepts. This often leads to hallucinations and overfitting to specific viewpoints. However, we believe that Large Multi-modal Models (LMMs) have the capability reason about and mitigate these hallucinations. To demonstrate this, we designed a two-phase inquiry involving LMMs, specifically using high-performing GPT-4V (88 and LLaVA VA (66 examples. The dialogue depicted in the figure indicates that although LMMs were not explicitly trained with geometric constraints, they could identify inconsistencies in the 3D renderings and categorize them as negative prompts. Additionally, LMMs can standardize their output format based on a one-shot reference, making it easier for us to extract negative prompts.
Specifically, in our model, we input one 2D rendered image alongside 3D-aware inquiry prompts, denoted as Py. into the multi-modal large modal to assist in automatically identifying inconsistencies present during the 3D generation process. To further mitigate hallucinations and correct inconsisten- cies, we have standardized the output format of the LMM, enabling it accurately generate negative prompts based on the provided shots. These negative prompts can then be used to rectify distorted images in subsequent steps. Given their effectiveness in purposefully addressing inconsistencies, refer to them as "Enhanced Negative Prompts". We formalize this process as follows:
PE = (x, Pı):
(5)
where D& is the LMM parameterized by 4h. and PE is the enhanced negative prompt.
3.4 Prompt- Enhance Re-consistency
With the enhanced negative prompt PE introduced in Sec. 3.3 straightforward method to refine
2D renderings involves employing image editing algorithms address inconsistencies. However,
5

---

existing approaches predominantly focus on adjustments to the null prompt [3 or the modification of positive prompts [14], which are generally ineffectual for altering the geometric structures in 2D images derived from 3D models. To address this limitation, we introduce a novel module for achieving re-consistency in 2D renderings, termed "Prompt-Enhanced" Re-consistency," which leverages PE to effectively refine the geometric fidelity of the rendered images.
We regenerate the 2D rendered image X under the guidance of PE. Specifically, to preserve the original semantic information of Xo, we employ Denoising Diffusion Implicit Models (DDIM) [ST] to invert theimage X to its noisy representation Xr. Subsequently, we apply DDIM sampling to generate the consistent versions of the image, denoted as Xy, from XT as follows:
a1-1 & + (v1 01-1 a1-1(1 ar) az
-1-1-
tt,
(6)
Ox
P- where Eo ( PF) Sec. 3.2 and is adjusted by Classifier-Free Guidance (CFG) [16], with the null text prompt replaced by the enhanced negative prompt P7. This approach ensures that the regenerated image retains its core semantic integrity while improving its multi-view consistency. After completing T iterations as delineated by Eq. []] we successfully achieve the re-consistent image Xo. effectively reconciling the image consistency with its original semantic information.
is the denoising strategy incorporated the attention mechanism Attn(.) ) in
Finally, we train the 3D model using the MSE loss Lcg between X, and Xn in the image space:
Log - E [(xu - x,)],
It is worth noting that apply Prompt-Enhanced Reconsistency only when the rendered image exhibit complete semantic structure. Our detector, D.. assesses the semantic completeness of the image. If the semantic structure is deemed incomplete unclear, Di. returns None, precluding further processing. This ensures that enhancements are only applied to images that are adequately prepared. The dependency of our enhancement process the state of semantic completeness directly influences the formulation of the final training loss for the 3D model, as detailed below:
(()) = LSDS wLCG, if (x, Pt) not None, otherwise.
(8)
LSDS,
where is set balance the magnitude of Lsns and LCg. By incorporating Lca. which is only applied when Da confirms the semantic readiness of the image, we ensure that our model focuses on enhancing well-formed images. This selective application of LoG prevents further exacerbating the quality of images already of poor quality. Simultaneously, it avoids misallocating resources to images that do not benefit from the intended enhancements, thereby improving the efficiency and effectiveness of our training process. For more implementation details, see the Appendix A
4 Expremients
In this section, we comprehensively evaluate Hallo3D's performance within two categories of 3D generation frameworks: text-to-3D and image-to-3D. We present comparative results with other baseline models to highlight its capabilities. Additionally, to further substantiate the effectiveness of Hallo3D in enhancing multi-view consistency in 3D generation, we have conducted an extensive user study. Finally, we designed ablation experiments to validate the necessity of the framework's 's design.
4.1 Experiment Setup
Baselines. We evaluated our method against several established baselines, demonstrating strong performance across diverse frameworks. These include text-to-3D models like GaussianDreamer
[63]. Score Jacobian Chain (SJC) [58), DreamFusion-I1 40). and Magic3D [24]. as well as image-
to-3D models such as DreamGaussian (53] and Zero-1-to-3 28 an extension of DreamFusion. We also included methods based on NeRF 33 and 3DGS 20 for comprehensive comparison. Identical parameter configurations and seed values were maintained for fair comparison, using default hyperparameters from the baselines' open-source implementations. We employed the Threestudio library [3] for SJC and Magic3D, and the official codebases for the other methods. Additionally, we conducted experiments to evaluate the time consumption of Hallo3D, detailed in the Appendix B

---

GaussianDreames
SJC
DreamFusion-II
Magic3D
Baselines
90"
270
90"
270
90"
270
90"
270
Hallo3D
prompt: An elegant flamingo standing tall with long legs and pinkish-white feathers.
Baselines
45
225
45
2255
45"
225"
45
225e
Hallo3D
prompt: sculpture of dog in medieval style, sitting upright under a spotlight.
Baselines
285
105
285
105
285
105
2850
105
Hallo3D
prompt: An electric sports car with aerodynamic curves and glossy blue appearance.
Figure 5: Qualitative comparison in text-driven 3D generation of HalloD and baseline models. To provide more straightforward comparison, we rendered both Hallo3D and the baseline models from two identical and complementary angles.
Table 1: Quantitative comparisons in text-driven 3D generation
Metrics
GaussianDreamer
Hallo3D
SJC
Hallo3D
DreamFusion-II
Hallo3D
Magic3D
Hallo3D
CLIP-Score B/32
21.31
24.53
14.09
22.15
14.93
CL.PP-Score B/16
22.67
27.00
21.36
26.36
15.98
23.79
16.41
24.29
CLIP-Score 1/14
26.72
Table 2: User study in text-driven 3D generation
Metries Multi-view Consistency 1
GrussianDreamer
Hallo3D
SIC
Hallo3D
DreamFusion-BB
Hallo3D
Magic3D
Hallo3D
6.00
8.87
4.53
7.63
4.63
6.33
5.13
7.53
Overall Quality
5.53
8.67
4.77
7,80
4.17
7.37
4.60
8,03
Aligument Prompt 1
The field of 3D generation struggles with the absence of ground truth, complicating the development of unified evaluation metric. To address multi-view consistency, we reviewed existing evaluation methods and identified 3D inconsistencies using CL.IP-ccore 43 We generated 80 unique
Metrics.
3D prompts using ChatGPT 37 and arranged cameras a 360-degree configuration around the z-axis. The average CLIP-Score across all views measured consistency.

---

DreamGaussian
Hallo3D
Zero-I-to-3
Hallo3D
DreamGaussian
Hallo3D
Hallo3D
Figure 6: Qualitative comparison in image-driven 3D generation of Hallo3D and baseline models. To facilitate a more direct comparison, we rendered both Hallo3D and the baseline models from two complementary angles and magnified specific details.
Table 3: User study in image-driven 3D generation
Metrics
DreamGaussian
Hallo3D
Zero-I-to4
Hallo3D
Multi-view Consistency
8.40
9.15
7.25
7.81
Overall Quality 1
9.23
9.52
6.10
7.20
Alignment with Prompt
8.55
8.00
8.30
8.95
4.2 Quantitative Comparison with Baselines
In our qualitative evaluation for text-driven 3D content generation, we randomly selected three prompts from dataset of 80 and used two high-definition images from Google Images for image- driven 3D generation. The results, shown in Fig. [ and Fig. (6) reveal significant enhancements in multi-view consistency. Baseline models often produced flawed figures, such as headless "flamingos" or "dogs" with multiple heads and ears. In contrast, our models achieved more realistic and consistent outputs, confirming the effectiveness of our approach. The 360-degree visualization is shown in Appendix[[ C
4.3 Qualitative Comparison with Baselines
Computational results. Following 63 53 we evaluated the CLIP-Score to assess the quality and consistency of 3D generated contents, as presented in Tab The results indicate that our approach outperforms all baseline models, confirming the effectiveness of our method. should be noted that the existence of a ground truth corresponding to the front view in image-driven 3D generation generally leads to higher generation quality.
60). Consequently, for image-3D tasks, we adhered to the experimental setup outlined in [3]]60]. Specifi- cally, we selected 60 objects from the GSO g and Objaverse (6) datasets, replacing overly simple objects to ensure more robust evaluation. These objects were rendered in frontal views at res- olution of 256x256. To comprehensively assess performance, we utilized Chamfer Distance (CD) and Volume IoU (Vol. IoU) for evaluating geometric accuracy, along with PSNR, SSIM, and LPIPS for measuring visual quality. As presented in Tab the experimental results clearly indicate that our method surpasses the baseline across all metrics, achieving significant improvements in both geometry and texture quality. This further substantiates the broad applicability of our approach, demonstrating its capacity to enhance both text-to-3D and image-to-3D tasks.
8

---

Hallo3D
Iallo3D
w/o.
w/o
w/o
w/o
w/o
w/o
prompt: majestic lion standing on rock.
prompt: deer grasing on grassy plain.
Figure 7: Ablation study of our method. In the figure, module A represents Multi-view Appearance Alignment Sec. 3.2 module stands for Multi-modal Hallucination Detection in Sec. 3.3 and module C denotes Prompt-Enhancee Re-Consistency in Sec. 3.4 We conducted ablation studies on each of these three modules respectively.
Table 4: Quantitative comparisons image-driven 3D generation
Metries
DreanGaussian
Hallo3D
Zero-I-to-3
Hallo3D
CD
0.0185
0.0171
0.0370
0.0283
Vol. loU1
0.5861
0.6099
0.4824
0.5602
PSNR
16.502
16.518
13.433
14.930
SSIM
0.8543
0.8793
0.7210
0.7527
LPIPS
0.2025
0.1726
0.3926
0.3328
User study. In our user study, we recruited 58 volunteers with expertise in artificial intelligence to participate our experiment. To comprehensively assess the quality discrepancies among various generated 3D models, we developed an extensive scale for the volunteers to fill out. Specifically, we generated 120-frame videos for each 3D model, totaling video sets. Our comparative approach involved evaluating each model independently on three criteria: "Multi-view Consistency," "Overall Quality," and "Alignment with Prompt," with ratings on scale from to 10. The findings were summarized by compiling the average scores, and can be seen in Tab and Tab
4.4 Ablation Study
We conducted ablation experiments on the three Hallo3D modules, shown in Fig. 7 Starting from the complete model, we independently removed each module and assessed their effects. Notably, in the "w/o C" setting, the output of LMM, PE is applied to LSDS calculations demonstrate the necessity of Leg- Additionally,
conducted ablation "w/o C & PE". where PT is not applied anywhere in the "w/o C" setting, to further highlight the effectiveness of the module.
In Fig 7 we focus on how module A primarily affects color and texture, while module B and module enhance cross-view consistency. Specifically.
In Row 2, w/o A: The lion appears significantly darker than in Row 1, and the deer exhibits blurry halo accompanied by an unnatural color shift.
In Row 3, w/o B: The lion's head is noticeably deformed in both the first and third columns, while the deer entirely loses head.
Row 4, w/o C: The "second face" appears the lion's back in the third column, and the right deer's image shows clear Janus Problem, with multiple legs and a distorted body visible in the second and fourth columns.

---

Loss and Score vs Iteration
LSDS
0.3L
0.230
CLI Score w/o L_
0.25
30003
1055
20000
0.700
0.205
0.200
200
400
600
D00
3000
1200
Remation
Figure 8: Loss curves for Log and LSDS, along with the CLIP-Score curves with and without Log-
Table 5: Quantitative Ablation Results
Metrics
Hallo3D
w/o
w/o B
w/o
w/o C& PE
Baseline
CLIP-Score B/321
24.25
23.98
23.65
22.46
22.23
21.27
CLIP-Score B/16
26.83
25.88
25.10
23.59
23.23
22.67
CLIP-Scone L/14
30.00
29.36
28.71
26.92
25.58
23.71
Additionally, conducted quantifiable ablation experiments, shown in Tab4.44 further demon- strating the effectiveness and necessity of each module in Hallo3D. Moreover, the better performance of "w/o C" compared to "w/o C & P also supports the necessity of introducing LCg.
4.5 Balance between LoG and LSDS.
In our experiments, we observed that the loss function Leg. which computed on a per-pixel basis, typically exhibits larger magnitude in comparison LSDS- To achieve balanced scale between these losses, we assigned weight of w = 0.1 to LeG in Eq is important to note that this adjustment in weight does not diminish the importance of LeG in any way. Specifically, as shown in Fig even after scaling Lcg by the factor w, it maintains larger magnitude compared to LSDS- This demonstrates that Leg provides ample guidance for 3D generation, ensuring effective optimization throughout the process.
Conclusion
In this paper, we introduce Hallo3D, novel approach designed enhance 3D content generation through both text-driven and image-driven methods. We demonstrate the capability of large multi- modal models to infer geometric structures and detect hallucination arising from 2D diffusion models. By combining LMM with diffusion models, we achieve re-consistent 2D images applicable in the 3D domain. Extensive experimental evidence substantiates that our method significantly improves consistency and mitigates hallucinations in 3D content generation. Additionally, we thank Haoyang Tong from the University of Chinese Academy of Sciences for contributions to this work.
Acknowledgements
This research is supported in part by several funding sources, including the National Natural Science Foundation China (Grant No. 32341009, 62206277, U21B2045, 62425606). Additional support has been provided by the Beijing Nova Program (20230484276), well as the Youth Innovation Pro- motion Association of the Chinese Academy of Sciences (Grant No. 2022132). These contributions have been instrumental in enabling the advancement of this work.
10

---

References
[1] Badour AlBahar, Shunsuke Saito, Hung-Yo Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang. Single-Image 3D Human Digitization with Guided Diffusion. SIGGRAPH, 2023.
[2] Mohammadreza Armandpour, Ali Sadeghian, Huangjie Zheng, Amir Sadeghian, and Mingyuan Zhou. Re-imagine the Negative Prompt Algorithm: Transform 2D Diffusion into 3D, alleviate Janus problem and Beyond. arXiv:2304.0496 2023.
[3] Jomathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, Peter Hedman. Mip-NeRF 360: Unbounded Anti-Aliased Neural Radiance Fields. In CVPR, pages 5460-5469, New Orleans, LA, USA, 2022.
[4] Mingdeng Cao, Xintao Wang, Zhongang Qi. Ying Shan, Xiaohu Qie, and Yinqiang Zheng. MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editing. In ICCV, pages 22560-22570 2023.
[5] Zilong Chen, Feng Wang, and Huaping Liu. Text-to-3D using Gaussian Splatting, arXiv:2309. 2023.
[6] Matt Deitke, Dustin Schwenk. Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsami, Aniruddha Kembhavi, and Ali Farhadi. Ohjaverse: A universe of annotated 3d objects. In CVPR, 2023.
[7] Yu Deng, Jiaolong Yang, Jianfeng Xiang, and Xin Tong. GRAM: Generative Radiance Manifolds for 3D-Aware Image Generation. In CVPR, 2022.
[8] Laura Downs, Anthony Francis, Nate Koenig, Brandon Kinman, Ryan Hickman, Krista Reymann, Thomas B McHugh, and Vincent Vanhoucke. Google scanned objects: A high-quality dataset of 3d scanned household items. In ICRA, 2022.
[9] Yilun Du Shuang Li, and Igor Mordatch. Compositional visual generation with energy based models. In NeurlPS, volume 33, pages 6637-6647, 2020.
[10] Lincong Feng, Mayu Wang, Maoyu Wang, Kuo Xu, and Xiaoli Liu. MetaDreamer: Efficient Text-to-3 Creation With Disentangling Geometry and Texture. arXiv:2311.1013,, 2023.
[11] Yang Fu. Sifei Liu, Amey Kulkarni, Jan Kautz, Alexei A. Efros, and Xiaolong Wang. COLMAP-Free 3D Gaussian Splatting. CVPR, 2023.
[12] Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. GraphDreamer: Compo- sitional 3D Scene Synthesis from Scene Graphs. In CVPR, 2023.
[13] Yuan-Chen Guo, Ying-Tian Liu, Ruizhi Shao, Christian Laforte, Vikram Voleti, Guan Luo, Chia-Hao Chen, Zi- Xin Zou, Chen Wang. Yan-Pei Cao, and Song- Hai Zhang. Threestudio. 2023.
[14] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfr Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to- Prompt Image Editing with Cross Attention Control. In ICLR, 2022.
[15] Jonathan Ho, Ajay Jain, and Picter Abbcel. Denoising Diffusion Probabilistic Models. In NeurlPS, 2020.
[16] Jonathan Ho and Tim Salimans. Classifier- Diffusion Guidance. In NeurlPS, 2021.
[17] Susung Hong. Donghoon Ahn, and Seungryong Kim. Debiasing Scores and Prompts of 2D Diffusson for View-consistent Text-to-3D Generation. In NeurlPS, 2023.
[18] Yifan Jiang, Hao Tang, Jen-Hao Rick Chang, Liangehen Song, Zhangyang Wang, and Liangliang Cao. Efficient3DDiM: Learning Generalizable Single-image Novel-view Synthesizer in One Day. In ICLR, 2023.
[19] Heewoo Jun and Alex Nichol. Shap-F: Generating Conditional 3D Implicit Functions. arXiv:2305.0244 2023.
[20] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkuchler, and George Drettakis. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Trans. Graph., 42(4))1-14, 2023.
[21] Chenghao Li, Chaoning Zhang, Atish Waghwase, Lik-Hang Lee, Francois Rameau, Yang Yang, Sung- Ho Bae, and Choong Seon Hong. Generative AI meets 3D: A Survey on Text-to-3D AIGC Era.
vrXiv:2305.06133 2023.
[22] Ming Li. Pan Zhou, Jia-Wei Liu, Jussi Keppo, Min Lin, Shuicheng Yan, and Xiangyu Xu. Instant3D: Instant tD Generation. IJCV, 2023.
[23] Xiaoyu Li. Qi Zhang, Di Kang, Weihao Cheng, Yiming Gao, Jingbo Zhang, Zhihao Liang, Jing Liao, Yan-Pei Cao, and Ying Shan. Advances in 3D Generation: A Survey, arXiv:2401. /7807 2024.
[24] Chen-Hsuan Lin, Jun Gao, Luming Tang. Towaki Takikawa, Xiaohui Zeng. Xun Huang. Karsten Kreis, Sanja Fidler, Ming-Yo Yu Liu, and Tsung- Yi Lin. Magic3D: High-Resolution Text-to-33 3D Content Creation. In CVPR, 2023.
11

---

[25] Fangfu Liu, Diankun Wu, Yi Wei, Yongming Rao, and Yueqi Duan. Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D Prior. In CVPR, 2023.
[26] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual Instruction Tuning. In AAAI, 2023,
[27] Jin Liu, Huaibo Huang. Chao Jin, and Ran He. Portrait diffusion: Training- free face stylization with chain-of phiinin. arXiv preprint arXiv:2312.02212, 2023.
[28] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick.
Zero-I-to-3: Zero-shot One Image to 3D Object. ICCV. 2023.
[29] Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang. SyncDreamer: Generating Muntiview-consistent Images from Single-view Image. In ICLR, 2024.
[30] Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J. Black, and Bernhard Schölkopf. Ghost on the Shell: An Expressive Representation of General 3D Shapes. In ICLR, 2023.
[31] Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theohalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In CVPR, 2024.
[32] Jonathan Lorraine, Kevin Xie, Xiaohui Zeng, Chen- Chen-Hsuan Lin, Towaki Takikawa, Nicholas Sharp, Tsung- Yi Lin, Ming-Y Yu Liu, Sanja Fidler, and James Lucas. ATT3D: Amortized Text-to-3D Object Synthesis. In ICCV, 2023.
[33] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jomathan T. Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. Commun. ACM, 65(1):99 106, 2022.
[34] Ron Mokady, Amir Hertz, Kfrr Aberman, Yael Pritch. and Daniel Cohen-Or. Null-text Inversion for Editing Real Images using Guided Diffusion Models. In CVPR, 2023.
[35] Jisu Nam, Heesu Kim, Donglae Lec, Siyoon Jin, Scungryong Kim, and Scunggyu Chang. DrcamMatcher: Appearance Matching Self-Attention for Cenuaniialll-Connisstee Text-t--lmage Personalization. In CVPR, 2024.
[36] Alex Nichol, Heewoo Jun, Prafulla Dhariwal, Pamela Mishkin, and Mark Chen. Point-E: A System for Generating 3D Point Clouds from Complex Prompts. arXiv:2212.08751, 2022.
[37] OpenAL. CharGPT.
[38] OpenAL. GPT-4 Technical Report. 2023.
[39] Hao Ouyang. Kathryn Heal, Stephen Lombardi, and Tiancheng Sun. Text2Immersion: Generative Immersive Scene with 3D Gaussians. arXtv:2312.09242 2023.
[40] Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall, DreamFusion: Text-to-3D using 2D) Diffusion. In ICIR, 2022.
[41] Guocheng Qian, Jinjie Mai, Abdullah Hamdi, Jian Ren, Aliaksandr Siarohin, Bing Li, Hsin- Ying Lee, Ivan Skorokhodov, Peter Wonka, Sergey Tulyakov, and Beraard Ghanem. Magic123: One Image to High-Quality 3D Object Generation Using Both 2D and 3D Diffusion Priors. In ICLR, 2023.
[42] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. 3d. In CVPR, 2024.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In JCML, 2021.
[44] Amit Raj. Srinivas Kaza, Ben Poole, Michael Niemeyer, Nataniel Ruiz, Ben Mildenhall, Shiran Zada, Kfir Aberman, Michael Rubinstein, Jonathan Barron, Yuanzhen Li, and Varun Jampani. DrcamBooth3D: Subject-Driven Text-to-3D Generation. In ICCV, 2023.
[45] Olaf Ronneberger. Philipp Fischer, and Thomas Brox, U-net: Convolutional networks for biomedical image segmentation. In MICCAI, 2015.
[46]
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- Booth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In CVPR, 2023.
[47] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Scyed Ghasemipour, Burcu Karagol Ayan. S Sara Mahdavi, Raphael Gontijo-Lopes, Tim Salimans, Jonathan Ho, David J Fleet, and Mohammad Norouzi. Photorealistic Text-to-lmage Diffusion Models with Deep Language Understanding. In NeurIPS, 2022.
[48] Hoigi Sco, Hayeon Kim, Gwanghyun Kim, and Se Young Chun. DITTO-NeRF Diffusion based Iterative Text To Omni- miriidirectional// 3D Model. arXiv:2304.02827, 2023.
12

---

[49] Junyoung Seo, Wooseok Jang, Min-Seop Kwak, Jaehoon Ko, Hyeonsu Kim, Junho Kim, Jin-Hwa Kim, Jiyoung Lee, and Seungryong Kim. Let 2D Diffusion Model Know 3D-Consistency for Robust Text-to-3 Generation. ICLR, 2023.
[50] Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. MVDream: Multi-view Diffusion for 3D Generation. In ICLR, 2024.
[51] Jiaming Song. Chealin Meng, and Stefano Ermon. Denoising Diffusson Implicit Models. In ICLR, 2022.
[52] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. CVPR, 2022.
[53] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation. In ICLR, 2023.
[54] Junshu Tang, Tengfei Wang, Bo Zhang, Ting Zhang. Ran Yi, Lizhuang Ma, and Dong Chen, Make-It-3D: High-Fidelity 3D Creation from A Single Image with Diffusion Prior. In ICCV, pages 22762-22772, Paris, France, 2023.
[55] Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-Play Diffusion Features for Text-Driven Image-to-lmage Translation. In CVPR, pages 1921-1930, 2023.
[56] Alexander Vilesov, Pradyumna Chari, and Achuta Kadamhi, CG3D: Compositional Generation for Text-to-3D via Gaussian Splatting. urXiv:2311.17977, 2023.
[57] Vikram Voleti, Chun-Han Yao, Mark Boss, Adam Letts. David Pankratz. Dmitry Tochilkin, Christian Laforte, Robin Rombach, and Varun Jampani. SV3D: Novel Multi-view Synthesis and 3D Generation from a Single Image using Latent Video Diffusion. arXiv:2403. 12008, 2024.
[58] Haochen Wang, Xiaodan Du Jiahao Li, Raymond A. Yeh, and Greg Shakhnarovich. Score Jacobian Chaining: Lifting Pretrained 2D Diffusion Models for 3D Generation. In CVPR, 2022.
[59] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su. and Jun Zhu. ProlificDreamer:: High-Fidelity and Diverse Text-to-3D Generation with Variational Score Distillation. In NeurIPS, 2023.
[60] Kailu Wu, Fangfu Liu, Zhihan Cai, Runjic Yan, Hanyang Wang, Yating Hu, Yueqi Duan, and Kaisheng Ma. Unique3d:
and efficient 3d mesh generation from a single image. arXiv:2405.20:: 2024.
[61] Jianfeng Xiang, Jiaolong Yang, Binbin Huang, and Xin Tong. 3D 3D-aware Image Generation using 2D Diffusion Models. In ICCV, 2023.
[62] Jiayu Yang, Ziang Cheng, Yunfci Duan, Pan Ji, and Hongdong Li. ConsistNet: Enforcing 3D Consistency for Multi-view Images Diffusion. In CVPR, 2023.
[63] Taoran Yi, Jiemin Fang, Junjie Wang, Guanjun Wu, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Qi Tian, and Xinggang Wang. GrussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models. In CVPR, 2023.
[64] Lvmin Zhang and Maneesh Agrawala. Transparent Image Layer Diffusson using Latent Transparency. arXiv:2402.1 1.1112240..1713.. 2024.
[65] Minda Zhao, Chaoyi Zhao, Xinyue Liang, Lincheng Li, Zeng Zhao, Zhipeng Hu, Changjie Fan, and Xin Yu. EfficienDDreamer: High-Fidelity and Robust 3D Creation via Orthogonal-view Diffusion Prior. In CVPR, 2023.
[66] Linqi Zhou, Andy Shih, Chenlin Meng, and Stefano Ermon. DreamPropeller: Supercharge Text-to-3D Generation with Parallel Sampling. In CVPR, 2023.
13

---

A Implementation details
Selection of focal view. We use Fovy, the camera's vertical view, as our selection standard. The first view with Fovy exceeding 120% of each baseline's 's default becomes our focal view, enabling broader shot capturing more object details.
Details of LMM setup. We used different Pr than in Fig The primary purpose of Fig 4 is to use case study demonstrate how LMMs infer structural consistency and respond in specific formats. To highlight this capability, we employed two dialogues. In practice, we used a single interaction to query the LMM to achieve faster runtime. The specific setting is follows.
"You are master of 3D generation, and please refer the 'Prompt' and 'Negative Prompt' below to identify the inconsistency in the image I provided you with, with body shape, perspective, texture, and so on. Reference:
'Prompt': '3d render of xx, front view, standing, high quality, 4K',
"Negative Prompt': 'multi-head, unnatural lighting, smooth appearance, distorted color, long neck, two-nosed, extra limbs"
For LMM, we chose the locally deployed LLaVA 26. using the version Ilava-v1. (.te-v1l..334b
General prompt. Our method acts as a universal enhancement for 3D generation, considering the
common of general negative prompts in baseline methods 66 58 40 setting is as follows.
53
24
28
The specific
"unnatural colors, poor lighting, low quality, artifacts, smooth texture".
B Time Consumption
We recorded the runtime using two baselines: Gaussian 63 based on 3DGS with fewer iterations and faster speed, and DreamFusion 40 based on NeRF with more iterations and slower speed, on NVIDIA V100.
To optimize the process, we begin calculating Leg later in the training and only every 4 iterations in our experiments. This approach is consistent with the statement in Sec that "this module only works when the rendered images exhibit complete semantic structures. " The rationale is twofold: first, during the early stages of training, the 3D assets are relatively disorganized and lack clear semantic structures, making it challenging for LMMs to reason accurately. Therefore, we delay the introduction of Log. Second, we empirically found that calculating LeG every iterations does not affect performance, allowing us to reduce training time. The results are presented in the Tabl6]
Notably, since our method includes the "Multi-Vine View Appearance Alignment" module, which requires attention calculations across four differently angled rendered images, we set the batch size to 4 for all baselines. To ensure fair comparison, we reduced the number of iterations 1/4 of the original. For example, DreamFusion originally trained for 10,000 iterations, and we adjusted it to 2,500 for optimization. CaussianDreameriiteraioo- 1200) already uses batch=4, so we matched its iteration count at 1,200.
The experimental results indicate that while method introduces some additional time overhead, this fully justified by the significant improvements in performance and quality, especially considering the challenging nature of addressing the Janus Problem.
Additional Experiments
C.1 The Effectiveness and Necessity of LeG
To further underscore the necessity of incorporating Lea, we plotted the curve of LCg over the course of iterations. In conjunction with this, we also plotted the CLIP-Score for both the complete model and an ablated version that omits Log. As illustrated in Fig it is evident that Log steadily decreases with increasing iterations, contributing to marked improvement in the CLIP-Score. In
14

---

Bascline
IL.r-Score: 0.3362
+Perp-Neg
CLIP-Score 0.3426
+Debias
CLIP-Score 0.3435
+Hallo3D
CLIP-Score: 0.3440
prompt: A 3D model of an adorable cottage with a thatched roof.
Figure 9: Comparison experiments with Perp-Neg and Debias.
Table 6: The time consumption introduced by Hallo3D.
Baseline
Iteration
Lea Start Rounds
Original Time
Extra Time
Total Time
Greaser
1200
1000
~8 min
10 min
~3 min
DreamFusion
2500
2200
51 min
115 min
~66 min
contrast, the CLIP-Score for the model without Leg exhibits only marginal improvement. These findings clearly highlight both the necessity and effectiveness of incorporating Leg into the model.
C.2 Comparison Experiments with Other 3D Consistency Enhancement Methods.
To further demonstrate the advantages of our method, compared it with other approaches 2 17 aimed at improving 3D consistency. As shown in Fig 9 Hallo3D more effectively addresses the Janus problem and achieves greater improvement in CLIP-Score.
C.3 360-degree Visualization Results
Due to space constraints, 5i in the main text only presents 3D generation results from selected viewpoints. The full 360-degree visualizations can be found in Fig 10 and Fig 11
D
Limitation
Our method has shown improvements in 3D generation consistency across various baselines, including both text-based and image-based approaches. However, method focused on enhancing view consistency, the quality of our experimental results is inherently tied to the performance of the baseline models. Moreover, the potential misuse of advanced 3D generation technologies poses risks to social trust and information integrity. Looking ahead, we will prioritize the Janus Problem as key research direction and are committed contributing further to the field of 3D generation alongside our fellow researchers.
15

---

Figure 10: 360-degree visualization results in Fig 5 (1).
16

---

Figure 11: 3e0-ee visualization results in Fig 5 (2).
17

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We describe it in Section 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We describe it in Section D
18

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Without theory assumptions and proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We describe it in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
19

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We are currently organizing the code and plan release it as open-source in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee.c www.ddddeeeooeuuummmceco ssi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
20

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
We describe it in Section 4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our main results are based on user study.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe it in Section
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
21

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httppp/s/bburrpss.cc/obbb...llad
Answer: [Yes]
Justification: We conducted the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We describe it in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: None.
Guidelines:
22

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All existing assets such as code, data, and models used in our research are properly credited to their respective creators owners. We have explicitly mentioned the licenses and terms of use for each asset in our documentation. Additionally, we have ensured that all terms of assets with restrictive licenses. This practice supports ethical research standards and ensures legal compliance in the use and distribution of third-party resources.
strictly adhered to, including obtaining necessary permissions for
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: We are currently organizing the code and plan to release it as open-source in the future.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/codeemoodll as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
23

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification:
We have included data collected from the survey in the supplementary
materials.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: None.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "EvMXtiasbs", "reviewer": "Reviewer_gwDu", "summary": "This paper aims to alleviate the multi-Janus problem in SDS-based 3D generation tasks. Inspired by the spatial structure inference capability of large multimodality models (LMMs), they propose a novel automatic negative prompt strategy. Specially, they input rendered images and 3D-aware inquiry prompts to LMM to obtain negative prompts. To keep the semantic consistency, they regenerate rendered image guided by a negative prompt and calculate regularization loss between the originally rendered image and the regenerated one.", "questions": "- How to select a focal view in the Multi-view Appearance Alignment module is unclear. Is the focal view chosen randomly? Or the front view as a focal view always be chosen for each iteration? \n- Why the shown examples are low quality? Can the proposed method work for high-quality 3D generation?\n- Why only provide one 360-degree example? Why not consider providing complete results in an appendix?", "limitations": "Based on the provided examples, it seems like this work just works for very low-quality 3D generation and simple prompts.\n\n---------------------\n\nThis work presents the integration of VLM into text-to-3D generation, framing the Janus problem as hallucination detection. It automatically produces adaptive negative prompts via VLM. While the effect is constrained, this approach, devoid of training and 3D priors, offers insights into leveraging LLM in 3D generation. The primary constraint lies in the additional time overhead.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- A novel and interesting strategy to introduce LMM automatically generating negative prompts to alleviate Janus issue.\n    - Different from direct use generated negative prompts, this work introduces a prompt-enhanced re-consistency scheme that regenerates render image guided by negative prompt and calculates an MSE loss to help SDS optimization.\n- Multi-view Appearance Alignment module calculates self-attention between the focal view of noise x and others to introduce appearance consistency across different views.\n- The proposed method is applied to various baselines, including different 3D representations (NeRF and Gaussians) and tasks (text-to-3D and image-to-3D), to show robustness and generalization ability.", "weaknesses": "- Lack of detaild of implementation and ablation study. I doubt the reproduction of this work.\n    - For Multi-modal Hallucination Detection, is P_I  always the same as the prompts in Figure 4? All rendered images are inputted to LMM?   Which LMM is used in the final experiments, given GPT-4V and LLaVA in Figure 4? \n    - For Prompt-Enhanced Re-consistency, how to balance loss_SDS and loss_CG? Can you provide the loss curve of them? The author mentions that this module only works when the rendered images exhibit complete semantic structures. So what is the proportion in the training does it work on average? Why Dψ can produce \"None\"? I cannot find the corresponding instruction in Figure4's prompts.\n    - For the Ablation Study, does w/o B mean that there are no adaptive negative prompts and a general negative prompt is used in the Prompt-Enhanced Re-consistency module to calculate L_CG? So what is the general negative prompt in w/o B ablation study? In w/o C study, does this mean that the generated negative prompt is directly used in equation (3) and no L_CG?\n\n- Lack of some necessary comparison.\n    - Please show the training time compared with multiple baselines. In my aspect, inference LMM for each iteration may introduce more training time.\n    - The ablation study and analysis are not complete. It's interesting to show that replacing P_E^− with a general negative prompt. And why not introduce L_CG all the time? What is the impact of P_I for Hallucination Detection?\n    - Magic3D is very similar with DreamFusion-IF except for the additional finetune phrase. Actually, I think finetune stage has less impact for geometric structures. I suggest authors replace Magic3D with ProlificDreamer. Prolificdreamer has a serious Janus problem but it produces clearer texture which is beneficial for Multi-modal Hallucination Detection.\n    - Lack of comparison with mentioned prompt engineering methods: \"prep-neg\" and \"debiasing scores and prompts\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your recognition of our method's innovation and applicability. Here are our responses to your comments.\n\n**W1. \"I doubt the reproduction of this work.\"**\n\nThanks for pointing this out. We'll share our code upon publication to help with reproducing our work.\n\n**W1.1-1. \"Is P_I always the same as the prompts in Figure 4?\"**\n\nA1.1-1. We used different P_I than in Fig.4. The primary purpose of Fig. 4 is to use a case study to demonstrate how LMMs can infer structural consistency and respond in specific formats. To highlight this capability, we employed two dialogues. In practice, we used a single interaction to query the LMM to achieve faster runtime. Please refer to Q4.1 in the Global Review for the specific setting of P_I.\n\n**W1.1-2. \"All rendered images are inputted to LMM?\"**\n\nA1.1-2. For efficiency, we input only one of the four rendered images into the LMM, selecting it randomly to avoid over-intervening in a specific view.\n\n **W1.1-3. \"Which LMM is used?\"**\n \nA1.1-3. For LMM, we chose the locally deployed LLaVA, using the version llava-v1.6-34b.\n\n**W1.2-1. \"How to balance loss_SDS and loss_CG? \"**\n\nA1.2-1. Thanks for your pointing this issue. There's actually a typo in Eq.8. To balance L_SDS and L_CG at the same order of magnitude, we set w=0.1  for L_SDS. We'll correct this in the further version. Here's the correct formula:\n\n$$\n\\mathcal{L}(\\theta) = \\mathcal{L} _ {SDS} + w\\mathcal{L} _ {CG}, \\text{\\quad  if } D _ \\psi(x, P_I) \\text{ is not None}\n$$\nAnd the other case is:\n\n$$\n\\mathcal{L}(\\theta) = \\mathcal{L} _ {SDS}, \\text{\\quad  if } D _ \\psi(x, P_I) \\text{ is None}\n$$\n\n**W1.2-2. \"Can you provide the loss curve of them?\"**\n\nA1.2-2. Please refer to Q3 of the Global Review for the detail.\n\n**W1.2-3. \"What is the proportion does L_CG work?\"**\n\nA1.2-3. We provided quantitative ablation results in Q2 of the Global Review. The analysis indicates that adding L_CG increases the average CLIP-Score from 24.32 to 27.03, contributing significantly to the overall improvement. Additionally, the CLIP-Score curve in Fig.4 (PDF) further demonstrates the effectiveness of L_CG.\n\n**W1.2-4. \"Why Dψ can produce 'None'?\"**\n\nA1.2-4. When Dψ receives low-quality or poorly angled images, LMMs may fail to generate the expected negative prompt. The output then cannot be recognized by our regex and returns None. This process is visualized in Fig.5 in the PDF.\n\n**W1.3. \"About general negative prompt in w/o B and w/o C.\"**\n\nA1.3. w/o B means that P_E^- is not involved in L_CG, while w/o C indicates that P_E^- is involved in L_SDS during the ablation of L_CG. Our method acts as a universal enhancement for 3D generation, considering the common use of general negative prompts in baseline methods[1]. The general negative prompt used in w/o B can be found in Q4.2 of the Global Review.\n\n**W2.1. \"About the training time.\"**\n\nA2.1. We detailed the training time in Q1 of the Global Review. LMM does introduce additional training time, but we believe this is acceptable given the significant improvement in performance.\n\n**W2.2-1. \"About the ablation.\"**\n\nA2.2-1. More ablation study and analysis are detailed in Q2 of the Global Review. \n\n**W2.2-2. \"Replacing P_E^− with a general negative prompt?\"**\n\nA2.2-2. As discussed in A1.3, we adopt a generative negative prompt (which is used by baseline methods) in all the ablation study.  Specifically, w/o B removes  P_E^− and kept general negative prompt.\n\n**W2.2-3. \"Introduce L_CG all the time?\"**\n\nA2.2-3. L_CG is designed to constrain the view consistency, but in the early stage, when image quality is too low and lacks sufficient structural information, constraining with L_CG is less meaningful. Additionally, applying L_CG only in the later stages can also reduce training overhead.\n\n**W2.2-4. \"What is the impact of P_I for Hallucination Detection?\"**\n\nA2.2-4. The two dialogues in Fig.4 correspond to the two roles of P_I:\n\n1. Activating the LMM's ability to infer 3D view consistency. By including query necessary information about inconsistencies in P_I, the LMM is prompted to identify specific issues in the image.\n2. Standardizing the LMM's output format. By providing a one-shot example, the LMM's output is guided to match the negative prompt format, making it easier for our designed regular expression to recognize.\n\n**W2.3. \"Replace Magic3D with ProlificDreamer.\"**\n\nA2.3. Thanks for suggestion. We added experiments on ProlificDreamer. The results in Fig.2 (PDF) show our method can improve its consistency.  We will include more detailed discussion about  ProlificDreamer in the final version.\n\n**W2.4. \"Comparison with 'perp-neg' and 'debiasing scores and prompts'.\"**\n\nA2.4. Thanks for the suggestion. We added experiments on Perp-Neg and Debiasing Scores and Prompts. Results in Fig. 3 (PDF) show that our method improves consistency better than both approaches, as reflected in the quantitative and qualitative outcomes.\n\n**Q1. \"How to select a focal view?\"**\n\nWe use Fovy, the camera's vertical view, as our selection standard. The first view with Fovy exceeding 120% of each baseline's default becomes our focal view, enabling a broader shot capturing more object details.\n\n**Q2. \"Why the shown examples are low quality?\"**\n\nA-Q2. Our method aims to boost view consistency in 3D generation, with quality hinging on the baseline models. We've tested this on high-quality models like GaussianDreamer,  DreamGaussian, and ProlificDreamer.  The results confirm our method's versatility, elevating both lower and higher-quality 3D generations.\n\n**Q3. \"Provide 360-degree example?\"**\n\nA-Q3. Thanks for suggestion. We have displayed the 360-degree view of the results in Fig.1 (PDF).\n\n---\nWe hope our response resolves your concerns. Due to character limitations, we've provided a concise answer to your questions. We look forward to further communication with you!\n\n\n[1] Yuan-Chen Guo, et al. Threestudio. https://github.com/threestudio-project/threestudio, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iYuLEGc2oB", "reviewer": "Reviewer_bBV6", "summary": "Hallo3D presents a tuning-free method, empowering 3D content generation frameworks via multi-modal LLM. The paper aims to solve the hallucination and inconsistent problems in SDS-based 3D content generation pipelines. With the design of multi-view appearance alignment and enhanced negative prompts, Hallo3D is able to improve the quality of generated 3D objects. The effectiveness of the proposed method was demonstrated through experimental verification using multiple frameworks.", "questions": "1. What does consistency refer to in the paper? The SDS-based method is optimized to generate a complete 3D content, not like directly using 2D diffusion models to generate multi-views. Does inconsistency refer to Janus problem?\n2. Why need an additional $L_{cg}$ loss? Why not add negative prompts directly to the 2D diffusion models in the top left of Figure 3 for $L_{SDS}$ loss?\n3. Is it necessary to first use the baseline methods to obtain a raw 3D content before cooperating with Multi-modal LLM for subsequent operations.", "limitations": "The paper has addressed the potential negative societal impact: the potential misuse of advanced 3D generation technologies could undermine social trust and compromise information integrity. And there are no other obvious potential social impacts.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1. The paper integrates various SDS-based 3D generation frameworks, including text-to-3D and image-to-3D frameworks. And the effectiveness of the proposed method on these frameworks has been demonstrated through qualitative results.\n2. Introducing multi-modal LLM to optimize the generation of diffusion is effective, and the output of multi-modal LLM can be directly applied to the diffusion through negative prompts. This design is concise and clever.\n3. The paper also performs some quantitative results to validate the effectiveness of the proposed method.\n4. The design is flexible and adaptable to the rapid advancements in SDS-based methods.", "weaknesses": "1. The method description is not clear enough.\n- In lines 148-150, the paper claims defining a new denoising strategy, aiming to enhance the appearance consistency. But there's no further details about how to apply the new denoising strategy with appearance attention.\n- Lines 184-186 provide some description, but lack a detailed explanation of how to utilize appearance attention, which is an important contribution claimed by the paper.\n2. The ablation study lacks quantitative analysis.\n- Figure 7 presents qualitative results; however, the lack of quantitative analysis makes these results less convincing.\n3. Lack of enough quantitative analysis showcasing performance improvements on image-to-3d frameworks.\n- Image-to-3D models can use images rendered from original 3D objects as input and ground truth, and then use the metrics of 2D images for measurement, such as SSIM, PSNR, etc. The GSO dataset or Objaverse dataset can be used.\n4. The impact of methods on training time is not introduced. \n- The paper refine 3D contents generated by baseline models, but the impact of training time is not addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your recognition of our method's innovation and effectiveness. Here are our responses to your comments.\n\n**W1. \"In lines 148-150, the paper claims defining a new denoising strategy, but there's no further details. Lines 184-186 provide some description, but lack a detailed explanation of how to utilize appearance attention.\"**\n\nA1. Thank you for pointing this out. We’d like to clarify that the \"new denoising strategy\" mentioned in lines 148-150 refers to the introduced strategy in Sec.3.2.  \n\nIn the denoising strategy $\\tilde{\\epsilon}_\\phi(\\cdot)$, $\\rm{AAttn}(\\cdot)$ in Eq.4 functions as cross-attention. The key and value are derived from the focal view, with each of the four views calculating a distinct query. This ensures that each view aligns its features with the focal view, achieving consistent appearance, and we will make it clearer in further versions.\n\n**W2. \"The ablation study lacks quantitative analysis.\"**\n\nA2. We have added quantitative analysis in Q2 of the Global Review, and the quantitative results further verify the effectiveness of our method. \n\n**W3. \"Lack of enough quantitative analysis showcasing performance improvements on image-to-3d frameworks, such as SSIM, PSNR, etc. The GSO dataset or Objaverse dataset can be used.\"**\n\nA3. Thank you for the suggestion. We followed the experimental setup in [1], randomly selecting 30 objects from both the GSO and Objaverse datasets, totaling 60 objects. To ensure variety, we replaced objects with overly simple structures. We rendered their frontal views at 256x256 resolution for input into our method. Performance was assessed using Chamfer Distance (CD) and Volume IoU (Vol. IoU) for geometric quality, and PSNR, SSIM, and LPIPS for visual quality. The results are shown in the table below.\n\n| Metrics            | DreamGaussian | Hallo3D | Zero-1-to-3 | Hallo3D |\n| :----------------- | :-----------: | :-----: | :---------: | :-----: |\n| CD$\\downarrow$     |    0.0185     | 0.0171  |   0.0370    | 0.0283  |\n| Vol. IoU$\\uparrow$ |    0.5861     | 0.6099  |   0.4824    | 0.5602  |\n| PSNR$\\uparrow$     |    16.502     | 16.518  |   13.433    | 14.930  |\n| SSIM$\\uparrow$     |    0.8543     | 0.8793  |   0.7210    | 0.7527  |\n| LPIPS$\\downarrow$  |    0.2025     | 0.1726  |   0.3926    | 0.3328  |\n\nThe experimental results show that our method outperforms the baseline across all metrics, improving both geometry and textures. This further confirms the broad applicability of our approach, enhancing both text-to-3D and image-to-3D tasks.\n\n**W4. \"The impact of methods on training time is not introduced.\"**\n\nA4. We measured the time overhead for both 3DGS-based and NeRF-based baselines, and the results show minimal additional time costs, as detailed in Q1 of the Global Review.\n\n**Q1. \"What does consistency refer to in the paper? The SDS-based method is optimized to generate a complete 3D content, not like directly using 2D diffusion models to generate multi-views. Does inconsistency refer to Janus problem?\"**\n\nA-Q1. Yes, inconsistency refers to the Janus problem. The term consistency refers to a 3D object maintaining its normal visual structure and appearance across multiple views.\n\n**Q2. \"Why need an additional $\\mathcal{L} _ {\\rm{CG}}$ loss? Why not add negative prompts directly to the 2D diffusion models in the top left of Figure 3 for $\\mathcal{L} _ {\\rm{SDS}}$ loss?\"**\n\nA-Q2. Thanks for your suggestion. Indeed, adding negative prompts directly to the 2D diffusion models is what we do in the w/o C configuration. We also introduced an ablation version, excluding the $P _ E ^ -$ output by LMM from the $\\mathcal{L} _ {\\rm{CG}}$ calculation, to isolate $P _ E ^ -$ impact. The outcomes from both ablation versions thoroughly evaluate the role of $\\mathcal{L} _ {\\rm{CG}}$, confirming its necessity.\n\n**Q3. \"Is it necessary to first use the baseline methods to obtain a raw 3D content before cooperating with Multi-modal LLM for subsequent operations?\"**\n\nA-Q3. Yes. We initially employ baseline methods to generate a raw 3D content. The method we proposed is designed to perform hallucination detection and mitigation on the results produced by the 3D baseline. Therefore, it's necessary to have 3D content prior to the detection and correction process.\n\n---\nWe hope our response is helpful in addressing your concerns. We look forward to continuing our communication with you.\n\n\n[1] Long X, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In *CVPR*, 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CpFvzkrkvf", "reviewer": "Reviewer_isRS", "summary": "The authors point out that generating 3D content with Score Distillation Sampling leads to multi-view inconsistencies. To address this challenge, they propose a novel, tuning-free method called Hallo3D. Specifically, they utilize large multi-modal models (e.g., GPT-4V) to detect and correct these inconsistencies during optimization. Notably, their method can be implemented in a plug-and-play manner with several text-to-3D generation baselines, achieving satisfying visual results.", "questions": "1.\tSection 3.4 is not clear enough. From lines 180-183, the authors seem to use the “DDIM Inversion” technique to regenerate an image without the so-called hallucination. However, generating such an image with CFG without damaging the original structure of the image is not an easy task (see Null-text Inversion). More illustration of this part should be provided.\n\n2.\tFrom Figures 3 and 4, it appears that you only feed a single image into the LMM for hallucination detection. However, a less serious multi-view inconsistency (e.g., a rabbit with three eyes) can only be observed from certain view directions. How can the effectiveness of Hallo3D be proven in this context?\n\n3.\tThe generation-detection-correction paradigm introduced in this paper appears to require frequent use of the LMM. Given that generating 3D content with SDS optimization typically requires around 5,000-10,000 iterations, what is the overall time cost of your algorithm?\n\nI’m willing to give this paper a higher rating if these three questions are well addressed!", "limitations": "Yes, the authors discuss the limitations of the proposed Hallo3D clearly at the end of the \"Conclusion and Discussion\" section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe idea of using large multi-modal models to implement a generation-detection-correction paradigm for text-to-3D generation is both novel and interesting. Moreover, the experimental results demonstrate that this approach works quite well.\n\n2.\tThe entire \"Methodology\" section is presented quite well. The preliminaries are written accurately, and the subsequent subsections clearly introduce the overall motivation and design.\n\n3.\tThe experiments are also solidly conducted, including both qualitative and quantitative comparisons on text- and image-driven 3D generation. The ablation studies are sensible and include user studies as well.", "weaknesses": "The overall quality of the paper is quite good, but some problems still exist:\n\n1.\tIn Figure 2, the \"Illustration of the Janus problem\" seems to understate the performance of current baselines. In the first two rows of the figure, eight faces of the dog are visible around the sculpture, which does not align with the experimental results from methods like DreamFusion. This is acceptable as an illustration, but it would be better to be more rigorous.\n\n2.\tEquation 6 in line 183 does not seem to be derivable from Equation 12 in the original DDIM paper. Please check this carefully.\n\n3.\tThe ablation study section lacks clarity. Rows 2-4 do not show a clear visual difference, and the text in Section 4.4 does not provide an easy-to-follow illustration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your feedback. We appreciate your recognition of our method's innovation and effectiveness. Here are our responses to your comments.\n\n**W1. \"Fig.2 seems to understate the performance of current baselines.\"**\n\nA1. We appreciate your attention to the details in Fig.2. Actually, this illustration is based on the Score Jacobian Chain[1] (a text-to-3D baseline method). The figure's primary purpose is to visually explain the Janus problem by highlighting an example where the multi-faces phenomenon is particularly pronounced. To better illustrate the Janus Problem, we included the expected result under normal conditions as achieved by our method, focusing on the problem's severity rather than comparing different methods. We understand the importance of precision and will consider adding a note in the figure caption to ensure this context is clearer in future versions.\n\n**W2. \"Equation 6 in line 183 does not seem to be derivable from Equation 12 in the DDIM paper.\"**\n\nA2. Thank you for your thorough review and for pointing out this issue. After carefully revisiting our derivation, we confirmed that there is indeed a typo. We have corrected the derivation by setting $\\sigma =0$ in Eq.12 of the DDIM paper. The correct form is as follows: \n\n$$\n\\hat{\\mathbf{x}} _ {t-1} = \\sqrt{\\frac{\\alpha _ {t-1}} {\\alpha _ t}} \\hat{\\mathbf{x}} _ t +  (\\sqrt{1-\\alpha _ {t-1}}-\\sqrt{\\frac{\\alpha _ {t-1} (1-\\alpha _ t)} {\\alpha _ t}}) \\tilde{\\epsilon} _ {\\phi} (\\hat{\\mathbf{x}} _ t, t, P^{+}, P^{-} _ E)\n$$\n\nYour feedback has been instrumental in ensuring the precision of the paper. We have revised the formula accordingly in the updated version to enhance its accuracy.\n\n**W3. \"Ablation study section lacks clarity. Rows 2-4 do not show a clear visual difference.\"**\n\nA3. To demonstrate the necessity of each module in our method, we conducted quantitative ablation experiments, including one that ablates both C and $P _ E ^ -$, i.e., \"w/o C & $P _ E ^ -$\" to validate the importance of $\\mathcal{L} _ {\\rm{CG}}$. The results in Q1 of the Global Review confirm that each module in Hallo3D enhances performance.\n\nIn Sec.4, we focus on how module A primarily affects color and texture, while module B and module C enhance cross-view consistency. Specifically:\n\n- Row 2, w/o A: The lion is darker than Row1, and the deer shows a blurry halo and unreasonable color shift.\n- Row 3, w/o B: The lion's head is derormed in the first and third columns, and the deer misses its head.\n- Row 4, w/o C: The \"second face\" appears on the lion's back in the third column, and the right deer's image shows a clear Janus Problem, with multiple legs and a distorted body visible in the second and fourth columns.\n\n**Q1. \"The authors use the “DDIM Inversion” to regenerate an image without hallucination. However, generating such an image with CFG without damaging the original structure is not easy.\"**\n\nA-Q1. That's an excellent question. We agree with your point that CFG-based techniques often struggle to maintain structure in regenerated images in image editing or reconstruction. However, in generative tasks, we use DDIM Inversion to regenerate the image as a guidance for constraining consistency, not as the final output. Our primary focus is on the final 3D assets, with the images obtained using rendering techniques that are independent of the diffusion model.\n\nSpecifically, generative tasks involve repeated optimization, where each iteration may cause some structural loss, but these intermediary images are not visible. Therefore, as long as the final result is consistent and high-quality, the loss of information is relatively less important. This contrasts with image editing or reconstruction, where only a single image is produced, making structural details critical.\n\nOur experiments confirm this. As shown in Figures 5 and 6, even with structural information loss during the DDIM Inversion process, we achieved consistent results.\n\n**Q2. \"It appears that you only feed a single image into the LMM. However, a less serious inconsistency can only be observed from certain view directions. How can the effectiveness of Hallo3D be proven?\"**\n\nA-Q2. Yes, we only input one image at a time to reduce the inference cost of LMM. However, since the camera pose for each rendered image is random, multiple iterations ensure that all sides of the 3D object are addressed, effectively avoiding inconsistencies in specific views.\n\nOur experiments also show that this method effectively addresses even minor multi-view inconsistencies. For instance, in the second image of the fifth row in Fig.5, the car's right rear wheel appears doubled, but our method corrected this. Similarly, in the second example in Fig.6, our method improves subtle Janus problems on the figure's body.\n\n**Q3. \"The method appears to require frequent use of the LMM. Given that generating 3D requires 5000-10000 iterations, what‘s the time cost?\"**\n\nA-Q3. We measured the time overhead for both 3DGS-based and NeRF-based baselines, finding minimal additional costs, as detailed in Q1 of the Global Review. While LMM inference can add time, our model uses single-stage dialogue and queries at intervals during later training to improve efficiency. Additionally, due to the Multi-View Appearance Alignment, our method processes four images at a time with batch=4, reducing the number of SDS iterations.\n\n---\nWe hope that our response addresses your concerns sincerely. Looking forward to further communication with you!\n\n[1] Wang H,et al. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In *CVPR*, 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive feedback and valuable insights, which have significantly contributed to the improvement of our research. We are grateful for your thoughtful suggestions. 

Our work has been recognized for ***the innovative introduction of the LMMs*** (gwDu, bBV6, isRS), ***the broad applicability*** (gwDu, bBV6), ***the significant performance improvements*** (bBV6, isRS), ***the solid comparative experiments*** (isRS), and ***the clear method description*** (isRS).

**Q1. "The time consumption introduced by Hallo3D."**

**A common concern is the additional time consumption introduced by Hallo3D**, particularly regarding the extra inference time for LMMs. To address this concern, we recorded the runtime using [1] (*based on 3DGS with fewer iterations and faster speed*) and [2] (*based on NeRF with more iterations and slower speed*) as baselines, on NVIDIA V100.  

**Specifically, we begin calculating $\mathcal{L} _ {\rm{CG}}$ later in the training process and do so every 4 iterations in our experiments.** This approach aligns with the statement in Sec 3.4 that *"this module only works when the rendered images exhibit complete semantic structures."* The rationale is twofold. First, in the early stages of training, the 3D assets are relatively disorganized and lack clear semantic structures, making it difficult for LMMs to reason accurately. Therefore, we introduce $\mathcal{L} _ {\rm{CG}}$ later in the training. Second, we empirically found that calculating $\mathcal{L} _ {\rm{CG}}$ every 4 iterations does not impact performance, so we adopted this approach to reduce training time. The results are shown in the table below.

| Baseline            | Iteration | $\mathcal{L} _ {\rm{CG}}$ Start Rounds | Original Time | Extra Time | Total Time |
| :------------------ | :-------: | :------------------------------------: | :-----------: | :--------: | :--------: |
| GaussianDreamer [1] |   1200    |                  1000                  |    ~28 min    |  ~10 min   |  ~38 min   |
| DreamFusion[2]      |   2500    |                  2200                  |    ~51 min    |  ~15 min   |  ~66 min   |

Combining the table above with Fig.4 in the PDF, it can be seen that **Hallo3D efficiently enhances cross-view consistency with acceptable time costs**. 

**Q2. "The quantitative ablation experiment and analysis."**

**We conducted quantitative ablation experiment** to further demonstrate the necessity of each module. Identical to the setup in paper, module A represents Multi-view Appearance Alignment, module B stands for Multi-modal Hallucination Detection, and module C denotes Prompt-Enhanced Re-Consistency. Additionally, we included experiments for the scenario "w/o C & $P_E^-$", where without providing $P ^ {-} _ E$ for $\mathcal{L} _ {\rm{SDS}}$ when w/o C. The results are as follows.

| Metrics                   | Hallo3D | w/o A | w/o B | w/o C | w/o C & $P ^ {-} _ E$ | Baseline |
| ------------------------- | :-----: | :---: | :---: | :---: | :-------------------: | :------: |
| CLIP-Score B/32$\uparrow$ |  24.25  | 23.98 | 23.65 | 22.46 |         22.23         |  21.27   |
| CLIP-Score B/16$\uparrow$ |  26.83  | 25.88 | 25.10 | 23.59 |         23.23         |  22.67   |
| CLIP-Score L/14$\uparrow$ |  30.00  | 29.36 | 28.71 | 26.92 |         25.58         |  23.71   |

The better performance of "w/o C" compared to "w/o C & $P ^ {-} _ E$" also supports the necessity of introducing $\mathcal{L} _ {\rm{CG}}$.


**Q3. "The loss curve between $\mathcal{L} _ {\rm{SDS}}$ and $\mathcal{L} _ {\rm{CG}}$."**

The curves are shown in Fig.4 of the PDF. Additionally, we've also included the CLIP-Score for the full model and the ablated version without $\mathcal{L} _ {\rm{CG}}$. It can be observed that $\mathcal{L} _ {\rm{CG}}$ decreases with the number of iterations and significantly improves the CLIP-Score, whereas the CLIP-Score without $\mathcal{L} _ {\rm{CG}}$ shows only a minor improvement, demonstrating the effectiveness of $\mathcal{L} _ {\rm{CG}}$.

**Q4. "The Prompt settings."**

**Q4.1. "The $P_I$ setting in LMM."**

> *"You are a master of 3D generation, and please refer to the 'Prompt' and 'Negative Prompt' below to identify the inconsistency in the image I provided you with, with body shape, perspective, texture, and so on.*
> 
> *Reference:*
> 
> *'Prompt': '3d render of xx, front view, standing, high quality, 4K',* 
> 
> *'Negative Prompt': 'multi-head, unnatural lighting, smooth appearance, distorted color, long neck,  two-nosed, extra limbs' ".*

**Q4.2. "The general negative prompt setting in LMM."**

> "unnatural colors, poor lighting, low quality, artifacts, smooth texture".

---

We believe that Hallo3D can be a valuable supplement to the NeurIPS community, particularly with the enhancements made based on the reviewers' feedback, which have helped us better convey the effectiveness of our method.

Thank you! The Authors.


[1] Taoran Yi, et al. GaussianDreamer: Fast Generation from Text to 3D Gaussians by Bridging 2D and 3D Diffusion Models. In *CVPR*, 2024.

[2] Ben Poole, et al. DreamFusion: Text-to-3D using 2D Diffusion. In *ICLR*, 2022.

Meta-review: In the initial reviews, all reviewers acknowledged the innovative introduction of the LMMs, the significant performance improvements, and the solid comparative experiments. However, some reviewers pointed out the missing quantitative ablation experiment and analysis, and confused term such as consistency in the paper. During the rebuttal phase, the authors addressed these concerns with additional experiments, leading to positive feedback from all reviewers. AC has also carefully reviewed the paper, the reviews, and the rebuttal, and concurs with the reviewers' opinions. Therefore, AC recommends acceptance. It would be highly beneficial to include the additional results from the rebuttal phase in the final version.
Decision: Accept (poster)