DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Authors: Zhenglin Zhou
Xiaobo Xia
Fan Ma
Hehe Fan
Yi Yang
Tat-Seng Chua
Venue: ICML 2025 poster (2025)
Keywords: 3D Generation
Human Preference
Direct Preference Optimization
Primary area: applications->computer_vision
TL;DR: DreamDPO is a framework that incorporates human preferences into text-to-3D generation, achieving state-of-the-art results with improved quality and fine-grained controllability.

Abstract: Text-to-3D generation automates 3D content creation from textual descriptions, which offers transformative potential across various fields. However, existing methods often struggle to align generated content with human preferences, limiting their applicability and flexibility. To address these limitations, in this paper, we propose DreamDPO, an optimization-based framework that integrates human preferences into the 3D generation process, through direct preference optimization. Practically, DreamDPO first constructs pairwise examples, then validates their alignment with human preferences using reward or large multimodal models, and lastly optimizes the 3D representation with a preference-driven loss function. By leveraging relative preferences, DreamDPO reduces reliance on precise quality evaluations while enabling fine-grained controllability through preference-guided optimization. Experiments demonstrate that DreamDPO achieves state-of-the-art results, and provides higher-quality and more controllable 3D content compared to existing methods. The code and models will be open-sourced.

Xiaobo Xia DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Zhenglin Zhou 12
Xiaobo Xia
Fan Ma
Hehe Fan? Yi Yang 12 Tat-Seng Chua
Abstract
create high-quality 3D content, which requires substantial time and effort, even for professionals. In response, text- 10-3D gemeration has emerged as a solution by automating 3D generation from textual descriptions, which archives remarkable advancements in the field (Poole et al., 2022; Wang al., 2022; Yu al., 2023; Wang et al., 2024; Shi et al., 2023; Katzir et al., 2023; Chung al., 2024; Wu et al., 2024b). Nevertheless, some researchers (Xie et al., 2024; Ye et al., 2025) emphasize that 3D content generated by ex- isting methods often fails to align with human preferences fully, highlighting the need for continued refinement and innovation these methods.
Text-to-3D generation automates 3D content cre- ation from textual descriptions, which offers trans- formative potential across various fields. How- ever, existing methods often struggle to align generated content with human preferences, lim- iting their applicability and flexibility. To ad- dress these limitations, in this paper, we pro- pose DPOamDPO, an optimization-bae based frame- work that integrates human preferences into the 3D generation process, through direct preference optimization. Practically, DreamDPO first con- structs pairwise examples, then compare their alignment with human preferences using reward or large multimodal models, and lastly opti- mizes the 3D representation with a preference- driven loss function. By leveraging pairwise comparison to reflect preferences, DreamDPO reduces reliance precise pointwise quality eval- uations while enabling fine-grained controllabil- ity through preference-guide optimization. Ex- periments demonstrate that DreamDPO achieves competitive results, and provides higher-quality and more controllable 3D content compared to existing methods. Code publicly available at: https//ggthbbb.omm///eeennnn////
Previous work (Ye et al., 2025) has leveraged reward models to integrate human preferences into the generation process, leading to enhanced 3D generation outcomes. The core idea is to regularize the generated 3D content to achieve high pointwise score from the reward model. Despite these improved results, several issues remain to be addressed. First, heavily depends on the reward model's ability to accurately evaluate the pointwise quality of generated con- tent, which places significant demands on the reward model. Second, since the reward model can only provide quality- relevant scores, it lacks the flexibility to enable controlla- bility from other perspectives. The issues reduce the appli- cability and adaptability of the current method. This falls short of meeting diverse requirements or providing broader control in 3D generation, which is never our desideratum.
To relieve the issues of prior work and better align 3D gen- eration with human preferences, propose DrcamDPO. Essentially, DreamDPO is an optimization- for text-to-3D generation. It achieves alignment through direct preference optimization, leveraging preferences de- rived from either reward models or large multimodal models. Specifically, DreamDPO operates by initializing 3D rep- resentation (Mildenhall et al., 2020; Kerbl et al., 2023) and optimizing it through a three-step iterative process. First, puirwise examples are constructed on-the-fly by applying different Gaussian noise. Second, reward model or a large multimodal model ranks these examples based on their matching with the input text prompt or specific instruc- tions, which matches human preferences about the pairwise
1. Introduction
method
3D content generation is pivotal in driving innovation across diverse fields, including product design, medical imaging, scientific visualization, and the rapidly growing domains of virtual and augmented reality (Li et al., 2023a). De- spite its extensive applications, it remains challenging to
Staate Key Laboratory of Brain-machine Intelligence, Zhe- jiang University. China *ReLER, CCAI, Zhejiang University, China 'National University of Singapore, Singapore. Corre- spondence to: Yi Yang grangyics@2ju.edu.coo <xbx Exbx@nus.cde exbx@@ns.cdu..g>..
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

DreamDPO; Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
examples'. Finally, a reward loss is computed from the pairwise preferences, which guides the update of the 3D representation. By incorporating human preferences into the optimization loop, DreamDPO generates 3D assets that achieve superior alignment with textual inputs, along with enhanced texture and geometry quality.
DPeamDPO outperforms 13 state-of-the-art methods, achieving the best quantitative performance across two key metrics while delivering highly impressive and controllable qualitative results.
The rest of this paper is organized as follows. 2 introduces some background knowledge relevant to this work. $ 3 presents the technical details of our proposed DreamDPO. Experimental results are analyzed and discussed in $ 4. Conclusions are given in $ 5.
DreamDPO can be justified as follows. While absolute quality evaluation inherently provides a ranking on pair- wise examples, ranking requires only the scores to dis- tinguish relative preferences, but need not be perfectly ac- curate (c.f., (Zhang et al., 2024d)). DreamDPO takes ad- vantage of this distinction, and changes the previous score- guided optimization (Ye et al., 2025) to crrferencc-guudder optimization. Therefore, it lowers the demand for precise scoring and requires only distinguishable scores. Addition- ally, DreamDPO can make use of the preferences provided by large multimodal models. By constructing preferred and less preferred examples based on specific instructions about the attributes of generation content (e.g., the object num- ber and motion), it directs the optimization process to align more closely with the desired outcomes. This strategy en- hances adherence instructions and introduces lineiggained controllability, meeting diverse requirements effectively. Moreover, we conduct series of experiments to justify our claims. Empirical results demonstrate that our method flexi- bly accommodates either reward models or large multimodal models, which enables the generation of nigher-quality and more controllable 3D content.
2. Preliminaries
Text-to-3D generation aims to create high-quality 3D as- sets aligned with a given text prompt y. The pipeline typically distills knowledge from a parametrized diffusion model eø (Rombach et al., 2022a; Shi al., 2023) into learnable 3D representation with parameters 0 O (e.g., NeRF (Mildenhall et al., 2020), DMTet (Shen et al., 2021), and 3DGS (Kerbl et al., 2023)), where Q is the space of 0 with the Euclidean metric. Score distillation sam- pling (SDS) (Poole et al., 2022) is used to guide the dis- tillation process.
Diffusion models. The diffusion model has been widely used in generative tasks (Sohl-Dickstein et al., 2015; Song et al., 2022; Bao et al.. 2022; Peebles & Xie, 2023). Gen- erally, it involves forward process to gradually add Gaus- sian noise to data points and a reverse process to trans- form Gaussian noise into data points from target distribu- tion Pdata- The reverse process starts from an initial noise ZT ~ (O,I). At each diffusion step t, the model refines noisy data z, into cleaner one Z until finally producing x Pdata- Therefore, the transitions p( p2,,11 124) can be learned effectively by the diffusion model.
Before delving into details, we clearly emphasize our con- tribution follows.
Conceptually, we propose DreamDPO that shifts the paradigm of text-to-3D generation by reducing depen- dence on precise absolute quality evaluation. Instead, it leverages distinguishable relative preferences and integrates large multimodal models, which achieves remarkable alignment with human preferences while enabling fine- fine-grained controllability.
Score distillation sampling (SDS). SDS was proposed in DreamFusion (Poole et al., 2022), and has been widely studied (Wang et al., 2024; Yu et al., 2023; Katzir al., 2023; Chung et al., 2024; Wu et al., 2024b; Zhuo et al., 2024; Ye et al., 2025). Technically, for rendered image from 3D representation, random noise c is added at timestep t. A pre-trained diffusion model predicts this noise. The SDS loss is computed as the difference between predicted and added noise, which optimizes a set of parameters The gradient of the SDS loss with respect to is:
Technically, DrEamDPO pioneers a three-step opti- mization process, combining online pair construction, preference ranking, and ranking-driven updates. This innovative design ensures superior alignment with in- put prompts, significantly enhancing the quality and adaptability of generated 3D content.
dx a0
VoLSDS = [Er,((w())( (X.rN,
(I)
Empirically, extensive results establish DreamDPO as new benchmark for text-to-3D generation. Both quan- titative and qualitative results are thoroughly analyzed, supported by detailed discussions and ablation studies.
where x; 0;xo O,E, w(t) a weighting function, and s is pre-defined scalar of classifier-free guidance (CFG) (Ho & Salimans, 2022). The minimization of the SDS loss follows the score function of the diffusion model to move x to the text description region, ensuring the generated 3D representation aligns with the given text prompt. Note that to make a continuous and smooth presentation, we provide
'Reward models are trained using pairwise data reflecting hu- man preferences, while large multimodal models inherently aligned with these preferences (Sun et al., 2023).
2

---

DrcamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
STEPI: Pairwisc Examplc Construction
Back Propagation
glose
lose One-slep Prodiction
lose
lose
Render
Text-to-Imagee Diftusion
STEP2: Pairwise Comparison
STEP3:
Add Noise
Preference-
9
E Ewin
win
win
Optimization
3D Represeniation
y
abalone sholl, with jugged contours, hosting a miniature, tranquii Zen garden.....
Comparison by Reward Model
Comparison by Large Multimodal Modal (LMM)
The Gradieni of Reward Loss Ve V
Text Prompt
[Imaae
Omange
(Answer (Anseer
(m( (ea sy. a0
dx
Ao:No
Al:
Syap <T.
2.67 lose
LMM
Reward
Model
2.72 win
Zen garden fenture a micro bonsai?
(Question)
[Score 1]
0 lose
[Sco
Eg (w())
(x emin) (Cs (xlo00,), close)
80
dx
otherwise.
win
Figure 1. Overview of our method. DreamDPO first constructs pairwise examples, then compares their alignment with human preferences using reward or large multimodal models, and lastly optimizes the 3D presentation with a preference-driven loss function. The loss function pulls the win example x closer and pushes the 1 csse example x, away. As piecewise objective, it selectively pushes x only when the preference score gap 8eup exceeds a threshold T, preventing chaotic gradients from overly similar x
a detailed review of related work in Appendix A.
model G6 (Shi et al., 2023; Rombach et al., 2022a) and generate corresponding predictions:
3. Method
x 01E (x/ :.,
x] =
Overview. DreamDPO is an optimization- based text-to-3D generation method. It begins by initializing 3D representa- tion, egg., NeRF (Mildenhall et al., 2020). In each training it- eration, the optimization procedure involves three key steps: (1) pairwise example construction: pairwise examples are generated online by applying different Gaussian noises dur- ing the diffusion process; (2) pairwise example compari- son: reward model or large multimodal model (LMM) compares the generated examples based on their alignment with the desired text prompt: and (3) preference-guiee optimization: a piecewise reward loss calculated using the pairwise comparison, and the 3D representation is up- dated accordingly. Overall, our DreamDPO guides the op- timization process with human preferences, leading to 3D assets with improved alignment to input text and enhanced texture/geometry quality. The framework overview of our method is provided in Figure 1. A complete algorithm flow of our method can be checked in Appendix B.
at
(3)
x x D2E (x7; v.
Ot.
where x] and x are predicted xo of single step for x! and x². respectively (Song et al., 2021).
Pairwise comparison. After pair construction, at step I, we utilize rank model denoted by r(-) to compare x! and x² This yields preferred prediction x-iin and less preferred one x!ex,, where x = x and Jose x². or vice versa. It is worth noting that our DreamDPO supports both reward models (Xu et al., 2024b; Wu et al., 2023) and LMM-based AI annotators (Bai et al., 2023; Yang et al., 2023), where reward models are used as default.
Preference-guided optimization. The proposed method leverages the pairwise comparison (xę*t,, xc(oee enable efficient sampling via optimization to yield human preferred 3D assets. To achieve this, we need a differentiable loss function, where preferred images have low losses and less preferred images have high losses. To this end, inspired by (Rafailov et al., 2024; Meng et al., 2024; Wallace et al., 2024), we reformulate SimPO (Meng al., 2024) to elim- inate the need for reference model and derive a differen- tiable objective:
3.1. Algorithm Details
Pairwise example construction. Given a sampled cam- era pose, an RGB image x can be rendered from the 3D representation using renderers. Then two different Gaus- sian noise el and e are added to x at timestep [0, resulting in pairwise noisy images x! and x?:
LReward = -E, [wo(t) ( ||emin
win - E (x, win .y 1)
x = 0x + 0,6', x² = 0x:00 + o
(4)
- | cx(xx t)1)
where X = x, a and oi are hyperparameters satisfying Og ~ 1,00 R 0, z 0, oT v 1 (cf., (Sohl-Dickstein et al., 2015; Ho et al., 2020)). Afterward, we feed the pair- wise noisy images into a pre-trained text-to-image diffusion
where e win and losse denote Gaussian noise for x win and x! respectively. Intuitively, PReward encourages Es to pull x" closer and push xlose further away.
3

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Table Qualitative comparisons on 110 prompts generated by GPTEvaI3D (Wu et al., 2024a). We calculate the ImageReward (IR) (Xu et al., 2024b) for human preference evaluation, the CLIP score (Radford et al., 2021) I) for text-image alignment evaluation, and GPTEvaI3D (Wu al., 2024a) for comprehensive 3D quality evaluation. The best performance in each case shown bold.
GPTEva13D f
Method
IR t
Alignment
Plausibility
T-G Coherency.
Geo Details
Tex Details
Overall
DreamFusion (Poolc et al., 2022)
-1.51
1000.0
1000.0
1000.0
1000.0
1000.0
1000.0
DrcamGaussian (Tang et al., 2023)
-1,56
1100.6
953.6
1158.6
1126.2
1130,8
951.4
Fantasia3D (Chen al., 2023)
-1.40
1067.9
891.9
1006.0
1109.3
1027.5
933.5
Instamt3D (Li et al., 2023c)
-0.91
1200.0
1087.6
1152.7
1152.0
1181.3
1097.8
Latent-NeRF (Metzer et al., 2023)
-0.42
1222.3
1144.8
1156.7
1180.5
1160.8
1178.7
Magic3D (Lin et al., 2023)
-1.11
1152.3
1000.8
1084.4
1178.1
1084.6
961.7
Point-E E (Nichol et al., 2022)
-224
725.2
689.8
688.6
715.7
745.5
618.9
ProlificDreamer (Wang et al.. 2024)
-0,50
1261.8
1058.7
1152.0
1246.4
1180.6
1012.5
Shap-E (Jun & Nichol, 2023)
2.10
842,8
842.4
846.0
784.4
862.9
843,8
SIC (Wang ct al., 2023a)
-0.82
1130.2
995.1
1033.5
1079.9
1042.5
993.8
SyneDreamer (Liuu et al., 2023c)
-1.77
1041.2
968.8
1083.1
1064.2
1045.7
963.5
Wonder3D (Long et al., 2024)
-1.70
985.9
941.4
931.8
973.1
967.8
970.9
MVDream (Shi et al., 2023)
-0.58
1270.5
1147.5
1250.6
1324.9
1255.5
1097.7
DreamDPO (ours)
-0.35
1298.9
1171.9
1276.4
1373.2
1296.9
1203.1
Following (Poole et al., 2022), we consider the gradient of CRwadd and omit the U-Net Jacobian term for effective opti- mization. It leads to the following gradient for optimizing 3D representations with preference pairwise comparisons:
advanced methods are then presented and discussed ($4.2).
Finally, we carry out an analysis study to further elaborate on and discuss the superiority of our method ($4.3).
4.1. Experimental Setups
V Reward =E, [wt(t) (co(x) win. y
ewin)
Ox
Datasets and measurements. We here evaluate the pro- posed method with 110 prompts from GPTEval3D (Wu et al., 2024a), which covers range of creativity and com- plexity use cases. Based on this, two evaluation strategies are exploited. (1) We utilize a text-to-image reward model named ImageReward (Xu et al., 2024b) to evaluate human preference for 3D assets. We calculate the average prefer- ence score across 120 rendered images of a 3D asset and its corresponding text prompt. (2) We use GPT-4V to perform pairwise comparisons with baselines, generating Elo ratings that align with human judgments text alignment, 3D plau- sibility, and cexture-goomete coherence, etc. More details of the two measurements can be found in Appendix C.I.
(xx(xx V.
-c Jose)
00
However, in practice, the gradient in Equation (5) fails to produce realistic results (refer to Figure 6). Delving into the optimization process, we observe that the pairwise compar- ison results can be overly similar, leading to nearly equal scores. this case, directly pushing x away leads to chaotic gradients. To address this, we introduce a piecewise optimization loss that selectively pulls x xwin when the prefer- ence score gap geap small. The gradient of the final loss is defined as:
V RRewan =
Ei (t) (x,"x"" ()) dx E, (t) (Arin A 70
ae the
Sgap < T, otherwise.
Juse
(6)
Baselines.
Following GPTEva13D (Wu et al., 2024a), we benchmark our method against 13 basclines catego- rized into ext--guided and image-guide approaches respec- tively. Specifically, the text-guided group includes Dream- Fusion (Poole et al., 2022), DreamGaussian (Tang al., 2023), Instant3D (Li et al., 2023c), Fantasia3D (Chen et al., 2023), Latent-NeRF (Metzer et al., 2023), Magic3D (Lin et al., 2023), MVDream (Shi et al., 2023), Point-E (Nichol et al., 2022), ProlificDreamer (Wang et al., 2024), Shap- E (Jun Nichol, 2023), and SJC (Wang et al., 2023a). Be- sides, the image-guided group includes SyncDreamer (Liu et al., 2023c) and Wonder3D (Long al., 2024).
where T = 0.001 is a pre-defined threshold, A win (x."in y, e win and A Jose = (x
[x/o0e y, lose
Note that 8gap == (xxinx,))
r(x!oee, y) indicates the dis-
lose
crepancy of preference scores between xX and x!ose In- struction questions can also be incorporated into LMM- based ranking models provide explicit guidance (see em- pirical evaluations in $4.3).
4. Experiments
In this section, a series of experiments are conducted to justify our claims. We first detail experiment setups ($4.1). The comprehensive results and comparison with previous
Implementation.
We conduct experiments using Py-

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Deorribiien Drommrrsion
Magicti
Ponest
Stepest:
SBC
Syschmamm
MVltmam
Qur
A small, rustic cabin sits alone in peaceful, snow--oovree forest
assortment of vintage. frogrant perfumes display
A cat pondering the mysteries of the universe
boat floating calm water
A rough rock Figure 2. Qualitative comparisons the benchmark of GPTEva13D (Wu al., 2024a). Existing methods struggle with text matching, marked in red. DreamDPO improves text matching, which provides better human preference results. (Zoom in to see the details.)
Torch (Paszke ct al., 2019) and threestudio (Guo et al., 2023), with MVDream (Shi et al., 2023) as the backbone our method. Note that we use PyTorch auto-differentiation to compute analytic normals for geometry evaluation GPTE- val3D and do not use the Lambertian shading trick (Lin al., 2023) due to memory limitation. We follow the train- ing strategy of MVDream and use HPSv2 (Wu et al., 2023) as the default reward model. The optimization process takes around two hours on single NVIDIA RTX A6000 GPU.
demonstrate that our method not only improves text align- ment (e.g., ensuring "leaves a trail of flowers" appears under the bicycle shown the first row Figure 7) but also en- hances geometric and texture details (e.g.. generating more luxuriant oak shown in the second row in Figure 7). More visualization results can be found in Appendix D.2.
4.2.2. QUANTITATIVE COMPARISONS
We provide extensive quantitative comparison results to jus- tify our claims. Human preference evaluations are first con- ducted using ImageReward, as illustrated the first column of Table Our method achieves competitive performance compared to existing methods, highlighting the benefits of preference-guided optimization via our reward loss. Then we perform comprehensive evaluation using GPTEval3D. The results indicate that our method outperforms previous state-of-the-aa (SOTA) methods, and ranks first across all metries. Specifically, our method achieves improvements in text-asset alignment (+28.4), plausibility (+24.44), cextegeometry alignment (+25.8), texture details (+48.44, geometry details (+41.44, and overall performance (+24.4). It showcases the superiority of the proposed method in en- hancing text and geometry details while maintaining 3D consistency.
4.2. Comparison with Prior Methods
4.2. QUALITATIVE COMPARISONS
We conduct three qualitative evaluations, which include comparing 13 benchmarks of GPTExa13ID, MVDream (Shi et al., 2023), and DeeamReward (Ye et al., 2025). The eval- uations show improvements text alignment, generation stability, and eextuee-geometry details, respectively. As seen in Figure 2, while the comparing baselines produce high- fidelity results, they often fail in text alignment, marked in red. For instance, in the prompt "A small, rustic cabin sits alone in a peaceful, snow-covered forest", most exist- ing methods miss key elements like the forest (the first row Figure? 2). In contrast, our method accurately captures both objects, showcasing its effectiveness for improving text alignment. Further comparisons with MVDream, are shown in Figure 3. Although MVDream is capable of gen- erating multiview consistent 3D) assets, it struggles with long prompts (e.g., the second and fourth rows in Figure 3).
4.3. More Analyses and Justifications
Evaluation on different backbones. We here further in- vestigate the impact of backbone selection on our method. The performance of DreamDPO using Stable Diffusion v2.1 (SD2.1) (Rombach et al., 2022a) is provided. Note
Instead, our method performs well across both short to long prompts. Lastly, the comparisons with DeeamReward
5

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
MVDream
DreamDPO (Ours)
MVDream
Ours HPSv2)
Ours ImageReward)
redinlly sjaseriical, Midercee abolase jogget conpoars, loning tuaquai Zon garden copplit nay, roled miro
filled with steaming coffee
Variou hollnn, acymsuonnnsl, instane ianoalll, collectd sond-filled civar glan, with and tock, dopional wondowall
Ewisted lower
Figure 5. The analysis of reward models. We present the results of DreamDPO using ImageReward (Xu et al., 2024b). DreamDPO demonstrates effective performance with ImageReward, highlight- ing its potential to leverage stronger reward models to further enhance generation quality.
hvtryy foyeral, axpasmtrical wastr quill, a puishesri pf falenz in greess, filded of the of walhande
SDS
T= 0.01
T= 0.005
== 0.002
T
Allyfi
like dorls
Figure Qualitative comparisons with MVDream (Shi al., 2023). DreamDPO performs well across short to long prompts, offering better human preference results, marked red. (Zoom in the details.)
elderty nomadle with deuply lined face, wearing fur-lined clook and a traditional woven headdress adorned with beads.
ProlificDreamer
MVDrrem
Our MVDreamm
close-up on elderly woman deep-set wrinkles around her eyes, high cheokbones, and than wearing lnitted hot with intricate patterss.
A rubberr cachti
Figure 6. The analysis of the score gap threshold We conduct 2D toy experiments with 7 ranging from 0.01 to 0. The results indicate that small but non-zero effectively filters overly similar losc examples, leading to more detailed outputs.
A lomp covting a ware glow
Figure 4. The analysis of backbone. We present the results of DreamDPO using Stable Diffusion v2.1 (SD2.1) (Rombach et al., 2022a). DreamDPO demonstrates effective performance with SD2.1, highlighting its potential to leverage more advanced back- bone diffusion models further improvements.
ward successfully capture the coffee, and ImageReward places greater emphasis the steam. While IewgeReward demonstrates improvement over the baseline, HPSv2 yields superior results due to its better generalization across di- verse image distributions (Wu et al., 2023). Therefore, we adopt HPSv2 as our default reward model. It highlights the potential of leveraging stronger reward models, e.g., VisionReward (Xu al., 2024a), to further enhance genera- tion quality. Furthermore, DreamDPO is compatible with rule-based reward metrics, such image quality metrics (e.g.. BRISQUE (Mittal et al., 2011)). It does not require additional reward models and is suitable for scenarios with limited computational resources. As shown in Figure 10, we include a case study demonstrating DreamDPO with BRISQUE, a no-reference image quality evaluator, to show its effectiveness.
that as the previous method PromificDreameer (Wang et al., 2024) also utilizes SD2.1, we compare DreamDPO with S against PremieicDreamee. As shown in Figure 4, our method can perform effectively with SD2.1 and achieve competitive results compared to PromificDreamer. Impor- tantly, DreamDPO does not need LoRA training and is more efficient than PromificDreamer..
Evaluation on different reward models. We study the impact of reward model selection on our method. Specif- ically, ImageReward (Xu et al., 2024b) is used, which is an image-based reward model with similarity function comparable to HPSv2 (Wu et al., 2023). As shown in Fig- ure 5, the results demonstrate that our method performs effectively across different reward models, demonstrating its flexibility and scalability, For instance, in the prompt "A mug filled with steaming coffee", both HPSv2 and ImageRe-
Evaluation different impact of the score gap Specifically, 2D toy experiments with range from 0.01 to 0 are conducted. We provide results Figure 6, which show that a smaller 7 produces more detailed outputs. Note that a small 7 means choosing to lose examples with scores close
gaps. We investigate the
samples, and
6

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
MVDream
DreamReward
DreamDPO (Ours)
A bicycle that leaves a trail flowers
A pen leaking blue ink
A colorful kite tangled in the branches of an oak tree, fabric fluttering in the wind
An ultra-detailed illustration of a mythical Phoenix, rising from ashes, vibrant feathers in fiery Palette
Figure 7. Qualitative comparisons with DreamReward (Ye et al., 2025). DreamDPO improves both text matching (marked in red) and geometric/texture details.
focusing the training process on hard cases. However, we observe that v 0 (the last column in Figure 4) results in chaotic gradient. To balance high-fidelity generation and stable training, we suggest using a small but non-zero т, which excludes overly similar lose examples.
use large visual- language models, such as Qw (Bai al., 2023), rank paired results. Specifically, we tract "yes" or "no" questions from the text prompt, query the LMM with rendered paired examples, and count the "yes" responses to calculate the reward score. Then, we use Equation (6) with threshold r = for 3D assets generation. As shown in Figure 8, our method effectively improves text alignment by using LMM to guide the optimization with user instruction (e.g., correcting the number and attribute method flexibly supports of 3D assets). Additionally, optimization. For example, using 1 lose examples guide given the prompt "A dancing elephant", the baseline gen- erates an elephant standing rather than dancing, By setting "elephant stays on the ground" as the lose example and pushing away, our method encourages the elephant to lift its leg, leading dancing pose. It highlights the potential of method integrate LMMs into 3D generation. More implementation details of the LMMs-based comparison be found in Appendix B.2.
Evaluation on different pair examples. The influence of different pair example generation methods is studied. Specifically, we compare: (1) different noises, by adding different Gaussian noises with the same timesteps; (2) differ- steps, by adding the same Gaussian noise with different timesteps. As shown in Figure 9, using different Gaussian noise yields better results than different timesteps. We at- tribute that noisy latents with different timesteps are easier to distinguish, making them less effective as challenging examples. highlights the importance of generating mean- ingful and challenging lose examples. Accordingly, we adopt different noises the default setting.
Ranking model design. We explore the potential of our method to leverage large multi-modal models (LMMs) for explicit guidance. Instead of relying on a reward model, we
Discussion on potential bias from reward models. The

---

DreamDPO; Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
1000 iter
L1100 tter
2000 iter
10000 iber
1000-ster
H400-iter
2000-iter
10000-iter
MVDream
Ours Quen)
Are dhere non shors in the pretoos?
nie logf shoating"
No: oose
Lose
Yes. Win
Yos Win
pair liking boon caked and doorstep of
shouning leaf
20000-iter
6000-ater
10000-iteer
4000-10cr
10000
MVDrem
Ours Quem)
Are torre store shate ned syylles the pienere?
Dows the olephnn on the gresss??
No.
Four npe apples it basket
dancing elephare
Figure 8. The generation results of DPO with large multi-modal models (LMMs). We explore the potential of our method to leverage LMMs, such as QwenVL (Bai al., 2023) for explicit guidance in correcting the number and attribute of 3D assets. The left corner shows the details of pairwise comparisons using the LMM, including the question and win/lose criteria. By carefully designing the question, DreamDPO can leverage both win and lose examples guide optimization. (Zoom in see the details.)
Ours
Ours (w. Different Timestep)
DecuniiPO
DecamDRO
DreanieO
MVDream
(w. Different Noiss)
Jolaytéé Evvieunnnnnn rommavior shaped the loninang
Aagg,ille with www.wigg
Figure 9. The analysis of pairwise example construction. We compare (1) different noises: adding different Gaussian noises with the same timesteps, and (2) difference timesteps: adding the same Gaussian noise with different timesteps.
Figure 10. The analysis of ruru-ueaaae buse-based reward metrics and ensemble reward models. (1) DreamDPO is compatible with image quality metrics (e.g., BRISQUE (Mittal et al., 201 1)). (2) DreamDPO could combine two rankings reduce reliance single model and mitigate unwanted biases.
reward bias is a known issue in RL-based optimization. One solution is to scale the training data and model size of the reward model. As shown in our experiments in Section 4.3, the reward model HPSv2, which has a stronger general- ization ability compared to ImwgeReward, demonstrates superior generation performance in most cases. Moreover, DreamDPO supports an ensemble of reward models, com- bining their rankings to reduce reliance single model and mitigate unwanted biases (see Figure 10).
some potential artifacts, offering a more reliable evaluation for 31) consistency.
Discussion on LMM-based reward model enhancements. We investigate strategies enhance performance when us- ing large multimodal models (LMMs) as reward models, in- cluding increasing the number of comparison candidates and employing stronger LMMs. First, we extend DreamDPO to support multi-sample comparisons. Specifically, we modify STEPI to construct multiple candidate examples (e.g.. 4 or 6), enabling more diverse comparisons. From these, we select the candidate with the highest reward score as the win example and the one with the lowest score as the lose example. As shown in Figure 12 (first and second columns), this approach proves effective, as it allows DreamDPO to mine more positive samples and form higher-quality positive- pairs. Second, we evaluate the impact of the LMM scale by conducting experiments with three models: Qwen2.5-VL-3B,, Qwen2.5-VL-32B, and Qwen-
Discussion on potential limitations on 2D metrics. In this paper, we mainly adopt 2D reward models, due to the fact that they are relatively mature, with abundant data and good reward performance. However, they have limitations. While they improve prompt alignment, such correcting character attributes (e.g., changing "man" to "knight"), they struggle to address inherent Janus issues (e.g., characters with three legs). Our DreamDPO is flexible with both 2D and 3D rewards (Ye et al., 2025). Experiments in Figure 11 demonstrate that effectively mitigates Janus issues and
8

---

DreamDPO; Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
DPe (W. HPSv2)
Dre
MVDremm
(w. Reward31)
DSLR portrait of elderly man with a long white beurd, deep-set eyes, and a fur-lined cap
A DSLR photo a knight chopyning woud
A head of am Groot
Figure 13. The further application of DreamDPO, We conduct toy experiments on texi-to-avatar generation by combining DreamDPO with Gaussian-basee avalar generation framework (Zhou et al., 2024). More details can be checked in Appendix B.3.
Various hollow, axymmeetrical, textured seashells, collected in a sand-filld,, clear glass with twine-sied neck, displayed wdndowzilt
Figure 11. The analysis of the 3D reward models. DPe is flexible with both 2D and 3D rewards. The 3D reward model (e.g.. Reward3D (Ye et al., 2025)) effectively mitigates Janus issues and some potential artifacts.
tion (Zhang et al., 2024a), etc.
5. Conclusion
Owen: Qwes-VL-Ples VL-Ples
Candidates
Candidates Qwen-VL-PPon Vlos
Candidases Qwen2 V-L-3b
Candidates S-VL335
In this work, we propose DreamDPO, an optimization-based 3D generation method that offers human preferences and fine-grained control for generation. The method is built on three key steps: pairwise example construction, pairwise example comparison, and preference-guice
200-itor
400-iter
optimization. Unlike existing methods that rely on precise pointwise qual- ity evaluations, DreamDPO uses pairwise comparison in- formed by reward models or large multimodal models. It enables a more flexible optimization process. By incor- porating human preferences directly into the optimization, DreamDPO generates 3D assets that are better aligned with input text and exhibit enhanced texture and geometry qual- ity. Comprehensive experimental results demonstrate that DreamDPO surpasses previous state-of-the-ar methods in both output quality and controllability. Lastly, we hope DreamDPO paves the way for more refined, adaptable, and human-aligned 3D content generation solutions.
600-iter
800-iter
100-iter
300-itee
These vibrunt bollowws ried togyther
The analysis of the LMM-based reward model. The results show that increasing the number of comparison candi- dates helps to better mine positive samples, while high-performing LMMs could enhance DreamDPO's performance.
Figure 12.
Limitations and future work. While DreamDPO has shown improvements aligning 3D generation with human preferences. several avenues for future research could fur- ther enhance its performance and applicability. The primary limitations of DreamDPO are as follows: (1) Al feedback can be used for guidelines, but is largely limited to the inher- ent power of generative models. (2) Open API can provide more freedom, but actually brings more instability, where instruction prompts should be designed carefully.
VI.-Plus (Bai et al., 2023). Notably, Qwen2.5-VL with only 3B parameters, a relatively weak model. Results in Figure 12 (third column) indicate that when the capability of LMM is limited (e.g., Owen2.5-V1--3B), it struggles with tasks such as number correction. In SDS-based optimization, early correction of numerical content is crucial, which re- quires strong LMM. Thus, larger and more capable LMMs can enhance DPeasDPO's performance.
To address these limitations, we suggest the following di- rections for future work: (1) Enhancing generative models. Incorporating image prompts (Chen al., 2024) to intro- duce explicit guidance could improve alignment with user expectations by providing a more detailed context for gener- ation. (2) Improving the robustness models in pairwise comparison. Exploring prompt- methods, such as using object detection models (Wang et al., 2023c) or ground- ing models (Oquab et al., 2023), for number and attribute correction, might reduce dependencies on prompt design.
Further application. To showcase the potential of our method, we provide empirical results on text-to-avatar gen- eration. Specifically, we replace the score sampling loss in HeadStudio (Zhou et al., 2024) which is 3DGS (Kerbl et al., 2023) based avatar generation framework, with our re- ward loss. As illustrated in Figure 13, our method achieves great generation results. This underscores the broader ap- plicability of our method to various generation tasks, e.g., 4D generation (Bahmani et al., 2024) and scene genera-

---

DrcamDPO; Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Acknowledgments
al. Training a helpful and harmless assistant with rein- forcement learning from human feedback. arXiv preprint arXiv:2204.085 2022a.
This work was supported in part by the National Natural Science Foundation of China (U2336212), Fundamental Research Funds for the Zhejiang Provincial Universities (226-202-000288), Earth System Big Data Platform of the School of Earth Sciences, Zhejiang University. Xiaobo Xia is supported by MoE Key Laboratory of Brain-inspired Intelligent Perception and Cognition, University of Science and Technology of China (Grant No. 2421002).
Bai, Y., Kadavath, S., Kundu, S., Askell, A., Kernion, J., Jones, A., Chen, A., Goldie, A., Mirhoseini, A., McKin- non, C., et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:22/2.08800 2022b.
Bao, F., Li, C.,, Zhu, J., and Zhang, B. Analytic-dpm: an ana- lytic estimate of the optimal reverse variance in diffusion probabilistic models. In ICLR, 2022.
This research/project is supported by the National Research Foundation, Singapore under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP- 2024-002). Any opinions, findings and conclusions or rec- ommendations expressed in this material are those of the author(s) and do not reflect the views of National Research Foundation, Singapore.
Barron, J. T., Mildenhall, B., Verbin, D., Srinivasan, P. P., and Hedman, Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, pp. 5470-5479, 2022.
Black, K., Janner, M., Du, Y., Kostrikov, I., and Levine, Training difffusion models with reinforcement learning. arXiv preprint arXiv: 13301, 2023.
Impact Statement
Brooks, T., Holynski, A., and Efros, A. A. Instructpix2pix:
This paper advances the field of text-to-3D generation by introducing DreamDPO, an optimization- based framework that integrates human preferences into the 3D generation process. By aligning generated content more closely with human preferences, our approach enhances the quality, con- trollability, and applicability of 3D content creation across various domains, including gaming, virtual reality, design, and digital media.
Learning to follow image editing instructions. In CVPR, pp. 18392-18402, 2023.
Cao, Y., Cao, Y.-P., Han, K., Shan, Y., and Wong, K.- Y. K. Dreamavatar: Text-and-shape guided 3d human avatar generation via diffusion models. arXiv preprint 230400300 2023.
Chen, R., Chen, Y., Jiao, N., and Jia, K. Fantasia3d: Disen- tangling geometry and appearance for high-quality text- to-3d content creation. In ICCV, pp. 1244622256, 2023.
From an ethical perspective, our work raises considerations related to bias in human preference data, potential misuse in deepfake or unauthorized content generation, and the environmental impact of computationally intensive opti- mization processes. To mitigate these risks, we promote transparency by open-sourcing
Chen, Y., Pan, Y., Yang, H., Yao, T., and Mei, T. Vp3d: Unleashing 2d visual prompt for text-to-3d generation. In CVPR, pp. 4896-4905, 2024.
code and models, allow- ing the research community to further evaluate and refine our approach. Additionally, we encourage responsible use of preference-driven generation techniques to ensure ethical applications.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In NeurlPS, volume 30, 2017.
Chung, J., Lee, S., Nam, H., Lee, J., and Lee, K. M. Lucid- dreamer: Domain-free generation of 3d gaussian splatting scenes. In CVPR, 2024.
References
Bahmani, S., Skorokhodov, I., Rong, V., Wetzstein, G., Guibas, I.., Wonka, P., Tulyakov, S., Park, J. J., Tagliasac- chi, A., and Lindell, D. B. 4d-fy: Text-to-4d generation using hybrid score distillation sampling. In CVPR, pp. 7996-8006, 2024.
Clark, K., Vicol, P., Swersky, K., and Fleet, D. Directly fine-tuning diffusion models on differentiable rewards. arXiv preprint arXiv:2309. 11099.774000 17400, 2023.
Cohen-Bar, D., Richardson, E., Metzer, G., Giryes, R., and Cohen-Or, D. Set-the-scene: Global-local training for generating controllable nerf scenes. arXiv preprint arXiv:2303.13444 2023.
Bai, J., Bai, S., Yang, S., Wang, S., Tan, S., Wang, P., Lin, J., Zhou, C., and Zhou, Qwen-vl: A frontier large vision- language model with versatile abilities. arXiv preprint ar0iv:2308.1 /2966 2023.
Deitke, M., Schwenk, D., Salvador, J., Weihs, L., Michel, O., VanderBilt, E., Schmidt, L., Ehsani, K., Kembhavi, A., and Farhadi, A. Objaverse: A universe of annotated
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T.
3d objects. In CVPR, pp. 13142-13153, 2023.
10

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Deitke, M., Liu, R., Wallingford, M., Ngo, H., Michel, O., Kusupati, A., Fan, A., Laforte, C., Voleti, V., Gadre, S. Y., et al. Objaverse-xl: A universe of 10m+ 3d objects. In NeurlPS, volume 36, 2024.
Jiang, R., Wang, C., Zhang, J., Chai, M., He, M., Chen, D., and Liao, J. Avatarcraft: Transforming text into neural human avatars with parameterized shape and pose control. arXiv preprint ar.ivv:3303.I11... 17606, 2023.
Fan, Y., Watkins, O., Du, Y., Liu, H., Ryu, M., Boutilier, C., Abbeel, P., Ghavamzadeh, M., Lee, K., and Lee, K. Rein- forcement learning for fine-tuning text-to-image diffusion models. In NeurlPS, 2024.
Jun, H. and Nichol, A. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.0244 024666, 2023.
Kamata, H., Sakuma, Y., Hayakawa, A., Ishii, M., and
Narihira, T. Instruct 3d-to-3d: Text instruction guided 3d-to-3d conversion. arXiv preprint arXiv:2303.15780 2023.
Gal, R., Alaluf, Y., Atzmon, Y., Patashnik, O., Bermano, A. H., Chechik, G., and Cohen-Or, D. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208. 016188.....1....1.011 2022.
Katzir, O., Patashnik, O., Cohen-Or, D., and Lischin- ski, D. Noise-free score distillation. arXiv preprint arXiv:2310. 17590, 2023.
Guo, Y.-C., Liu, Y.-T., Shao, R., Laforte, C., Voleti, V., Luo. G., Chen, C.-H., Zou, Z.-X., Wang, C., Cao, Y.- P., and Zhang, S.-H. threestudio: A unified framework for 3d content generation. httpp:///githi com/ threestudio perreetudio-projec///aoee ptwjecttttreesttteteeeeeeee
Kerbl, B., Kopanas, G., Leimkühllrr T., and Drettakis, G. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023.
2023.
Han, X., Cao, Y., Han, K., Zhu, X., Deng, J., Song. Y.-Z., Xiang, T., and Wong, K.-Y. K. Headsculpt: Crafting 3d head avatars with text. arXiv preprint arXiv:2306.03008 2023.
Lee, H., Phatale, S., Mansoor, H., Lu, K. R., Mesnard, T., Ferret, J., Bishop, C., Hall, E,, Carbune, V., and Rastogi, A. Rlaif: Scaling reinforcement learning from human feedback with feedback. arXiv preprint arXiv:2309.0000 00267, 2023a.
Haque, A., Tancik, M., Efros, A., Holynski, A., and Kanazawa, A. Instruct-nerf2neff:: Editing 3d scenes with instructions. In ICCV, 2023.
Lee, K., Liu, H., Ryu, M., Watkins, O., Du, Y., Boutilier, C., Abbeel, P., Ghavamzadeh, M., and Gu, S. S. Aligning text- to-image models using human feedback. arXiv preprint arXhv:2302.12199 2023b.
Ho, J. and Salimans, T. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12598, 2022.
Li, C., Zhang, C., Cho. J., Waghwase, A., Lee, L.-H., Rameau, F., Yang, Y., Bae, S.-H., and Hong, C. Gen- erative ai meets 3d: survey on text-to-3d aige era. arXiv preprint arXiv:2305.000 001.0333...3,,. 2023a.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion proba- bilistic models. In NeurlPS, volume 33, pp. 6840-6851, 2020.
Li, C., Zhang, C., Waghwase, A., Lee, L.-H., Rameau, F., Yang, Y., Bae, S.-H., and Hong, C. S. Generative ai meets 3d: A survey on text-to-3d in aigo era. arXiv preprint arXhv:2305.0613/ 2023b.
Ho, J., Chan, W., Saharia, C., Whang, J., Gao, R., Gritsenko, A., Kingma, P., Poole, B., Norouzi, M., Fleet, D. J., et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303, 2022.
Li, J., Tan, H., Zhang, K., Xu, Z., Luan, F., Xu, Y., Hong, Y., Sunkavalli, K., Shakhnarovich, G., and Bi, S. Instant3d: Fast text-to-3d with sparse-view generation and large reconstruction model. arXiv preprint arXiv:2311.0mm 14, 2023c.
Höllein, I.., Cao, A., Owens, A., Johnson, J., and Nießner, M. Text2room: Extracting textured 3d meshes from 2d text-to-image models. arXiv preprint arXiv:2303.11988 2023.
Lin, C.-H., Gao, J., Tang, L., Takikawa, T., Zeng, X., Huang, X., Kreis, K., Fidler, S., Liu, M.-Y., and Lin, T.-Y. Magic3d: High-resolution text-to-3d content creation. In CVPR, pp. 300-309, 2023.
Hong, Y., Zhang, K., Gu, J., Bi, S., Zhou, Y., Liu, D., Liu, F. Sunkavalli, K., Bui, T., and Tan, H. Lrm: Large reconstruction model for single image to 3d. In ICLR, 2023.
Liu, M., Xu, C., Jin, H., Chen, L., Varma T, M., Xu, Z., and Su, H. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. In NeurIPS, volume 36, 2024a.
Jain, A., Mildenhall, B., Barron, J. T., Abbeel, P., and Poole, B. Zhoo text- lext-guided object generation with dream fields. In CVPR, 2022.

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Liu, R., Wang, X., Wang. W., and Yang, Y. Bird's --yee-vew scene graph for vision-language navigation. In ICCV, 2023a.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H., Szafraniec, M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F,, EI- Nouby, A., et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719. 2023.
Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., and Vondrick, C. Zero-1-to-3 Zero-shot one image to 2023b.
3d object. In ICCV, pp.
Ouyang, L.,. Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C.,, Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
Liu, R., Wang, W., and Yang, Y. Vision-language navigation with energy-based policy. In NewrIPS, 2024b.
Liu, R., Wang, W., and Yang, Volumetric environment representation for vision- saiiugla navigation. In CVPR, 2024c.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., al. Pytorch: An imperative style, gigh-performance deep learning library. In NeurlPS, 2019.
Liu, X., Tu, T., Ma, Y., and Chua, T.S. Extending visual dynamics for video-to-music generation. arXiv preprint arXiv:2504.07594, 2025.
Peebles, W. and Xie, Scalable diffusion models with transformers. ICCV, pp. 4195 $195-4205, 2023.
Liu, Y., Lin, C., Zeng. Z., Long, X., Liu, L.., Komura, T., and Wang, W. Synedreamer: Generating multiview- consistent images from single-view image. arXiv preprint arXiv:2309.03453, 2023c.
Poole, B., Jain, A., Barron, J. T., and Mildenhall, B. Dream- fusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209. 1.X8:2200.144888, 2022.
Radford, A., Kim, J. W., Hallacy, C.,, Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In JCML, pp. 8748-8888 8763, 2021.
Long, X., Guo, Y.CC., Lin, C., Liu, Y., Dou, Z., Liu, L.., Ma, Y., Zhang, S.-H., Habermann, M., Theobalt, C., et al. Wonder3d: Single image 3d using cross-domen diffu- sion. In CVPR, pp. 9970-9980, 2024.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In NeurIPS, 2024.
Meng, Y., Xia, M., and Chen, D. Simpo: Simple preference optimization with reference-free reward. arXiv preprint 2.X1::4005.14344 2024.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10699 2022a.
Metzer, G., Richardson, E., Patashnik, O., Giryes, R., and Cohen-Or, D. Latent-nerf for shape-guided generation of 3d shapes and textures. In CVPR, pp. 12663-12673, 2023.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. High-resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022b.
Mildenhall, B., Srinivasan, P., Tancik, M., Barron, J. T., Ramamoorthi, R., and Ng, R. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject- generation. arXiv preprint a2xiv: /2242, 2022.
Mittal, A., Moorthy, A. K., and Bovik, A. C. Blind/referenceless image spatial quality evaluator. In ASILOMAR, pp. 723-727. IEEE, 2011.
Shen, T., Gao, J., Yin, K., Liu, M.-Y., and Fidler, S. Deep marching tetrahedra: a hybrid representation for high- resolution 3d shape synthesis. In NeurIPS, pp. 6087- 6101, 2021.
Nichol, A., Dhariwal, P., Ramesh, A., Shyam, P., Mishkin, P.. McGrew, B.. Sutskever, I. and Chen, M. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2/ 10744) 2021.
Shi, Y., Wang, P., Ye, J., Long, M., Li, K., and Yang, X. Mvdream: Multi-view diffusion for 3d generation. arXiv preprint arXiv:2308.1653 2023.
Nichol, A., Jun, H., Dhariwal, P., Mishkin, P., and Chen, M. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751,, 2022.
Sohl-Dickstein, J., Weiss, E., Maheswaranathan, N., and Ganguli, S. Deep unsupervised learning using nonequilib- rium thermodynamics. In ICMI., pp. 2256-2265, 2015.
12

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Song, J., Meng, C.,, and Ermon, S. Denoising diffusion implicit models. In ICLR, 2021.
Wu, T., Yang, G., Li, Z., Zhang, K., Liu, Z.. Guibas, L., Lin, D., and Wetzstein, G. Gpt-4v (ision) is human- aligned evaluator for text-to-3d generation. In CVPR, pp. 2024a.
Song, K., Han, L., Liu, B., Metaxas, D., and Elgammal, A. Diffusion guided domain adaptation of image generators. arXiv preprint arXiv:22 2.04473, 2022.
Wu, X., Hao, Y., Sun, K., Chen, Y., Zhu, F., Zhao, R., and Li, H. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09541, 2023.
Su. J., Zhong, Q. Ma, Y., Liu, W., Chen, C., Zheng, X., Yin, J., and Chua, T.-S. Distilling transitional pattern to large language models for multimodal
rec- ommendation. arXiv preprini arXiv:2504. 10538, 2025.
Wu, Z., Zhou, P., Yi, X., Yuan, X., and Zhang, H. Consis- tent3d: Towards consistent high-fidelity text-to-3d gener- ation with deterministic sampling prior. In CVPR, 2024b.
Sun, Z., Shen, S., Cao, S., Liu, H., Li, C., Shen, Y., Gan, C., Gui, L.-Y., Wang, Y.X.., Yang, Y., et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Xiang, J., Lv, Z., Xu, S., Deng. Y., Wang, R., Zhang, B., Chen, D., Tong, X., and Yang, J. Structured latents for scalable and versatile 3d generation. In CVPR. 2025.
Tang, J., Ren, J., Zhou, H., Liu, Z., and Zeng. G. Dream- gaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprini arXiv:2309. 16653, 2023.
Xic, D., Li. J., Tan, H., Sun, X., Shu, Z., Zhou, Y., Bi, S., Pirk, S., and Kaufman, A. Carve3d: Improving multi- view reconstruction consistency for diffusion models with rl finetuming. In CVPR, 2024.
Tang, J.. Chen, Z., Chen, X., Wang. T., Zeng, G., and Liu, Z. Lgm: Large niew gaussian model for high- resolution 3d content creation. In ECCV, 2024.
Xu, J., Huang, Y., Cheng, J., Yang, Y., Xu, J., Wang, Y.. Duan, W., Yang, S., Jin, Q., Li, S., Teng, J., Yang, Z., Zheng, W., Liu, X., Ding, M., Zhang, X., Gu, X., Huang, S., Huang, M., Tang, J., and Dong, Y. Visionreward: Fine- grained multi-dimensional human preference learning for image and video generation, 2024a.
Voynov, A., Aberman, K., and Cohen-Or, D. Sketch-guided text-to-image diffusion models. In SIGGRAPH, pp. 1-11, 2023.
Wallace, B., Dang, M., Rafailov, R., Zhou, L.., Lou, A., Pu- rushwalkam, S., Ermon, S., Xiong, C., Joty, S., and Naik, N. Diffusion model alignment using direct preference optimization. In CVPR, 2024.
Xu, J., Liu, X., Wu, Y., Tong, Y., Li, Q., Ding, M., Tang, J., and Dong. Y. Imagereward: Learning and evaluat- ing human preferences for text-to-image generation. In NeurIPS, 2024b.
Wang, H., Du, X., Li, J., Ych, R. A., and Shakhnarovich, G. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2022.
Yang, Z., Li, L.., Lin, K., Wang, J., Lin, C.-C., Liu, Z.,, and Wang, The dawn Imms: Preliminary explorations with gpt-4v (ision). arXiv preprint arXiv:2309. 174.2. 2023.
Wang, H., Du, X., Li, J., Yeh, A., and Shakhnarovich, G. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, pp. 12619-12630 2023a.
Ye, J., Liu, F., Li, Q., Wang, Z., Wang, Y., Wang, X., Duan, Y., and Zhu, J. Dreamreward: Text-to-3d generation with human preference. In ECCV, pp. 259-276, 2025.
Wang. T., Zhang, B., Zhang, T. Gu, S., Bao, J., Baltrusaitis, T., Shen, J., Chen, D., Wen, F., Chen, Q. et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. CVPR, pp. 4563 4573, 2023b.
Yu, X., Guo, Y.-C,, Li, Y., Liang, D., Zhang, S.-H., and Qi, X. Text-to-3d with classifier score distillation. arXiv preprint arXiv:2310.19415 2023.
Wang, Z., Li, Y., Chen, X., Lim, S.NN., Torralba, A., Zhao, H., and Wang, S. Detecting everything in the open world: Towards universal object detection. In CVPR. pp. 11433- 1 11443, 2023c.
Zhang, B., Tang, J., Niessner, M., and Wonka, P. 3dshape2vecset: A 3d shape representation for neural fields and generative diffusion models. ACM Transac- tions on Graphics, 42(4):1-16, 2023a.
Wang, Z., Lu, C., Wang, Y., Bao, F., Li, C., Su, H., and Zhu, Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2024.
Zhang, C., Zhang, C., Zhang, M., and Kweon, I. Text-to- image diffusion model in generative ai: A survey, arXiv preprint arXiv:2303.07909, 2023b.
13

---

DrcamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Zhang, J. Li, X., Wan, Z., Wang, C., and Liao, J. Text2nerf: Text-driven 3d scene generation with neural radiance fields. IEEE Transactions Visualization and Computer Graphics, 2024a.
Zhang, L., Rao, A., and Agrawala, M. Adding condi- tional control text-to-image diffusion models. In ICCV, 2023c.
Zhang, I.., Wang, Z.,, Zhang. Q.. Qiu, Q. Pang, A., Jiang, H., Yang. W., Xu, L., and Yu, J. Clay: controllable large- scale generative model for creating high-quality 3d assets. ACM Transactions on Graphics, 43(4):1-20, 2024b.
Zhang. S., Yang, X., Feng, Y., Qin, C., Chen, C.-C., Yu, N., Chen, Z., Wang, H., Savarese, S., Ermon, S., al.
Hive: Harnessing human feedback for instructional visual 2024c.
editing. In CVPR, pp.
Zhang, Z.-Y., Han, S., Yao, H., Niu, G., and Sugiyama, M. Generating chain-of- tothoghf-thoughts with direct pairwise- comparison approach searching for the most promising intermediate thought. In ICML, 2024d.
Zhao, W., Zhao, Y., Lu, X., Wang. S., Tong, Y., and Qin, B. Is chatgpt equipped with emotional dialogue capabilities? arXiv preprint arXh::200.009582. 2023.
Zhou, Z., Ma, F,, Fan, H., Yang, Z., and Yang, Y. Head- studio: Text to animatable head avatars with 3d gaussian splatting. In ECCV, 2024.
Zhu, J., Zhuang, P., and Koyejo, S. Hifa: High-fidelity text-to-3d generation with advanced diffusion guidance. arXiv preprint 1xXx:23055.1866, 2023.
Zhuo, W. Ma, F., Fan, H., and Yang, Y. Vividdreamer: invariant score distillation for hyper-realistic text-to-3d generation. In ECCV, 2024.
Zou, Z.-X., Yu, Z., Guo, Y.C.., Li, Y., Liang, D., Cao, Y.- P., and Zhang, S.-H. Triplane meets gaussian splatting: Fast and generalizable single-view 3d reconstruction with transformers. In CVPR, pp. 10324-10335, 2024.
14

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Appendix
Related Work
16
A.1 Text-to-Image Generation
16
A.2 Text-to-3D Generation
16
A.3 Learning from Human Preferences
16
Additional Implementation Details
17
B.1 Pseudo-Code for DreamDPO
17
B.2 Details of LMM-bbsed Pairwise Comparison
17
B.3 Details of Text-to-Avatar Generation
18
Supplementary Experimental Settings
18
C.1 Details of Measurement Metrics
18
D Supplementary Experimental Results
19
D.1 Discussion
19
D.2 More Qualitative Results
20
15

---

DreamDPO; Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
A. Related Work
A.1. Text-to-lmage Generation
With the development of vision-language models (Radford et al., 2021) and diffusion models (Sohl-Dickstein et al., 2015; Ho al., 2020), great advancements recently have been made in text-to-image generation (Nichol et al., 2021; Ho et al., 2022; Zhang et al., 2023b). In particular, Stable Diffusion (Rombach et al., 2022b) is a notable framework that trains the diffusion models latent space, leading to reduced complexity and detailed preservation. In addition, with the emergence of text-to-2D models, more applications have been developed (Liu et al., 2023a; 2025; 2024c;b), e-g., spatial control (Voynov et al., 2023; Zhang et al., 2023c), concept control (Gal et al., 2022; Ruiz et al., 2022), and image editing (Brooks et al., 2023).
A.2. Text-to-3D Generation
The success of the 2D generation is incredible. However, it is challenging to transfer image diffusion models to 3D because of the difficulty of 3D data collection. Fortunately, Neural Radiance Fields (NeRF) (Mildenhall et al., 2020; Barron et al., 2022) provided new insights for 3D-aware generation, where only 2D multi-view images are needed in 3D scene reconstruction. Combining prior knowledge from text-to-2D models, several methods, such as DreamField (Jain et al., 2022), DreamFusion (Poole et al., 2022), and SJC (Wang et al., 2022), have been proposed to generate 3D objects guided by text prompts (Li et al., 2023b). However, the vanilla score distillation sampling loss (Poole al., 2022) suffers from issues such as over-saturation, over-smoothing, and Janus problems, etc. Recently, several works have proposed improvements to enhance generation quality (Wang et al., 2024; Yu et al., 2023; Zhu et al., 2023; Katzir et al., 2023; Chung et al., 2024; Wu et al., 2024b; Zhuo et al., 2024), and inspire multiple applications that include but are not limited to text-guided scene generation (Cohen-Bi Bar et al., 2023; Höllein et al., 2023), text-guided 3D editing (Haque et al., 2023; Kamata et al., 2023), and text-guided avatar generation (Cao et al., 2023; Jiang et al., 2023; Han et al., 2023; Zhou et al., 2024). Additionally, with the availability of large 3D datasets (Deitke et al., 2023; 2024), some works (Liu et al., 2023b; Shi et al.. 2023; Liu et al., 2024a; 2023c; Long et al., 2024) leverage multi-view information to address the Janus problem more effectively.
Recent advancements in multi-view diffusion models have enabled the development of large reconstruction models (Hong et al., 2023; Tang et al., 2024: Zou et al., 2024) that can generate 3D objects from multi-view inputs within minutes, In paraliel, 3D generative models (Zhang et al., 2023a; Wang et al., 2023b; Zhang et al., 2024b; Xiang et al., 2025) have emerged, enabling the direct creation of 3D objects guided by text or images. These feed-forward methods significantly accelerate the 3D generation process, fostering broader applications in areas such as gaming and 3D printing. Aligning feed-forward methods with human preferences is meaningful but remains underexplored, primarily due to the significant computational resources required. DreamDPO offers valuable insights for feed-forward methods. Specifically, by constructing candidates with varying noise and using manking--uided optimization, DreamDPO provides potential pathway for reinforcement learning-basse based optimization of feed-forwar methods.
A.3. Learning from Human Preferences
Learning from human preferences is essential for improving the alignment and performance of generative models across various domains, including large language models (LLMs) (Bai et al., 2022a;b; Lee et al., 2023a; Zhao et al., 2023; Su et al., 2025), text-to-image diffusion models (Black et al., 2023; Lee et al., 2023b; Clark et al., 2023; Xu et al., 2024b; Wallace et al., 2024; Fan et al., 2024; Zhang et al., 2024c), and text-to-3D generation (Xie et al., 2024; Ye et al., 2025). Reinforcement Learning from Human Feedback (RLHF) (Christiano et al., 2017) has been proven effective in refining LLMs to better align with human preferences. It often involves collecting human feedback datasets, training a reward model, and fine-tuning the language model using reinforcement learning. InstructGPT (Ouyang et al., 2022) is notable work that employs two-stage fine-tuning strategy to align GPT-3 with human instructions, which leads to more coherent and contextually appropriate outputs.
The success of RLHF in LLMs has inspired the applications in text-to-image diffusion models to enhance image generation quality and align with human preferences. In more detail, RWR (Lee et al., 2023b) first introduces human feedback- reward fine-tuning for diffusion models, which fine-tunes Stable Diffusion (Rombach et al., 2022a) using log probabilities of the denoising process. ImageReward (Xu et al., 2024b) proposes a reward model specifically for text-to-image tasks and further develops reward feedback Learning for refining diffusion models. DiffusionDPO (Wallace et al., 2024) uses DPO to optimize difffusion models using human comparative data, while DPOK (Fan et al., 2024) integrates policy optimization
16

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
with KL regularization for improved alignment. Recently, advancements in multi-view diffusion models have facilitated significant progress in text-to-3D generation. For instance, Carve3I) (Xie et al., 2024) enhances text-to-3D generation with Multi-view Reconstruction Consistency (MRC) metric for improved consistency and quality. DreamReward (Ye et al., 2025) improves text-to-3D models using human feedback. It collects a 3D dataset with human annotations, trains Reward3D as a multi-view reward model, and introduces DreamFL. to create 3D assets aligned with human preferences. However, these works rely heavily on large-scale datasets to train reward model or utilize it as preference feedback, which is very expensive for generation.
B. Additional Implementation Details
B.1. Pseudo-Code for DreamDPO
A more detailed pseudo-code for DreamDPO is presented in Algorithm 1.
Algorithm Pseudo-code for DreamDPO
Input: Text-to-image diffusion model Eg. Ranking model Learning rate for 3D representation parameters. A prompt y. Evaluating threshold of score gap
1: Initialization A 3D representation presenting with NeRF A
2: while not converged do
3: Randomly sample a camera pose c, pairwise 2D noise el and e², and timestep & Uniform( ((0. (0..... .T))
4: Render pose с get a image Xo.
5: Add noise e' and c2 Xy and get x/ and x², respectively.
6: Denoise with the predicting noise:
81 x: (x) y,t)
x o2 xx: v.
7: Score the prediction ($t) x3) online via rank model r. yielding the pairwise comparison (xtiin x
8:
Compute the score gap:
r(x, win .) r r(x lose ,y).
gapp
9:
if gap < r then
10:
0x
V Reeawdddd = Eg w(() (x .y.t))
00
11: else
12:
0x
;2, VeLReward E, w(() (EE)(x) :2, - einn) (Eb luse, ; E clos))
30
13:
end if
0
V Rewaard
end while
B.2. Details of LMMebased Pairwise Comparison
We detail the implementation of the LMM-based pairwise comparison. We use the large visual-langure model "qwen-vl- plus-latest" from VL (Bai et al., 2023) the default LMM. Given pairwise examples, we conduct the comparison query sequentially. For each query, we first insert a predefined "yes" or "no" question into the comparison prompt, such as "Is the leaf shouting?" for the prompt "A shouting leaf." Then, the LMM performs visual question answering based on the provided image and query. Finally, we extract the number of "yes"" responses as the score. The following prompts are used for the queries:
17

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
Comparison Query
You are an expert in evaluating the alignment between given text description and an image. Your task is answer each of the alignment questions with either "Yes" or "No" based on the image. Provide your responses in the format specified below.
[Task Description]:
[Evaluation Instruction]:
1. Carefully analyze the provided image and answer questions based on the image. For each question, answer with either "Yes" or "No". Do not provide explanations or additional information.
[Evaluation Question(s)]:
Q1: {
[Output Format]:
Al: [Yes/No]
B.3. Details of Text-to-Avatar Generation
We detail the toy exploration of text-to-avatar generation using DreamDPO. Specifically, we integrate the reward loss into HcadStudio (Zhou et al., 2024), Gaussian-basee avatar generation framework. HcadStudio is an optimization-on based method that utilizes score sampling loss (Katzir al., 2023) with ControlNet (Zhang et al., 2023c) to optimize an animatable head prior model. By replacing the score sampling loss with the reward loss, we leverage ControlNet generate avatars.
C. Supplementary Experimental Settings
C.1. Details of Measurement Metrics
In the main paper, we employ two evaluation strategies to demonstrate the superiority of the proposed method. Here we supplementary the details the measurements.
Evaluation with IeageReward. ImageReward (Xu al., 2024b) is a text-to-image human preference reward model. Due its effectiveness, has been broadly used for human preference evaluation in rext-to-image generation (Fan al., 2024) and text-to-II generation (Ye al., 2025). Given (text, image) pair, it extracts image and text features, combines them with cross-attention, and uses MLP to generate scalar for preference comparison. For each 3D asset, uniformly render 120 RGB images from different viewpoints. Afterward, the ImageReward score is computed from the multi-view renderings and averaged for each prompt.
Evaluation with GPTEva13D. We utilize GPTEva13D (Wu et al., 2024a), which is a comprehensive benchmark for text-to-3D generation evaluation. GPTEva13D includes 13 baseline methods M, 110 text prompts, and 5 criteria that are text-asset alignment, 3D plausibility, texture details, geometry details, and texture-geonne settme-g coherency respectively. For new method, GPTEva13D employs GPT-4V to compare 3D assets generated by this new method and one of the baseline methods with the same input text prompt. These pairwise comparison results are then used to calculate the Elo rating for each model. Specifically, let A be matrix where Aij represents the number of times that the i-th model outperforms the j-th model in comparisons. The Elo ratings for the models are computed by optimizing the following objective:
Asj log (1 -0(0s-)0))))0))))
a arg max
(8)
$
where o, E R is the Elo rating of the i-th model. In this work, we calculate Elo ratings within the existing tournament, initializing, and freezing baseline scores as specified in the official code?. For interested readers, please refer to (Wu et al., 2024a).
18

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
MVDream
DreamDPO (10000 Steps)
DreamDPO (4000 steps)
book left on a park bench
A thorny rose
Figure The analysis of reducing generation time, DreamDPO could reduce generation time by 25% (to 1.5 hours) while maintaining
improved performance.
0.001
==
Figure 15. The analysis of the gap threshold We conduct 3D experiments with ranging from I 0. The results indicate that 0 results in over-saturation, and a small but non-zero leads to more detailed outputs.
D. Supplementary Experimental Results
D.I. Discussion
Discussion on training time. The vanilla SDS takes around hour optimize. Our method takes approximately hours due to the pairwise example construction. To speed up the process, SDS for the first 6000 iterations and then switching to DreamDPO. As shown in Figure 14, this approach reduces the generation time by 25% (to around 1.5 hours) while maintaining improved performance.
adopt simple yet effective strategy: performing
Discussion on the score gap threshold in the 3D setting. We present the ablation study the score gap threshold 7 the 3D setting (see Figure 15). The results indicate that a large 7 (e.g., т = 1) degrades DreamDPO to the SDS loss, resulting in may result over-saturation some cases, such purplish sunlight
an over-smoothing issue. Conversely, setting appearance in rocks. Therefore, we recommend using a small but non-zero T.
19

---

DrcamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
D.2. More Qualitative Results
We present additional qualitative results Figure 16 and Figure 17. The comparisons demonstrate that method generates human-prefeen 3D assets, with improved text alignment and enhanced geometric and texture details.
20

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
MVDream
DreamDPO (Ours)
A book left park bench
small, solid, geometrically spherical, metallic orb, with glossy ruby finish, nestled in of black velvet, untouched and gleaming under spotlight
An ensemble of hollow, irregularly shaped musical instruments, including saxophone, violin, and a drum, resting on a stage before jazz concert"
A bunch of colorful marbles spilling out of a red velvet bag
ceramic upside down yellow octopus holding blue green ceramic cup
A DSLR photo of lion reading the newspaper
DSLR photo of humanoid robot using laptop
Figure 16. More qualitative results using DreamDPO.
21

---

DreamDPO: Aligning Text-to-3D Generation with Human Preferences via Direct Preference Optimization
MVDream
DreamDPO (Ours)
A DSLR photo of a blue jay standing large basket of rainbow macarons
A DSLR photo of cat lying its side batting ball of yarn
A DSLR photo of cat mugician making white dove appear
A DSLR photo of a knight chopping wood
A beagle in detective 's outfit
A collection of solid, irregularly shaped hand tools, with wooden handles and metal ends, well-used and slightly rusty, hanging on a pegboard in a workshop
DSLR photo of a basil plant
Figure 17. More qualitative results using DreamDPO.
22


Reviews:
[{"review_id": "tl31kQv4O8", "reviewer": "Reviewer_x6sm", "summary": "The paper introduces the DreamDPO framework, designed to enhance text-to-3D generation by aligning generated content more closely with human preferences. Traditional methods often fall short due to their heavy reliance on precise evaluations, restricting flexibility and applicability. DreamDPO employs an optimization-based approach that incorporates human preferences through direct preference optimization.\n\nThe methodology consists of three key steps:\n\n1. **Pairwise Example Construction**: The framework constructs pairwise examples by applying varying Gaussian noise to the diffusion model.\n2. **Pairwise comparison**: It uses either reward models or large multimodal models to rank these examples based on their alignment with the provided textual prompts.\n3. **Preference-guided optimization**: A preference-driven loss function guides the optimization of the 3D representation.\n\nThese steps allow DreamDPO to minimize dependence on exact scoring while granting robust control over the generation process.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive reviews. We provide our responses below.\n\n> **Q1:** Dependence on External Models.\n\n**A1:** DreamDPO is compatible with external models and rule-based reward metrics, such as image quality metrics (e.g., BRISQUE [1]) and 3D-consistency evaluation. By leveraging 3D geometry information (depth and camera transformation matrices), we could establish stereo correspondence between views to evaluate multi-view consistency. These approaches do not require additional reward models and are suitable for scenarios with limited computational resources. As shown in [Figure 1-1](https://imgur.com/a/XdVx2sn), we include a case study demonstrating DreamDPO with BRISQUE, a no-reference image quality evaluator, to show its effectiveness.\n\n> **Q2:** How to mitigate the potential bias from reward model?\n\n**A2:** Reward hacking is a known issue in RL-based optimization. One solution is to scale the training data and model size of the reward model. As shown in our experiments in Section 4.3, the reward model HPSv2, which has a stronger generalization ability compared to ImageReward, demonstrates superior generation performance in most cases. Moreover, DreamDPO supports an ensemble of reward models, combining their ranking to reduce reliance on a single model and mitigate unwanted biases (see [Figure 1-2](https://imgur.com/a/XdVx2sn)).\n\n\n> **Q3:** How does the LMM in STEP2 work, and where do its questions come from?\n\n**A3:** We detail the LMM-based pairwise comparison in Section B.2. In STEP2, given pairwise examples, LMM conducts the comparison query sequentially. For each query, LMM performs visual question answering based on the provided image and query, and we extract the number of \"yes\" responses as the score. The questions for LMM can be customized by the user or generated from a template. For instance, LLM can automatically extract questions from a given prompt, such as generating \"Is the leaf shouting?\" from \"A shouting leaf\". Alternatively, users can define custom questions, such as \"Does the elephant stay on the ground?\" for \"A dancing elephant\".\n\n> **Q4:** Why is the generation time long, and can it be improved?\n\n| Method | MVDream | DreamDPO (10000 Steps) | DreamDPO (4000 Steps) |\n| ------ | ------- | -----------------------| ----------------------|\n| Computation Time | 1 hour | 2 hour | 1.5 hour|\n\nTable 1-1: Reducing generation time for DreamDPO while maintaining improved performance.\n\n**A4:** The vanilla SDS takes around 1 hour for optimization. Our method takes approximately 2 hours due to the pairwise example construction. To speed up the process, we can adopt a simple yet effective strategy: performing SDS for the first 6000 iterations and then switching to DreamDPO. As shown in [Figure 1-3](https://imgur.com/a/XdVx2sn), this approach reduces the generation time by 25% (to around 1.5 hours) while maintaining improved performance.\n\n[1] Anish et al. Blind/referenceless image spatial quality evaluator. ASILOMAR 2011.", "claims_and_evidence": "The claims made in the paper regarding the DreamDPO framework for text-to-3D generation are well-supported by clear and convincing evidence based on the findings presented.\n\n1. **Performance Claims**: The paper asserts that DreamDPO outperforms existing techniques. This claim is supported by extensive experiments outlined in section 4, which detail experiment setups and comprehensive results comparing DreamDPO with 13 state-of-the-art methods.\n2. **Quality and Control Claims**: The paper emphasizes improvements in the texture and geometry quality of the generated 3D assets. Direct reference to qualitative results is made, indicating that the method delivers high-quality outcomes and offers fine-grained control over the generation process. These aspects are discussed through quantitative and qualitative analyses, supplemented by ablation studies that validate the claims regarding quality and adaptability.\n3. **Innovative Contributions**: The paper introduces a novel optimization-based approach integrating human preferences via direct preference optimization. This conceptual contribution is bolstered by detailing the three-step optimization process, highlighting its significance and effectiveness in achieving better alignment with human preferences compared to traditional methods.\n\nHowever, while the paper’s claims are largely supported, additional context or clarification could enhance the arguments surrounding the advantages of using large multimodal models in the optimization loop and the implications of reduced reliance on precise scoring. Providing a more detailed discussion on how these elements distinctly outperform existing approaches could help mitigate doubts regarding the robustness of these claims since large multimodal models sometimes could have poor performance.\n\nOverall, the evidence presented supports the claims made throughout the paper.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria in the paper make sense for the problem of enhancing text-to-3D generation. The DreamDPO framework introduces an innovative approach that integrates human preferences through a systematic process of direct preference optimization. This method comprises constructing pairwise examples influenced by Gaussian noise, ranking these examples against textual prompts using reward models or large multimodal models, and optimizing the 3D representation.\n\nThe evaluation criteria further support this methodology effectively. The paper utilizes two robust evaluation strategies: the ImageReward model, which scores 3D assets based on human preferences by assessing multi-view renderings, and the GPTEval3D benchmark, which compares the generated outputs against 13 baseline methods across five critical criteria—text-asset alignment, 3D plausibility, texture details, geometry details, and texture-geometry coherence.\n\nBoth the methodology and the evaluation benchmarks are well-founded and specifically tailored to address the challenges of generating 3D assets that align closely with human expectations, as reflected in the document’s results showing significant improvements in various metrics. Thus, they are appropriate for evaluating text-to-3D generation.", "theoretical_claims": "The support of proposed framework appears to be on empirical evaluations and quantitative comparisons rather than theoretical proofs. Thus, there are no specific theoretical claims or proofs outlined that were checked for correctness in the paper. The issues addressed mainly revolve around the limitations of previous methods and how DreamDPO aims to overcome these challenges, rather than validating theoretical aspects through proofs.", "experimental_designs_or_analyses": "The paper outlines a series of experiments that were conducted to evaluate the effectiveness of the DreamDPO framework in aligning text-to-3D generation with human preferences. Specifically, it mentions two evaluation strategies employed:\n\n1. **Comparison Using a Text-to-Image Reward Model (ImageReward)**: This model was utilized to assess human preferences for the generated 3D assets based on their alignment with provided text prompts. The average preference scores were calculated across 120 rendered images of the 3D assets.\n2. **Pairwise Comparisons with GPT-4V**: This strategy involved generating Elo ratings that reflect human judgments on various criteria, including text alignment, 3D plausibility, and texture-geometry coherence. The pairwise comparison results were used to calculate ratings that position the performance of DreamDPO against baseline methods.\n\nThe paper does not explicitly state any issues encountered with these experimental designs. However, it does highlight the limitations of prior methods that DreamDPO aims to address, particularly the reliance on accurate pointwise quality evaluations from reward models, which can hinder flexibility and adaptability. Therefore, while the experimental setup leverages innovative measures of evaluation, the potential weaknesses in the dependence on reward models and previous designs may still pose challenges.\n\nOverall, while the paper provides a sound experimental framework, the discussion on limitations suggests that there are ongoing concerns regarding the robustness of the evaluation methods used.", "relation_to_broader_scientific_literature": "The key contributions of the paper build on several important concepts and findings in the broader scientific literature, particularly in the fields of generative models and human preference integration.\n\n1. **Shift from Absolute to Relative Preference Evaluation**: The paper highlights that traditional methods of evaluating 3D generation heavily depend on precise pointwise quality assessments from reward models, which can be restrictive. Previous work has attempted to incorporate human preferences in 3D content generation, but still lacked flexibility and adaptability due to reliance on absolute scoring. DreamDPO shifts this paradigm by utilizing relative preferences, enabling better alignment with human expectations and enhancing the flexibility of the generation process.\n2. **Integration of Large Multimodal Models**: The framework proposed in DreamDPO incorporates insights drawn from large multimodal models. This aligns with recent advancements in text-to-image generation, where models have successfully been used to infer and generate content based on textual inputs. By applying such principles to 3D models, DreamDPO takes advantage of the robust features learned from multimodal datasets, thereby establishing a strong connection with ongoing research in multimodal AI.\n3. **Direct Preference Optimization**: DreamDPO’s method of optimization through direct preference ranking addresses previous limitations noted in the literature, wherein automated systems struggle to meet diverse user expectations. The incorporation of a preference-driven loss function within the generation process also ties back to reinforcement learning algorithms that have shown efficacy in similar tasks, using human preferences to guide learning in various contexts.\n\nIn summary, the contributions of DreamDPO are deeply interwoven with established theories and methods in the literature, pushing the envelope forward by offering a more adaptable and human-aligned approach to text-to-3D generation.", "essential_references_not_discussed": "I don’t find essential references that are not discussed.", "other_strengths_and_weaknesses": "The paper presents the DreamDPO framework, which reflects several notable strengths and weaknesses concerning originality, significance, and clarity.\n\n**Strengths:**\n\n1. **Originality:** The DreamDPO framework introduces a fresh approach to text-to-3D generation by emphasizing direct preference optimization based on human feedback, diverging from traditional methods that rely on pointwise quality evaluations. This optimization-based strategy enhances flexibility and adaptability in generating 3D content, setting it apart from existing techniques that often fail to fully align with human preferences.\n2. **Clarity and Structure:** The paper is well-organized, clearly delineating the methodology, experiments, and outcomes. The three-step process of constructing pairwise examples, comparing them, and guiding optimization through preference-driven loss functions is explained clearly, making the complex ideas accessible.\n3. **Empirical Validation:** Comprehensive experiments demonstrate that DreamDPO outperforms existing methods, providing robust evidence of its efficacy. This empirical validation strengthens the paper’s claims and offers a solid foundation for the proposed approach.\n\n**Weaknesses:**\n\n1. **Dependence on External Models:** While the framework utilizes reward models or large multimodal models for preference ranking, this dependence may limit the applicability of DreamDPO, especially in scenarios with limited access to such models. The necessity for high-performing models could also restrict its use in real-world applications where computational resources are constrained.\n2. **Potential for Bias:** The integration of human preferences into the generation process raises concerns about the inherent biases present in the training data of the reward models. This could lead to outputs that reflect unwanted biases, which may necessitate additional measures to ensure fairness and ethical considerations in generated content.\n3. **Time Consuming**: The optimization process takes around two hours on a single NVIDIA RTX A6000 GPU which is too long for text-to-3d generation since some latest method could produce results in less than 1 hour.", "other_comments_or_suggestions": "I don’t have other comments or suggestions.", "questions_for_authors": "1. Could you explain how the LMM worked in STEP2 more specific? How to get the questions for LMM? Is it provided by the user or generated from template?\n2. Could you explain why you need to use such long time for generation? Is there any methods to speed up the generation process?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xBek80LJdD", "reviewer": "Reviewer_YAnn", "summary": "This paper introduces DreamDPO, an optimization-based framework for text-to-3D generation. The authors propose to integrate human preferences into the generation process through direct preference optimization. The authors claim that the key innovation is leveraging pairwise comparisons to guide optimization instead of relying on absolute, pointwise quality scores. In specific, DreamDPO constructs pairwise examples, evaluates them using reward models or large multimodal models (LMM), and then optimizes the 3D content with a designed preference-driven loss function. This paper shows superior alignment with textual inputs and improved controllability over existing methods on established benchmark.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive reviews. We provide our responses below.\n\n> **Q1:** Ablation studies on the score gap threshold in the 3D setting.\n\n**A1:** We present the ablation study of the score gap threshold $\\tau$ in the 3D setting (see [Figure 2-1](https://imgur.com/a/XdVx2sn)). The results indicate that a large $\\tau$ (e.g., $\\tau=1$) degrades DreamDPO to the SDS loss, resulting in an over-smoothing issue. Conversely, setting $\\tau=0$ results in over-saturation, such as a purplish sunlight appearance in rocks. Therefore, we recommend using a small but non-zero $\\tau$.\n\n\n> **Q2:** Show some failure cases to highlight DreamDPO’s limitations.\n\n**A2:** While DreamDPO has shown improvements in aligning 3D generation with human preferences, there are still some failure cases. For example, number and attribute correction are crucial for prompt alignment, but DreamDPO sometimes fails to address these issues shown in [Figure 2-2](https://imgur.com/a/XdVx2sn). We found that this limitation arises from the generative model's capacity. When the positive sample is hard to generate, DreamDPO struggles to construct effective positive-negative pairs, causing it to degrade into SDS.\n\n> **Q3:** Increasing the number of comparison candidates per iteration.\n\n**A3:** Thank you for your valuable suggestion. We further extend DreamDPO to multi-sample comparisons. In specific, we expand STEP1 to multi-example construction, enabling the creation of more comparison candidates (e.g., 4 or 6). We then select the candidate with the highest score as the \"win\" example and the lowest score sample as the \"lose\" example. Experiments in [Figure 2-3](https://imgur.com/a/XdVx2sn) (the first and second columns) demonstrate that this improvement is effective, as it better mines positive samples and enables DreamDPO to construct positive-negative pairs more effectively.\n\n> **Q4:** More evaluation on DreamDPO with large multi-modal models.\n\n**A4:** Following your recommendation, we evaluated the robustness of DreamDPO with various LMMs. Specifically, we conducted experiments using three large multimodal models: Qwen2.5-VL-3B, Qwen2.5-VL-32B, and Qwen-VL-Plus. Notably, Qwen2.5-VL-3B has only 3B parameters, making it a relatively weak LMM. The results in [Figure 2-3](https://imgur.com/a/XdVx2sn) show that when the LMM performance is relatively poor (e.g., Qwen2.5-VL-3B in third column), it fails to correct numbers effectively. In SDS-based optimization, number correction should perform early, which requires strong LMM capabilities. Therefore, high-performing LMMs can improve DreamDPO.", "claims_and_evidence": "The claims presented are generally supported by substantial evidence through qualitative and quantitative experiments. The experiments demonstrate that DreamDPO somewhat generates higher-quality and more controllable 3D assets compared to existing methods, supported by human preference evaluation scores, GPTEval3D scores, and comprehensive ablation studies.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are sensible and appropriate. The benchmark GPTEval3D dataset is commonly used in text-to-3D generation. Meanwhile, the human preference evaluation using ImageReward also provides robust evaluation metrics. The experimental setups align with standard practices for evaluating generative models, ensuring validity and comparability with prior works.", "theoretical_claims": "No explicit theoretical proofs are provided for this paper.", "experimental_designs_or_analyses": "The experimental designs and analyses appear sound and rigorous. Detailed ablation studies (e.g., different Gaussian noise, score gap thresholds, reward models, and backbones) are clearly conducted, strengthening the validity of the results. The experimental framework effectively validates the method’s key contributions.", "relation_to_broader_scientific_literature": "The most related paper is DreamReward. Both DreamReward and DreamDPO focus on the human preference alignment problem in text-to-3D generation. DreamReward collected and labeled a multi-view preference dataset and accordingly trains a multi-view reward model, and use this model for 3D human preference alignment. \nDifferent with DreamReward, the authors claim that DreamDPO shifts from absolute scoring to direct preference optimization, and introduce a pair-wise optimization loss, which reduce reliance on precise pointwise quality evaluations. Therefore, DreamDPO could use the image-aware reward model and LMM for preference alignment generation.", "essential_references_not_discussed": "The paper appears comprehensive in its references.", "other_strengths_and_weaknesses": "Pros:\n1. The motivation is clear, which addresses the need for improved human alignment and controllability in 3D generation tasks. Besides, the writing is easy to follow.\n2. The method is simple but effective. It integrates human preferences through pairwise comparisons, which improves the alignment of the generated 3D content with user expectations and instructions. l believe it can improve other 3D generation methods based on optimization and have a wide generalizability.\n3. The experiments are convincing. Experiments show DreamDPO generates higher-quality outputs in terms of geometric and textural details compared to baseline models like DreamFusion, DreamGaussian, and MVDream. Besides, it offers explicit and fine-grained control over attributes.\n\nCons:\n1.\tThe ablation study of the score gap threshold $\\tau$ in the 3D generation setting is lacking, limiting insights into its influence on performance.\n2.\tThe analysis of failure cases is insufficient, which constrains a deeper understanding of DreamDPO’s limitations and operational boundaries.\n3.\tThe potential for performance improvement through increasing the number of comparison candidates is worth exploring and should be considered in future work.\n4.\tThe applications of various large multimodal models are underexplored. A more detailed discussion is encouraged to assess how LMM quality affects alignment with human preferences.", "other_comments_or_suggestions": "The authors do not report the CLIP score in Table 1, which should be removed accordingly in the caption.", "questions_for_authors": "1.\tThe current parameter analysis is conducted on 2D (Figure 6). To provide a more comprehensive evolution, I suggest including ablation studies on the score gap threshold $\\tau$ in the 3D setting. \n2.\tIncluding failure cases would offer valuable insights into the limitations of DreamDPO and help delineate the boundaries of its effectiveness.\n3.\tWhile the pairwise comparison method is effective, its potential could be further explored by increasing the number of comparison candidates per iteration. For example, investigating group-based loss functions, such as GRPO loss in DeepSeek-R1, might lead to improved performance.\n4.\tAlthough the authors acknowledge that fine-grained controllability is currently limited by the capabilities of the LMM, it would be worthwhile to explore whether advancements in LMMs could mitigate this constraint.\n5.\tIt is recommended to conduct a broader evaluation across more LMMs, ranging from weaker to more advanced models, to gain a deeper understanding of their impact on controllability and overall performance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5XjRQePsxC", "reviewer": "Reviewer_yvaT", "summary": "This paper proposes DreamDPO, an optimization-based method to better align 3D generation with human preferences for text-to-3D generation. In detail, it constructs pairwise examples to formulate a reward loss function for preferred images with lower loss and less preferred images with higher loss. It conducts comprehensive experiments with 13 baselines and shows the superiority of the proposed method.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive reviews. We provide our responses below.\n\n> **Q1:** The computational efficiency comparison to other baselines.\n\n**A1:** Thank you for your question. We summarize the computational cost of our method compared with other text-to-3D generation baselines as follows:\n\n| Method             | DreamFusion | Fantasia3D | Latent-NeRF | ProlificDreamer | MVDream | DreamDPO |\n| ----------------- | ---------- | ---------- | ---------- | --------------- | ------- | -------- |\n| Computation Time   | 1.5 hours  | 1.5 hours  | 30 minutes | 10 hours         | 1 hour  | 2 hours  |\n\nTable 2-1: The analysis of the time consuming of text-to-3D generation methods.\n\nWhile introducing additional computational overhead due to pairwise example construction, our method achieves the best results on the GPTEval3D Benchmark, demonstrating superior alignment with human preferences. Meanwhile, we can mreduce generation time by 25% (to ~1.5 hours) while maintaining performance (see R1-Q4).\n\n> **Q2:** How are $x_t^{win}$ and $x_t^{lose}$ determined?\n\n**A2:** We utilize a reward model to compute the scores of pairwise examples. The sample with a higher score is regarded as the \"win\" example ($x_t^{win}$), while the sample with a lower score is regarded as the \"lose\" example ($x_t^{lose}$).\n\n> **Q3:** Can current text-to-3D generated items be used in real-world manufacturing?\n\n**A3:** Yes. Text-to-3D generation is increasingly suitable for real-world manufacturing, particularly in customized product design. It allows engineers to quickly create 3D models from textual descriptions. While further refinement may be needed to meet precise industrial standards, the technology is proving valuable in automating design workflows and enabling mass customization.", "claims_and_evidence": "Yes, the claims are clear and convincing.", "methods_and_evaluation_criteria": "Yes, the author utilizes ImageReward score for human preference evaluation, CLIP score for text-image alignment evaluation, and GPTEval3D for 3D quality evaluation to evaluate their proposed method. Their method is effecient by using those meaningful evaluation criteria.", "theoretical_claims": "This paper does not include the theoretical discussion.", "experimental_designs_or_analyses": "I checked the soundness of the experimental designs and analyses. The main experiment demonstrates the effeciency of the proposed method compared with the other baselines with more delicate 3D generation. The authors also discussed the effect of different backbones, reward models, score gaps, pair examples, model design, and further applications in the experiment parts. I think the experiment is comprehensive and meaningful.", "relation_to_broader_scientific_literature": "N/A.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "* Strengths:\n1. This paper is well-written and well-organized. \n2. The research topic is insteresting and meaningful. Text-to-3D can be utilized in many practical applications such as games and designing.\n3. The proposed method, DreamDPO, is much more efficient than existing baselines, which generates more delicate 3D images.\n\n* Weaknesses:\n1. How about the computational efficiency of the proposed method compared with the other baselines?\n2. I am still confused about the determination of $x_t^{win}$ and $x_t^{lose}$. Do you assign the sample with lower loss to $x_t^{win}$ and the sample with higher loss to $x_t^{lose}$?\n3. If current 3D iterms generated from text-to-3D generation can be used in manufacturing? I ask this question because I am very curious about the practical application of this field.", "other_comments_or_suggestions": "Please see the weaknesses.", "questions_for_authors": "Please see the weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqCfXqrbUv", "reviewer": "Reviewer_kB21", "summary": "This paper introduces DreamDPO, an optimization-based framework for text-to-3D generation that aligns the generated 3D content with human preferences. Instead of relying on absolute quality scores from reward models, DreamDPO employs Direct Preference Optimization (DPO). The method operates in three iterative steps: 1) Pairwise Example Construction, 2) Pairwise Comparison, and 3) Preference-Guided Optimization, where a novel piecewise loss function, derived from the pairwise preference, guides the update of the 3D representation parameters. This piecewise loss is designed to prevent noisy gradients when the compared examples are very similar.\n\nThe paper demonstrates the proposed method's effectiveness through experiments on the GPTEval3D benchmark, comparing it against 13 existing text-to-3D generation methods. Quantitative evaluations using ImageReward, CLIP score, and GPTEval3D's metrics to show good results in terms of text-image alignment, 3D plausibility, and texture/geometry detail. Qualitative comparisons also support these claims. The paper also explores the use of MLLMs/LMMs for providing preference feedback. Ablation studies investigate the impact of different diffusion model backbones, reward models, and the score gap threshold used in the piecewise loss.\n\n\n## Update After Rebuttal\nI thank the authors for their detailed rebuttal. The provided quantitative comparison with DreamReward (Rebuttal Table 2-1) is appreciated and clarifies their relative performance. The authors also committed to better contextualizing their work regarding feed-forward methods.\n\nHowever, the justification for the $\\tau$ hyperparameter remains unclear, and the potential limitations of using 2D metrics for 3D evaluation were not fully resolved. Most importantly, the concern regarding the method's slow computational speed compared to state-of-the-art feed-forward approaches was not addressed.\n\nDue to these remaining issues, particularly the unaddressed speed limitation, my assessment has not changed.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the positive reviews. We provide our responses below.\n\n> **Q1:** Quantitative comparisons with DreamReward.\n\n**A1:** Thanks for your suggestion. We conduct a quantitative comparison on GPTEval3D to evaluate human preference, including the correction of numbers. In specific. we report the number accuracy and CLIPScore. The results below show that DreamDPO demonstrates competitive performance in prompt alignment and significant improvement in number correction.\n\n| Method       | Number Accuracy $\\uparrow$ | CLIPScore $\\uparrow$ |\n| ----------- | --------------- | ------------------ |\n| DreamReward | 41.7%             | 0.2855           |\n| DreamDPO    | 71.7%             | 0.2787           |\n\nTable 2-1: Quantitative comparisons of DreamDPO and DreamReward on GPTEval3D. We calculate the CLIPScore for prompt alignment and number accuracy for number correction.\n\nWe attribute this to that DreamDPO does not rely on precise reward scores, allowing it to leverage various black-box AI models for scoring, which helps correct numbers and attributes and improves human preference alignment.\n\n> **Q2:** The potential limitations on 2D metrics.\n\n**A2:** Yes. 2D image reward models are relatively mature, with abundant data and good reward performance. However, they have limitations. While they improve prompt alignment, such as correcting character attributes (e.g., changing \"man\" to \"knight\"), they struggle to address inherent Janus issues (e.g., characters with three legs). Our DreamDPO is flexible with both 2D and 3D rewards. Experiments in [Figure 3-1](https://imgur.com/a/XdVx2sn) demonstrate that it effectively mitigates Janus issues and some potential artifacts, offering a more reliable evaluation for 3D consistency.\n\n> **Q3:** More exploration of LMM capabilities.\n\n**A3:** Thanks. We have further analyzed the failure cases of DreamDPO with LMMs (see R2-Q2). Additionally, we demonstrate that increasing the number of comparison candidates can effectively mine positive samples, leading to improved performance with LMMs (see R2-Q3). Lastly, we show that DreamDPO works effectively across various LMMs (see R2-Q4).\n\n> **Q4:** Additional discussion with feed-forward methods.\n\n**A4:** Thanks. Aligning feed-forward methods with human preferences is meaningful but remains underexplored, primarily due to the significant computational resources required. DreamDPO offers valuable insights for feed-forward methods. Specifically, by constructing candidates with varying noise and using ranking-guided optimization, DreamDPO provides a potential pathway for RL optimization of feed-forward methods. We will polish the paper with better contextualize within the broader landscape of 3D generation.\n\n> **Q5:** Ablation studies on the score gap threshold in the 3D setting.\n\n**A5:** Thanks. Please kindly check R1-Q1.\n\n> **Q6:** More results with other SDS-based text-to-3D generation methods.\n\n**A6:** Thanks. We evaluate the performance of DreamDPO with different backbones in Section 4.3. In specific, we evaluate the performance of DreamDPO with Stable Diffusion v2.1 (SD2.1). The results demonstrate that DreamDPO works effectively with SD2.1 and achieves competitive results compared to ProlificDreamer. While SD2.1 shows improvements over the baseline, MVDream outperforms due to its superior 3D consistency. Therefore, we adopt MVDream as the default backbone.", "claims_and_evidence": "Overall, most of the claims made in the paper are backed up by good evidence. The results on the GPTEval3D benchmark (Table 1) and the visual examples (Figures 2, 3, 7, 11, 12) show that DreamDPO can generate reasonable 3D objects that match the text descriptions well. The piecewise loss function also seems to be important, as shown by Figure 6 and the explanation in Section 3.1. The experiments in Section 4.3 and Figure 8 demonstrate the flexibility of the method.\n\nThe paper does compare DreamDPO to DreamReward, but only qualitatively (Figure 7). While these visual comparisons are helpful, it would be much stronger to also include a quantitative comparison. This is a missed opportunity to directly compare DreamDPO to a closely related method that uses a different approach (a custom-trained 3D reward model).", "methods_and_evaluation_criteria": "The methods proposed in this paper show some promise for text-to-3D generation. The DreamDPO framework and the piecewise loss function are interesting ideas. However, there are some aspects of both the methods and the evaluation that could be strengthened. The reliance on pairwise comparisons and 2D image metrics, while common practice, introduces potential limitations that aren't fully explored, such as the gap between 2D-3D perception. More thorough ablation studies focusing on the piecewise loss would be beneficial. The evaluation, while using a standard benchmark, could be significantly improved by including direct 3D mesh quality metrics and, crucially, by providing a quantitative comparison with DreamReward. The current evaluation doesn't provide fully convincing evidence of the method's superiority, especially considering the reliance on rendered 2D views only.", "theoretical_claims": "This paper is primarily empirical and does not present theoretical claims or proofs.", "experimental_designs_or_analyses": "The use of the GPTEval3D benchmark and multiple metrics is generally a good approach, and the comparisons to a wide range of baselines are valuable. Still, the lack of a quantitative comparison with DreamReward is a significant issue. The ablation studies are helpful in understanding the impact of different components, but the exploration of LMM capabilities could be more extensive, and the justification of the $tau$ relies more on the 2D toy example. While the qualitative evaluations provide visual support, they are inherently subjective. Maybe a more thorough user study could strengthen the claims of improved human preference alignment.", "relation_to_broader_scientific_literature": "This paper connects its contributions to the broader literature on (SDS-based) text-to-3D generation and learning from human preferences.", "essential_references_not_discussed": "I did not find any essential references that were missing from the paper's discussion.", "other_strengths_and_weaknesses": "In addition to the points already raised, there are a few minor weaknesses. The discussion of limitations could be more thorough, particularly regarding the reliance on pre-trained models and 2D metrics. While the experiments analyze the impact, more justification about the hyperparameter selection would also make the paper clearer and stronger.\n\nCrucially, the paper is very slow compared to many state-of-the-art 3D generation methods that use feed-forward approaches. This is a major practical limitation that needs to be addressed more directly.", "other_comments_or_suggestions": "It would be helpful for the paper to better contextualize its approach within the broader landscape of 3D generation. While the SDS-based method using 2D diffusion models is a valid area of research, many of the current state-of-the-art results in terms of visual quality and speed are coming from feed-forward methods that use large-scale 3D models (diffusion or autoregressive) trained on massive 3D datasets. Mentioning these other approaches, and explaining why this paper focuses on the SDS-based method, would make the paper more complete.", "questions_for_authors": "1. The paper is using 2D image reward models (HPSv2, ImageReward) to evaluate 3D assets. Could you elaborate on the potential limitations of this approach? Are there specific types of 3D inconsistencies or artifacts that these 2D models might miss? Have you considered any ways to mitigate this domain gap (e.g., incorporating some form of 3D consistency check)?\n\n2. The piecewise loss uses a threshold of $tau = 0.001$. Figure 6 shows this works in 2D, but could you give a bit more explanation, or maybe some 3D-specific results, to justify this particular value?\n\n3. The paper employs MVDream as the backbone. Have you tried other SDS-based text-to-3D generation methods? How does the proposed method perform with them?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces DreamDPO, an optimization-based framework for text-to-3D generation that aligns outputs with human preferences via Direct Preference Optimization (DPO). The method constructs pairwise examples, ranks them using reward models or large multimodal models (LMMs), and optimizes 3D representations via a preference-driven loss. Experiments on the GPTEval3D benchmark demonstrate superior performance over 13 baselines in terms of text-asset alignment, 3D plausibility, and texture/geometry quality.

DreamDPO presents a novel, empirically validated approach to preference-aligned 3D generation, with clear improvements over reward-based methods (e.g., DreamReward). Most reviewers think that while slower than feed-forward alternatives, its flexibility (LMM/reward-agnostic) and controllability make it a valuable contribution.
Decision: Accept (poster)