Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment with 2D Rewards
Authors: Qingming LIU
Zhen Liu
Dinghuai Zhang
Kui Jia
Venue: NeurIPS 2025 poster ()
Keywords: 3D Generation
Diffusion Model
Alignment
Reward Finetuning
Reinforcement Learning
Primary area: applications
TL;DR: We extend a new RL-based gradient-informed finetuning method to the task of reward finetuning/alignment for 3D native diffusion models.

Abstract: Generating high-quality and photorealistic 3D assets remains a longstanding challenge in 3D vision and computer graphics. Although state-of-the-art generative models, such as diffusion models, have made significant progress in 3D generation, they often fall short of human-designed content due to limited ability to follow instructions, align with human preferences, or produce realistic textures, geometries, and physical attributes. In this paper, we introduce Nabla-R2D3, a highly effective and sample-efficient reinforcement learning alignment framework for 3D-native diffusion models using 2D rewards. Built upon the recently proposed Nabla-GFlowNet method for reward finetuning, our Nabla-R2D3 enables effective adaptation of 3D diffusion models through pure 2D reward feedback. Extensive experiments show that, unlike naive finetuning baselines which either fail to converge or suffer from overfitting, Nabla-R2D3 consistently achieves higher rewards and reduced prior forgetting within few finetuning steps.

Nabla-R2D3: Effective and Efficient 3D Diffusion Alignment
with 2D Rewards
Qingming Liu!.'
Zhen Liu
Dinghuai Zhang² Kui Jia!
'he Chinese University Hong Kong, Shenzhen MMicrosoft Research
'Equal contribution
'Corresponding author
Project page: nabla-r2d3, 122d...gthub....
NABLA-R2D3
Base 3D-Native Oiffusion Model
ALIGN
Finetuned 30-Native Diffusion Model
Texture
Geometry
Figure I: Our Nabla-R2D3 can efficiently and robustly finetune 3D-native diffusion models with differentiable reward models learned from human preferences on appearance, geometry and many other attributes.
Abstract
Generating high-quality and photorealistic 3D assets remains a longstanding chal- lenge in 3D vision and computer graphics. Although state-of-the-an generative models, such as diffusion models, have made significant progress in 3D generation, they often fall short of numan-designed content due to limited ability to follow instructions, align with human preferences, or produce realistic textures, geome- tries, and physical attributes. In this paper, we introduce Nabla-R2D3, a highly effective and sample-efficient reinforcement learning alignment framework for 3D-native diffusion models using 2D rewards. Built upon the recently proposed Yabla-GFlowe method, which matches the score function to reward gradients in a principled manner for reward finetuning, our Nabla-R2D3 enables effective adaptation of 3D diffusion models using only 2D reward signals. Extensive ex- periments show that, unlike vanilla finetuning baselines which either struggle converge suffer from reward hacking, Nabla-R21) -2Daa-22D3 consistently achieves higher rewards and reduced prior forgetting within few finetuning steps.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Recent advances in 3D generative models have enabled non-experts to produce batches of 3D digital assets at low cost for downstream tasks such as gaming, film, and robotics simulation. However, these assets are rarely production-reade and often require post-processing due to low visual fidelity, suboptimal geometry, ethical biases, and poor instruction following (in text-to-3D setups). These issues largely stem from training datasets that diverge from human preferences and 3D constraints.
A common solution is to first obtain a reward model that reflects human preferences and later perform reward finetuning on the generative model-a process commonly known reinforcement learning from human feedback (RLHF). Originally developed for aligning autoregressive language models, reward finetuning has also been successfully applied diffusion models, which are among the most popular generative models in the vision domain. It is therefore natural to consider applying similar techniques to 3D diffusion models.
However, extending RLHF to 3D diffusion models is non-trivial due to the lack of high-quality 3D reward models. In 2D settings, reward finetuning typically relies on 2D reward models; by analogy, 3D finetuning would require 3D reward models. Yet collecting diverse and high-quality 3D data remains longstanding challenge, making it difficult build reward models that reflect human preferences on attributes such as aesthetics, geometry, instruction following, and physical plausibility.
Inspired by the success of the "lifting from 2D" approach for 3D generation, where optimizes 3D shape such that each view is of high likelihood under a pretrained 2D generative model, we propose to similarly finetune 3D-native diffusion models [46, 61, 38, 23, 24, 44] with 2D reward models. Using a 3D-native model, we can sample different camera views and perform the lifting operation in an amortized fashion across training instances, rather than optimizing each object individually. However, sampling from 2D views yields high variance during optimization and can lead to instability and overfitting of 3D-native diffusion models. In light of state-of-the-art reward finetuning method called Nabla-GFlowNer [25], which proves to be highly robust, efficient and effective on 2D diffusion models, we propose Nabla-R2D3 (short for Reward from 2D for Diffusion Alignment in 3D) via Wabla-GFlowwhe which adapts this method to finetune 3D-native diffusion models with 2D) rewards. We empirically show that our method produces 3D-native models that are better aligned with human preferences and avoid major artifacts, such as unexpected floaters, commonly produced by prior methods. Furthermore, we demonstrate the effectiveness of different 2D reward models for aligning 3D-native diffusion models for different preferences.
We summarize major contributions in this paper below:
To the best of our knowledge, Nabla-R2D3 is the first method to effectively and robustly align 3D-native diffusion models with human preferences using only 2D reward models.
We demonstrate several examples of 2D reward models, including appearance- geometry-based ones, for aligning 3D-native generative models on different attributes.
and
Our extensive experiments show that, compared to the proposed vanilla reward finetuning effectively, efficiently and robustly finetune 3D-native
baselines, our Nabla-R2Ds generative models from 2D reward models with better preference alignment, better text- object alignment and fewer geometric artifacts.
2
Related Work
Reward Finetuning of Diffusion Models. The earliest attempt at reward finetuning for diffusion models, named DDPO [2], views the denoising process of diffusion models as trajectory sampling in Markov decision process (MDP), in which each state is a tuple of a noisy image and the corresponding time step; a sampled trajectory starts from a random Gaussian noise at time T and, through the iterative stochastic denoising process, reaches sample image at time O. With this MDP defined, DDPO leverages the classical policy gradient method in reinforcement learning (RL) to finetune diffusion models. As reward models are typically learned with neural nets and thus differentiable, DDPO does not effectively leverage the available first-order information in the reward model. To address this deep computational
issue, methods like ReFL. [48] and DRaFT [4] treat each sample trajectory graph and, by assigning reward values to the states, use back propagation to directly optimize model parameters with respect to reward signals in an end-to-end fashion. While efficient in practice, these

---

methods lack probabilistic grounding their training objectives are not designed to approximate the seward-weighter distribution and thus tend to overfit to the reward model. A parallel path is to adopt stochastic optimal control (SOC) that frames the alignment objective as an optimal control problem. The SOC approach in theory may achieve ideal results, but the proposed methods [42] so far are either ineffective or computationally expensive. Recently, new RL-baaed diffusion finetuning methods are proposed in the framework of generative flow networks [1, 58, 55, 27, 30, 56, 59, 52] (or GFlowNets in short) that builds generative models on directed acyclic graph to generate samples according to a reward distribution. The resulting finetuning methods, albeit derived in GFlowNet language, are also deeply rooted in (and in many cases equivalent to) soft Q-learning [9] in reinforcement learning. These new RL methods are constructed from first principles and are shown to be effective and efficient in creating finetuned diffusion models that generate diverse samples.
3D Generation via "Litting from 2D" Due to the scareity of 3D data, several works propose generating 31) shapes with only 2D signals. One early work in this direction is Dream Field [14], which initializes a 3D shape and optimizes the shape with CLIP scores on images rendered from randomly sampled camera views of the shape. Since the CLIP model is not generative model, it is hard, not impossible, to yield detailed appearance and geometry in generated 3D shapes. Following Dream Field, DreamFusion [31] was proposed to use score distillation sampling (SDS) that replaces CLIP scores with diffusion losses on rendered views. Such an approach is used not only for 3D object synthesis from scratch with 2D models, but also for texture generation known geometry [36]. 3D object synthesis with video generative models [15] and so on. However, the lifting approach is highly unstable and requires extensive hyperparameter tuning [31]. Another issue is the famous Janus problem that implausible 3D objects may be generated even if most of the 2D views reasonable- demonstaannnn the issue of the lack of 3D priors. In addition, these lfftingbbssed 3D generation methods take a long time to sample only one single object and are less suitable for downstream tasks due high computational cost. Our proposed method is free from these issues because it directly works on and infers from 3D-native diffusion models.
Alignment in 3D Generation. Apart from the per-sample alignment with SDS-based "lifting from 2D" approaches [51, 63, 13], probably the methods most relevant to ours are MVReward [43] and Carve3D [47], both of which finetune multi-view diffusion model with separate salli-vvee-bbssee reward model. These methods assume a multi-view representation of 3D objects, a representation that does not guarantee 3D consistency. Moreover, since this representation does not encode ground-truth normal or depth maps, external estimators must be employed to improve geometry using geometric rewards. Another line of 3D alignment is through direct preference optimization (DPO) [35, 63], where a model is finetuned on preference dataset and without any explicit reward model. While DPO is conceptually simple, alignment with a reward model is generally better [29] and applies to scenarios where we have analytical and/or expert-designed reward models. There is a recent trend of post-training alignment with test-time scaling [16, 28] to filter undesired samples or dynamically adjust sampling strategy during inference. Such strategy is applied to 3D generation [7], but it is typically costly and do not leverage the gradient information in reward models.
3
Preliminaries
3.1
Diffusion models and RL-based finetuning
Diffusion models are powerful class of generative models that generate samples through sequential denoising process. To be specific, it typically starts from time T with a point øT sampled from a standard Gaussian distribution, and gradually generate cleaner samples through a learned backward process p(z,-1 (x)) until it obtains the final sample To. The backward process p(r+-1 (Tt)) is trained to match a forward process g(xz) (xre((==))) 1). typically set simple Gaussian distribution. For instance, in a popular diffusion model DDPM [11]. the corresponding noising process is ((<2+1/<>) (x+) =
N /o///////// /OLTA, /1 - 0 J) and (( = (N/or vT- or/) with a noise schedule J. To train DDPM model on dataset D, we use the score matching loss:
Ez- ((. (I.....)).==N(..))).............
+ 11-1aace
d|2
where w(() is a weighting scalar, and cœ(xr, 1) is a neural net that predicts the noise vector from Iz. The stochastic sequential denoising process can be treated as a Markov decision process (MDP) in which (Ie, t) is a state, (Iz, (T) is the initial state, (zg, 0) is the final state, p( ( is the transition function. With such an MDP defined, can align the underlying diffusion model with any reinforcement learning algorithm [2] and optimize some terminal reward R(xo).
3

---

3.2 Diffusion alignment via gradient-informed RL finetuning
To preserve prior in the pretrained model Pbase (2x [x+1), a typical finetuning objective is to match the "tilted" reward distribution Phose (a)se()) Rt (x) where 3 controls the amount of prior information in the finetuned model. With the MDP defined in Sec. 3.1, it is shown that we may collect on-policy trajectories {(TT. .... To) from the finetuned model pe(zz; (x(+1) (where K is the batch size) and optimize some RL objective. A recent method called GellaeGFlowNett shows that we may efficiently and robustly finetune a diffusion model with "score-matching -iike" consistency losses:
Cforward (I=-1:t) Vx,- -1 log Po (2 (xx) 7.8V V 1--1 log (((0((((11)))
go( (2)
(It- = Ve, log (e (x) + N. V2, log (())(x))) + (z))))
(3)
and the terminal loss Ciemmiaa = ( (20) 12, where log log Ponse Phuse represents the log-density ratio between the finetuned and base models.. (2x))) = (xt (x:))) is the expected one-step prediction of æg (i.e., R[ro æx) under the finetuned model, cg(x)) ) the noise prediction network of the finetuned model (a, and o, denote the signal and noise scales from the forward process). V is the stop-gradient operation. 3 controls the relative strength the reward with respect to the prior of the pretrained model and "/ is the decay factor of the guessed gradient.
The total loss follows (where uy is a non-negative weighting scalar): Cerminal + >' + wy Lreverse)
Loaal = (u (orr (1..ooo (tsa (1)))))))
(4)
This loss is originally derived within the framework of GFlowNet a generative model with the MDP defined on directed acyclic graph in which the terminal states are sampled with probability proportional to the corresponding rewards. It is shown [25] that this special set of losses is indeed equivalent to a gradient version of the soft Q-learning loss [9] in the reinforcement learning literature.
Method
4.1 Efficient and Robust 3D Diffusion Alignment with 2D Rewards
Suppose that we have a 2D reward model R(zo) that maps some image xg the corresponding reward. We adopt common assumption in the text-to-3D literature: the 3D reward model can be derived from 2D reward model via multi-view rendering.
log (2))
E log R(h(zo, c)),
(5)
enl
where 20 a clean 3D shape, h(zo, e) is the rendering function that maps 2g to the image with camera pose e sampled from a set of camera poses C. Notice that if R(-) is approximated with the negative
diffusion loss with a pretrained 2D diffusion model, the 3D reward is basically the score distillation
sampling (SDS) objective in DreamFusion [31] (with slight differences in implementation):
log RAD, (20) c~,N (o.,( t-
We (2.) <ll?.
(6)
where EpD is the e-prediction network of the 2D diffusion model and w, is some weight factor.
Based on the above assumption, derive the following DD losses for finetuning 3D diffusion
models on 2D reward models (with the terminal loss staying the same as Luceminat (20) = |l (zo) 11
Liorward (orrard(24100 (2)-1:)) = E one
V log Po 12)))) NB
V Z 1 log R(h(29(2(-1).
(7)
Vz, log po (= + BV ZZ, log ((h(E)())),)))) + +0o(z))l?
Treverse (zt=
E. anl
(8)
Empirically, we find that the following approximate loss alone works well:
Tuppoux (2)-1:t)=
V = log (0 V
V - log
c)))
with which we assume that our educated guess of the "reward gradient" is accurate and therefore there is no need to learn the correction term gø with the reverse-direction loss anymore. We use this simple loss throughout the rest of the paper.

---

4.2 Practical 2D Reward Models for 3D Native Diffusion Models
2D rewards from appearance. It has been shown that RGB appearance in 2D views alone can support high-quality 3D generation, as demonstrated by lifting Setiin-ffrom-25 from-22 methods [31]. We follow recent methods SDS-based reward-guider 3D generation and consider the following reward models for finetuning 3D-native diffusion models: 1) Aesthetic Score [17], trained on the LAION-Aesther dataset [17] to measure the aesthetics of images, and 2) HPSv2 [45], trained on HPDv2 dataset [45] to measure general human preferences over image quality and text-image alignment.
2D rewards from geometry. In many cases, multiview RGB information can still fall short in creating fine-grained and 3D-consistent geometric details due to the lack of sufficient 3D priors or regularization. Inspired by the recent progress in single-view depth and normal estimation, we propose to explicitly encourage the consistency between rendered RGB images and the 3D geometric predictions inferred from those images. In our experiments, we employ state-of-thee ti-thhearr art normal map estimators and take as the reward the inner product between rendered normal maps (with approximate volumetric rendering for Gaussian splatting and NeRF representations) and the normal maps predicted from rendered RGB images:
(Moomaal (z, fnormal (/RGB (z, ct)
Ruornal (2) = E and
(10)
where Anarmal (z, c) and BRGB (2, denote the rendered normal map and rendered RGB images, respectively, from some 3D representation at camera pose e in some camera pose set C and f mnomall is some pretrained normal map estimator, for which we use the one-step normal map prediction model in StableNormal [50] in our experiments.
Inspire by [53] which proposes use depth-normal consistency improve the geometric quality of reconstruction, we also propose to use the Depth-Normal Consistency (DNC) reward:
(Irommalz,, c), T (haepth (z, c))
RDNC (z)
= IE onc
(11)
where the pseudo normal map T Thacopu (z, c)) can be computed by taking finite difference of 3D coordinates computed from the rendered depth map.
5
Experiments
5.1 Base model, baseline, metries and prompt dataset
Base models. We consider two state-of -fte-----hh-an- art and open-sourced base models: DiffSplat [18], available in two variants finetuned from PixArt-E [3] and StableDiifusion--i..5 [37] (SD1.5 for short), and GaussianCube [54]. We use 20-step and 50-step first-order SDE-DPM-Solver [26] for DifiSplat- Pixart-E and GaussianCuhe, respectively, and 20-step DDIM [39] for DittSplat-SabllDDiffuuionn Unless otherwise specified, we use the DiffSSplat -
[3] for experiments.
Baseline methods. With the 3D reward defined an expectation of 2D rewards, other alignment methods can be similarly applied once we use stochastic samples to compute 3D rewards. Specifically,
consider these baseline methods: 1) DDPO [2], which finetunes diffusion models with the vanilla policy gradient method, 2) ReFL [48], which directly optimizes R(2g(z) with a truncated
computational graph 2++1 - 2t En (22) and randomly sampled t, and 3) DRaFT [4], which
directly optimizes R((0)) with a truncated computational graph zK > ZK
- zo. For ReFL,
sample from 15 to 19; for DRaFT,
K 1.
Evaluation metric. Following alignment studies in 2D domains [25, 60, 6]. we consider three metrics: 1) average reward value, 2) multi-view FID score [10, 8] for measuring prior preservation and 3) multi-view CLIP similarity score [34] for measuring text-object alignment. The multi-view metrics for any given text prompt are computed by first computing metrics on each view and then taking the average over all views. Similarly, we compute metrics for each prompt and average them to obtain the final multi-view metrics. We use 60 unseen random prompts during the finetuning process for evaluation. For each prompt, we sample a batch of 3D assets (of size 32) to compute the metrics.
Prompt dataset. We use the prompt sets in G-Objaverse [32], high-quality subset of the large 3D object dataset Objaverse [5]. For experiments on geometry rewards, we filter out the prompts for which the base models yield very low reward values.
5

---

Ours
ReFL.
DRaFT
DDPO
430
CLIP Sianilarity (10-3)
Reward
250
25.0
58
150
22.5
206
17.5
100
200
300
902
600
0
100
205
300
410)
530
510
100
200
300
4001
500
500
Update Step
Update Stop
Update Step
Figure 2: Convergence curves metrics for different finetuning methods on Aesthetic Score. Our method achieves faster finctuning with better prior preservation and text-object alignment than ReFL and DRaFT, In addition, our method produces results with significantly lower variance in FID and CLIP similarity.
Ours
RcFL
DRaFT
DDPO
32.5
CLIP Similan
200
27.5
20.6
100
17.5
4.5
5.5
555
60
6.5
4.5
5.0
5.5
Reward
Reward
Figure 3: Trade-offs among reward maximization, prior preservation, and text-object alignment for various reward finetuning methods Aesthetic Score experiments. Each data point represents evaluation results of model checkpoints saved at every 20 finetuning iterations. Models with higher rewards, higher CLIP similarity scores, and lower FID scores are considered superior. Our Nabla-R2D3 shows better trade-off's between swward-improvement and other metrics and consistently outperforms the baselines.
Implementation Details. We follow [25] and regularize the gradient updates: Lreg
A lleo (Ix) Egt (Iz) where øf is the diffusion model parameters in the previous update step and À is a positive scalar. We set À 3e3, 5e3, le4 for Aesthetic Score, HPSv2 and Geometry Reward respectively. During training, we sub-sample 40% of the transitions from each collected trajectory. For HPSv2 and Aesthetic Score experiments, we set the reward temperature 3 to 2e6 and le7, respectively; for geometry rewards, four orthogonal views (front, left, back and right) with randomly sampled elevation +:20 and then apply azimuthal perturbations by adding random offsets within predefined range +.60%. We use learning rate of 10 4 for ReFL, DDPO, DRaFT and Nabla-R2D3, The rest of the implementation details are elaborated in the appendix.
set to le6. To sample camera views c, we first sample
5.2 Results
General experiments. In Tab. 1, show the metrics (average over random runs) of models finetuned on different reward models with different finetuning methods. Our Nabla-R2D3 is shown be capable of achieving the best reward value the fastest speed, and the same time preserving the prior from the base model plus text-object alignment. We show in Fig. the evolution different metrics (both the mean value and the standard deviation) for different methods. As higher rewards inevitably lead to worse prior preservation and text-object alignment, we illustrate which method achieves the best trade-off by presenting the Pareto frontiers of all methods. Specifically, we plot the results from different checkpoints saved at various fine- teninunnng iterations of each independent run Fig. 4, and 6, we visualize the generated
(with different random seeds) in Fig. 3. Furthermore, assets with the corresponding reward values from different finetuning methods and demonstrate that our method can qualitatively yield better alignment for various reward models meaningful for generation. To illustrate that our method leads to more robust finetuning, we show in Fig. 7 the assets generated using the same random seeds with models finetuned with Nabla-R2D3 and DRaFT. The shape with the DRaFT-finctuned model exhibits the severe Janus problem, where rendered multi-view

---

Egyptian bead stone base.
colorful coffee cup and with spoon, straw, and Swiss content design.
Green and gold Celtie treasure chest with intricate designs.
Mario lab coat holding pill.
Prompt
Pretrained Model 4.70
Ours
ReFL 5,02
DRaFT 5.30
DDPO 4.76
Figure Qualitative comparison of 3D assets produced by models finetuned with different methods Aesthetic Score. show for each method on the the average reward the visualized assets. For fair comparison, we pick the model checkpoints that generate the highest rewards but without significant overfitting patterns.
images display inconsistent or contradictory characteristics from different viewpoints, while the one from Nabla-R2Ds Sala--R233ffineee finetuned model does not.
Table 1: Quantitative comparison between our method and the baselines on different reward models. Since it would be possible to trade FID with reward, we further present what the rewards if FIDs are similar (with early stopping) in Tab. 5.
Normal Estimator
Reward
FID,
CIIP--iin
Reward
FID.
CIIPP-Sim >)
Roward((0-
FIDI
CLIP-Sim f(10
Base Model
4.72
53.26
34.58
22.86
35.26
34.58
89.48
55.26
34.58
RaFi.
5.82
352.97
21.80
24.92
274.44
32:40
90.60
112.06
33.90
DDPO
4.54
172.95
32.44
22.23
69.59
31.35
$9.45
63.93
34,56
DRaFT
5.5]
337.77
22.89
32.65
224.64
33.99
91.11
296.23
28.
35.35
92.03
34.18
DiniSplat-SDL.
GaussianCube
Prompt
three-ffe tier cake with icing sprinkles.
A Chinese Sakura honsai sculpture with pink flowers on a wooden hase.
three- tier cake with icing and sprinkles.
Chinese Sakum bonsai sculpture with pink flowers on a wooden base.
Pretrained Model
Ours
Figure 8: Finetuning results on two base models, DifISplat-SD1.5 [18] and GaussianCube [54]. Our method generalizes well to models beyond DifSSpliat-ixAm--...
Different 3D-native generative models. We experiment with different base models, including Dif1 [18] and GaussianCube [54], to show that our method is universally applicable. Our
7

---

A black and gold soccer trophy.
A low poly snowmam with top hat, scarf, and outstretched
A green and pink Pokémon holding flower, with a flower
A pink and blue teupol with dark base.
Prompt
Pretrained Model 21.91
Ours 27.07
RellL 24.73
DRaFT 32.07
22.94
Figure Quantitative comparison HPSv2 [45]. For cach object we present the front and back views. Prompts highlighted red indicate unsuccessful instruction following by the base models. We further show the severe Janus problem of DRaFT in Fig. 7.
possibly bear roboi, standing on blue buse.
Spiderman piagy hank, with spides, and homs.
silver tcapot.
Prompt
Pretrained Model
93.81
91.26
DRaFT 91.59
DDPO 88.01
Figure 6: Quantitative comparison of different finetuning methods the normal estimator reward (Eqn. 10). Left: front-via view RGB image. Middle: front-view rendered normal map. Right: Zoomed details.

---

DRaFT
Ours
Figure 7: 360 visualization of 3D shapes generated by models finetuned with Nabla-R2D3 and DRaFT. DRaFT-finctuned model is prone to overfitting and suffers from the Janus problem.
Table 2: Comparison of finetuning methods on different base models with the Aesthetic Score reward.
DiffSplat-SD1.5
GaussianCube
Method
Reward
FIDI
CLIP-Sim 1(10-))
Reward
FID +
CLIP--im f (10-3)
Base Model
4.81
67.76
34.66
4.40
64.22
28.60
ReFL
6.08
396.79
20.66
5,50
296.46
17.31
DDPO
4.80
54.79
34.61
4.06
230.48
18.90
DRaFT
395.84
19.08
352.16
15.52
154.61
30.95
234.96
22.57
results (Fig. 8 and Tab. 2) show that our model consistently outperforms other finetuning methods and delivers desirable 3D assets.
Comparison with 2D-SDS-based lifting align-
Table 3: Comparison with 3D-SDS.
ment method. We compare in Fig. 10 our
Method
Reward
FID | CLIP-Sim f
method with DreamReward [51], method that
incorporates reward gradients into 2D-liftingg- based 3D generation. Our method produces more visially-desirable shapes not only because 2D-lifiing methods are less robust at synthesiz- ing 3D shapes [31, 44] but also because the 3D-native generative model provides more 3D prior.
3D-SDS (1 = 3)
5.38
194.98
0.31
3D-SDS (1
5.27
97.99
0.34
Ours (200 steps)
5.29
114.45
0.32
Ours (600 steps)
6.24
205.16
0.26
Comparison with native 3D prior guided SDS baseline. To underscore the benefit of inference with finetuned 3D na-ivtive diffusion model compared to the SDS based sampling approaches, we experiment with SDS on the DiffSplat base model with 3D diffusion loss plus the multi-view 2D reward model of Aesthetic Score: VL <3D(21,) t)
v/ so log Rap(20) with reward strength SDS approach indeed yields worse appearance and geometry, even with better 3D prior from the base 3D-native diffusion model and increasing the strength 1j does not improve results (Tab. 3). Moreover, running the 3D-SDS inference takes around minutes, wherear inference with the finetuned model takes only around 8 seconds.
7). We observe (Fig. 9) that
Pretrained Model
3D-SDS n n-1
3D-SDS 1 3
Ours (200 step)
Ours (600 step)
piselssed model wearing strow
Spidemmas piggy spider, webs,
Prompt
Model (ID-aativ)
2D-SDS
Ours
Figure 9: Comparison with 3D-SDS baselines on Aesthetic Score. We show two opposite views of the presented assets.
Figure 10: Comparison with 2D-SDD-based alignment method [51].

---

Blue and gold steampunk telescope on stand.
red and blue gouse statue base.
silver leaput.
Prompt
Pretrained Model
81.00
Ours
Figure 11: Qualitative comparison with pretrained model on DNC reward.
Results on DNC reward. We show the results of finetuning with different methods on DNC reward in Tab. 4. Our method achieves the best reward im- provement without incurring much loss in FID and Clip-similarity In Fig. 11, we visualize the geometry from the pre- trained model and the model finetuned on the DNC reward. The results demonstrate that the DNC reward, independent of any priors from
Table 4: Comparison between the base model and the one fine-
tuned with different methods on Depth-Normal Consistency.
Method
Reward? (10 ?)
FID
CLIP--inn (10
Base Model
87.81
$5.26
34,58
ReFL
89.15
158.19
32,74
DDPO
87.99
69.12
34.48
DRaFT
88.13
74.82
34,52
Ours
89.53
99.01
34.08
external normal estimators, can improve the sample geometry quality of 3D-native diffusion models. We further show the error map for the base model and the model finetuned with our method in Fig. 13.
Visualization of the evolution process. We visualize the evolution (every 50 update steps) of our method in the Aesthetic Score experiments (Fig. 12). We use the same seed, prompt, and initial noise, and we render the generated assets from the identical camera pose.
Pretrained
Figure 12: Visual evolution of the generated object with Figure 13: Results on DNC reward. From left to right: the same random seed during the finetuning process. rendered RGB image, rendered normal map, depth- induced normal map, and the corresponding error map.
Discussions
Reward finetuning test-time scaling. One may incorporate reward signals during inference using extra computational resources, a strategy known as test-time scaling [52]. Reward finetuning be treated amortized process such that does not have pay the typically high cost of reward evaluation. Moreover, reward finetuning benefits from implicit regularization during network training, particularly in the case of LoRA finetuning [12].
Limitations. Our finetuning method suffers from the same issues as the fiftingffrom-22 approaches: no supervision for shape inner structures. Our method requires expensive gradient computations during the forward pass, underscoring the need for improved numerical algorithms and architectural designs for finetuning [33, 21.. Furthermore, our method focuses solely on parameter-leve alignment and does not explore prsedt-based alignment strategies [52].
7 Conclusion
We propose an efficient alignment method, dubbed Nabla-R2D3, for finetuning 3D-native generative methods with 2D differentiable rewards in manner that avoids overfitting issues commonly seen in lifting-from-211 from-2D approaches. We demonstrate that Nabla-R2D3 outperforms baseline methods across both appearance- and geometry- based reward models, well across different base architectures. The development of better alignment methods for diffusion models, including this work, contributes toward constructing virtual worlds aligned with human values.
10

---

Acknowledgements
This project supported by Key-Area Research and Development Program of Guangdong Province, China under Grant 2024B0101040004.
References
[1] Yoshua Bengio, Salem Lahlou, Tristan Deleu, Edward Hu, Mo Tiwari, and Emmanuel Bengio. GFlowNet foundations. Journal of Machine Learning Rexearch, 2023.
[2] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. In ICLR, 2024. 2,3,5
[3] Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt-E: Weak-to-Strong Training of Diffusion Transformer for 4K Text-to-Image Generation. In ECCV, 2024. 5,
[4] Kevin Clark, Paul Vicol, Kevin Swersky, and David Fleet. Directly fine-tuning diffusion models on differentiable rewards. In ICLR, 2024. 2,5
[5] Matt Deitke, Dustin Schwenk. Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, Ali Farhadi. Objaverse: A universe annotated 3d objects. In CVPR, 2023.
[6] Carles Domingo-Enrinn Michal Drozdzal, Brian Karrer, and Ricky T. Q. Chen. Adjoint matching: Fine- tuning flow and diffusion generative models with memoryless stochastic optimal control. In ICLR, 2025. 5, 17
[7] Yuan Dong, Qi Zuo, Xiaodong Gu, Weihao Yuan, Zhengyi Zhao, Zilong Dong, Liefeng Bo, and Qixing Huang, Gpld3d: Latent diffusion of 3d shape generative models by enforcing geometric and physical priors. In CVPR, 2024.
[8] Jun Gao, Tianchang Shen, Zian Wang, Wenzheng Chen, Kangxue Yin, Daiqing Li, Or Litany, 7an Gojeie, and Sanja Fidler. Get3d: A generative model of high quality 3d textured shapes learned from images. In NewrIPS, 2022.
[9] Tuomas Haarnoja, Haoran Tang, Pieter Abbeel, and Sergey Levine. Reinforcement learning with deep energy-based policies. In ICLR, 2017. 3,4
[10] Martin Heusel. Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a time-scale update rule converge to local nash equilibrium. In NeurlPS, 2017. 5
[11] Jonathan Ho, Ajay Jain, and Pieter Abheel. Denoising diffusion probabilistic models. In NewrlPS, 2020. 3
[12] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. 10, 18
[13] Savva Victorovich Ignatyev, Nina Konovalova. Daniil Selikhanovych, Oleg Voynov, Nikolay Patakin, Ilya Olkov, Dmitry Semushkin, Alexey Artemov, Anton Konushin, Alexander Filippov, Peter Wonka, and Evgeny Burnaev. A3d: Does difffusion dream about 3d alignment? In ICLR, 2025. 3
[14] Ajay Jain, Ben Mildenhall, Jonathan T Barron, Pieter Abbeel, and Ben Poole. Zero-shot text text-guided object generation with dream fields. In CVPR, 2022. 3
[15] Hyeonho Jeong. Jinho Chang, Geon Yeong Park, and Jong Chul Ye. Dreammotion: Space-time self-similar score distillation for zero-shot video editing. In ECCV, 2024. 3
[16] Sunwoo Kim, Minkyu Kim, and Dongmin Park. Test-time alignment of diffusion models without reward over-optimization. In ICLR, 2025.
[17] LAION. Laion aesthetic score predictor. https://laion http:::/laaoooaa/bll//aaaooo htt.o//aaaaab/o/llaaaeeeeeeteiie cessed: 2024-09-22 09227.99255
Ac-
[18] Chenguo Lin. Panwang Pan, Bangbang Yang, Zeming Li, and Yadong Mu. Diffsplat: Repurposing image diffusion models for scalable 3d gaussian splat generation. In ICLR, 2025. 5, 7, 18
[19] Yaron Lipman, Ricky TQ Chen, Heli Ben HamuH Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In ICLR, 2023. 17

---

[20] Jie Liu, Gongye Liu, Jiajun Liang. Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan. Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505.05477 2025. 17
[21] Weiyang Liu, Zeju Qiu, Yao Feng. Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In ICLR, 2024. 10
[22] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. ICLR, 2023. 17
[23] Zhen Liu, Yao Feng, Michael Black, Derek Nowrouzezahrai, Liam Paull, and Weiyang Liu, Meshdiffu- sion: Score- based generative 3d mesh modeling. In ICLR, 2023, 2
[24] Zhen Liu, Yao Feng, Yuliang Xiu, Weiyang Liu, Liam Paull, Michael J. Black, and Bernhard Schölkopf. Ghost on the shell: An expressive representation of general 3d shapes. In ICLR, 2024. 2
[25] Zhen Liu,, Tim Z. Xiao, Weiyang Liu, Yoshua Bengio, and Dinghuai Zhang. Efficient diversity-preservin preserving diffusion alignment via gradient-informee gflownets. In /CLR, 2025. 2, 4, 6, 17
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. NewrIPS, 2022. 5
(27] George Ma, Emmanuel Bengio, Yoshua Bengio, and Dinghuai Zhang. Baking symmetry into gfoownets. arXiv preprint arXiv:2406. 05426, 2024. 3
[28] Nanye Ma, Shangyuan Tong. Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang. Xuan Yang, Yandong Li. Tommi Jaakkola, Xuhui Jia, al. Inference- time scaling for diffusion models beyond scaling denoising steps. arXiv preprint ar0iv:2501 009732, 2025.
[29] Qinwei Ma, Jingzhe Shi, Can Jin, Jenq-Neng Hwang, Serge Belongie, and Lei Li. Gradient imbalance in direct preference optimization. arXiv preprint urXiv:2502.20844 2025. 3
[30] Ling Pan, Dinghuai Zhang, Moksh Jain, Longbo Huang, and Yoshua Bengio. Stochastic generative flow networks. UAI, 2023. 3
[31] Ben Poole, Ajay Jain, Jonathan Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023. 3, 4, 5,9
[32] Lingteng Qiu, Guanying Chen, Xiaodong Gu, Qi Zuo, Mutian Xu, Yushuang Wu, Weihao Yuan, Zilong Dong, Liefeng Bo, and Xiaoguang Han. Richdreamer: A generalizable normal-depth diffusion model for detail richness in text-to-3d. In CVPR, 2024.
[33] Zeju Qiu, Weiyang Liu, Haiwen Feng. Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In NeurlPS, 2023. 10
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 5
[35] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon. and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model. In NeurIPS, 2023.
[36] Elad Richardson, Gal Metzer, Yuval Alaluf, Raja Giryes, and Daniel Cohen-Or. Texture: Text-guided texturing of 3d shapes. In ACM SIGGRAPH 2023 conference proceedings, 2023.
[37] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent difffusion models. In CVPR, 2022.
[38] Yawar Siddiqui, Antonio Alliegro, Alexey Artemov, Tatiana Tommasi, Daniele Sirigatti, Vladislav Rosov, Angela Dai, and Matthias NieBner. Meshgpt: Generating triangle meshes with decoder-only transformers. In CVPR, 2024. 2
[39] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In ICLR, 2021. 5
[40] Yang Song, Jascha Sohl- bib--Dicsttei, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In ICLR, 2021. 17
12

---

[41] Jiaxiang Tang, Zhaoxi Chen, Xiaokang Chen, Tengfei Wang, Gang Zeng, and Ziwei Liu. Lgm: Large multi-view gaussian model for high-resolution 3d content creation. arXiv preprint 2Xii:2402.000544 2024. 18
[42] Masatoshi Uchara, Yulai Zhao, Kevin Black, Ehsan Hajiramezanali, Gabriele Scalia, Nathaniel Lee Diamant, Alex M Tseng, Tommaso Biancalani, and Sergey Levine. Fine-tuning of continuous-time diffusion models as catropy-reguarizee control. arXiv preprint xxXi:24402.1514,
2024. 2024,
[43] Weitao Wang, Haoran Xu, Yuxiao Yang, Zhifang Liu, Jun Meng, and Haoqian Wang. Mvreward: Better aligning and evaluating multi-view diffusion models with human preferences. In AAAI, 2025. 3
[44] Zhengyi Wang. Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurlPS, 2023. 2, 9
[45] Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid henchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint 2306 09341, 2023. 8,8
[46] Jianfeng Xiang, Zelong Lv, Sicheng Xu, Yu Deng, Ruicheng Wang, Bowen Zhang, Dong Chen, Xin Tong, and Jiaolong Yang. Structured latents for scalable and versatile 3d generation. arXiv preprint WwXiv22412.01506 2024. 2
[47] Desai Xie, Jiahao Li, Hao Tan, Xin Sun, Zhixin Shu, Yi Zhou, Sai Bi, Sören Pirk, and Arie E Kaufman. Carve3d: Improving multi-view reconstruction consistency for diffusion models with rl finetuning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6369-6379,
2024. 3
[48] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Neur!PS, 2023. 5.5
[49] Zeyue Xue. Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo. Weilin Huang, al. Dancegrpo: Unleashing grpo visual generation. arXiv preprint arXiv:2505.07811 2025. 17
[50] Chongjie Ye, Lingteng Qiu, Xiaodong Gu, Qi Zuo, Yushuang Wu, Zilong Dong, Liefeng Bo, Yuliang Xiu, and Xiaoguang Han. Stablenormal: Reducing diffusion variance for stable and sharp normal. ACM Transactions on Graphics (TOG), 2024. 5
[5i] Junliang Ye. Fangfu Liu, Qixiu Li. Zhengyi Wang. Yikai Wang, Xinzhou Wang. Yueqi Duan, and Jun Zhu. Dreamreward: Text-to-3d generation with human preference. In ECCV, 2024. 3,9
[52] Taeyoung Yun, Dinghuai Zhang, Jinkyoo Park, and Ling Pan. Learning sample effective and diverse prompts for text-to-image generation. In CVPR, 2025. 3, 10
[53] Baowen Zhang, Chuan Fang, Rakesh Shrestha, Yixun Liang, Xiaoxiao I Long, and Ping Tan. Rade-gs: Rasterizing depth in gaussian splatting. arXiv preprint arXiv:2406.01460 2024. 5
[54] Bowen Zhang, Yiji Cheng, Jiaolong Yang, Chunyu Wang, Feng Zhao, Yansong Tang, Dong Chen, and Baining Guo. Gaussiancube: Structuring gaussian splatting using optimal transport for 3d generative modeling, arXiv preprint arXiv:2403. 19655, 2024. 5, 7,
[55] Dinghuai Zhang, Ricky Q Chen, Nikolay Malkin, and Yoshua Bengio. Unifying generative models with CFlowNets and beyond. arXiv preprint wtw.:i22299000000000 2022. 3
[56] Dinghuai Zhang, Ricky TQ Chen, Cheng-Hao Liu, Aaron Courville, and Yoshua Bengio. Diffuusion generative flow samplers: Improving learning signals through partial trajectory optimization. In ICLR, 2024. 3
[57] Dinghuai Zhang, Hanjun Dai, Nikolay Malkin, Aaron Courville, Yoshua Bengio, and Ling Pan. Let the flows tell: Solving graph combinatorial problems with gilownets. In NeurlPS, 2023.
[58] Dinghuai Zhang, Nikolay Malkin, Zhen Liu, Alexandra Volokhova, Aaron Courville, and Yoshua Bengio. Generative flow networks for discrete probabilistic modeling. In ICML, 2022. 3
[59] Dinghuai Zhang, Ling Pan, Ricky T. Q Chen, Aaron Courville, and Yoshua Bengio. Distributional GFlownets with quantile flows. Transactions Machine Learning Research, 2024. 3
13

---

[60] Dinghuai Zhang, Yizhe Zhang, Jiatao Gu, Ruixiang Zhang. Joshua M. Susskind, Navdeep Jaitly, and Shuangfei Zhai. Improving GFlownets for text-to-image diffusion alignment. Transactions Machine Learning Research, 2025.
[61] Longwen Zhang, Ziyu Wang, Qixuan Zhang. Qiwei Qiu, Anqi Pang, Haoran Jiang. Wei Yang. Lan Xu, and Jingyi Clay: controllable large-scale generative model creating high-quality 3d assets, ACM Transactions on Graphics (TOG), 2024. 2
[62] Qinqing Zheng. Matt Le, Neta Shaul, Yaron Lipman, Aditya Grover, and Ricky TQ Chen. Guided flows for generative modeling and decision making. arXiv preprint arXiv:2311.13443, 2023. 17
[63] Zhenglin Zhou, Xiaobo Xia, Fan Ma, Hehe Fan, Yang. and Tat-Seng Chua. Dreamdpo: Aligning text-to- 3d generation with human preferences via direct preference optimization. arXiv preprint arXiv:2502.04370.
2025. 3
14

---

Appendix
Table of Contents
A Overall algorithm
16
B Proof of Unbiasedness of Babla-R2D3
17
C Application to Flow Matching Models
17
D Implementation Details
18
E Comparison with Multi-view Generative Model
18
F More Ablation Studies and Visualization
19
G More Qualitative Results
20
Failure Cases
23
15

---

A Overall algorithm
Algorithm 1 3D-Native Diffusion Alignment with 2D Rewards using Nabla-R2D3
Inputs: Pretrained diffusion model with sampling probability Prese ( 2D reward model R(-)
2: Initialization: Model to finetune with sampling probability po(x)(2++1) where O
3: Sample the initial batch of trajectories Dprvv {(TT,,... .... Toli I==.... with the current finetuned diffusion model po.
4: Set HJ + A
5: while not converged do
6:
Sample a batch of trajectories Deum = {(2T, ... =n);): I...N with the finetuned diffusion model.
Subsample the time steps to train with: the full set T, {0, .... 1} or the
sampled set 7, = Sample-N((10) ((0, .... - 1}) where Sample-N is some unbiased sampling algorithm to randomly pick N samples.
8: Append the last timestep / Sample a set of camera poses Cı, .... CM-
the sample trajectory set above.
9:
10:
Compute the loss (ʃ is the output of the diffusion denoising network)
M(N+1) ΣØ= ELen.
V[ Po - w.3y 2. log ((M(202)))
+ All.fo (2++1) for (11+1))12
11:
Set 01 +- 0.
12: Gradient update with the loss function.
13:
Set Dprev Dear.
14:
end while
15: return finetuned model fø.
16

---

B Proof of Unbiasedness of Nabla-R2D3
We start with the original Nabla-GFlowNee forward loss (Eqn. for random variable 2 but with the 3D reward specified in Eqn. 5:
L
V a-1
log (2 now Vnni and
log ((h((2(((2)))))))) g0 (0()))
(12)
The corresponding gradient is
Vol. - Vo x log 7B V=-- E one
log R(M(E(((-1))0)) 80(=1-1)
V8 V
log (1-1
VoV sc log Do (2 NBV VVn. log ((h(26((2-1)))))) -0(((-1)
E enc
V 2 (0- 120) 2 0
a (2t-
(13)
which proves the unbiasedness of the proposed loss in Eqn. 7, The proof for the reverse loss is similar.
C Application to Flow Matching Models
As discussed in prior works [6. 25], the popular generative model flow matching [19] that samples 31 xx() via v(x, f), zo =00) -N(0,, I) can be turned into an equivalent diffusion model (but with a non-linear noising process). The sampling (denoising) process of this equivalent diffusion model is:
dX, [o(Xin))
xtt)²
de Xx dt dB) Xo ~o~N((0,1) (o(Xi,t) at
28:
a du 3, - B.)
(14)
where (t)) is an arbitrary diffusion term. We may therefore use Nabla-R2D3 to obtain finetuned diffusion model (and therefore the corresponding probability flow ODE [40]) from pretrained flow matching model.
For the special case of rectified flows [22] with .N I), the velocity field v(x, t) and the probability flow p(z, t) are related via the following formula ([62), Lemma 1):
v Vlogp(,,1) - [ 1
o(z, 0)
(15)
The corresponding reverse process of the equivalent diffusion model therefore [20, 49]:
oî (xr ()( + owdw
dx u(z
(16)
2t
17

---

D Implementation Details
General experiments. We use LoRA [12] parametrization with rank of 16 on all attention layers plus the final output layer for DiffSplat-Pixart-- [3]. and rank of 8 for DiffSplat-SDI. 1.5 [18] and GaussianCube [54]. The CFG scales are 7.5, 7.5, and 3.5 for DiftSplat-Pixart- DiffS SD1.5, and GaussianCube, respectively. All experiments were conducted with either two Nvidia Tesla V V100 GPUs or GeForce GTX 3090 GPUs. takes no more than one day finetune models with our method and all other baselines.
3D SDS. During 3D) SDS optimization, sample timesteps [0, 400) and train the object for 1000 steps. For each step, the rewards are evaluated from four randomly sampled views. To compute test metries, we sample 32 objects for each of 30 selected prompts, compute the mean per-prompt metrics and take the metrics averaged over all prompts.
E Comparison with Multi-view Generative Model
To emphasize the importance 3D native representations for 3D consistency after finetuning, we use method to finetune MVDream, multi-view diffusion model, on Aesthetic score and compare the results with those with DiffSplat. The generated multi-view images are passed to Large Multi-view Gaussian Model (LGM) [41] to build reconstructed 3D objects that they can be directly compared to 3D objects generated by 3D native models. Qualitative comparisons are presented in Fig. 14. The results from MVDream exhibit noticeable artifacts (illustrated in the green boxes in the figure), whereas the objects produced by the finetumed 3D native models do not.
Procpet: bear souttaae
MV MVDream
Proerpt: A pink and bike topor with dark beso.
MVDream
Ours
Figure 14: Qualitative comparison with MVDream. For MVDream, show the multi view images directly generated by the finetuned model the gray dashed boxes; all other images MVDream are rendered from the objects reconstructed by LGM (with the corresponding generated multi-view images).
18

---

More Ablation Studies and Visualization
Effect of different reward temperatures. We experiment with different temperature values 3 {5e5, 1c6, 2c6). and observe in Fig. 15 that higher temperature leads faster convergence at the cost of worse text-object alignment and prior preservation. Since previous experiments (Fig. 2) showed that the variance of our method's 's metric is minimal, the statistics for the standard deviation (std) are
omitted here.
_ 265
B leb
2eb
CLIP Similarity (10-5)
150
Reward
6:2
28
26
54
100
400
302
530
500
Update Step
Update Step
Update Step
Figure 15: The relationship between the temperature parameter and Reward, FID scores, and CLIP- CLIP-Similarity scores. Higher values of B result in faster convergence, but at the cost of worse text-object alignment and diminished prior preservation.
Effect of different learning rates. As illustrated in Fig. 16, higher learning rates lead to faster convergence, but with compromises in prior preservation and text-object alignment.
Learning Rate 1o-4
Leaming Rate So-5
200-
Reward
6.
5.5
100-
200
400
500
800
203
400
600
800
200
400
600
800
Update Step
Update Step
Update Step
Figure 16: Convergence
of metrics different learning rate.
Comparison of metrics given the same FID level on HPSy2 Reward. We show the results on HPSv2 (with DiffSplat-Pixant Σ) in Tab. where we select checkpoints with roughly the same FID. Specifically, we pick checkpoints for models finetuned with ReFI. and DRaFT such that their FIDs are approximately equal that of our model Tab. 1. The results clearly show that, given a similar FIDD level, our method achieves higher reward and CLIP-Sim scores.
Table 5: Results on HPSv2 (with DiftSplat-Pixart-)) ).
Method
Reward 1110-2)
CLIP-Sim
FID
ReFL
22.18
34.11
127
DRaFT
25.65
34.41
143
Ours
27.85
35.35
131
19

---

G
More Qualitative Results
360 videos of the generated objects can be found our project website: https://nallx-R233...g gi...guu...............
Pretrained Model
Ours
Figure 17: More qualitative results on Aesthetic Score.
20

---

Pretrained Model
Ours
Procup, A Souch Park carkoon character diving green military jeap with a bat on
Prompe Mickey candy apple, piselated
popsicle
Prompe: Seientisl clumaser with lair and lui, conducting experimnts table plant and beaker.
Prempe: Yellow sphrre clemazte with black hat and slees,
Procup: Pupplc green loy on pupple
Prompe: A dragon head with yellow eyes,
Proene:: warting husdic trimgle
Figure 18: More qualitative results on HPSv2.
21

---

Pretrained Model
Prompt: Wooden tible with blue cup
Prompt: Lov-puly Chridmas tnoz with prosetts and bows.
Prompt: of pears podestal gray cylinder
Prompt: A tall glves display sturd with a logo ind boond sign.
Prompt: Perple and green tyy rebor purple base.
Figure More qualitative results on the normal estimator reward.
22

---

H
Failure Cases
Similar to alignment for the 2D image domain, our method still suffers from issues: 1) imperfect rewards and 2) reward hacking. The first one can easily be observed in our normal-estimator based reward model, which inevitably will hallucinate non-existing geometry based on single-view RGB cues (Fig. 20). If we finetune the target diffusion model for too long, reward hacking becomes apparent as the generated shapes tend to over-optimize the rewards the way that the natural geometry and semantics are gradually forgotten (Fig. 21).
Prompt: A Farbello Gold grape juice carton with grapes on
Prompt: Scientist character with green hat, conducting experiments on table plant and beaker.
Figure 20: Failure cases the normal estimator reward. Left: rendered RGB image, Middle: rendered normal map. Right: estimated normal map. estimator produces wrong normal maps therefore guides diffusion model to generate wrong geometry.
Prompt: green and blue striped toy Ilama.
Prompt: goose on
Figure 21: Failure cases of the Aesthetic Score (first row) and HPSv2 (second row). The left column shows results from the finetuned model, while the right column presents results from the pretrained model. In both cases, the animals become unnatural after finetuning.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. Our main contributions are also detailed in Sec. 1. Also see Sec. 5.2 for more experimental evidence.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Yes, please see Sec. for limitations.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

Justification: We provide proof to the unbiasedness of the proposed loss that use 2D rewards in Appendix. All other formula are justified in reference papers.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper fully discloses necessary information required to reproduce the main experimental results in the Sec. 5 and Appendix. D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have released the whole set of code, instructions and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/ Bellliicggiies//odeSuuu
www.sssssiiiiiiiicooocoiiiiiiiic for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: Cooee/oodeeeeeeeeeeeeee / cttpsssccppbbccccc/pppcppp /otaa//ge/deeeeeeeeebaaiiiiii issiisssion onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
The paper provides detailed descriptions of all experimental setups in Sec. and Appendix. D, including prompts, base model, and hyperparameter configurations and evaluation metrics.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: Following common practice, we provide standard deviation based on repeated random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
26

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details the GPU platform used Appendix. D
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics http:: tttt:::///eurip... http://trrppp.ccpplttcc/ee/eeeee i
Answer:
[Yes]
Justification:
We followed the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the positive impacts of this work in the conclusion section (Sec.7).
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
27

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Not applicable for our case.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we credited them in appropriate ways.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi twcode.cooo.... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
28

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not include any crowdsourcing activities or research involving human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does not include any crowdsourcing activities or research involving human participants.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
29

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper does LLMs important, original, non-standard compo- nents of the core methods.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to LLM policy (https: //tp://pne..eeeeeeeeeeeeeeeeeeee 2000 /LLM) for what should or should not be described.
30


Reviews:
[{"review_id": "tqWFAbjaIO", "reviewer": "Reviewer_2qff", "summary": "This paper proposes a novel finetuning method for 3D generative diffusion models that relies solely on pretrained 2D models to provide reward signals. The method enables efficient optimization of 3D generation quality, resulting in models with richer geometric details and improved stability. The core of the approach is a gradient-based reward mechanism: by rendering multi-view images from 3D shapes and scoring them using a pretrained 2D model, reward gradients with respect to the latent variables are computed and used to guide the corresponding gradients on the side of the 3D diffusion model for optimization. In addition, optimization constraints are applied to both ends of the diffusion trajectory between adjacent latent variables to improve training stability. The method demonstrates significantly higher rewards and mitigates prior forgetting, even with a limited number of finetuning steps.", "questions": "1. It seems that the model doesn't respond well to color-related prompts. Does it show similar insensitivity to other types of prompts as well?\n2. The generated assets appear to share a relatively uniform style. Is the model capable of producing assets with diverse styles?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Compared to other finetuning methods, this approach generally produces results with better 3D consistency across views.\n\n2. Generated textures typically show higher quality and clearer details relative to alternative finetuning techniques.\n\n3. The pipeline achieves fast finetuning convergence and low inference latency.\n\nWeaknesses:\n\n1. The alignment between generated results and input prompts is not always consistent. For example, as demonstrated in the paper (e.g., variation in flower color), some prompt-driven attributes are not accurately retained.\n\n2. There can be noticeable deviations in visual style compared to the baseline model.\n\n3. The overall performance, including prompt fidelity and visual quality, largely depends on the quality of the pre-trained 2D backbone.", "was_revised": true, "final_justification": "The rebuttal has addressed most of my concerns. I will keep my rating.", "rebuttal": "We thank the reviewer for their time and efforts in the reviewing process and their acknowledgement of our contributions.\n\n> The alignment between generated results and input prompts is not always consistent. For example, as demonstrated in the paper (e.g., variation in flower color), some prompt-driven attributes are not accurately retained.\n> There can be noticeable deviations in visual style compared to the baseline model.\n\n\nThis behavior depends on two factors: 1) how much we would like the model to follow the reward instead of the pretrained prior and 2) what outcomes the reward model is rewarding. For instance, the reward model of Aesthetic Score favors image aesthetics and does not take any text prompt as input. Therefore, with a high value of $\\beta$ (the parameter to control the strength of the reward), the generated shapes inevitably deviate from the prompt. Such a problem can be alleviated by carefully picking the $\\beta$ parameter. We may also consider combining different reward models, which we leave for future drafts of this paper or future work.\n\n> The overall performance, including prompt fidelity and visual quality, largely depends on the quality of the pre-trained 2D backbone.\n> \n\nYes, this is another inevitable fact: just as finetuning larger LLMs is more effective than finetuning smaller ones (examples can be found in RL finetuning of LLMs for math reasoning), if the base model is largely incapable of generating reasonable geometries and textures, we cannot expect finetuning to greatly improve it.\n\n> It seems that the model doesn't respond well to color-related prompts. Does it show similar insensitivity to other types of prompts as well?\n\nYes, we did observe insensitivity to some other types of prompts:\n\n- Text-generation-related (e.g., generate the word \"Apple\" as a 3D shape)\n- Small objects (e.g., a spoon on the tea cup)\n- Directions (left or right)\n\nWe believe that these issues are mostly due to the incapability of the base models. In the meanwhile, it possibly reveals that we need better reward models, especially dense rewards (e.g., [1]).\n\n> The generated assets appear to share a relatively uniform style. Is the model capable of producing assets with diverse styles?\n\nThanks for bringing this question up. We also notice that after finetuning for too long, the styles tend to converge. The reason behind this behavior, we believe, is hinted by your questions above: the base model is still not that capable, plus that we use a relatively large $\\beta$ value. We do observe that if we do early stopping, the styles are far more diverse, which partially verifies the claim.\n\n---\n\n[1] Rich Human Feedback for Text-to-Image Generation. Liang et al. CVPR 2024. https://arxiv.org/abs/2312.10240"}, {"review_id": "Vjj6d6fhuk", "reviewer": "Reviewer_qnSr", "summary": "The paper proposes to improve 3D generation from 2D diffusion by RLHF. Here the reward is not from human, but from models judging the aesthetics and geometry quality, applied on 2D rendered views.", "questions": "None.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n1. The idea is a straight-forward adaptation of RLHF for LLMs to 3D generation. It is relatively easy to understand and simple to implement.\n2. The reward is obtained by looking at the 2D rendered views instead of directly on the 3D asset, allowing us to use much more powerful reward models trained on 2D data.\n3. Qualitative results (Figure 3, 4, 7) show that the method effectively improves the texture and geometry quality.\n\n**Weaknesses:**\n1. The idea is so straight-forward that novelty might be a concern. It is not a problem for me though. Proving that this idea can work in the 3D generation setting with strong evidence is good enough for me.", "was_revised": true, "final_justification": "The method is straight-forward and effective. I keep my original positive rating.", "rebuttal": "We greatly appreciate the reviewer for acknowledging our contributions. Partially to respond to other reviewers as well, here we would like to emphasize that although the method is straightforward, all other variants do not work well, including:\n\n- Vanilla finetuning methods (ReFL, DRaFT, DDPO)\n- SDS-based generation with 2D base model + 2D reward\n- SDS-based generation with 3D-native base model + 2D reward\n\nGiven these results on baselines, we would like to argue that such a path to combine 3D-native model and 2D rewards may be overlooked without our attempt to use the recent method of Nabla-GFlowNet."}, {"review_id": "LW9AI6aVkv", "reviewer": "Reviewer_Xv7Y", "summary": "This work aims to solve the fine-tuning of 3D generative models using 2D reward functions. It is based on the general framework of GFlowNet and instantiates it in the context of 3D generative models. The proposed framework uses a collection of multiple 2D rewards from prior works. The experiments show that the proposed method performs similarly to prior alignment fine-tuning methods in 3D generation.", "questions": "* Please clarify in the rebuttal the motivation of using the “approximated loss” (Eq. 9) and why it works better than the accurate loss.\n* Please clarify the confusing mathematical terms as I mentioned in the weakness section.", "limitations": "While the authors did discuss some limitations of the proposed method, I think it would be helpful to also discuss why the quantitative study shows very little advantages of the proposed method against prior works.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- **Strengths**:\n\t- Even though the results presented by the author didn’t show promising improvement, I believe using 2D reward function to fine-tune 3D generative models is a promising direction to address the lack of good large-scale 3D data.\n\n- **Weakness**:\n\t- **Incremental technical contribution with weak results**: Technically, this paper substitutes terms in the general framework of GFlowNet (Sec. 3.2) with specific instantiations for 3D object generation (Sec. 4.1). Additionally, the reward functions used in Sec. 4.2. are all commonly used objectives in prior works, as the authors already mentioned. Yet, a potentially valuable study of understanding the importance of each reward used is missing. Therefore, the technical novelty is limited. This would have been fine if such a simple and straightforward adaptation leads to surprisingly better results than before. However, from Tab. 1-3, the results are merely on-par with prior arts, if not worse in some cases. So, it’s hardly convincing that the proposed simple adaptation marks some progress.\n   - **A critical design choice seems arbitrary**: Eq. 9 presents an “approximated loss” of the losses in Eq. 7 & 8. The only explanation of this design choice is a seemingly post hoc “assumption” that the hφ term is not needed. This is not a principled justification of a critical design choice. Is the hφ term hard to compute? If not, what is the motivation behind dropping it, except a random or even accidental experiment? Meanwhile, if it is true that the “approximated loss” works better than the accurate loss, does it indicate any flaws of the theoretical foundation? Much rigorous discussion can be added here to reveal many potentially interesting insights. Yet, they are missing in the submission.\n\t- **Confusing mathematical notations**: (1) Undefined term, hφ. This term is first used in L129 and then used a few times in L151-154. It’s part of the objective so it’s an important term in my opinion. However, there’s no formal definition of what it means and how to get this function. The authors should clarify this. (2) The use of gφ is also quite confusing. It’s first used in Eq. 2 & Eq. 3, again without clear definitions. Then, when I compare Eq. 7 & 8 with Eq. 2 & 3, it seems this gφ term becomes hφ, and there’s a different g() function defined as the rendering function in L145. The authors should make these notations more clear and accessible.", "was_revised": true, "final_justification": "After reading the rebuttal and having further discussions with the authors during the rebuttal stage, most of my concerns from the first round were addressed. Hence, I'm leaning positive for my final reviews. However, due to the limited technical novelty as mentioned in my review and other reviewers' comments, I wouldn't recommend stronger accept for this submission. My final score is therefore borderline accept.", "rebuttal": "We thank the reviewer for their time and efforts in reviewing our paper and raising their concerns over our claims.\n\n> Even though the results presented by the author didn’t show promising improvement\n\n> Therefore, the technical novelty is limited. This would have been fine if such a simple and straightforward adaptation leads to surprisingly better results than before. However, from Tab. 1-3, the results are merely on-par with prior arts, if not worse in some cases. So, it’s hardly convincing that the proposed simple adaptation marks some progress.\n\nWe first respectfully disagree with the reviewer on this \"weak improvement\" point. We believe the misunderstanding comes from our miss in explaining how the methods should be evaluated and we elaborate it below:\n\nThe metrics cannot be taken at face values because we are facing a **\"multi-objective optimization problem\"**: we would like the reward to be higher, but higher reward inevitably hurts instruction following and prior preservation. For this reason, we would argue against simply looking at the \"bolded\" figures in the table; instead, probably the best way to evaluate the methods is to fix certain metric to be some value and compare other metrics. Indeed, we **present the Pareto front in Figure 16 in the supplementary material**: we evaluate and visualize the **reward-FID and reward-CLIPScore pairs** for each saved checkpoint of finetuned models during the training process. On the reward-FID subplot, the lower-right corner is the best (high reward + low FID); on the reward-CLIPScore plot, the upper right corner is the best (high reward + high CLIPScore). In these two subplots, our method is clearly much better than the other ones. We will bring this Figure 16 to the main text for better clarity.\n\nOn qualitative results:\n\n- We would like to point the reviewer to Figure 6 in which we compare our method with DRaFT. The shapes generated by DRaFT-finetuned model shows the Janus problem (multi-head; all views look like the front view) while ours do not. We believe that this is a very solid evidence to prove that our method is very robust.\n- We also want to highlight some details in Fig 7: a) the baseline methods (ReFL and DRaFT) are more likely to produce \"unaligned artifacts\", for instance the 1st and 3rd instance on the DRaFT line; b) after finetuning, our model removes the incorrect estimation on the geometry (for instance, in the base model, specular reflections are mistakenly treated as the true geometry instead of surface material effects.\n- As the reward model of HPSv2 is trained with a preference dataset of text-image pairs, the results of Fig 4 should be evaluated in two dimensions: 1) aesthetics and 2) text-shape/texture alignment (or instruction following). For instance, the shape generated by the ReFL-finetuned model with the prompt \"A low poly snowman with a top hat, scarf, and outstretched arms\" does not have \"scarf\" in it.\n\nBesides, in the figures we purposely pick shapes with the highest rewards (averaged across views) yet without catastrophic forgetting of pretrained priors to demonstrate the advantage of our methods. Indeed, we can sacrifice the quantitative figures and pick shapes with lower rewards but better looking.\n\n\n> Yet, a potentially valuable study of understanding the importance of each reward used is missing.\n\nThank you for bringing this up. Our paper aims to show that these rewards are available and with our method we can effectively use them for 3D-native generation. We do not mix these reward models together and in each experiment we only use one. We do share the same thought with the reviewer that it is fruitful to explore a good mixture of reward models, but since our purpose is to show the task itself, we leave this to the future work.\n\n> A critical design choice seems arbitrary\n\nThe design itself is not that arbitrary. Using this one step estimation for the \"value function\" (or basically leveraging the Tweedie's formula) is common in probabilistic modeling literature (see Equation 9 and 10 in [1]) and also used by the people working on applications (although without theoretical justification, e.g. [2]). In the Nabla-GFlowNet paper [3], the 2nd-line in Table 1 (with $w_B = 0$) shows that even if the correction term is not properly trained (due to the discard of some equations that making the correction term underdetermined), the performance is good enough. We follow the existing literature and use the best approximation we can do to accelerate training.\n\n> Please clarify in the rebuttal the motivation of using the “approximated loss” (Eq. 9) and why it works better than the accurate loss.\n\nWe apologize for not stating it clearing in our main text, but we do **not** claim in the paper that it is a better loss in the sense of final performance. It is, however, a much more efficient one because we do not need to train the correction term with potentially involves 2nd-order gradients and large computational graphs.\n\n> 1) Undefined term, hφ. This term is first used in L129 and then used a few times in L151-154. It’s part of the objective so it’s an important term in my opinion. However, there’s no formal definition of what it means and how to get this function. (2) The use of gφ is also quite confusing. It’s first used in Eq. 2 & Eq. 3, again without clear definitions. Then, when I compare Eq. 7 & 8 with Eq. 2 & 3, it seems this gφ term becomes hφ, and there’s a different g() function defined as the rendering function in L145. The authors should make these notations more clear and accessible.\n\nWe apologize for the unclear and undefined notations in the main text. $g_\\phi$ is the residual term that corrects the educated guess on the \"flow\" (or \"value function\" in the RL sense if we turn this problem into a suitable Markov decision process with some well defined intermediate rewards), and $h_\\phi$ should be $g_\\phi$ (we accidentally use two distinct symbols for the same thing). We will revise the text in the next draft of the paper and explain the preliminaries & notations in a clearer way.\n\n---\n\nFinally, we would like to underscore our contributions to the community of 3D generation that are not clear enough or not mentioned in the paper:\n\n- Empirically, all other methods do not work well: slow reward convergence, not robust, Janus problem, etc. Ours is the only one that is both efficient and robust. Given these results on baselines, we would like to argue that such a path to combine 3D-native model and 2D rewards may be overlooked without our attempt.\n- In addition, there are, to our knowledge, few papers that use RL for post-training in 3D large generative models (most people use DPO-based methods [4, 5]; the name of Mesh-RFT [5] is a bit over-claiming as they use DPO methods instead of online RL ones). By showing that RL finetuning can leads to good performance with simple reward models, we view our paper as a signal to encourage the people in the field of 3D generation to try different approaches (especially given that RLHF in general behaves better than DPO if carefully tuned in LLM research).\n\nWe hope that our answers may resolve some of the concerns of the reviewer.\n\n---\n\n[1] Practical and Asymptotically Exact Conditional Sampling in Diffusion Models. We et al. NeurIPS 2023. https://openreview.net/forum?id=eWKqr1zcRv&noteId=MlaVPZvkC6\n\n[2] Human Motion Diffusion Model. Tevet et al. ICLR 2023. https://arxiv.org/abs/2209.14916\n\n[3] Efficient Diversity-Preserving Diffusion Alignment via Gradient-Informed GFlowNets. Liu et al. ICLR 2025. https://arxiv.org/abs/2412.07775\n\n[4] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Rafailov et al. NeurIPS 2023. https://arxiv.org/abs/2305.18290\n\n[5] Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuning. Liu et al. https://arxiv.org/abs/2505.16761"}, {"review_id": "GZNtX4JBRd", "reviewer": "Reviewer_qR33", "summary": "This paper tackles the problem of improving a 3D generative model through RL-based fine-tuning with respect to 2D rewards. For this, the authors adopt a recently proposed Nabla-GFlowNet [22] framework and apply it to the 3D generation domain with some specific 2D rewards, e.g., appearance and geometry-based rewards. Experiments on 60 unseen prompts demonstrate that the proposed approach aligns the model better with the rewards (higher rewards values).", "questions": "1. My main concern is around the claim that\n> (L53) Nabla-R2D3 can effectively, efficiently, and robustly finetune 3D-native generative models from 2D reward models with better preference alignment, better text-object alignment, and fewer geometric artifacts.\n\nI do not find this convincing, especially with quantitative metrics in Tab. 1 and 2 as FID and CLIP-Sim seem quite inferior (in Tab. 2, the rewards are not good either). Further, I do not find the qualitative results in Figs 3, 4, 5, and 7 compelling either.\n\nEven on the convergence speed, e.g., Fig. 2, my sense is that the baseline approaches, e.g., ReFL or DRaFT, perform better.\n\nCan authors clarify?\n\n2. About the novelty,\n> (L48) to the best of our knowledge, the first method that effectively aligns 3D-native diffusion models with human preferences using only 2D reward models.\n\nI am not fully convinced by this as it is a common practice to use 2D metrics to supervise 3D learning (on generations), e.g., DreamFusion [28], and a lot of works using LPIPS, a 2D perception metric, to provide learning signals.\n\nFurther, the framework largely mimics Nabla-GFlowNet [22] and does not present strong novelty when it comes to the methodology.", "limitations": "See the points in \"questions\".", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "1. Quality-wise: the qualitative results displayed in the paper looks good\n2. Clarity-wise: the paper is generally well-written\n3. Significance-wise: the paper tackles an important problem of how to align 3D generative assets with human preferences\n4. Originality-wise: the paper demonstrates the potential for utilizing 2D rewards for 3D asset generation quality improvement", "was_revised": true, "final_justification": "After reading the authors' rebuttal, some of my concerns have been addressed. \n\nAs discussed with the authors, though I agree that the findings are promising, I still think that, methodologically, it is straightforward. I think this paper should be positioned as a study paper, which requires clearer and careful refinement of the contributions. Thus, I raised my score to \"borderline accept\".", "rebuttal": "We thank the reviewer for their time and efforts in reviewing our paper and raising their concerns over our claims.\n\n> I do not find this convincing, especially with quantitative metrics in Tab. 1 and 2 as FID and CLIP-Sim seem quite inferior (in Tab. 2, the rewards are not good either). Further, I do not find the qualitative results in Figs 3, 4, 5, and 7 compelling either.\n\nWe would like to first emphasize that we are facing a **\"multi-objective optimization problem\"**: we would like the reward to be higher, but higher reward inevitably hurts instruction following and prior preservation. For this reason, we would argue against simply looking at the \"bolded\" figures in the table; instead, probably the best way to evaluate the methods is to fix certain metric to be some value and compare other metrics. Indeed, we **present the Pareto front in Figure 16 in the supplementary material**: we evaluate and visualize the **reward-FID and reward-CLIPScore pairs** for each saved checkpoint of finetuned models during the training process. On the reward-FID subplot, the lower-right corner is the best (high reward + low FID); on the reward-CLIPScore plot, the upper right corner is the best (high reward + high CLIPScore). In these two subplots, our method is clearly better than the other ones. We will bring this Figure 16 to the main text for better clarity.\n\nOn Table 2: besides the \"multi-objective optimization\" issue, we would like to bring up the question: how effective one model is in finetuning models of different capacity? Just as finetuing larger LLMs is far more effective than finetuning small ones, in 3D generative models we have the same scaling law argument. We therefore argue that it is more important that one method works on large models.\n\nOn qualitative results: \n- We would like to point the reviewer to Figure 6 in which we compare our method with DRaFT. The shapes generated by DRaFT-finetuned model shows the Janus problem (multi-head; all views look like the front view) while ours do not. We believe that this is a very solid evidence to prove that our method is very robust. \n- We also want to highlight some details in Fig 7: a) the baseline methods (ReFL and DRaFT) are more likely to produce \"unaligned artifacts\", for instance the 1st and 3rd instance on the DRaFT line; b) after finetuning, our model removes the incorrect estimation on the geometry (for instance, in the base model, specular reflections are mistakenly treated as the true geometry instead of surface material effects.\n- As the reward model of HPSv2 is trained with a preference dataset of text-image pairs, the results of Fig 4 should be evaluated in two dimensions: 1) aesthetics and 2) text-shape/texture alignment (or instruction following). For instance, the shape generated by the ReFL-finetuned model with the prompt \"A low poly snowman with a top hat, scarf, and outstretched arms\" does not have \"scarf\" in it.\n\nBesides, in the figures we purposely pick shapes with the highest rewards (averaged across views) yet without catastrophic forgetting of pretrained priors to demonstrate the advantage of our methods. Indeed, we can sacrifice the quantitative figures and pick shapes with lower rewards but better looking.\n\n\n\n> Even on the convergence speed, e.g., Fig. 2, my sense is that the baseline approaches, e.g., ReFL or DRaFT, perform better.\n\nThanks for bringing this question up and we believe it demonstrates the key difference between 2D generation and \"lifting-from-2D\" generation. On 2D images, ReFL and DRaFT are indeed very fast. However, in lifting-from-2D, one samples a view and optimize the shape responsible for that view with limited 3D regularization. While one single view can be optimized at a fast speed, the optimization process may \"oscillate\" due to signals from different views. Here, stability and low gradient variance in training play greater roles in determining the convergence speed, which is why the reviewer observes the \"counter-intuitive\" results on convergence.\n\n> I am not fully convinced by this as it is a common practice to use 2D metrics to supervise 3D learning (on generations), e.g., DreamFusion [28], and a lot of works using LPIPS, a 2D perception metric, to provide learning signals.\n\nWe respectfully disagree with the reviewer on this point. Here, we would like to first emphasize that our focuses are\n\n- to finetune a **3D-native** diffusion model with 2D rewards in a principled way, and\n- to show a set of plausible rewards for improving 3D-native diffusion models.\n\nWe by no means ignore the predecessors in the direction of \"lifting-from-2D\" (e.g., DreamFusion as the reviewer mentioned, and many other earlier ones such as Dream Field [1] and GRAF [2]) and claim that we are the first to leverage 2D signals for 3D generation. Indeed, we have explicitly compared with SDS-based methods (i.e. DreamFusion) in Figure 9 and Table 3 and show that our method is better than these two alternatives.\n\nThe key differences between out method and DreamFusion (or SDS-based methods in general) are that\n\n1) SDS-based method aims to generate samples through optimization: initializing some random $x$ and use the diffusion loss $\\mathbb{E} \\|\\| \\epsilon_\\theta(\\text{noising}(x, t)) - \\epsilon \\|\\|^2$ to iteratively update $x$. Our method finetunes a diffusion model and the samples are generated by conventional sampling procedures (DDPM, DDIM, DPM-solver, etc.) which starts from a Gaussian noise and follows a prescribed denoising schedule.\n2) Our method is derived from a principled method with clear assumptions. By \"principled\", we mean that we know exactly what the error terms and how to learn the correction term. It is only that we for efficiency do not choose to learn this correction term. In DreamFusion, however, one does not have the theoretical guarantee that the optimization with the diffusion loss leads to the same distribution, both due to its sampling procedure with the SGD noise hard to characterize and due to that the diffusion loss is only a approximation of the ground-truth likelihood function (see Theorem 3 in [3]).\n\n> Further, the framework largely mimics Nabla-GFlowNet [22] and does not present strong novelty when it comes to the methodology.\n\nWhile it seems that we merely borrow an existing ML method, we argue that our contributions are non-trivial to the domain of 3D generation. In particular, we argue that\n\n1) We are the first to introduce this idea of RL finetuning of 3D diffusion models with 2D rewards.\n2) Empirically, all other methods do not work well: slow reward convergence, not robust, Janus problem, etc. Ours is the only one that is both efficient and robust. Given these results on baselines, we would like to argue that such a path to combine 3D-native model and 2D rewards may be overlooked without our attempt to use the recent method of Nabla-GFlowNet.\n3) We empirically show that some rewards, especially the geometric ones, can be used to improve 3D-native diffusion models. This is also something not done in the 3D RL alignment setting.\n\nWe would like also to point out that the field of 3D generation is still suffering from the lack of data. The largest publicly available dataset is Objaverse-XL, which contains 10m uncleaned 3D objects. People typically use only a subset of about 1 to 2 million shapes. Considering the dimensionality (3D vs. 2D) and the number of images we have in 2D domain (billions of images), there is still a huge gap. Our method aims to connect the 3D foundation models with the rich knowledge we have in 2D domain, just as DreamFusion does, but in a more direct approach. \n\nIn addition, there are, to our knowledge, few papers that use RL for post-training in 3D large generative models (most people use DPO-based methods [4, 5]; the name of Mesh-RFT [5] is a bit over-claiming as they use DPO methods instead of online RL ones). By showing that RL finetuning can leads to good performance with simple reward models, we view our paper as a signal to encourage the people in the field of 3D generation to try different approaches (especially given that RLHF in general behaves better than DPO if carefully tuned in LLM research).\n\n---\nWe hope that our answers may resolve some of the concerns of the reviewer.\n\nCitations:\n\n[1] Zero-Shot Text-Guided Object Generation with Dream Fields. Jain et al. CVPR 2022. https://arxiv.org/abs/2112.01455\n\n[2] GRAF: Generative Radiance Fields for 3D-Aware Image Synthesis. Schwarz et al. CVPR 2020. https://arxiv.org/abs/2007.02442\n\n[3] Maximum Likelihood Training of Score-Based Diffusion Models. Song et al. NeurIPS 2021. https://arxiv.org/abs/2101.09258\n\n[4] Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Rafailov et al. NeurIPS 2023. https://arxiv.org/abs/2305.18290\n\n[5] Mesh-RFT: Enhancing Mesh Generation via Fine-grained Reinforcement Fine-Tuning. Liu et al. https://arxiv.org/abs/2505.16761"}]

Author rebuttal:

Meta-review: Reviewers agree that this is a significant step forward in the field. The rebuttal and post-rebuttal discussion clarified initial questions. Congratulations, this submission is accepted.
Decision: Accept (poster)