CPO: Condition Preference Optimization for Controllable Image Generation
Authors: Zonglin Lyu
Ming Li
Xinxin Liu
Chen Chen
Venue: NeurIPS 2025 poster ()
Keywords: Diffusion Models
Controllable Image Generation
ControlNet
Primary area: applications

Abstract: To enhance controllability in text-to-image generation, ControlNet introduces image-based control signals, while ControlNet++ improves pixel-level cycle consistency between generated images and the input control signal. To avoid the prohibitive cost of back-propagating through the sampling process, ControlNet++ optimizes only low-noise timesteps (e.g., $t < 200$) using a single-step approximation, which not only ignores the contribution of high-noise timesteps but also introduces additional approximation errors. A straightforward alternative for optimizing controllability across all timesteps is Direct Preference Optimization (DPO), a fine-tuning method that increases model preference for more controllable images ($I^{w}$) over less controllable ones ($I^{l}$). However, due to uncertainty in generative models, it is difficult to ensure that win--lose image pairs differ only in controllability while keeping other factors, such as image quality, fixed. To address this, we propose performing preference learning over control conditions rather than generated images. Specifically, we construct winning and losing control signals, $\mathbf{c}^{w}$ and $\mathbf{c}^{l}$, and train the model to prefer $\mathbf{c}^{w}$. This method, which we term \textit{Condition Preference Optimization} (CPO), eliminates confounding factors and yields a low-variance training objective. Our approach theoretically exhibits lower contrastive loss variance than DPO and empirically achieves superior results. Moreover, CPO requires less computation and storage for dataset curation. Extensive experiments show that CPO significantly improves controllability over the state-of-the-art ControlNet++ across multiple control types: over $10\%$ error rate reduction in segmentation, $70$--$80\%$ in human pose, and consistent $2$--$5\%$ reductions in edge and depth maps. The error rate is defined as the difference between the evaluated controllability and the oracle results. Our project is available \textcolor{blue}{\href{https://zonglinl.github.io/CPO_page}{here}}.

CPO: Condition Preference Optimization for Controllable Image Generation
Zonglin Lyu Ming Li Xinxin Liu Chen Chen
Institute of Artificial Intelligence University of Central Florida Orlando, FL 32816 (zonglin.1p) lyu, ming li, xinxin. chen. chen) edu
Abstract
To enhance controllability text-to-image generation, ControlNet introduces image-based control signals, while ControlNet++ improves pixel-level cycle con- sistency between generated images and the input control signal. To avoid the prohibitive cost of back- propkaarpggg through the sampling process, ControlNet++ optimizes only low-noise timesteps (c.g.. t < 200) using a single-step approxi- mation, which not only ignores the contribution of high-noise timesteps but also introduces additional approximation errors. A straightforward alternative for op- timizing controllability across all timesteps is Direct Preference Optimization (DPO), a fine-tuning method that increases model preference for more control- lable images (1")) over less controllable ones (L'). However, due to uncertainty in generative models, it is difficult to ensure that win-lose image pairs differ only controllability while keeping other factors, such as image quality, fixed. To address this, we propose performing preference learning over control conditions rather than generated images. Specifically, we construct winning and losing control signals, c and c', and train the model to prefer e. This method, which we term Condition Preference Optimization (CPO), eliminates confounding factors and yields a low-variance training objective. Our approach theoretically exhibits lower contrastive loss variance than DPO and empirically achieves superior results. More- over, CPO requires less computation and storage for dataset curation. Extensive experiments show that CPO significantly improves controllability over the state-of- the-art ControlNet++ across multiple control types: over 10% етror rate reduction in segmentation, 70-80% in human pose, and consistent 2-5% reductions in edge and depth maps. The error rate is defined the difference between the evaluated controllability and the oracle results. Our project is available here
Introduction
Recent advancements in diffusion models i have achieved state-of-the-art performance in text-to- image (T2I) generation. Text prompts serve sparse conditioning signals, typically conveying only global attributes such as background, foreground, and salient objects. However, when multiple objects are present when fine-grained details, such as edges or 2D/3D spatial structure, are involved, text prompts struggle to capture such information precisely. These details are more effectively conveyed through dense visual conditions, such as segmentation maps [4, depth maps [5] and Canny edges [6] To incorporate such dense control, several works 77 introduce additional modules that extract control signals on top of base T2I models, such as Stable Diffusion (2). However, these methods do not define an explicit training objective to ensure precise controllability.
ControlNet++ [ introduces pixel-level cycle consistency loss between the generated image x and the input control condition explicitly model controllability during training, as shown in
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Detaset
Canny
Filder: (R( TR(P)> &
Ground truth condition
DPO example
Generation model
Desect conditio compane with
20 Images
best aligned with condition
worst aligned with condition
(a) Data generation Process of DPO
Ground truth
Perturbed condition dl
G ennrration model
Detection Model
example
(c) Examples in Canny Figure I: (a) Data generation Process of DPO. We generate 20 images and find images that the best and the worst aligned with the input condition. The ImageReward (IR) score of the winner is 0.2 higher; otherwise. the example is filtered out. (b) Data generation process of CPO. Our process is much simpler. We generate one image to perturb the ground truth condition since the generation model is not perfectly controllable. (c) Examples on Pose and Canny, c 's are the ground truth conditions. In the Pose example, the red circle indicates an artifact the winning example DPO. In Canny example, the difference in Canny edge is too hard discern raw pixels in DPO, but our method directly compares conditions.
(b) generation
Fig. 2 The cycle consistency is enforced via a loss term between and è, where è is extracted from the generated image x using a condition detector. For example, in depth-to-immae generation, this corresponds to mean-squared error loss. However, sampling from arbitrary timesteps t with back- propagation is computationally infeasible due to memory constraints. To mitigate this, ControlNet++ employs a single-step reparameterization of DDPM of this approach is the distribution mismatch between the reparameterized diffusion process and sampling processes [1, leading to noisy approximations. As result, ControlNet++ is only applicable for low-noise steps (t < 200), beyond which the sampled image & becomes non-recognizable [ As result, this paradigm fails optimize high-noise timesteps that are important for image structure generation 12H19.
approximate sampling. A key limitation
To avoid noisy approximation and optimize all dif- fusion timesteps, one approach is to apply Direct Preference Optimization for diffusion models (Dif- fusion DPO) (0). to fine-tune a pretrained model. Diffusion DPO computes pairwise contrastive loss at each timestep without requiring an approximation. Let I be image and x, be its corresponding latent representation. Given preference pairs (I" I')) with the same condition c, where 1 is the winning image that better aligns with (i.e., has higher controllability score), the model learns to prefer x over x² for any timestep l. However, while Diffusion DPO offers general preference optimization capabilities, faces fundamental limitations for controllable generation tasks. The inherent uncertainty in generative models makes it challenging create reliable win-lose image pairs that isolate controllability from other quality factors, thus in- troducing potential noise. To highlight the limitations, we generate a dataset using the pipeline shown in Fig. a), focusing on pose control due computational constraints| To construct high-quality preference pairs, we generate 20 images per prompt using ControlNet 77 and select the sample with the highest controllability score 1d and the one with the lowest I'. To ensure that T" is meaningfully better, we use ImageReward [21] as quality filter, requiring I" to score at least = 0.2 higher than /¹. This threshold sufficiently large that around half of the generated pairs are filtered out. Increasing the threshold further results in too few usable training examples. Despite this setup, several challenges remain for adapting DPO to controllable generation.
Inpet Image
Denoising
aneo pPV
Condittion Dretcctor
One-stap
sample
Impout Condition:
Consistency Loss
Figure Illustration of ControlNett+
First, factors unrelated to controllability introduce noise, increasing loss variance and potentially confusing the model. In the pose example shown Fig. (c), although the winning image aligns
We generate DPO training data only for Pose, creating Canny dataset requires over 5K GPU-days, which is computationally prohibitive. However, we follow the same procedure to generate Canny examples.

---

better with the pose, it contains a round artifact on the rear of the motorcycle, highlighted in the red circle. Optimizing on such pairs may degrade image quality. Experimental results in Fig. ff support this claim.
Second, some control types are hard rec- ognize in raw images. In the Canny edge example shown in Fig. [[c), differences Canny edge are hard to discern, while color variations dominate visual perception.
Deaeting
+
SODN ppY
IK
Third, the selection process is computa- tionally expensive. Since images gener- ated using the same model, constructing a preference pair requires multiple inference calls (we use 20) to ensure a diverse range of controllability scores.
Linat
Livet
(a) Diract Proferance O ppmiization
[b) Condition Preforance Optimization
Figure 3: (a) Training of DPO. DPO is trained prefer 1 over I'. (b) Training of CPO. CPO is trained to prefer over
The first limitation arises because images encode much more information than just alignment with control signal c. Varying the generated image affects not only its alignment with but also unrelated attributes such as background, color, and style. If we could vary only the controllability score while keeping all other aspects fixed, this issue would be mitigated. Importantly, the aligmment between an image I and a condition с is bi-directional. That is, one can compare two images control signals c and
T and 71, where T aligns better with c, equivalently, compare c', where e aligns better with a fixed image T. We adopt the latter formulation by setting en as the ground-truth condition and constructing c' via perturbation. Specifically, we estimate control signal from an image generated using c" as the condition. In the (1",," formulation, the model must infer controllability differences indirectly from raw image content, which is particularly challenging when conditions involve fine-grained details such as Canny edges. In contrast, the (c c²) formulation allows the model to observe controllability differences directly the condition level, thereby addressing the second limitation. The third limitation of DPO lies in its sampling cost: multiple images must be generated to obtain meaningful preference pair (I") I') with significant difference in controllability. In our approach, this burden is reduced. Since generative models are far from perfectly controllable, the difference between c and its perturbed version cf is naturally large enough to serve as a training signal, requiring only single image sample for perturbation. Based on this insight, we propose Condition Preference Optimization (CPO) for controllable image generation by constructing preference pairs (c") .c)..). This method eliminates noise introduced by irrelevant visual factors and provides a cleaner training objective. Our data generation process is illustrated in Fig. [Tb,, and the distinction from DPO training is shown in Fig. Our contributions
are summarized as follows:
Low variance training objective than DPO. Our method provides a low-variance (less noisy) training objective than Diffusion DPO in the control-to-imant generation task, and we provide theoretical analysis in Appendix C and experimental support in Sec. 4.2
Generalization beyond ControlNet+ Our method generalizes beyond ControlNet++ in the sense that we can optimize arbitrary timesteps for improved controllability and support human pose.
New Dataset. We curate and will open-source our Condition Preference (CPO) Dataset. There 60K examples for human pose, 20K and 118K examples for ADE-20K [22] and COCO-Stuff 23 segmentation, and 2.8M examples each for Canny edges, HED, Lineart, and Depth maps. The CPO dataset is more efficient in curation and storage than the DPO dataset, with details in Appendix D
State-of oo--he---ttt----eeeeee Results. Our method achieves state-of-the-- art performance in various tasks. Specif- ically, achieve over 10% error rate reduction in segmentation, 70-80% error rate reduction in human pose, and around 2-5% consistent error rate reduction on edges and depths after fine-tuning. The error rate is defined the difference between the evaluated controllability and the oracle result.
2 Related Works
Diffusion Models. Denoising Diffusion Probabilistic Models (DDPM) introduced diffusion models for image generation by employing a forward diffusion process (a Markov chain) that progressively adds noise to image until it becomes standard Gaussian noise, and reverse sampling
3

---

process that progressively denoises the Gaussian noise to an image using a UNet [24). However, DDPM discretizes the sampling process into 1000 steps, leading to inefficiencies. To mitigate this, DDIM 25 proposed non-Markovian formulation that reduces the number of sampling steps without significantly degrading generation quality. Subsequent works (26 have developed more advanced ODE solvers to further improve efficiency and sample quality. Beyond sampling steps reduction, other studies have explored leveraging intermediate UNet features to reduce computational cost, such as by reusing feature maps [29 attention maps [30]. Additionally, latent diffusion models (LDMs) incorporate VQGAN 3I or VAE 322 to project images into a compact latent space for better efficiency. With advancements in model architecture and training scale, LDM is delivered as Stable Diffusion. Building on this foundation, recent works have proposed specialized architectures 7 10 8 33
and diffusion processes 34 35 tailored to downstream tasks.
Controllable Image Generation. Controlling image generation at a fine-grained level, such as aligning with Canny edges or depth maps, is difficult to achieve using text prompts alone, as they offer sparse and coarse control. To address this limitation, ControlNet 7 introduces efficient fine-tuning modules top of pretrained T2I models (e.g.. Stable Diffusion) to incorporate dense visual conditions such as depth and edge maps. UniControl and UniControINet [10, extend this framework to support multiple control types within a unified architecture. ControlNet++ [ further improves controllability by enforcing cycle consistency between generated images and input control signals. However, due to the iterative nature of diffusion sampling, it remains infeasible to apply cycle consistency at arbitrary timesteps, as back- paokkpropaggting through intermediate states requires prohibitive GPU memory [
Beyond ceffusiio-baseed approaches, ControlAR 36 introduces controllable autoregressive models with multi-resolutio training and inference. It employs DINO-v2 (37 as the control signal extractor and achieves competitive performance. However, due to the sequential nature of autoregressive generation, it is significantly less efficient than diffusion models equipped with ODE solvers.
Direct Preference Optimization. Direct Preference Optimization (DPO) 38 reinforcement learning from human feedback (RLHF) method [39] originally developed for natural language processing. It aims maximize the log- pogg-rbbbillity difference between uuman-preferred and non- preferred responses, encouraging the model to generate preferred outputs. Diffusion-DPP (20
adapts DPO to diffusion models by leveraging a high-quality, human-annotated dataset 40 to
enhance image generation quality. Subsequent works introduce alternative optimization techniques, such as KTO 41 and preference score matching 42 to further improve performance 43
However, in the context of controllable image generation, comparable human-annotated datasets are unavailable, Moreover, applying DPO in this setting presents several limitations, including noisy training objectives, imperceptible control differences (e.g., in edge maps), and high computational overhead, as discussed Sec.
3 Methodology
We first present the preliminaries in Sec. 3.1 and then describe our proposed method in Sec. 3.2
3.1 Preliminaries
Diffusion Models. Diffusion models 1 define forward diffusion process q(x: that adds Gaussian noise to an image Xg, and a reverse sampling process pe xx,1)xx) that reconstructs the image from noisy latent variables. These processes are defined as:
(xxx|x0) Nixx Va,xo (1 at I)).), where at =II' (1 3z).
Po (x(x = Nxx
(2)
where j.
Dr P T- (xt VI-ai ))
1
B, 1- ai
1-a
Be, e~ (0,1).
(3)

---

Here, St is predefined small constant. In Eq. (3). the noise term e is unknown and is predicted by neural network Ep (x,, t). Conditioning terms are omitted here for clarity. practice, they serve as additional inputs to the model.
Diffusion-DPO. In the setup of Diffusion-DPO (0). preference pairs (x5 xx) provided, and the objective is to train the pretrained model eg to prefer the winner x over the loser x Using the Bradley-Terry model [45]. I the training objective is defined as [20]:
)
DDO -.xxy xg log
Exp Pol (x (x (x))
(log
Ps (xx lc) Pert
log Do lc)
(xb
xprrpp ( (xir)xx))
With appropriate algebraic and statistical simplifications (see
20 for derivation), this reduces to:
CDo -B xb log o
( (18Tw((e) (xx" ; t) ||e - (x,r ;
(5)
1)l2 Gxr 013))
Here, is sampled from (00, I), and x and x! are obtained via Eq. U The networks eg and <ve are initialized with the same weights, but only Eg is updated during training. The scalar x is defined I- and w (Xe) is practically set to 0.5 [20].
3.2 Proposed Condition Preference Optimization
Formulation. For notational simplicity, we omit the text prompt. Given a triplet (Xo, c",, c'). where Xg is the latent representation an image J, and ct e' are control signals (e.g.. depth maps), we have e aligning better with than c/. Our Condition Preference Optimization (CPO) loss
Do (xooe"" CCPO - ,'' x log [xi (x01/)1x0)) (log
Pe (xco le') Pert (xoooo)oo))
)]
Pret(Xol c log
(6)
Similar to Eq. (5) the expression can simplified (full derivation is provided in Appendix B as:
Loro -Eie" elo))e
logo (BBTw(A)) (
cx.x cm ,),,)) ||e varlxe, c",,)1 -Ilc- co(x, ,01l - car(X., c', (111)))))))
Gradient Analysis for the Improved Formulation. We begin by defining the following terms:
a= BTw, de = ||c c .0) [|c- co(x,, c d),e).. dur ||c Gar(xi, e",, .0..) l|e c(. (8)
The gradient of Lcpo (ignoring the expectation for clarity) is: VoLcro = - (a (do - dret)) Vo (-a (dg drer))
= =a a (a (do - dret)) Vodg.
This is equivalent to scaling the gradient of loss defined by dg. The term dg resembles a triplet loss, where serves as the anchor, < (x c. ,t) as the positive sample, and c,(x, c'. f) as the negative sample, but without a margin term and zero-clipping. The margin term and zero-clipping are introduced to prevent excessive contrast between positive and negative pairs.
Importantly, in CPO, the negative sample does not imply "incorrectness", but rather a weak alignment between image and condition. Therefore, we include triplet margin to avoid overly contrasting the two samples. addition, since a is typically set large value (c.g., 2500) [20,, can amplify the gradient and destabilize training. To mitigate this, we remove the scaling factor a from the outside.
Our final CPO loss is formulated as:
Lcro Eye= E(cm,.e.xx),t AcPo max(do m, O)]
(10)
where AcPO = sg (a (dg dret)), and d. ||e Ex,, c² t)]? - ||e (Xe, c',
5

---

Table I: Quantitative comparison in controllability. 1 means higher better, means lower is better. The best results are boldfaced. Oracle indicates the performance of the condition detection model on ground truth images, which is the upper-bound of controllability. We only compare models with the same CFG Sales for fairess.- No publicly available checkpoints or evaluation results are available. "Models are not open--oprcee sourcsd
but results are copied from their papers. Evaluated with their provided checkpoints.
Segmentation
Pose
Canny
HED
Lineart
Depth
Method
T21 Model
ADE20K COCO.Stuff
COCO- Pose
HumanAri
MultiGen 20M
mloU |
mAP1
F1
SSIM 1
SSIM
RMSE -
ComuolAR (CFG- 4.00)
LlamaGen-XX XL
39.95
36.78'
0.8184
0.7922
Ours (CFG = 4.0)
SD1.5
46.38
36.10
39.68
0.8299
0.8589
25.98
T21-Adapter
SD1.5
12.61
23.65
Gligen
SD1.4
23.78
87.84
39.88
26.94
0.5633
38.83
Jni-ControlNet
SD1.5
19.39
12.50
4.23
27.32
0.6910
40.65
UniControl
SD1.
25.44
75.29
ControlNet
SDI.5
32.55
27.46
72.53
34.52
34,65
0.7621
0.2054
35.90
ControlNete+
SD1.5
43.64
34.56
38,03'
0.8097
0,8399
28.32
Ours
SDL.5
44.81
35.49
87.98
45.71
39.28
0.8201
0.8447
27.49
Oracle
55.07
41.32
92.54
50.03
100.0
1.0
1.0
0.0
Here, sg(.) denotes the stop-gradient operation. Since de diverges from the original diffusion model objective, add the pretrained diffusion loss regularization term. The final loss becomes:
Lioat Lcro + Xpretrain:
where
Cpretrain
E E(xo,e") t", t'
||c' co((x, c²,, (t)
(11)
The variables e' and denote independently resampled noise and timesteps. A proof that the variance of the CPO loss is lower than that of DPO (i.e., it is less noisy) is provided in Appendix C
CPO Dataset Curation. Suppose we have detection model R that extracts control signals from images, a generative model G that generates images conditioned on control signals, and a dataset consisting of images I and control signals (text prompts are omitted for simplicity). Our data curation process proceeds follows: (1) Generate an image î using G conditioned on c; (2) Set the winning condition e and obtain the losing condition c¹ R(f); (3) If the original control unavailable, fallback to e R(T); (4) Store the triplet (I, (,,c,,,ct) c²).
We select G to ControlNet++ when available; otherwise, we use ControlNet. A detailed comparison with DPO dataset curation is provided in Appendix D
4 Experiments
Datasets and Curation. Following ControlNet++ we use ADE20K 22 and COCO-Stuff 23 for segmentation-to-imagg generation, and MultiGen-20M [10] for edge and structure-based con- ditions, including Canny edges, HED, Lineart, and depth maps. In addition, we include COCO- Pose [46] and HumanArt [47 for pose-to-image generation. Note that only train the pose-to-image generation model on COCO-Pose and test it on both COCO-Pose and HumanArt. We curate CPO datasets using COCO-Pose, ADE20K, COCO-Stuff, and MultiGen-200 20M.. For COCO-Pose, erate images using Non 7 and apply YOLO-1 generated images, which serve the perturbed condition c'. For the remaining datasets, ControlNet++ to generate images and apply the same condition detectors as in ControlNet++.
gen-
[48] to extract keypoints from the
Evaluation and Metrics. Following ControlNet++ [1I. we resize all images to 512x512 for evaluation. We use mean Intersection over Union (mloU) for segmentation, root mean squared error (RMSE) for depth, and structural similarity index (SSIM) for Lineart and HED. For Canny edges, we report the Fl score, and for human pose, we follow conventions in pose estimation [49] to report mean Average Precision (mAP). To ensure fair comparison, we use the same condition detectors as ControlNet++ to extract conditions from the generated images. For pose estimation, use YOLO- - 48 to detect human keypoints. In addition to controllability metrics, we report Fréchet Inception Distance (FID) using the CleanFID implementation (50) and CLIP similarity scores (5T]. For consistency across all experiments,
fix the classifier-free guidance scale to 7.5 (the standard for based models) and use the UniPC sampler (28) with 20 inference steps, matching the ControlNet+ setup. Training details provided in Appendix
Baselines and Base Models. We compare against several strong controllable generation baselines, including T21-Adapter (8), GLIGEN (33), Uni-ControlNet 99. UniControl [10], ControlNet (

---

Original Image
Ours
ControlNett+
ControiAR
ControiNet
HED
Canny
Depth
Segmemation
Figure 4: Qualitative Comparison in Controllability. Red boxes indicate the area where our method achieves better controllability.

---

Table 2: Results in FIDJCCLPT. Best results are boldfaced. We only compare models with the same CFG
scales.
Segmentation
Powe
Canny
HED
Lincart
Depth
Method
121 Model
ADE22K
COCO Stuff
COCO Pose
HummaAri
MultiGes 20M
ContrlAR (CR) =4 Do
LlmmaGin-X X1
27.15%/-
14.51/31 00
19 007220 12"
14.00 /30
12.41/--
17,20'200 19"
Ours (CRR (C +40)
SD15
27. 11.5931.99
17.01/32.30
18.4931.15
12,2431.55
12.96/31.44
14,62/31.22
T21 A2i-Adupter
SD1.5
39.13/90.65
15 15.96/31771
22.52/33.44 46
Gligen
SD14
33.02231.12 12
41.3652.18 13
48,92231 20
18.89"/31.77
19.36/51.75
Usi ContolNee
SD1.5
30.97:30.59
46.19/31.76
31.26/23.30
17.14/31 84
17.08/21.94 944
20.27/31 666
UniConirel
SD1.5
46.34/9.92
44.74/31 922
46, 99/33
19.94/31 997
15 15.99/32.002
18.66/32.4 45
CoatedNet
SD1.5
33 23,2631.55
21.33/32.21
42.02/52.05
32.22/33..3
14.73/32.15
15.47/32.33
17,44/32.4 46
17.76/32-4 45
CouteeSeeve
SD1.5
30.24 131 196
19.79 32
20.167731.87
15.00/322.05
13,88/51 955
16.66/92.11
Ouss
SD1.5
30.30231.97
19.30/32.36
39.21/32.90
29,94/3299 99
19,09/3188
13.35/32.07
13,38/51.99 99
15,89/32.31
ControlAR [36]. and ControlNet++ [1] As our method is designed as a fine-tuning approach, requires initialization from a pretrained model. We use ControlNet as the base model for human pose generation and ControlNet++ for all other tasks.
4.1 Experimental Results
Fair Evaluation. To ensure fair comparison, we re-evaluate the results of ControlNet++ and Con- trolAR using their official checkpoints. If substantial differences are observed due to variations in the machine environment or implementation details, we report our re-evaluated results instead. Specifically, bostrol methods use the Kornia implementation for Canny edge detection, whereas ControlAR uses an OpenCV-based implementation with a different threshold, which can lead to inconsistencies. Additionally, there are subtle differences in depth and HED preprocessing: ControlNet-based methods round these maps to integers during image generation and evaluation, while ControlAR retains floating-poot point values. All re-evaluated results are marked with * Further- more, ControlNet- based methods are evaluated with a classifier free guidance (CFG) scale of 7.5 [52], while ControlAR is evaluated at CFG scale 4.0. To ensure fairness, we also evaluate our method at CFG scale 4.0 and only compare methods under the same CFG scale.
Controllability Results. Controllability eval- uation results are presented in Tab. [ In the MultiGen-20M. dataset, labels are generated us- ing the same condition detectors employed for evaluation, enabling oracle controllability scores to be perfect when extracting conditions from ground-truth images. In contrast, labels for hu- man pose and segmentation tasks are human- annotated, making evaluation upper-bounded by the accuracy of the condition detector. The error rate is defined the difference between the evaluated controllability score and the oracle result. Our method consistently yields sub- stantial improvements over the base models it fine-tunes. Specifically, we observe approxi- mately 10% error rate reduction on ADE20K, 14% on COCO-Stuff, and over 70% on both pose datasets. For edge-based control types, we observe 2% error rate reduction on Canny and Lineart, 5% on HED, and 3% on depth. The relatively modest improvements for fine-gra fine-grained signals like Canny and Lineart may be due to the sparsity of supervision in the latent space. In contrast, HED captures coarser structural cues and thus enables more noticeable gains. For object-level control conditions such as human pose and segmentation, our method achieves significantly more pronounced improvements. Compared to ControlAR, our approach demonstrates substantially stronger controllability performance across most settings, except for COCO-Strr Stuff.
Control Pose
CombrolNNot
(a) Human Pose
Original Image Lineart
ControlNet=#
(b) Lineart
Figure 5: Qualitative Comparison in Pose and Lineart.
FID and CLIP. We report FID and CLIP scores in Tab. 2 Compared to ControlAR, our method CLIP scores. Relative to ControlNet+++ our method has minimal impact on both metrics, suggesting that the controllability improvements from fine-tuning do not compromise overall generation quality or semantic alignment. We note that our FID score on HumanArt is worse than that of the base model, ControlNet. This
achieves comparable FID scores while significantly outperforming
8

---

Table 3: User study controllability.
Task
CPO
ControlNet++
ControlAR
HED
47.73%
37.87%
14.40%
Depth
46.67%
40.27%
13.06%
Table 4: Ablation studies on the effect of DINO-v2 feature extraction on controllability. t means higher is better, L means lower is better. The best results boldfaced. Oracle indicates the performance of the condition detection model on ground truth images, which is the upper-bound of controllability.
Segmentation
Pose
Canny
HED
Lineart
Depth
Method
T21 Model
ADE20K COCO-Stuff
COCO-Pose
HumanArt
MultiGen-20
20M
m 1
mAP 1
F11
SSIM 1
SSIM 1
RMSE
Ours
SD1.5
44.81
35.49
87.98
45.71
39.28
0.8201
0.8447
27.49
Ours-DINOv2
SD1.5
43.72
39.25
88.45
49.98
44.81
0.8376
0.8301
26.92
Oracle
55.07
41.32
92.54
50.03
100.0
1.0
1.0
0.0
is likely due to a domain gap: HumanArt consists predominantly of cartoon-style images, while our training data (COCO-Pose) comprises real-world photographs. A similar pattern is observed with GLIGEN, which is also trained on COCO-Pose. However, our method generalizes better than GLIGEN on this out-of-distribution dataset, indicating stronger robustness despite the domain shift. Finally, we observe that the classifier-fre freee guidance (CFG) scale significantly affects both controllability and FID. We include an ablation study on this in Sec. 4.2
Qualitative Comparison. We present qualitative comparisons with baseline methods in Figs. 4 and 5 For HED and Canny edge conditions, other methods produce undesired edge artifacts within the highlighted red boxes, while ControlAR additionally misses key edge segments, indicated by the red arrows. In the depth example, the original image depicts shallow river near the camera, where the riverbed is visible. ControlAR fails to capture this detail, ControlNet++ introduces an undesired tree, and ControlNet generates an over-height mountain. In the segmentation example, ControlAR fails to generate the ketchup and the sandwich, while both ControlNet++ and ControlNet omit the sandwich. Notably, our method is the only one that correctly generates the table in the upper-right corner. The black region in the input condition represents background and is not evaluated for correctness, In the pose example (Fig. 5). both GLIGEN and ControlNet mistakenly generate multiple humans, whereas our method accurately renders single person. In the Lineart case, our method produces more coherent results in generating the doud-shaped object highlighted in the red box.
User Study. We conduct a user study on Con- trollability among our methods, Nettr,llet+++ and ControlAR. We randomly sample 25 ex- amples each from HED and depth and ask 15 annotators to select which one is the most con- sistent with the input condition. The results in
88.0
Ours
87.8
DPO
87.6
87.4
87.2
win rate (%) are shown in Tab. 3
87.0
4.2 Ablation Study
86.8
86.6
CPO vs. DPO. Since DPO requires additional dataset curation, we provide a quantitative com-
37.0
37.5
38.0
38.5
39.0
39.5
40.0
40.5
FID 4
parison COCO-Pose [6] only. We evaluate five checkpoints evenly spaced throughout train- ing and report results in Fig. 6 Our method
Figure 6: Comparison with DPO. We evaluated 5 check- points equally throughout training and observed that under the same mAP level, our method gets better FID. consistently achieves higher mAp at matched FID levels, or lower FID at matched mAP levels, demonstrating improved controllability-qual
trade-offs over DPO.
DINO-y2 adaptation. ControlAR adopts pretrained DINO-v2 [37] to extract features of conditions, while ControlNet-basse methods use simple CNNs. The DINO-v2 is also adaptable to ControlNet- based methods and generally improves image quality and controllability, shown in Tab. 4 and 5 Ours-DINOv2 is obtained by pretraining a ControlNet with the DINO-v2 adapter until convergence, followed by our CPO finetuning for at most 4000 steps
We train ControlNet++ for COCOStuff and Depth for 500 steps and 2000 steps. respectively. For other tasks, do not train ControlNett+.

---

Table S: Ablation studies the effect of DINO-v2 Teature extraction on FIDJ/CLIPT. Best results boldfaced.
Segmentation
Pose
Canny
HED
Lineart
Depth
Method
T21 Model
ADE20K
COCO-Stuff
COCO-Posc
HumamAri
MulliCka-20M
30.30/1197
19.30 19.30V32.00
39.21/32.90
39,94/32.98
19.69/3
13.35/32.07
13.35/31,80
15.88/32.31
Oun--DINOv2
SDL.5
30.02/32.02
17,57/32.53
37,58/32.75
45.78/52.45
14.62/32.077
12.06/32.00
13.12/32.03
15.50/32.55
Table 6: Ablation Studies on CFG Scales.
Table 7: Ablation Studies of margin and regular- ization strength ADE-20K dataset.
CPG scale
Seg. (ADE-20K)
Canny
MMloU//ID//CLIP
FI//PID/CLIPI
Margin
Reg. Strength
mIoUY//ID//CLIPP
48.18/28. 28/30.95
09.49/20.45/29
m 0.01
0.05
14.81/30.30/311 9.77
3.0
06.400/7.21/33 .68
0.1
A= 0.05
11/33..47/31.99
4.0
0..38/77.9//3..0 80
39.68/18.49/11. 15
no margin
A= 0.05
44.83/30.79/31.
7.5
44,81/30,3000 30/331
19.28/19.69/3188 69/3183
m 0.01
A= 0.15
0..99/99.8//22.
ControlAR (4.0)
39.95/27.15//
36.78/19.00/2..2
m 0.01
X == 0
46.01/31. 137/3 ..92
m1 0.01
A 1
39.01/30.80/31.77
Classifier- Free Guidance. We conduct experiments on varying the classifier-free guidance (CFG) scale for our method in Tab. [ Additional results are in Appendix F We observe that increasing the CFG scale beyond a certain point (around 3-4) leads to a degradation in both FID and controllability. while CLIP scores continue to improve. This trade-off between FID and CLIP scores has also been observed in prior work (2], and our results show similar trend extending to controllability. These findings highlight an open research question: how can we fairly evaluate controllable image generation, given the inherent trade-offs among controllability, FID, and CLIP, all of which are influenced by the CFG scale.
Regularization Strength and Margin. We conduct an ablation study on the margin m and regular- ization strength À in Eqs. (9) and (TO) evaluate the effect of each component in our design. Results are reported in Tab. 7 In the first three rows, we fix À and vary the margin m. In the last three rows, we fix and vary A. The case of "no margin" corresponds using an untruncated contrastive loss. To better interpret the results, we evaluate checkpoints that achieve similar mIoU when varying m. For the regularization strength X, models are trained for 10K steps. Interestingly, when À is set too high, performance deteriorates. This may be due to the pretraining loss dominating the CPO objective, causing the model to revert toward the original ControlNet behavior. When the regularization term removed (i.e., X= = 0). controllability improves substantially, reducing the error rate by around 10%, but both FID and CL.PP scores degrade. At 0.15, controllability gains are smaller, but FID and CLIP scores improve. Notably, although models trained without regularization (A = 0) achieve worse FID, the drop in visual quality is not always perceptible to humans, as shown in our Appendix F
Conclusion
In this paper, propose Condition Preference Optimization (CPO), a low-variance training objective compared to DPO. CPO achieves better FID scores at matched Controllability, benefiting from reduced variance and disentangled training signal. In addition, CPO supports training arbitrary timesteps, allowing it to generalize beyond ControlNet++ and achieve state-of-the-arr controllability across variety of tasks. Through evaluation across different classifier-fres guidance (CFG) scales, we also uncover a broader challenge in the fair evaluation of controllable image generation: specifically, how to jointly assess controllability, FID, and CLIP scores.
Acknowledgment
This work was supported by the intramural research program of the U.S. Department of Agriculture, National Institute of Food and Agriculture via grant number 2024-67022-41788. Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and should not be construed to represent any official USDA or U.S. Government determination or policy.
10

---

References
[1] Jonathan Aja Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6885 6851, 2020.
[2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[3] Prafulla Dhariwal and Alex Nichol. Diffusion models beat gans on image synthesis, 2021.
Evan Shelhamer, Jonathan Long, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. IEEE transactions on pattern analysis and machine intelligence, 2015.
[5] David Eigen, Christian Puhrsch, and Rob Fergus. Depth map prediction from a single image using a multi-scale deep network. Advances in neural information processing systems, 2014.
[6] John Canny, A computational approach to edge detection, IEEE Transactions on pattern analysis and machine intelligence, 1986.
[7] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models, 2023.
[8] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAl conference on artificial intelligence. 2024.
[9] Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shanzhe Hao, Lu Yuan, and Kwan- Yee K Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 2023.
[10] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang. Juan Carlos Niebles, Caiming Xiong, Silvio Savarese, al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint arXiv:2305. I 147, 2023.
[11] Ming Li, Taojiannan Yang. Huafeng Kuang, Jic Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnct++: Improving conditional controls with efficient consistency feedback: Project page: liming-ai. github. iofcontrolnet plus_plus. In European Conference on Computer Vision, 2024.
[12] Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-or. Prompt-to- prompt image editing with cross-attention control. In ICT.R, 2023.
[13] Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmid- huber. Cross-attentics makes inference cumbersome in eexttooimaae diffusion models. arXiv preprint arXiv:2404.0274/ 2024.
[14] Yogesh Balaji. Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song. Qinsheng Zhang, Karsten Kreis. Miik Aittala, Timo Aila, Samuli Laine, al. ediff-i: Text-to-image diffusion models with ensemble of expert denoisers. arXiv preprint arXiv:22/1.00 (tt::2//1.01324, 2022.
[15] Or Patashnik, Daniel Garibi, Idan Azuri, Hadar Averbuch- Elor, and Daniel Cohen-Or. Localizing object- level shape variations with text-to to-image diffusion models. In ICCV, 2023.
[16] Mingyang Yi, Aoxue Li, Yi Xin, and Zhenguo Li. Towards understanding the working mechanism of text-to-image diffusion model. arXiv preprint arXiv:2405. 1533000 2024.
[17] Qin Guo and Tianwei Lin. Focus on your instruction: Fine-grained and multi-instruction image editing by attention modulation. In CVPR, 2024.
[18] Enis Simsar, Alessio Tonioni, Yongqin Xian, Thomas Hofmann, and Federico Tombari. Lime: localized image editing via attention regularization in diffusion models. arXiv preprint arXiv:2312.09256 2023.
[19] Ming Li. Xin Gu, Fan Chen, Xiaoying Xing, Longyin Wen, Chen Chen, and Sijie Zhu. Superedit: Rectifying and facilitating supervision for instruction- based image editing. arXiv preprint arXiv:2505.02377, 2025.
[20] Bram Wallace, Meihua Dang, Rafael Rafailov. Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong. Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.

---

[21] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong. Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: learning and evaluating human preferences for text-to-image generation. In Proceedings of the 37th International Conference on Neural Information Processing Systems, 2023.
[22] Bolei Zhou, Hang Zhao, Xavier Puig. Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 2019.
[23] Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018.
[24] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted InterventionMIICAl. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, 2015.
[25] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations. 2021.
[26] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Informaiion Processing Systems, 35:5775-5787, 2022.
[27] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint
2022.
[28] Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor-correcto framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, 2023.
[29] Senmao Li, taihang Hu, Joost van de Weijer, Fahad Khan, Tao Liu, Linxuan Li, Shiqi Yang. Yaxing Wang. Ming-Mine -ing Cheng, and jian Yang. Faster diffusion: Rethinking the role of the encoder for diffusion model inference. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[30] Wentian Zhang, Haozhe Liu, Jinheng Xie, Francesco Faccio, Mike Zheng Shou, and Jürgen Schmidhuber. Cross-attention makes inference cumbersome in text-to-image diffusion models. arXiv e-prints, pages arXiv-2404, 2024.
[31] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high nighuresolutton image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2021.
[32] Diederik P Kingma, Max Welling, et al. Auto-encoding variational bayes, 2013.
[33] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jac Lec. Gligen: Open-set grounded text-to-image generation. CVPR, 2023.
[34] Zonglin Lyu, Ming Li, Jianbo Jiao, and Chen Chen. Frame interpolation with consecutive brownian bridge diffusion. In Proceedings of the 32nd ACM International Conference on Multimedia, 2024.
[35] Bo Li, Kaitao Xue, Bin Liu, and Yu-Kun Lai. Bbdm: Image-to-image translation with brownian bridge diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern Recognition, 2023.
[36] Zongming Li, Tianheng Cheng, Shoufa Chen, Peize Sun, Haocheng Shen, Longjin Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang. Controlar: Controllable image generation with autoregressive models. In International Conference Learning Representations, 2025.
[37] Maxime Oquab, Timothée Darcet, Théo Moutakanni. Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2024.
[38] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Teirth- Conference on Neural Information Processing Systems, 2023.
[39] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 2022.
12

---

[40] Yuval Kirstain, Adam Polyak. Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick-a- pic: An open dataset of user preferences for texi-to-image generation. Advances Neural Information Processing Systems, 2023.
[41] Kawin Ethayarajh. Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01306 2024.
[42] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13556, 2020.
[43] Shufan Li, Konstantinos Kallidromitis. Akash Gokul, Yusuke Kato, and Kazuki Kozuka. Aligning diffusion models by optimizing human utility. arXiv preprint arXiv:2404.04465, 2024.
[44] Huaisheng Zhu, Teng Xiao, and Vasant Honavar. DSPO: Direct score preference optimization for diffusion model alignment. In The Thirteenth International Conference on Learning Representations, 2025.
[45] Ralph Allan Bradley and Milton E Terry. Rank analysis of incomplete block designs: I. the method of paired comparisons. Biometrika, 1952.
[46] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECC EEC- 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part v 13, 2014.
[47] Xuan Ju, Ailing Zeng, Jianan Wang, Qiang Xu, and Lei Zhang. Human art: A versatile human dataset bridging natural and artificial scenes. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
human-centric centric
[48] Glenn Jocher and Jing Qiu. Ultralytics yolol1, 2024. URL https://ggthub..com/ullaaaaa ultralytics
[49] Jie Yang, Ailing Zeng, Ruimao Zhang, and Lei Zhang. X-pose: Detection any keypoints. ECCV, 2024.
[50] Gaurav Parmar, Richard Zhang, and Jun-Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
[52] Jonathan Ho and Tim Salimans. Classifier-Free diffusion guidance. arXiv preprint arXiv:2207.12598 2022.
[53]
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
[54] Nlya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[55] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models, Advances neural information processing systems, 35:25278 25294, 2022.
13

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No], or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] providedd proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
See Appendix. A
14

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
See Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We will open-source code, checkpoints, dataset, and required versions of packages. Training parameters shown in Appendix E
Guidelines:
The answer NA means that the paper does not include experiments.
15

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g.. with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We will open-source everything, but we may not be able submit it anony- mously at this point.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/public/eee.c www.eeddeee.oeuuummmcecoed ssi ssionyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
16

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: See Sec. 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
This is not computationally feasible.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
17

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///wwoorg..ccaaaaall/acllca
Answer:
[Yes]
Justification:
We conform with code of ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [No]
Justification: Our dataset does not contain unsafe issue, but for models, we are unable to provide
Guidelines:
18

---

The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We uses the code and Models from ControlNet and ControlNet++ and properly cite them. Datasets are also cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of license (e. CC-BY 4.0) should included for each asset. For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssiiipssiiiiaea... phcerssithcode thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of derived asset (if has changed) should be provided. this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We properly documented.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details the dataset/code/moddll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
19

---

Justification:
Not involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not involved.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification:
Core method does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy http:s/s/neerrppecc/////oeeereee for what should or should not be described.
20

---

Appendix
Our appendix is structured as follows:
Limitations and Broader Impact.
Full derivation of the CPO loss.
Proof of claim that the CPO loss has lower variance than DPO.
Comparison between the CPO and DPO datasets, including analysis of computation and storage costs.
Training details, including hyperparameters and hardware specifications.
Additional ablation studies.
Additional visual results and selected failure cases.
A Limitations and Broader Impact
Linitation and Future Work. Recent works such as ControlAR 33 enable multi-resolution training and inference, while ControlNet-based methods (using Stable Diffusion v1.5 as the base T2I model) typically trained and evaluated fixed 512 512 resolution. Additionally, may be interesting to apply our method to other RLHF algorithms such as KTO 41
Broader Impact. As controllable generative model. method may have positive societal impact in areas such as privacy preservation for example, generating images conditioned on Lineart extracted from humans obscure identity. However, similar to general-purpose generative models, carries risks of malicious misuse, including the generation unsafe or inappropriate content (e.g.. nudity).
B Full Derivation of CPO loss
Our Condition Preference Optimization (CPO) loss is defined as:
LCPO = -E ,e xo log a Exprr (xarr (xo) (log Pe (x (c") log pa (xxlee))
per et) )] (12)
a is the sigmoid function. Since sampling from pe is intractable, we approximate it using the diffusion process defined in Eq. @ Because the log-sigmoid function is convex, can apply Jensen's inequality to move the expectation outside, resulting in the following (condition с omitted for notational simplicity):
LcPO 5 -x o'), logo ( (gL( Xo, t) po Xx, )))
DKL DKL(q(X;-1
Xo, t) Pret(xx-1 X1, c²))
(13)
DDxI((Xx,1
Xo,, t) || Pa (X X,,
+ DKI( t) (rer(X;-1 Xt.
By 1. each KL. divergence term be simplified, yielding the following final form:
Lcro Elee..exx logo (-BIw()))
(lle (
cx, ,n .0)11 - ||e c,,(x,, ct
1)1 .
(14)
[11--60xx c¹, t)1 - -1e-- - c¹ , t)133)
21

---

C Proof of Loss Variance
We include the proof that our CPO loss achieves lower variance here.
Decomposition of inputs. We write each of the two inputs ut ("winning") and u ("losing") as
/= + 8 + 8 Quiss
(15)
For example, u= is x for DPO and X2, c.' for CPO. u defined similarly for the losing example. is shared baseline, specifically original input x,, C, Setrl is the deviation due to condition alignment, and ónuis collects all other deviations. Define
Actil 8 dar
Amuis
muist
ut - uuuua
Actrl + Anuis-
(16)
nuis
Centering nonzero means. If the deviations have nonzero mean, set
Metrl
E[Actt],
Mnuis
E[Anuis]:
(17)
and define residuals
Actrl Actel
Metrl,
Aniis
Anuis
Pmuis.
(18)
3. Linear approximation of the score difference. We first define: su(u) - -llea( el12 - -xeee(xe, c) el12.
(19)
We ignore input as it is not relevant. Let (.) By first-order Taylor expansion,
As = so(u*) so(u" = (g. a+ - u = (g. Metrl + Mmuis) + (g, Aetrl + Amuis)
(20)
Variance decomposition. The variance of As
Var[As] = (1g, Z) B[(g. 2)1)²]
(21)
= E[(9, 2) - (El 2)1)²
(22)
B (0 (Actril - T Amnis)) (g [e|Actr + Ammis))
(23)
= E[(a + Ammn + Amnis) " g (E[A + Amuis (E[ + Amnis) (24)
Var[As] T
[Coo(Aaan))
Cov Cov(Anuis) +2 Cov(A Amis) g.
(25)
Vesei
Vanta
Vorces
DPO case. Due to the inherent randomness of sampling process, both Aeul and Amuis vary, giving Var = g (Vear) + Vauis + Verows (26)
Vetrl > 0, Vnuis > O, Verosss *
CPO case. Since we fixed images and only varied conditions, we have
Viuia 0, Vaross 0, Vetrl Cov(Aatt) > 0
- Var|As] = Ven g > 0.
(27)
Note that we have
53
A - Ag 9" Bg
(28)
Since covariance is always positively semi-definite, by comparing Eqs.
26 and 27 we can find that
the variance of CPO case is smaller since it only contains variance from perturbing the condition. Thus, the training objective only focuses on controlabiiiiyy
22

---

Table The optimal storage storing preference pair is equivalent storing 1.66 RGB images). Note segmentation maps can stored as labels, and poses can be stored as keypoints.
preference pair in terms of the number of RGB images (i.e. 1.66 means
With original image
Without original image
DPO
3.66
2.66
Ours
1.66
1.66
Table 9: Training Hyperparameter.
Prompter
Segmentation
Pose
Canny
HED
Lineart
Depth
ADE-20K
COCO-Stuff
COCO-Pose
MultiGen- 20M
Learning rate Optimizer
le-8
le-8
le-7
3e-9
1e-8
3e-9
1e-8
AdamW
[54]
A
0.05
0.05
0.1
0.05
0.05
0.02
0.05
m
0.01
0.005
0.01
0.005
0.01
0.05
0.01
Number of steps
10K
10K
20K
10K
10K
10K
10K
Batch size
16
16
16
256
256
256
256
D Comparison to DPO dataset
To generate DPO dataset, we need sample 1", I' using generative model G (e.g., ControlNet or ConrolNet++ (II).. However, since both samples are drawn from the same distribution (i.e., the sampling process of G), their alignment with may be nearly indistinguishable. In practice, must sample multiple images and select the most and least aligned ones. In our experiments shown in Fig. we sampled 20 images. We assume the controllability score s of generated image follows a distribution S, where S denotes the distribution of controllability scores for all images sampled by model G. According to order statistics, for any distribution S. si...., $20 S, then P (min(f* [sn))2 (==)) < s' < max((s,, )
n-1 Here, s' is new sample drawn from S. When n+11
= 20, this probability is approximately 90%, implying that I" achieves controllability score in 90th percentile compared to a random sample Even under this setup, we still cannot observe clear differences in controllability from raw images, shown in Fig. Te).
Moreover, this case, DPO requires 20x more computation than our method, which severely limits scalability. In practical applications, the storage requirement for the DPO dataset is also larger, as shown in Tab. 8 If the DPO dataset does not store the original image, our method requires only 5 of the storage. If the DPO dataset includes original images to support regularization terms, our method & of that required by DPO.
reduces storage II
Beyond efficiency, our method may also mitigate safety concerns by storing only c'. DPO dataset curation involves generating full images, which can raise safety issues. In contrast, since our method involves only the generation of control signals (e.g., edges, depth, segmentation maps), it avoids such risks.
E
Training Details
We include our training details on hyperparameters Tab. 9 The training with batch size 16 and 10K steps takes approximately hours finish with 2 H100 GPUs. Readers may assume linear scaling when increasing the number of steps and batch size. Non-mentioned hyperparameters are in their default setup, such optimizer configuration.
F
More Ablation Studies
Ablation Studies on DINOv2. ControlAR 36 reveals the effectiveness of extracting image condition features with the DINOv2 model 37,, which improves the image quality and controllability. We find that the DINOv2 is also effective in diffusion models, shown in Tab. and Tab. In general,
23

---

Table 10: More Ablation Studies CFG Scales.
CRG scale
Seg. (ADE-20K)
Seg. (COCO-Suff)
Conny
HED
Lincart
Depch
mFal///FDD,/CLA
mla///FIDyCLL JP
SEYRDD.CLPPP
SSIMYEIDD
SSIMYFIDJXCIIP
RMSEL/IDDJCLLPP
1.5
45. 18/28 25/30.95
34,21/17. 10/31.0
39,49/20 (0922044//29.0
38829/12.73/30.5
0.8607/14 30
25.35/17,5429966
46.4022
16.0//6.30/32.00
3,582/13,28/31111
2561711.71/1.44
38/27 5W51
(011701/32.22
39.68/88.9/5111
0.899112.24/1155
2.99/12
25.014462111
7.5
44,81/30 30751.97
35. 49.4 S0/32.36
39.28/19, 6XX31
0.8201/1..353222
0.8457/13.25/31.
25 (44915.88/32.1
ConrrolAR (4.0)
39, 39.95/27.15/.
37.49/14,51/31.09
36,78/19 06.78/19.00029.3
12
0.8184/4.03230.82
0.7922/1244/
29.3817.7029919
Ours
Controttat++
Seg (ADE2OK)
(COCOStuff)
Canry
46.1
20.5
9001
45.0
942011
20.0
46.3
46.0
20.5
38.0
CTO
Depth
0.800
26.0
0.825
0.835
21.0
0.820
26.5
0.545
0810
0.840
25.5
200
1,5
15
Figure 7: Comparisons with ControlNet++ under different CFG scales.
D2 feature extraction improves controllability image quality, and text-to-image alignment in most scenarios.
Additional Ablation Studies CFG Scales. We include additional ablation results in Tab. 10 We observe that after the CFG scale reaches 3-4, further increasing the CFG scale leads to a degradation in controllability. We also include graphical illustration of controllability improvement under different CFG scales in Fig. 7 Under CFG scales 1.5, 3.0, 4.0, and 7.5, our method outperforms ControlNet++ with notable margin.
Visual Comparisons on Different Regularization Strengths. As discussed, increasing controllabil- ity by adjusting the regularization strength leads to slight drop in FID score, but the difference is not visually noticeable. Visual comparisons are presented Fig. where quality differences are minimal and difficult perceive due to their subtlety.
G Additional Visual Examples
We include additional visual comparisons with SOTAs in Fig. 9 and visual comparisons between CPO and DPO Fig.
TT We implement our method on Lont--oottol for the Lineart task and include the visual comparisons between our methods and promeAl's 's open-sourced FLUX-Come eet for Lineart in Fig.
10 The training data for NL obtained from a subsample of To generate the losing condition, we use our CPO models to generate samples, detect conditions, and resize c' to 1024 pixels. For demo showcase, we include additional visual
LAION-5B 55
examples for Pose in Fig.
12 Segmentation in Fig. 13
HED in Fig. 14 Lineart in Fig. [5] Canny in
Fig. 16 and Depth Fig. 17
Failure Cases. We present failure cases in Fig. 18 Our method struggles with extremely dense control conditions and with generating fine details for small human faces-a common limitation across all diffusion- based models.
24

---

Gnerated Images & Extracted Condtions
Input& & Condition
== 0
À = 0.05
2 = 0.15
FIMIBNUL
Figure 8: Visualization with different A.
25

---

Original Image Condition
Ours
ControlNett+
ControlAR
ControINet
HED
Canny
Depith
Segmentatio
Figure 9: Additional Qualitative Comparison in Controllability. Red boxes indicate the area where our method achieves better controllability.
26

---

Original Image & Condition
Generated Images & Extracted Conditions
PromeAl FLUX- ControiNet
PromoAI FLUX- ControiNet
Ours
Figure 10: Comparison with PromeAl
27

---

Original Image
Pose
CPO
DPO
Figure 11: visual Comparisons between CPO and DPO in Pose task.
Generated Images
Original Image
Pose
Figure 12: Additional Visual Examples for Pose.
28

---

Original Image & Condition
Generated Images Extracted Conditions
Figure 13: Additional Visual Examples for Segmentation maps. Note that black areas in ground truth segmenta- tion are "background" and not considered for correctness.
29

---

Original Image & Condition
Generated Images & Extracted Conditions
Figure 14: Additional Visual Examples for HED.
30

---

Original Image & Condition
Generated Images & Extracted Conditions
Figure 15: Additional Visual Examples Lineart.
31

---

Original Image & Condition
Generated Images & Extracted Conditions
DRENOE
Figure Additional Visual Examples for Canny.
32

---

Original Image & Condition
Generated Images & Extracted Conditions
Figure Additional Visual Examples for Depth.
33

---

(a) Highly dense control signals
Input and condition
Generated images and extracted conditions
(b) Small human faces
Condition
Generated images
Figure 18: Failure When the condition is highly dense, the generated images hardly follow it. (b) Similar the problem with all diffusion models, our method struggles with human generation when humans are small but require details in their faces.
34


Reviews:
[{"review_id": "DL3emx77Xi", "reviewer": "Reviewer_HcdE", "summary": "The paper proposes the CPO (Condition Preference Optimization) method, which can improve the compliance of controllable image generation with control signals. The CPO method trains the model to prefer winning control signals by generating both winning and losing control signals. Compared to DPO, it requires significantly fewer computation for training data generation, making it more efficient. Compared to SoTA methods, it achieves better controllability. Under control signals such as segmentation and pose, it demonstrates a notable reduction in errors.", "questions": "- Is the improvement related to the correlation between the detector model for data generation and evaluation? If a different detector model is used during evaluation, will the same improvement still be achieved?  Taking Pose condition as an example, the CPO training data was generated using YOLO-11x-Pose, and the mAP calculation also used YOLO-11x-Pose. Could it be possible that visually, the pose of a person in the image appears incorrect, but YOLO-11x-Pose considers the pose more consistent? For the Pose mAP evaluation in Table 1, could another pose estimation model be used for assessment?\n- The comparison with DPO is not sufficiently thorough. Could you provide more qualitative results?  \n- Could you also provide more qualitative comparative results?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work. \"yes\".", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strength\n- The paper demonstrates a clear improvement in controllability, such as achieving  10%+ error rate reduction in segmentation, 70–80% error rate reduction in human pose, approximately 2–5% reduction on edges and depths after fine-tuning. This error reduction is relative to the model before fine-tuning.\n- The writing of the paper is good, and it is quite readable. There is theoretical proof that the variance of CPO is smaller than that of DPO.\n- The motivation of the paper is clear. The method is significantly more efficient than DPO, and easier to generate training data.\n\nweakness\n- The number of images for qualitative evaluation is too limited. In Section G, the focus is mainly on the generation results of this method, lacking more qualitative comparisons with other methods. The evaluation results are primarily based on the quantitative outcomes of  algorithms, without comparative assessment through a user study.\n- There is a risk that training is favorable for a specific detection model, rather than improving the image information. The model used to generate data is the same as the one used for evaluation, such as YOLO-11x-Pose for pose estimation.", "was_revised": true, "final_justification": "My main concern is about overfitting to detection model for data generation. The rebuttal has successfully addressed this concern.\n\nIt is not possible to access more qualitative results, but given the large numerical improvement and the managable risk of overfitting, I tend to believe the qualitative quality without seeing more images.\n\nThe method is simple, effective and widely applicable.\n\nConsidering these factors, I decide to raise the rating to accept.", "rebuttal": "We thank the reviewer for providing constructive comments. The reviewer appreciates our motivation, writing, and results.\n\n**Questions:**\nThe number of images for qualitative evaluation is too limited, without comparative assessment through a user study.\n\n**Answer:**\nWe’d like to let the reviewer know that we follow the practice of ControlNet++ and ControlAR to provide a qualitative Comparison (one group of visual comparison for each task (in the main paper), and several visual examples without comparison (supplementary Figure 8 - 13)). We are, of course, willing to provide additional qualitative results. However, this year Neurips does not allow us to upload a PDF with figures, nor the links. We will include more in the final version.\n\nFor the user study, we conducted a user study on controllability on hed and depth. We randomly select 25 groups of images for each task and ask 15 annotators to annotate.\n \nHere are the results in win rate (%):\n\n| Task| CPO  | ControlNet++| ControlAR|\n|----------|-----|------------|-----------|\n|HED |47.73% |   37.87%  | 14.40%  |\n|Depth |46.67% |  40.27%  | 13.06%|\n\nIn hed, we outperform ControlNet++ by 10 percentage points and 6 percentage points in Depth.\n\n---\n\n**Questions:**\nIs the improvement related to the correlation between the detector model for data generation and evaluation? If a different detector model is used during evaluation, will the same improvement still be achieved? Could it be possible that, visually, the pose of a person in the image appears incorrect, but YOLO-11x-Pose considers the pose more consistent? For the Pose mAP evaluation in Table 1, could another pose estimation model be used for assessment?\n\n\n**Answer:**\nWe do understand the reviewer’s concern that the improvement might be because we overfit to the pose detector. We provide two types of experiments to resolve the concern. First, our experiments show that changing the detector during dataset curation (i.e., the detector to construct $c^w$ and $c^l$) almost does not change the results. See the table below\n\n| Task & detector    | Pose + YOLO11x| Pose + mmpose | ADE20K + UperNet-ResNet |ADE20K + UperNet-ConvNext |Depth + DPT-Large |Depth + MiDaS |\n|----------|------------|------------|------------|------------|------------|------------|\n| Our | 87.98 |   87.71 |  44.81   | 44.90 | 27.49 | 27.56|\n| Base Model| 72.53 |   72.53 |  43.64   | 43.64 | 28.32 | 28.32|\n\nThe differences are primarily due to the randomness of the generative model, and we can conclude that there is almost no difference.\n\nWe also include mAP results where the dataset is curated with YOLO11x-pose but evaluated with a different model, mmpose (HRNet), below.\n\n\n|Methods| Cocopose | HumanArt|\n|----------|------------|-----------|\n| CPO| 95.48 |   67.29 | \n| Base Model| 91.34 |   60.44 |  \n\n\nWe can see that with HRNet, CPO still outperforms base models. HRNet tends to result in better evaluation results than YOLO11x-Pose. We will change the evaluation model to HRNet and update the entire table in our final version to resolve the reviewer’s concern. \n\n---\n\n**Question:**\nThe comparison with DPO is not sufficiently thorough. Could you provide more qualitative results? Could you also provide more qualitative comparative results?\n\n**Answer:**\nThis year, Neurips does not allow us to provide any PDF or links. Therefore, we are not able to provide during the rebuttal. However, our structure follows previous SOTA ControlNet++ and ControlAR to provide one set of qualitative comparisons (in the main paper) and more visualizations with our models only (in the appendix). We will, of course, provide more qualitative comparisons in our final version, including CPO vs DPO and ours vs others."}, {"review_id": "gsGYc5Nlte", "reviewer": "Reviewer_Q7TW", "summary": "The paper introduces a new Condition Preference Optimization (CPO) method that aligns the conditions directly with DPO instead of using images for controllable image generation. It uses the original condition as the winning condition while treating the generated condition as the losing condition. Experiments are conducted on segmentation, depth, canny, edge and pose conditions across 5 different benchmarks.", "questions": "How does the performance vary with different detectors? How many images are sampled to obtain the losing condition? Is it just one?\n\nHow does the performance compare to training baseline models for longer (same amount of time as CPO) with the new data? Is the improvement solely because of CPO?\n\nWhat is the effect of this alignment on the background regions? The model’s performance is specifically optimized to improve the controllability in the regions specified by the conditions. Does it degrade the performance for those regions outside the conditions?\n\nWill there be any interference when CPO is performed on top of DPO or vice-versa?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**: The paper proposes a novel method for improving the alignment and controllability in conditional image generation with the CPO method that uses conditions as preference images instead of their RGB counterparts. Experimental results show improved performance over the baseline ControlNet or ControlNet++ across pose, depth, segmentation and sketch conditions.\n\n**Weaknesses**: \nThe proposed method uses a discriminative model (condition detector) to obtain the losing conditions from the generated images of the baseline model. However, this assumes that the detector is perfect which is not the case in practice. This results in error propagation to the CPO where even good images will be rejected. So, it is sensitive to the detector used. How does the performance vary with different detectors? How many images are sampled to obtain the losing condition? Is it just one?\n\nThe proposed method requires a strong pretrained model as initialization such as the ControlNet or ControlNet++ and so the computational complexity increases. How does the performance compare to training baseline models for longer (same amount of time as CPO) with the new data? Is the improvement solely because of CPO?\n\n The performance improvements when evaluated under the same CFG configuration of 7.5 listed in Tables 1 and 2 are not very significant compared to the additional complexity involved in training. What is the performance difference between the methods when using the CFG scale of 4.0? It is not clear why the performance changes a lot at CFG 7.5 especially since the initialization is from a base model that uses a CFG of 7.5.\n\nWhat is the effect of this alignment on the background regions? The model’s performance is specifically optimized to improve the controllability in the regions specified by the conditions. Does it degrade the performance for those regions outside the conditions? This can also be seen from the FID scores reported in Table 2 where the performance is comparable or slightly inferior to the baseline models.\n\nIn practice, we require both controllability and high-quality images. So, preference along both dimensions is essential. Will there be any interference when CPO is performed on top of DPO or vice-versa?", "was_revised": true, "final_justification": "I appreciate the authors in providing additional experimental evaluations that addressed most of my concerns on error propagation, performance improvement and training time. Overall, the method shows good empirical gains over the baselines but it still suffers from the limitations of the original DPO method such as the noisy preference issue. A more thorough discussion and analysis on the noisy preference problem in the paper will be beneficial. Therefore, I increase my rating to borderline accept (4).", "rebuttal": "We thank the reviewer for the constructive comments. The reviewer does agree with our novelty.\n\n**Question 1: On detector sensitivity and error propagation.**\nThe reviewer noted that our method uses a condition detector that is not perfect, which could lead to error propagation and the rejection of good images.\n\n**Answer:**\nThank you for this question. Our method fundamentally avoids this issue for the following reasons:\n1. We Contrast \"Conditions,\" Not \"Images\": Our method (CPO) contrasts a \"winning condition\" with a \"losing condition,\" not the images themselves. Both conditions are applied to the same input image. The winning condition is always the ground truth or an oracle. Therefore, the losing condition is, by definition, less aligned with the input image than the winning one. This mechanism prevents good conditions from being mislabeled as losers, thus eliminating the risk of error propagation.\n2. Robustness to Different Detectors: To further validate this, we experimented with different detectors for segmentation (ADE20K), pose (COCOPose), and depth (MultiGen20M) tasks. As shown in the table below, the performance shows almost no difference, demonstrating our method's strong robustness to the choice of detector.\n3. Yes, we only sample one image to obtain the losing condition. \n\n| Task & detctor    | Pose + YOLO11x| Pose + mmpose | ADE20K + UperNet-ResNet |ADE20K + UperNet-ConvNext |Depth + DPT-Large |Depth + MiDaS |\n|----------|------------|------------|------------|------------|------------|------------|\n| Our | 87.98 |   87.71 |  44.81   | 44.90 | 27.49 | 27.56|\n| Base Model| 72.53 |   72.53 |  43.64   | 43.64 | 28.32 | 28.32|\n\n---\n\n**Question 2: Is the improvement solely due to longer training, not CPO itself?**\n\n**Answer:**\nThe performance improvement stems entirely from CPO, not from longer training. We demonstrate this with the following experiments:\n1. Extended Training of the Baseline: We trained the baseline model, ControlNet++, for the same training step as CPO. As shown below, it began to overfit on ADE20K (performance degraded) and showed negligible improvement on other tasks. In contrast, our CPO method consistently improves controllability without sacrificing FID.\n| Task    | ControlNet++| ControlNet++ more training | ours  |\n|----------|------------|------------|------------|\n| Seg (ADE20K)    | 43.64/30.24  |   40.91/32.57       |  44.81/30.30   |\n| Canny| 38.03/20.16 | 39.01/22.50  | 39.19/19.68    |\n| HED   | 80.97/15.01 | 80.81/14.92    | 82.01/13.35   |\n| Lineart  | 83.99/13.88  | 84.04/13.80    | 84.31/13.84    |\n| Depth | 28.32/16.66 | 28.15/16.78   | 27.49/16.64    |\n\n2. Applying CPO on a Fully Converged Model: On the COCOStuff dataset, we train the baseline model (ControlNet++) until full convergence (5000 more steps) because we observe the under-convergence of ControlNet++. Further training beyond 5000 steps does not improve controllability without sacrificing FID a lot. The performance is approximately 90% of Oracle, which is extremely challenging to further improve. Even so, our CPO still achieved a noticeable improvement, proving its effectiveness beyond simple training extension.\n| Task    | Fully converged ControlNet++| ours  |\n| --- | --- | --- |\n| Seg(COCOStuff)     | 38.02/19.24 | 38.44/19.20 |\n\n---\n\n**Question 3: Insignificant performance improvement and the effect of CFG-scale.**\n\n**Answer:**\n1. On the Significance of Improvement: The improvement might seem modest for two main reasons:\n  - The baseline is already very strong: ControlNet++ achieves 80-84% of oracle performance on most tasks, leaving limited room for improvement.\n  - Balancing Controllability and FID: Our goal is to enhance controllability while maintaining or improving image quality (FID). As shown in the previous answer, longer training of ConrolNet++ either fails to maintain FID (canny) or improve further on controllability (other tasks), whereas CPO succeeds in achieving better controllability. \n2. On CFG-scale: It is important to clarify that the CFG-scale is an inference-time technique, unrelated to model training or its parameters. It adjusts the strength of conditional guidance using the formula $pred = (c+1) \\times pred_{cond} - c \\times pred_{uncond}$. $pred_{cond}$ means the model output with full text input, and $pred_{uncond}$ means the model output with empty text input. Different CFG scales will thus naturally alter the output and the resulting evaluation scores. This is a standard practice for diffusion models and not a unique aspect of our method. \n\n**Question 4: Effect on background regions.**\n\n**Answer:**\n1. Most Tasks Lack a Distinct \"Background\": In tasks like edge detection (Canny, HED, Lineart) and depth mapping, there is no strict \"background\" region. Every part of the image contains relevant conditional information. For example, the black region of edge detection tasks means **no edges** instead of **no requirement**.\n\n2. Background Quality is Not Degraded: For tasks with backgrounds, such as segmentation and pose estimation, Figures 8 and 9 in our appendix clearly show that the quality of background regions is not degraded.\n3. On the FID Score for HumanArt: The drop in FID on this dataset is not due to lower quality but to a data distribution shift. Our model was fine-tuned on the realistic COCO-Pose dataset, whereas HumanArt contains numerous cartoons and artistic paintings. Fine-tuning shifted our model's output distribution closer to real-world photos, naturally increasing the FID distance to the artistic style of HumanArt. This demonstrates that our model successfully learned the target data distribution. A similar pattern was observed in GLIGEN's results, as it was also trained on COCO-Pose only.  We will add examples to showcase it for a better explanation in our final version.\n\n---\n\n**Question 5: Potential interference when combining CPO and DPO.**\n\n**Answer:**\nThis is an interesting direction, but combining CPO with DPO presents practical challenges.\n1. Challenges in Data Curation: CPO leverages the model's inherent imperfection in controllability, requiring only one generated image to obtain an effective \"losing condition.\" In contrast, to ensure a \"winning image\" is superior in both quality and controllability, DPO would require generating and filtering a large number of samples, which is computationally prohibitive. Importantly, **DPO cannot be performed in quality only** since the alignment to control signals cannot be identical. However, CPO is performed for controllability only.\n\n2. Negative Interference: Our experiments confirm this. We split our training steps equally for CPO and DPO. As shown in the table below, combining DPO and CPO, regardless of the order, results in performance inferior to using CPO alone, especially in terms of FID. The reason is that we apply a worse method on top of a better one. Figure 6 in our main paper shows that DPO performs worse than our method.\n\n| DPO   | CPO| CPO then DPO| DPO then CPO |\n|----------|-----|------------|------------|\n|87.64/41.28  |   87.98/39.94    | 87.89/41.07  | 87.94/41.00  |\n\nWe do think this is an interesting idea for future improvement and research, and appreciate reviewers’ comments."}, {"review_id": "bqS8g26h5Y", "reviewer": "Reviewer_yv4M", "summary": "This paper aims to enhance the controllability of image generation using ControlNet. It begins by examining the shortcomings of existing methods based on Direct Preference Optimization, which often struggle with uncertainties inherent in generative models—specifically, a winning image may compromise on other factors such as overall image quality. To address these challenges, this paper proposes Conditioned Preference Optimization, which constructs preference condition pairs rather than relying solely on image pairs. The proposed method demonstrates superior performance compared to previous approaches.", "questions": "For the condition pair, how to control the differences? If the difference is very large, it might be invalid. The authors should detail the process of constructing the pair, such as pose, canny, depth, segmentation ,etc.\n\nThe backbone is still SD1.5. To better validate its effectiveness, please use SDXL, even SD3 and FLUX.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\nThe proposed method has clear motivation and analysis, providing analysis for previous methods.\n\nThe proposed method is simple but effective, with similar formulation to previous methods.\n\nThe paper also provides theoretical proof for the proposed method, lending it greater validity.\n\nThis paper is well-written. For example, the Fig.1 and Fig.2 can clear illustrate the motivation and strategy.\n\nWeaknesses:\n\nFor the condition pair, how to control the differences? If the difference is very large, it might be invalid. The authors should detail the process of constructing the pair, such as pose, canny, depth, segmentation ,etc.\n\nThe backbone is still SD1.5. To better validate its effectiveness, please use SDXL, even SD3 and FLUX.", "was_revised": true, "final_justification": "Thank the authors for their responses. I agree that due to the time limitation, the additional experiments are hard to complete. As the authors have made efforts to give explanations, I keep my initial rating.", "rebuttal": "**Question:**\nFor the condition pair, how to control the differences? If the difference is very large, it might be invalid. The authors should detail the process of constructing the pair, such as pose, canny, depth, segmentation ,etc.\n\n**Answer:**\nWe currently do not have a process to control the difference. The reviewer’s comments are helpful and may possibly provide a way to improve our work. Our pair construction process is described in lines 195-200. \n\n1. We first obtain the ground truth (segmentation/pose) or oracle (canny, lineart, hed, depth) as $c^w$. \n2. Then, we generate an image $I$ with ControlNet++ controlled by $c^w$.  \n3. Finally, we use the condition detector to extract $c^l$ from $I$.\n\nFor example, in the canny edge task, the condition detector is just a canny edge detector.\n\nSince ControlNet++ achieves strong controllability, we assumed the difference would not be large. However, this is more about curating a better dataset, and we will follow this direction and try to improve our dataset by adding some difference-control mechanisms. \n\n---\n\n**Question:**\nThe backbone is still SD1.5. To better validate its effectiveness, please use SDXL, even SD3, and FLUX.\n\n**Answer:**\nWe are actively working on FLUX training. However, since FLUX requires $1024 \\times 1024$ resolution images as inputs, we collected a high-resolution dataset first. We filtered out 3M images from the LIAON-Aesthetic-5+ dataset with an aesthetic score higher than 5.6, then we captioned all images with qwen-2.5-vl-32b for a high-quality text description. We are currently training FLUX-ControlNet, but FLUX takes weeks to train with 64 A100 80G gpus, and we only finished the first 5K steps of training FLUX-ControlNet in lineart task, but we do not have results now. It will take one more week to finish FLUX-ControlNet, if everything works as expected. Then we can proceed to CPO. We will include available results in our final version."}, {"review_id": "EYF2AudP6j", "reviewer": "Reviewer_YcYa", "summary": "This paper proposes using reinforcement learning for ControlNet-based image generation. Prior work such as ControlNet++ optimizes only low-noise timesteps, which can introduce optimization errors. While Direct Preference Optimization (DPO) can optimize across all timesteps, it lacks fine-grained control, making it difficult to improve controllability without affecting other generation factors. To address this, the authors introduce a novel Condition Preference Optimization approach that constructs winning and losing control signals and trains the model to prefer the winning ones. Experimental results demonstrate superior performance over both DPO and ControlNet++ across various control types and evaluation metrics.", "questions": "While the method shows good controllability scores, it sometimes yields lower FID and CLIP scores. It would be helpful if the authors could discuss this trade-off and provide insights into how controllability affects image quality and alignment with the text.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The proposed idea is novel and well-motivated. To the best of my knowledge, no prior work has attempted to apply the DPO method for conditional image generation following ControlNet. The authors also identify the challenges in directly applying DPO and introduce a new approach—Conditional Preference Optimization (CPO)—to address these issues.\n\n2. The paper is generally well-written and easy to follow.\n\n3. The proposed method achieves strong controllability scores, demonstrating the effectiveness of CPO. Additionally, the authors conduct thorough ablation studies comparing CPO with DPO and analyzing hyperparameter choices, which further strengthen the experimental results.\n\n\nWeaknesses\n1. Despite the novelty of the approach, it raises questions regarding the theoretical equivalence between the proposed CPO and the original DPO. As stated in the introduction, while the relationship between the control and the generated images may be bidirectional, directly switching to CPO changes the data distribution compared to DPO. If CPO is not strictly equivalent, some clarification or refinement in the introduction would be helpful to better explain the motivation and implications.\n\n2. In practice, when applying perturbations, how can we ensure that the perturbed control signals extracted from generated images share a similar distribution with real control signals? The paper mentions that the perturbations are derived from the generated images—but are they of comparable quality to the original control signals? Why not directly perturb the original control inputs instead?\n\n3. The authors should consider conducting a user study to better evaluate the results. Human judgment can provide a more reliable assessment of how well the generated images align with the control inputs, compared to simply comparing the extracted control signals from generated images. Based on the current results, the quality of the extracted control signals is highly dependent on the extraction method used. For instance, the depth map for ControlAR in Figure 4 does not appear meaningful. Additionally, the HED results represent particularly challenging cases, and I would not conclude that ControlAR performs worse than the proposed method in those examples. Furthermore, in Figure 5(a), the alignment between the human subject and the control input is not very accurate.", "was_revised": true, "final_justification": "My questions have been well addressed. I would appreciate a more detailed quantitative comparison between CPO and DPO, as well as additional visual comparisons. While another reviewer noted that the limited performance improvement might be a concern, I believe the proposed method shows noticeable improvements, as demonstrated in the tables and figures.", "rebuttal": "Before our response, we would like to explain a little about our intuition so that the reviewer can have a clearer understanding. Simply speaking, we construct a preference pair $c^w$ and $c^l$ and let the model learn that $c^w$ aligns better to the input image than $c^l$, where the input image is from the real-world dataset. \n\n**Question:**\nDespite the novelty of the approach, it raises questions regarding the theoretical equivalence between the proposed CPO and the original DPO. As stated in the introduction, while the relationship between the control and the generated images may be bidirectional, directly switching to CPO changes the data distribution compared to DPO. If CPO is not strictly equivalent, some clarification or refinement in the introduction would be helpful to better explain the motivation and implications.\n\n**Answer:**\n1. DPO and CPO are similar. They cannot be strictly equivalent due to their difference in formulation and intuition. They are similar in training loss formulation. In plain words, DPO does not want to sample worse images (i.e., keep the sampling trajectory far from the trajectory of worse images). This raises concerns mentioned in our intro, such as the inability to discern the condition from the image to perform meaningful preference. CPO deals with the condition instead. It tries to keep the sampling trajectory far from the less controllable trajectory. Therefore, it functions similarly to DPO. However, they are different in the way the CPO deals with the condition part (C) of a probability distribution $P(X|C)$, but DPO deals with the variable part (X).\n2. We did not mention the data distribution change in CPO. We did mention that the distribution mismatch in the DDPM forward and sampling process causes some limitations for ControlNet++. \n3. If the reviewer has concerns about the generation data distribution, then our method aligns better to real-world images than DPO. DPO requires high-quality model-generated data for training, and therefore, its training distribution diverges from the real-world image. However, we still keep the real-world images, but we contrast conditions. This may raise a concern about conditions distribution (the reviewer’s next question). But the condition distribution still aligns. For Canny, Lineart, Depth, and HED, conditions are extracted with the same mode for both winning and losing conditions, and therefore they are in the same distribution. For pose and segmentation, conditions are drawn with a pre-defined visualization method. For example, in segmentation, we map labels (argmax of output distribution) to predefined colors. In pose, we draw lines with predefined colors between keypoints. This process can also be understood as coloring models that map labels/keypoints, and the coloring models are identical for the winning (ground truth) condition and the losing condition. Therefore, the winning and losing conditions are in the same distribution because they are mapped to the image (pixel) space by the same model. Importantly, in the same distribution does not mean that the losing conditions extracted from generated images are correct. They will be incorrect due to the non-perfect controllability, which is what we desire to construct the losing condition $c^l$. \n\n---\n\n**Question:**\nIn practice, when applying perturbations, how can we ensure that the perturbed control signals extracted from generated images share a similar distribution with real control signals? The paper mentions that the perturbations are derived from the generated images—but are they of comparable quality to the original control signals? Why not directly perturb the original control inputs instead?\n\n**Answer:**\nWe include the response for the distribution in our previous answer. For the quality of the control signal, please see our previous response. We intentionally make the extracted control signals from the generated images worse to perform contrastive learning so that the model knows the winning signal is better than the losing one. For perturbing the original control inputs, we did think of doing such a processing at the beginning. However, the issue is that we need to carefully consider the richness of perturbation. Here are some problems that we encountered and found too hard to solve. Take the pose as an example, the direction and distance that we should move each keypoint is hard to determine. The perturbed keypoints look reasonable, so that the model can learn meaningfully from telling winning conditions are better? For edges, how can we perturb edges? Direct shifting to left/right does not change the shape of edges.  Given these challenges, we came up with our method. The perturbation process will be automatic, and the perturbed conditions still seem reasonable.\n\n---\n\n**Question:**\nThe authors should consider conducting a user study to better evaluate the results. Human judgment can provide a more reliable assessment of how well the generated images align with the control inputs, compared to simply comparing the extracted control signals from generated images.\n\n**Answer:**\nWe conduct a user study on hed and depth with controllability. We randomly select 25 groups of images for each task and ask 15 annotators to annotate. \n \nHere are the results in win-rate(%):\n\n| Task| CPO  | ControlNet++| ControlAR|\n|----------|-----|------------|-----------|\n|HED |47.73% |   37.87%   | 14.40%  |\n|Depth |46.67% |  40.27%  | 13.06% |\n\nWe see that we outperform ControlNet++ by 10 percentage points in HED and 6 percentage points in Depth. \n\n---\n\n**Question:**\nWhile the method shows good controllability scores, it sometimes yields lower FID and CLIP scores. It would be helpful if the authors could discuss this trade-off and provide insights into how controllability affects image quality and alignment with the text.\n\n**Answers:**\nThe only time the FID and CLIP are apparently lower is on HumanArt. We mentioned it in lines 270-273 in our main paper. The drop in FID on this dataset is not due to lower quality but to a data distribution shift. Our model was fine-tuned on the realistic COCO-Pose dataset, whereas HumanArt contains numerous cartoons and artistic paintings. Fine-tuning shifted our model's output distribution closer to real-world photos, naturally increasing the FID distance to the artistic style of HumanArt. This demonstrates that our model successfully learned the target data distribution. A similar pattern was observed in GLIGEN's results, as it was also trained on COCO-Pose only. CLIP score is impacted in a similar way. We will add examples to showcase it for a better explanation in our final version.\n\nFor any other datasets, the changes in FID/CLIP are ignorable due to the inherent randomness of generation models. If you evaluate the dataset twice with different environments, machines, random seeds, etc, results will be slightly different. Therefore, it is inconclusive for the FID/CLIP on other datasets.\n\nHowever, we do observe that, if we do not use regularization, the controllability will increase further with worse FID and CLIP, and we do not visually find a strong difference in image quality (we kindly refer the reviewer to Figure 7 in our appendix). One reason is that the model tries to overfit on controllability, and some unobservable features might be captured by FID/CLIP, which is the reason we use original training to regularize."}]

Author rebuttal:

Meta-review: The submission proposes Condition Preference Optimization (CPO), a new approach for enhancing controllability in image generation. Reviewers agreed that the paper is clearly written, well-motivated, and supported by sound theoretical analysis. Empirical results show consistent improvements across segmentation, pose, depth, and edge conditions, with notable efficiency advantages over DPO. The authors further strengthened their case through ablation studies, detector-robustness checks, and a user study, which confirmed better alignment compared to strong baselines such as ControlNet++. Some concerns were raised about limited qualitative comparisons, reliance on detector accuracy, and the use of SD1.5 rather than newer backbones. Additional experiments during the rebuttal—including cross-detector validation, CFG-scale analysis, and initial steps toward FLUX training—addressed these points effectively. While the performance gains are sometimes modest given strong baselines, the method remains simple, effective, and broadly applicable. Overall, the reviewers reached consensus that the paper makes a solid contribution to controllable generation research and warrants acceptance.
Decision: Accept (poster)