Ctrl-U: Robust Conditional Image Generation via Uncertainty-aware Reward Modeling
Authors: Guiyu Zhang
Huan-ang Gao
Zijian Jiang
Hao Zhao
Zhedong Zheng
Venue: ICLR 2025 Poster (2025)
Keywords: Conditional generation
semantic alignment
uncertainty learning
Primary area: generative models
TL;DR: To address the adverse effects of inaccurate reward feedback by enforcing alignment for conditional image generation, we introduce an uncertainty-aware reward modeling approach, which adaptively regularizes the reward learning process.

Abstract: In this paper, we focus on the task of conditional image generation, where an image is synthesized according to user instructions. The critical challenge underpinning this task is ensuring both the fidelity of the generated images and their semantic alignment with the provided conditions. To tackle this issue, previous studies have employed supervised perceptual losses derived from pre-trained models, i.e., reward models, to enforce alignment between the condition and the generated result. However, we observe one inherent shortcoming: considering the diversity of synthesized images, the reward model usually provides inaccurate feedback when encountering newly generated data, which can undermine the training process. To address this limitation, we propose an uncertainty-aware reward modeling, called Ctrl-U, including uncertainty estimation and uncertainty-aware regularization, designed to reduce the adverse effects of imprecise feedback from the reward model. Given the inherent cognitive uncertainty within reward models, even images generated under identical conditions often result in a relatively large discrepancy in reward loss. Inspired by the observation, we explicitly leverage such prediction variance as an uncertainty indicator. Based on the uncertainty estimation, we regularize the model training by adaptively rectifying the reward. In particular, rewards with lower uncertainty receive higher loss weights, while those with higher uncertainty are given reduced weights to allow for larger variability. The proposed uncertainty regularization facilitates reward fine-tuning through consistency construction. Extensive experiments validate the effectiveness of our methodology in improving the controllability and generation quality, as well as its scalability across diverse conditional scenarios, including segmentation mask, edge, and depth conditions.

Published as a conference paper at ICLR 2025
CTRL-U: ROBUST CONDITIONAL IMAGE GENERATION VIA UNCERTAINTY-AWARY REWARD MODELING
Guiyu Zhang*1.2 Huan-ang Gao*? Zijian Jiang?
Hao Zhao
Zhedong Zheng"!
1 FST and ICI, University of Macau 2 AIR, Tsinghua University
ABSTRACT
In this paper, focus the task of conditional image generation, where image is synthesized according to user instructions. The critical challenge under- pinning this task is ensuring both the fidelity of the generated images and their semantic alignment with the provided conditions. To tackle this issue, previ- ous studies have employed supervised perceptual losses derived from pre-trained models, ie., reward models, to enforce alignment between the condition and the generated result. However, we observe one inherent shortcoming: considering the diversity of synthesized images, the reward model usually provides inaccu- rate feedback when encountering newly generated data, which can undermine the training process. To address this limitation, we propose uncertainty-awart re- ward modeling, called Ctrl-U, including uncertainty estimation and uncertainty- aware regularization, designed to reduce the adverse effects of imprecise feed- back from the reward model. Given the inherent cognitive uncertainty within re- ward models, images generated under identical conditions often result in a relatively large discrepancy in reward loss. Inspired by the observation, we ex- plicitly leverage such prediction variance the uncertainty estimation, we regularize the model training by adaptively rectify- ing the reward. particular, rewards with lower uncertainty receive higher loss weights, while those with higher uncertainty are given reduced weights to allow for larger variability. The proposed uncertainty regularization facilitates reward fine-tuning through consistency construction. Extensive experiments validate the effectiveness of our methodology in improving the controllability and generation quality, as well its scalability across diverse conditional scenarios, including segmentation mask, edge, and depth conditions. Codes are publicly available at https://grenoble-zhangg httphamggggguuuubo//CO--..
uncertainty indicator. Based on
INTRODUCTION
Driven by the emergence of large-scale image-text datasets (Schuhmann et al. 2021 Changpinyo 202T Schuhmann al., (0222) and the development of diffusion models (Dhariwal & Nichol
et al.
Nichol et al. 2021 Rombach et al.] (022)), text-to-image (T2I) diffusion models (Ramesh) 2021: Saharia al. 2022) have become fundamental in the field of controllable visual gen- eration. These models excel at producing realistic, high-quality images that accurately reflect the descriptions provided in natural language. However, due their inherent properties, text-based con- ditions often struggle to convey the detailed controls required for final generation results efficiently. This limitation becomes especially evident in certain scenarios where text prompts alone can not capture all necessary details, such as when depicting unique artistic styles accurately rendering complex scenes. To this end, alongside text descriptions, a substantial body of research focuses on integrating novel conditional controls, such user-drawn sketches and semantic masks, into T2I diffusion models (Qin et al.) 2023 Ye et al. (2023] Zhang et al.) 2023a, Mou et al.) 2024). Despite investigations into the controllailiity of T2I diffusion models and their expanded applications (Gao
2021
al.
ct al. 2024 Li et al. 2024b; Xu et al.) 2024 Gu et al., 2024), achieving precise and fine-gra graind
control remains challenging. The primary issue lies in ensuring both the fidelity of the generated
"Equal contribution. The work is done during G Zhang's internship at the University of Macau. 1 "Corrsponniig authors.

---

Published as a conference paper at ICLR 2025
Sowe
0,99
0.25
Real Image
0.15
0.30
Input Condition
0.0 100
200
300
400
60.0 7C0
800
500
1000
(b)
Figure 1: Given test image and the layout condition, we employ diffusion model generate new images by adding noise and then recovering from the noisy input. (a) Ground-truth segmentation results with the category illustration. (b) Here we show the reward changes, i.e. mloU error, on newly generated images at different timesteps. The horizontal axis represents the current timestep t and the vertical axis shows the error, i.e., 1-mloU. As shown, even mloU errors. As increases, although the visual layout aligns with the condition, the reward model tends to increase the error, leading to the backpropagation of incorrect gradients.
O, there are non-zero
images and their semantic alignment with the provided conditions. To tackle this issue, some ef- forts, e.g., ControlNet++ (Li et al. 2024a) attempt to employ pre-trained reward model to extract the corresponding condition from the generated images and enforce alignment between the specified condition and the output. However,
observe that the reward model inevitably produces inaccu- rate feedback (see Fig. T]. During the diffusion training, we add different levels of Guassian noise to the input, which increases the reconstruction challenge. It also leads the distribution discrepancy between the generated image and the real image. As the timestep t increases, such generation dis- crepancy increases. The reward model has not "seen" such generation discrepancy before, resulting false segmentation prediction, even if the generation is correctly aligned with the condition. Due to the diffusion compression process, we observe that even if O, there is some mis-alignment feedback as well. If we do not rectify such misleading rewards, enforcing alignment between the provided conditions and the inaccurate predictions on newly generated data will consistently com- promise the training process.
To mitigate the adverse effects of inaccurate rewards, we introduce a robust, controllable image gen- eration approach via uncertainty- awaree reward modeling (Ctrl-U). Motivated by observation, we explicitly model the reward uncertainty to facilitate the reward back- back-propagaae propagation. The proposed ancertainty-based method consists of two phases, í.e., uncertainty estimation and uncertainty regu- larization. (1) In the uncertainty estimation phase, we forward the identical input condition twice with different noise timesteps, yielding two similar generation results. We explicitly leverage the reward variance between these two generations as an uncertainty indicator. Notably, we do not in- troduce extra parameters and thus do not impact the inference efficiency of the diffusion model. (2) Following the uncertainty estimation, we adaptively adjust the loss weights of different reward feed- back. Specially, the alignment reward for each pixel are rectified by its corresponding pixel-wise uncertainty. Rewards with lower uncertainty. indicating greater stability, should be given higher weights encourage the model to learn from these reliable signals. Conversely, rewards with higher uncertainty, which are less stable, should be assigned reduced weights to minimize the negative im- pact of potentially inaccurate feedback. Quantitative and qualitative experiments verify the efficacy of the proposed method controllability and image quality across various conditions.
We observe an inherent drawback in enforcing alignment for conditional image generation using a pre-trained reward model, as the reward model often fails to generalize to newly generated data. To address the adverse effects of inaccurate reward feedback conditional image generation, we introduce an uncertainty- reward modeling approach, termed Ctrl-U, which adaptively regularizes the reward learning process.
Through extensive experiments on five benchmarks across three datasets, i.e., ADE20k, COCO-Stuff, and MultiGen-20Me. we validate the effectiveness of our methodology in im- proving controllability and generation quality. Our approach also shows scalability across various conditional scenarios, including segmentation masks, edges, and depth conditions.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORKS
Conditional Generation. Recently, diffusion probabilistic models (Ho et al., 2020 Song et al. 2020a) have become important cornerstone in general image generation. As the quality of gen- erated images improves, a critical challenge remains, i.e., achieving precise control over these gen- erative models to meet the intricate and varied demands of real-world applications adequately. Mo-
tivated by the development of guidance mechanism (Song et al. 2020b] Ho & Salimans] 2022). T21 diffusion models such as GLIDE (Nichol al. 2021). Imagen (Saharia et (0022), DALL-E 2 (Ramesh et al. 2022) excel at modeling fine-grained structures and texture details. To facilitate
the training of diffusion models on limited resources, Latent Diffusion Model (LDM) (Rombach) et al., p022) maps the diffusion process from pixel space to the latent space. Building upon the foundation of LDM framework, Stable Diffusion (SD) exhibits exceptional capabilities in T2I gen- eration, gaining widespread usage within the community due to its open- models. Despite the astonishing capabilities of T2I diffusion models, language, with its sparse and high-level semantic nature, is unsuitable for processing intricate and low-level control images, e.g., depth maps. To achieve conditional control in T2I diffusion models, researchers are exploring the integration of var- ious control signals with text descriptions. ControlNet (Zhang et al. (023a) integrates image-based conditions by incorporating an additional encoder copy into frozen T21 diffusion models via zero convolutions. This additional module links the original UNet layers facilitates the integration of conditional inputs and prevent degradation in performance. Similarly, GLIGEN (Li et al. [203) and T21-Adapter (Mou et al. 2024) propose utilizing independent adapters (or extra modules) syn- chronize internal knowledge within T2I models with external conditions. Furthermore, Cocktail (Hu al., ) integrates multi-modal control information to yield more high-quality outputs. With the rapid development of large language models, some recent works have explored prompt engineering for regulated generation. For instance, ReCo (Zhang et al. 2023b) specifies text descriptions of dif- ferent regions to model the fine-grained structure and aesthetic features. Control-GPT (Zhang et al.) 2023b) applics GPT-4 (Achiam et al., 2023) as the sketch generator to incorporate control signals. However, one drawback is that their generated images usually deviate from input conditions. In this work, we take a closer look at denoised results and explicitly introduce the uncertainty to impose consistency constraints adaptively.
Uncertainty-aware learning. As data-driven technologies rapidly advance, the imperative for model reliability grows. Measuring the "confidence" of predictions has been a long-standing prob- lem. As a result, researchers are progressively focusing on uncertainty as a viable solution. [endall & Gal (2017) categorize uncertainty into two primary types: epistemic uncertainty and aleatoric uncertainty. The former epistemic uncertainty denotes model uncertainty, reflecting variations in model weights even when trained on the same dataset. The representative approach in this direction is Bayesian networks (Jensen& & Nielsen 2007,Neat 2012), which focuses on learning the distribu- tion of weights and estimates uncertainty by assessing the distribution variance. Similarly, Monte Carlo Dropout (Gal & Ghahramani] 2016) is proposed to randomly use varying dropout masks to simulate variational Bayesian approximation effectively. Another line of research on epistemic un- certainty (Raghu et al. 2019; Nandy et al.] [020 Lee & AlRegib] (2220) adapts the original model to measure prediction uncertainty directly. However, incorporating this additional uncertainty es- timation typically compromises both the training efficiency and predictive accuracy of the model. In addition, ensemble methods (Lakshminarayanaa et al. 2017] Malinin et al.) 2019; Wenzel et al.) 2020) combine various deterministic models in the prediction process improve prediction accu- racy, but is constrained by the computational burden associated with operating multiple independent networks and the requisite diversity across ensemble models. Another family of research tackles the inherent uncertainty in data, arising from noise interference or ambiguous annotations. This methodology for addressing uncertainty has been applied across various fields: such as person re- identification (Zhang et al. 2022), image retrieval (Chen et al.) 2022), image classification (Litrico) al. 2023), hallucination detection (Zhang et al., (0024b) and 3D object detection /Zhang et al. 2024a). In particular, nou et al. (2022) effectively simulates uncertainty by utilizing model predic- tion variance to inject noise into the latent space. In a manner akin to (Kendall & Gal 2017). dy- namic uncertainty-basee loss is introduced to enhance training stability. Unlike existing uncertainty- based works, we explicitly introduce noise at varying intensities into the diffusion model to enable the quantification of uncertainty. It is worth noting that we explore the utilization of uncertainty in the untapped conditional image generation task, preserving the original diffusion learning and mitigating the adverse effects of inaccurate feedback from the reward model.
3

---

Published as a conference paper at ICLR 2025
(a) Controllable Generation
(b) Uncertainty Learning
"A bedroom with a large bed.....""
Image Decoder
Text Encoder
Text
ft
22
7.2
UNet
Add Noise
X
80
Image Encoder
Add Noise
P
2
X
Z1
Control
Encoder
Liiffusion
fc
U1
U2
C
Figure 2: A brief overview of our pipeline. Here, we take the segmentation mask conditional generation example. (a) Conditional Generation. Given text, source image To. and the conditional control c, we extract feature 20- ft. fer respectively. Then, we fine-tune the Diffusion model to generate two intermediate features for the image decoder. (b) Uncertainty Learning. Given the two features, we decode the two images, i.e., ai and a.. Then we apply the reward model obtain the two layout predictions è and é2 We leverage the KL--divergence prediction discrepancy between é, 02 as the uncertainty indicator U1, U2 (see E. Based on U1,U2, we then calculate the rectified reward loss between the predicted label é1, 02 and the ground-truth label Eq.
3
METHOD
3.1
UNCERTAINTY ESTIMATION
We provide a brief overview of the pipeline in Fig. To simplify the explanation, here we take the segmentation task an example, if not specified. Given the triplet input, i.e., one source image input Zo. one text prompt, and the conditional control c, the basic encoders extract the visual feature 2g of Zo, textual feature ft and the control feature fe of c. We fix the weight of the off- the-shelf image encoder and the text encoder, while finetuning the control encoder. During the conditional diffusion training, we first add Gaussian noise the feature map 2o as the noisy latent. In particular, we conduct two generation forward with identical condition but different t and 12, and resampled Gaussian noise e. Therefore, the two noisy latents 2 and Z2 can be formulated as:
(I) Following the ControlNet pipeline (Zhang et al. 2023b), we further fuse text condition fe and image condition fc to predict the added noise. After removing the predicted noise, obtain the recovered latent, ie., =/ and = Then, given the latent $/ and =g the pretrained decoder is to reconstruct the input image
21 = V 20 + VI - āt, 22 = / / + VI 0:y5, e~ (0,1
2 and ag. respectively. To align the condition within the generated images, we follow the existing works and apply an off-the-shelf reward model D) to quantify the consistency between the input condition and the corresponding output condition of the generated images. How- ever, as shown in Fig. Tb the reward model usually contains inaccurate feedback, even if the visual layout has already aligned with the condition. consistency between inaccurate rewards and conditions predicted on the newly generated data, the backpropagation of incorrect gradients will significantly compromise the model. To estimate inac- crate rewards, we explicitly leverage two diffusion forwards for the identical input conditions. We compare the reward discrepancy between extracted conditions é 82 from generated images, which can be considered tion, considering the probability output, we quantify uncertainty by calculating the KL-divergence between the extracted conditions of two generations the pixel level as:
consistently apply strong supervision on the
reward indicator the current timestep. For the segmentation map condi-
C1
((ab))
U1 = E
è log
= E
( log
82
D(#)))
(D(2))
U2 E
è2 log
E
D(#3) log
D(
For other non- nonbbabbyyyl conditions, e.g., edge and depth, we adopt the E distance Un = U2 = |è dz aa the uncertainty indicator.
4

---

Published as a conference paper at ICLR 2025
Why not auxiliary network to directly regress uncertainty? Some exist- ing methods (Zheng & Yang 2021; Jain et al.) (024) introduce extra modules to model the uncer- tainty. However, directly regressing uncertainty often leads to overfitting, where the model predicts zero uncertainty for all samples or a large, uniform value. This line of approaches is challenging to optimize effectively. Therefore, instead of directly regressing uncertainty, we perform predictions twice to estimate the prediction variance, which serves as a cognitive uncertainty indicator for the reward model. Additionally, this method has the side benefit of not introducing the extra training parameters. 2). How about choosing the same timestep for the two-time generations? Using the same timestep for the two-time generations is possible, and the samples will still differ due to the inherent randomness in However, this approach limits the diversity of the generated images, which hinders the full utilization of the reward model. our experiments, we observe that using different timesteps for the two generated images helps to learn a better and more robust generation model after rectifying the reward.
Discussion. 1).
3.2 UNCERTAINTY REGULARIZATION
The existing controllability modeling methods (Li al. 2024a) usually adopt consistency loss be- tween the input condition and the extracted condition, which can be regarded as a pixel-wise su- pervision. For example, when using segmentation mask as input condition, L can be defined as per-pixel cross-cntropy loss as:
Ci = clog(()), CS -clog0)
where denotes the imput condition. a è2 represent extracted conditions from generated images. Our objective is to adaptively rectify the inaccurate reward feedback. To achieve this, we adapt the original consistency loss by incorporating our estimated uncertainty:
C
CS
CT
exp (U)) + +
C2
exp (U2) + +
(4)
where denotes the regularization factor. The second term consistently predicting high uncertainty across all samples. The first term diminishes the impact of reward feedback when significant disagreement is evident, yet retains its influence when predictions are consistent. Notably, when the uncertainty value is constant, the back-propagation gradient is identical to the original consistency loss. Finally, to optimize robust conditional image generation, adopt combination of the diffusion loss Ldittusion and the proposed loss CT. Cy. The original diffusion loss at timestep t1 and to can be reformulated as:
is to prevent the model from
Laitlusion = E [1lc tı, fr, fc) - cl12 + [|co(22, 12, fi, fe) -<1
(5)
Therefore, the total loss is as follows:
Lusal
Laillusion + μ (CT + L2),
(6)
During training, we balance the ratio of diffusion training and reward feedback by:
Ho 1 (0
if < fume
(7)
if fihre
where p4 is the consistency weight. If > tthre. we only consider the diffusion optimization as existing works (Zhao al. 2023a). if S bime we incorporate uncertainty-awar reward modeling.
Discussion. 1). Why set timestep threshold 'the for μ Considering that when I, i.e., 4 and l2, is large, 24 approaches the random noise e, leading to more diverse recovered èo (see Fig. [.. In such cases, it becomes difficult to ensure that two outputs with identical conditions remain close, which is necessary for our uncertainty estimation, as it requires visually similar inputs to evaluate the reward model. In practise, we, therefore, empirically set threshold to avoid the large timesteps for the reward modeling. We have conducted the experiment on the choice of fhere (see Section 4.3). 2). What is the advantage of uncertainty regularization in the reward feedback? The advantage of uncertainty regularization in the reward feedback is that it helps to mitigate the adverse effects of imprecise feedback from the reward model. Specifically, by incorporating uncertainty estimation, the proposed method, Ctrl-U, can adaptively adjust the weights the rewards during training. Re- wards with lower uncertainty are given higher loss weights, ensuring that more reliable feedback
5

---

Published as a conference paper at ICLR 2025
has stronger influence the training process. Conversely, rewards with higher uncertainty given reduced weights, allowing for greater variability and preventing the model from being overly influenced by potentially inaccurate feedback. This adaptive rectification of rewards based on their uncertainty improves the consistency and reliability of the training process. As a result, the model becomes more robust to the diverse and sometimes unpredictable nature of newly generated data, leading to better controllability and generation quality.
4
EXPERIMENT
4.1 SETTINGS
Datasets. Our experiments are conducted using three datasets: ADE20K (Zhou et al. 2017 2019). COCOStuff [Caesar et al. 2018) and MultiGen-20M dataset Qin et al. 2023). adhering to the dataset construction principles of Controlnet++ (Li et al. 2024a More specifically, we utilize the ADE20K and COCOStuff for the segmentation task. For Hed and Lineart conditions, we utilize pre- trained models, which is the same ControlNet++,, to generate annotations for the MultiGen-20M dataset. Considering that existing datasets usually include masks for pixels with unknown depth values, adapt the MultiGen20M depth dataset in manner similar to the dataset construction method used by ControlNet (Zhang et al.] (2023a). For datasets without image captions, we use the captions generated by MiniGPT-4 (Zhu et al. (2023). The training and inference are conducted at resolution of 51 5x2x112 for all datasets and methods.
Evaluation and Metrics. To evaluate semantic segmentation and depth map controls, adopt mloU and RMSE as the metrics, respectively. For the Hed edge and Lineart tasks, we calculate SSIM to compare the difference between the extracted control and the ground truth. For Sampling, we employ the UniPC (Zhao et al., 2023b) sampler, implementing 20 denoising steps to generate images using the original text prompts in accordance with ControlNet v1.1 (Zhang al.) 2023a), without incorporating any negative prompts. For comparative methods, we utilized their publicly available code to generate images, ensuring fairness by adhering to their original inference settings.
Implementation Details. In our experiments, we first fine-tune the pre-trained ControlNet model convergence, using Adam as the optimizer with a learning rate of 1e-5, weight decay of le-2, and momentum of 0.9. Then, we use the same optimization settings to perform 10k iterations of uncertainty-aware reward fine-tuning. It worth noting that we employ an one-step efficient reward strategy (Li et al., 2024a) to enhance training efficiency. Following the settings of previous work, we choose slightly weaker models as the reward model and stronger models for evaluation to guarantee fairness in assessment. See also the supplementary material for detailed reward model settings.
our evaluation, we mainly compare with competitive methods, including Control- Net++ (Li et al. 2024a). T2I-Adapter (Mou et al., 2024). ControlNet v1.1 (Zhang et al. 2023b)
Baselines.
GLIGEN Li et al. (2023), Uni-ControlNet (Zhao et al. (023a), and UniControl (Qin et al. 2023 All of the aforementioned methods have been fine-tuned on datasets across various tasks, and Con- trolNet++ introduces reward-based post-training on the foundation of ControlNet. These models perform well in controllable text-to-image generation, providing open-source model weights for re- implementation. For fair comparison, all models are tested under the same image conditions and text prompts. Although these methods utilize the user-friendly SD1.5 for controllable text-to-image generation, recent advancements have led to the adoption of SDXL (Podell et al. 2023) by sev- eral methods. Accordingly, we also present controllability results for ControlNet-S Adapter-SDXL. Since currently there are no officially released models, we deploy the third-party ControlNet-SDDL for the following experiments.
and T2I-
4.2 EXPERIMENTAL RESULTS
Comparison of Controllability. We present the results on five benchmarks in Table 1 Our uncertainty-aware method Ctrl-U significantly outperforms the previous state-of-the-art method ControlNett+ (Li al. 2024a) by 6.53% in ADE20K and 8.65% in MultiGen20M depth, respec- tively. Regarding Hed and Lineart edge, the model with uncertainty regularization has obtained +3.76% and +1.06% increase SSIM. The most significant improvement is observed in COCO- Stuff for segmentation masks, with an increase of 44.42%. This performance enhancement confirms that reducing the adverse effects of imprecise feedback from the reward model can significantly

---

Published as a conference paper at ICLR 2025
Table Controllability comparison under various conditional controls and datasets. 1 denotes higher result is better, while & indicates lower is better. '-" signifies the absence of publicly avail- able model for testing. The best result in each column marked bold and the second is underlined. We generate four groups of png images and report their average result reduce random errors.
Condition
Seg. Mask (mloU f)
Hed Edge (SSIM t)
LineArt Edge (SSIM t) MultiGen-20M,
Depth Map (RMSE ()
T2I Model
(Metric)
Dataset
ADE20K
COCO-Stuff
MultiGen-200
MultiGen-200
ControlNet
SDXL
40.00
T21-Adapter
SDXL
0.6394
39.75
T21-Adapter
SD1.5
12.61
48.40
Gligen
SD144
23.78
0.5634
38.83
Uni Ne-
SD1.5
19.39
0.6910
40.65
UniControl
SD1.5
25.44
0.7969
39.18
ControlNet
SD1.5
32.55
27.46
0.7621
0.7054
35.90
ControlNct++
SDI.5
43.64
34.56
0.8097
0.8399
28.32
Ctrl-U (Ours)
SD1.5
46.49
49,91
0.8401
0.8488
25.86
Table 2: FID (1) comparison under various conditional controls and datasets. 1-" signifies the absence of publicly available model for testing. The best result in each column is marked bold and the second is underlined. We generate four groups of png images and report their average result to reduce random errors.
T21
Seg. Mask
Hed Edge
LineArt Edge
Depth Map MultiGen-20M
Method
Model
ADE20K
COCO-Stuff
MultiGen-20M
MultiGen-200
Gligen
SD1.4
33.02
18.36
T2I-Adapter
SD1.5
39.15
22.52
UniControlNet
SD1.5
39.70
17.08
20.27
UniControl
SD1.5
46.34
15.99
18.66
ControlNet
SD1.5
33.28
21.33
15.41
17.44
17.76
ControlNet++
SD1.5
29.49
19.29
15.01
13.88
16.66
Ctrl-U (Ours)
SD1.5
28.61
15.79
11.59
11.99
15.48
improve the model controllability. We use the same prrammprrrmeter settings those in the Con- trolNet++ experiments, verifying the effectiveness of our uncertainty-avee reward modeling.
Comparison of Image Quality. We employed the Fréchet Inception Distance (FID) (Heusel al. 2017) to measure the distribution distance between generated and real images in Table 2 We could observe that, compared with existing methods, Ctrl-U exhibits superior FID values in various con- ditional generation tasks. Notably, in the EDCO-Stuff for segmentation masks and MultiGen20M for Hed edge, Ctrl-U achieves impressive improvements of 14% and 22.74%, respectively. This significant boost indicates that method not only enhances the controllability but also improves the quality of image generation, underscoring the efficacy of our proposed uncertainty-awar reward modeling in adaptively adjusting the weights of the rewards during training.
Comparison of CLIP Score. To assess the potential impact on text controllability. we reported the CLIP-Score metrics across various conditional generation tasks in Table We discovered that Ctrl-U generally exhibits comparable or superior CLIP-Scone results in most cases, suggesting that our approach not only enhances the controllability of conditional controls but also maintains the original proficiency of text-to-image generation. To ensure more comprehensive comparison, we re-implemented the Cc.P-Score using the official checkpoint for ControlNet++ (Li al.) 2024a). and marked the results with and gray in Table
Qualitative Analysis. We show the visual comparisons between our Ctrl-U and previous state-of- the-art methods across various conditional scenarios in Fig. 3 Given the same text prompts and image-based conditions, we observe that existing methods usually generate images with areas that do not align with the image conditions. Other methods, taking the segmentation mask generation task as an example, produce content on the building that irrelevant to the provided condition, reflecting relatively poor controllability. Similarly, under edge and depth conditions, other methods fail to accurately represent the nuances of various image controls. In contrast, images generated by Curl-U exhibit better consistency with the input conditions.
Human Evaluation. Following ControlNet++ (Li al. 2024a). we choose four types of conditional controls (segmentation masks, Hed edge, Lineart edge, and depth conditions) for human evaluation.

---

Published as a conference paper at ICLR 2025
Table 3: CLIP-score (1) comparison under various conditional controls and datasets. '-' signifies the absence of publicly available model for testing. The best result in each column is marked bold and the second underlined. and gray denote the CLIP-score we re-implemented using the checkpoints provided by Controlnet++. We generate four groups of png images and report their average result to reduce random errors.
T21
Seg. Mask
Hed Edge
LineArt Edge
Depth Map
Method
Model
ADE20K
COCO-Stuff
MuliiGen-20M
MultiGen-ZOM
MultiGen-20MI
Gligen
SDLA4
31.12
31,48
T21-Adapter
SD1.5
30.65
31,46
UniControlNet
SD1.5
30.59
31.94
31.66
UniControl
SD1.5
30,92
32.02
31.68
ControlNet
SD1.5
30.16
31.18
31.46
31.26
32.11
ControlNet++
SD1.5
31.96
13.13
32.05
31.95
32.09
ControlNet++
SD1.5
31.24
30.93
30.49
30.34
30.02
Cur-U (Ours)
SD1.5
31.26
31.23
32.05
31.87
31.72
Table 4: Results of Human Evaluation
Evaluation Category
Ours
ControlNet++
ControlNet
UniControl
UniControlNet
Image-Condition Alignment
72.5%
7.5%
6.2%
7.5%
12.5%
Image Quality
56.2%
10.0%
15.0%
15.0%
13.7%
Image-Text Alignment 50.0%
27.5%
22.5%
7.5%
12.5%
We invite 20 participants to rank each result based on three distinct criteria. According to the average rankings shown in Table users prefer the results produced by our method over those generated by comparative methods.
4.3 ABLATION STUDIES AND FURTHER DISCUSSION
Design of Uncertainty Estimation. We present an ablation study the design of the two-time gen- eration in Table 5a To mitigate the adverse effects of inaccurate rewards, we forward the identical input condition twice with different noise timestep estimate uncertainty. Specifically, we adjust the disparity between the two timesteps, i.e. [ tz] A short interval, such as 1 = t22 where the only randomness stems from resampled noise c, limits the diversity of the generated images. Con- sidering the generated images are too similar, the reward discrepancy is small, and could not serve as uncertainty indicator. Conversely, a long interval indicates a significant gap between the two noisy latents, which, in turn, increase the generation discrepancy. Too large discrepancy also impacts the accurate uncertainty estimation, and thus compromises reward modeling. When (t - t2l 1, the model achieves the optimal FID and relatively strong mloU and CLIP-score.
Impact of Different Timestep Threshold furre- We investigate the impact of varying timestep threshold fime (see Eq. [ in Table 5b] We observe that, the ture increases, the scope of our uncertainty-aware reward rectification broadens, leading to improved mloU. However, while in- creasing the rectification range improve alignment between generated images and input ditions, setting the threshold too high will negatively affect image quality, resulting in higher FID scores. We find that the thhre 400 setting strikes an optimal balance, facilitating uncertainty esti- mation and regularization to adaptively correct the reward Icarning process.
Impact of Regularization Weight We conducted study analyze the impact of varying uncer- tainty regularization weight À in Table 5c When using low value for X, which imposes a minimal penalty for high uncertainty, leads to excessively high uncertainty values across all samples. Con- versely, a high value of X imposes a strong penalty high uncertainty, suppressing the impact of uncertainty during reward fine-tuning. The setting with À achieves an optimal balance. This configuration provides enough model capacity accurately fit reward feedback while preventing overfitting to inaccurate rewards. As a result, the adaptive rectification rewards based on uncer- tainty regularization improves the consistency and reliability of the training process.
Impact of the Consistency Weight Aag We conduct an ablation study on the consistency weight No shown in Table [d) We notice that the model shows insensitivity to the μo change in terms of CLIP-score. Setting Ho to yields the highest mloU but at the cost of FID performance. This is due to the high weight of the consistency loss, which biases model training towards reward learning.
8

---

Published as a conference paper at ICLR 2025
Table Ablation studies on the ADE20K dataset. We report mloU, FID and CLIP-score to evaluate controllability and image quality respectively. (a) We show the impact of the interval between & and 12. We find that a short interval limits the diversity of the generated images, making challenging to estimate uncertainty. In contrast, long interval impacts accurate uncertainty estimation and thus compromises reward modeling. (b) We study the timestep threshold hhre- We find that the fhe 400 setting strikes optimal balance, facilitating uncertainty estimation and regularization to adaptively correct the reward learning process. (c) The impact of the regularization weight A. With À = 1. we obtain best results. This setting provides enough model capacity to accurately fit reward feedback while preventing overfitting to inaccurate rewards. (d) The impact of the consistency weight Ao- Considering the balance between diffusion training and uncertainty-aware reward learning, we set 140 = 0.1 to ensure enhanced controllability while improving generation quality.
(c)
It t2
3
7
9
À
0.05
0.1
mIoU
45.33
46.49
46.72
45.94
55.94
45.11
44.88
mloU
43.06
43.47
46,49
FID
29.01
28.61
29.01
29.81
29.10
28.94
FID
30.15
30.60
28,61
CLIP-score
30,92
31.26
31.01
31.06
30.93
31.09
CLIP-score
30.88
30.80
31.26
(b)
(d)
fur
200
300
400
500
600
700
No
0.05
0.1
mIoU
43.82
43.89
46.49
46.72
47.92
50.11
mloU
42.37
46.49
49.48
FID
29.70
30.34
28.61
31.04
32,98
34,21
FID
30.16
28.61
30.89
CLIP-score
31.13
30.68
31.26
31.07
31.01
30.98
CLIP-score
31.02
31.26
30.84
thereby enhancing controllability but compromising generation quality. When μ4 = 0.1, FID per- formance reaches its optimum, and mloU remains strong, effectively balancing diffusion training and uncertainty-awarr reward learning without causing the uncertainty either vanish or explode.
Efficacy of Uncertainty-aware Reward Modeling. We validate the superiority of our learnable uncertainty-avve reward method by comparing it with the vanilla reward learning. As depicted in Fig. 4 under the identical input conditions and timesteps, although the generated images align with the given conditions, reward learning without uncertainty produces inaccurate predictions. On the contrary, our uncertainty-aware reward fine-tuning generates higher-quality images. Addition- ally, the reward model can accurately extract conditions from these images. This improvement is attributed to the reward uncertainty. which adaptively rectifies inaccurate rewards and improves the reliability of the training process. Specifically, rewards with lower uncertainty are given higher loss weights, enhancing the influence of precise feedback in the training process. Conversely, rewards with higher uncertainty are assigned lower weights, leading to greater variability and ensuring the model is not excessively influenced by potentially inaccurate feedback. Consequently, the model becomes more robust to the diversity of the newly generated images, enhancing both controllability and the quality of generation.
CONCLUSION
In this work, we focus on the challenge of ensuring both high fidelity and semantic alignment in conditional image generation. We highlight a significant limitation in existing methods that use pre- trained reward models for enforcing alignment. Yet these models often provide inaccurate feedback when encountering diverse, newly generated data, which can negatively impact the training process. To alleviate this issue, we propose an uncertainty-awan reward modeling approach, termed Ctrl-U, which incorporates uncertainty estimation and adaptive regularization. This method assigns higher loss weights to rewards with lower uncertainty and reduces the weights for highly uncertain rewards, thereby enhancing the consistency and reliability of the training process. Our extensive experiments across five benchmarks on three datasets, i.e., ADE20K, COCO-Stuff, and MultiGen-200 validate the effectiveness of our methodology in improving controllability and generation quality. Addition- ally, Ctrl-U shows robust scalability, performing well across various conditional scenarios, including segmentation masks, edges, and depth conditions. These results indicate the potential of our method to contribute to advancements in conditional image generation.

---

Published as a conference paper at ICLR 2025
teogna
Segnentattin
Dayph
Med
TRAM CondEon
CoardSates
Linwer
Figure 3: Qualitative comparisons with different conditional controls unseen test images. We observe that our generated image preserves condition alignment with good visual quality. Some models do not have open-source weights for Hed or Lineart condition, and thus skip them.
100
200
300
400
Image
(a) w/o uncertainty
Condition
(b) w/ uncertainty
Figure 4: Ablation study on the uncertainty- reward modeling. Here, we show the recov- ered test image with different denoising timesteps. Since we mitigate the negative impact of noisy rewards, our output maintains consistent semantic conditions with fewer blurred areas.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is supported by the University of Macau Start-up Research Grant SRG2024-00002-FFT and Multi- Year Research Grant MYRG-GGRGG2244007775TTTU
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744 2023.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209- 1218, 2018.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3558-3568 2021.
Yiyang Chen, Zhedong Zheng, Wei Leigang Qu, and Tat-Seng Chua. Composed image retrieval with text feedback via multi-grained uncertainty regularization. arXiv preprint arXiv:2211.07399, 2022.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in neural information processing systems, 1188780-8794, 2021.
Zhaopeng Dou, Zhongdao Wang. Weihua Chen, Yali Li, and Shengjin Wang. Reliability-awaan pre- diction via uncertainty learning for person image retrieval. In European Conference on Computer Vision, pp. 188-605.55 Springer, 2022.
Yarin Gal and Zoubin Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In international conference machine learning, pp. 1050-1059.
PMLR, 2016.
Huan-ang Gao, Mingju Gao, Jiaju Li, Wenyi Li, Rong Zhi, Hao Tang, and Hao Zhao Sep-diff: Photo-realistic semantic image synthesis with spatial-categorical joint prior. arXiv e-prints, pp. arXiv-2403, 2024.
Xin Gu, Ming Li, Libo Zhang, Fan Chen, Longyin Wen, Tiejian Luo, and Sijie Zhu Multi-reward as condition for instruction-bar based image editing. arXiv preprint arXiv:2411.00 047113, 2024.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.
Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho and Tim Salimans.
Classifier-free diffusion guidance.
arXiv preprint
arXiv:2207. 12598, 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Minghui Hu, Jianbin Zheng, Daqing Liu, Chuanxia Zheng, Chaoyue Wang, Dacheng Tao, and Tat- Jen Cham. Cocktail: Mixing multi-modality control for text-conditional image generation. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
Nishant Jain, Karthikeyan Shanmugam, and Pradeep Shenoy. Learning model uncertainty as puriance-miiimize instance weights. In The Tweifth International Conference on Learning Representations, 2024.
Finn Jensen and Thomas Dyhre Nielsen. Bayesian networks and decision graphs, volume 2. Springer, 2007.
11

---

Published as a conference paper at ICLR 2025
Alex Kendall and Yarin Gal. What uncertainties do need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
Jinsol Lee and Ghassan AlRegib. Gradients as a measure of uncertainty in neural networks. In 2020 IEEE International Conference on Image Processing (ICIP), pp. 2416-2420. IEEE, 2020.
Ming Li, Taojiamnan Yang. Huafeng Kuang, Jie Wu, Zhaoning Wang. Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback: Project page: liming-ai. github. io/controlnet, Baii__puu_______ In European Conference on Computer Vision, pp. 129-147. Springer, 2024a.
Wenyi Li, Haoran Xu, Guiyu Zhang, Huan-ang Gao, Mingju Gao, Mengyu Wang, and Hao Zhao. Fairdiff: Fair segmentation with point-image diffusion. In International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 617-628. Springer, 2024b.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to- text-to-image generation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 22511-225211 2023.
Mattia Litrico, Alessio Del Bue, and Pietro Morerio. Guiding pseudo-labels with uncertainty estima- tion for source-free unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7640-7650, 2023.
Andrey Malinin, Bruno Mlodozeniec, and Mark Gales. Ensemble distribution distillation. arXiv preprint arXiv:1905.0000 2019.
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI conference artificial intelligence, volume 38, pp. 4296- 4304, 2024.
Jay Nandy, Wynne Hsu, and Mong Li Lee. Towards maximizing the representation gap between in-domain & out-of-distrbbtionn examples. Advances in neural information processing systems, 33:9239-9250, 2020.
Radford M Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arxiv preprint 23x.::2307.00922 2023.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang, Juan Car- los Niebles, Caiming Xiong, Silvio Savarese, et al. Unicontrol: A unified diffusion model for controllable visual generation in the wild. arXiv preprint 3rXiv:2305.1114: 2023.
Maithra Raghu, Katy Blumer, Rory Sayres, Ziad Obermeyer, Bobby Kleinberg, Sendhil Mul- lainathan, and Jon Kleinberg. Direct uncertainty prediction for medical second opinions. In International Conference on Machine Learning, pp. 5281-5290. PMLR, 2019.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alee Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In International conference on machine learning, pp. 8821-883 Pmlr, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2266:22244. 1(2):3, 2022.
12

---

Published as a conference paper at ICLR 2025
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494, 2022.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open dataset of clip-fittered 400 million image-text pairs. arXiv preprint arXiv:2111.0211 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 35:25278-25294, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song. Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 13456, 2020b.
Florian Wenzel, Jasper Snoek, Dustin Tran, and Rodolphe Jenatton. Hyperparameter ensembles for robustmess and uncertainty quantification. Advances in Neural Information Processing Systems, 33:6514-6527, 2020.
Zhiyuan Xu, Yinhe Chen, Huan-ang Gao, Weiyan Zhao, Guiyu Zhang, and Hao Zhao. Diffusion- based visual anagram multi-task learning. arXiv preprint arXiv:2412.02693. 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for -m-age- diffusion models. arXiv preprint arXiv:2238..08 067221, 2023.
Lymin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEE/CVF international conference computer vision, pp. 3836-3847, 2023a.
Ruiyang Zhang, Hu Zhang, Hang Yu, and Zhedong Zheng. Harnessing ancertainty-awar bounding boxcs for unsupervised 3d object detection. arXiv preprint arXiv:2408.000 00619, 2024a.
Ruiyang Zhang, Hu Zhang, and Zhedong Zheng. Vl-uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation. arXiv preprint (rXiv:24/1.1191) 2024b.
Tianjun Zhang, Yi Zhang, Vibhav Vineet, Neel Joshi, and Xin Wang. Controllable text-to-image generation with gpt-4. arXiv preprint arXiv:2305. 18505, 2023b.
Xinyu Zhang, Dongdong Li, Zhigang Wang. Jian Wang, Errui Ding, Javen Qinfeng Shi. Zhaoxiang Zhang, and Jingdong Wang. Implicit sample extension for unsupervised person re-identification. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2022.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan- Yee Wong. Uni-controlnet: All-in-one control to text-to-image diffusion models. Advances in Neural Information Processing Systems, 36: 1e111122111111 2023a.
Wenliang Zhao, Lujia Bai, Yongming Rao, Jie Zhou, and Jiwen Lu. Unipe: A unified predictor- corrector framework for fast sampling of diffusion models. Advances in Neural Information Processing Systems, (6:49842-49886 2023b.
Zhedong Zheng and Yi Yang. Rectifying pseudo label learning via uncertainty estimation for domain adaptive semantic segmentation. International Journal of Computer Vision, 129(4): 106-:1100611111000 120, 2021.
13

---

Published as a conference paper at ICLR 2025
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 333-641,201 2017.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision, 127:302-321, 2019.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
14

---

Published as a conference paper at ICLR 2025
5.1 MORE IMPLEMENTATION DETAILS.
Dataset Details. We the ADE20K dataset for segmentation masks, which includes 20,210 images in the training set and 2,000 images in the validation set. The dataset contains 150 types of segmentation annotations, covering most objects found in daily life scenes. Since ADE20K (Zhou let al.] 2017 2019) lacks text prompts, we follow ControlNet++ (Li et al.) (2024) and use MiniGIT- (Zhu al. (023) to generate image captions by instructing: "Please briefly describe this image in one sentence". Similarly, COCO-Stuff provides segmentation annotations, with 118,287 images in the training set and 5,000 in the validation set.
challenging to find datasets with accurate annotations for Hed and LineArt edge. To tackle this issue, we adhere to the dataset construction principles of ControlNet (Zhang et al.) 2023a) and train our model using the MultiGen20M (Qin) et al., 2023) dataset, which includes annotations generated by the model. The images and their corresponding captions are displayed in Fig. 5
GOWD
group cows grazing lush green hillside.
wooden billiards table with green felt surface and ornate wooden trim, surrounded by stone wall with a chandelier hanging from it.
yellow sports car is driving on track in front of a large crowd of people watching.
group of upright pianos arranged in row in a room.
Figure 5: Text prompts generated by MiniGPT-4
Reward Model Details. We provide details of the models and evaluation metrics in Table Consid- cring that the Hed and Lincart edge extraction models are neural networks without non-differentiable operations, we achieve differentiability by modifying the forward code. We have improved the reward fine-tuning process by the proposed uncertainty-aware reward modeling, resulting in sig- nificant enhancement in controllability and generation quality across four control conditions. Our research will expand to encompass additional control scenarios, such as human pose and scribbles.
Table 6: Details of the reward model, evaluation model, and training loss under different conditional controls. ControlNet* indicates that we utilize the same model for condition extraction employed in ControlNet (Zhang et al. 2023a).
Seg. Mask UperNet-R50
Depth Edge
Hed Edge
LineArt Edge ControlNet
Reward Model (RM)
DPT-Hy brid
ControlNet
RM Performance
ADE20K (mloU): 42.05
NYU (AbsRel): 8.69
Evaluation Model (EM)
Fask2Former
DPT--Lage
CentrolNet
Cen
EM Performance
ADE20K (mloU): 56.01
(AbsRel)) 8.32
Consistency Loss
CrossEntropy Loss
MSE Loss
MSE Loss
MSE Loss
5.2 EXPLANATION OF UNCERTAINTY VISUALIZATION
We present additional qualitative results in Fig. As shown in Fig. 6b] we could observe that the high variance usually exists in the area with incorrect reward. The proposed rectified loss assigns different thresholds to different areas. For example, for the location with coherent rewards between two generations, the variance regularization drives the model trust reward feedback. For the area with ambiguous rewards, the variance regularization drives the model to neglect reward feedback. Our uncertainty-and aware framework adaptively adjusts the loss weights of different reward feedback, facilitating more effective reward optimization.
15

---

Published as a conference paper at ICLR 2025
Real Images
Generated Images
Real Condition
Reward Condition
Uncertainty
cushion
sky
mirror
bed
Fildd
painting
lamp
wall
plant
pillow
unlabel
televiston
(a)
(b)
Figure 6: Illustration of the estimated uncertainty inaccurate reward. (a) Detailed category illus- tration. (b) The areas in the reward condition, where generated images align with the real condition, but receive incorrect rewards, obtain large value of the prediction variance. Meanwhile, we could observe that the high-varinn varianer area has considerable overlaps with the wrong segmentation feedback from the reward model.
5.3 MORE VISUALIZATION
More visualization results across different conditional controls for our image generation on the val- idation set are shown Figures 8 TO, We observe that our uncertainty-an -wwaree framework is capable of generating diverse images that align well with the given inputs across semantic masks, edge, and depth conditions. Attributed to our proposed uncertainty estimation and regularization, the adaptive rectification of inaccurate rewards enhances the diversity of the generated data and its alignment with the given conditions.
16

---

Published as a conference paper at ICLR 2025
Image & Condition
Generated Images & Extracted Conditions
Figure 7: More visualization results of ours on unseen test images (Segmentation Mask)
17

---

Published as a conference paper at ICLR 2025
Image & Condition
Generated Images & Extracted Conditions
Figure More visualization results of ours on unseen test images (Depth Map)
18

---

Published as a conference paper at ICLR 2025
Image& & Condition
Generated Images & Extracted Conditions
VEL
VEL
VEL
VL
VEL
LL
EL
EL
U
D
PO
FOU
FOX
FOJ
EMHYSS
POX
EMNYSS
FO
EMIN VS
FO
EMNYYS
F-
EMIRYS
FCX
RCX
RO
ROX
RCX
CO
PC
RCC
P
RC
Figure 9: More visualization results of ours unseen test images (Hed Edge)
19

---

Published as a conference paper at ICLR 2025
Image & Condition
Generated Images & Extracted Conditions
Figure 10: More visualization results of ours on unseen test images (LineArt Edge)
20


Reviews:
[{"review_id": "3R7phbPTJu", "reviewer": "Reviewer_QrcQ", "summary": "The authors investigate the reward model used in conditional generation diffusion models and find that it tends to output wrong conditions (e.g., segmentation masks) on the de-noised images. They propose leveraging the output uncertainty on two different generated results from different denoising timestamps. They reduce the loss weight of pixels with high uncertainty while increasing ones with low uncertainty.", "questions": "1. The annotation in Fig. 2 and Eq. 2 needs to be clarified. Specifically, is the KL divergence computed between the input control (c) and c1, c2 separately? Or is it computed only between c1 and c2? While Fig. 2 denotes the former, Eq. 2 denotes the latter.\n2. What is the range of t1 and t2? Also, do the optimal t1 and t2 vary on different datasets/tasks?\n3. Visualization of inaccurate prediction from the reward model and the corresponding uncertainty prediction. Are they aligned? In other words, does the uncertainty reduce the weight loss of positions where the predicted condition matches the input but somehow has a significant loss (the reward model predicts the wrong segmentation mask)?\n4. Line 360, can the authors elaborate more on \"After communicating with the authors, we re-implemented the CLIP-Score for ControlNet++ (Li et al., 2024b) and marked it with ∗ in Table 3\"?\n5. Does the reward model impact the performance if a different one is used? Also, does the consistency prediction generalize well to different distributions? (e.g., different segmentation datasets)", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method is intuitive and easy to incorporate.\n2. The quantitative results outperform the baselines by a large margin (although some metrics used seem to be questionable, e.g., mIoU from another segmentation model; see Weaknesses)", "weaknesses": "1. If I understand correctly, the authors leverage another more powerful segmentation model to evaluate mIoU. However, this seems strange to me. For instance, in Fig. 3, while the authors circled out the additional green masks by ControlNet++, I think this shouldn't be regarded as an error. Because the green boxes seem to be windows, and the powerful segmentation model predicts these masks, it could be that ControlNet++ did a great job generating the windows based on the concept of the pinkish building mask.\n2. Please see Questions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oCZ3h7kHeq", "reviewer": "Reviewer_gccp", "summary": "This paper addresses the issue of conditional image generation. The authors identify a significant flaw in the perceptual loss used for conditional image generation, where it may provide incorrect supervisory signals for some newly generated images, thereby affecting the results. To address this, the authors propose a simple yet effective approach to assess the reliability of supervisory loss by estimating the uncertainty in the generated images. Specifically,  during the training process they measure the variability between two different images generated under the same condition as an indicator of the uncertainty in the supervisory loss, suggesting that a high variability implies an unreliable signal. This method automatically corrects the supervisory signals, thereby enhancing the quality of the generated models. The conclusions are supported by detailed experimental evidence.", "questions": "see Weaknesses", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is straightforward and effective, and the paper is well-organized and easy to understand.", "weaknesses": "1. There is a lack of important comparisons, particularly the absence of direct comparison with methods that use auxiliary networks to measure uncertainty (e.g.  Learning model uncertainty as\nvariance-minimizing instance weights(ICLR 24), It can be used to conduct an ablation experiment to compare who is the better way to measure uncertainty for Conditional Image Generation), discussed on line 214, which is a significant oversight.\n\n2. The motivation section in the introduction is somewhat abrupt and some statements, such as those mentioned on line 74 regarding \"the denoised image at t = 0 aligns well with the input condition,\" are not intuitive and require clearer explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SS1npDMnGI", "reviewer": "Reviewer_4HGt", "summary": "Targeting  conditional image generation, the authors proposed an uncertainty-aware reward modeling method, named Ctrl-U. This method includes uncertainty estimation and uncertainty-aware regularization, aimed at mitigating the negative effects of inaccurate feedback from the reward model. The authors use the prediction variance as an uncertainty indicator, and based on this, they adaptively adjust the reward during model training. Rewards with lower uncertainty are given higher loss weights, while those with higher uncertainty are given reduced weights, allowing for more variability.", "questions": "1. As specified in L215-217, authors stated that directly regressing uncertainty can incur indistinguishable predictions over samples. However, In L239-240, the proposed methods also require additional terms to prevent indistinguishable predictions. Considering that the proposed method can introduce double training time, what’s the major advantage of the proposed method?\n2. I'm confused by the value of t in equ. 7. Specifically, how does t in equ.7 relate to t1 and t2 introduced in equ.5?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper targets the uncertainty problem existing in training with reward models, which is critical and meaningful.\n\n2. Despite increasing training time for each training step, the proposed method avoids incorporating additional modules to assess reward uncertainty.", "weaknesses": "1. Lack of novelty. The major contribution of this work is the methodology that generates two images, calculates their uncertainty, and reweights the reward loss. Such a solution seems to be trivial, double the time of each training step, and bring neglective quantitative improvement. \nMaybe the author can report the computational cost required before and after using the proposed method. \nIn addition, why do not the authors collect a dataset composed of paired generated images and their uncertainty scores and then train a simple predictor to predict uncertainty for each image to remove the necessity to generate two images in one step? In my opinion, such an uncertainty predictor can also be adaptive to different generative models and can significantly improve the training efficiency compared to the proposed method. Since the predictor can be more lightweight than generative models and can be trained on image latents.\n\n2. Unfair comparison. The method proposed in this paper is mainly used for post-training. In other words, authors need to pretrain a conditional model such as ControlNet before employing the proposed training losses (Line 292-295). However, during quantitative comparison, authors compare their methods with pretrained models. Such a comparison seems to be inappropriate. It may be better to report the performance improvement when applying the proposed method to different base models, showing the generalization ability of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OxRHMsSMOT", "reviewer": "Reviewer_xjaQ", "summary": "This paper made improvements to the controllability of diffusion-based conditional generative models by introducing the concept of uncertainty.\nThe authors designed loss functions for probabilistic and non-probabilistic conditions by assessing the differences between images in two inference processes.\nThe authors claim that they have achieved SOTA performance on both objective and subjective metrics.", "questions": "Please refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "I appreciate the motivation behind this paper, and I believe the method presented is quite sound.", "weaknesses": "1. We found that the loss function proposed in this paper can lead to artifacts, such as the linerart in Fig 3. The proposed method can achieve alignment between the outputs and the conditions, but the authors do not discuss how to maintain the visual plausibility of the generated results while ensuring this alignment.\n\n2. In Tab.5a, there seems to be no clear evidence of what the optimal value of delta t is. We hope the authors can further discuss the choice of delta t. I am curious about why a relatively small delta t (1% of total 1000 timesteps) can encourage the diversity of generated samples.\n\n3. From Fig.2, it can be seen that the gradient obtained from 2 different timesteps is back-propagated, and the parameters of the UNet will be also optimized. We hope the authors can provide some discussion on the training computational cost. Have the authors tried to only optimize the parameters of the control net?\n\n4. This point does not affect our rating, but we want to know why the ControlNet method on SDXL in Tab.1 performs significantly worse than on SD1.5.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)