Follow-Your-Preference: Towards Preference-Aligned Image Inpainting
Authors: Yutao Shen
Junkun Yuan
Toru Aonishi
Hideki Nakayama
Jack Ma
Venue: ICLR 2026 Poster (2026)
Keywords: Image Inpainting
Preference Alignment
Diffusion Models
Flow-based Models
Primary area: generative models
TL;DR: This paper studies image inpainting with preference alignment, revealing insights into its effectiveness, scalability, and challenges.

Abstract: This paper investigates image inpainting with preference alignment. Instead of introducing a novel method, we go back to basics and revisit fundamental problems in achieving such alignment. We leverage the prominent direct preference optimization approach for alignment training and employ public reward models to construct preference training datasets. Experiments are conducted across nine reward models, two benchmarks, and two baseline models with varying structures and generative algorithms. Our key findings are as follows: (1) Most reward models deliver valid reward scores for constructing preference data, even if some of them are not reliable evaluators. (2) Preference data demonstrates robust trends in both candidate scaling and sample scaling across models and benchmarks. (3) Observable biases in reward models, particularly in brightness, composition, and color scheme, render them susceptible to cause reward hacking. (4) A simple ensemble of these models yields robust and generalizable results by mitigating such biases. Built upon these observations, our alignment models significantly outperform prior models across standard metrics, GPT-4 assessments, and human evaluations, without any changes to model structures or the use of new datasets. We hope our work can set a simple yet solid baseline, pushing this promising frontier. Our code is available at: https://github.com/shenytzzz/Follow-Your-Preference.

Published as a conference paper at ICLR 2026
BOLLLW-YOUR-PPEFERENCE TOWARDS PREFERENCE-ALIGNED) IMAGE INPAINTING
Yutao Shen'* Junkun Yuan Toru Aonishi Hideki Nakayama' Yue Ma The University of Tokyo *Zhejiang University TTsnnguua University
A paper boat drifts lake.
SpongeBob with cazy smaile.
A cat clinging man's back.
Shiba Inu's face the toast.
origami unicorn is grazing.
Girl in burnng scarf.
Figure I: Results of our model. The inpainting outputs generated by our model are visually coher- ent, semantically aligned with text prompts, and consistent with human aesthetic preferences.
ABSTRACT
This paper investigates image inpainting with preference alignment. Instead introducing novel method, go back to basics and revisit fundamental prob- lems achieving such alignment. We leverage the prominent direct preference optimization approach for aligmment training and employ public reward models to construct preference training datasets. Experiments are conducted across nine reward models, two benchmarks, and two baseline models with varying structures and generative algorithms. Our key findings are as follows: (1) Most reward mod- els deliver valid reward scores for constructing preference data, even if some of them are not reliable evaluators. (2) Preference data demonstrates robust trends in both candidate scaling and sample scaling across models and benchmarks. (3) Observable biases in reward models, particularly in brightness, composition, and color scheme, render them susceptible to cause reward hacking. (4) A simple en- semble of these models yields robust and generalizable results by mitigating such biases. Built upon these observations, our alignment models significantly outper- form prior models across standard metrics, GPT-4 assessments, and human eval- uations, without any changes to model structures or the use of new datasets. We simple yet solid baseline, pushing this promising frontier. Your--Yeternee Preference
hope work
Our code is available at: https:/githhb.commseeeeyyz/////i
INTRODUCTION
Image inpainting (Bertalmio et al. 2000) aims to user-specified regions of an image visually coherent and realistic manner. It holds great value in applications such photo restora- tion (Liang et al.) 2021). content creation (Zhuang et al.] (204)), and image editing (Zhang et al.] 2023a). With the unprecedented success of diffusion models (Ho et al.) 2020) and flow-based mod-
els (Lipman et al. 2022), image inpainting has become a prominent research focus in recent years.
Equal contribution.
Corresponding author.

---

Published as a conference paper at ICLR 2026
Aligning human preferences in visual generation has emerged as focal point of research ef- forts (Wallace et al., 2024] Xue et al. 2025 Black et al. [023] Fan et al. 2023). While great progress
has been made in image inpainting (Wu et al., 2025 Ju et al. 2024 Zhuang et al. 2024 Manukyan
et al. 2023), research on aligning inpainting results with human preferences remains limited.
This paper explores image inpainting with preference alignment. Given the limited work on this task, our goal is not to present a novel method, but rather to rethink foundational questions. To this end, we adopt the prominent Direct Preference Optimization (DPO) (Rafailov al.) 2023
Wallace et al. 2024 Liu et al. 2025) to conduct studies due to its simplicity and efficiency, Instead of relying on costly and non scalabee human annotations, we employ public, off- tff-the-shelff reward models for constructing preference training datasets. Our study focuses on several key questions:
(1) How effective are these reward models in scoring and constructing high-quality preference data?
(2) How scalable is preference data with respect to the candidate quantity and the sample quantity?
(3) How does reward hacking (Pan et al.) 2022) oceur, and what method can be used to mitigate it?
To answer these questions, we conduct experiments across nine widely used reward models (e.g.. HPSv2 (Wu et al. 2023), PickScore (Kirstain et al., (023)), two representative evaluation bench- marks (BrushBench (Ju et al., 2024), EditBench (Wang et al. 2023)), and two buseline inpainting models (BrushNet (Ju et al.) 2024). FLUX.1 Fill (BlackForestLas,, 2024b)) with diverse architec- tures (U-Net (Ronneberger al. (213)) Transformer-basee
(aaswani et al. 2017)) and generative algorithms (diffusion (Ho et al. (020), flow-hased (Lipman et al. (022)). Our findings reveal that:
(1) Most reward models provide valid reward signals for constructing effective preference training data, despite some being unreliable as evaluators and exhibiting shared biases. (2) Preference data shows consistent trends in both candidate scaling and sample scaling across baseline models and benchmarks. However, biases in certain reward models (e.g.. HPSv2) can lead to reward hacking, undermining scaling effectiveness. (3) We identify explicit biases in reward models particulaary in brightness, composition, and color scheme- echeme-making making them vulnerable to reward hacking. For ex- ample, HPSv2 tends to favor images with bright lighting, complex composition with rich details, and vivid colors; PickScore shows the opposite tendency. We also find that BrushNet generates vi- brant images, making PickScore suitable for it; while FLUX.1 I Fill produces plain images, aligning well with the property of HPSv2. (4) A simple ensemble of these reward models exhibits strong versatility across models, producing balanced and aesthetically pleasing inpainting results.
Building on these observations, we propose simple yet effective models via reward ensemble. With- out modifying model architectures or introducing new datasets, our models substantially outperform oftte-of-the-ar models across standard metries, GPT-4 assessments, and human evaluations. Visu- alizations show that our models generate more coherent and visually appealing results than competi- tors. We hope our work can establish simple yet strong baseline advance this research field.
2 RELATED WORK
Image inpainting (Bertalmio et al. 2000) is the process of filling in missing regions of an image.
Great progress (Manukyan et 20033 Wang et al.] 2025c Yuan et al.| [el]||203pp b) has been achieved
by employing diffusion (Ho al. 2020) and flow- (Lipman et al. 2022 Esser et al. 2024
Chen et al. 2024 Ma et 20244)) models, which iteratively transform random noise into structured
data. Some pioneering efforts make attempts (Avrahami et al. [023 Rombach et al. 2022 Zhang ct al. 2023b) to incorporate these generative models into image inpainting, and others (Ju et al.,
2024 Zhuang et 2024 Yuan et al., 2023a) later advance and refine it. For example, BrushNer (Ju
et al.
2024) is a branch model that decouples masked image feature extraction from generation.
FLUX (BlackForestLabs 2024a) is rectified flow transformer that generates impressive images.
Its variant, FL.UX.1 Fill ( (BlackFooesst..abs) 2024b), is tailored specifically for image inpainting.
Image generation with preference alignment an emerging field that seeks to align synthesized images with human preferences (Hu et al.) 2026) Yuan et al.) 2022). Some previous works (Black) ct al. 2023; Fan et al. 2023) employ reinforcement learning (RL) (Sutton et al.) 1998) to fine-tune the models using feedback derived from reward models. Recent works have begun exploring meth- ods based on Direct Preference Optimization (DPO) (Rafailov et al. 2023). without the RL process. For instance, Diffusion- Riffusion-DPOO (Wallace et al. 2024) is the first attempt to align the generation with human preferences by optimizing diffusion based pairwise feedback. PrefPaint (Liu et al. (024b) refines image inpainting results by using a reward model trained human-annotater data.
2

---

Published as a conference paper at ICLR 2026
3 PRELIMINARIES
3.1 DIFFUSION MODELS AND FLOW-BASED MODELS
Diffusion models (Sohl-Dickstein et al. 2015 Song al. 2020), such DDPM (Ho et al. 2020
are a class of generative models that learn to reverse a gradual noise corruption process. DDPM assumes a forward process that gradually applies noise to real data. At timestep i, the real data To is destroyed to 22: (=))) = Wa V (1 0z T), where ōg is noise scheduling hyper- parameters. It has reparameterizatioo formula: 24 v VāgTo -I where noise NN I). DDPM learns reverse process using a denoising model eg with parameters 0, inverting the forward process: po (Tt- [xa) = =N(po(rx); ((()))) The denoising model Eg can be trained by minimizing:
LDDPM Ex, [[|c colvorto vi-
(1)
Flow-based models (Esser al., 2024) are generative models that learn to model data distributions using invertible transformations. Recently, Flow Matching (Lipman et al. (022) has emerged as
prominent approach for visual generation (Ma et al. 2024a] Gao et al. 2025a). It usually learns a continuous-time flow that transforms simple prior distribution into the data distribution by solving an ODE. The process, with an optimal-transpor path, employs a linear interpolation scheme: 2, = (1 t] + te. A denoising model vo is trained to predict the velocity field by minimizing:
LFlow = =o,z. [llvo((] ((1 - t) + tc, t) - (c - =
(2)
U-Net (Ronneberger et al., 2015) is used as the basic model structure by many previous denois-
ing models (Ho et al. 2020 Song et al. 2020). U-Net is a symmetric encoder-decoder architecture
that captures multi-scale features through progressive downsampling and upsampling. Transform-
ers (Vaswani et al. 2017), , employed in recent works (Gao et al. 2025b a Wan et al.) 2025 Kong
ct al. 20244), process all data elements in parallel using attention, facilitating training scalibility.
To improve the reliability and generalization of conclusions drawn in our studies, we conduct investi- gations using two different baseline models BrushNet (Ju et al. 2024) and FLUX.1 Fill (Black- LabsstLabs] 2024b). introduced in (sccionn 2 BrushNet is built on a U-Net-like architecture and trained with the DDPM loss, while FLUX leverages transformers and learns via Flow Matching.
3.2 PREFERENCE ALIGNMENT
The standard pipeline for training large-scale models typically involves pre-training, supervised fine- tuning, and preference alignment. Preference alignment refines model outputs to better match human values. Reinforcement Learning from Human Feedback (RLHF) (Bai et al. (022) is a popular align- ment approach. It utilizes human preferences on model outputs to train a separate reward model, which subsequently provides rewards for alignment via reinforcement learning algorithms such as PPO (Schulman et al., 20.7) and GRPO (Guo et al. (025). In comparison, Direct Preference Op- timization (DPO) (Rafailov et al. 2023), which performs direct supervised learning, offers higher training efficiency. It constructs a preference dataset that comprises preferred samples and dispre- ferred samples. DPO learns human preferences implicitly contained within the data by maximizing:
[z) Tret (y)()) 3 log
To(y/1>) Tyer [z) )
Enyy of [log ar (8 log
(3)
where a is the sigmoid function; ia and Tref are the policy and the reference policy respectively. In image generation, given a text prompt x, ydd and y denote the generated preferred image and dis- preferred image, respectively. The dyper-parameter 3 controls the strength of regularization: large
value of 3 increases regularization pressure, dampening preference learning. In visual generation, Equation 3 can be derived to yield a simplified loss (Wallace al. 2024 Liu al. 2025)
(4) where Cy and C. denote the loss (Equation Equation 2) applied the policy on preferred samples and dispreferred samples, respectively; similarly, Lhe and Cies denote the loss applied to the reference policy. This loss function aligns the distribution of generated samples with the preferred data distribution and diverges from the dispreferred distribution. Due to the simplicity, efficiency, and stability of DPO, this paper will explore preference aligument for image inpainting by optimizing Equation 4 on different preference datasets that are constructed for investigation.
LDPO = -E]log (- - C
(C), )l),
3

---

Published as a conference paper at ICLR 2026
Table 1: Comparisons of reward models using BrushNet on BrushBench and EditBench.
CLIPScore
Aesthetic
ImageR
PhckSSore
HPSY2
VQASccoee
UnitiedR
Perception
HPS15
GP14
reward mode!
Brash
Fdle
Brash
Edi
Brash
Edit
Brash
Fdi
Brash
Fdie
Brush
Edi
Brash
Fdir.
Brash
Fdit.
Brash
Edir.
Brush
Edi
Bosoline
26.415
27.357
6-425
5.592
12:317
-1.296
22.133
20,616 21,009
23,056
9.050
6.770
3:30
2,100
25,290
26
26,410
5,399
0,403
79..91
57.066
Random
26.441
27.633
6.424
5,392
12,055
-1.136
22.
130 20.642
27,501
23,067
9.050
6,917
3.302
2,110
26,292
26.422
5.738
0.425
79.177
56.753
CLIPSoove
26.461
27.710
6.430
5,395
12,752
00770
22,
146116
20,600
27.582
23.316
9.062
6,594
3.34
2.124
26,324
26.548
0.640
79.661
$7.539
Aeshede
26.465
27.355
6.477
5,520
12,990
00577
22.221
20,689
27.594
23.166
9.065
6,525
3.343
2,140
26.295
26.342
5.922
0.597
81.003
58.603
ImageR
26.471
27.539
6.462
5,434
12,891
0.577
22.155
20.701
27.672
23.467
9:036
6.761
3.334
2.344
25,305
26.501
5.113
0.782
50.341
57.806
PxcSsoree
26.397
27.199
6.454
5:454
12,893
1,364
22.254
20.732
27.322
22,933
9.002
6,573
3.353
2.178
25.273
26.427
5.750
0.469
82.726
59.550
HPS-2
26.481
27.677
6.476
5.495
12,800
0.128
22.137
20.725
27,818
23.742
9J173
6,818
3.832
0.155
26.361
26.678
1.061
70.914
$7.658
VQASaue
26.442
27.524
6.429
5:407
12,658
-0.800
22,
126 20.667
27.527
23.234
0.038
6.579
3,311
2,139
26,
26.826
26:406
5.723
0.555
78.877
56.975
UnidedR
26.428
27.505
6.453
5,402
12,764
-0.812 22,157 20.675 27.562
23,204
9.061
6,857
3.329
2.155
26.520
26
0.540
90.333
$7.185
Persppiinn
26,448
27.484
6.428
5,395
12,789
-0.073 22. 177 20.660 27,519
23.111
9.060
6.59L
3.327
2,160
26,310
26.515
5.764
0.433
80.277
57.254
HPS
26.461
27.547
6.464
5/448
12,922
10.14 22. 22.176 20.713 27.758 23,491
9.063
6,850
3.344
2.158
26.387
26
53.555
0.563
90.623
57,485
Enscuble
26.535
27.395
6.485
5.497
13,037
-0,352 22 230 20.735 20.792 23.522
9:053
6.592
3,365
2.176
26.538
26
6,074
1,15
$2.172
58.986
Bold values denote besuls Underlined values Benoti socondd Best results. Values blue danote results below bascline
chunce.
3.3 REWARD MODELS
Reward models play an important role in preference alignment: they provide real-time rewards in RLHF (Schulman et al., 2017), and offer scores for constructing offline preference data in DPO (Wang et al. 2025b Lee et al. (025). However, prior works (Wang et al. 2025a Xue et al. 2025 directly employ the-the-shelf reward models for visual preference alignment without suffi- cient evaluations. In this paper, we evaluate the effectiveness of these reward models in constructing preference data via extensive studies. Specifically, we examine the following public reward mod- els: (1) CLIPScore (Hessel al., [222]] measures semantic alignment between images and text prompts by calculating cosine similarities of their CLIP embeddings (Radford et al. 2021 (2)
Aesthetic (Schuhmann et al. 2022) predicts human aesthetic preferences on top of the CLIP em- beddings. (3) Lewae (ImageR) (Xu et al. 2023) is trained by fine-tuning BLIP Li et al.) 2022) on 137K preference samples. (4) PickScore (Kirstain et al.] (202) is a Cased image scoring model, trained on over 500K synthesized image samples with users' preference choices. (5) HPSv2 (Wu et al., 2023) is also a CLIPP-ased model that evaluates both image quality and text-image alignment by learning from 798K human preferences on 433K sample pairs. (6) VQAS- core (Lin et al. 2024) provides a semantic alignment score by computing the probability of a VQA model answering "yes" to each question: "Does this figure show [text]?". (7) Reward (Uni- fiedR) (Wang et al. 2025b) is a unified model that assesses both visual generation and understand- ing. (8) Perception Encoder (Perception) (Bolya et al. 2025) is trained by contrastive visual- language pre-training, producing semantically aligned multimodal embeddings.
(9) HPSv3 (Mal al. 2025b) is trained on 1.5M annotated sample pairs using Qwen2VL.-7B (Wang al. 2024).
To assess their efficacy, these models are employed to assign reward scores to candidate samples which are generated by the baseline models with different random seeds. The resulting highest- and lowest-scoring samples from each text prompt are subsequently utilized as the preferred and dis- preferred samples for DPO training. Based on the evaluation of training results, the top oop-perforning ones are designated as the most effective reward models to provide accurate rewards, and vice versa.
4 HOW EFFECTIVE ARE REWARD MODELS?
The ability of reward models to accurately predict human preferences is critical to the performance of preference alignment algorithms. To evaluate this capability, we apply DPO on the preference data constructed by the reward models and evaluate the model performance after training. Specifically, based on the popular dataset of BrushData (Ju et al. 2024), we generate 16 candidate inpainting results with varied random seeds for each prompt and the corresponding masked image. The can- didates are scored by the reward models, and the highest-scoring (preferred) and lowest-scoring (dispreferred) samples form preference pairs for DPO training. Following (Ma et al. 2025a Wang et al. 2025b), the reward models are employed to serve two purposes: (1) providing scores to con- struct training data, and (2) evaluating performance after training. All experiments adhere to the same training configurations by default (e.g., learning rate is 1c-7, B is 2000, 2000 training steps, etc.), with the only variation being the reward model used to score and construct the training data.
4

---

Published as a conference paper at ICLR 2026
Table 2: Comparisons of reward models using FLUX.1 Fill on BrushBench and EditBench.
CLIPScore
Aestheicc
ImageR
PhckSSore
HPSY2
VQAScooe
UnitiedR
Perception
HPS15
GP14
reward mode!
Brash
Fdle
Brash
Edi
Brash
Edit
Brash
Fdi
Brash
Fdde
Brush
Edi
Brash
Fdir.
Brash
Fdit.
Brash
Edir.
Brush
Edi
Bosoline
26.246
27.003
6-429
5,458
12,200
4910
22,527
21121
21,676
24076
9.051
8,012
3,300
2,485
25
9455
26
6,055
2,470
83,905
06.979
Random
26.239
27.075
6.431
5/499
12,772
4,955
22
508
21.211
27.475
24,100
9.077
8,000
3.356
2
25.944
26.558
6.056
2,490
83.517
66.942
CLIPSoove
26.235
27,072
6.452
5,477
12,791
4,997
22,529
21.215
27.457
24.121
9:071
8,071
3.361
2,512
25,948
6.056
2,499
53,942
66.997
Aeshede
26.250
27,200
6.432
5.478
12,823
5.175
22.537
21.219
27.520
24.142
9:075
8,001
3,363
2.507
25,954
26.578
6.075
2.577
83.950
67.906
ImageR
36.251
27.321
6.434
5,481
12,825
5.001
22.336
21,211
27.538
24.143
9.050
7,917
3.362
2.536
25,946
26
2,550
54.176
67.785
PxcSsore
26.236
27.195
6.436
5,475
12,879
5.134
22.341
21.223
27,530
24.154
9.076
8,003
3,383
2.514
25,955
26.900
6.105
2.548
19.198
67.100
HPS-2
27.160
6.441
5.475
12,904
5.145
22.356
21,232
27.666
34.202
9.08S
8.028
3,363
2.553
25.063
26
2.605
$4.699
68.186
VQASaue
26.243
27.10
6.432
5.466
12.781
4.926
22,
529
21.215
27.486
24.104
9.075
8.000
3,353
2.501
25.050
26 26.861
6,046
2.475
85.854
66.793
UnideeR
26.235
27.133
6.454
5.473
12,769
5,084
22,531
21:212
27.485
24.120
9.076
8,014
3.366
2.517
25.054
26 36.853
2.524
83.050
67.372
Pereppiinn
26.251
27.068
6.433
5.466
12,752
4.975
22,
531 21.209
27.479
24.109
9.082
8.005
3.356
2.514
25.951
26:315
6.064
2.504
894/122
68.024
HPS
27.223
6.441
5:465
12,855
5,002
22,
21:226
27.534
24.155
9.083
8.000
3.377
2.497
25.957
26 26:359
6.106
2.568
94.615
Enccuble
26,239
27.155
6.442
5.472
12.884
5,239
22
546
21:215 27,560
24.146
9.082
8.086
3.367
2.507
25.963
26.915
6.151
2.523
$4.628
67.549
Bold values dencte resuls,
Underlined values Benoti socord best results. Values blue danote results below bascline nundomn chunce.
Note that we may encounter an oracle reward model (Ma et al. 2025a), where the same model is used both for data construction and performance evaluation within one experiment. We assess results on two benchmarks, i.c., BrushBench (Ju et al. (2224) and EditBench (Wang et al. (002)..
Here, we introduce a new reward model- that constructs samples based on the average ranking of all the reward models. We make GPT-4 (Achiam et al. (2023) serve as a "fair" evaluator by assessing aesthetic quality, structural coherence, and semantic alignment of the results (see details in Appendix). We report two other results. Baseline: The model's 's performance prior to DPO train- ing. Random: It involves training with randomly sampled preferred and dispreferred pairs. Results are reported in Table and Table 2
We have the following observations and conclusions.
Some reward models are not reliable evaluators. It is believed that an accurate and robust reward model should assign high evaluation scores to models trained on its own preference dataset (i.e., the oracle reward model setting). Surprisingly, we find that CLIPScore, VQAScore, and Perception fail to meet this requirement-in Table 2] their scores can be even lower than baseline or random results. We hypothesize that the fail of CLIPScore and Perception stems from their large-scale yet potentially coarse contrastive pre- training; the fail of VQAScore likely arises from its simplistic, VQA-like evaluation approach. In light of this, we exclude these models from subsequent analyses.
Most reward models provide valid reward scores. Most reward models capable of offering valid reward scores for preference data construction, as they outperform both the baseline and ran- dom selection across most evaluation results especialy GPT-4. Even though CLIPScore and Per- ception are observed to be less effective at accurately evaluating on small-scale benchmarks, they re- main viable when their reward scores are incorporated into larger-scale preference training datasets. In this context, we continue to attribute VQAScore's 's limitations to its simple scoring methodology.
Reward models may share common biases. We find that the model trained on HPSw2-constructedd data outperforms most competitors when evaluated using public reward models. Specifically, when trained using BrushNet, it ranks first or second in 4 out of 12 evaluations: when trained using F Fill, it ranks first or second in 9 out of 12 evaluations. This pattern aligns with GPT-4's results when using FLUX. Fill but diverges when using BrushNet-under the latter condition, the model is largely outperformed by PickScore. We posit that HPSv2 and many other models may share some common biases, which can potentially lead to reward hacking (Pan et al. 2022).
Ensemble is accurate and robust reward model. It shows that Ensemble ranks first second in 11 out of 12 public model evaluations when using BrushNet, and 7 out of 12 when using FLUX.1 Fill. Besides, Ensemble ranks first or second in 3 out of 4 GPT-4's evaluations across both baseline models, demonstrating its robustness in constructing effective preference data. We hypothesize that its versatility arises from the bias of reward models being weakened in Ensemble.
Discussion. Part of the above analysis is based on an untested assumption GPT-4 is an ideal eval- uator. We will examine its validity as well reward hacking in insstiiionn 6 and the Appendix.
5

---

Published as a conference paper at ICLR 2026
Aceherik
6.54
5,29
12:314
1956
29.776
24.00
5
22:500
20826
$1.10
(0
6.111
<570
13497
2.544
25.476
21,006
20.
16 700
sounss
19.44
6.457
13:00
23.772
20.
28.76)
50
0,4%
-9.55
23.373
In
15
16
16
16
condicote somples
condidane comples
candidate somples
condicate somples
candidate samples
6.542
5624
12:00
1694
25.
25.700
22:517
23.49
$4.54
15:734
13.11
46.855
13667
4.01
22,003
23.380
50.000
0.49
13.90
5942
27.002
34.772
22-22
21.356
85-47
60.533
16
condicate samples
candalate samples
candidute sumples
condiCase samples
candilate samples
(a) Candidate scaling using HPPS22.
23.000
Sada
12.77
20le
26/722
60.200
59887
0.41
5.000
236
20
22:11
26 o0
$1,000
59.22
12.172
cundidate samplees
condidane camples
Candidane somples
condicase samtlee
candidate samples
Anshviic
8.55
Iangeit
12380
12.033
27.532
24.142
24.156
22
22.541
222224
22.200
$4.291
60:22
60.352
21216
16
12010
27:311
16
24.14
04.18
3
16
nandktte samples
candalate samples
candidute samples
csadiiitte sampple
candillate samplee
(b) Candidate scaling using PickScore.
Arttheit
13.09
23.201
20011
8.77
5.00
13.50
23,50
22:315
20/773
91.76
19.596
12919-
28.711
$1.02
condicaae samplex
candichne comple
candidone somples
candičate sommlle
candicitte samples
Acchurik
5005
13.000
1347
20000
22:
23:175
27.707
72.597
Soin
13x45-
25:534
22:529
314499
71.245
24.90
22-771
90.50
13:44
24,022
22,41
In
16
In
16
condicate samples
candidate samples
candidate samples
condicate camples
candidate samples
(c) Candidate scaling using Ensemble.
5628
12:000
29.716
24166
22:307
20835
$1.200
19915
13.
20.
22.191
18.00
90.777
59:318
6.59
23,711
COMIIARCatto training steps
training steps
training steps
training
training steps
12677
6.88
24.547
22.92
13.355
12.11
22.00
12594
853 1200 2006
sce 1000
1200
990
900 1200 I0CC 2006
maining steys
truining steps
muining steps
traiiing steps
tuining steps
(d) Sample scaling using HPSv2.
115
12.94
101s
20093
20:250
28,743
20
00:21
12.15-
$2,301
59.759
#1.11
6.37 903 120 1930 2000
123000
1.574
23.579
900
22.000
22.135
401 B98
1200 1000 2010
Str0e
81.201
491 991 1200 HIGC
triniing steps
training steps
traiting steps
training steps
training steps
Ansiveii
CARAE
24.113
21216
12.79-
surrs
2406
21.00
20.355
Save
24.15
22.30
1200
1200
Inaining slaps
sioge
steps
Iraining slipss
Insining slops
(c) Sample scaling using PickScore.
13.22
23,000
22.00
20000
60.272
<.00
13:24
21,000
22.306
200111
10.701
23,053
0.48
5.16
109
500
22.353
387799
22.335
833 100 19:30 2006
2001 S00 L20100022000
403 90U 1209 1000 2000
190 EX0 2000
300 901 1200 Ioo 2006"
training staps
training steps
training stepe
training stops
triining steps
6.590
13:87
7.73
29
20122
STA16
12.350
13.716
2b:221
0.505
6553
25.421
22.50
21.470
09.531
8.000
28,14
22.402
27.400
1200
mrining steps
training
steps
training
training steps
(1) Sample scaling using Ensemble.
Figure 2: Candidate scaling (a-c) and sample scaling (d-f) using HPSv2, PickScore, and Ensemble.
We employ Aesthetic, ImageR, HPSv2, PickScore, and GPT-4 for evaluation. The first and second row of each sub-figure is based on BrushNet and FLUXX11 Fill, respectively. We use training steps to indicate the consumed samples to align the scaling across models (their batch-sizes are different).
5 HOW SCALABLE ARE PREFERENCE DATA?
The results in section 4 have shown that IIPSv2, PickScore, and Ensemble are promising reward models. Building on this finding, we conduct an investigation into the scalability of preference data using these reward models. Specifically, we explore along two dimensions: (1) Candidate scaling. As the number of candidate samples generated from different random seeds increases, their diversity expands. This augmentation in diversity would enhance the accuracy of the construction of preferred
6

---

Published as a conference paper at ICLR 2026
(a) Examples from models trained using BrushNet.
(b) Examples from models trained using FLUX.1
Figure 3: Bias studies. each sub-figure, the four images (from left right) display: the masked image. followed by inpainting results from models trained using HPSv2, PickScore, and Ensemble. We omit text prompts for brevity. Zoom in to see details. Find more examples in the Appendix.
(a) Examples from models trained using BrushNet.
(b) Examples from models trained using FLUX.1 Fill.
Figure 4: Qualitative results of ablations. each figure, the three images (from left to right) display: the masked image. followed by inpainting results from baseline models and baseline models preference alignment using Ensemble. omit text prompts for brevity. Zoom to seed details.
and dispreferred samples by enhancing their differences. (2) Sample scaling, A larger dataset en- ables the model to capture nuanced patterns more comprehensively, leading to deeper learning of preferences. Based on the insights in infsection 4] we select Aesthetic, ImageR, HPSv2, PickScore, and conduct
GPT-4 as the evaluation models. To enable the model to achieve optimal performance, a search over two typical hyper--arameters--1 and learning rate (see details in the Appendix), be- fore the scaling experiments. For each experiment, tune scaling dimension and fix the other dimension. The results are reported in Figure 2 We have the following results and discoveries.
Consistent scaling trends across models and benchmarks. First, observe that data scaling demonstrates robust trends regardless of the model used-BrushNet and FLUX. Fill, in the first and second rows of each sub-figure, respectively. Second, we find that similar scaling trends emerge when evaluating on different benchmarks, as evidenced by the comparable patterns of the two lines within each sub-figure. These findings indicate that the observed scaling behavior is robust and generalizable. However, we also observe some inconsistent phenomena: when using PickScore as the reward model, ImageR/HPSv2 exhibit opposite trends on BrushNet and FLUXX.1 Fill. This issue caused by the characteristics of both reward models and baseline models, analyzed in snsectioon 6
Reward hacking from HPSv2 undermines training. When evaluated by Aesthetic, ImageR, HPSv2, and PickScore, using HPSv2 the reward model shows benefits from both candidate scal- ing and sample scaling. However, its GPT-4 results deteriorate significantly scaling. This observation aligns with finding in section 4, where HPSv2 achieves good results under public model evaluations but sometimes loses to others when assessed by GPT-4. We hypoth- esize that this degradation stems from some shared common biases among these reward models.
the later stages of

---

Published as a conference paper at ICLR 2026
Table 3: Ablation studies on new dataset of I Dream My Painting (Fanelli et
2025
Inpaisting model CLiPScore
Aesthetic
ImageR
PickScore
IPPS2
VQAScore
UnisedR
Perception
HPSS
GPT4
BevehNer
24 24.840
5,223
-0.345
20.550
19.749
8.583
2,317
27:317
-0551
72.609
BruPA (ours)
25,460
6.111
2153
20.735
21,156
8663
2,463
28.294
1,265
73.739
FIJUX. Fill
194
6017
0,544
20,55
30.301
8.667
2,476
26.627
0.547
76.191
PlaPA (ours) 25,500
6,488
5.61
21,407
23.770
9.831
2.544
283688
5003
79.255
Bold values desote the best resalls.
Table 4: Comparisons of state-of- art image inpainting models on BrushBench and EditBench.
iepainiing medk)
CLJPSecee
Accholic
InageR
Picköcane
HPS
VQJAsoo::
Unifsllf
Percepiion
HTSv3
GP/141
Brah:
Edi
Briad
Edit
Brash.
Bruds
Edit
Basahs
Brush.
Edit.
Bruds
Edit
Brash.
Brush.
Edit.
Brish
SDI
26,364 26.506
6.318
5,377
12,005
-1.100
20,751
37,079
25,215
KMM61
0,925
3,268
2,069
190
25,362
5,300
0.549
75.004
60,751
CNI
26.34] 26.972
6,305
5,382
11,421
-1.044
21,953
20.874
26.035
230076
8,890
6,906
3.218
2.125
26,150
25,894
4,546
0.894
74.173
63.921
BED
36.15 29 666
6,760
5,523
11.161
0.963
21
30:000
76
775
2389
7-467
1,30
2,27%
76,
178
22,093
4.599
1.114
11.794
00,000
Powerthint
20.200 27.291
6.312
5,418
11.771
0.720
22,009
20.912
27.055
23:347
KM11
7.238
3.271
2.219
20.125
26.264
5,113
1008
78.241
63/00
PrefPaint
26,208 25.509
6.377
5,296
11,795
22.125
20,066
263355
22.241
3.271
1.951
26.115
24.264
5,305
50.327
00.915
SuDiffusio
23,872 21.298
5:330
4,405
0.063
16.2222
19.147
21.431
16.816
2,417
1,236
23.381
20.102
2.245
34.255
25,200
HD-Paittr
26.307 26.9/3
6,400
1,630
12,913
22.314
21.016
37.931
23:51
9,019
3.349
2.
26.214
25.721
6.204
19855
151010
00.000
ASUKA
24.347 20.542
6.294
5,075
5,208
134.110
21.601
19,003
25.255
18,702
7,681
3,631
2,862
1,242
25,059
19.017
3,556
13,506
75.140
50,00
BraskNct
26.415
25.333
6:425
5.M0
12:717
-1.796
22.155
20.66
27.900
23,076
9.000
6.770
3.301
2.
1.000
26.290
26.410
5.749
0.405
79.391
$9,046
BraP' (ours)
26,547
27.044
0.116
5,577
13.11
10.463
20.219
20.844
25.012
739933
0.01
7.44
1,121
2. 1919
26.990
5,226
1.998
83054
61.10
FLUX.I
20,244
27,
6:429
5,458
12,700
22.327
21,211
27:476
34,076
60.979
FluPA (ours)
26.436
27.813
6.546
5.681
13,899
22.577
21,559
28.735
353922
34577
2,649
26,096
27.617
7,000
$7.009
73.307
Roil
uxing officiaal implomooeeiinnnsseoo...oooo blending
Ensemble offers robust data scaling by resisting hacking, Although PickScore demonstrates good scaling behavior, its performance remains sub-optimal. In contrast, the Ensemble approach achieves the best results across benchmarks, model structures, evaluation models, and scaling dimensions. This is likely because Ensemble averages the preference choices of different reward models, which eliminates the biases of the employed reward models and improves its resistance to the hacking.
6 HOW REWARD HACKING HAPPENS?
We identify potential biases in reward models that may lead to reward hacking, as discussed in section 4 and Section 5] In this section, we delve deeper into exploring these intriguing biases- examining their nature and how they make reward hacking happen. To investigate it, we sample inpainting examples in Figuue 3 and Figure 4 We report the following findings and insights.
Reward models exhibit biases in brightness, composition, and color scheme. As evidenced by the results from HPSv2 and PickScore the second and third images in each sub-figure of Figure 3 respectively, we observe notable biases in their preferences. HPSv2 tends to favor images with bright lighting, complex composition with rich details, and vivid colors. In contrast, PickScore shows a preference for dim lighting, simple composition with few details, and muted colors.
Biases in reward models affect different baseline models in distinet ways. Although each re- ward model has its own inherent biases, we find that their influence varies across baseline models. For instance, BrushNet trained using HPSv2 produces inpainting outputs characterized by exces- sively bright lighting, overly intricate details, and unnaturally vivid colors-they they seem deviate from human aesthetic preferences. In contrast, FLUX.1 Fill trained using HPSv2 generates visually pleas- ing results. PickScore shows a similar disparity in performance. stems from the characteristics of baseline models shown in Figure 4 BrushNet generates vibrant images, making PickScore par- ticularly suitable for it; while FLUX.T Fill produces plain images, aligning with HPSv2's property.
Ensemble shows generality and generalization by mitigating biases. Ensemble, a simple and straightforward method implemented through reward ensembling, exhibits strong versatility across models by producing balanced and aesthetically pleasing inpainting results, as shown in Figure 3 and Figure 4 It likely stems from Ensemble's ability to mitigate biases inherent in reward models.
7 ABLATION STUDIES AND COMPARISONS WITH STATE-OF-THE-ARRT
We name our methods BruPA and FluPA BrushNet and FLUX.1 Fill with Ensemble- based pref- erence alignment. We compare them with tfate-of-the-ar art image inpainting models. Specifically, the following methods are compared (they are introduced in [section 2): SDI (Rombach et al. 2022), CNI (Zhang al. 2023b), BL. (Avrahami al. 2023). PowerPaint (Zhuang et al. 2024)
8

---

Published as a conference paper at ICLR 2026
A girl is painting pumpkin with cat.
colorful cat with splash paint its face.
A cartoon girl with long hair and backpack.
Cartoon fox driving car with cute face.
Conition
SDI
CNI
BLD
PowerPaint
PrePPuin
StrDiffusion
HD-Palnter
ASUKA
Ours
Figure 5: Qualitative comparisons with state-of-the-ae image inpainting models.
ASUKA
(23%)
Painter
St.Dul (99)
FlotA (70%)
(20%))
Pre:hirr (10%)
TulA (47%)
Bi (35%)
PowaxPhiia (15%)
(75%)
de (117%)
BLD (14%))
(75%)
tie (157%)
CNI (86)
HoxtA (60%)
He (1895)
SDI (.%))
Talts (15%)
tee (901)
FLUX: i Fill (18%)
HriPA (40%)
Tic (47%))
BrsstNer (17%)
IM0
Figure 6: User studies. We compare each pair of models by randomly sampling 100 pairs from their inpainting results. We invite 30 volunteers participate in blind assessment determine which one is better ("A win", "B win", or "tic") based on their preferences. We report the winning rates.
BrushNet (Ju et al. 2024), PrefPaint (Liu et al. 2024b), StrDiffusion (Liu et al., (024a), FLUX. Fill FolastForestLanh Labs 2024b). HD-Painter (Manukyan et al. 2023). and ASUKA (Wang et al, 2023c), where BrushNer and FLUX.1 I Fill are also the baseline models for ablation studies.
Ablation studies. We report quantitative and qualitative ablation studies, before and after preference alignment training, in Table 4 and Figure 4, respectively. After preference alignment using Ensem- ble, our method significantly surpasses the baseline models by achieving much better results and yielding visually appealing results. We further conduct ablation studies on a new dataset, reported in Table It also confirms that our improvement is generalizable across data distributions.
Comparisons with state-of-the-art. Table 4 reports the results. Our BruPA and FluPA set new state-of-the-arts attaining the best results on all evaluations and the second-bent results in nearly half of the cases. Notably, even on coarser metrics CLIPScore, VQAScore, and Perception (analyzed in section 4) 4ur methods still outperform competitors. Besides, BruPA and FluPA significantly outperform BrushNet and FL.UX.1 Fill, i.e, the baselines before applying preference alignment. The qualitative results are reported in Figure 5] and our model generates images with better aesthetics.
User studies. As shown
Figure 6. our models align with human preferences better.
CONCLUSION AND LIMITATION DISCUSSIONS
We conduct extensive studies on image inpainting with preference alignment and obtain key insights into the effectiveness, scalability, and challenges in achieving alignment. We find that simple en- semble method mitigate biases and achieve non-trite non-trivial results. Yet, our work is confined to images and DPO training; future extensions generalize these findings to video, 3D data, and RLHF.

---

Published as a conference paper at ICLR 2026
Reproducibility statement. We conduct experiments using the official implementations of Brush- Net (Ju et al. 2024) and FLUX.1 1 Fill (BlackForestLass] 2024b). 20024)) The DPO loss is implemented based on the official code from Wallace et al. (2024). Additional implementation details are also provided in section Section 5) and the Appendix. Our code will be open-sourced to ensure reproducibility.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint 1rXXiv2303.0877:: 2023.
Omri Avrahami, Ohad Fried, and Dani Lischinski. Blended latent diffusion. ACM transactions on graphics, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assis- tant with reinforcement learning from human feedback. arXiv preprint arXiv:2204. 055622, 2022.
Marcelo Bertalmio, Guillermo Sapiro, Vincent Caselles, and Coloma Ballester. Image inpainting. In Proceedings of the 27th annual conference Computer graphics and interactive techniques, 2000.
Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. 2023.
BlackForestLabs, I-ux. http:///aannnaaaccooocccccssssss 2024a. Ac-
cessed: 2025-07-12.
BlackForestl.abs. I-ill-dv... 2024b. Accessed: 2025-07-12.
Daniel Bolya, Po-Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13188 2025.
Qihua Chen, Yue Ma, Hongfa Wang. Junkun Yuan, Wenzhe Zhao, Qi Tian, Hongmei Wang. Shaobo Min, Qifeng Chen, and Wei Liu. Follow-your-canvas: Hesher-resolution video outpainting with extensive content generation. arXiv preprint crXiv:2409.01053 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et Scaling rectified flow transformers for high-resolution image synthesis. In International conference on machine learning, 2024.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lec. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models, Advances in Neural Information Processing Systems, 2023.
Nicola Fanelli, Gennaro Vessio, and Giovanna Castellano. dream my painting: Connecting milms and diffusion models via prompt generation for text-guided multi-mask inpainting. In IEEE/CVF Winter Conference on Applications of Computer Vision, 2025.
Yu Gao, Lixue Gong, Qiushan Guo, Xiaoxia Hou, Zhichao Lai, Fanshi Li, Liang Li, Xiaochen Lian, Chao Liao, Liyang Liu, al. Seedream 3.0 technical report. arXiv preprint arXiv:2504.11344 2025a.
Yu Gao, Haoyuan Guo, Tuyen Hoang, Weilin Huang, Lu Jiang, Fangyuan Kong, Huixia Li, Jiashi Li, Liang Li, Xiaojie Li, et Seedance 1.0: Exploring the boundaries of video generation models. arXiv preprint (wx::2206609111111 091 2025b.
Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint ar0iv:2501. 12948, 2025.
10

---

Published as a conference paper at ICLR 2026
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference- frre evaluation metric for image captioning. arXiv preprint arXiv:1104.08718, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 2020.
Zijing Hu, Junkun Yuan, Kairong Han, Yunze Tong, Shengyu Zhang, Fei Wu, and Kun Kuang. Reinforcement learning in generative multimodal ai: A survey. Authorea Preprints, 2026.
Xuan Ju, Xian Liu, Xintao Wang, Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and- play image inpainting model with decomposed dual-branch diffusion. In European Conference on Computer Vision, 2024.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for tex--t-immae generation. Advances neural information processing systems, 2023.
Weijie Kong, Qi Tian, Zijian Zhang, Rox Min, Zuozhuo Dai, Jin Zhou, Jiangfeng Xiong, Xin Li, Bo Wu, Jianwei Zhang, et al. Hunyuanvideo: A systematic framework for large video generative models. arXiv preprint arXhv:2412.03600 2024.
Kyungmin Lee, Xiahong Li, Qifei Wang, Junfeng He, Junjie Ke, Ming- Hsuan Yang, Irfan Essa, Jinwoo Shin, Feng Yang, and Yinxiao Li. Calibrated multi-preference optimization for aligning diffusion models. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, 2022.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang, Luc Van Gool, and Radu Timofte. Swinir: Image restoration using swin transformer. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Zhiqiu Lin, Deepak Pathak, Baigi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, 2024.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Haipeng Liu, Yang Wang, Biao Qian, Meng Wang, and Yong Rui. Structure matters: Tackling the semantic discrepancy in diffusion models for image inpainting. In IEEE/CVF Conference Computer Vision and Pattern Recognition, 2024a.
Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint arXiv:2500 25X.::2501113/11 13918, 2025.
Kendong Liu, Zhiyu Zhu, Chuanhao Li, Hui Liu, Huanqiang Zeng, and Junhui Hou. Prefpaint: Aligning image inpainting diffusion model with human preference. Advances in Neural Informa- fion Processing Systems, 2024b.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Sain- ing Xie. Sit: Exploring flow and diffusion-based generative models with scalable interpolant transformers. In European Conference on Computer Vision, 2024a.
Nanye Ma, Shangyuan Tong, Haolin Jia, Hexiang Hu, Yu-Chuan Su, Mingda Zhang, Xuan Yang, Yandong Li, Tommi Jaakkola, Xuhui Jia, et al. Scaling inference time compute for diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025a.
Yue Ma, Hongyu Liu, Hongfa Wang. Heng Pan, Yingqing He, Junkun Yuan, Ailing Zeng, Chengfei cineocontoo and expressive
Cai, Heung-Yee Yeung Shum, Wei Liu, et al. follow-your-ent freestyle portrait animation. In SIGGRAPH Asia 2024 Conference Papers, 2024b.

---

Published as a conference paper at ICLR 2026
Yuhang Ma, Xiaoshi Wu, Keqiang Sun, and Hongsheng Li. Hpsv3: Towards wide-spectrum human preference score. arXiv preprint arXiv:250s.03789, 2025b.
Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Hd-painter: high-resolution and noompt-fatthfin text-guided image inpainting with diffusion models. In International Conference on Learning Reprexentations, 2023.
Alexander Pan, Kush Bhatia, and Jacob Steinhardt. The effects of reward misspecification: Mapping and mitigating misaligned models. arXiv preprint arXiv:2201.0333 2022.
QwenTeam. Qwen3-vl. https://qwen.ai/bbbt http::/qween.ibbbogdddd-aaaaaaa 2025. Accessed: 2025-11-20.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, 2015.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 170v.01777.06477 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, 2015.
Jiaming Song. Chenlin Meng. and Stefano Ermon. Denoising diffusion implicit models. preprint arXiv:2010.02502, 2020.
arXiv
Richard S Sutton, Andrew G Barto, et al. Reinforcement learning: An introduction. MIT press Cambridge, 1998.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 2017.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Ste- fano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Diffusion model alignment using direct preference optimization. In IEEF/CVF Conference on Computer Vision and Patlern Recog- nition, pp. 8228 8038-8238,777 2024.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haim- ing Zhao. Jianxiao Yang, et Wan: Open and advanced large-scale video generative models. arXiv preprint arXiv:2503.203/ 2025.
Junke Wang. Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504.1 1455, 2025a.
12

---

Published as a conference paper at ICLR 2026
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing lision-langual model's ' perception of the world at any resolution. arXiv preprint mmX1:2449.121111, 2024.
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi Pont-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guide guided image impainting. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multi- modal understanding and generation. arXiv preprint arXiv:2503.05236. 2025b.
Yikai Wang, Chenjie Cao, Junqiu Yu, Ke Fan, Xiangyang Xue, and Yanwei Fu. Towards enhanced image inpainting: Mitigating unwanted object insertion and preserving color consistency. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025c.
Sihao Wu, Xiaonan Si, Chi Xing, Jianhong Wang, Gaojie Jin, Guangliang Cheng, Lijun Zhang, and Xiaowei Huang. Preference alignment on diffusion model: comprehensive survey for image generation and editing. arXiv preprint arXiv:2502.07822, 2025.
Xiaoshi Wu, Yiming Hao. Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306. 09341, 2023.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 2023.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.0755
2025.
Junkun Yuan, Xu Ma, Defang Chen, Kun Kuang, Fei Wu, and Lanfen Lin. Label-efficient domain generalization via collaborative exploration and generalization. In Proceedings of the 30th ACM international conference on multimedia, pp. 2361-2370 2022.
Junkun Yuan, Xu Ma. Defang Chen, Kun Kuang, Fei Wu, and Lanfen Lin. Domain-specific bias filtering for single labeled domain generalization. International Journal of Computer Vision, 131 (2):552-571, 2023a.
Junkun Yuan, Xu Ma, Defang Chen, Fei Wu, Lanfen Lin, and Kun Kuang. Collaborative semantic aggregation and calibration for federated domain generalization. IEEE Transactions on Knowl- 12528)--1228812 2023b.
edge and Data Engineering,
Junkun Yuan, Xinyu Zhang, Hao Zhou, Jian Wang. Zhongwei Qiu, Zhiyin Shao, Shaofeng Zhang, Sifan Long, Kun Kuang, Kun Yao, et al. Hap: Structure-aware masked image modeling for human-centric perception. Advances in Neural Information Processing Systems, 2023c.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guiie image editing. Advances in Neural Information Processing Systems, 2023a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023b.
Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. In European Conference on Computer Vision, 2024.
13

---

Published as a conference paper at ICLR 2026
A DETAILS TO MAKE GPT-4 AS AN EVALUATOR
Given GPT-4's (Achiam al. 2023) strong multi- motiil understanding capabilities, we use it to evaluate image inpainting results. Specifically, we provide GPT-4 with (1) system prompt, (2) an input image, (3) the mask impaint, (4) an inpainting prompt, (5) and the inpainting result.
We randomly selected 500 inpainting pairs and invited volunteers to judge which result is better. Our human raters were drawn completely at random from large and diverse volunteer pool. This random selection ensures that the evaluation is representative. To quantify agreement, the alignment accuracy between the reward models and human raters, defined as the average propor- tion of cases in which both the raters and the model prefer the same result. In our evaluation, GPT-4 achieves 86% accuracy; HPSv2: 82%, PickScore: 80%, ImageReward: 77%, and Aesthetic: 80%. We acknowledge that, due to the complexity of real-world scenarios, the raters are able to recruit may still exhibit minor biases.
compute
To eliminate the potential biases in design of prompts, we provide results the weights of the citeria-aesthetie quality, structural coherence, and semantic alignment, in Table 5| Our BruPA and FluPA consistently outperform the baseline models by large margin, showing the significant and robust improvement of our method in GPT-4 evaluations.
Table GPT-4 evaluation results with different weight combinations: [aesthetic [oherence]-[ssmannee alignment]. "auto" represents the weight is determined by GPT-4 itself.
inpuinting model
30-30.40
BrushNet
79.391
78.423
77.937
78.169
77.388
BruPA
83.054
82,538
82,388
82.226
81.036
FLUX.I Fil
83.935
83.746
83.229
83.217
81.649
FlaPA
87.609
87.559
87.368
87.243
85.309
Here, to further evaluate the results, we include an open source vision-language foundation model- Qwen3-Ve VL JOwenTeamm 2025) for evaluation. The results with means and standard deviations are reported in Table 6 Evaluation results of GPT-4 and Qwen3-VL are broadly stable, with our BruPA and FluPA delivering consistently remarkable performance gains.
Table 6: Evaluation by GPT-4 and Owen3-VL
GPT-4
Qwen3-VL
inpainting model
Brush.
Edit.
Brush.
Edit
SDI
79.004 + 0.203
63.921 1: 0.410
90.323 + 0,245
88.300 0,310
CNI
74.173 1 0.817
60.751 1: 0.590
88.101 =: 0,277
85.519 1 0.337
BLD
71.794 1 0.657
62.690 0.736
85.719 1 0.461
86,886 0.336
PowerPoint
78.241 + 0.912
63. 03.092 0,738
89,763 + 0.250
85 68.682 0.425
PrefPaint
80.327 + 0.293
60.815 0.392
91.249 0.220
83.269 0.449
SuDiffusion
34.255 + 0.159
25,200 0.236
49,560 : 0.394
40, 867 0.336
HDD-ainter
85.016 0.626
59.089 0.628
92.361 0.026
87.389 0.325
ASUKA
75.140 1.022
$8.686 0.477
84.597 0.458
73.614 0.980
BrushNet
BruPA (ours)
92.255
FLUX.1
83 93.935--4 + 0.414
0.463
92.752 + 0.145
0.306
FluPA (ours)
87.609 + 0.291
72.307 0.560
94.501 + 0.154
91.603 0.220
The system prompt designed by is given below:
14

---

Published as a conference paper at ICLR 2026
GPT-4 System Prompt for Image Inpainting Evaluation
You are human expert analysis of image inpainting. Please evaluate the image inpainting result based on the following three criteria:
Aesthetic Quality (0-40 points):
Visual appeal in color harmony, composition, style coherence
Texture realism and naturalness
Structural Coherence (0-30 points)
Preservation of geometric structures and content continuity
Seamlessness at mask boundaries
Semantic Alignment (0-30 points)
Faithfulness the Text Prompt instructions
Contextual consistency added restored content
For each criterion, provide:
A sub-score.
A 1-2-sentence justification.
Then compute the total score (0-100).
B SEARCHES OF HYPER-PARAMETERS
13.522
1,300
26-27
24.00
22:20
18806
20,450
68 90
56-5
13.25
0.51
24.24
22:11
26-722
$1.01
59 718
6.535
0-61
26.11
24.00
22140
20/766
78429
15110
1030
2900
400)
10.00
1000
2001
2000
3900
2000
(000
2009
1000
2000
a000
600
1106
2000
8006
0
8
0,553
5.74
14.100
1.01
25/129
20,98
22:00
21,200
14844
0.31
17.072
2121
22.400
2300
1008
2000
4000
5000
1000
2000
4000
5000
/
/
(a) Sensitivity analysis on
6.500
5600
17.001
28.100
24.077
22:207
28/077
$0.250
68:425
5.90
11.240
20.300
19.015
6.45
159
23.000
1648
16-7
10w
10-5
10-7
:046
Bod
1286
1077
Bob
1255
16-7
1046
1o-5
1oll
1x
Ir
6.553
5.09
13.905
1368
25
22. 99
31,500
$9.821
12.590
0.41
16.7
5.902
12.712
10.7
4.033
34,44
22.521
$1.200
1r
1r
(b) Sensitivity analysis on
Figure 7: Searches 3, ie.. sub-figure (a), and learning rate (Ir), i.c., figufi (b). The first and second row of each sub-figure is based BrushNet and FLUX.1 Fill, respectively.
We conduct paraaeear searches for Ensemble, as shown in Figure 7 For Ensemble, finally adopt a learning rate of le-6, and set 3 = 4000 for BrushNet; while using a learning rate of le-5, and set 2000 for FLUX.1 Fill. For HPSv2, we use learning rate of le-5 with 4000 for BrushNet; and a learning rate of le-6 with B 8000 on FLUX.1 Fill. For PickScore, we set the learning rate 1e-7 and use B 2000 for both BrushNet and FLUX.1 Fill.
We also find that fine-tuning these paramerasseetrs is not critical. Even when using fixed moderate value (3 = 4000 and
6, as suggested by Wallace al. (2024) across all experiments, our method still far exceeds the baseline w/o DPO: BruPA on BrushBench (83.133 vs. 79.391), BruPA on EditBench (72.442 vs. 57.046), FluPA on BrushBench (86.337 vs. 83.935), FluPA on EditBench (77.908 vs. 66.979) The new results remain the SOTA by outperforming previous best method: BrushBench (86.337 vs. 85.016) and EditBench (77.908 vs. 69.087). Besides, Ensemble less sensitive the change of hyper-paramett variations than single reward models like HPSv2 and PickScore. We hypothesis that this robustness aries because the Ensemble integrates different reward models, thereby mitigating the sensitivity inherent to individual reward model.
15

---

Published as a conference paper at ICLR 2026
C DETAILS OF OUR ENSEMBLE METHOD
Definition of our Ensemble: For set of images generated with different seeds but the same prompt, first rank all images by the score of each reward model and record their ranks. Then average the ranks of single image across all reward models to obtain its Ensemble rank. The image with the highest Ensemble rank selected as the winning sample, and the one with the lowest rank is selected as the losing sample. In this way, the preferences from different reward models are effectively integrated.
Algorithm 1:
Ensemble Method
Input : set of images generated under different seeds but the same prompt:
M: set of metrics;
8 s|m]]]]: score of metric m E M for image E I
Output:
Ensemble((] for each i I
begin
for metric E EM do for image E I do
rank[m]]i] arg sort sort s(m][i] end for end for for image E I do
for metric m. E M do Ensemble + m rank|m]] rank(m][)//MM)
end for end for
Winning sample < arg min,zz Ensemble[i]] Losing sample t- arg m Ensemble[i]]
12
14
end
D DETAILS OF VLM-BASED EVALUATION
We report the details of VL.Mbbased evaluation, i.e., GPT-4 and Qwen3-VL., Table According to both GPT-4 and Qwen3-VL, our BruPA consistently improves image quality across all aspects aesthetic appeal, structure accuracy, and semantic alignment.
Table Details of VLM-based evaluations.
(a) GPT-4
(b) Qwen3-VL
inpainting model overall
aesthetics
inpainting model
ovenal
aesthetics
structure
semantics
BrushNet
79.391
30.301
24.035
25.055
BrushNet
89,930
27.395
27.393
BruPA
83,054
33.516
24.069
25.469
BruPA
92.255
36.032
28.018
28.204
FluPA
83.935
32.617
24.997
26.321
FLUX.1 Fill
92.752
36.244
28.080
28.426
FLUX.1 Fill
87.609
35.100
25.627
26.883
FluPA
94.501
36,972
28.477
29.052
E COMPUTE COSTS OF EACH REWARD MODEL
We report the compute cost of each reward models for giving to a single sample in Table g Note that our Ensemble method only introduces additional cost for scoring samples and constructing preference pairs; the cost for training remains the same in standard methods.
Table 8: Compute for giving
single sample.
reward model
CL.PPSoore
Aesthetic
ImageR
PickScore
HPSv2
VQAScore
UnifiedR
Perception
HPSv3
cost (scconds)
0.154
0.048
0.0759
0.0319
0.219
0.288
3.051
0.155
0.379
16

---

Published as a conference paper at ICLR 2026
F MORE RESULTS ON REWARD MODEL BIAS STUDIES
(a) cartoon drawing of kitchen.
(b) A pink lotus flower blooming with green leaves.
(c) A couch with pillows and wall behind it.
(d) dog sitting the beach.
(e) house floating over a lake.
(f) Sunflowers in vase with pears on table.
(g) A vase filled with colorful flowers table.
(h) A notebook, glasses and camera map.
(i) A pigeon sitting on ground.
(j) purple chair with black seat and back.
(k) A sheep with pink fur is standing.
(I) A teacup and saucer with spoons.
(m) lamb standing in a field green grass.
(n) vase with blue flowers sitting on a table.
(o) A vase with some flowers in it.
(p) A kitten is playing with flower.
Figure 8: More results on reward model bias studies using BrushNet. In each sub-figure, the four images (from left right) display: the masked image, followed by inpainting results from models trained using HPSv2, PickScore, and Ensemble. For optimal detail, view figures zoomed in.
17

---

Published as a conference paper at ICLR 2026
(a) A woman in white holding bouquet of flowers.
(b) A cartoon boy driving car.
(c) Wolf howling at moon.
(d) A cat is shown low polygonal style.
(e) A couch with winged chair window.
(f) A bright moon on the sea.
(g) fruit bowl with a pink flower on top.
(h) A small cabin in the snow near a lake.
(i) A girl with pink hair and flowers her face.
(i) Cartoon fox driving car with cute face.
(k) A lamp and cup of tea on a table.
(l) person standing on tile floor with a rug.
(m) A rabbit sitting hill with trees.
A red cabin sits the shore of lake.
(o) teacup and saucer with spoons.
(p) vase with flowers it on a dark background.
Figure 9: More results reward model bias studies using FLUX.1 Fill. In each sub-figure, the four images (from left to right) display: the masked image, followed by inpainting results from models trained using HPSv2, PickScore, and Ensemble. For optimal detail, view figures zoomed in.
18


Reviews:
[{"review_id": "WIE3cNc0Ph", "reviewer": "Reviewer_mUgh", "summary": "This paper investigates preference alignment for image inpainting, a relatively underexplored but increasingly important topic.\nRather than proposing a novel architecture, the authors systematically study how to align inpainting models with human aesthetic preferences through Direct Preference Optimization (DPO).", "questions": "Did you try other ensemble strategies (e.g., weighted average or learned combination)?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper is among the first to systematically analyze preference alignment in image inpainting, bridging the gap between text-to-image alignment studies and conditional image editing.\n\nThe experimental design is rigorous: nine reward models, multiple benchmarks, two architectures (diffusion and flow-based), and both automatic (GPT-4) and human evaluations.\n\nThe paper is well-written and easy to follow.", "weaknesses": "The paper shows limited algorithmic novelty, as the core approach mainly combines existing DPO techniques with a straightforward averaging ensemble.\n\nThe reliance on GPT-4 for evaluation may introduce instability and reproducibility concerns, as model-based judgments can vary over time and lack full transparency.\n\nWhile the study highlights potential biases related to brightness, color, and composition, it lacks quantitative analysis or formal metrics to substantiate these observations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iAY23kh9GL", "reviewer": "Reviewer_XjVN", "summary": "This paper conducts a systematic and extensive investigation into aligning image inpainting models with human preferences. Instead of proposing a novel architecture, the authors revisit fundamental questions about using preference alignment in this domain. They employ the DPO algorithm, leveraging nine different public, off-the-shelf reward models to automatically construct preference datasets without costly human annotation.", "questions": "See Weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Pros:\n1. It provides the community with valuable insights into the behavior of reward models and the dynamics of preference alignment.\n2. The analysis and visualization of \"reward hacking\" (Figure 3) clearly demonstrates a crucial pitfall in naive preference alignment and provides a strong motivation for their proposed solution.\n3. The proposed Ensemble method shows that significant gains can be achieved by thoughtfully combining existing tools rather than inventing complex new ones.", "weaknesses": "Cons:\n1. The paper states the Ensemble is based on \"the average ranking of all the reward models\" but could be more explicit. Does this involve normalizing scores before averaging, or is it a direct average of rank positions? A more formal definition or algorithmic description would improve reproducibility. \n2. In your analysis, some reward models (like VQAScore) were found to be unreliable evaluators. However, they seem to be included in the final Ensemble method. Did you test if an ensemble composed of only the most reliable/diverse reward models would perform better or worse than an ensemble of all nine? Is there a benefit to including weaker, but potentially diverse, signals?\n3. More details on the Hyper-parameter Sensitivity are desired. The appendix shows searches for the DPO hyper-parameter β and the learning rate. How crucial was fine-tuning these parameters for achieving top performance? Is the Ensemble method more or less sensitive to these hyper-parameters compared to using a single reward model like HPSv2?\n4. Does the DPO process primarily learn aesthetic qualities (e.g., better lighting, composition), or does it also improve semantic alignment to the text prompt? The results suggest an aesthetic improvement, but it would be interesting to know if prompt faithfulness also improved systematically.\n5. The evaluation benchmarks (BrushBench, EditBench) primarily focus on object replacement, addition, or style modification. The paper does not explore how preference alignment performs on other critical inpainting tasks, such as: Object Removal: Where the primary goal is seamless and plausible background completion, not creative generation. Image Restoration: Such as removing scratches or blemishes, where fidelity and realism are paramount. It's unclear if aligning for \"aesthetic preference\" would be beneficial or even detrimental in these contexts, where the model should be as unobtrusive as possible.\n6. The paper does identify biases (e.g., brightness, complexity). However, it stops short of deeply investigating why these biases exist. This is likely tied to the data on which the reward models were trained (e.g., HPSv2 was trained on a dataset of human-preferred images, which may have skewed towards professionally shot, well-lit photos). A more thorough discussion on the provenance of these biases would strengthen the paper's contribution and provide a clearer path toward creating less biased reward models in the future.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jE45RmXN2c", "reviewer": "Reviewer_854V", "summary": "This paper provides a comprehensive evaluation of preference alignment with multiple reward models for the task of image inpainting. \nIt provides useful insights into how different reward models affect the final performance of the model by studying biases in them that could cause reward hacking, and studying data scaling trends. Finally, it proposes that an ensemble of reward models can produce generalizable results and shows strong quantitative performance.", "questions": "It would be great if the rebuttal could include the comparison with the paper mentioned in the weaknesses section or a justification of why it cannot be included.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper provides a thorough analysis of different reward models for image synthesis, such as 1) demonstrating how some reward models are not reliable, while valid reward models can still exhibit biases, 2) scalability of preference data, and 3) analysis on reward hacking,  through a systematic study. These can be useful analyses for the researchers working with preference alignment. \n- It is the first work showing a systematic study of preference alignment with reward models for image synthesis.\n- Experiment design is clear and well thought out, and the result is written clearly. \n- This paper demonstrates a strong result by identifying reliable reward models and ensembling them, and this can be a baseline that the community could build upon.", "weaknesses": "The reviewer did not find significant weaknesses of the paper other than:\n- Calibrated Multi-Preference Optimization for Aligning Diffusion Models (Lee et al., CVPR 2025) also considers a multi-reward formulation and could be compared against, but this work is only cited very briefly in the related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WfBTCOfzH8", "reviewer": "Reviewer_9M76", "summary": "The paper studies image inpainting with preference alignment by revisiting basic questions rather than proposing a new architecture. Using Direct Preference Optimization, the authors construct preference datasets from nine public reward models and explore (i) which reward models yield useful training signals, (ii) how preference data scale with more candidates and samples, (iii) how reward hacking emerges, and (iv) whether a simple ensemble of reward models mitigates biases. Experiments span two inpainting backbones (BrushNet and FLUX.1 Fill) that cover diffusion and flow matching, evaluated on BrushBench and EditBench. Results show improvements across automatic metrics, GPT‑4 scoring, and human studies, without architectural changes or new datasets.\n\nThe paper offers a careful, evidence‑driven empirical study of preference alignment for inpainting, including scaling analyses and bias diagnostics. The ensemble strategy consistently improves or matches the best reward models across backbones and benchmarks. However, the evaluation pipeline relies heavily on GPT‑4 as a “fair” judge with limited validation and reporting details. The ensemble design and computational cost are underspecified. The bias analysis is qualitative mainly and could benefit from quantitative measurements.", "questions": "- The paper exclusively relies on GPT-4 as the “fair evaluator,” reporting 86% accuracy against volunteer judgments, yet it omits any cross-validation with other Vision-Language Models (VLMs) such as GPT-4o, Gemini, or Qwen2.5-VL, which are widely available and architecturally diverse. \n- The “Ensemble” method is merely the average ranking of all reward models, without exploring learned or heuristic integration schemes, or even VLM-based approaches.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Evaluates nine reward models and two distinct backbones: BrushNet (diffusion) and FLUX.1 Fill (flow matching), across two benchmarks. This breadth supports generalizability claims.\n- The paper concisely derives the DPO loss specialized to visual generation, supporting technical soundness.\n- The paper documents brightness/composition/color‑scheme biases, e.g., HPSv2 favoring bright, detailed, vivid images; PickScore favoring dimmer, simpler, muted ones, providing novel qualitative insight.", "weaknesses": "- GPT‑4 is positioned as a “fair” evaluator with 86% accuracy vs. volunteers, but rater selection, agreement metrics, and sampling protocol are under‑reported. This affects evaluation validity. The evaluation prompt fixes specific criteria and weights (0–100), which may embed subjective biases. This limits generality. Details such as inter‑rater reliability, demographic diversity, and task difficulty balance are not provided. No direct evidence was found in the manuscript. This reduces statistical rigor.\n- Ensemble is defined as average ranking of all reward models with no exploration of weighting schemes or subset selection, limiting technical transparency.\n- The paper omits compute/latency costs for scoring with nine reward models and generating 16 candidates per prompt. No direct evidence found in the manuscript. \n- Bias findings (brightness/composition/color) rely on visual inspection without quantitative measures, limiting scientific rigor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)