PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference
Authors: Kendong Liu
Zhiyu Zhu
Chuanhao Li
Hui LIU
Huanqiang Zeng
Junhui Hou
Venue: NeurIPS 2024 poster (2024)
Keywords: diffusion model
image inpainting
human feedback reinforcement learning
Primary area: diffusion_based_models
TL;DR: This paper makes the first attempt to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework.

Abstract: In this paper, we make the first attempt to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework, significantly improving the quality and visual appeal of inpainted images. Specifically, instead of directly measuring the divergence with paired images, we train a reward model with the dataset we construct, consisting of nearly 51,000 images annotated with human preferences. Then, we adopt a reinforcement learning process to fine-tune the distribution of a pre-trained diffusion model for image inpainting in the direction of higher reward. Moreover, we theoretically deduce the upper bound on the error of the reward model, which illustrates the potential confidence of reward estimation throughout the reinforcement alignment process, thereby facilitating accurate regularization.
Extensive experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, demonstrate the effectiveness of our approach, showing significant improvements in the alignment of inpainted images with human preference compared with state-of-the-art methods. This research not only advances the field of image inpainting but also provides a framework for incorporating human preference into the iterative refinement of generative models based on modeling reward accuracy, with broad implications for the design of visually driven AI applications. Our code and dataset are publicly available at \url{https://prefpaint.github.io}.

PrefPaint: Aligning Image Inpainting Diffusion Model with Human Preference
Kendong Liu'; Zhiyu Zhu! Chuanhao Lif
Hui Liu",
Huanqiang Zeng* Junhui Hou!
CCity University of Hong Kong Yale University Saint Francis University "Huaqiao University
(xdliu2-c, zhiyuzhu2-ch @my
cityu. edu.
chuanhao 11. www.cc166yall...... edu h2liuesfu edu hk jh..ouucityu. eeu edu. hk
Abstract
In this paper, make the first attempt to align diffusion models for image in- painting with human aesthetic standards via reinforcement learning framework, significantly improving the quality and visual appeal of inpainted images. Specifi- cally, instead of directly measuring the divergence with paired images, we train reward model with the dataset we construct, consisting of nearly 51,000 im- ages annotated with human preferences. Then, we adopt a reinforcement learning process to fine-tune the distribution of pre-trained diffusion model for image inpainting in the direction of higher reward. Morcover, we theoretically deduce the upper bound on the error of the reward model, which illustrates the potential confidence of reward estimation throughout the reinforcement alignment process, thereby facilitating accurate regularization. Extensive experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, demonstrate the effectiveness of our approach, showing significant improvements in the alignment of inpainted images with human preference compared with state- of-the-art methods. This research not only advances the field of image inpainting but also provides framework for incorporating human preference into the iterative refinement of generative models based on modeling reward accuracy, with broad implications for the design of visually driven Al applications. Our code and dataset are publicly available at htthtts: pettps://prefpanttt github.io
Introduction
Image inpainting, the process of filling in missing or damaged parts of images, is critical task in computer vision with applications ranging from photo restoration 2 to content creation Traditional approaches have leveraged various techniques, from simple interpolation (5777 to complex texture synthesis [ 9,, to achieve visually plausible results. The recent advent of deep learning, particularly diffusion models, has revolutionized the field by enabling more coherent and contextually appropriate inpaintings [3] 12. Despite these advancements, a significant gap remains between the technical success of these models and their alignment with human aesthetic preferences, which are inherently subjective and intricate. As shown in Fig. [ the existing stable diffusion-baat based inpainting model tends to generate weird and discord reconstruction.
Human preference for visual content influenced by complex and mutual factors, including not limited to personal background, experiences, and the context in which the content is viewed [4) This makes the task of aligning inpainting models with human preference particularly challenging, as
"Equal Contribution
1 Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Phompe
Ours
Runway
Prompt
Ours
Runway
Prompt
Ones
LUNG HOCERT HUAN
Runway
Figure 1: Visual comparisons of the results by the diffusion-bee based image inpainting model named "Rumway", and the aligned model through the proposed method.
it requires the model not only to understand the content of the missing parts but also to predict and adapt the diverse tastes of its users.
Inspired by recent advancements in the reinforced alignment of large pre-trained models 15H19, we propose to align diffusion models for image inpainting with human preference through reinforcement learning. Specifically, our approach is grounded in the hypothesis that by incorporating human feedback into the training loop, can guide the model toward generating inpainted images that are not only technically proficient but also visually appealing to users. Technically, we formulate the boundary of each reward prediction in terms of the model's alignment with human aesthetic preferences, thereby leveraging the accuracy of the reward model to amplify the regularization strength more reliable samples. Extensive experiments validate that the proposed method can consistently reconstruct visually pleasing inpainting results and greatly surpass state-of-the-aee methods.
In summary, the main contributions of this paper lie in:
we make the first attempt to align diffusion models for image inpainting with human preferences by integrating human feedback through reinforcement learning:
theoretically deduce the accuracy bound of the reward model, modulating the refinement process of the diffusion model for robustly improving both efficacy and efficiency; and we construct a dataset containing 51,000 inpainted images annotated with human preferences.
Related Work
a paradigm of machine learning where an agent learns to make decisions by taking actions in an environment to maximize some notion of cumulative reward. The foundational theory of reinforcement learning rooted in the concepts of Markov decision processes (23), which provide a mathematical structure
Reinforcement Learning Model Alignment. Reinforcement learning 20

---

for modeling decision-maken making in environments with stochastic dynamics and rewards [24 25] With the advent of deep learning, deep reinforcement learning [2] 26 has significantly expanded the capabilities and applications of traditional reinforcement learning. The integration of neural networks with reinforcement learning, exemplified by the Deep Q-Network [27 28 algorithm, has enabled the handling of nigh-dimensional state spaces, which were previously intractable. Subsequent innovations, including policy gradient methods [29] 30 like Proximal Policy Optimization [31] and actor-critic frameworks [32] 33 like Soft Actor-Critic [34], have further enhanced the efficiency and stability of learning in complex environments.
The recent surge in popularity of large-scale models has significantly underscored the importance of reinforcement learning in contemporary Al research and application [65] As these models, including large language models [35] 36 and deep generative networks 37,39,, become more prevalent, reinforcement learning increasingly employed to fine-tune, control, and optimize their behaviors in complex, dynamic environments. This integration is particularly visible in areas such as natural language processing, where reinforcement learning techniques are used improve the conversational abilities of chat-bots and virtual assistants, making them more adaptive and responsive to user needs. [[7-19] Moreover, in the realm of content recommendation and personalization, reinforcement learning algorithms are instrumental in managing the balance between exploration of new content and exploitation of known user preferences, significantly enhancing the user experience. The growing intersection between large models and reinforcement learning not only pushes the boundaries of what's ' achievable in AI but also amplifies the need for sophisticated reinforcement learning techniques that can operate scale, adapt in real-time, and make decisions under uncertainty, thereby marking a pivotal evolution in how intelligent systems are developed and deployed.
Diffusion Model. Diffusion models have recently emerged as powerful class of generative models [38/42], demonstrating remarkable success in generating high-quality, coherent images 44 44J. These models work by gradually transforming distribution of random noise into a distribution of images, effectively 'diffusing" the noise into structured patterns [45] 46). In the context of image inpainting, diffusion models offer significant advantage by leveraging their generative capabilities to predict and fill in missing parts of images in a way that is contextually and visually coherent with the surrounding image content [47 48).
Recent studies have showcased the potential of diffusion models in achieving state-of-the the-art results in image inpainting tasks, outperforming previous generative models like Generative Adversarial Networks in terms of image quality and coherence [49 50] However, while these models excel in technical performance, there remains a gap in their ability to cater to diverse human aesthetic preferences. Most existing works focus on the objective quality inpainting results, such as fidelity to the original image and coherence of the generated content, with less attention given to subjective satisfaction or preference alignment.
3 Proposed Method
Due to the random masking within the task of image inpainting, there may not be a definitive causal relationship between the known and inpainted contents, which manifests as a one-to-many issue. Consequently, stringent per-pixel regularization inherently results in unnatural reconstruction. which significantly diverges from samples conforming to human preference. To this end, we propose a reinforcement learning-based alignment process involving human preferences to fine-tune pre-trained diffusion models for image inpainting, aiming to improve the visual quality of inpainted images (Sec. 3.1). More importantly, we theoretically deduce the upper bound on the error of the reward model (Sec. 3.2). Based on this deduction, we formulate a reward trustines-aware alignment process that is more efficient and effective (Sec.
3.1 Reinforced Training of Diffusion Models for Image Inpainting
Diffusion models [38 40 iteratively refine a randomly sampled standard Gaussian distributed noise, resulting in a generated image. To adjust the distribution of diffusion models, we introduce human feedback rewards to measure and regularize the distribution of model sampling outputs. To be specific, instead of applying standard prlicy-gradient descent [51] that is hard to converge to high-quality models, inspired by classical methods, e.g., TRPO/PPO 31 [2], which introduces a model trust region to avoid potential model collapse during the training process, we achieve the reinforced
3

---

training of diffusion model as
VoPo(m)
Vx](x)=-
R(x) + WooDPPll Por Pe (x)
where R(-) represents the reward model, which quantifies the quality of diffusion samples; is the reconstructed sample; Po and Pe: represent the probabilistic functions of training and reference models parameterized with 0 and 9', respectively; Vo calculates the derivative 0: D(-I-) is the divergence measurement of the given two distributions for regularizing the distribution shifting; the hyperparameter x. balances the two terms. Inspired by recent work (53 [4], we take the same step to measure the probability of each diffusion step via calculating the probability of perturbation noise, e.g., the DDIM sampling algorithm 40)
Dt
ac (I:t)
+ V 04 - o7c, + osct.
Dt-1 = Va--1
Vai
where (.) is the noise estimate diffusion network; & is the diffusion step; the scalar oz controls the noise to signal ratio; and oye, represents a Gaussian noise to increase the sampling diversity. Thus, the result of each reverse step consists of a deterministic component žę-1 and a probabilistic component
-t-1- Since 0;-1 ~ (i otI), we calculate the density of each step / = 11-1-1111 to ap- proximate the probability, where (--) denotes the density function of standard Gaussian distribution.
3.2 Bounding Reward Model Error
The precision of the reward model R(-) assumes a pivotal function within this learning framework directs the optimization trajectory. In this section, we theoretically derive its error upper bound, which can facilitate the re- inforced training process in terms of both efficacy and efficiency.
Boundary Line
Emers Reword Model
0.6
0.4
Denote by X = [211 22 IN E N and y [y1, 1/, .... yNl Rixi; a set of N inpainted im- ages of dimension D by diffusion models and correspond- ing ground-truth reward values from human experts, re- spectively. By dividing the reward model into two parts, i.e. feature extractor F()) and linear regression weights rb E DR x1, we thus can represent as (xxt), vb), where ()--) calculates the inner product of two vectors. We can formulate the learning process of the last weights as the result of the following optimization process:
0.2
0.4
1.0
12V
Figure 2: Experimental plot of reward prediction error vs. || on the val- idation set, where dashed line is an upper boundary of error, positively rela- tive to ||
S argmin (F(()) v) v1l2 + All
(3)
where || is the E norm of vector, À 0, and the second term is used for alleviating the over-fitting phenomenon. Thus, we have
/ = (z + AI) 'Z y,
(4)
3 th = (z + AI) - XZZ 4 -b.
(5)
where Z E N F(X) is the set of embeddings of X. 2b. is the ideal weight with y Zab, + C and Ç a noise term between Zab, and y. Then, the following term also holds:
z Φ
+. = (z Tz (I) 'z' AzT (ZTZ + AI) "...
(6)
where x E RR D'xl F(x) ) is the reward embedding typical input sample. Based on Cauchy- Schwarz inequality, there is
1z T +
2 _Tb.
|| cllv-1 + 1 [bb))

---

Serm
4442
15
-200
2030 200C
40 53 25
40 (d)
Detalla Scor
(6)
(b)
(c)
Figure 3: Statistical characteristics of the dataset constructed. (a) the distribution of the images across different selected datasets; (b) the comparison between the distribution of the average score and score for details; (c) and (d) show the numbers of images with different mask ratios the outpainting and warping splits, respectively.
where V zTz AI and | == || Moreover, based on Theorem of 55 for 5 > O, we have 8 probability to make following inequality stands
122-
Iz /z'elv-= s BV2 log( det dete(\)/))t(()))) )
(8)
5
where det(-) computes the determinant of the matrix, and B is scalar. We have
1 2 z zTab.l || BV BB/2 log( det det(V)!! 1/2 + A 11/311/b_11
(9)
5
Chaund
Due to Chound being constant after the training of the reward model, we can conclude that
sup z>p(()
z z"ab_l x ||
(10)
where sup represents the upper bound. Such theoretical bound also experimentally verified in Fig. 2
3.3 Reward Trustiness-Awrr Alignment Process
According Eq. (TO), it can be known that the error of the reward model is bounded by || there might be relatively large error for those ||=||-- quite large. Thus, we further propose a weighted regularization strategy amplify the penalty strength of those samples in the high-trust region. Specifically, we calculate the amplification factor as
7 -|]==ly-+++b
(11)
where the hyperparameters & = 0.05 and b - 0.7 are used for scaling the regularization strength. We then define the overall gradient of the reward trustiness-aware alignment process as
VoJ'(x) VoJ(()
(12)
where Jaxx) is the weighted reward loss for sample x. For the updating of the reference model 0', we adopt (53) update the model in each optimization step. Correctly amplifying the scaling factor can both speed up the convergence speed and reconstruction effectiveness, as experimentally
validated in Sec. 5.2
4 Human Preference-Centric Dataset for Reward
We first randomly selected 6,000, 4,000, 6,000, and 1,000 images with diverse content from
ADE20K [56 177. ImageNet (58) KITTI [59], and Div2K 60 61 datasets, respectively. We then applied the operations described below generate prompt images (i.e., incomplete images), which were further fed into the diffusion model for image inpainting named Runway (62), produc-
ing inpainted images. To mitigate the potential bias of the reward model on different rewards,

---

Table I: Quantitative comparisons of different methods. indicates the small model (non SD-based); "S" is the number of sampling times. For the calculation of WinRate, we first derive the best sample of the compared method among sampling times. Then, we calculate as Tr. where Tw indicates the number of compared samples that surpass the results of Runway (S 1) and 'i' is the total number of prompts. "f (resp. 4)" means the larger (resp. smaller), the better. We normalized the predicted reward values with the dataset distribution. "Var" calculates the variance of different sampling times, showing the consistency of generation quality. (See the Supplementary Material for more details.)
Prompt Methods
Outpainting Prompts
Warping Prompts
WinRate (%)
f
Reward Meant
WinRate (%)
f
Reward Mean
Metrics
S l
=
S 10
Varl
S
=
S 10
Var
Runway
62
73.40
89.32
0.07
75.74
91.42
0.06
SD v1.5
03
11.95
20.67
30.24
-0.43
0.05
11.38
21.22
32.85
-0.38
0.06
SD v2.1
64
10.73
18.51
26.82
-0.44
0.04
11.68
22.11
34.22
-0.36
0.06
SD xl 65
14.56
22.58
31.09
-0.31
0.04
15.43
25.43
36.77
-0.26
0.05
SD xl +++
66
21.15
33.25
45.51
-0.13
0.05
18.66
30.53
43.07
-0.18
0.04
Compvis [67
50.51
66.39
78.21
+0.03
0.03
47.35
65.08
78.01
-0.01
0.04
Kandinsky [68]
14.06
22.73
32.16
-0.37
0.04
11.38
19.46
29.20
-0.42
0.05
MAT * 669
15.06
17.97
20.51
-0.40
0.01
7.17
9.97
12.96
-0.56
0.01
Palette * 41
10.96
16.92
21.37
-0.38
0.02
13.41
20.18
27.37
-0.34
0.03
Ours
70.16
84.65
93.14
+0.38
0.01
72.38
87.10
93.85
+0.36
0.01
Metric
T21
CLIP BLIP Aes.
CA
IS
Rank
V2.
0.50
0.51
0.48
0.37
0:27
0.00
0.12
0.02
Model
S4
70
72
73
74
S2.
0.38
0.28
0.07
Kandirsky
0.30
0.29
0.09
SDv1.5
-1.67
0.19
0.44
4.52
0.38 17.07
5.17
SDv2.1
-1.37
0.20
0.45
4.62
0.39
17.07
4.33
50at
0.83
0.62
0.61
0.50
0.35
0.11
0.15
0.03
Kand.
-3.49 0.18 0.39
5.19
0.39
17.06
5.33
0.65
0.50
0.17
SD xl ++
0.63 0.21 0.46 4.77
0.40 18.95
3.17
Compuis
0.91
0.89
0.63
Runway
3.16 0,22 0.48 4.61 0.43 20.30
2.33
Runway
0.68
0.89
0.87
0.85
0.80
0.51
0.50
0.30
Platte
-1.76 0.22 0.46 4.08
0.37 16.24
5.33
0.37
0.94
0.72
Ours
4.49 0.23 0.49 4.55 0.45 23.71
1.67
90
sandeoth spot
Commavi
Rumntyy
Dd>
Table 2:
Comparison across metrics: higher values are better for all metrics except "Rank".
Figure 4: WinRate comparison heat-map between different methods.
repetitively generated from a given prompt image three distinct inpainted images. Consequently. we obtained 51,000 inpainted images in total, which were scored by human experts following the criteria described below.
Generation of Incomplete Images. We considered two distinctive image completion patterns: inpainting and outpainting. For inpainting, we simulated warping holes on images by changing the viewpoints, where we derived the depth of the scene using [75]. Following past practice [76). we defined camera sequence that forms a sampling grid with three columns for yaw and three rows for pitch. The resulting nine views feature yaw angles within a +0.3 range (i.e., a total range of 35") and pitch angles within a +0.15 range (i.e., a total range of 17"). As the range of views increases, the task of inpainting becomes progressively more challenging. For outpainting, we randomly masked the boundary of the image through two types of random cropping methods: (1) square cropping. where the size of the prompt ranges from 15% to 25% of the image size (512 x 512) randomly; (2) rectangular cropping, where the height of the prompt matches the image size, while the width is randomly sampled between 35% and 40% of the image size. Each cropping method accounts for half of the outpainting prompts.
Scoring Criteria. In the scoring process, we employed three criteria, i.e., (1) structural rationality representing the rationality of the overall structure, whether illogical objects and structures are generated; (2) feeling of local texture showing whether strange textures are generated that do not conform to the characteristics of the object; and (3) overall feeling indicating the impression capturing the overall feeling upon first glances at the image. The score value is in the range of I to 7, indicating

---

Prompt
Kandinsky S1.
SD v2.1
Palette
xl
Compvis
Runway
Ours
Figure 5: Visual comparisons of approach and SOTA methods. The prompted images of 5th and 7th rows are generated by boundary cropping, while the remaining rows by warping. All images were generated with the same random seeds.
from the least most favorable. The final score for reward model training is derived by averaging those three scores using the weights [0.15, 0. 0,15, 0.7]. We refer readers to the Supplementary Material for more details about dataset process and labeling scheme.
Statistical Characteristics of our Dataset. As illustrated in Fig. (a). the score distribution of images from different datasets is generally uniform. Meanwhile, from Fig. (b), we can see that the details and overall score are independent of each other, showing the correctness of the scoring scheme and the necessity of each score. Finally, Figs. (c) and (d) show the ratio of images with different mask sizes, where can be seen that the outpainting has more uniform distribution than warping based hole, which rely on depth map and may not have uniform distribution.
Experiments
Evaluation Metrics. We adopted seven metrics to measure the quality of inpainted images, i.c.,
the predicted reward value by our trained reward model. T2I reward [54). CLIP 770] BLIP (71). Aesthetic (Aes.) [72), Classification Accuracy (CA) [73), and Inception Score (IS) 74). Specifically,
T2I reward, CLIP, BLIP, and Aes. directly measure the consistency between the semantics of inpainted images and the language summary of corresponding prompt images. While, CA and IS indicate the quality of the generative model. Due to the fact that our inpainting reward directly measures the reconstruction quality, we adopted as the principle measurement of our experiment.
Implementation Details. We partitioned the dataset in Sec. 4 into training, validation and testing sets, containing 12,000, 3,000 and 2,000 prompts (with 36,000, 9,000 and 6,000 images), respec- tively. The reinforcement fine-tuning dataset contains the prompts from the original reward training dataset. We employed the pre-trained CLIP (ViT-B) checkpoint as the backbone our reward model R(-) with the final MLP channel equal to 256. We utilized cosine schedule to adjust the learning rate. Notably, we achieved optimal preference accuracy by fixing 70% of the layers with learning rate of le 5 and batch size of 5. We trained the reward model with four
7

---

Table 3: Left: Ablation studies amplification factors, where "static" refers employing constant factor to replace y in Eq. (12). The column "Factor" indicates an average magnitude of amplification strength, i.e. p() (7). For our method, we coordinate the value of k in Eq. (11 to change the E(y) shown in the Table. "Acl." signifies accelcration, calculated by Thp the convergence iterations baseline and compared methods, respectively. For all metrics, the larger, the better. Right: Performance of the reward model trained with two manners based on pre-trained CLIP (70] with various fix rates (FRs). "Acc" and "Var" stand for the accuracy and variance of the reward estimation, respectively. "Bd." is the ratio of data below the same upper boundary. (The underlined settings are selected.)
Th - with TV and Tm. being
RL Amp.
Factor
WinRate
Reward
Acl.
Training
Classiiicutio-driienn
Regression-driven
a)
x
50.00%
0.00
Width
MLP-128 MLP-256
MLP-
MLP-256
b)
V
1.0
69.57%
+0.36
00.00%
Metrics f Acc.
Var.
Acc.
Var.
Acc.
Var.
Bd.
Acc.
Var.
Bd.
c) V
Static
1.4
65.95%
+0.34
+84.28%
FR. 0.3 73.49 0.24 74.29 0.18
74.68
0.41 92.1 74.88
0.38 93.2
d) V
Static
1.6
65.11%
+0.35
+203.53 %
FR 0.5 74.73 0.25 75.32 0.26
74.78
0,44 92.3
75.32
0,44 94.4
e)
V
Ours
1.4
71.27%
+0.37
+106.40%
FR 0.7 74.89
0.28 74.53 0.25
75.91
0.46
85.0
75.94
0.45 97.3
0
V Ours
1.6
70.47%
+0.36
+102.83%
FR 0.9 74.09 0.16
72.76 0.14
74.73
0.41 78.8
74.83
0.40 98.0
NVIDIA GeForce RTX 3090 GPUs, each equipped with 20GB of memory. With the trained re- ward model, we subsequently fine-tuned the latest diffusion-basee image inpainting model, namely Runway 622. four 40GB NVIDIA GeForce RTX A6000 GPUs as our PrefPaint. During fine-tuning, we employed half -aef-precision computations with learning rate of 2e - 6, and a batch size of 16.
Table 4: Performance under various param- eterizations of amplification factor y where f || (The underlined settings are se- lected.)
Amplification factor (r)
WinRate f Rewand
e -+f+b
0.050
0.70
71.27%
0.37
-5444
0.065
0.90
70.47%
0.36
-f + b / + b2
0.100
(0.10,
0.85
70.07%
0.37
5.1 Results of Image Inpainting
1J + b1/ / + b2
0.120
(0.80,
0.85|
69.95%
0.36
We compared our PrefPaint with SOTA methods both quantitatively and qualitatively to demonstrate the advantage of our method. We compared the WinRate and Reward under various sampling steps in Table and Fig. 4 where it can be seen that method actually greatly improves the probability of high high-quality numan-preferm samples, since with a single inference step, our method achieves more than a 70% WinRate, which is similar to the baseline model (Runway) with S 3. The higher reward value and lower variance also indicate that our method can consistently generate high-quality samples. Comparisons with other metrics in further support the superiority of the proposed method. In addition, we visually compared the
kf +
1.900
0.06
60.28%
0,28
1.43
65.95%
0.34
Table 2 inpainted images by different methods in Fig. 5] where it can be seen that our method can generate more meaningful and reasonable content, which is consistent with the style of the prompt region. We refer readers to the Supplementary Material for more details about more comparison results.
Finally, we have also carried out user study to evaluate our superiority. We have randomly selected about 130 groups of results and conducted a user study involving 10 users. The WinRate map. as depicted in Fig. IT demonstrates that our reward scoring is highly accurate and capable of assessing the results of inpainting under criteria based on human preference. We also present some examples scored by our reward model in Fig. ( illustrate the scoring criteria system.
5.2 Ablation Study
Reward Trustiness-Awant Scheme. We validated the advantages of our reward trustiness-awan training scheme, As shown on the left side of Table b) and e), we can see that this scheme improves the reward by 1.7% and accelerates the algorithm efficiency of 106.40%. Although constant amplification, such as c) and d) in Table 3 can expedite the training process even faster than the proposed method, it compromises model accuracy, evidenced by the reduced WinRate and Reward. This demonstrates the necessity and efficacy of adaptively managing the sampled trajectory and underscores the superiority of the proposed method.
Training Manner of Reward Model. We investigated two types of strategies to train the reward model R(-) i..., innfficaationdriveer and regression-driven. Specifically, the former classifies the
8

---

Figure 6: Results of image FOV enlargement by method on (a) and (b), where the prompt region (the given image) is delineated by the central area between white dashed lines.
Prompt
[b]
Giver-
[d]
Prompt
Idl Giveguaer
(d) Result
Frompt
(o/ihven View
(et Result
Prompt
M Gleen View
Result
Figure 7: Novel view synthesis on KITTI dataset of 6 scenes from (a) to (f). For each scene, we give the "Prompt", which is warped from the "Given View", with the white regions referring to holes/missing regions. "Result" is our in in-painting result from "Prompt". Note that for the synthesized novel view, there is ground-truth available.
discrete scores of annotated reward samples, while the latter directly makes a regression on the scores. As listed the right side of Table 3. it can be that the regression-driven training generally outperforms the classification-driis regression-drivee -rrivi training show the strong discriminative ability. Based accuracy and boundary performance, we finally selected fixed rate of 0.7 and MLP-256 with the regression-driven training manner as the configuration of our reward model.
one on reward accuracy. Moreover, the larger variances from the
Parameterize amplification factor 7. To parameterize the amplification factor in Eq. 10 we investigated various functions to parameterize shown in Table 4 The experimental results indicate that the exponential function provides the best regularization effect. contrast, the linear function and static constant do not fully exploit the regularization effeet of the reward upper boundary.
5.3 Further analysis
Reward errors distribution. We make statistics of reward estimation errors, and the results are shown in Fig. g Although the proportion of very large error samples is not large, the incremental performance of our method lies in more suitable choice of amplification function, evidenced by Table
Necessity of Our dataset. Although existing metrics such as BERT Score 777 provide general measure of quality, we emphasize that our dataset is specifically tailored for the task of image inpainting, where human-labeled scores are both essential and more precise. To substantiate this claim, we conduct a comparison between the BERT score and our dataset's score, as illustrated in Fig. T0 The results reveal significant divergence between human judgments and BERT's preferences, underscoring the necessity and superior accuracy the proposed dataset for this specific task.

---

Score: 0,4332
Score: J,3227
Score: 0.3155
Score: 0,0148
Score: 0:2021
Soore: O 6277
Score: 0.7623
Score: - 0.6512
Soore: -0 0.6284
Spore: 0.5201
Score: - 0,5984
Score: 1.71
Score: 0.1048
Score: 0.7073
Figure 8: Visualization of various image in-painting results and associated rewards from our model. Our model effectively evaluates in-painting reconstructions based on human preference.
ASL
ATP
940
825
60
sent
-05
QM
945
941
-20
300 8.20
6.50
6.5
1.09
1.0
125
1.75
Figure 9: Reward error distri-Fige Figure 10: Plot of GT reward by our Figure 11: WinRate heat-map butions of the proposed reward dataset (x-axis) and the Bert Score user study. The winrate at model. The distribution of re- re-(y-axis) on the validation set, where specific locations shows the ra- ward error percentages is de- each point indicates a sample. The tio of superior top methods to picted the y-axis to the right. darker indacated the larger error. those at the bottom.
5.4 Applications of Image FOV Enlargement and Novel View Synthesis
We also applied approach two additional tasks: (1) image field of view (FOV) enlargement, where we iteratively extended the boundaries of typical image strip in the horizontal direction to create wider FOV image; and (2) novel view synthesis, where warped given image using the predicted depth image through the method in (75) to generate novel viewpoint and subsequently applied diffusion models fill the holes/missing regions of the Warped view. As depicted in Fig. 6 (a)-a oil painting and (b)-a realistic photography, our method yields natural and visually pleasing results that can be seamlessly integrated with the prompt regions. As illustrated in Figs. 7 our method can generate more reasonable novel views and visually appealing reconstruction. We refer readers the Supplementary Material for more details about more application visual demonstrations.
Conclusion and Discussion
We have presented PrefPaint, an innovative scheme that leverages the principles the linear bandit theorem align diffusion models for image inpainting with human preferences iteratively. By integrating human feedback directly into the training loop, we have established a dynamic framework that not only respects the subjective nature of visual aesthetics but also continuously adapts the evolving preferences of users. We conducted extensive experiments to demonstrate the necessity for alignment in the task of image inpainting and the significant superiority of PrefPaint both quantitatively and qualitatively. We believe that our method, along with the newly constructed dataset, has the potential bring significant benefits the development of visually driven AI applications.
PrefPaint aligns with a distribution that corresponds to the preferences a specific group. However, important to recognize that individual preferences for image styles vary. Therefore, after achieving alignment with the general preferences of group, is advisable to develop personalized rewards and corresponding reinforeed alignment model to ensure complete alignment with the preferences reward- free alignment process, as discussed [16),
of each user. Recently, the implementation has gained popularity. Consequently, exploring reward model-free training methods for alignment in diffusion-based models represents a promising avenue for future research. Furthermore, there is opportunity explore the potential applications of our in in-painting algorithm for additional 3D reconstruction tasks.
10

---

Acknowledgement
This work was supported in part by the National Natural Science Foundation of China Excellent Young Scientists Fund 62422118. 18, and in part by the Hong Kong Research Grants Council under Grant 11219324 and 1219422,, and in part by the Hong Kong University Grants Council under Grant UGC/FDS11/E02/22.
References
[1] Zili Yi, Qiang Tang. Shekoofch Azizi, Daesik Jang, and Zhan Xu. Contextual residual aggrega- tion for ultra high-resolution image inpainting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 7508-7517, 2020.
[2] Jingyuan Li, Ning Wang, Lefei Zhang, Bo Du, and Dacheng Tao. Recurrent feature reasoning for image inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7760-7768, 2020.
[3] Titas Anciukevičius, Zexiang Xu, Matthew Fisher, Paul Henderson, Hakan Bilen, Niloy Mitra, and Paul Guerrero. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12608-12618, 2023.
[4] David Svitov, Dmitrii Gudkov, Renat Bashirov, and Victor Lempitsky. Dinar: Diffusion inpainting of neural textures for one-shot human avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7062-7072, 2023.
[5] Efsun Karaca and M Alper Tunga. Interpolation-basel image inpainting in color images using high dimensional model representation. In 2016 24th European Signal Processing Conference (EUSIPCO), pages 2425-2429. IEEE, 2016.
[6] Pablo Arias, Vicent Caselles, Gabriele Facciolo, Vanel Lazcano, and Rida Sadek. Nonlocal variational models for inpainting and interpolation. Mathematical Models and Methods in Applied Sciences, 22(supp02): 2012.
[7] Mashail Alsalamah and Saad Amin. Medical image inpainting with rbf interpolation technique. International Journal of Advanced Computer Science and Applications, 7(8), 2016.
[8] Xiefan Guo, Hongyu Yang, and Di Huang, Image inpainting via conditional texture and structure dual generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14134-14143 2021.
[9] Jitesh Jain, Yuqian Zhou, Ning Yu, and Humphrey Shi. Keys to better image inpainting: Structure and texture go hand in hand. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 208-2011111 108-217,20 2023.
[10] Avisck Lahiri, Arna Kumar Jain, Sanskar Agrawal, Pabitra Mitra, and Prabir Kumar Biswas. Prior guided gan based semantic inpainting. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 13696-13705, 2020.
[11] Xian Zhang, Xin Wang, Canghong Shi, Zhe Yan, Xiaojie Li, Bin Kong, Siwei Lyu, Bin Zhu, Jiancheng Lv, Youbing Yin, et al. De-gan: Domain embedded gan for high quality face image inpainting. Pattern Recognition, 124:108415, 2022.
[12] Shiyuan Yang, Xiaodong Chen, and Jing Liao. Uni-paine A unified framework for multimodal image inpainting with pretrained diffusion model. In Proceedings of the 31 ACM International Conference Multimedia, pages 3190-3199, 2023.
[13] Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 2096-2105, 2023.

---

[14] Prem Nandhini Satgunam, Russell L Woods, P Matthew Bronstad, and Eli Peli. Factors affecting enhanced video quality preferences. IEEE transactions on image processing. 22(12):5146-5157, 2013.
[15] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36, 2024.
[16] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
[17]
Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning summarize with human feedback. Advances in Neural Information Processing Systems, 33:3008-3021, 2020.
[18) Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, al. Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint urXiv:2307.15217, 2023.
[19] Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, et al. Nash learning from human feedback. arXiv preprini urXiv:23/2.00880 2023.
[20] Leslie Pack Kaelbling, Michael L Littman, and Andrew W Moore. Reinforcement learning: A survey. Journal of artificial intelligence research, 4:237-285, 1996.
[21] Kai Arulkumaran, Mare Peter Deisenroth, Miles Brundage, and Anil Anthony Bharath. Deep reinforcement learning: A brief survey. IEEE Signal Processing Magazine, 34(6):26-38, 2017.
[22] Vincent Trancois-Lavet, Peter Henderson, Riashat Islam, Marc G Bellemare, Joelle Pineau, al. An introduction to deep reinforcement learning. Foundations and Trends@ in Machine Learning, 11(3-4):219-355 2018
[23] Eugene A Feinberg and Adam Shwartz. Handbook of Markov decision processes: methods and applications, volume 40. Springer Science Business Media, 2012.
[24] Martijn Van Otterlo and Marco Wiering, Reinforcement learning and markov decision processes. In Reinforcement learning: State-of-the-art, pages 3-42. Springer, 2012.
[25] Ather Gattami, Qinbo Bai, and Vaneet Aggarwal. Reinforcement learning for constrained markov decision processes. International Conference on Artificial Intelligence and Statistics, pages 6656-2664, PMLR, 2021.
[26] Hao-nan Wang, Ning Liu, Yi-yun Zhang, Da-wei Feng, Feng Huang, Dong-sheng Li, and Yi-ming Zhang. Deep reinforcement learning: survey, Frontiers of Information Technology & Electronic Engineering, 21(12):1726-177 2020.
[27] Hao Yi Ong, Kevin Chavez, and Augustus Hong. Distributed deep q-learning. arXiv preprins wwXiv:1508.04188 2015.
[28] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv: 1312.523 2013.
[29] Songtao Lu, Kaiqing Zhang, Tianyi Chen, Tamer Başar, and Lior Horesh. Decentralized policy gradient descent ascent for safe multi-agent reinforcement learning. In Proceedings of the AAAI Conference Artificial Intelligence, volume 35, pages 8767-8775, 2021.
[30] Shauharda Khadka and Kagan Tumer. Evolution-guided policy gradient in reinforcement learning. Advances in Neural Information Processing Systems, 31, 2018.
[31] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
12

---

[32] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. actor-critic algorithm for sequence prediction. arXiv preprini arXiv: 1607.07086, 2016.
[33] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. In International conference on machine learning, pages 1587-1596.
PMIR, 2018.
[34] Tuomas Haarnoja, Aurick Zhou, Kristian Hartikainen, George Tucker, Sehoon Ha, Jie Tan, Vikash Kumar, Henry Zhu, Abhishek Gupta, Pieter Abbeel, et Soft actor-critic algorithms and applications. arXiv preprint arXiv:1812.05905, 2018.
[35] Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education. Learning and individual differences, 103:102274, 2023.
[36] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. Al Open, 2023.
[37] Lars Ruthotto and Eldad Haber. An introduction to deep generative modeling. GAMM- Mitteilungen, 44()):2021000088, 2021.
[38] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-683 6851, 2020.
[39] Jascha Sohl-Dickstedl Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference machine learning, pages 2256-2263 PMLR, 2015.
[40] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[41] Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPH 2022 Conference Proceedings, pages 1-10, 2022.
[42] Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02300 2022.
[43] Jinhui Hou, Zhiyu Zhu, Junhui Hou, Hui Liu, Huanqiang Zeng, and Hui Yuan. Global structure- aware diffusion process for low-light image enhancement. Advances in Neural Information Processing Systems. 36, 2024.
[44] Meng You, Zhiyu Zhu, Hui Liu, and Junhui Hou. Nvs-solver: Video diffusion model as zero-shot novel view synthesizer. arXiv preprint vrXi:22405.15364, 2024.
[45] Yang Song. Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. 2023.
[46] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 2020.
[47] Andreas Lugmayr, Martin Danelljan, Andres Romero, Fisher Yu, Radu Timofte, and Luc Van Gool. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 1 11461-11471, 2022.
[48] Ciprian Cornean,, Raghudeep Gadde, and Aleix M Martinez. Latentpaint: Image inpainting in latent space with diffusion models. Proceedings of the IEEF/CVE Winter Conference on Applications of Computer Vision, pages 4334-4343, 2024.
13

---

[49] Michal Stypuikowski, Konstantinos Vougioukas, Sen He, Maciej Zięba, Stavros Petridis, and Maja Pantic. Diffused heads: Diffusion models beat gans on talking-face generation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 5091-5100, 2024.
[50] Gustav Müllle-Frrazzs, Jan Moritz Niehues, Firas Khader, Soroosh Tayebi Arasteh, Christoph Haarburger, Christiane Kuhl, Tianci Wang, Tianyu Han, Sven Nehelung, Jakob Nikolas Kather, et al. Diffusion probabilistic models beat gans on medical images. arXiv preprint arXiv:2212.07501. 2022.
[51] Sham M Kakade. A natural policy gradient. Advances in neural information processing systems, 14, 2001.
[52] John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMI.R, 2015.
[53] Yinan Zhang, Eric Tzeng, Yilun Du, and Dmitry Kislyuk. Large-scale reinforcement learning for diffusion models. arXiv preprint arXiv:2401. /2244, 2024.
[54] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36, 2024.
[55] Yasin Abbasi Yadkori, Dávid Pál, and Csaba Szepesvári. Improved algorithms for linear stochastic bandits. Advances neural information processing systems, 24, 2011.
[56] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 633-641, 2017.
[57] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 127:302-321, 2019.
[58] Jia Deng, Wei Dong. Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pages 248 255. Ieee, 2009.
[59] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In Conference Computer Vision and Pattern Recognition (CVPR), 2012.
[60] Eirikur Agustsson and Radu Timofte. Ntire 2017 challenge on single image super-resolution: Dataset and study. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, July 2017.
[61] Radu Timofte, Shuhang Gu, Jiqing Wu, Luc Van Gool, Lei Zhang, Ming-Hsuan Yang. Muham- mad Haris, et al. Ntire 2018 challenge on single image super-resolution: Methods and results. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2018.
[62] Runway, Runway model on huggingface, 2023. URL htlLi//pp////ggngiieeee////////
[63] Runway, Runway model on huggingface, 2023. URL http::::///uggggggcccc///aaaaaaa stable-diffuusion-ii55
[64] StabilityAl. Stabilityai model on huggingface, 2023. URL
co/
[65] StabilityAI. Stabilityai model on huggingface, 2023. URL htttps//huggingtee co/
S base-1.0
14

---

[66] Diffuser. Stabilityai model on huggingface, 2023. URL https: helses//hggnnggagg...o
wttp://tttteeddfffiioooo x1 10--npanntinn-O....
[67] Compvis. Compvis github website, 2023.
URL http::://gttuubcoo..
Sstent-diffusie
[68] Anton Razzhigaev, Arseniy Shakhmatov, Anastasia Maltseva, Vladimir Arkhipkin, Igor Pavlov, Iyy Ryabov, Angelina Kuts, Alexander Panchenko, Andrey Kuznetsov, and Denis Dimitrov. Kandinsky: an improved text-to-image synthesis with image prior and latent diffusion. arXiv preprint ar 2rXi:22310.035000 2023.
[69] Wenbo Li, Zhe Lin, Kun Zhou, Qi, Wang, and Jiaya Jia. Mat: Mask-aware transformer for large hole image inpainting. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10758-10763 2022.
[70] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 11488-763..2 PMLR, 2021.
[71] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language- image pre-training for unified ussionllanguag understanding and generation. In International conference machine learning, pages 12888-12900. PMLR, 2022.
[72] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278-25222 2022.
[73] Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 281 18-6228.,, 2016.
[74] Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelenn with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517. 2017.
[75] Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401. 10891, 2024.
[76] Jianfeng Xiang, Jiaolong Yang, Binbin Huang, and Xin Tong. 3d-aware image generation using 2d diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2383-2393, 2023.
[77] Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305.15 13301, 2023.
[78] Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. arXiv preprint arXiv:2312.03594 2023.
[79] Xuan Ju, Xian Liu, Xintao Wang. Yuxuan Bian, Ying Shan, and Qiang Xu. Brushnet: A plug-and-play image inpainting model with decomposed dual-branch diffusion. arXiv preprint arXiv:2403.0m 06976, 2024.
[80] Hayk Manukyan, Andranik Sargsyan, Barsegh Atanyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Hd-painter: high-resolution and frompt-faithfu text-guided image inpaint- ing with diffusion models. arXiv preprint arXiv:2312. /4091, 2023.
[81] Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36, 2024.
15

---

[82] Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge. Jiaxin Chen. Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8941-8951, 2024.
16

---

Appendix
Table of Contents
Dataset Details
A.1 Labeling Scheme
18
Labeling Platform
A.3 License
21
B
More Application Visual Demonstrations
22
Application of Image FOV Enlargement
22
B.2
Application of Novel View Synthesis
22
Reward Scoring Statistics Reward Normalization Configuration
Score Distribution across Different Categories
23
Normalization Factors
24
More Visualizations
24
D.1 More Visualizations Comparisons
24
D.2 More Visualizations of Multiple Sampling
27
E More experimental results
27
Comparisons with Additional inpainting Methods
27
Comparisons with other reinforcement learning methods,
28
Compared with ++
28
F Details & Reward
29
F. Training Curve
29
F2 Details between Eq. 4-5
30
17

---

A
Dataset Details
In this section, we visualize our labeling platform and the corresponding labeling scheme.
A.1 Labeling Scheme
The reward score contains the following terms:
Structural Rationality. measures the correctness of the model understanding of image content. Score (0-2): Failure generation, e.g. strange and weird objects; random patterns completed; inap- propriate strings; abrupt bar completions. Objects are generated but obviously do not belong to the scene, e.g. There shelves unique homes the side of the highway; there is marble floor blocking the front of the washing machine. Score (3-5):Thhe understanding of the scene roughly correct, e.g. the inpainted sky next the sky pattern; the behind the beach. Score (6-7): The overall structure of the generated scene is reasonable and do not seem to be any major problems. To illustrate the scoring scheme, we show some examples and corresponding reasons in Table A-I
Table A-1: Illustration of different scoring examples structural rationality.
Prompt with Reconstruction
Prompt with Reconstruction
Score: O. Reason: there should be no mar- ble in front of the washing machine. Wrong
Score: 1. Reason: the appearance of racks
on the highway is unreasonable, semantically incorrect, and misunderstanding of the scene.
semantics and wrong understanding of the
scene.
KECOM. th
Reason: the model understands that this is volleyball match and try to com- plete relevant people.
Score:
Score: Reason: Although the quality of the details generated here is not rated high, the model understands that the scene on the beach, and the overall structure is rated.
Score: 6. Reason: although there are some flaws, understand the content of the highway scene and complete the relevant content.
Score: Reason: Understand the scenario and correctly add relevant content.
Feeling Local Texture. It measures whether the texture the object correct and consistent with objective facts. Score (0-2): Failure generation (e.g. strange and weird objects; random patterns
18

---

inappropriate strings; abrupt bar completions). The generated objects do not conform to objective facts (e.g. dog has two heads; person without an upper body). The texture of the generated objects is very poor (e.g. two car bodies stuck together; poor quality sky). Score (3-5): The generated objects are generally reasonable (e.g. complete leopard, complete corn, but the corn kernels are strange, there are flaws in the complete object;). The generated texture not obtrusive but will have flaws if you zoom in. Score (6-7): Details are complete and reasonable. We also show some examples and corresponding reason in Table A-2
completed;
with very long legs;
Table A-2: Illustration of different scoring examples the feeling of local texture.
Prompt with Reconstruction
Prompt with Reconstruction
k ch
Score: 0. Reason: it can be seen that the model understands the scene correctly and wants to add dog, but the quality is not high and does not conform to the objective facts.
Score: 1. Reason: it wants to generate people
the generation quality is not high.
BRAHIA
BRAHHAN
ilu
Gailmn
Score: 4. Reason: The completion is gener- ally reasonable.
Score: 4. Reason: The completion is gener-
ally reasonable.
Score: 6. Reason: wants to patch up the leopard but the legs is not consistent with common sense, but can also be seen that is complete leopard.
Score: 7. Reason: The completion quality is good. good.
Overall Feeling, The overall feeling given by the picture, whether it is reasonable and consistent with the objective facts. We directly show some examples and corresponding reasons in Table A-3
A.2 Labeling Platform
The visual demonstration labeling platform is shown in Fig. A-1 right-side, we aggregate three reconstruction results with the prompt images. The labeling annotated in the left-side, where for each image, people need label scores by aforementioned 3 different criteria, ranging from to 7.
19

---

Table A-3: Illustration of different scoring examples on the overall feeling. Prompt with Reconstruction (from left to right are Recon-1, 2, and the prompt )
Score: 3, 2, Reason: The quality of Recon-3 is significantly better than Recon- 1, and there are no obvious flaws, so it has the highest score of 6.
Score: 5, Reason: Recon-1 is the most reasonable, but the steel basin structure is bit strange. Recon-2 may want to make up for the melon seeds. The details are quite good but the scenes relatively rare. Recon-3 is very strange.
Score: 1, 1, 4. Reason: The overall structure of Recon-1 is wrong, and the quality of the corn heads Recon-2 is relatively poor, and it looks like there is glue stuck on it. Recon-3 has reasonable structure, but the proportion of the head is relatively large, and the grains of the corn head are strange, so the score only 4 not very high.
Score: 7, Reason: The shape the car on the right side of Recon-1 too weird and affects the look and feel. Recon-2 is of very good quality and very natural. The house on the right side of Recon-3 has poorer perspective.
Score: O, 4, 1. Reason: Recon-1 was randomly generated, Recon-3 tried to generate a traffic light, but failed. Similar branches in Recon-2 don't look inconsistent.
20

---

04 01
Figure A-1: Labeling platform demonstration. In each group, from top left bottom right, reconstruction 1, 2, 3 and the prompt image.
A.3 License
The license of the proposed dataset is CC BY-NC 4.0, which allows creators to share their work with others while retaining certain rights but gives a restriction on commercial use.
21

---

B More Application Visual Demonstrations
To further verify the effectiveness of the proposed method, this section further visualizes more scenes for the application of the proposed method (as demonstrated in Sec 5.4).
B.1 Application of Image FOV Enlargement
For the task of Image FOV Enlargement, the results shown in Fig. [-2 picture from Internet and apply center crop on those pictures. We can see that no matter the style of prompt image, c-g., nature photography, oil painting or Chinese painting, the proposed method can consistently generate meaningful and visually pleasing results, which demonstrates the superority of the proposed method.
We randomly select some
Dutperding
Outpairting
(d Durpaining
Detpaining
Outpolrting
Outpaliiing
Promut
Peceept
Pecrept
Figure B-2: Application FOV enlargement, where visualize 9 scenes from (a) to (i) with corresponding prompt cropped image and enlarged result.
B.2 Application of Novel View Synthesis
We also visualize more novel view synthesis examples on the KITTI and DIV2K datasets in Figs. B-3 induced inpainting hole is much more irregular than the FOV Enlargement, which makes the task more challenging. However, the proposed method successfully filled up the missing region. The resulting high-quality reconstructions verify the superiority of the proposed method.
B-4] The warping- induued
22

---

Prompt
de) Given View
140 Result
Bb) Prompt
Ib Given View
[b) Result
View
Result
Viven
Result
Prompt
tc Glvcc
View
tel Result
+9 Frompt
Given View
|f| Result
Prompt
Given
AEl Result
Ch Prompt
The Givan Vi
(b) Result
Figure B-3: Application of novel view synthesis on KITTI dataset, where we visualize scenes from (a) to (h) with corresponding prompt warped image, given view and reconstruction result.
Prommit
Given View
EB Promptt
Drew
IA Rook
FOO Pop
Given View
(ni Prompt
Inj Srven View
Ranuih
Promppt
IP Gliven View
10 Rebslt
Drampt
Grean View
Rusult
Thi Given View
thi Risoki
wane
Figure B-4: Application of novel view synthesis on DIV2K dataset, where we visualize 8 scenes from (a) to (h) with corresponding prompt warped image, given view and reconstruction result.
C Reward Scoring Statistics & Reward Normalization Configuration
C.1 Score Distribution across Different Categories
We count the ratio of prompt with different semantic class in Fig. C5
Most methods, including SOTA diffusion-based methods, e.g., stable diffusion or Runway, have large ratio samples with negative rewards. However, the proposed method achieves the most positive rewards, as shown in the last figure, which validates the necessity and effectiveness of applying such an alignment task the diffusion in in-painting model.
23

---

282
200
imoges
Mamber of Inwages
X30
120
320
100
0.33
0.25
2.00
330
250
200
120
150
220
100
$1.00
Reward
Rowaadd Scores
Rowardd Scores
Pulette
Runmay (Base)
Ours
132
300
200
392
Nimberaa of
200
120
100
555 -0.8 -0.8
-0.6
-0.3
+0.75 -0.00 -233
0.3
6.mh
-0.4
0.2
00
10
Roward Scores
Reward Scores
Reward SCOrES
Figure C-5: Detailed score statistics of the proposed dataset.
C.2 Normalization Factors
To facilitate the training of reward model, we normalize the score with s-mean where 8 is raw score data. To remove the data bias for training the reward model, the normalization factors calculated within different datasets and inpainting patterns. We show the resulting factors in Table C44 The relatively balanced factors indicate that our labels do have large biases on different datasets or inpainting pattern.
ADE20K Warping Outpainting Warping Outpainting Warping Outpainting
KITTI
ImageNet
Div2K Warping Outpainting
Dataset
Mean
3.46
3.12
3.02
2.87
2.85
2.50
2.99
2.34
Variance
2.77
4.42
3.04
2.69
3.03
3.08
3.26
3.60
Table C-4: The normalization factor of score to facilitate for reward model training.
D More Visualizations
D.1 More Visualizations of Comparisons
In this section, we visually compare the reconstruction results of different methods. As shown in that
Figs. [D-6] and D-7 The comparison of different methods in Figs. D-6 and proposed method could generate high-quality reconstruction compared with other SOTA methods. Notably, the results from our approach show improved clarity and detail retention, which are essential for practical applications.
24

---

Prompts
Compsis
Kandinsky
Pulette
Runway
SD x
SD v2.1
Ours
Figure D-6: Qualitative comparison between the proposed method and other SOTA methods.
25

---

Prompts
Compvis
Kandinsky
Pulette
Runway
SD x
SD v2.1
Ours
Figure D-7: Qualitative comparison between the proposed method and other SOTA methods.
26

---

D.2 More Visualizations of Multiple Sampling
To verify the robustness and consistency of the inpainting model after alignment with our method. We run 5 times with different random seeds on the same prompt to derive different reconstruction results. The results shown in Fig. d-edeonnstettee the stability of the proposed method. We can see the proposed method generate high-quality and visually pleasing results under different conditions.
Figure D-8: Qualitative comparison of different sampling times of the proposed method.
E More experimental results
E.1
Comparisons with Additional inpainting Methods
We have experimentally compared with some inpainting Methods, i.e., PowerPaint (78. Brush-
Net [79] and Hdpaint [80] For all these methods, we assessed performance using their publicly released models. As shown in the Table E-1 and Table E-2 our method significantly outperforms all
27

---

the compared methods, particularly in perceptual metrics such as T2I and our Reward score. Besides, BrushNet employs roughly twice the number of parameters as our model, and HdPaint requires 15 times more computational time (experimented on NVIDIA GeForce RTX A6000 GPUs), making it impractical for real-world applications.
Table E-1: Comparison across metrics: higher values are better for all metrics except "# Param".
Metrics
T21
BLIP
CLIP
CA(Incep.)
Reward
# Param(M) Infer. Time(s)
PowerPaint(v-1)
4.44
0.46
0.21
0.42
-0.057
819.72
5
PowerPain((--ruuhNNer
-3.84
0.46
0.20
0.42
-0.036
1409.88
16
BrushNet
1.26
0.46
0.22
0.43
0.137
1409.86
16
HdPaint
-4.57
0.47
0.21
0.44
-0.059
451.47
60
PrefPaint(OUrs)
11.60
0.49
0.23
0.45
0.374
819.72
5
Table E-2: Quantitative comparisons of different methods. "S" the number of sampling times. For the calculation of WinRate, we first derive the best sample of the compared method among $ sampling times.
WinRatef (v.s. Runway)
S=1
S=2
S=3
PowerPaint(y-1)
27.06%
39.92%
47.38%
CiverPain((-BrushNNee
29.86%
43.12%
52.01%
BrushNet
49.49%
62.83%
69.22%
HdPaintdss--inp)
33.37%
43.41
49.03%
PrefPaint(Ours)
71.27 %
85.88%
93.50%
E.2
Comparisons with other reinforcement learning methods.
we have experimentally compared our method with some other reinforcement learning methods in
Table E-3
Specifically, simply summarize the implementation of each method. Human Preference
Score 3 learns negative prompt to map the diffusion process to low-quality samples. Then, the inference process, the negative sample is utilized in the classifier- freeerffeeeeeee guidance (CFG) push the generation trajectory away from lua-quality samples. ImageReward [54] trains a reward model and then applies the reward model
loss metric to end-to-end optimize the diffusion model accompanied by a reconstruction loss. We also conduct the ablation study on reward training strategy in Table 3 in our paper. Our method employs regression-drive a classification-drive strategy. DPOK [81] simultaneously optimizes the whole trajectory of a reverse diffusion process and utilizes the KI. divergence panel the regularization, avoiding large distribution shift. D3PO (82) adopts the RL strategy from direct performance optimization (DPOK), directly optimizes the model the reward labeling data minimize the probability of low-quality samples and increase the probability of high-quality samples.
training strategy, while ReageRewarr
E.3
Compared with SD xl ++
Currently, their also a kind of diffusion-inpaintiog network like SD xl ++ [66), which requires complete image to redraw the mask region and can generate a visually pleasing result. However, their performance would be dramatically reduced without the complete image. To validate the characteristics of SD xl ++, carry experiments to examine its performance.
28

---

Table E-3: Quantity evaluations of different reinforcement learning methods.
Methods
WinRate
T21
Reward
CLIP
BLIP
CA
Human Preference Score
58.03%
16.67
0.26
0.20
0.47
0.40
ImageReward
65.10%
13.12
0.29
0.22
0.48
0.44
DPOK (KL weight=0.1)
64.59%
11.43
0.32
0.21
0.48
0.43
DPOK (KL weight=1.0)
62.67%
9.36
0.30
0.21
0.48
0.43
D3PO
59.74%
19.20
0.26
0.21
0.46
0.41
PrefPaint
71.27 %
21.53
0.37
0.23
0.49
0.45
Table E-4: Alignment evaluation across various generative methods, where represents the small model (non SD-based); "S" refers to the number of sampling times. For comparing the WinRate, the best inpainting results are selected through multiple sampling and compared with the Runway Inpainting model with only sampling. "* (resp. J)" means the larger (resp. smaller), the better. The WinRate calculated against Runway 662
Prompt Methods
Outpainting Prompts
Warping Prompts
WinRate (%) f
Rewards Meant
WinRate (%)
Rewards
Metrics
Meant
SD v1.5 w/ +
21.15
-0.13
18.66
-0.18
SD v2.1 w/ xl
26.07
-0.07
22.26
-0.12
Kandinsky w/ xl +
17.09
-0.29
12.04
-0.37
MAT w/ xl ++
19.12
-0.34
8.58
-0.56
Palette * w/ ++
27.99
-0.07
25.82
-0.09
Runway w/ xl ++
56.94
+0.03
45.71
+0.02
Ours
70.16
+0.38
72.38
+0.36
Details & Reward
F.1 Training Curve
We present training curves various methods for comparison. The curves for '1.4BaseLine' and 1.4Boundary' (Ours) demonstrate our modification's acceleration, reaching 0.35 rewards first. Our approach converges with the fewest training iterations.
0.5
0.4
Mean Rewards
0.3
0.2
BaseLine
(WR=69.57%)
0.1
1. 4BaeeLne (WR=65.95%)
..oundarryyy (WR=71.27%)
0
1000
2000
3000
4000
5000
Steps
Figure F-1: Training curves different experimental setups.
29

---

F.2 Details between Eq. 4-5
According to the [55], we derive a similar process for the linear reward regression on feature space, where Z denotes the latent of an image.
S -ZZ
(13)
+
(14)
XI - + +7"c] = - + +Z" + -11
(15)
(16)
3 レ-
x- "b.
(17)
30

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The authors correctly summarized the contribution in the abstract and intro- duction sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
The authors have discussed the limitations in the section of the conclusion and discussion.
Justification:
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
31

---

Answer: [NA]
Justification:
No theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We have detailedly illustrated our implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
32

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have given the example data and code for supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (https
/ "pippscc/puullc//uuu /ttee//ggudddeeeedSuuuummbbbbi ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We have given the implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: N.A.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
33

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the computational resources.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follows the instructions for preparing the paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: N.A.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
34

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: N.A.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the utilized papers.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
35

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We have discussed the new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We have given the instructions in the appendix.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: N.A.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
36


Reviews:
[{"review_id": "589LnKTTcZ", "reviewer": "Reviewer_85gD", "summary": "This paper attempt to align diffusion models for image inpainting with human aesthetic standards through reinforcement learning framework. To train the model, this paper construct a dataset containing 51,000 inpainted images annotated with human preferences. Extensive experiments on inpainting comparison and downstream tasks, such as image extension and 3D reconstruction, is provided in this paper.", "questions": "See weekness", "limitations": "The authore have include limitation in the paper, but I would recommend the authore discuss more on the limitation and negative societal impact.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well presented and easy to read.\n2. This paper provids detailed experiments to validate the effectiveness of the proposed method.\n3. This paper have collect a small scale dataset for human preference of image inpainting results, which can be useful for the ressearch field.", "weaknesses": "1. This paper seems to use a common reinforcement learning pratice on text-to-image diffusion model. What is the novelty of the proposed method? Why would this practice better than simply finetuning the model on high-quality (human prefered) inpainting data?\n\n2. There are missing citations and comparisons with the following methods:\n   - A Task is Worth One Word: Learning with Task Prompts for High-Quality Versatile Image Inpainting\n   - BrushNet: A Plug-and-Play Image Inpainting Model with Decomposed Dual-Branch Diffusion\n   - Hd-painter: High-resolution and prompt-faithful text-guided image inpainting with diffusion models\n\n3. Can you provide an ablation of how the training data quantity influence final results? The training dataset only contain 17,000 images and 51,000 inpainted samples, which is quite small in diffusion model training.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer 85gD\n\n### Q1. Difference between the Proposed Method, Common Reinforcement Learning, and Supervised Fine-tuning\n\nIn the following, We first clarify that our method is **NOT** a straightforward practice of the common reinforcement learning on diffusion model alignment. Then, we address your **confusion** about supervised fine-tuning (i.e., directly tuning on high-quality data) and reinforcement learning-based tuning. \n\n(1) **Not the same task as text alignment RL task.** While some related works exist in the area of text-to-image tasks involving human preference, our work is the first to introduce reinforcement learning (RL)-based alignment into the task of image inpainting. Initially, our proposal of the human preference inpainting dataset enabled the feasibility of this task. Without such a dataset that includes high-quality labels based on human preference, undertaking inpainting alignment would be impossible. Note that our dataset is annotated by professional data annotation company.  The **technical novelty** of the proposed method, compared to conventional RL, primarily lies in our modeling of reward accuracy and the adaptive control of reward regularization strength using the reward error upper bound, as detailed in Secs. 3.2 and 3.3 of our paper. \n\n\n(2) **Supervised fine-tuning cannot  address our task.**  We first clarify some experimental results that our baseline model (the pretrained diffusion model) has already trained on plenty of GT samples for image inpainting, which is even better than our high-preference samples. However, the limited performance of the pretrained model, with comparison of Runway and Ours in Tables 1 and 2 of our paper, indicates the supervised fine-tuning model is hard to accurately and effectively modifying model generation on the perspective of preference alignment.  \n\nMoreover, to further address your concerns, we have experimentally validated the neural network performance using fine-tuning based methods.\n\n| No.  | Method|Train Dataset| Val Dataset|T2I| BLIP|CLIP|CA| Reward|\n|----|----|----|----|----|----|----|----|----|\n| (a)  | Original (Best Samples) | -- | Training Prompt | 3.61 | **0.49** | 0.22   | 0.45     | 0.19     |\n| (b)  | FineTuning-Model | Training Prompt | Training Prompt | -4.89 | **0.49** | 0.22     | 0.44     | 0.12     |\n| (c)  | FineTuning-Model | Training Prompt | Val Prompt| -13.61    | 0.48     | 0.21     | 0.40     | 0.02     |\n| (d)  | RL-Model(Ours) | Training Prompt | Val Prompt  | **14.90** | **0.49** | **0.23** | **0.45** | **0.37** |\n\nwhere (a) signifies the high-quality subset generated from the original samples and selected by our reward labels; (b) represents the reconstructed images by the fine-tuned model; (c) corresponds to the fine-tuned model's performance on the validation dataset; (d) indicates the generated samples by the RL-based model evaluated on the validation dataset. We observe that the fine-tuned model performs not well on the training set and shows even more undesirable performance on the validation set. In contrast, the proposed method accurately aligns with human preferences.\n\n### Q2. Comparisons with Additional inpainting Methods\n\nWe have experimentally compared with your mentioned papers. For all these methods, we assessed performance using their publicly released models. We utilized our test dataset to comprehensively evaluate their performance. As shown in the table below, our method significantly outperforms all the compared methods. \n\n| Metrics (the larger the better) | T2I       | BLIP     | CLIP     | CA(Incep.) | Reward    | \\# Param(M) | Infer. Time(s) |\n| ------- | ---- | ------ | ------ | ----- | ----- | ------- | -------- |\n| PowerPaint(v-1)  | -4.44     | 0.46     | 0.21     | 0.42       | -0.057    | 819.72      | **5**  |\n| PowerPaint(v-BrushNet)  | -3.84     | 0.46     | 0.20     | 0.42       | -0.036    | 1409.88     | 16  |\n| BrushNet(realistic-V15VAE) | 1.26      | 0.46     | 0.22     | 0.43       | 0.137     | 1409.86   | 16  |\n| HdPaint(ds8)  | -4.57     | 0.47     | 0.21     | 0.44       | -0.059    | **451.47**  | 60  |\n| **PrefPaint(Ours)** | **11.60** | **0.49** | **0.23** | **0.45**   | **0.374** | 819.72  | **5**  |\n\nMoreover, we also tested the Winrate compared with our baseline model (Runway). Our method also greatly surpass all compared methods, which validates the effectiveness of the proposed RL-based alignment scheme.\n\n| WinRate (v.s. BaseModel) (the larger the better) | S=1         | S=2         | S=3         |\n| ----- | ----| ------ | ----- |\n| PowerPaint(v-1)[ECCV2024]                       | 27.06\\%     | 39.92\\%     | 47.38\\%     |\n| PowerPaint(v-BrushNet)[ECCV2024]                | 29.86\\%     | 43.12\\%     | 52.01\\%     |\n| BrushNet(realistic-V15VAE)[ECCV2024] | 49.49\\%     | 62.83\\%     | 69.22\\%     |\n| HdPaint(ds8)[Arxiv2023]                     | 33.37\\%     | 43.41\\%     | 49.03\\%     |\n| **PrefPaint(Ours)**                             | **71.27\\%** | **85.88\\%** | **93.50\\%** |\n\n### Q3. Size of Training Dataset\n\nWe believe you have a **misunderstanding** here. We clarify that the dataset is **primarily used to train the reward model rather than the diffusion model**. With an accurate reward model, we can easily train diffusion models on scaled datasets by allocating more prompts. We empirically validate that the dataset size is sufficient for training a reward model. Specifically, as shown below, we train the reward model with 10K, 20K, 30K, and 50K data samples, respectively. \n\n| Size              | 1W     | 2W     | 3W     | 5W     |\n| ----- | ----- | ----- | ----- | ---- |\n| Reward Accuracy ↑ | 72.5\\% | 74.0\\% | 75.3\\% | 75.9\\% |\n\nWe observe that the accuracy of the reward model becomes saturated as the size of the training dataset increases. Therefore, we believe that 50K data samples are sufficient for training an accurate reward model. Additionally,  the proposed method has already outperformed SOTA, validating its effectiveness.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FTofIBbfUx", "reviewer": "Reviewer_yfx6", "summary": "This paper is the first to use reinforcement learning in diffusion-based image synthesis. This significantly improves the quality since image synthesis is usually a one-to-many mapping, which may not be suitable for conventional learning methods. To generate reward functions for RL, this paper also gathers and releases a new dataset on image synthesis with human preference annotation. Additionally, the paper computes the theoretical upper bound on the error of the reward for more efficient RL training.", "questions": "1. In L152, how often do these \"relatively large errors\" occur? Will the relatively smaller errors balance out these large errors? Since the amplification $\\gamma$ greatly improves the performance, does this suggest that the errors occur frequently so that the static $\\gamma$ do not handle these very well? From Fig.2, it is shown that the range of $\\| z \\|_{v^{-1}}$ is [0, 1]. Does amplification $\\gamma$ for $k=0.05$ and $b=0.7$ have a range of [1.9, 2.0] (which seems narrow and thus static)?\n\n2. How does this paper compare to [1]? [1] leverage the BERT model for reward scoring. How do you justify the choice of human evaluation instead of foundation models for reward function? Do you have ablation experiments to backup the decision?\n[1] Black, Kevin, et al. \"Training diffusion models with reinforcement learning.\" arXiv preprint arXiv:2305.13301 (2023).", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. Release a new dataset for image inpainting and outpainting benchmarks with human preference annotation. The generation process of the dataset is well-documented.\n2. First incorporate RL with diffusion-based method on image inpainting and outpainting.\n3. The authors provide extensive experiments, supplementary materials with results on multiple datasets, and a project page to demonstrate their results.", "weaknesses": "1. Minor question on the motivation of amplification factor.\n2. Possible missing references.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer yfx6\n\n### Q1. Error of Reward Model \\& Amplification Factor\n\nWe make statistics of reward estimation errors, and the results are shown in Fig. **S1** of the uploaded one-page PDF file. Here we make a table to briefly show the results.  Although the proportion of very large error samples is not large, the incremental performance of our method may lie in a more suitable choice of amplification function, as evidenced by the table in our response to the Q2 from the reviewer **Hftn**.\n\nSorry for causing the confusion of the reward range. Actually, the x-axis of Fig. 2 of the manuscript is normalized. The range of `$||z||V^{-1}$` lies approximately between 1.5 and 14. Thus, the range of the amplification factor is approximately between exp(0)=1 and exp(0.625)=1.86.\n\n| Reward Error | [0, 0.25) |  [0.25, 0.5) |  [0.5, 0.75) |  [1.0, 1.25) |  [1.25, 1.5) |  [1.5, 1.75) |  [1.75, +inf) |\n| ------------ | --------- | ----------- | ----------- | ----------- | ----------- | ----------- | ------------ |\n| Percentage   |   43.99%    |   30.57%      |   16.24%      |   6.53%       |   1.97%       |   0.59%       |   0.06%        |\n\n### Q2. Reward Model\n\nWe have experimentally validated the consistency between the proposed method and the BERT score [1]. As shown in the Fig. **S2** in the uploaded one-page PDF file, the deeper color indicates the larger errors in the BERT score. We can see that there are plenty of samples whose BERT scores are **unrelated** to human labeling even contradictory, shown as the upper left or lower right regions. We also provide some examples scoring by our reward model in Fig **S4** in the uploaded one-page PDF file, the assessment of inpainted images is closely aligned with human preference.  Thus, our proposed human-labeled dataset **is necessary** for the alignment of inpainting tasks. Note that our dataset is labeled by professional data annotation company. We will add more discussion with [1] in the final version. Thanks for the advice.\n\n[1] Black, Kevin, et al. \"Training diffusion models with reinforcement learning.\" arXiv preprint arXiv:2305.13301 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LCE3sGDpNV", "reviewer": "Reviewer_Hftn", "summary": "This paper makes the first attempt to align diffusion models for image inpainting with human preferences by integrating human feedback through reinforcement learning. The authors theoretically deduce the accuracy bound of the reward model, modulating the refinement process of the diffusion model to robustly improve both efficacy and efficiency. Additionally, they construct a dataset containing 51,000 inpainted images annotated with human preferences.", "questions": "Please refer to the weaknesses.", "limitations": "The authors provide the future work in the last section.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. This paper provides theoretical insights into the upper bound of the error of the reward model, ensuring the reliability and accuracy of the reinforcement learning process.\n2. This paper presents a dataset for image inpainting tasks that incorporates human aesthetic preferences. This dataset will facilitate further research into the evaluation of image inpainting models and aid in generating high-quality images that better align with human aesthetics.", "weaknesses": "1. Since reward models and feedback learning trained based on human preferences have been introduced in Text-to-Image Generation (T2I), it seems they are simply applying the same process from T2I to Image Inpainting. This work should clarify the distinctions from T2I to highlight its innovation.\n2. The core contribution of this paper is Equation 11; however, its rationale, implementation details, and effectiveness lack explanation and validation. First, the selection of hyperparameters $k$ and $b$ is not addressed, despite their importance. Second, there is a lack of detailed explanation from Equation 11 to its implementation in the model. Third, the choice of using $e$ instead of other functions needs to be validated. Finally, the description of the ablation study on $k$ in Table 3 is unclear, making it difficult to assess its effectiveness. There is also a lack of ablation analysis for the $b$ and $e$.\n3. The weights assigned to the three scores in the manuscript are [0.15, 0.15, 0.7]. Why are these scores combined into a weighted score instead of being used individually? Furthermore, how were these weights determined? The basis for this weighting is unclear.\n4. What does the \"Rank\" metric in Table 2 signify? It lacks explanation.\n5. There are details missing regarding the dataset annotation. Such as the number of annotators employed, whether training was required for them, and the amount of time each individual spent on the annotation task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer Hftn\n\n### Q1. Differences from T2I Methods\n\nWe confirm that our method is **NOT** simply applying the text-to-image alignment scheme to image inpainting.  The **technical novelty** of the proposed method primarily lies in modeling reward accuracy and adaptively controlling the regularization reward strength, which has not been investigated by T2I methods before. Moreover, the ablation studies presented in the right part of Table 3 in the main paper validate the superiority of our design, which can achieve a **+106\\%** speed acceleration while maintaining high performance (even increasing WinRate by up to **1.7\\%**). We will add further discussion comparing our method with current T2I methods. **More importantly**, we want to note that this is the **first** time to explore the diffusion model human preference alignment problem on the task of image inpainting. To enable the feasibility of this task, we built the **first dataset**, where inpainted images are labeled with human preference through a professional data annotation company. Without such a dataset, it is almost **impossible** to conduct this work. We believe it will advance this field. Our dataset holds significant research value and application scenarios in several related areas, including image quality assessment tasks and other tasks involving human preference. We also provide some examples scored by our reward model in Fig **S4** in the uploaded one-page PDF file, the assessment of inpainted images is closely aligned with human preference.\n\n### Q2. More Explanations of Eq. (11)\n\n**Selection of Hyper-parameters**. The critical thing for pasteurization is the region of the weight factor $\\gamma$. In the left of Table~3 (e) and (f) of our manuscript, we experimentally validate the two sets of $k=0.05,b=0.7$ and $k=0.065,b=0.9$. The experimental results show that our selection is better than the other settings. Moreover, please refer to our response to the third question, where we detailedly investigate the different hyper-parameters and weighting functions.\n\n**Implementation details**.  We first prepare the matrix $\\mathbf{V}^{-1}$ by calculating $\\mathbf{V} = \\mathbf{Z}^T\\mathbf{Z} + \\lambda \\mathbf{I}$, where $\\mathbf{Z}$ represents the concatenation of feature embeddings before the last MLP layer of the reward model, encompassing all training dataset samples. During the diffusion training process, for each sample, we obtain a feature $z$ from the reward model. We then calculate $\\gamma$, which serves as the weight factor for the final RL loss, adaptively adjusting the magnitude of the gradient. Thanks for the comments; we will include the corresponding content in the final version.\n\n**Other Functions \\& Ablation Study**. To further resolve your concern, we have applied another linear function to parameterize $\\gamma$ as table below. Specifically, $\\gamma =  -1.9*\\| \\boldsymbol{z}\\|_{\\mathbf{V}^{-1}} + 0.06$. **The experimental results indicate that the exponential function provides the best regularization effect. In contrast, the linear function and static constant do not fully exploit the regularization effect of the reward upper boundary.**\n\n| Function ($x = \\| \\boldsymbol{z}\\|_{\\mathbf{V}^{-1}}$)| Range |k | b | WinRate | Reward |\n|----|----|----| ---- | ----| ---- |\n| $\\gamma = e^{- kx + b}$(Ours) | [1.00, 1.87] | 0.05| 0.7| **71.27\\%** |**0.37**|\n| $\\gamma = e^{- kx + b}$  | [1.00, 2.23] | 0.065 | 0.9| 70.47\\% | 0.36|\n| $\\gamma = e^{- kx} +  b_1/x + b_2$ | [1.10, 1.78] |0.10| [0.1, 0.85]| 70.07\\% | 0.37|\n| $\\gamma = e^{- kx} + b_1/x + b_2$ | [1.10, 2.22] |0.12| [0.8, 0.85]| 69.95\\% | 0.36|\n| $\\gamma = -kx + b$|[1.10, 1.81] |1.9 | 0.06| 60.28\\% | 0.28 |\n| $\\gamma = b$ |--|--|1.43|65.95\\%|0.34|\n\n### Q3. Weight Range\n\nSince RL requires a reward value to guide the training direction, we must combine scores to decide whether to push the diffusion model away from the reconstruction sample or bring them closer together. We determine these weights using our scoring scheme. Specifically, the first two scores focus on partial aspects, such as structure and texture, respectively. The third score, however, reflects the overall impression, which is more comprehensive. Therefore, it is reasonable to assign lower weights to the first two metrics and a higher weight to the final metric. Additionally, to further address your concern, we had the labeling supplier company directly rank these samples to validate the consistency between our weighted combination of scores and the human labeling rank. Specifically, the rank of several reconstructions by both human experts and previous label weighting schemes. Then, we calculate the consistency between the human ranking and the weighted score ranking. The experimental results are shown in the following table. The proposed weighting combination scheme is validated by its high consistency with the human scoring scheme. The scoring system is shown in Fig. **S6** of the uploaded one-page PDF file.\n\n|Weighted Score|Rank1(\\%)|Rank2(\\%)|Rank3(\\%)|All(\\%)|\n|-----|-----|----|-----|-----|\n| **0.15, 0.15, 0.7 (Ours)** |**0.93**|**0.92**|**0.93** |**0.93** |\n| **0.10, 0.10, 0.8**|**0.93**|**0.92**|**0.93**|**0.93** |\n| 0.20, 0.20, 0.60| 0.92| 0.92|0.92|0.92|\n| 0.30, 0.30, 0.40| 0.87|0.86|0.88|0.87|\n| 0.50, 0.40, 0.10|0.85|0.87|0.84|0.85|\n| 0.80, 0.10, 0.10|0.84|0.85|0.85|0.85|\n\n\n### Q4. Meaning of the Metric \"Rank\"\n\n“Rank” in Table 2 indicates an average order (from the best to the worst) of all metrics. Thus, a lower rank indicates better performance. \n\n### Q5. Dataset Details\n\nNote that the labeling is provided by a **professional data annotation company**. All annotators are professionals trained in similar tasks. The annotation time for each pair (3 samples) averages about 2 minutes. A total of 24 annotators were employed. We will provide more details in the final version.\n\n### Q6. Limitations\n\nWe have **indeed** discussed some limitations in the final paragraph of the conclusion section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xfj6omQPXd", "reviewer": "Reviewer_G8Cx", "summary": "This paper tries to align diffusion models for image inpainting with human aesthetic standards via a reinforcement learning framework.", "questions": "See Weaknesses.", "limitations": "The authors outlined potential future directions in the conclusion.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper proposes aligning diffusion models for image inpainting with human preferences by integrating human feedback through reinforcement learning, which improves the quality of generated images.\n2. This paper presents a dataset containing 51,000 inpainted images annotated with human preferences, addressing the lack of evaluation datasets for image inpainting.\n3. The proposed method can be applied to various applications, such as image extension and novel view synthesis, providing visually impressive results.", "weaknesses": "1. There has been a lot of work introducing human preferences into diffusion models, such as Human Preference Score (ICCV 2023), ImageReward (NeurIPS 2023), DPOK (NeurIPS 2023), and D3PO (CVPR 2024).\n\n   a. This paper does not fully elaborate on the work related to diffusion models with human preferences.\n\n   b. It is necessary to supplement the comparison with these methods, including differences in methodology and advantages in experimental results.\n\n2. A user study is necessary to evaluate the quality of the generated results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "## Reviewer G8Cx\n\n### Q1. Differences from Other Diffusion Alignment\n\n\n**Our work is significantly different from existing diffusion alignment work**. While some related works exist in the area of text-to-image tasks involving human preference, our work is **the first to align diffusion-based image inpainting with human preference through reinforcement learning (RL)**. Initially, our proposal of the **human preference inpainting dataset** enables the feasibility of this task. Without such a dataset that includes high-quality labels based on human preference, undertaking inpainting alignment would be **impossible**. \nTechnically, we propose a reward accuracy-aware weight strategy into the RL process to further accelerate the training process and boost performance, which has also not been investigated in the mentioned works. We will add further illustrations about the works related to diffusion model alignment with human preference.\n\n### Q2. Experimental Comparisons with your Mentioned Methods\n\nMoreover, to further resolve the reviewer's concern, we have experimentally compared our method with your mentioned methods. Specifically, we simply summarize the implementation of each method. \n\n(1) Human Preference Score (ICCV 2023)[1] learns a negative prompt to map the diffusion process to low-quality samples. Then, in the inference process, the negative sample is utilized in the classifier-free guidance~(CFG) to push the generation trajectory away from low-quality samples. \n\n(2) ImageReward (NeurIPS 2023)[2] trains a reward model and then applies the reward model as a loss metric to end-to-end optimize the diffusion model accompanied by a reconstruction loss. We also conduct the ablation study on reward training strategy in Table 3 in our paper. Our method employs a regression-driven training strategy, while ImageReward (NeurIPS 2023) [2] a classification-drive strategy.\n\n(3) DPOK (NeurIPS 2023) [3] simultaneously optimizes the whole trajectory of a reverse diffusion process and utilizes the KL divergence to panel the regularization, avoiding a large distribution shift.\n\n\n(4) D3PO (CVPR 2024) [4] adopts the RL strategy from direct performance optimization (DPO)[5],  directly optimizes the model on the reward labeling data to minimize the probability of low-quality samples and increase the probability of high-quality samples.\n\n**The experimental results shown in the table below further validate the advantage of the proposed method.**\n\n| Methods (All metrics the larger the better) | WinRate     | T2I       | Reward   | CLIP     | BLIP     | CA       |\n| ------------------------------------------- | ----------- | --------- | -------- | -------- | -------- | -------- |\n| (1) Human Preference Score                  | 58.03\\%     | -16.67    | 0.26     | 0.20     | 0.47     | 0.40     |\n| (2) ImageReward                             | 65.10\\%     | 13.12     | 0.29     | 0.22     | 0.48     | 0.44     |\n| (3) DPOK (KL weight=0.1)                    | 64.59\\%     | 11.43     | 0.32     | 0.21     | 0.48     | 0.43     |\n| (3) DPOK (KL weight=1.0)                    | 62.67\\%     | 9.36      | 0.30     | 0.21     | 0.48     | 0.43     |\n| (4) D3PO                                    | 59.74\\%     | -19.20    | 0.26     | 0.21     | 0.46     | 0.41     |\n| **PrefPaint(Ours)**                         | **71.27\\%** | **21.53** | **0.37** | **0.23** | **0.49** | **0.45** |\n\nMoreover, the authors will provide further discussion in the final version of the paper.\n\n### User Study\n\nFirst, we clarify that the labeling of our dataset is provided by a **professional data labeling company**, with all annotators being professionals trained in similar tasks. The reward scoring is highly accurate and capable of assessing the results of inpainting under criteria based on human preference. We also provide some examples scoring by our reward model in Fig **S4** in the uploaded one-page PDF file, the assessment of inpainted images is closely aligned with human preference. Moreover, to alleviate your concerns, we have also carried out a user study to evaluate our superiority. We have randomly selected about 130 groups of results and conducted a user study involving 10 users, as detailed below. The WinRate map and the diagram of the user study platform can be found in Fig. **S3** and Fig. **S5** of the uploaded one-page PDF file.\n\n| Methods | Kandinsky | Palette | SDv2.1 | Runway (BaseModel) | **PrefPaint (Ours)** |\n| ------- | --------- | ------- | ------ | ------------------ | -------------------- |\n| Rank    | 4.74      | 3.79    | 3.11   | 2.26               | **1.10**             |\n| Var     | 0.35      | 0.49    | 0.60   | 0.48               | **0.24**             |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
## General Response

We thank all reviewers for their time and constructive comments. We sincerely thank Reviewer **yfx6** for the affirmation of the motivation and novelty behind our task, as evidenced by comments such as "Release a new dataset..." and " First incorporate RL with image inpainting ..." , as well as acknowledging the thoroughness of our experimental evaluations, as highlighted by comments like  "The authors provide extensive experiments, ...".  

Furthermore, we are thankful to Reviewer **Hftn** for acknowledging the theoretical contributions outlined in our paper, as illustrated by comments such as "This paper provides theoretical insights into ...", and for recognizing the significance of our dataset in addressing this emerging task, as indicated by the comment "This paper presents a dataset for image in...". We value the insightful feedback provided by all reviewers, which has greatly enriched our work.

We believe we have **clearly and directly** addressed all concerns. Here, we would like to summarize a few key clarifications regarding the contributions of our work.

(1) Our method makes the **first** exploration of the diffusion model human preference alignment problem on the task of image inpainting and proposes a novel benchmark with human experts labeled preference score. While many T2I models use RL to enhance the consistency between text meaning and image content, our method specifically focuses on the task of image inpainting. As far as we know, the alignment of the diffusion model in this context has not been explored before. 

(2) We propose a **human preference-aware inpainting dataset** enabled the image inpainting alignment based on Reinforcement Learning. Without such a dataset that includes high-quality labels based on human preference, undertaking inpainting alignment would be impossible. Note that our dataset is annotated by a professional data annotation company.

(3) The **technical novelty** of the proposed method mainly lies in modeling the upper bound of reward estimation error and using it to adaptively control the reinforcement learning regularization strength, an approach that, to the best of our knowledge, has not been previously investigated.

(4) The reason for using RL for diffusion alignment lies in the differences between the training and testing process of diffusion models. Specifically, the diffusion model is trained on individual steps from a decoupled probabilistic flow, while its inference process involves running the entire trajectory and projecting random noise onto data samples. Since the score function (noise) for each step cannot be accurately estimated, the accumulated errors for each reversing step may cause the reconstructions to drift away from the targets. RL is a method that can optimize the entire trajectory (Markov Chain) and account for the accumulated error of each step, as the reward is measured based on the final reconstruction

Thanks again for the time and effort. We appreciate any further questions and discussions.

**Last but not least, we will make the reviews and author discussion public regardless of the final decision. Besides, we will include the newly added experiments and analysis in the final manuscript/supplementary material.**

Meta-review: - Reviewers acknowledged the contributions of applying RL to align the diffusion-based image inpainting/outpainting models with human preferences and of the newly introduced dataset.
- Reviewers also raised many technical questions: 1) the motivation and the implementation details of the reward weighting factor (yfx6, Hftn), 2) comparing with direct fine-tuning models and recent image inpainting models (85gD), 3) comparing with RL-based methods for T2I diffusion models (Hftn, G8Cx), 4) the details of data annotation (Hftn) and some ablation studies.
- In the rebuttal, the authors did an excellent job by providing thorough and direct responses and successfully addressed most of the reviews’ questions.
- The AC recommends its acceptance after reading all the reviews, rebuttals and discussions. It's a solid work, although its contributions may be limited by the chosen reward model and the specific task of non-guided image inpainting. The AC strongly requests the authors to incorporate the tables and results in the responses to the final version, and to consider comparing with diffusion-DPO (https://arxiv.org/abs/2311.12908) as noted in the limitation section.
Decision: Accept (poster)