PreferThinker: Reasoning-based Personalized Image Preference Assessment
Authors: Shengqi Xu
Xinpeng Zhou
Yabo Zhang
Ming Liu
Tao Liang
Tianyu Zhang
Yalong Bai
Zuxuan Wu
Wangmeng Zuo
Venue: ICLR 2026 Poster (2026)
Keywords: Image Preference Assessment;Multimodal Large Language Model；Chain-of-Thought
Primary area: interpretability and explainable AI

Abstract: Personalized image preference assessment aims to evaluate an individual user's image preferences  by relying only on a small set of reference images as prior information. Existing methods mainly focus on general preference assessment, training models with large-scale data to tackle well-defined tasks such as text-image alignment. However, these approaches struggle to handle personalized preference because user-specific data are  scarce and not easily scalable, and individual tastes are often diverse and complex. To overcome these challenges, we introduce a common preference profile that serves as a bridge across  users, allowing large-scale user data to be leveraged for training profile prediction and  capturing complex personalized preferences.  Building on this idea, we propose a reasoning-based personalized image preference assessment framework that follows a \textit{predict-then-assess} paradigm:  it first predicts a user's preference profile from reference images, and then provides interpretable, multi-dimensional scores and assessments of candidate images based on the predicted profile. To support this, we first construct a large-scale Chain-of-Thought (CoT)-style personalized assessment dataset annotated with diverse user preference profiles and high-quality CoT-style reasoning, enabling explicit supervision of structured reasoning. Next, we adopt a two-stage training strategy: a cold-start supervised fine-tuning  phase to empower the model with  structured reasoning capabilities, followed by reinforcement learning to incentivize the model to explore more reasonable assessment paths and enhance generalization. Furthermore, we propose a similarity-aware prediction reward to encourage better prediction of the user's preference profile, which facilitates more reasonable assessments exploration. Extensive experiments demonstrate the superiority of the proposed method. Our code and dataset will be  publicly released.

Published as a conference paper at ICLR 2026
PREFERTHINKER: BEASONING-BASED PERSONALIZED IMAGE PREFERENCE ASSESSMENT
Shengqi Xu Xinpeng Zhou Yabo Zhang Ming Liu - Tao Liang* Tianyu Zhang Yalong Bai Zuxuan Wu", Wangmeng Zuo
Fudan University
Harbin Institute of Technology
iN2X
Project Page
ABSTRACT
Personalized image preference assessment aims evaluate individual user's image preferences by relying only on small set of reference images as prior infor- mation. Existing methods mainly focus on general preference assessment, training models with large-scale data to tackle well-defined tasks such as text-image align- ment and aesthetics. However, these approaches struggle handle personalized preference assessment because user-specific data are typically scarce and not easily scalable, and individual tastes are often diverse and complex. To overcome these challenges, introduce common preference profile that serves as bridge across users, allowing large-scale user data to be leveraged for training profile prediction and capturing complex personalized preferences. Building this idea, we pro- pose a reasoning- based personalized image preference assessment framework that follows a predict-then-asser paradigm: it first predicts user's preference profile from reference images, and then provides interpretable, multi-dimensional scores and assessments of candidate images based on the predicted preference profile. To support this, we first construct large-scale Chain-of- Thought (Cole)-style personalized assessment dataset annotated with diverse user preference profiles and high-quality CoT-sttyle reasoning, enabling explicit supervision of structured reasoning. Next, we adopt two-stage training strategy: a cold-start supervised fine-tuning phase to empower the model with structured reasoning capabilities, followed by reinforcement learning to incentivize the model to explore more rea- sonable assessment paths and enhance generalization. Furthermore, we propose a similarity- prediction reward to encourage better prediction of the user's ss preference profile, which facilitates more reasonable assessments exploration. Ex- tensive experiments demonstrate the superiority of the proposed method.
INTRODUCTION
Image preference assessment is essential for both evaluating and aligning generative models with human preferences, playing a key role in applications like content recommendation. However, most existing works (Kirstain et al., 2023; Xu al., 2023) focus assessing general preferences, such as text-image alignment and aesthetics, relying on large-scale general preference data. The practical but challenging task of personalized preference assessment remains largely underexplored, where only limited amount of personalized data is available for each user as prior information to assist in the assessment. As shown Fig. (a), personalized preference assessment is partieularly challenging for two main reasons. First, unlike easily scalable general preference data, where most users share similar assessment criteria, each user's personalized data is typically limited and distinet from others, making large-scale, user-specific training difficult. Second, personalized preferences are often complex and diverse, covering multiple dimensions such as art style, color, and art medium, which are more challenging than clear general human preferences (e.g. text-image alignment and aesthetics).
Current research on image preference assessment mainly falls into two categories: CLiP-based methods and Multimodal Large Language Model (bLLM)-bused methods. CLIP- methods (Wu et al., 2023b;a) typically fine-tune the CLIP model using large-scale text-image pairs with similarity to assess general preferences. However, they struggle with personalized preferences since the amount of personalized data is typically limited for large-scale training. Besides, simple similarity fails to capture complex individual preferences and lacks interpretability, it outputs only numerical score.

---

Published as a conference paper at ICLR 2026
General Preference
Persenalioed Preferenee
Conoscoi Profernce Flamerss
Walskpteed Prclennoo
Complee Prffamec
apai Cirasl
Saatabee Daaa
Use A Portennc Bas Conter Dass
Coler (omur Gold
Coler Ciru Drunge
is Weilew Sp-ey
Mesioon Juom Caring
Stamaden Nemor
Seuraleee abood'
doak delvvl
Dead Sulce
Tinker
Obeted
Coler Junen Log"
Color Crankery Aes
de Makan Calieed Fonsl
Avr Makiow Diguul Hocking
User Perioo Dara
Senveleer Nera
Srewetterr Andevar
Doail Dilkol
Arsil Fler
Dher Ala
Graraal Padfor chan Precaiin Puude
Cherl
Dur E Pedkreee Froday Uur it porlero Protl
(5) Challenges of personalized preferencc general preference
(b) Common preferenoe profile for bridging various
Figure I: Illustration of challenges and motivation. (a) The general preference data is easily scalable since users share common assessment criteria, while personalized preference data for each user is typically limited and unscalable, as each user's 's preferences are distinct. Besides, general preferences are often clear (e.g. text-image alignment and aesthetics), while personalized preferences are typically complex and diverse. (b) We propose preference profile comprising multiple common visual elements, based on the observation that although each user's personalized preferences are unique, the key visual elements that shape them are shared and can therefore serve bridge to connect users.
MI...-bbsed methods (Wang et al., 2025c; Xu et al., 2024) achieve interpretable preference assess- ment by fine-tuning the MLLM on large-scale visual ceesstion-anweriing (VQA) pairs. However, the scarcity of personalized images makes it difficult to obtain sufficient VQA pairs for large-scale training, which limits their effectiveness for personalized preference assessment. Overall, most existing methods are designed for general preferences and therefore struggle to handle personalized preference. Recently, an MLLM-based method named ViPer (Salehi et al., 2024) was proposed for personalized preference assessments. It directly feeds both personalized reference images and candidate images into an MILM and fine-tunes the model using a supervised score regression strategy. Nevertheless, it only implicitly utilizes reference images to assist in evaluating the candidate images, failing to fully leverage the critical prior information they contain. Morcover, it lacks interpretable reasoning steps that explain how to produce its final score based on the individual reference images.
To address the above issues, propose a reasoning- based personalized preference assessment system with preference profile prediction, named Prefer Thinker. Our key insight is to bridge various users by introducing preference profile comprising multiple visual elements (e.g. color, art style), as shown in Fig (b)) Alihough each user's personalized preferences are unique, the fundamental elements that form them are common and can serve as bridge across users. This profile offers three main advantages. First, the element- level profile allows us to characterize personalized preferences, which alleviates the challenge of complex individual tastes. Second, since these visual elements can be shared and learned across users, we can leverage large-scale user samples for training to effectively predict each user's visual preference profile, thereby mitigating the issue of limited personalized data. Third, it provides a solid foundation for subsequent interpretable and multi-dimensiona assessments.
Specifically, given user's personalized reference images (preferred and non-preferred) and two candidate images, where reference images may reflect multiple aspects of user's tastes. PreferThinker follows pedicttthen-assess CoT--style structure for personalized preference assessment. As shown in Fig. 2. the process consists of two stages: 1) Profile Prediction: PreferThinker first predicts the user's visual preference and non- prefffrrfeeee profiles comprising multiple visual elements based on the reference images. 2) Multi-dimensional and Interpretable Assessment: Using the predicted profile, PreferThinker provides interpretable scores for the candidate images across multiple dimensions and produces final result based on the total score. This predict-then-asser framework not only enables accurate assessment, but also achieves interpretable scoring grounded in the user's reference images.
To this end, we first construct large-scale CoT-style personalized image preference assessment dataset, named PreferImg-Con which contains 60,000 user samples, annotated with preference profiles and high-quality CoT-style assessments, enabling explicit supervision of structured reasoning. Next, we adopt a two-stage training strategy: a cold-start supervised fine-tuning (SFT) phase to elicit the model structured reasoning, followed by reinforcement fine-tuning (RFT) based on Group Relative Policy Optimization (GRPO) to guide model toward exploring more reasonable assessments and enhancing generalization. Moreover, we find that accurate preference profile prediction facilitates exploring reasonable assessments, which motivated us to propose similarity-awan prediction reward to better predict user's preference profile. Overall, our main contributions are summarized as follows:
We propose PreferThinker, reasoning- based personalized image assessment system with prefer- ence profile prediction, which achieves accurate assessment along with interpretable and multi- dimensional scoring based on the predicted profile.
2

---

Published as a conference paper at ICLR 2026
Reference IuEErc fighastiy Vulype Preteasse
Civee sact's octaomes magee (hech prckerad and maspeclored braact and tro caddalic mmsel, for orki suntl showel profeower peufie and sinual was prgferreer pegfie food refirence irage, soner and smen whid Imagges Refle ring Single Prgfeonc
insao boke zukho and prdlorass
pegfontee peoflie
proficc
tayke codfti fr. Coker Tropial Ar Mediawr Coeple Semeaber Wiidd Desth Scletre
Nespeterra ine got
profermte profle 3
rallaad porformeet praflees
chrisal proferenee profite-
Preference Profile Prediction
cabast nanprettrrnne proflle
cistnaal prefiernce profite
Prefironcc Profile Prodiction
prefercicc praffle
chiase mongrrofirnee profflee
An sy moie Cokor is dl Win, fur Morka Aiaoo d. Salusaion ai Book Ruc
cotoaat iss pusppogonnne profltes
Non-preffoeen Profile Prodiction
sare shareatie Coler frat. fut Medioen Vorme tesday stigt reant tead Tut
nen unngrrefernee profle
Predfiction
tos peroadd the snvediordoonee aleua diy saloo oos
Gandla clyass whan adjont....
J.t
kogertt Augeeat th ficc wott op otige aecfenyy with Crdflt 1ath
All
her oo floure porc highht decei ther preon bete fimer portectl the peefored
match
C.kar::
Liug lall Jiag contined to aieer while ownr wich la oa unsili camas si dee so afllcr
1 A22
th of dhe funus and badkga grea I Luggosch bare to vich dustccii kinding gie
J.T Matiun:
bug: fell
alpay yo corll orhond with pime anc and per tore, shol de direen Arromati horeer
4. S.staration:
L.Stteralie::
Yrag All
Eanoo sigly een rehly oelnot foecn tn dorp ofhenee and mmgggna un pec comer
5. D.Detal:
kugg
Detal:
Anai parto wir velre aeploess aug oakey and
3 + ol
unddig witll yro beleor reuhh
kill holdl gloves herd whih slay ollrrs.. poowink parree cal
redured sncottentie.
Total Score:
Teags Al
heage 3-iiede--33 31
Imagg At 513-4 10000) lenge ergee
Multi dinerviional Scoring and Aveermmnnt
Image
Multiolimensiond Seoring and Assessment
Figure 2: Examples of PreferThinker for personalized image preference assessment. In the think stage, red text denotes alignment with preference profiles, while blue text denotes alignment with non-preference profiles. See Appendix D for more complete reasoning examples.
We construct PreferImg-CoT, the first CoT-style personalized assessment dataset, annotating with preference profile and CoTT-style reasoning, facilitating the supervision of model interpretability.
We adopt a training strategy comprising cold-start and GRPO-based RFT to enables the system to master the reasoning capability. Furthermore, we propose similarity-awant prediction reward to better predict the user's preference profile, facilitating more reasonable assessments exploration.
We compare PreferThinker with existing methods on the proposed and existing datasets. Extensive experiments confirm that PreferThinke outperforms SOTA methods. Moreover, the predicted preference profile of PreferThinker can also benefit personalized image generation.
2 RELATED WORK
Image Preference Assessment is crucial for both evaluating and aligning generative models with human preferences. While metrics such as FID (Heusel et al., 2017) and CLIP scores (Radford et al., 2021) assess image quality and text-image consistency, they are limited in capturing human preferences. To address this, most works (Kirstain et al., 2023; Liang et al., 2024; Li et al., 2024; Zhang al., 2024; Ba et al., 2025; Chen et al., 2025a) employ large-scale preference data to fine-tume CLIP models for assessing general preferences. However, they lack interpretability, acting as black boxes that output only a score. Recently, MLLM-based methods(Zhou et al., 2025; Wang et al., 2025b; Mo et al., 2025a; Gambashidze et al., 2025) have emerged, benefiting from the powerful multimodal understanding of MLLMs. For example, (Xu et al., 2024) and (Wang et al., 2025b) fine- tune MILMS using large-scale visual gunswrin-answering pairs to achieve interpretable assessments. Nevertheless, most existing methods focus on general preferences and struggle with personalized preferences due the scarcity of personalized data and complexity of individual tastes. Though (Salchi et al., 2024) stands out by proposing a personalized assessment method, it naively feeds reference and candidate images into an MLLM for score regression, thus lacking interpretability. In this work, we propose a reasoning- based personalized assessment system that achieves both accurate assessments and interpretable scoring.
Image Preference Assessment Dataset is essential for training preference assessment models. (Xu et al., 2023) proposes ImageRewardDDB, the first dataset for training models to assess general preferences (e.g. text-image alignment, aesthetics and safety). containing 137K candidate image pairs. Further, (Kirstain et al., 2023) and (Wu et al., 2023a) respectively developed PickaPic_v2 and HPD_v2, two large-scale and diverse datasets for learning human preferences. However, most existing public datasets focus on training models to capture general preferences, with limited consideration for personalized preferences. While (Salehi et al., 2024) curates simulated personalized preference dataset, it remains unreleased. In this work, we construct a large-scale personalized assessment dataset annotated with diverse preference profiles and CoT-style reasoning to advance this field.
3

---

Published as a conference paper at ICLR 2026
Reasoning-based Multimodal Large Language Models have achieved significant progress in multimodal reasoning abilities through CoT--baedd or RL-based fune-tuning. Deepseek-R R1 (Guo et al., 2025a) employs the GRPO algorithm (Shao et al., 2024) with the rule-based reward to improve reasoning abilities of L.LM without critic models. Inspired by this, several works have incorporated GRPO-based post-training into MLLMs to enhance the multimodal reasoning in various tasks, including visual perception (Jiang et al., 2025; Huang et al., 2025; Liu et al., 2025b; Zhang et al., 2025a; Yu et al., 2025; Liu et al., 2025a; You & Wu, 2025) and visual understanding (Chen et al., 2025b; Pan et al., 2025; Ni et al., 2025), math problem solving (Guo et al., 2025b; Zhang et al., 2025b; Li al., 2025b), and visual quality assessment (Wu al., 2025; Li al., 2025a; Cai al., 2025). Closest to ours, (Wang et al., 2025b) explores reasoning-basee image preference assessment through RL-baaed post-training. However, this method focuses on assessing general preferences. Distinetly, PreferThinker is the first ceasoning-based personalized assessment system. Inspired by Deepseek-RI, we adopt a two-stage training strategy comprising cold-start SFT and GRPO- based RFT to enable the system to master the reasoning capability Furthermore, we propose manilarrtyy-warr prediction reward to better predict user's 's preference profile, facilitating more reasonable evaluations exploration.
3 REASONING-BASHR PERSONALIZED IMAGE PREFERENCE ASSESSMENT
In this work, we propose PreferThinker, reasoning- assessment system with preference profile prediction. Given a few individual reference images (both preferred and non-preferred) and two candidate images, wreferThinner assesses the candidate images following predict- thesss--hen--sssess CoT- style structure. As shown in Fig. 2, it first predicts the user's 's visual preference and non-pre non-preference profiles based on the reference images. Next, it utilizes the predicted profiles criteria to score the candidate images across multiple dimensions in an interpretable manner and produces a final result based the total score. By leveraging this predict-then-eee achieves accurate assessment but also providing interpretable scoring based on the predicted profiles.
paradigm, PreferThinke not only
Specifically, first introduce preference profile composed of multiple common visual elements to bridge various users (Section 3.1). Next, we construct a large-scale CoT-style dataset annotated with preference profiles and high-quality reasoning to provide reasoning supervision (Section 3.2). We then employ a two-stage training strategy comprising cold-start SFT and RL-based post-training to enable the system to master the reasoning capability (Section 3.3). Moreover, we discover that more accurate profile prediction facilitates exploring more reasonable assessments, which motivates us to propose a similarity aware prediction reward to better predict preference profile (Section 3.4).
3.1 VISUAL PREFERENCE PROFILE
Most existing methods mainly rely on large-scale, easily scalable general preference data to train models for general preference assessment. However, they struggle to handle personalized preference due to the limited personalized data for each user and user heterogeneity. To address this issue, we propose preference profile comprising multiple common visual elements to bridge various users. We argue that although each user's personalized preferences are unique, the key visual elements that shape these preferences are shared and can serve as a intermediate bridge to connect various users.
To formalize a user's ' complex personalized visual preferences, we identify 15 visual elements that most frequently appear in Lexica's text prompts and strongly influence user preference toward personalized image generation. We then conduct a user study with 100 participants, asking each to select the five most important visual elements visual preference profile. The result (See Fig SI in the Appendix) reveals that art style, color, detail, art medium and saturation are voted as the most representative visual elements for characterizing personalized preferences. To ensure preference profile diversity, we collect rich vocabulary of related terms for each visual element, totaling 288 words (See Fig. $2 in the Appendix), laying solid foundation for constructing a large-scale and diverse dataset for personalized image preference assessment.
3.2 LARGE-SCALE COT-STYLE PERSONALIZED PREFERENCE ASSESSMENT DATASET
To enable the model to master CoT-style reasoning capabilities, we construct a large-scale CoT-style dataset that provides high-quality reasoning supervision. Since most existing public datasets focus on general preference assessment, we first construct new dataset called PreferImg, dedicated to
hhtps//lexica.an//
4

---

Published as a conference paper at ICLR 2026
Youal Preference Prufile
Preferred Referenes Images
Candidare Image
Randon Sumple
Coter: Sladh Park
Art vole: Nasga
Rocaption
T21 Model
Satoration: Nokeraic
Detil:: Fase
Finaat Proffreeen
Aow
Imgge therey
sight
Weights
Sime
Alaha be
Color
de mrsiiu
Art syyk Guulii:/ut
Caker Spods Blue
Mediumn Darylic Parting
Racepiin
121 Madkl
Runshr
Sateration: Vihreal
Decuit: Sell
Yaual Nan-prelerene Frofile
Nan-prefeered Refervare
Candidate Image
(a) Personalized preference generation pipeline
CoI-style Rapane
Relaiing Baaee
Reference Tuage
Meta Inforuations
System Pronupt Examppie-
criaad peofornce prgfie-
Questio Pronpt
fermel wen proffemed
Gitve aure' perooaaioo offiveec Anager Bash pre-
chttnal perfiorace peofles
Preferencs Profic Pradeciio
Image
Prefermns:
prudiuae she ass's rouul pnglirstor pegfie and setgl
reftrossr peofle Banad as made Apforustion Das
<hinaal von greffroccc profiics
Seinaal profernnce proflee-
Nax Nox-profrrnne Proflle Prodiction
Nec-reforeao Frofle
coer for the cowsller image f Allee upyl onlor wediee wieraton
Moll-ffonerriond fosersmnnnt
Iunget
Inngee Is
Annea: Iange B
coumpiio dlons ZolosZzooppin
Clansor Imagge <lascsors
(b) CoT-style dataset construction
(c) Multimodal information and system prompt for Claude 3.7
(d) CoT-style response template
Figure 3: Illustration of the proposed dataset Preferlmg-Coff (a) Personalized preference data generation pipeline. (b) Overview of Co'T-style dataset construction: Claude annotation and data filtering. (c) Prompt design for Claude 3.7 to generate CoT-style response. (d) CoT-style response template, including preference profiles prediction, multi-dimensional assessment and answer.
personalized image preference assessment, annotated with diverse visual preference profiles. We then employ the advanced Claude 3.7 model (Anthropie, 2024) to generate high-quality CoT--tyyle annotations on the proposed Preferlmg, ultimately producing our CoT-style dataset, PreferImg-Col
Personalized Preference Assessment Data Generation. Due to the difficulty in acquiring person alized preference data with annotated preference profiles, such datasets currently unavailable. To fill this gap, inspired by (Mo et al., 2025b), we construct a large-scale personalized assessment dataset containing 80K simulated users, each with distinct preference profile. The data generation pipeline is shown in the Fig. 3(a). Specifically, we first randomly sample five visual preference elements to assign visual preference and non-preference profiles to each user. To account for the fact that real users may have multiple personalized preferences, several preference profiles are assigned to subset of users. Then, we combine these profiles with initial prompts and feed into a text-to-image model to generate each user's 's reference images (preferred and non-preferred) and two candidate images. To further guarantee content diversity, we follow the PrefGen (Mo et al., 2025b)to select 190 K prompts from Lexica, DiffusionDB (Wang et al., 2022), and COCO (Lin et al., 2014) 4) initial prompts. Ultimately, we establish a dataset annotated with preference profiles, comprising 80 K users, 20K of whom are associated with multiple preference profiles, along with 1.36 million images, laying a solid foundation for subsequent CoT-style dataset construction.
CoT-style Dataset Construction. To construct a high-quality CoT-style dataset, design two- stage construction process: Claude annotation and data filtering, as shown in Fig. 3(b). First, we employ Claude 3.7 with prompting to generate CoT style responses. Figure 3(c) shows that the prompt contains four key components: (1) user reference images (both preferred and non-preferred), (2) two candidate images, (3) meta-inforration including the question prompt, visual preferences and non-preferences, and the correct answer, and (4) system prompt along with correct response examples. Based on this prompt, Claude 3.7 follows a predict-then-assees structure. It first predicts the user's visual preference profile from the reference images, which serves as a criterion for assessment. It then generates a CoT-style analysis that includes milti-dimensional,, interpretable scoring and assessments for the candidate images, and finally outputs the final result. The response template is shown in Fig. 3(d). To ensure annotation quality, we further filter out any illogical or incorrect CoT-sttye responses, such as those with inconsistent reasoning or a mismatch between the prediction and ground--rutte Finally, we curate PreferImg-Ca - large-scale CoT-style dataset with 60,000 user samples based on the proposed Preferimg dataset, which serves as the foundation for our subsequent cold-start SFT.
3.3 TRAINING STRATEGY
We employ a two-stage training strategy to elicit and incentivize the model's structured reasoning capabilities. First, we conduct cold-start supervised fine-tuning (SFT) phase teach the model how
5

---

Published as a conference paper at ICLR 2026
(a) Stage 1: SFI cold-start
(b) Stage 2: RL posi post-training
(c) Profile prediction reward
CoT-tyle Asseismeett
Gooeg Foran
Proto
Benard
Aclous mach Prodati Songing Proii
Frodetke Rewent
Heddtig
Reved
Tou Stalarty
Prable
Statide
EaEagh Respansn
hnlsscs Proll
Medi
Sapertbaad Fine-tuard Ohw1
Image
Tokeniset
Innege
Tes Tokenazer
Images Impuc
Questio Prompi
Tmnage
Iuputs
Question Pronept
Fyredies Magee Mloggleee
Figure 4: Illustration of training strategy and proposed prediction reward: (a) Cold-start SFT to teach
structured reasoning: (b) RL-based post-training to explore more reasonable assessments and enhance
model generalization. (c) Similarity-awan prediction reward for better preference profile prediction.
to perform structured reasoning. Then, we employ reinforcement learning (RL)-based post-training to encourage the model to explore more reasonable assessment paths and enhance its generalization.
Stage 1: Supervised Fine-tuning for Cold-start Initialization. We utilize Qwen2.5-VL (Bai et al., 2025) as the baseline model and conduct a SFT with the proposed CoT-style dataset Preferimg- COT, as shown in Fig. 4(a). During the cold-start phase, the model learns to first predict the user's preference profiles based on reference images and then leverages the predicted profiles as criteria to produce interpretable, nulti-dimensional scores and assessments for candidate images. We train the model using a standard autoregressive language modeling objective and apply a token-level cross-entropy loss to provide strong supervision throughout the generation process:
(SFr(0) == (a..)))-DoT log P (yt 2, 0)
where DCOT our Preferlmg-CoT, and (x,,) the input query and CoT-style target response.
Stage 2: GRPO-based Reinforcement Learning for Post-training. Although cold-start endows the model with structured reasoning capabilities, its supervised training paradigm limits the exploration of diverse reasoning paths and restricts generalizability. To explore more reasonable assessments and enhance generalizability, we employ the Group Relative Policy Optimization (GRPO)-bases reinforcement learning for exploration-driven post-training, shown in Fig. 4(b). Unlike traditional policy optimization methods (Schulman et al., 2017), GRPO efficiently generates diverse reasoning responses by optimizing policy gradients sample group without requiring critic model.
Group Relative Policy Optimization. The MILM generates a group of G CoT-style outputs {01, 02,,02,...,44 for each input query from current policy лg. Each output contains preference profiles prediction, multi-dimensional assessment and a final answer. For each o,, we compute a scalar reward ri, and normalize these rewards to estimate its group-relative advantage A::
r; max (fri, r2, ... ,rr))) std (fr1, 72, ... ,ra))
A =
(2)
Then, the GRPO training objective is as follows:
To(a; x)) Te (o; |x)
(f) =E Pmx) No (O)x) G 1=1
min
Ai,
(3)
To(0; x) Th (o; x))
clip
e, +0 A. -1
where and 33 are hyperparameters, and RSFT, ny, and Tu are the model after cold-start SFT, the optimized model and the old policy model. The KI. divergence term Dx (SFT)) promotes controlled policy updates, balancing exploration and stability throughout training.
3.4 SIMILARITY-AWARE PREFERENCE PROFILE PREDICTION REWARD
We find that accurate profile prediction facilitates the exploration reasonable assessments, moti- vating us propose a prediction reward to improve the accuracy of profile predictions. Figure

---

Published as a conference paper at ICLR 2026
User Reference Images
Candidatc Immees
Preferred Images
Non Non-preferred Images
B
robnol peofrreked proflin
Coker Doer Meckem: Woodea famurstar Vames Betal: abenned
-hrosd proforunee proflli-
Similarity with profile GT: 69.08%
critoaal profivence profllin
Similarity with profile GT:
ratoaal entepe prefereeee proflirs
sotaee nnnvengooenn pmallic
Goay,
Right
oollnee songgiifinnnce proffics
Similarity profile 80,
"cennal Ass preffenncc profiles
Similarity with profile
Liit manig sport on cptuase as les an vuar plln dos dat ded Abscc
lncippaaa
de delienitty dagei thr
procrad conventer arterne on way lirking the speeff yluct woed reeriee profecd
Aut blddaee
ewagee denggee lee dlou them reroorts wdider
lado Buddyy
alverhy
-m-er <he te war
raddiiiiiggg shaly otfir sesens foom motn wou peooiing
priment
sheel' qrally cos bas cite
1aag Aill.. ag bevvese officra corperciss cas the dedge au oligg with volly du Deffuct dinall
Newwr
Auog regirm di in Ben.
vierassonable
Ansrgc CT: Irenge
Incorrect
Connec
(a) Unreasonable assessment result without prediction reward
(b) Reasonable assessment result with prediction reward
Figure 5: Effectiveness of the proposed prediction reward (PR). (a) The model without PR predicts profiles differed from GT, leading to unreasonable assessment and incorrect answer. (b) The model with PR predict the profiles closer to GT and provide more reasonable assessment with correct answer.
illustrates a comparison of personalized assessment cases with and without the prediction reward. The results show that without the reward, the predicted profiles differ from the ground-truth (GT), leading to unreasonable assessments and an incorrect answer. Conversely, the model with the reward predicts the profiles closer to GT and provide more reasonable assessments with a correct answer.
Similarity-aware Prediction Reward Design. To incentivize the model to predict the user's preference profile more accurately, we propose text-image similarity- prediction reward in Fig. 4(c). Given a user's personalized reference images, PreferThinker first predicts the user's visual preference profile V+ and non-pre non-preference profile V_. First, we measure the prediction accuracy via computing the text semantic similarity Stett between the predicted and ground profiles:
Stext Simeeu (V., V+) + Simeeu (V_, V-):
where Simeet measures the similarity between two texts, V+ and V_ are ground-truth preference and non-preference profiles. We use SBERT (Reimers & Gurevych, 2019) to compute the text similarity.
Second, considering that image and text are the two sides of the same coin and should be tightly coupled. Therefore, we also measure the prediction accuracy via computing the image similarity Sing between the generated images based on the predicted and ground-trute -tuth visual preference profiles. Specifically, given an initial text mintial, we recap it by incorporating the predicted and ground-truth visual preference profless, and then generate images using T2I model (Labs, 2024):
1 = [R2 V+)), 4 = T==1(11AA (R(Tiiiaa V+)),
(5)
i_ T21 (R(Tiniiiinnnn initial V-)), 1_ T2- (R(Tinitaal, V-)),
where T2I is the text-to-image model, R denotes the recaption process. Then, we compute the image similarity between the images generated with predictions and ground-truth:
Sing~ Simm (4,, [4) + amg (i_, I_),
(6)
where Simangg measures the similarity between two images. We use DreamSim (Fu et al., 2023) to compute the image similarity. Finally, the profile prediction reward function is a weighted combination of two similarity measures, designed to balance their contributions:
Tpredict = Wing-Sing + WhextSuxtt
where Wing and Whes are hyperparameters that balance the contribution of each similarity.
Hybrid Reward Modeling. Apart from considering the prediction reward Tpredict. we utilize format correct reward Tformat and an assessment accuracy reward raccuracy to encourage the model to produce the correct response format and assessment results:
7

---

Published as a conference paper at ICLR 2026
Preferred reference images
Prodicted preference profile
Initial image
Personalized image
Initial image
Personalized image
Art style: Naive ari
Color: Coral pink
Art medium: Puper mache
Saturation: Vibrant
Detail: Textured
"A smiling eyberpunk girl" Figure 6: Personalized image generation with the predicated preference profile.
"Sturry night"
Format Correctness Reward (fformat). The format reward Nformat is awarded when the response follows the required structure: 1) the predicted preference and non-preference profiles are en- closed within the <visual pre preference
profile> and <vi suall non-pre ferencee profile> tags, respectively, 2) the multi-dimensional assessment is placed in the <think> tags, 3) the final result is contained in the <answer> tags; otherwise, the Tformat is set to O.
Assessment Accuracy Reward (r (Faccuracy). Given the ground truth of Assessment result E' and the produced assessment result É. The reward racccurayy is set to when E E, and 0 otherwise.
Furthermore, during the RL-based post-training, we utilize weighted combination to balance these reward functions. The total reward r for response is weighted sum of the three rewards:
Up"predics + W Tfoomat + W laccuracyy
(8)
where Wlp, wy and u, are hyperparameters that balance the contribution of each reward.
4 EXPERIMENTAL RESULTS
4.1 EXPERIMENTAL SETTINGS
Datasets. We conduct experiments on the proposed PreferImg and real user dataset PickaPic (Kirstain et al., 2023). Based on the PreferImg, we construct a benchmark of 1,500 users, categoriz- ing them into innle-preffernce (SP) and multi-preference (MP) groups based on whether reference images reflect singular or multiple prefer- ences. To evaluate generalization, we further divide the benchmark into seen and unseen subsets based on whether users' profiles are included in the training data. Since no real-user dataset exists for personalized pref- erence assessment, we process Pick- aPic, originally collected from real user interactions for general prefer- ence assessment, by grouping sam- ples by user IDs, resulting in a user- specific benchmark with 894 user sam- ples. Note that although the samples are grouped by users, the assessment labels still reflect general preferences, unlike PreferImg, which targets personalized preferences.
Table I: Quantitative comparison assessment accuracy with existing methods on the proposed Preferimg and real user dataset PickaPic. SP denotes that the user's ss reference image reflects sin- gle preference, while MP is multiple preferences. t denotes the dataset's labels are for general preference assexsment. * denotes the methods that support reference images as input for in-context learning. The best and second-best performances are highlighted.
Unsecn
Mathod
Prztering Prztrring Pielcllng Picterling (SP)
Avg
(MP)
(SP)
(MP)
CJIP-aaned Madile
Pick&oooe sess::
986M
49.6
65.4
51.2
56.4
67.9
$4.7
ImagelRewad persamm)
478M
52.2
51.2
49.0
57.2
60.7
$4.1
HPS acc
2B
51.4
5013
50.8
57.6
63.3
54.6
CLIPScore soiaza
428M
50.2
54.0
52.6
52.0
58.0
53.4
Aesthetcs addeaan
304M
50.6
48.0
48.5
50.8
48.5
49.3
Cycl-Reeand precnsss
477M
45.0
$0.8
50.0
56.0
62.6
53.5
MLLM-Juwd Midlels
UnifeedReward UniifedReendd- eeand--hin perernt
7B
49.0
46.0
30.6
57.2
61.7
52.9
7B
45.6
47.3
48.0
52.8
60.5
51.4
LLaVA Reward ViPer"
58.4
62.2
55.6
8D
78.0
93.4
30,0
62.2
81.2
MLLMs
Qwen2.5 V-VL-B (Busse) InternVI -3.588 GL. TVii--nhiniigg
66.4
69.3
76.3
Claude
83.6
OpenA1 GPT-40
92.2
Doubao vision pro. Gonnini-2
80.2
83.6
67.2
71.7
Prefer Thinker' prper 1s. prrs S67A 15. Base Model
92.8
88.7
+3.0
+5.1
+21.2 +30.
+34.4
+25.0
+7.7
+20.3
Implementation Details. PreferThinker is initialized from Qwen2.5-VIL-78 and trained on cluster of 8 NVIDIA A A100 GPUs. We utilize the I.LMA-Facctor (Zheng al., 2024) for cold-start SFT, and train for only (Shen et al., 2025) for RL-based post-training. We set a learning rate of lc-5 with 6 rollout samples per input. The reward weights Wp. w and w. set to 0.7, 0.3 and 1.0. See Appendix A for details.
epoch to prevent overfitting. After the SFT, we utilize the VLM-RI framework
Baselines. We compare PreferThinker with (1) CLIP- based models: PickScore (Kirstain et al., 2023), ImageReward (Xu et al., 2023), HPSv2 (Wu et al., 2023a), CLIPScore (Radford et al., 2021), Aesthetics (Schuhmann et al., 2021), CycleReward (Bahng et al., 2025); (2) MLLM-based methods: UnifiedReward (Wang et al., 2025c) UnifiedReward Think (Wang al., 2025b), L.LVV- VA-Rewarr
8

---

Published as a conference paper at ICLR 2026
Table 2: Ablation study of PreferThinker. Ass.: Assessment
Table 3: Performance and speed of
aceuracy. Pred.: Profile prediction accuracy.
PreferThinker without explicit COT.
Seen-SP
Unseen-SP
Seen-MP Unseen-MP
Accuracyt
Base SFT RL PR
Method
Speed
Axs. Pred.
Ass. Pred.
Ass. Pred.
Ass.
Pred.
Preferimg PickaPie
75.4 70.4
72.0 70.88
62.0 71.1
64.8
71.1
ViPer
86.0
62.2
2.19
92.0 84.2
91.8 85.3
81.2 73.9
81.6
74.2
OpenAI-GPT-4o
88.0
65.7
12.92
/
89.6 79.6
91.0
80.3
78.4
76.9
77.2
73.5
Claude
88.3
64.9
18.22
/
/ d
94.9 84.1
95.2
85.2
90.4
72.5
89922
74.1
Doubao- 1.100--------0 1-Svisson
84.3
63.8
5.39 s.
1
96.6 85.4
96.4 86.2
92.0 78.6
92.8
80.7
Ours (w/o Co'T
93.2
65.2
0.80
ProteTRisker ---- vie O
GLM-LIV - 100VVL-35
PreferThinice
Viter man
Ques-25.VL
GLM4.1V - 1n
Robusmess Seen-SP
Robustness Seen MP
Robustnest on Unseen-SP
Robustness on Unscen-MP
100
95
100
95
Accassy (1)
Accuracy (N)
Accurcy (%)
75
80
80
80
70
70
55
60
Num. of Reference Frames Pairs
Num. of Reference lrmmee Pairs
Refermme
Num. Relinence
Seen domain performance curves
(b) Unseen domain performance curves
Figure 7: Robustness to number of personalized reference images on PreferImg.
(Zhou et al., 2025), and ViPer (Salehi et al., 2024); (3) Open-souree MLLMs: Qwen2.5- (Bai et al., 2025), Intern (Wang et al., 2025a), GLM-4.1 V-9B (Hong et al., 2025); (4) Closed-source MILMs: Claude 3.7, GPT-4o, Doubao- 1.5-vision, and Gemini-2.5. For fair comparison, we categorize the methods based on whether support reference images as input for in-context learning.
4.2 COMPARISON RESULTS
Comparison Seen User Data. In Table 1, we compare PreferThinker with existing methods the seen user data of PreferImg. It is observed that CL.PP-bsed and most MLLM-basee methods perform poorly, since they lack modeling for personalized preferences. Though ViPer achieves relatively good results on the SP data, its accuracy decreases when handling the MP data, and it lacks interpretability as it only outputs a numerical score. Further comparisons with the open-MLLMM reveal that directly utilizing middle-sized models struggle with personalized preference assessment. In contrast, closed- source models achieve comparable performance, thanks to their powerful prior knowledge. Overall, PreferThinke achieves the highest accuracy and provides interpretable assessments and scores.
Comparison on Unseen User Data. To validate generalization, we compare performance on unseen users in Table 1. PreferThinker maintains the highest accuracy on Preferlmg, revealing strong generalization. However, nearly all methods perform poorly on the real user benchmark, PickaPic, due to its complexity. Though we group PickaPic by user ID and assign user-specific reference images, the labels still reflect general preferences. Thus, methods designed for general preferences perform better on PickaPic. PickScore achieves the best performance since PickaPic serves seen dataset for it, while our method ranks second on unseen PickaPic data, further confirming its generalizability. Please refer Appendix D.3 for CoT-style assessment visualization of real users.
4.3 ABLATION STUDY AND DISCUSSION
Effectiveness of Cold-start SFT and RL Post-training, Table demonstrates the critical roles of both cold SFT and RL training phase. Removing either phase leads to a notable degradation in model performance, confirming their importance for accurate preference assessment. Moreover, the use of RL after SIT significantly improves the model's performance on unseen user data, particularly for multi-preference (MP) users, revealing its effectiveness in enhancing model generalization.
How dose the Prediction Reward (PR) Facilitate Reasonable Assessment? We further study the importance of the proposed prediction reward. Table 2 shows that adding the prediction reward effectively improves the preference profile prediction accuracy. Besides, Figure shows that incorpo- rating the prediction reward enables accurate profile prediction, which facilitates more reasonable assessment and correct answer, further demonstrating the effectiveness of the prediction reward.
Personalized Generation with Predicted Preference Profile. We further discuss how to utilize the predicated profile to generate personalized images in Fig. 6. We first predict a user's 's preference profile based reference images. Then, we optimize the initial texts based on the predicated profile

---

Published as a conference paper at ICLR 2026
to generate personalized images. The results show that the generated images align well with reference images, affirming the potential of the TreferThinket for facilitating personalized image generation.
Performance and Speed of PreferThinker without Explicit COT. We study whether PreferThinker can implicitly use reasoning capabilities for direct preference assessment. Table shows that still outperforms SOTA methods on PreferImg and achieves comparable accuracy on PickaPic even without explicit CoT, while delivering the fastest inference speed, further demonstrating its superiority.
Robustness to Number of Reference Images. To evaluate the robustness of PreferThinker, different number of reference images as user's prior information. Figure shows that PreferThinker outperforms other baselines across different numbers of reference images both seen and unseen data, revealing its robustness in leveraging limited information for personalized preference assessment.
5 CONCLUSION
We introduce PreferThinker, the first reasoning-based personalized image preference assessment system with preference profile prediction. We propose preference profile to bridge various uses, allowing large-scale user data to be leveraged for training profile prediction and capturing complex personalized preferences. We construct PreferImg-Cofs erence profiles and high-quality reasoning for interpretability supervision. We adopt a two-stage training strategy comparing Cold-start SFT and reinforcement learning to empower the model with reasoning capabilities. We propose a similarity-aware prediction reward to improve profile prediction, facilitating reasonable assessments. Extensive experiments verify the superiority of PreferThinker.
CoT-style dataset annotated with pref-
ACKNOWLEDGMENTS
This work was supported by National Key RD Program of China under Grant No. 2022YFA1004110
REFERENCES
AI Anthropic. The claude 3 model family: Opus, sonnet, haiku. Claude-3 Model Card, 1(1):4, 2024.
Ying Ba, Tianyu Zhang, Yalong Bai, Wenyi Mo. Tao Liang, Bing Su, and Ji-Rong Wen. Enhancing reward models for high-quality image generation: Beyond text-image aligmment. arXiv preprint arXiv:2507.1900 19002, 2025.
Hyojin Bahng, Caroline Chan, Fredo Durand, and Phillip Isola. Cycle consistency as reward: Learning image-text alignment without human preferences. arXiv preprint arXiv:2506. 02095, 2025.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge. Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 25Xiv: 13923, 2025.
Zhuoxuan Cai, Jian Zhang, Xinbin Yuan, Peng-Tao Jiang, Wenxiang Chen, Bowen Tang, Lujian Yao, Qiyuan Wang, Jinwen Chen, and Bo Li. Q-ponder: A unified training pipeline for reasoning- visual quality assessment. arXiv preprint 2.XY::2506.03884 2025.
Daiwei Chen, Yi Chen, Aniket Rege, Zhi Wang, and Ramya Korlakai Vinayak. Pal: Sample-efficient personalized reward modeling for pluralistic alignment. In The Thirteenth International Conference on Learning Representations, 2025a.
Lei Chen, Xuanle Zhao, Zhixiong Zeng. Jing Huang, Yufeng Zhong, and Lin Ma. Chart-rl: Chain-of-thought supervision and reinforcement for advanced chart reasoner. arXiv preprint arXiv:2507.15509, 2025b.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions human visual similarity using synthetic data. arXiv preprint arXiv:2306.099: 2023.
10

---

Published as a conference paper at ICLR 2026
Alexander Gambashidze, Li Pengyi. Matvey Skripkin, Andrey Galichin, Anton Gusarov, Konstantin Sobolev, Andrey Kuznetsov, and Ivan Oseledets. Listener-rewarded thinking in vlms for image preferences. arXiv preprint arXiv:2506.228833 2025.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms reinforcement learning. arXiv preprint arXiv:2501. /2948, 2025a.
Zixian Guo, Ming Liu, Zhilong Ji, Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Decoupled visual interpretation and linguistic reasoning for math problem solving. arXiv preprint arXiv:2505.1766 2025b.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Wenyi Hong, Wenmeng Yu, Xiaotao Gu, Guo Wang, Guobing Gan, Haomiao Tang, Jiale Cheng. Ji Qi, Junhui Ji, Lihang Pan, et al. Glm-4.1 v-thinking: Towards versatile multimodal reasoning with scalable reinforcement learning. arXiv e-prints, pp. arXiv-2507, 2025.
Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao, Zheyu Ye, Fei Zhao, Zhe Xu, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models. arXiv preprini arXiv:2503.06749) 2025.
Qing Jiang, Xingyu Chen, Zhaoyang Zeng, Junzhi Yu, and Lei Zhang. Rex-thinker: Grounded object referring via chain-of-thought reasoning. arXiv preprint urXiv:2506.04034 2025.
Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joe Penna, and Omer Levy. Pick- a-pic: An open dataset of user preferences for text-to-image generation. Advances neural information processing systems, 36:36652-36663 2023.
Black Forest Labs. Flux. attps://gitte wttp://w....oooocccccccooeeeeeee horesacc--ooortttttto httpp//a..////tt.aa.ooooooooooor 2024,
Hanting Li. Hongjing Niu, and Feng Zhao. Stable preference: Redefining training paradigm of human preference model for text-to-image synthesis. In European Conference Computer Vision, pp. 250-266 Springer, 2024.
Weiqi Li, Xuanyu Zhang, Shijie Zhao, Yabin Zhang, Junlin Li, Li Zhang, and Jian Zhang. Q-insight: Understanding image quality via visual reinforcement learning. arXiv preprint arXiv:2503.22679, 2025a.
Yuting Li, Lai Wei, Kaipeng Zheng, Jingyuan Huang, Linghe Kong. Lichao Sun, and Weiran Huang. Vision matters: Simple visual perturbations can boost multimodal math reasoning. arXiv preprint (.Xiv:2506609736. 2025b.
Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, et Rich human feedback for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
19401-19411, 2024.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yuqi Liu, Bohao Peng. Zhisheng Zhong, Zihao Yue, Fanbin Lu, Bei Yu, and Jiaya Jia. Seg- zero: Reasoning-chain guided segmentation via cognitive reinforcement. arXiv preprint arXiv:2503.06520, 2025a.
Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint arX.v:2503.07785, 2025b.
Wenyi Mo, Ying Ba, Tianyu Zhang, Yalong Bai, and Biye Li. Learning user preferences for image generation model. arXiv preprint 250::2500008200 2025a.

---

Published as a conference paper at ICLR 2026
Wenyi Mo, Tianyu Zhang, Yalong Bai, Ligong Han, Ying Ba, and Dimitris N Metaxas. Prefgen: Multimodal preference learning for iifferencc-condittion image generation. arXiv preprint arXiv:2512.06020, 2025b.
Minheng Ni, Zhengyuan Yang, Linjie Li, Chung- Lin, Kevin Lin, Wangmeng Zuo, and Lijuan Wang. Point-rft: Improving multimodal reasoning with visually grounded reinforcement finetuning. arXiv preprint arXiv:2505. 2202225555.9222... 2025.
Jiazhen Pan, Che Liu, Junde Wu, Fenglin Liu, Jiayuan Zhu, Hongwei Bran Li, Chen Chen, Cheng Ouyang, and Daniel Rueckert. Medvlm-rl: Incentivizing medical reasoning capability of vision- language models (vlms) via reinforcement learning. arXiv preprint 1.Xiv:2552.11634 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748 8763. PmLR, 2021.
Nils Reimers and Iryna Gurevych. Sentence-bert Sentence embeddings using siamese bert-networks. arXiv preprint 1008, 2019.
Sogand Salehi, Mahdi Shafiei, Teresa Yeo, Roman Bachmann, and Amir Zamir. Viper: Visual personalization of generative models via individual preference learning. In European Conference on Computer Vision, pp. 391-406. Springer, 2024.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Yang Wu, et al. Deepseekmath: Pushing the limits of mathemat- ical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable rl rl-style large vision-language model. arXiv preprini arXiv:2504.07615. 2025.
Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internv13. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.1826: 2025a.
Yibin Wang, Zhimin Li, Yuhang Zang, Chunyu Wang, Qinglin Lu, Cheng Jin, and Jiaqi Wang. Unified multimodal chain-of-thoughe reward model through reinforcement fine-tuning. arXiv preprint arXiv:2505.03318, 2025b.
Yibin Wang, Yuhang Zang, Hao Li, Cheng Jin, and Jiaqi Wang. Unified reward model for multimodal understanding and generation. arXiv preprint arXiv:2503.0522 2025c.
Zijie J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondb: A large-scale prompt gallery dataset for text-to-image generative models. arXiv preprint arXiv:2210, 0et.:2210..44800 14896, 2022.
Tianhe Wu, Jian Zou, Jie Liang. Lei Zhang, and Kede Ma. Visualquality-rl: Reasoning-indueed image quality assessment via reinforcement learning to rank. arXiv preprint 250..::500...4660 2025.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306.09334 2023a.
12

---

Published as a conference paper at ICLR 2026
Xiaoshi Wu, Keqiang Sun, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score: Better aligning text-to-image models with human preference. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2096-2105, 2023b.
Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903-15999 2023.
Jiazheng Xu, Yu Huang, Jiale Cheng, Yuanming Yang, Jiajun Xu, Yuan Wang, Wenbo Duan, Shen Yang. Qunlin Jin, Shurun Li, et al. Visionreward: Fine-gramd ame-ggained multi-dimensional human preference learning for image and video generation. arXiv preprint arXiv:24/2.21006 2024.
Zuyao You and Zuxuan Wu. Seg-rl: Segmentation can be surprisingly simple with reinforcement learning. arXiv preprint 25t.::::0066222222 2025.
En Yu, Kangheng Lin, Liang Zhao, Jisheng Yin, Yana Wei, Yuang Peng, Haoran Wei, Jianjian Sun, Chunrui Han, Zheng Ge, al. Perception-rl:: Pioneering perception policy with reinforcement learning. arXiv preprint arXiv:2504407795. 2025.
Bob Zhang, Haoran Li, Tao Zhang, Cilin Yan, Jiayin Cai, Xiaolong Jiang, and Yanbin Hao. Improving the reasoning of multi-image image grounding in milms via reinforcement learning. arXiv preprint arXiv:2507.00748, 2025a.
Jingyi Zhang, Jiaxing Huang, Huanjin Yao, Shunyu Liu, Xikun Zhang, Shijian Lu, and Dacheng Tao. R1-vI: Learning reason with multimodal large language models via step-wise group relative policy optimization. arXiv preprint ar Xrv:: 12937, 2025b.
Sixian Zhang, Bohan Wang. Junqiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Learning multi-dimensional human preference for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8018-8027, 2024.
Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng, and Yongqiang Ma. Llamafactory: Unified efficient fine-tuning of 100+ language models. In Pro- ceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024. Association for Computational Linguistics. URL httt::///// / /wwpiv..iivv..///////aa//aaaaa org/ /los/2403.133
Shijie Zhou, Ruiyi Zhang, Huaisheng Zhu, Branislav Kveton, Yufan Zhou, Jiuxiang Gu, Jian Chen, and Changyou Chen. Multimodal Ilms as customized reward models for text-to-image generation. arXiv preprint arXiv:2507.2139), 2025.
APPENDIX
For better understanding of the main paper, we provide additional details in this supplementary material, which is organized as follows:
(Section A) We provide more details about the experimental settings.
We provide more statistics details of the PreferImg test benchmark in Sec. A.1.
We provide more training details for the SFT cold-start in Sec. A.2.
We provide more training details for the reinforcement learning post-training in Sec. A.3.
(Section B) We provide more details about the constructed dataset PreferImg.
We provide more details of visual elements comprising the preference profile in Sec. B.
We show several examples of the proposed PreferImg in Sec. B.2.
(Section C) We provide additional details about our designed prompt templates.
We provide prompt templates for generating CoT-style cold-start data with Claude 3.7 in Sec. C.I.
13

---

Published as a conference paper at ICLR 2026
Table S1: Preferimg test benchmark data categories and statistics
Data Categories
Number
Singe-prefcrence
500
In-Domain
Multi-preferenees
250
Singe-preference
500
Out-of-Domain
preee-preeesences
250
Total
1500
We provide prompt templates using during SFT cold-start and RI. post-training phase in Sec. C.2.
(Section D) We present more visualization results of personalized image preference assessment on the proposed dataset PreferImg and real user dataset PickaPic.
We show visualization results the Preferimg where users's reference images reflect single preference in Sec. D.I.
We show more visualization results on the Preferimg dataset where users's reference images reflect multiple preferences in Sec. D.2.
We show more visualization results PickaPic to demonstrate our method's generalizability to real-world users in Sec. D.3.
(Section E) We conduct further discussions on the effectiveness and performance of proposed PreferThinker, along with limitation and future work.
We investigate the effectiveness of the cold-start SFT Sec. E.I.
We discuss the transferability of the PreferThinker towards assessment and scoring of single image in Sec. E.2.
We study the effectiveness of PreferThinker to facilitate personalized image generalization in Sec. E.3.
We discuss the limitation PreferThinker and the future work in Sec. E.4.
(Section F) We describe how we utilize LLMs to assist our work. (Use of LLMs)
A EXPERIMENTAL SETTINGS
A.1 PREFERIMG TEST BENCHMARK.
To comprehensively evaluate the performance of personalized image preference assessment, we construct test benehmark comprising 1500 user samples based on the proposed PreferImg dataset. We categorize the samples into singleprrfference (SP) and multi- multi-preference (MP) groups based on whether reference images reflect singular or multiple preferences.
evaluate the generalization of PreferThhnked we further divide the benchmark into in-domain (ID) and out-of-domain (OD) subsets based on whether users' profiles are included the training data. The detailed statistics of the benchmark are provided in Table S1.
A.2 TRAINING DETAILS SFT COLD-START PHASE.
We utilize Qwee2.55VLL-77 Instruct model as our base model and perform supervised fine-tuning using LLaMA-Factory (Zheng et al., 2024). We train the model on the proposed CoT-style cold-start dataset PreferImg-Cof for one epoch. The dataset includes 50,000 user samples with single preference and 10,000 user samples with multiple preferences. we freeze the parameters of the vision tower and the multi-moda. projector, and fine-tune the LLM. We employ DeepSpeed Zero optimization strategy to handle the memory requirements of large models. The training settings of SFT are detailed in the Table S2.
14

---

Published as a conference paper at ICLR 2026
Table S2: Training settings for cold-start SFT stage.
batch size
I
maximum gradient norm
precision
b6
gradient accumulation
2
learning rate scheduler
cosine
epochs
learning rate optimizer
Ic-6
max length
32768
times
21.1h
Adam 0.1
decpspeed
zcro2
GPU
8x A800
warm up ratio
weight decay
0.0
trainable module
LLM
Table S3: Training settings for GRPO-based RL training stage.
batch size per device
2
num of rollout
6
precision
bf16
gradient accumulation
2
B 0.04
epochs
0.5
learning rate
le-6
lemperature
0.9
times
73h
optimizer
AdamW 0.03
deepspeed
zero3
GPU
6x A100
warm up ratio
weight decay
0.01
trainable module
LLM
A.3 TRAINING DETAILS OF REINFORCEMENT LEARNING POST-TRAINING PHASE.
Following the cold-start SFT, we employ GRPO-based reinforcement learning for post-training. We utilize the VLM-RI framework for this phase. For GRPO training, we use 10,000 user samples with single preference and another 10,000 user samples with multiple preferences. The number of reference frames pairs (preferred and non non-preferred images) used training is 5. The training data of the phase does not include CoT-style reasoning, since we aim to incentivize the model to explore more reasonable assessment paths. The training settings of RI are detailed in the Table S3
B DATASET DETAILS
B. VISUAL PREFERENCE ELEMENTS OF PREFERENCE PROFILE.
To formalize a user's complex personalized visual preferences, we identify 15 visual elements that most frequently appear in Lexica's text prompts and strongly influence user preference toward personalized image generation. We then conduct a user study with 100 participants, asking each to select the five most important visual elements visual preference profile. The result in Fig (a) reveals that art style, color, detail, art medium and saluration are voted as the most representative visual elements for characterizing personalized preferences. To ensure preference profile diversity, we collect rich vocabulary of related terms for each visual element, as shown in Fig. S1(b), laying solid foundation for constructing a large-scale and diverse dataset for personalized image preference assessment.
To ensure the diversity of personalized visual preference profiles, we collect an extensive set of vocabulary related to five key visual elements, as illustrated in Fig. S2. Leveraging this lexicon, we construct large-scale and rich set of visual preference profiles, laying a robust foundation for constructing the dataset Preferimg and CoT--style dataset PreferImg-Cof..
B.2 EXAMPLES OF THE PROPOSED DATASET PREFERIMG.
We visualize several user examples of the proposed dataset PreferImg in Fig. S3 and Fig. S4. Figure. S3 shows examples where users' reference images reflect single preference, while Fig. S4 presents examples where users' reference images reflect multiple preferences. is observed that our dataset encompasses wide variety of image content and diverse visual preference profiles.
hhups//lexicaaam/
15

---

Published as a conference paper at ICLR 2026
Color
Art style
Art medium
Saturation
Detail
Hound
Mid te
Vibpant
Selective
Orange
Paintng
Vivid
Art
Intricate
Aberanee
Graye
Bold
Brown
Gollage
Intense
Fine
actile
le
Green
Object
Yallow B ltee Red Black
Pencil
Realism
Expression
Bright
Soft Rightedd
Textured
Modertin
Vieaa Iimment
(a) User study on key visual preference elements
(b) World clouds of five core visual elements
Figure S1: Key visual elements of preference profile. (a) User study result reveals that color, art style, art medium, saturation, and detail are voted five key elements representing the visual preference profile. (b) World clouds show that each element has a rich vocabulary associated with it.
Vocabulary Related
Vocabulary Related
Shoor/hrrman Anshaa to
L. flodd in
Anny Mloge
Cove Crange
Ax Apireee
de Reyt Aae Ports
Hon Park
Sul Gay
10 Arkin Larendee
(1 Berape
(C Aesiuman
15 Prnerter in
Daacy Avee
12 Noude Ale
11 Mer
Nink Pick
11 Make
Fall
Deep
Chy
Gevan Cipnationa
Gollean
Sun Dav
Bust Doay
Goopler
as J.paeee de
Dep Ale
2. Regosa
Kowns An
43 Caidds
Lanal Alroctioo
Chow Dror
4 Auae Oraag
11. Nura
N. Olere koow
45. Fouplee Spor
Mesnenicee An
Bake
Drekory
Amann Lorf
Swiwwer
Coet Phuh
15. thet Googg
52 Nauradia
31 Streeprrrioiin
M.Horiamire
15 dee
Niw
51. Galioned
94 Orui
51. Orex
Papuly
Rickin
Ae
Depp Augg
Clanaa Dock
Gassuo Svee
Songgpuh fou
Rune
Owt
76 Sontelee
Deatre
M. Dadilont gpter in Oleoe
1% 1ltn
Dee
Channe!
Chiene
7% mey
Sext Mar
Aweey
Herw
#4 Aiee
Vocabulary Related to Art Medium
Vocabulary Related Detail
2 Letoac
2. Selectre
Sarlle
A Spre Pack
P.Pacy
(e Chencel
I.. Pard
12 Goles
Goglee
Come
14 Codder
Stoppsi
12 Caler
28 Strreer Mosee
Bork
24 CaNa
Dedcal Pu
Dary
Degullt
Vocabulary
to Saturation
c5 Cyandyy
Da Sowe Choong BaatConnn C Vad
Roglt
<t Cromt
Ouer
37 Shenee
34, P.
1% Govmmnnd It Draych 12 Pean
17 Howeig
twiing
12 W..ah
24 Helhy
achone 12 Oregnal
18, of
17, Paat it Dadrrasd
Dest
Nany
25
22. Vabhelene Goeds
Sourciac
Figure S2: Vocabulary related to five core visual preference elements.
C PROMPT TEMPLATES
C.I PROMPT TEMPLATES FOR COT-STYLE COLD-START DATA GENERATION.
To enable the Claude 3.7 model to annotate high-quality. CoT-sstyle assessments for the Preferimg dataset, we design detailed prompting strategy that guides the Claude 3.7. This strategy directs Claude to use the meta information,
(1) user visual preference and non-preference profile, (2) question prompt, and (3) assessment answer to provide an explainable score and assessment for candidate images. The prompt mainly consists of an evaluation template and guidelines, follows a "predict-then-asseee
structure. First, based on a reference image, the model predicts the user's preference profile. Then, using this predicted profile, it evaluates the candidate images across multiple dimensions and assigns interpretable scores. Finally, the model selects the user's s preferred image based on the total scores. The prompts for generating the CoT-style assessment data are shown in Fig. S5 and Fig. S6. Figure S5 shows the prompt for single-preference users, while Figure S6 shows the prompt for multi-preference users.
C.2 PROMPT TEMPLATES USING DURING SET AND RL TRAINING.
We provide prompt templates for use during cold-start Supervised Fine-Tuning (SFT) and Group Policy Reinforcement Learning (GRPO RL) training. These templates, which include a system prompt and a question prompt, describe the specific task and provide correct output examples. This guides the model to perform explainable, personalized preference assessments for candidate images in the correct format. The prompt templates for SET and RL training are shown in Fig. S7 and Fig. S8. Figure S7 illustrates the prompt for single-preferent users, while Figure S8 shows the prompt for multi-preference users.
16

---

Published as a conference paper at ICLR 2026
Personlized Reference Images
Candidate Images
Prefered
Image A
User
Nom preferred
iinge
Imaoe
User
Non-prefereed
Image B
Preferel
Image
User 3
Nom ere-preferre
Image
Prefeated
Image
User 4
Non-prefened
Prefered
Imares
User 5
Preferred
Imaage
User
Non- preferred
Imagee
Figure S3: Examples of the proposed dataset Preferimg, where users' reference images reflect single preference. Red box indicates preferred image in the candidate images.
17

---

Published as a conference paper at ICLR 2026
Personlized Reference Images
Candidate Images
Preferred
Image A
User
Non-preferred
Image
Preferred
Image
User 2
Non-preferred
mage
Preferred
Image
User
Non-preferred
Image
Pre Preferred
Image A
User 4
Non-treferred
Imagee
Image
User 5
Non-preferred
Image
Figure S4: Examples of the proposed dataset PreferImg, where users' reference images reflect multiple preferences. Red box indicates preferred image in the candidate images.
D VISUALIZATION RESULTS
D.1 VISUALIZATION RESULTS ON USERS WITH SINGLE PREFERENCE
We present more visualization results the PreferImg where users's 's reference images reflect single preference in Figures $9 to $11. It is observed that PreferThinker Thinker first predicts the users' preference
18

---

Published as a conference paper at ICLR 2026
and con-peferenne profiles from the reference images, and then provides reliable assessment and score for candidate images based on these profiles, which not only enables accurate assessment but also provides interpretability.
D.2 VISUALIZATION RESULTS ON USERS WITH MULTIPLE PREFERENCES
We show more visualization results on the Preferimg where users's reference images reflect multiple Preferences in Fig. S12 and S13. Our model can accurately predict multiple preference profiles from reference images and provide reliable assessment and score for candidate images based on these profiles, revealing the effectiveness of
for users with multiple preferences.
D.3 VISUALIZATION RESULTS ON REAL USERS
To demonstrate the generalizability of our method real-world users, we further utilize the Prefer- Thinker to conduct personalized preference assessment for real users from PickaPic. As shown in Figures S14 to S17 Although the preferences in real users' reference images method can still effectively extract their primary preference profiles and provide a correct assessment.
complex,
E DISCUSSION
E. IMPORTANCE OF COLD START INITIALIZATION.
We further discuss the importance of cold-start SFT for correct structured reasoning. Figure S18(a) shows that directly using the base model (Qwen-2.5-V) VL. fails provide numerical scores and reasonable reasoning and leads to incorrect answer. Figure S18 (b) shows that while applying reinforcement learning alone on the based model can generate the total numerical scores, it fails to provide the correct output format and interpretable scores for each visual elements. In contrast, by first teaching the model proper structural reasoning with Supervised Fine- FinenTuning (SFT) before RL, the model can provide reasonable, explainable scores for each dimension. ultimately leading to correct final assessment based on the total score, as shown Figure S18 (c).
E.2 TRANSFERABILIT PERSONALIZED ASSESSMENT AND SCORING OF SINGLE IMAGE.
We further study the transfcrability of PreferThinker towards personalized assessment and scoring of single image. We directly modify the question prompt enable the PreferThinker provide an assessment and score for a single generated image. The visualization results are shown in Fig. S19 and Fig. $20. We can observe that PreferThinker can flexibly provide an interpretable assessment and score for single image.
E.3 PERSONALIZED IMAGE GENERATION WITH PREDICTED PROFILE.
To further demonstrate the benefits of the proposed preference profile, employ the predicted preference profile to achieve personalized image generation in Fig. S21. We first predict the user's preference profile based reference images. Then, optimize the initial text based on the profile to generate personalized images. The results show that the personalized images align well with reference images, revealing the potential of the proposed method to facilitate personalized generation.
E.4 LIMITATION AND FUTURE WORK
The main limitation of our method is that the personalized preference dataset built on synthetic profiles may not fully capture the complex and subtle real human preferences. Our current preference profile only contains five key visual attributes that influence human visual preferences. However, real users' preferences are often complex and subtle, shaped by multiple factors. Beyond visual attributes, they are also closely related to image content semantics and users' backgrounds (e.g., age, gender and culture). Consequently, preferences, as shown Fig. S22. In future work, we will incorporate additional key factors into preference profiles more comprehensively capture real users' preferences.
method may struggle with complex and hard-to artaiiuleltee real user
19

---

Published as a conference paper at ICLR 2026
Prompt for Claude to Generate CoT--style Data (User with Single Preference)
Given user's historical preferred reference images, non-preferred reference images, and candidate images, along with the following meta information: 1. User's visual preference and non preference profile: 2. 2.Question prompt:; 3. Assesameni Answer.
You are personlized preference analysis ossistant. Please strictly follow the requirements below to analyze the candidate im- ages generate Chain-of- bouuhh style oulput:
Your final output must include the following four tags:
<visual preference profile>, evisual preference profiles, <think>, and sanswer>
Within the =vvisual preference profile> tag, directly copy the visual preference profile provided user meta informatio- in ISON key-value format, for example:
<visual preference profile>-
Art style: Manga: Color; Blush Pink; Medium: Digital Painting; Saturation: Moderate Detail: Fine
</visual preference profile
Within the <visual non-preference profile> tag. directly copy visual non-preference profile provided in the inf- ormation in JSON sey-value formal, für example:
<visual non-pree preference profile >
style: Graffiti Art; Color: Powder Bluc; Medium: Acrylic Painting: Saturation: Vibrant: Detail: Soft
</visual non-preference profile-
Within the <think> lag, based on the preference and non-preference profile, score and reason about Image A and Image ac-
ross dimensions:
style", "Color", "Art Medium", "Saturation" and "Detail", scoring criteria are follows:
1. Score principle:
Preference attributes: If image features highly match preference attributes (e.g.. rich details), assign high score (4-5).
Non- Con-preference attributes: If image features resemble non non-preférence attributes (e.g. lack detail), assign score (1-2).
Neutral features: Assign medium score (3) when there no significant tendency.2
2. Naturally incorporate keywords the
analysis:
Attribute terms (e-g.. Performance Art) must integrated descriptions of specific scene elements. Listing them quotation marks prohibited.
"The dynamic street graffiti uses spontancously splattered paint. and its fluidity highly corresponds to the live-creation char- acteristic of Performance Art."
Output format must strictly follow:
<think>
Artistic style:
Image A( [[scorc])- [reason]: Image B( [[(soorc)) (reason]
2. Color:
Image [[score])- [reason]; Image B( [score]) [reason]
Total:
Image score- (scorel + [score2] (score]) (scored) [score5]][tott score A
Image B: total (son__ ++ (score2) + [ [score44) [scorc[] (total score </think>
Within the <answer> tag, directly output final choice provided in the user prompt ("Image A" or "Image B"), for example:
Lanswer>-lmage A</answer>
Figure S5: Prompt for Claude 3.7 to generate CoT-style assessment for users with single preference.
F USE OF LLMS
We only used LLMs to polish and correct the text and grammar in the Abstract and Introduction sections.
20

---

Published as a conference paper at ICLR 2026
Prompt for Claude to Generate CoT--tyyl Data (User with Multiple Preferences)
Given user's historical prefemed reference images, non non-preferred reference images, and two candidate images, along with the following meta information: 1. User's multiple visual preference non-peeference profile: Q.uustiion prompt; Assessment Answer.
You are an personlized preference analysis assistant. Please strictly follow the roquirements below to analyze the candidate im- ages and generate Chain-of wuff-houn bought slyle outpul:
Your final output must include the following four tags:
<visual preforencee profile>, <visual cre-prrfeeence profile>, <think>, and <answer>
Within the <visual preference profilee tag, directly copy the visual preference profile provided in the user meta informatio-
n in JSON key-value format. Please note that users may have multiple preference profiles. Here is an example: <visual preference profile>
1. Preference profile
Art style: Manga; Color; Blush Pink; Art Medium: Digital Painting; Saturation: Moderate Detail: Fine
2. Preference profile
Art style: Ceramic Art; Color: Vivid Purple; Art Medium: Acrylic; Saturation: Bright Detail: Focused
<Ivisual preference profile>
Within the evisual non- non-preference profile> tag, directly copy visual non-pre non-preference profile provided user meta inf- ormation in JSON key-value format. Plense note that users may have multiple preference profiles. Here is an example: svisual non-preference profile >
1. Non-preference profile 1
Art style: Graffiti Art; Color: Powder Blue; Art Medium: Acrylic Painting; Saturation: Vibrant; Detail: Soft
2. Non-preference profile 2
style: Ukiyo-e; Color: Hoi Pink; Art Medium: Spray Paint; Saturation: Grayish; Detail: Blurred
</visual non-preference profile-
Within <think> tag. based on preference and non- sun-prefeerenee profile, score and reason about Image and Image
ross five dimensions: "Art style", "Color", "Art Medium", "Saturation", and "Detail", The scoring criteria are as follows:
I. Score principle:
Pre Preference altributes: innage features highly match preference attributes (e.g.. rich details), assign high score (4-5). Non-preference attributes: If image Features resemble non- ponn-refeeence attributes (e.g. glcck of detail), assign a low score (1-2).
Neutral features: Assign medium score (3) when there no significant tendency.2
2. Naturally incorporate keywords
reason analysis:
Attribute terms (e.g. Performance Art) must be integrated into descriptions of specific scene elements. Listing them alone or in- quotation marks prohibited.
egg. "The dynamic street graffiti uses spontaneously splattered paint, and fluidity highly corresponds to the live-creation char- acteristic of Performance Art."
Ourput format must strictly follow:
<think>
I. Artistic style:
Image A( A(sccore)) [reason]; Image B( (score)) ) [reason)
2. Color:
Image A([score)- [reason]; Image B( ((scorr))) - [reason)
Total:
Image A: total ccore-[score |+ [score2] (score3) + [score4)+ (score5)= score A]
Image B: total score- [score2) (score3) + [scoro4]++ (scorce))sscs========-)))))) (toorcs)=ttotal_sooo score_B </think>
Within the Sanswer? tag, directly outgut the final choice provided in the user prompt ("Image A" or "Image B3), for example:
<answer>-mage A</answer>
Figure S6: Prompt for Claude 3.7 to generate CoT-style data for users with multiple preferences.
Prompt Template for SFT and RL Training (User with Single Preference)
You an personlized preference analysis assistant. The user provides set of historicall- y preferred and non-preferred images. Based on this historical data, please predict the user's 'se's visual prefere- nce profile and non-preference profile across five key visual elements: art style, color, artistic medium, sat- uration, and detail. The predicted results should be described using terms corresponding each of these f- ive visual attributes. Suu.buues..usssuuuttly... the user provides two candidate images: Image A and Image B. Using th- e predicted visual preference profile, assign an interprotability. score (0-5) for each of the five attributes for both images. A higher score indicates stronger alignment with the user's preference profile, while lower core suggests greater similarity to non-preference profile. Each score must he accompanicd by a rationale xplaining the rating.Finne Finally, sum the scores across all five attributes for each image. The image with the hig- her total score should be identified as the one the user is more likely to prefer. Below are some examples:
System Prompt:
Reesoningg-bas output example 1}
[Reasoning- bassd output example 2)
Question Prompt: Given a user's reference images (both preferred and non- non-preferred images) and two can- didate images, first predict the user's visual preference and non-preference profile based on the reference i- mages, then assess which of the two candidate images better matches the user's personlized preference.
Figure S7: The SET and RL prompt template for users with single preference
21

---

Published as a conference paper at ICLR 2026
Prompt Template for SFT and RL Training (User with Multiples Preferences)
You an personlized preference unalysis assistant. The user provides set of historicall- y preferred and non-preferred images. Based on this historical data, please predict the user's visual prefere- nee profile and non- non-prefcrence profile across five key visual elements: art style, color, artistic medium, sat- uration, and detail. Please note that user may has multiple preference profiles. The predicted results sho- uld be described using terms corresponding to each of these five visual attributes. Subsequently, the user pr- ovides two candidate images: Image and Image B. Using the predicted visual preference profile, assign interpretability score (0-5) for each of the five attributes for both images. A higher score indicates stronger a- lignment with the user's preference profile, while a lower score suggests greater similarity to non-preference profile. Fach score must be accompanied by rationale explaining the rating. Finally, sum the scores across a- Il five attributes for each image. The image with the higher total score should be identified as the one the use- r is more likely to prefer.
System Prompt:
Below are some examples:
leasoning-based output example 1
|Reasoning- output example 2]
Question Prompt:
Given a user's reference images (both preferred and non-preferred images) and two can- didate images, first predict the user's visual preference and non-preference profile based on the reference i- mages, then assess which of the two candidate images better matches the user's personlized preference.
Figure S8: The SFT and RI. prompt template for users with multiple preferences
User with single preference Preferred Non-preferred
<visaal proference profile>
Preference Profile Prediction
Style: Conceptual Color: Electric Lime, Art Medium: Paper Mache, Saturation: Crayish. Detail: Simplitied
sivinoal preference profiler.
Non-preference Profile Prediction Art Style: Impresioniset, Color: Bu Buryundy Wise, Art Medium:: Watereolor, Saturation; Vilund, Detail: Rough.
cvisual non crof-pefforecceeeee profile>
<niinall now preforence proffic-
Image The cartoon character depicted under large Multi-dimersional Scoring and Assessment
-ohink-
Aet style:
petal cslabits quilities Llusttative design than Concepoual. Art. Its whimsical nature scenee soft focus bockground lacks the imtelectual framework and provocative monentary sppically found in conceptual works, Instoad leaning soward the peraativ-foowsee storytelling seen in impressio nivr appraackes where subjects take procedence over shstncct corerpts.
Image R(4)- - The black cst figure positioned ummatically the addimar clameta foon estranndinary emiiosmental scale, delibarate sulversion propections chillenges viewer expectation through intentional absudity hafloarr of c'cucopinall ave that uses fumilizr objects it unespected co- mexss to provoke thought about perception versus reality relationsipps 2. Color:
overizzed lotos lowes enoric a striking jexisposttinn betw-
Image AC)- The deminant bergundy-red flower petals overshelm the composition, peecisely matching the explicilly stated non-prefernnee Rargundy Wine While morssadoosd by the parvaiie reddish--pik hucs dominatc the entite somne, creating excetly the type coke envisor- ment specilied as undesirable;
himts time-gram parts the tack vegetation, they'r
Image B15). - The promanent use of bright eleenic fine green lonus leaves perfectly matches the peeterred color uttrbuue. The ese vivid green elements stand out boldly against the neutral bockground. creating just enough contratt highlight the lime colored compenents while allowing them maintin prominence directly satisfying stated preference.
3. Art Mediam:
Imnge A(2) - The fuid, transparent quality of the flooes and backgyoond saggetts wateronfor trchniguer with their chamel- bleading edges
layeing ellects. way pagment appears flow naturally acound the chareeier pread orpanically acroas surfaces bears swong resemblunee wutcrcolor application methods spesitically mentioned non- preferred;
Image (64) The sculptural qualit of the car figure and lotus leaves gives them dimensional solidity reminiscent of molde- paper marke constraction, The clean-elgol forns lack the fluid ramuuuneccy of votencolers and have erifted, assembled quality suggestigg physical manigulttion meterial miker paint application alone
Saturation:
Image Al3) The image fentures hight- vibrant, zichly colored flowers with deep crimson and magenta tones that pop dram
atically against the softer background This level of color intensity directly contradicts the stated preference for gnaish, sub-
dood
values, instedd presenting hold chnomatic statements draw immodiate attention;
Imagge R(4)- - Despite having colorial elemenss like the lime green lavees, ovcaal presentation bean lened down with
poolaminnally pal: beigereeam backgroand and subdued lighting conditions. This restrartt is color intensly cratiic a mote understuted mood that better aliges with the proference for grajish, loss saturated compositions.
User Reference Images
Detail:
Imuge A(3) churacter is randered modertte detailing- cycs, bosic ficial structure are cleatly defined
Image A
Image B
implified compueed hyperrelliticc depiction. However, the background contains sumerous small details including individ uui flower petals and scatlered debris, creating middic ground benwees rough sketchiness and meticulous claboration; Image D(4) The figure and lotus leaves feature deliberately reduced details focusing primarily shape and silhcuette inrriaae textures. economical approach detail creates clean lines unchmered spaces that align he preferonce for simpliffied represattaioons nother than claborste ormmentiiion.
Total Score:
Image A: 2+1+2 111--11+-0 (+3-0; Image B: 4:5444444--21
<0hhlkk>
Candidate Images
Sunsserr Amage Bi/anows
Figure S9: Reasoning- personalized assessment for users with single preference.
22

---

Published as a conference paper at ICLR 2026
User with single preference
Stivaae proference profile-
Preference Profile Prodiction
Preferred Non-preferred
Style: Indian ARt Color: Fem Green, Art Medium:: Cerumic, Saturation: Natural, Detall: Tessured
<Wvimel proference profiie-
Non-preference Profile Prediction
Art Style: Naive Ant Color: Slate Blze, Art Medium: Pastel, Saturation: Vivid, Detall:: Inmreste
cobbaa non-greoorocce profile>
Multi-dimensional Scoring and Assessment
LAAt style:
Image A14)- istriaic mambiallkke pallem with symmetrical design elamants and umale deseling ryflects espects
fox der anstleter thrugh eistorate circular motif and speriual gymollinn w While not purely aurhernic Indian Ant the mppoction with concenric cirlles mirres iradtiional mendala designs commonlyy found in Indian religion iconography Image BC)) The simplifeed. childlike repeesettazion with disproporiinutte facial features and lack fornal structure beurs stonng rescublance Naive Art appeathees The siightffrood depistion lacking suppieiicatol penpactive iechriques and convertioall proportions afiger well with songrrferred Nutre Arr aesthetic.
Coler:
Image
rankly greass and nuteral Firm Grece color These oeganic hucs croke ratund unvirommani und tnaiitional pigmants uved in Iulian setwerk, crestnng sense of autherriitty
dominant
dhsoghout the piece cloaly mutches pegferned
Imuge BC2) The pecmisent use of slute blue backgmunds combined with reddish-rewn accenss creates a color scheme that
dinectly corresoonds in the ann-pogfereed State Elur attibatte Thuugh thene may be some worma highlight, the primary odl
impression remaiss fimily within the non peeferred color range.
Art Medium:
Image
solid, dimencioral quality the piecss sugense conamic contrantion muthods with visdtle thyy and
glaned fanns The way light csiche different parts the sculptuee rovesls the noterial propertiss typical of firal caramics
oligning with the prefereed medium;
inuge BC2) The sect blending elfocts and chally application secinique evidest in the portrall clearly indicate pante asage. Th
he powdary finish and shility to bladd colons smontily are ballmerk charactensiiss of peecil drawing toos, matching couely
with the non peeferred medium.
4. Suturation:
Image
speer naturally balanoed without appearing intificially enhanced averly civid. cethy
aintiin natoral sateration level aprope iate traltionll pigmenned usseriala like tlrse usei in Iulian srtworks;
image It-)) The image falls sontrvhere deneeen natural and shidd with modentely sonumted blues the background that hy
cut soward the nou prefemed sivid calquecy but other elements remaiin reluively sabducd. resaltngg in mniddle--rann socnd
S. Detail:
lmagee The sculptrral exhibits richty soxtared serfaces with varied tactlee qualities crested through what appeons
User Reference Inages
be
molded techeiques. diseenionnl work showssses intentioall iestuing that invites louch, perfectly
Image A Image
Image BG2) Despite being a pottat, the imagee relies heavilyy on lased details zaher that testural ozes. shspltfted
sthouling profermed
pproacch focuses primerity on outline work indeadd of building up dimeniooraliyy thrugg textural means, aligning closer to p
he non-proforred amrihue despii having some testural clessens.
Total Score:
Image A 4---444---1111 Image B: 2-242-342-9
<think-
Candidate Images
Image
Figure S10: Reasoning based personalized assessment for users with single preference.
User with single preference
staioae proforence profilec-
Preference Profile Prediction
Preferred Non-preferred
Art Style: Conceptial Art, Coler: Mint Cireen, Art Mediona: Macnme, Saturation: Grgiih, Detall: Testured
<isimaal proference profile>
strisaad con-rrefirnce profile-
Non-preference Profile Prediction
Stylec: Absercct Exprasiioiso, Coler: Doey Onang,, Art Medium: Drypuit, Saturation: Viteant Detail:: Simplified
<ivisuel non proo-rrefrrecee profiie-
Multi-dinennsonnll Scoring and Assessment
Art style:
Image A64) The mins-colored mouse character sis atop what speers be a stoee polestal with ogz deuil, preseningg
nncepial jectspoction where overlaa objects hoome symbolic repessrtations thrugh deliarne placement and context Th- Is shunlllff arraggmmeu aligns well with Concpntaf Arr prisciples ficusing ideas
aemteetis:
Imsge Br2)- The crage-huel
ugaint wann-lencel
eshibio gestaral beashstnske spontoroous
Amaking quolitis characioistic Absoart Expresiinionn loose emotive application of paint expressive reflect the non-preforree alspact expressionat acothetic rather flin conceptually driven aspecocles. 2. Calor:
Image A13) The dowinonr wint grees color palete perfectly monches the peeforred color ottribute. from soft toaf-greew b ackground to the pale wint bucs of the mosse inelf, every element incoepurates varions shales of mis green, creating a col- cavee and plansigg oolor wcheree that strinly sligns with stared preferoncen;
Image BCl) The image prominendy features duery orange towes droughose the backgroand and parts of the nonse's deds,
precisely matching the preferefl color amribunc These wamn umber-aange lazs domistte the composition, creating the of color anvinemmnnt specified indeurable.
Mediun:
Imtge A(4) While peintrilyy appearng a digina Illustation, the image incorporaees actual physical testures woven tope sound bacc the pedcsial and the Lacill-booking für the mussc These sungbble muterial alemonn croke the dim- cenional guality of suarmaes euggesting cnff-toend creation matheds degiit bring diginlly rinderdd
image B(2) The image uses this lices and shap commasts remdnóscnnt of dryoiit erching fcohniqwes, purtcollrtly evidest in the definaal outlines of the moiee figure This lincar spgrssh cantllal murk-nsking duros sinilaniias with the non-ue- femes drypint medum,, lacking the dimenconal testle quality mocrime
Saturation: hange
color polctic with sofienal min grams and sabele yours the backgrou- nd, coecring a subdued atmosghere. restnined use oolor intencry aliges well the proferred grajish saturation level alluwing foem lestue to take precedance over coloe streggh:
image emplayu adiibenttll
image Bt2)- The image featrees Aighly vibrant colors with strong chramanir intenalty porticalerly in the bright onagge back cund and vivid pitk ozr accenss This beightened color energy directly cocrespands to the ton, preferred whram satuation ottil bute, making the image visuily slmmlaing but conzary to statel prefenences. Detail:
User Reference Images
Imeae The mouse illustration showcases rick goxuunll eloments including the intricate whiskers, fivzzy
and
the dimensional rope wapping reward viewing- perfectly embodying the preferred tectured decaling
polotal. Thase tuctlle aarfaces
ineuastio, offaiig mulliple layss
Image
Image
informotion
Image B(2)- Despie having recognizable subjects, many eletoents opscar simplijted with mivinal sestargl varindin. The usee's fast and body lack asoured surface dennll, indicsd reying on bacic slopes and limited lonsl shills This strcenlinal a- ppecoch alloe with the won proferee sisplifed detall amibutt.
Total Score:
Image A 40dd444 5-21; Image R: 242224342-4
<ahinn>
Candidute Images
Figure S11: Reasoning- personalized assessment for users with single preference.
23

---

Published as a conference paper at ICLR 2026
User with multiple preferences
orisnal proference profile>
Preference Profile Prediction
Preferred Non-pretiereed
1.preferncce profile 1
Styte: Gafli Color: Trogical
Graphite Vivid,
preference profie
Color:
Medium: ation: Detall: Realitii
preforence profile-
cntswal proference profile>
Non-preference Profile Prediction
gre-ereferceee profiee
Art Stylee: Performacc Art. Color: Sülver Whit, Art Medium:: Quilting, Saturation: Desnurtel, Detail: Absincted
proference profiic
Medium: Virual Realit, Saturation: Highly saurtti, hnciisall non proference profilee
Mitinaiimn,
stkink>
Mulii-immensiondl Scoring and Assessment
A.
Image image wizard-ike figure teal
styiied poraa conmening clemente alles perficily with Grggit wher acahati snuhilito,, The pontuit doos centem-
drpptin pain
pocacy wist naditional imagery with is vibrant blu color sihene and ttamatc presenzation, reniniscest of sreet muro. imagrry.
that mandfrn
sthpets
Image BC1) inage diglays quilted testile pottaat monochommti sile: white wit acometic pattems and actte cleneats. The highly strottred, thee dinensioaa testle approach with is carefilly amarged quitescarefulll orazged quilied sections and decorative bution deneess clurey resembl Performance int intallatoo acabetics
Color:
lmage A)5) imags prassiuetlly foetunes miiking inalhfue poletly duat deminnies cmpusitoon, from nts hat to the glases and flowing bead. This rich impical leal crsts a viluatt foxal point spiitt the motochmmuis dience- sisual bicnashy The ooker perfasly mechos profered "Toopcaal Teal" stmibun: cofor effezt This allerrnce coleriess paletic deecdly muches sco profared "Slver
cxtbliching
BC The image
manocinmeiic ictile
color utribun alsc-
nee any color ridtness enaice clinical merrmee apponee: the conradicts the vituat orher profeenzze.
Art Mediam:
Image vosuace The pratisc
image
graphits-llle rendering drdipppss croscs hybral m-
facial foeurze combinal fhe painterly
edium approach thet incoppentes grapiie-style renderngg winh digital coler application.
Image Het)- The dipplas gulleed tealle artwork with pronoemced eltching panamn nd deceestive batuors The entir composttion Sabric manipulation tecluigues visille testure dinersional elements charictetiste Saturation:
Image A/5) The inage ethibis anease color saturdion in the teal blue elenents that create drunati visual impacct against th. motocimmnatcc
The whrdd blue
beand
sarking coler
User Reference Images
Image imare digllys compleely devetunated polene any values to create visua interest. The abscace of any color saturation creaes stack, clincsl appeurance.
color information, rubing solely white, an
Detail:
Image A
Image B
inase employs selectiv deall higk elemonts anougic concetration allowing other area to resale.
faciel crratos bicuncy pridee vieuer atiention to boy fiatroo
hoard while siepfii
Image (e3). The imagy dlyiaas obstranted denall with gesmemtie and slaptifted focial fratanne rendered in sectilo sctted oppournttion omeepuueily shameted simplifiaal
clke. White sodinically detaikal
Image 515-4-5-4-25 Image 1+1-1-1-2-6
<Whkik>>
Candidate Images
Sanswerr Image A Aommsees
Figure S12: Reasoning based personalized assessment for users with multiple preferences.
24

---

Published as a conference paper at ICLR 2026
User with multiple preferenees
<xtonol prgfooneeprefiie
Preferred Non-preferred
Egrefronce prsflle
Styke SeroqacAAt, Celan Foros Grea, Medianr Stene Carving Saturther Modcak; Octad:: Alsuadal prefirncce profil
Art Styke Noolggooseiinn Codar Dusty Rese, Art Medare: Sxocware Sattratken PepuoCiodi: Detadi Reslitt profereac pesfiic)
Style Avdlani Art Pouds Art Mediumn Asveribigg. Sataration: Clorc
<ovtean profenewee profiles
Preforence Profiie Prediction
ion non-grofrnterr profiles
Love-prliennor prafile)
Styks Revisseni
Depp
Malken: Manscys, Saturation Greyils, Detadt Scactive
ovs-grrfirmst pesfli 2
Air Stylec sockent Coodk Cakor Tersoor,, Art Median: Wend UioddCede Candaag Sateratian: Teaorse Detal:: Tecrord
oos crn-greffrmsse proffic)
Stakk Bed Piising Cotar: Combaryy Rei, Median:: Vinual Railt, Suturation: Naaral Dctal:: Pocuac
chibeal www-girrrrr profftte
Non-pireforence Profile Prediction
zagee styiisal pigial parmei with
prilifuraa gloses Heke backgaoenl
sieplifial and cake
ckeplifial
Relt alge with suders illonvatios.
foorase aherrn decertive francin compostee rensenccnt Acaderic Art' farnal the figane roluns duruper eaplook drweati ammrnnaaaion and ardistucuual frining dickcl
frone dececaiv sandbawk and and acchectees cleasrac contexaion
of sabjeck nalleti
2. Cake:
lomac AG2 The image decinasdlly a does Mac ncnadavenat pelcts
wootts This conittnnn biae pilttz olendy reuesble the
Doop Sie tlis disnd be the mnngrafinnce alrihains conting
consemaacy odlr schena.
lamac Bil Tacpurrat emplays
paicai faskaing puach puldeee siin spaied scaral uncitostanl clenen.. The wamn dioally alge with the Posch prefarace crrking hurn bukane bat the wdjat and samvnding doundii clenetts. J.Ar Medium:
troas Au2) The zaae appcan be creaed saing datal Btarette tcchiigues with encedh sooedhaniie gndiina and the coke avas The enasst, scrcaao- and sombisc oratts a consequency fed that doout' cspaae the plesica mascridly and cathourndi dai asecase with the pesfored aristi maada
zu. auteul crmiing a more siraal ronuts-onna cxperioner oypical of contopprrryy Alyied permuttee
lmmge Bi5)- The amge provent socipitan cualky with discesocnd cameol dienner frawnin the poraus. The touanes of dowenii anddl work and axciicttaaa chanatosinegly eockee shere carvnyy iichaipon with due danamiced edhid quaitii and bostaa variaiion The conmes belwees fhes smoods parmek orea and the tooarsl cenac medti-iinenioell effer.
Sateratiee
truge 407)- The waage foorre highly on te tom theraghre with rikieat cuiatis
lnneadty This oppoooh cwwess goy plic quality tha lig clouly with dhe Asseser socouation sccetieed sve proflrwec unillsson.
User Referencc Images
aity alorss paldichine gruer jucket were tocal poire withoer coanthattinng the compoction, coating scmone pape onler against more caural anchiuctand clement.
trags Def- The pormut derrenttove raddenne conpatice with clear oker deficitcee and ollctts emplise The halanced appraach inan
Image A
Image B
5. Denal:
trage 407 porro replays deeillig forvilg foomees glanee, and jewvbyy clepliiing Theape
pat
prickely
goitchuis
image
thrragisee purteclarty thoraghet stoupl
approuch 4-4rdhar-4221
Candidate Images
Mulii-dimensiomal Scoring and Aveesment
Figure S13: Keasoning-basee personalized assessment for users with multiple preferences.
Real user from PickPic v2
corsmal preference profile-
Preference Profile Prediction
User ID: 4095
Art Style: Fonunsm, Color: Ellcttic Lime, Art Mediums: Invironeental, Saturation: Vibrare, Detall: Intrictte
<ivisnal preference profile-
Preferred Non-preferred
<utouel non-proffronce profile>
Non--refeeence Profile Prediction
Style: Photorelism, Color: Mossy Grees, Mediumm: Photogram, Saturation: Soft, Detail: Ethrrcal sivinual non-profirencc profile-
<abdet
Multt-dimensional Scoring and Assessment
L stdl:
Image AC4) image showesees
movement through shoract gonmetic forms bold oveke sped
enengy kallmarks of Faturiso'ss calebration of sfichnologr and mution. Thcse sugihe cumpositions brok from static to
qpresentztios, unlike Photoeralsm's literal depiction reality:
Image B:21- While fhere are some stillaed dlmnens, the image predominaull anes photographic realisre in depicting archi
natural stmectres and enviromemall detrils, leaning cloue in the con-grefereed Photoratton coatetic where objects appo
eapurrd euacly Bay would photognplicaly.
2. Color:
Image A(5)- electii
domisate composition with briole yellow kighfisht euergize the catire
place, perfectly watching profereed Eleenic Line color atribuue while completely avoiding any mosey green tones,
Image B(3) image contaier some brighter green elemonts vegetstion, but also incorporati muny carthier Jonus ies
claving vome manty greens the fandscapinng placing somewchree between the preferee vivid lime color and non-peefo
rred mosy greess.
3. Art Mediam:
Image A/4) uttwakk appears designed interatt with physical dhroull anrironmental integration goome-
mic forms
influence sponial poropptian, aligning well Envinomentil principles despit being
primantly grppiic
Image BC2) The image keavly relles on photographic documostation tcolniguess with minimal manipulttion beyond basic
fraiing, sharing similaziies with the non- preforee Photognun mediun's dreet recording aature.
4, Saturation:
Image A(5) Colors througout composition maimain hight chromatic Incourty with silrast pollows, bizes and reds
strong
impact, perfictly embostring the Filrani saturation specifeed in preforences.
Image B(3) image balances sodorately saturated elencun with safter stnopperii qualitics skasdowe areas, resul
User Reference Images
ting in midble g gronn betwoee the prefered Viheatt and non-prefenee Soft satuntiin levels.
5. Detall:
Image A
Image B
Image A/41 The inmicate interplay a/averluping goomenic shaper creatos complex layering offects that reward close ins
offering mulipple layers visull information like those found Inticate detailing:
sull-oaus Ansely Extereel deseliig montioned in non-prefoentee.
lange BC21 Many
blumei boundeiia enaeing dnsamlike quditic alige
Scorc:
Image A: 4:54-4:5-4422 Image B: 2-3+2-342-12
<ihkind>>
Candidate Images
-aanswern Image A <ammerr
Figure S14: Reasoning- based personalized assessment for Real user from PickaPic.
25

---

Published as a conference paper at ICLR 2026
Real user from PickPic_ v2
<xtmaal profennnce profiie>
Preference Profile Prediction
Art Style: Misimaliam,, Color: Flasric Line, Art Medium: Legn, Sataration: Grayids, Betall: Definal
User ID: 4503
sivtaeel proferessee profiic-
Preferred Non-preferred
caluss cun-peefonencr profile>
Non--rreference Profile Prediction
Art Stylez Anime, Color: Clay Gray, Art Mulium: Teuper, Saturation: Madenie, Detail: Foausea
shiimal nongreefeenee profile>
Multi-dimensional Scoring and Assessment
L.rrtstle:
Image A(4)- The srehitatural strutuee in haage A echihiis strong winiimalist principle lhnughh clee lines nalnood firmal-
ity and finctionall simpliit The buildings gaimarie choigs with Lens gloe panch and minined unsicitation aligns soll with Mistmaltr aestieter where excess deliberntely elminsted unisse Anime's dabomit choracter designt
Image BX3) While Image doese't explicilty feanue usinss spple characers, is wann lonel interir some with soft lightn- and doomins dlments divergee
eeeerggeiinnieennn significntly from pure Minimalier.. The cory semospherc with firmiuue detalls and pas- ond lncher noumbla aeacii of dunzeic spaas ofton depiened animesinpiral illustationn, making cleas the peefereed sesthetic directon.
Imoge The prominent heigtt
cambined rolletiive glase crosios
quality that agen-
Flectric Line's brightoons potontial illuminntted propertj: These crisp whites the smak comnass enhance stretui clanty:
Image BL3)- The interioe seene atlizzes warmer earslyy lower inclading clay-aray slewnn wall textures Doocing.
pertially matching the noneprefemme coler atribite Hewever, there also brihhtr secemt oolors present the firrishings precent completc sligmment with oiher profenenes estrane.
S. Art Medium:
image AC)) - The photogmah captures what appeors to be concrese constration with indestnal finishes, which shares sonte modnior qualitcs with Lege's systomatic useeniiyy pelzsiple desphe being built difierenty The strucured nature of the arch- fezztre offers goometiis putterus that could mutiplanically relote Lago's Black mnnaiiatiin
B82) the intenor photography reveals painted surfaces wall tresmems resontlle emhede usef meditiona polsring The finidi byyrrd application tochriqpe the wall decorations reflect peopecties mentioned peffrrefarenes..
apalicattion
upena's spaque
4. Saturation: Imuge AL4) The prodomisantly whic facade with ninimal coloe varistiu achimos graprihi nonr due to overasai conditions and sholowing efferrs. controlled countion level sllos the stuchral olenents to aypear smplitiedd and inchstoeed, al- igning with the preforeed grovisht satvaaion atribate.
Imsge Bt3)- - The intetie stme balanes modense satration in woeder clement and falric componente with sume artas specaring mare subdued. This mishle- ground appoaac placer i spuarely in the non-greferred muderate saturation cangee- locking both the desine gragish indertoees and the prefered clectric lime brigttres.. S. Detall:
User Reference Images
Image A
Image B
A44) The archttectual photogmpp clearly defines ncturrl elewoots like windores,
and material benndar-
These distins delinzation illow
idaniify differani companetts the building, enbodyyng pezfaned
"defnne"
forused detailing spreffic elraoan fimiinr presos decorative Imago RK))- The incorpoettes becader comestual information cbout roem lopout and sponaal relationships. This balance between focus-
detail and em anvinnmmnaal united places non-preforrel anca.
Fotal Scare:
Image 4:44 31 4-19; brage 21392-993-13
Candidate Images
Sammeer Image A Anlonseyrs
Figure S15: Reasoning- personalized assessment for Real user from PickaPic.
Real user from PickPic v2
proference profile>
preference proflie
User ID: 4538
Style Postrudm
Sont Segn, Art Mediumn Saturation: Gryy sh, Detaili Tactlle
2. preference proflie
Preferred Non-preferred
Art Style: Poarillin, Color: Mist Greee, Art Midiiun: Auyie, Saturation: Vibuas, Detail: Selective <ivisual preference profile>
Preference Profile Prediction
<oatsoal now preference profile>
I. non- cutt-preferecce profile
Art Style Pop Caber Tarquino, Art Mediamn Digial Pasking Sateration Highle stansed, Detail Sieplifee
non-preference profile
Art Style Alease Expesixiion Colar/Wian Tomaont, Art Modiune Texpera.Sateraiin Unkerzadd, Detait lasiese
<hiiaae non proferencee profile-
Non-preference Profile Prediction
sthink-
Art style
Inugr Acts trane A depies a gray taity Litea with Nes eyes eendaree in bysuerestiii yet clightly an low moreer The contill stvuniee to
arcand and disonnkenl uadkinng refleh prstennire an's insdency in Moce malty with sleviiative The Enegirtfe companiion wit the cal
prslieee courally quinni will Kansl
Agreandd deman into comparicnaal sephini typial pooaneies wollo the chullcnge
servetional cratonal
with tright uuquiie tedippozad in lighly faod deeaet catocce like manner The cragammiad rises sirplilled lacal fastaro bold poke blockng iochnique streaa slign with Pay fnt analatcs wheure everdday suljects aleatial prasosatiio
BC)) image alann
2. Caler:
het fettees
kerdgrsssal thet perfocaly eunthes the prrfered cake unriuie resned grves
wooting tacistop entondra the nater whtle ring furmany exmual cake wirla gratetion badli
appiaiinnal landing becaning uncrpowering
hange lunge
whsd Aongusie buckgnnns thet protielly
atrikaee The
figes, canting jantag
Medium:
seckeipuess pericalatt
oanissed with rid,
apelcction capsine lggs artoeett angln
alloring prcces desting parmenty way light tesenne tajeec
posutlle
troge trange diploys belloerks dighal painting orig olpes cniform
and compate grocroed
faa agpi
chary fellenation
conterperayy (igial sefhrigees revesivily pop inagrry
longe lingg aaplap uoousined sttnaindd-attaiiiinn sialogy where und clenata ransis unddant to aduceed valae sagges owept for stri
king Haccyes contoilal agroncc tocollr inlunity crates contexplatrc mood whik alowwng entain fool puina Ohk cyca staed
cauralyy 2he grarisk quality asastioned profermmon everger saiity to essyy ancns keywed jur the bockground
User Reference Images
Image BCD image feuteres eomonntyy Atpp vntoonation dersughioaa pasticluste is the laoqusic hackaroand the vihrar eaage aold
bevaue coke mommmntt crases invnctions vieas inspact her centiecs storls with buth postoned cercaton unitrres "iayta" and "Vhhear Isos
encodte eartyy the "illyly comurasee quality queriiially laned snonng non preteree anbiaass.
Image A
Image B
S. Detall:
Irugr hels trage dovessstaate selectiv dewallig whew oriikal derrens noci reee foos an fie white other wees roluvis sotte reade
riu This sperodd avaac -oaal hirnacyy and daawe voeve locar to importus fanaes The tasile usalty masional is prefirecco bocoso cni dos copciaaly is low indoidaal stnnns of la sccn puppttlc ever vina niceel datialyy
Ionge BC2 bang enyohr a vinyliiiol apyroach in dosa'' wil ashood bostuing and flatadd Wlsk zeinsoaly prdiciast imng
pontuee boot shapes and coke relatientape over seanect eatfice battees.
Inage At ord 2---4--r 4nere-dd 21: Image B: 3-122
Candidate Images
-annserr Image A Vommmrr
Mulri-dimensional Scoring and Assessment
Figure S16: Reasoning-basee personalized assessment for Real user from PickaPic.
26

---

Published as a conference paper at ICLR 2026
Real user from PickPic v2
coboae Lpeeteeme profila
ooefete proferonce proflles
User ID: 9128
Artiiti Secal Rellon, Calor: Art Median: Saturation: Gaysa, Betxi: Thate
Preferred Non-prefemed
sivisual proference profilee
Preference Profile Prediction
ovtiaal non-proference profitee-
perfirmcc profte
Art Styke Postroker sn. Coler: Mons Bur Metium Waxreske, Saturaive:: Boxl Detail Selective
con-peeliemes grddle
CalanSlar Moliaan Grspbis,
Denal: Sasplfial
shimml aon preforecce profile-
Non-preference Profile Prediction
The smsigntieenaa
Aoshins
bodding fgom
yptual eaplaats
appers
Trage roonoroonnic prosodle prlote sabée beigotn hechoggreedd
sandones
onas
dauze cletanng sumerdiip alye
aecund profevnoor "Sontrraee" Deptic being entiadty
macoodrome,, imagge muinaano gustic warsis thet didinssikko froes cocke any palttes.
Image BI lange listace pronsacal eomoy goise Adlage
the ontrainng sidnsssaibseerss wooks
viskohci-
clettny consiination of eaneyy Arawas and dky goows directy consent somrenggonn coler anothanee The coolerned grean doe- compostton, crezzig cuacty type
whene spactiicaly moted
Medium:
tmane Acty Image vokibble wonk wloes Bons king and rour fun klng oedhiiguer daramettak of gropille poucll work purtistarly exddent
low light and ados ure rendeedi facal pusioss and fabis Sods The lave sallly und lueal grodition procady ranch the finst profosnnc
atvibato contocinn "Graphie"
Image BCT imaee diyvler fuaid warker inangeneet milu oysical watoraaer mechsipas, expecially ansocable in Booiing pvece
cleas and sof munatoes around the fiane. This modiass chicky dreele concspadd uo de fins ausgeettrcns staibasc iitia "Maazwoeer" with diatativc musluent qailyy und deilly to bieed vokes warkbey.
4. Satoration:
tmage Acd hmags employs restaned undertacdd approdh to value contnat with traned treal tange. The grapodhbbuug palezz with autsia
arorson croaes contorplon platve rood thet oligee wish the fioe pegfoveee anoihas wating "Goglaa" sutsratio The imaage soolds both oumnan
ceatie and compler recantation steingg roible gronn that fidl weheate yeu invrivuily moraan
User Reference Images
Image B:2) lange feseeonn hold, coloration pariiallr agens like the brigh: placokan und areas fellage The colera eepogaini arcuercattio roed energy contralic the muse velloautii profernnce. Detal::
Image
Image B
arudi-anll deaia wrtakk of the old mar' fass, tomure of ho ckobina, and indivada- of the vinaoe fana: The rxcicdlns appoech dleisting maande ckeaate alyes with die prddirmn aimbiu eguenlin "Tezale" of gras oxteas
inaae
drailing The
thet
hare.
102) oaker velocahr
amoll conegoodd www.eeff
empliynng oberr stemunee rasted aestioning "Seletiee devi
highlighiing
relviniidng lare bevilaenk pricetioe revelona S-4e5me-4
cevooee settring
2+1mi=2x2
Candidate Images
Multi-dimensional Scoring and Assessment
Figure S17: Reasoning-basse personalized assessment for Real user from PickaPic.
27

---

Published as a conference paper at ICLR 2026
User Reference Images
Candidate Images
Preferred
Image
Toghh
Non-preferred
Image
ART ARCEENT EGYPT
Sigh
Selle Rrweiow Coke doel Duy, tes Mochaes
Profermncn Profily Prodiction
Proflee Prodection
profinouu
Now pre-prefeeence
Seylan Afriie Egert Coleri Goce, Methen Deppeit, Sanrrber: Sabbuo. Deal: Abrrrect
Chinnel profirmee
you
strler
Malti- Malti-dlineessen anti--lleeeeonnll Scoving and Assenrmennt
mecalea where rignes wore doy not in porkic vee Wille thrch some realim the anrsi dip coe the an mvaarat Puly Th plllking eplle
malide sar ind duatty
for lonsiue Hoo Higligh light viell de the food shereer well lncee
will
de poduri vea ligh
ACl aneaiaas Sira peace evantaler palk pelleegma hoogg doedly camapad nesepciinn
Incorrect
on Goce onler atritat
vallu olic curky lana acc umvak:
coonas sad caral Saul Cony, Inttud
(a) Assessmeut result of Base ((-Ven-3..VV))
serpr prokee moeely ferees es ledt yru we ege recty kema and layllge
buty hooe meter harkgrannh
Proforen Peyfic Prodiction
Adl lacs entern lovive int bote drwwing tot puu whither Drganee meth
Proflee Profiction
(14) The
a anculd mere with natural lou lightng offece
funoennyyy light out
wit and
coppatiig caken prodeme brsanial wakk cleeeeenn cuall light
contrtin bes
This wand of gou well with the lovel whess" lovll
bor propuataety whi mades cnffrntionn Menoue out the
amao clrraca disaty viluoal ca aediiis, maikk the profered cauraic aattiac.
dhoo good baui dappt
derinng
Aul- The lame Equara randond with mikiaa dand -aaly busk au ciltee the fils and body shap, Blddo
click add iili have ao cleance ais aloadd
tock
wuilic vateaues temesties livery domm the heral prrrsims mata opennt cooproptonn camp and gooth orticiatios
Teal Sower
Incorrect
X
212 Image sedt.
Correct
(b) Assessment result
Assesement
Cold-Strt
Figure S18: Effectiveness of cold-start SFT for correct and structured reasoning.
Question
Given user's reference images (including both preferred and non pre-prefedeed images) and generated images, first predict the user's visual preference profile and non-preference profile based on the reference images, th- assess and score the image across multiple dimensions based on the predicted profiles.
Profered Non-prefioral
preferonce profilee
Preference Profile Prediction
Art Style: Pointilism Color: Dusty Orange, Art Medium: Decoupage, Saturation: Vibran, Detail: Simplifted
<ivional preference profile-
cvlsual non-greference profile>
Non-preference Profile Prediction Art Style: Graffii Art, Coler: Harvest Gold, Art Medium: Spmy Paint, Satoration: Modarate, Detail: Rough
shrisual non-preferowe profile>
<think
Multi-dimensione Scoring and Assessment
Art style (4) The otter churacter is depicted through clean lines and simpliified fomms with distintt dut- festures visibler
the background wall pattern, echoing Pointillis''s dotted technique despiie being applied digitally. Unlike Graffini Arts b nesthatic, this image uses controlled point-like morks to haild texture, aligning well with the preferred
ourlines
Peintilliom style:
2. Color (4) The dominant dusty orange backgronnd perfectly matches the preforred Dusty Orange color attributtr, creaing
warm, earthy atmosphere sround the subject. While there are
colors like white brown the figure.
he overall color scheme remains finaly uschoced in the prefemre dusty oraage spectrun:
3. Art Medium (3) The image appears
layered digitul technignes siuilar Decoupage where different elements app unditicnal decoupage with physical cutouts, this digital application doesn's
car to be placed upon one another. However,
achieve the some dimensional quility or betile foel of actual paper layering seen in trie decoupage work;
User Reference Images
Saturation (4) image displas vitma intensity especially in orange backgronnd
contrastin black
wiline of otter, creating strong visual impaat. Despite having moderate saturation diewhhre, the strtegic use highly sas- urated orange creates focal points that draw attention. parrially satisfying both the prefereed Vibrant and now preferved Mod asuett simallanenssty.
5. Detail (4) The otter character and enviroemen: have been rendered with deliberately simplifled forms smooth contours.
minimal shading, and reduzed complex detuils This steamlined approach focuses viewer ariention on essential churacteristis while dlimiratig unnocessarry intricacies, perfertly matching Simpliffed detail preference;
Total Score: 4:4 14-19 Vabink-
Image Assesament
Figure S19: Reasoning- based personalized assessment and scoring for single image.
28

---

Published as a conference paper at ICLR 2026
Question Given user's reference images (including both preferred and non non-preferred images) generated images, first predict the user's visual preference profile and non-preference profile based the reference images, th- en assess and score the image across multiple dimensions based on the predicted profiles.
Preferred Noo--rrffene
Preference Profile Prediction Art Styler Naive Art, Color: Burn Orange, Art Medium: Lithugnphy, Saturation: Vieram, Detail: Abstracsed
crtsuaa
profile
"trimaal preferencce profite>
<edsaal preferecee
Non-preference Profile Prediction
Art Style: Concepcual Ar, Color: Crabery Red, Art Medium: Witercolor, Saturation: Modarate, Detail: Intractte
non preference profile>>
Scoring and A Axvessment
Artistic style(2) The burger illustation doployy characteristies elover to Concrptuaa Art through is simplifsed presentat-
ion focusing pranarily conveying food infommtion ruther than embrecing Naive Art's childlike execation away
folk qua-
While image
cunceptual
siming represent specific
naive simplicity toward more polished commercial aesthetic:
2. Color(3) The dominant colors include vileant red tomators and grvens, but lack significtat presenee of Hurnt Orange ho- in preferences. golden-brown crast, resulting in only partial alignmant with color preferences,
appears minimal
ccarge except perhaps
Art Medium(2) The image exhibits watercolur-Fi like traneprooyyy offects around the where colors blend sofily These
other, especially visible
letuee
suggest witercoler tochniques rather
lishographic printing methods with their disticttive plames and sharp contrasts:
Reforence Imsage
Saturation(4) 4) image demonstrates moderately high saturation level, partcularly evident bright
and vivid lettuce layers Howwyer compared
Vibrant
with intense chrocnatic this falls slightl
hort to somewhat controlled application color inteusity. Detail(3) The burger
depicted with moderate abstraction -howing essentna elemenss like ty. vegetables but maintiining recognirable forms. avolds both extreme abstraction and Intricate detailing, striking a dee ground that doesnt fully embrace eitber preference direction.
Score: 2-2-2-4-3-13
<abinkk
Sanswer- (aaeeer---//n/eee 2//oswwer
Figure S20: Reasoning based personalized assessment and scoring for single image.
images
preference profile
images
Predicted preference profile
style digisl fintagy
vyde;
Calor: parpir
Color: atomi grean
Art Medium: sigial painting
An medium: Aim
Sanuratio:: vibvas
Suturatio:: dosannaiee
Detult: crieres!
Initial image
Pessonalized image
Initial image Personalized image
Initial Image
Personalized image
Initial image Personalized image
"A magical magic roaoo"
"Cybermetic dragon with tech
"Mona Lisa"
"Joneph Stafts"
Preferred reference
Predicted preference profile
Preferred reference images
Prodicted preference profile
Ari spll vilion on Color: warm tervaaata
Artep husiass
Color: auniue Javenier
digitl Suturation:
Al drypiir
Satvration pale
Denal: othevval
Drsall sestored
Initial image
Pensocalized image
Initial image
Personaliacd image
Initial image
Personalized image
Initial image
Persomalized image
"Dragon flying"
"Cyborg cate
harmonlous valley"
Drompton
Figure S21: Personalized image generalization with the predicted profile.
29

---

Published as a conference paper at ICLR 2026
Real user from PickPic_ v2
preference profilc-
Preference Profile Prodiction
Lpnderence profiie
User ID: 8619
Art Styles Cooispial Colec: Forza Cioes, Art Medium: Dight Painting Saturation: Vivid, Detail: Sclecivee
Preferred Non-preferred
<ivisual preference profile>
Art Styte Abanat Expresionion, Color: Jet, Medium: Temper,, Saturation: Bold, Detalli Intikate
proference profile>
Non-preference Profile Prediction
preferefeence proftle
Art Styte: Remoniiien, Caler: Electric Line, Art Mediune: Oil, Saturatien: lliglly satursre, Detall: Smooth
preferece profile
Art Style: Consegiual Iui. Color: Steel Cruy, Art Mediam: Collage, Setaretion: Undestaisl, Detai:: Sirplifed
chivoal non-prefirvnce profile?-
<ahin--
Mulii-dimensionl Scoring and Assessment
Ait stde:
Imoge image syllsrd
promenie duges angalar eegroseting building seroominnghultingg stacture. The composiioon has concepoual undesiininns vlere evenydy city dlaneats becoue reitzngioed thoough distraction and decorstacton. This approch illgns strogly with Concegina de's rewdewgr.
Imoge BC7) The
garls pestrral stmopphric perppetiiss
rolling hills,
fulds nendered somosfer idesk landooge
sing beauty
respome to acenery.
Color:
Image AC4) The fiaturs a polettc deminated by Nenagay punctuated carker shadkwwal snzs nacmbiing
blades nenn-taad
Imnge B(2) The imge sses palent consisting peanasly sft arees, bowwss and mured yellows depicting vegetation and scil. While these are hirta of forcw, centais foliage segiona, they seconday the peodemisare sinel filee un-
Image N3) The imege exlibits shorp edged goonetry and perfect grid ens segpeting computer generteed Imagery combie-
ed with permorly offoces The medium comtrnas sqees of digini piggtt painiig with eeepere-ilee sparly carvin sness, particll-
differest suffeees
clenerss particolarty noticeable gie fittere through mos and soros the lardkcape. The way color blrdd everiape mombii meltiinal painting tochnity
booge
imae displays
pamn-lkke taantions
tas, with vioiile brank smoker and amauphert ofoon achiosal teough bycring and glaring metted.
Saturatien:
areas of mby betwern iluminnsd surfasa desper mecoas stritgic use
Imege
image
light
poticular-eppttn fucal puins prido the sye
teough comples queial relationdips.
User Reference Images
Image B(2) The image daplass relutvels underttate samoration shroughout wost the compostion, with muted goens, bro-
blus
the Coloss
soctened
experinnce
Image A
Image
lmage N/4) The image emepooys selectie donaling, will cetain geanetie elssens recelving curetúi antotio while others
Decall:
sinpliffed implted
directs stewend gare anttegically cround composiion, highiggrinn
compancnes
allowing
viesally
Imoge BK2) The image procrts penonolly displified oppreac driel, bread srae colurand supp taking prosaden-
ceaver minute pertcclaar Trees, teman conmurs, and distatt elemeors recevve only basic definttion creating coltesve whole dinaily Image 2121112111
withu local inmiaacios This aimiiiectiinn
con-pefemet Simpiifual deail desil annhere.
Imeage
Candidate Images
Incorrect
Sonsowr Imago A Allemmve
Figure S22: Failed case of the Preferthinker for real users with complex preferences.
30


Reviews:
[{"review_id": "NMXx6vJwYb", "reviewer": "Reviewer_PbCi", "summary": "This work proposes PreferThinker, a system for personalized image reward scoring with chain of thought reasoning to predict a user's preference profile. To train this system, the authors first construct PreferImg-CoT, using Claude 3.7 to generate a reasoning trace of each individual user's score given reference images. The authors use this dataset to train Qwen-2.5-VL-7B with SFT and GRPO, and evaluate on their PreferImg dataset and Pick-a-Pic v1.", "questions": "* Which version of Pick-a-pic is used in your experiments, v1 or v2?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* This work proposes an interesting system to predict a user's preference profile for text-to-image generation, and then score generated images for that individual\n* This work contributes a new large scale synthetic preference dataset PreferImg created from 80K synthetic user preference profiles with attributes that the authors choose after a real-world user study. This dataset can be a valuable resource for the text-to-image reward modeling community\n* The authors demonstrate that their dataset can learn rewards that generalize well when new users have similar preference profiles to users seen during training (Tab 1) via cold start SFT and RL training of Qwen 2.5 VL", "weaknesses": "* The authors argue that *\"although each user’s personalized preferences are unique, the key visual elements that shape these preferences are shared\"* (L197-198), and they mention discrete attributes that users rank highly as important to them (art style, color, detail, art medium and saturation). I feel that this a strong assumption to make - what about individual preference differences that are more semantic in nature for a given prompt? Is it possible to discretize real-world user preferences of generated images in this manner?\n* **My primary concern**: PreferThinker appears robust to unseen users when the distribution of their preference profiles is shared with seen users (I.e. they are both sampled from PreferImg). How robust is PreferThinker to preferences outside this distribution? Since PreferImg is constructed with 80K synthetic preference profiles, it is unclear to me how robust these preference profiles will be to real-world datasets. While the authors do evaluate on Pick-a-Pic, PreferThinker's accuracy (67%) does not seem compelling when compared to extremely lightweight pluralistic reward modeling baseline [1], which gets 71% accuracy on V1 and 70.5% on V2 (no-leakage) with just 6M trainable parameters \n* The presentation has room for improvement, especially the figures, which are quite small, dense, and difficult to read (especially Figure 8). Figure 1 and 2 can be made bigger and Fig 3 moved to the Appendix. It is very hard to parse Table 2 and 3 as their captions are nearly contiguous.\n\n[1] Chen et al., \"PAL: Sample-Efficient Personalized Reward Modeling for Pluralistic Alignment\", ICLR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B9oQcPJbSi", "reviewer": "Reviewer_PeqQ", "summary": "This paper introduces PreferThinker, a new framework for personalized image preference assessment.\nThe authors pointed out that currently there is a lack of user-specific data.\nTo overcome this issue, the paper proposes a common preference profile for bridging various users.\nThis profile enables the model to leverage large-scale data to learn user preferences.\nThe PreferThinker framework operates on a \"predict-then-assess\" paradigm: It first predicts a user's visual preference and non-preference profiles based on a small set of reference images. Then, it uses this predicted profile as a criterion to generate interpretable, multi-dimensional scores and a Chain-of-Thought (CoT) assessment for candidate images.", "questions": "Given the primary limitation is the simulated dataset, could you discuss the feasibility of collecting a (perhaps smaller) \"gold standard\" test set with real personalized data?", "limitations": "", "rating": 8, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper tries to tackle the problem of personalized preference assessment by using the idea of common preference profile as a bridge between users. This idea is novel. Beyond this, the paper also introduces a new, large scale dataset for personalized assessment. The experiments are robust, covering seen vs. unseen profiles , single vs. multi-preference users , and robustness to the number of reference images.", "weaknesses": "The primary weakness is that the main dataset, PreferImg-CoT, is built on simulated user preferences. While the simulation pipeline is well-designed (based on a user study to find 5 key elements ), simulated profiles may not capture the full, complex, and sometimes contradictory or hard-to-articulate nature of real human preferences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cTLz2vhY5p", "reviewer": "Reviewer_2Qg5", "summary": "This paper presents PreferThinker, a reasoning-based personalized image preference assessment system. The key idea is to predict a preference profile to bridge various uses, allowing large-scale user data to be leveraged for training profile prediction and capturing complex\npersonalized preferences. To this end, a CoT-style dataset annotated with preference profiles and high-quality reasoning for interpretability supervision is constructed. A two-stage a two-stage training strategy comparing Cold-start SFT and reinforcement learning is utilized to enpower the model with reasoning capabilities. Experiments on the constructed dataset shows that the proposed method outperforms existing approaches.", "questions": "What is the limitation of the proposed method?\n\nHow does the effectiveness of the proposed method on personalized image generation?\n\nIn my opinion, the current 5-element visual preference profile mainly summarizes the visual tone of an image, I wonder whether it is possible to make the model learn to find the visual elements that determines his/her visual preference?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This paper addresses personalized image preference assessment from a novel visual preference profile based interpretable perspective. \n\nA large-scale Chain-of-Thought (CoT)-style personalized assessment dataset annotated with diverse user preference profiles and high-quality CoT-style reasoning is contructed, enabling explicit supervision of structured reasoning. \n\nExperiments demonstrate the superiority of the proposed method.", "weaknesses": "My main concerns is how the proposed method generalizes to real-world images. To costruct a large-scale CoT-style dataset that provides high-quality reasoning supervision, the authors propose to combine several random profiles with initial prompts and feed into a text-to-image model to generate each user’s reference images (preferred and non-preferred) and two candidate images. However, the generated images ,as shown in paper and supplementary, lack photorealism, and would also cover a very limtied range of categories. Though the experiments demonstrate the superiority of the proposed method, the main experiments are conducted on the collected dataset, failing to validate the generalization to real-world images. \n\nOn the other hand, it is still unclear to me whether the compared methods in paper are trained or fine-tuned on the collected dataset for fair comparison of assessment accuracy. Moreover, the evaluation on the PickaPic dataset is somehow confusion to me. As described in paper, the experiment on PickaPic reflects general preferences, rather than personalized preference. However, the results in Table 1 show that the proposed method ranks second on unseen PickaPic data for general preference assessment, this makes me doubt whether the proposed method can indeed extract and understand personalized visual preference profile. \n\nCurrently, the employed visual preference profile consists of five visual elements while the user study is conducted with 15 visual elements. Therefore, I would like to see experimental analysis on how the number of visual elements in a visual preference profile affects the final results. \n\nIn fact, it is difficult for a person to rate his/her visual preference to an image using an absolute score, but it is easy for a person to perform relative preference comparison between two images. Hence, I suggest the authors to evaluate the proposed method on such preference ranking data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8msCxepqOw", "reviewer": "Reviewer_oEnc", "summary": "This paper proposes a new CoT-based methodology for personalized image preference assessment and introduces a large-scale CoT-style dataset. Most existing work captures general human preferences and cannot handle personalized ones. The paper introduces the concept of \"common preference profile\" and proposes a two-stage \"predict-then-assess\" method. The interpretable preference profile and reasoning steps help MLLMs to achieve better alignment for each user's personalized image synthesis.", "questions": "Humans have inconsistent preferences across different topics and scenarios. How does the proposed elementary preference profile approach address this complexity? Can you clarify the motivation behind using independent element-wise preference profiles rather than capturing the continuous latent representation for users' preferences?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The innovative “Common Preference Profile” efficiently bridges the gap between large-scale general preference alignment and personalized preference modeling.\n---\n2. The proposed \"predict-then-assess\" paradigm decomposes the task into *profile prediction* (summarizing human preference across several pre-defined dimensions) and *assessment reasoning* (interpretation and scoring). This decomposition provides **better modularity for image generation** and interpretability.\n---\n3. The paper empirically verifies that carefully designed reasoning structures—incorporating CoT reasoning to (1) predict each user's preference profile and (2) assess candidate images with the uncovered preference profile—achieve state-of-the-art performance on human preference prediction.", "weaknesses": "The discrete elementary preference profile may oversimplify the real-world human's nuanced preference, as it reduces complex, context-dependent aesthetic judgments into a fixed set of categorical dimensions that may not capture the subtle variations and contradictions inherent in individual taste.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)