RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models
Authors: Yeongtak Oh
Dohyun Chung
Juhyeon Shin
Sangha Park
Johan Barthelemy
Jisoo Mok
Sungroh Yoon
Venue: NeurIPS 2025 poster ()
Keywords: Multi-Modal Large Language Models
Personalization
Post-Training
Primary area: applications
TL;DR: We propose a novel RL-based MLLM post-training framework named RePIC for the personalized image captioning task. Our method significantly outperforms SFT-based baselines on multi-concept personalized image captioning.

Abstract: Recent multi-modal large language models (MLLMs) often struggle to generate personalized image captions, even when trained on high-quality captions. In this work, we observe that such limitations persist in existing post-training-based MLLM personalization methods. Specifically, despite being post-tuned with large-scale caption data through supervised fine-tuning (SFT), these models frequently fail to produce faithful descriptions in real-world scenarios, such as multi-concept image captioning. However, acquiring large-scale, high-quality captions for such complex settings is both costly and difficult. To address the data-centric nature of SFT, we propose a reinforcement learning (RL)-based post-training framework. To the best of our knowledge, this is the first RL-based approach to post-train MLLMs for personalized image captioning. Our method significantly enhances both visual recognition and personalized generation capabilities of MLLMs, and consistently outperforms existing SFT-based baselines, especially in the challenging multi-concept image captioning task. Project page: https://github.com/oyt9306/RePIC

RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models
Yeongtak Oh', Dohyun Chung?, Juhyeon Shin', Sangha Park', Johan Barthelemy", Jisoo Mok*t, Sungroh Yoon 1.31
Depaatment of Electrical and Computer Engineering, Scoul National University
Department of Future Automotive Mobility, Scoul National University Interdisciplinary Program Artificial Intelligence, Seoul National University *Daegu Gyeongbuk Institute of Science and Technology NNVIDIA
(dualism9306, pissaitworks tworks, newjh12, wiarae, sryoon) (arooonnesnu... @snu. ac kr, jmokêdgist ac. ac.kr, kr, jbartheeeemmnniddia com
Abstract
Recent multi-modal large language models (MLLMs) often struggle to generate personalized image captions, when trained on high-quality captions. In this work, we observe that such limitations persist in existing post-training-base MILLM personalization methods. Specifically, despite being post-tuned with large- scale caption data through supervised fine-tuning (SFT), these models frequently fail produce faithful descriptions in real-world scenarios, such multi-concept image captioning. However, acquiring large-scale, high-quality captions for such complex settings both costly and difficult. To address the data-centric nature of SFT, we propose a reinforcement learning (RL)-based post-training framework. To the best of our knowledge, this the first RL-based approach to post-train MLLMs for personalized image captioning, Our method significantly enhances both visual recognition and personalized generation capabilities MLLMs. and consistently outperforms existing SFT-based baselines, especially in the challenging multi- concept image captioning task. Project page:
Introduction
The emergence of Large Language Models (LLMs) has greatly propelled the advancement of AI, particularly in natural language understanding and generation [1, 52, 13, 6]. These models demonstrate impressive general knowledge and achieve strong performance across a wide range of tasks [17, 55]. This progress inspired the development of Multimodal Large Language Models (MLLMs) [31, 25, 57, 4J. MLLMs integrate visual inputs using pretrained vision encoders, treating image embeddings similarly to tokens within unified architecture. This integration [31, 30, 4] extends the utility of LLMs to lision-language tasks, enabling image-grone -rgunddg dialogue [25] and captioning [25, 30, 4].
Unfortunately, despite being pre-trained on large-scale datasets, MLLMs struggle to perform per- sonalization by recognizing and incorporating personal, user-specific concepts, typically provided in the form of a reference image of the user and an associated textual description [40, 15). Figure illustrates MILMMs failure in personalized image captioning, one of the most widely-studied tasks in MLLM personalization. Given the reference image of 'thao" and the corresponding description about her, i.e., 23 23-year-old woman who adores her beloved dog, Bo an MLLM is prompted to provide personalized caption for new query image of "thao" that significantly differs
+Corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Name:: -thss
Q Give caplion for this image jking the gives information
Info: 23 23-year-old woman who adores her beloved dog,
COuPCITER Scipacas
Qwen-2.5 VL 7B: young woman stands confidenily in frunt af Computer Sciences building, ready for her day school.
Missing Name!
Does contain any personal informationt
Q) Give personalized caption. for the image.
this delighrful sutume seene, <hall> the cheerfull onimated mecooon "monster" the curious animated creature, and socter the excited animated other enjoy playful moment together amidst fallen leaves and a warm, golden backdrap.
Ours-2K:
Falthful Captioning
RAP-LLaVA-110:
<hall-- is having a great time with his friends in the forest. They'r to all smiling and having good time.
Identity Missing
RAP-Owen-210K: <ball> having fun with his friends in the forest Identity Missing
Multi-Concept Image Captioning with 3-Concepts
@ a pessonalized caption for the image.
v Ours-2K: A lively parade scene unfolds on a bustling street, featuring <monster, toyo, <slaths, <plushi, and steddy- colorfall costumes. The -monster toy> lends the way with a cheerful expression, followed by <sloth>, "plushi, and <teddy>, who carries drum, creating festive atmosphere as they march through the crowd.
Falthful Captioning
RAP-LIaX Human: -monster toy- ready to cross the streett
He's his
friends "plusb> Seddy) him, and they 're set this purade!
Identity Missing
RAP-Qwen-210K: smooster__sse toy> and friends are ready to march!
Identity Missing
<plak>
Multi-Concept Image Captioning with 4-Concepts
Figure 1: Visualizations of personalized image captioning results. In the first row, the zero-shot MLLM frequently fails to generate personalized captions. The used images are soureed from Yo' LL'L [37].
The remaining rows illustrate multi-concept scenarios at inference time. Compared to other SFT- based methods, our approach consistently produces faithful and detailed captions while accurately recognizing all provided identities, even for 3 or 4 concepts. All images are sourced from MuDI [22].
from the reference image in lighting, background conditions, and poses; a successfully personalized caption should accurately refer to 'thao' 2 and include faithful details about the query image. However, Qwen-2.5 VL 7B [4], one of the most performant open-source MLLMs, fails to recognize 'thao' and include her personal concepts in its caption. Thus, recent works [2, 37, 40, 15] have increasingly focused on the personalization of MLLMs.
Existing approaches for MLLM personalization can be categorized into two groups: those that require retraining as new personal concepts are introduced [2, 37], and those that do not [40, 15]. In real-world applications, where the types of personal concepts unpredictable and new ones continuously emerge, the former family of personalization approaches is inherently highly limited. The latter cases [40, 15], which enable MLLM personalization without the need for retraining when new concepts appear, fine-tune the LLM on large-scale label-annotated datasets composed of question-answerigg pairs using Supervised Tuning (SFT). As result, the post-tuned MILM can recognize corresponding personal concepts between reference and query images and generate outputs including the given personal information at inference time.
To curate training data composed of image-caption pairs, previous SFT-based approaches [40, 15] have relied on proprietary MILMM such as GPT-44 [21] and Gemini [5]] to generate large-scale personal captions. For instance, PVIT [40] used GPT-4o to create captions and manually validated them for individual human images. Similarly, RAP-MLLM [15] curated captions using Gemini. However, even after post-training with large-scale captions, we observe that existing SaT-b post-tuned MLLMs still often struggle to generate faithful personal captions for the query image.
As shown in Figure 1, these difficulties become more pronounced in real-world scenarios, such as involving 3 or 4 concepts. Following our investigation, the SFT-based method [15] underperforms primarily due to the scarcity of captions in training data for multi-concept settings (i.e., only 5.4% of the total dataset). However, considering the fact that the performance of SFT is highly sensitive to the quality of the training data [7, 48, 32], obtaining a large volume of high-quality personal captions for SFT is both costly and challenging. This challenge becomes particularly severe when
2

---

curating captions for images containing multiple distinct identities, as each caption must accurately incorporate detailed personal information corresponding to every identity represented.
To overcome these difficulties, we investigate the key capabilities that a MLLM should possess for personalized image captioning: (1) robust visual recognition ability: the ability consistently identify the same object across different images, even under variations in pose, location, lighting, and background. This ability induces the MLLM describe the query image accurately and faithfully; and (2) consistent personalized generation ability: the ability to incorporate personal information from demonstrations into its responses, including correctly referencing the provided names. MLLM equipped with this capability can perform personalized image captioning.
To strengthen the abovementioned key capabilities, we propose Reinforced post-training for Personalized Image Captioing (RePIC) framework. We leverage the strengths of reinforcement learning (RI) for method composed of three key components, outlined below:
Object Consistency: To strengthen the MLLM's recognition abilities, we propose an object consis- tency reward that provides direct positive and negative feedback for the output.
Visual Localization: To further reinforce the MILM's visual recognition ability, exploit the visual localization reward that predicts bounding box (BBox) coordinates based on a query instruction.
Identity Consistency: To enhance the MLLM's ability to generate personalized responses, we introduce an identity consistency reward that explicitly encourages the inclusion of target names in the output.
In our experimental results, we reveal that SFT-based personalization methods are highly limited for visual recognition and generalization abilities. Conversely, by integrating our proposed reward templates along with curated datasets and instructions, our method achieves significant performance improvements over existing baselines, particularly in multi-concept personalized image captioning benehmarks. To the best of our knowledge, this is the first work to present an RL-based post-training framework that enables MLLMs to perform personalized image captioning effectively.
2
Related Works
MILM Personalization To enable general- purerae- MLLMs [31, 31, 4] to perform personalized image captioning, several methods have been proposed, including those requiring retraining when new personal concepts emerge [2, 37] and those that do not [40, 15]. In the former case, My [2] uses external concept heads to identify user-specific concepts and learns embeddings for each to input into the LLM. Yo LL [37] encodes personal concepts as special textual tokens that serve as concept identifiers. However, these methods lack scalability to new concepts, as they require retraining the concept identifiers whenever a new concept emerges and do not guarantee sufficient training data per concept. To overcome these limitations, SFT-based post-training approaches have emerged. PVIT [40] uses special prefixes to encode indieicual-speeffic information, enabling MIL.Ms to answer queries about new individuals. RAP-MI. MLLMMMM presents pipeline combining retrieval for visual demonstrations and post-training for personalized generation based on the query and retrieved concept information. Compared to existing SFT-based approaches, we propose an RL-based post- training method that reduces reliance on large-scale, high-quality personal captions and demonstrates superior effectiveness, particularly in multi-concept personalized image captioning.
RL-based MILMM Post- Training Methods. RL has demonstrated substantial improvements in to
several tasks of LLMs [14, 46, 60, 36, 11]. Recent studies have extended RL-based
ILLMs-applyinn preference-based RL for hallucination mitigation [61] and model alignment [27], and basecy-based RL for visual reasoning [34, 20, 49]. For example, regarding policy-based RL, Visual-RFT [34] proposes reward templates to improve performance on visual perception tasks such as fine-grained image classification and few-shot object detection. Vision-R1 [20] combines cold-start initialization with RL training to enhance MLLM reasoning capabilities, particularly in tasks like bounding box prediction and few-shot classification. Reason-RFT [49] introduces two-phase RL. framework that integrates SFT-based and RL-based methods for visual reasoning. In contrast to these approaches, we leverage policy-based RL to enhance personalized image captioning in MLLMs.
3

---

Overll Frunewark Traisina
Object Contislency Templote
Verifeble Roward
Question:
from the first imae
Rettranca Media
Object Consiieency
the image?
cornect) lereng)
Vissal Londiization Templote
6RPO Loss
Question: Please provide describes cocrdinate of the
Visiel Locditation
rightt
1000 ON a Oatic oa
visiile.
Question: Give detail coption for the second refenencing the first names
Idantity Consihency Template
Terget Hooo
Eduntity Comsisiency Ineso a sutplt mome))
Infironce Step-Erttissl Serting
Q) Previde a paranal septise for the inage
Viean Encader
MILM
Decoder
li feers Edboy The cog teawiss calo
tis teartworming soone (Fegs French Builday and dlaby,.o Or thare a nomast together: dfel on floor diaby_.r lytng a play with a playful mabile
Figure 2: Overview of our RePIC framework: (a) training phase and (b) inference phase. An abbrevi- ated example of the prompt template shown; complete templates are provided the Appendix.
3 Method
In this section, we detail the GRPO [14] algorithm employed in our approach. To enhance both visual recognition and personalized generation capabilities. we introduce an RL-based post-training framework for MLLMs. Specifically, we outline the design verifiable rewards and the structuring of instruction and data templates. Figure 2 provides an overview of our proposed RePIC framework during both training and inference stages.
3.1 Preliminary: Group Relative Policy Optimization (GRPO)
GRPO [46] is an improved RI. algorithm of PPO [44] that exploits the rewards and introduces group- based learning based relative preferences. Specifically, as shown in Eq. (1), it optimizes clipped surrogate objective. similar to PPO, while simultancously minimizing the Kullback-Leible (KL) Divergence between the current and reference policies. For each state, task q is sampled from the distribution Q, and the policy model Tee generates set of G' responses, denoted as { which are referred to as rollouts. Then, each response is assigned with corresponding reward [ri)se This process can be formulated follows:
LGRPO (0)
Eg B Q (0.) ~
G 1o.1
min
1-
DKL (To Tret))
(I)
where the importance of sampling ratio (0)= x0(00.1g..1..c)) se,y. (o..e19.00,.................. c.. Af is the normalized advantage with y and being the mean and standard deviation group of rewards. Here, 3 controls the strength of the KL-regularization.
We leverage verifiable reward (VR) [14] with GRPO, enabling training without the need for an auxiliary reward model. In the context of personalization, the following sections discuss how the core ciciolittts-iiuuue recognition and personalized generation can be formulated using VRs.
3.2 Proposed Verifiable Rewards
Object Consistency Tuning (OCT) To improve the visual recognition capabilities of MLLMs by providing VR, we construct nositive-negative pairs. Positive pairs consist of images containing the same object, while negative pairs include images with different objects. We then use binary-response questions, such "Is <name> present in the second image?", and assign VR of 1 if the

---

model responds yes for a positive pair and no for a negative pair. Incorrect responses receive VR of O. Based on these pairs, we design reward template called OCT, as defined in Eq. (2).
1, correctly answer with yes or no TOCT
0, else
We use real datasets such as COCO [29], Objects365 [45], and CelebA [33], from which we crop object regions to serve reference images. However, as real data often lacks sufficient variation in attributes such as pose and lighting. we additionally incorporate high-quality, visually diverse synthetic images from Subject 200K+ [50]. These synthetic images, generated using diffusion models such as Flux [24], include variations in pose and background while preserving subject identity.
Visual Localization Tuning (VLT) To enhance MLLM by strengthening its localization capability, we adopt the loU-based accuracy reward template introduced in VIM-RI [47]. In this setup, the Intersection over Union (IoU) score serves as the reward criterion, where VR is assigned as 1 if the predicted BBox aligns with the GT BBox at an IoU threshold greater than 0.5, as described in E3)(
1,
if IoU 2 0,5
TVLT 0. otherwise
We refer to this reward template as VLT,, and by reinforeing VLT,, the model is enabled to localize the object within the image and can understand its relative location, such as right, left, or top. Specifically, we use Refcoco/+/g datasets [35, 59] commonly used for the general visual reasoning task of referring expression comprehension (REC). Notably, our empirical findings indicate that removing the REC task from the training set often leads to instability during RL-based post-training.
Identity Consistency Tuning (ICT) To force the model to consistently utilize the provided infor- mation from visual demonstrations in its responses, we consider positive pair composed of few reference images and query image pair. For the reward template of using one reference image per query, we call Single-ICT, and for the reward template of using multiple reference images per query, we call Multi-ICT. In this setting, we assign unique name to each reference image as <name> token. In detail, we prompt the model with response questions such as "Describe the query image while referencing the reference images. "' and assign VR of 1 is only assigned when the model accurately describes the query image using all the given names. Note, in our experiments, used a maximum of 3 reference images per query image. In detail, for single-ICT, assign VR as follows Eq. (4):
1, if <name> appears the output
SSngle-IIC
0. otherwise
We use the positive pair images used for OCT. Next, for multi-ICT, for a given set of m(< 3) names and n correctly mentioned names, we assign VR for the response as described in Eq. (5):
fn/m, if <name 1>, <name 2>,-.*, <name n> appears in the output
MMuliiICT
(5)
0,
otherwise
To construct multiple reference images, we use real multi-object images from COCO [29] and Objects365 [45], and manually curate high-quality examples by cropping two or three distinct objects from a single query image. The data templates used for each component are provided in the Appendix.
Furthermore, incorporate descriptive prompts (e.g., 'describe this image in detail." )) that elicit richer language generation in the training dataset. Additionally, we apply output length regularization ensure that captions exceed a minimum length, which helps avoid less preferable responses (e.g., 'This is <name>. 2).
4 Experiments
Baseline We compare our method with other post-training-batte approaches. As baseline models, we fine-tuned PVIT-LLaVA [40] using 210K subset of the 3M dataset. We also consider pre- trained RAP-LLaVA [15], which is fine-tuned using LoRA [19] from -.L. Vicuna 13B [30]. In addition, we fine-tune RAP-LLaVA using only 2K samples randomly selected from the full
5

---

Table 1: Single-concept personal grounding performance evaluation results.
Models
Seen Data
MyVLM [2]
Yo'LLaVA [37]
DreamBooth [43]
Pre.
Rec.
FI
Pre.
Rec.
Fl
Pre.
Rec.
F!
Skip- Ret-Reereeval Setting
PVIT-ILAVA
210K
17.1
1.8
3.3
20.1
2.1
3.8
26.5
16.5
20.3
RAP-ILLAVA
210K
100
92.9
96.3
100
95.5
97.7
97.3
91.8
94.5
RAP-LLAVA
2K
100
49.4
66.1
50.6
48.6
49.6
68.4
65.8
67.1
RAP-Qwen
210K
100
98.8
99.4
100
99.8
99.8
100
100
Qwen-2.5 VL
0
100
56.8
72.4
100
33.3
50.0
96.0
76.6
85.2
Ours
2K
100
96.2
98.1
99.7
96.1
97.9
100
98.1
99.0
Retrieval Serring
Retrieval (Top-2)
97.6
95.9
96.7
83.6
82.9
83.3
99.3
96.2
97.7
RAP-LLAVA
210K
95.6
79.1
87.8
82.7
79.9
81.2
96.0
91.1
93.5
RAP-LLAVA
2K
79.2
53.8
64.1
71.2
52.2
64.4
69.5
66.5
68.0
RAP-Qwen
210K
95.5
87.9
91.6
79.2
76.2
98.7
94.3
96.4
Qwen-2.5 VL
0
91.5
50.6
65.2
77.4
42.3
55.2
95.2
75.3
84.1
Ours
2K
99.0
83.2
90.4
84.4
69.7
76.3
98.6
90.5
94.4
Table 2: Multi- Multi-concept personal grounding performance evaluation results.
Models
Seen Data
2-Concepls
4-Concepts
Skip- Reip-Retrieval
Retrieval
Skip-Retrieval
Retrieval
Rec.
RAPAL
210K
100
93.9
96.9
99.3
89.6
94.5
52.9
7.9
16.7
3.1
5.2
RAP-LLaVA
2K
100
90.2
94.9
95.7
81.1
87.8
36.4
1,9
3.6
22.4
0.7
14
RAP-Qwen
210K
100
82.9
90.7
100
73.2
84.5
49.6
13.6
21.3
12.6
2.6
4.3
Qwen-2.5 VI.
100
75.0
85.7
98.1
64.0
77.5
73.3
22.9
34.8
22.5
6.4
10.0
Ours Full
2K
100
98.8
99.4
97.5
93.9
95.7
88.0
59.5
71.0
24.8
15.7
19.2
210K dataset, matching the amount of seen data used in method. In study, adopt the instruction-tunee Qwen2.5-VL. 7B [4] as a backbone MLLM. For a fair comparison, we also fine-tume tune it using LoRA the 210K instruction dataset [15], with data templates adapted for Qwen-Vi VL compatibility. The resulting model, fine-tuned using LLaMA-Factoon [64], is referred to as RAP. Qwen. Hyperparameter sensitivity details provided in Appendix C.
Dataset We consider both single and multi-concept datasets for evaluation. The single-concept data are sourced from Yo'LLaVA, MyVLM, and DreamBooth. These datasets consist of various single-concept images with variations in lighting, pose, and background conditions. For multi- concept evaluation,
the RAP-MLLM [15] dataset, constructed by collecting YouTube videos and extracting frames that include 2 concepts. To evaluate its personalization capabilitics more challenging scenarios, we experiment training. To this end, we curate a dataset by crawling images from movie teasers and award ceremonies where multiple celebrities appear together. We select those for evaluation dataset in which at least 4 distinct concepts are clearly present. Details on the used datasets are provided in the Appendix B.
images containing 4-concept cases, never seen during
We evaluate personalization capabilities under two different settings. First, in the case where GT demonstrations are directly provided at inference time, we refer to this as the skip-retrieval setting. Note that the demonstrations include several reference image-text pairs. Next, we consider the retrieval setting [15] that automates the manual selection of the demonstrations by retrieving the most relevant visual content from databasc. Further details are provided in Appendix B.5. Additionally, we newly evaluate personal grounding performance in the skip-retrieval setting using reference images that do not match the query. A lower score in this case implies that the MLLM does not merely duplicate the given demonstrations while performing personalized image captioning. Additional details for the used evaluation templates be found in Appendix D.
Evaluation
Implementation Details For the zero-shot model, we apply a detailed prompt (e.g., "Output the final answer, including its name in the answer "") to guide the model to mention the target identity. Note that in multi-concept settings, in-domain (ID) refers to using concepts, while out-of-domain (OOD) refers to using 4 concepts. In the retrieval setting, retrieve the top-2 most

---

Single Conceept
11a3 76
No Templet
vo 2ood aromer
Guavi
No Tox
130 2106
Cliao Toepkan
163
20
-0
100
Concepp
Oay-78
Olaa 2,
Oum-70
211K
ho Tamp mm
A/r Dical poret
Oan- 78
RAP L1.0A
Okar 26
No Tanziea
128-91T1310k
No aarpaa
Tempura
60
10c
20
40
00
130
Preforenoe 2X]
Comparisons with models
Ablation studies
Visualization of preference evaluation scores for single and 2-concept settings, correspond- ing the first and second rows, respectively. In (a), our model outperforms all other baseline models, while in (b), it surpasses all ablation variants.
Figure 3:
Give personalized ception image.
Ours
fiom
SCharatter
samding in dinlyy roum, lorking aupeisad cumaaraa The badkgrond suggots they might be incl: vchicle or suntrol noo.
Cornect Coptiening
Qwea-2.5.VL. : In dinly room. SChanese A and cCharater B stand Gacing cash odiet. their expeessions hinting a mix curiosity and anscipation
Whong Coptioning
RAP-LLaYA Chhrraetrr [I' as dy fox with big grin. Iie's always up for advennare.
Identity Mixinng
RAP-Qwen
ieneady holp!
Iduntity Misting
Give penomalized caption of this
Ours image, Ri and are canding ostsiée near buikiing colortil zural. They are both bolding small onjects in then hands and appear. so be enpaged at conversation acuisity together The scene is hight and sunny, with greenery Visible in the backegound.
Caphioning
Quen-2.5-V.. VI. man woman see sweding cntedl vanry day woman is worning sunglasses and a bloe polka doced dress while holding small object hand. The mas also wearing sunglasees and light colored tishin, hokiing similar object. They appear engaged is converation or activily logether.
Identity Misting
RAP-LLaVA
enjyingg some icc. enoms
Wrong Caphioning
RAP-Qwin
ampoying
Caphioning
Figure 4: Qualitative examples of 2-concept personalized image captioning.
relevant samples for single and 2- 2-concept settings, and the top-4 for the 4-concept setting. All training experiments are conducted using 8 A40 GPUs, with inference performed on a single A40 GPU, Additional details on the retrieval and experimental setup are provided in the Appendix B.
4.1
Personalized Image Captioning Evaluation
4.1.1 Personal Grouding Performance
We evaluate model performance for all settings using the same query evaluation prompts. To quantify how frequently the target <name> or Name or name appears the model's output, we compute precision, recall, and F1-score following the evaluation protocol of [15]. In detail, these scores represent how well the model generates the response while containing the personalized information, without considering the caption quality. We will refer to these scores as the personal grounding performance of the post-tuned MLLM. Here, recall reflects the proportion of correctly mentioned target concepts
of total concept names, while precision indicates the fraction of correct mentions out of all concept names that occurred. In the skip-retrieval setting, precision can be reported as 100%, the GT demonstrations provided. For multi-concepp settings, assign score of n/m the model correctly included n target names in the response from given m names.
As shown in Table 1, in the single-concept setting, the reproduced baseline PVIT [40] shows notably low personal grounding performance, even under the skip-retrieval setting. We conjecture that this poor performance likely stems from the training data, which is largely human-centric. The reproduced
7

---

Table 3: Image caption quality evaluation results with reference captions.
Types
Metrics
RAP-I.LaV
RAP-Qwen
Zero-Shot
Ours
BLEU [38] (10 2)
0,260
0.170
0.210
0,290
CIDEr [53]
0.193
0.185
0.208
0.194
Reference- based
METEOR [5]
0,242
0,267
0.271
0.321
SPICE [3]
0.104
0,084
0,083
0.086
BERTScore [63]
0.683
0.567
0.523
0.668
Table Image caption quality evaluation results without reference captions.
Types
Metrics
RAP-LLaVA
RAP-Qwen
Zero-Shot
Ours
Reference-
CLIPScore [18]
0.332 -0.094
0.316 0.087
0.323
0.339
free
ImageReward [56]
0.287
0.130
RAP-Qwen, trained on 210K samples, achieves the highest scores in the skip-retrieval setting, while method performs comparably and outperforms RAP-LLaVA. Under the retrieval setting, the top-performing model varies by dataset. Notably, RAP-LLaVA suffers a significant performance drop when trained only 2K samples, highlighting the data dependency of SFT. For reference, also report the top-2 retrieval performance as an upper bound, accounting for retrieval noise.
In the multi-concept settings, as shown in Table 2, the performance of SFT-based methods drops significantly in both and 4-concept scenarios. contrast, our proposed method consistently and substantially outperforms all SFT-based baselines under both skip-retrieval and retrieval settings. Notably, the performance gap becomes even more pronounced in the 4-concept setting. Note that the zero-shot Qwen outperforms other SFT-based models in terms of personal grounding across all cases, further highlighting the limitations of SFT-based post-training in generalizing to OOD scenarios. These results underscore the effectiveness of our RL-based post-training approach, particularly in extending it to real-world personalized image captioning tasks.
4.1.2 Preference Evaluation
We conduct human-level quality evaluation with GPT-4o [21] to assess the quality of generated personalized image captions. The quality evaluation is conducted where captions that merely duplicate the provided information or fail to accurately describe the image are considered wwwprrfeeence. The evaluation template is provided in the Appendix. In Figure 3, we present the preference evaluation on the single-concept YoLLaVA dataset and on the RAP-MLLM dataset, corresponding to the first and second rows, respectively. Those results indicate the superiority of our proposed post-tuning method in generating high-quality and faithful personalized captions compared to all other methods. Our proposed method significantly outperforms (a) all other baselines, and (b) all ablation models, including those without length regularization, without detailed prompts in the dataset, and with reasoning templates.
preference based assessment,
To better visualize the effectiveness of our method, we present two qualitative examples in Figure 4. In the first row, although the zero-shot model correctly references the given names, its generated captions are less accurate. SFT-based methods fail to recognize identities consistently, often missing given identities, resulting in unfaithful captions. In the second row, while the zero-shot model produces detailed captions, it fails to include the provided names. Conversely, although previous SFT-based methods mention the names correctly, but produce inaccurate descriptions due to limited visual recognition capability. However, in both cases, method consistently generates faithful and accurate personalized captions for the query images, demonstrating superior personal grounding and visual understanding. Please refer to further qualitative results in Appendix A.
4.1.3 Image Caption Quality Evaluations
We conduct both reference-based and reference-free evaluations of image captioning quality on the YoL.LVAA dataset. It is important to note that the employed metrics do not assess the degree of personalization achieved by the MILM; rather, they measure only the overall quality of the generated captions for the given query images. Table presents metric--bsed comparison between

---

Table 5: Personal grounding with wrong visual demonstrations.
Models
Seen Data
My (U)
YoLLaVA
[37](U)
(C)
DreamBooth [43] (J)
Pre.
Rec.
F1
Pre.
Rec.
F1
Pre.
Rec.
F1
RAP-LLAVA
210K
100
89.7
94.6
99.7
97.0
98.3
95.2
90.8
92.4
RAP-Qwen
210K
100
69.7
82.1
98.2
82.6
89.4
85.0
71.5
77.7
Qwen-2.5 VL
100
55.6
72.4
99.0
60.4
75.0
97.2
88.6
92.7
Ours
2K
98.9
54.4
71.5
93.8
64.0
76.1
82.6
63.3
71.7
Table 6: Ablation studies for personal grounding in 2-concept image captioning.
Models
Seen Data
Skip-Retrieval
Retrieval Recall
Pre
Recall
Fl
Pre
Fl
Shro-Shot
100
75.0
85.7
98.1
64.0
77.5
Reasoning Template Ablations
Ours <think>
100
90.9
95.2
97.2
84.2
90.2
Ours <observe>
2K
100
80.5
89.2
99.1
67.7
80.9
Reward Template Ablations
ICT
100
17.1
29.2
100
14.6
25.5
Ours w/o OCT
2K
99.2
73.2
84.2
99.1
67.7
80.4
Ours w/ VLT
2K
100
53.6
69.6
98.9
50.0
66.7
Ours only ICT
2K
100
29.9
46.0
98.0
25.0
39.5
Ours only OCT
2K
100
12.8
22.7
97.4
16.5
28.3
Ours only VLT
2K
100
17.7
30.1
98.9
18.3
29.9
Additional Component Ablations
Ours w/o length
2K
100
92.1
95,9
99.3
91.5
95.2
Ours w/o detail prompt
2K
100
86.0
92.5
98.4
74.4
84.7
Ours Full
2K
100
98.8
99.4
97.5
93.9
95.7
the generated captions and GT captions (not reference text in the database) for the query image. Since no GT captions available for the used evaluation dataset, generated them using GPT-4o by varying the seed three times. As a result, ours show relatively similar results trends, with minor differences in the top 1-2 rankings. In detail, in METEOR and BLEU, the second-best in CIDEr, SPICE, and BERTScore. Table 4 presents the results of image-text alignment evaluation without reference captions. CLIPScore measures the cosine similarity between image and text embeddings, while ImageReward is general-purpose reward model aligned with human preferences that evaluates image-text alignment quality. Our proposed method achieved the highest CLIPScore and the second-best ImageReward score.
proposed method achieves the best performance
4.2 Ablation Studies
Necessity of RL. To emphasize the necessity of RL, examine the limitations of SFT based experimental results. First, we analyze the algorithmic shortcomings in visual recognition for personal grounding. Although RAP-Qwen achieves the highest personal grounding scores in the single- concept setting (Table 1). its performance significantly deteriorates when incorrect demonstrations provided (Table 5). The results suggest that SFT-based methods struggle distinguish different objects between reference and query images, often resulting in inaccurate personalized captions. In contrast, RL-based method consistently yields lower scores when provided with incorrect demonstrations, highlighting its robustness in differentiating objects and its ability to avoid merely duplicating the given personal information when generating personalized captions.
Furthermore, we analyze why SFT-based methods struggle with personal grounding in multi-concept settings, shown in Table 2. To this end, we examine the training data compositions that include multiple identities within a single image. Interestingly, despite a comparable proportion of multiple- identity training data (i.e., RAP-MLLM: 5.4% of 210K, Ours: 4.7% of 2K). SFT yields only marginal improvements in the 2-concept setting (ID) and performs poorly in the 4-concept setting (OOD). In contrast, our method achieves substantial performance gains in both the 2 and 4-concept personalized image captioning tasks.
We attribute the shortcomings of SFT-based pethods-particulle their limited visual recognition and poor generalization to OOD scenarios-to fundamental algorithmic differences. As a data-centric approach, SFT tends to overfit to dominant patterns and struggles to learn from rare diverse

---

data [12]. Our experimental results further highlight why RAP-Qwen performs well in single-concept settings but fails significantly in OOD scenarios. These findings underscore the necessity of RL-based approaches to perform robust and generalizable personalized image captioning.
Efficacy of Each Component. The overall ablation studies are shown in Table 6. First, we evaluate the effectiveness of reasoning templates [14],
Meen oo
1.0
such as the <think> token used in visual reasoning tasks. As a result, using reasoning templates rather degrades the personal grounding per- formance, regardless of the two different special reasoning tokens. For reward template ablations, removing the OCT template from the training dataset weakens the scores, underscoring the importance of reinforced visual recogmition. Most notably, removing ICT causes drastic decline in performance, confirming its critical role in personal grounding. Removing the VLT also leads to performance drop, suggesting that reinforcing visual localization contributes to enhancing personal grounding. These findings emphasize that the integration of all proposed components is essential to achieve state-of-the-ary performance. For the ablation results on the additional components related to preserving the captioning quality, excluding length regularization or detailed query prompts results in minor accuracy drops. For the qualitative effects of these components image captioning quality, please refer to the Appendix.
0.8-
0.6
Averoge Seeeei
20
Sid. ofReendls
Average Sends
0.4
0.2
20
100
Seen Data (%6
Figure 5: Visualization of training stability.
Training Stability. In Figure 5, we report the mean and standard deviation of rewards across different random seeds to demonstrate the stability of our RI. training, independent of the data curriculum.
5 Conclusion
In this work, we propose RePIC, a novel RL-based post-training framework that alleviates the cost of collecting high-quality personal captions, enabling effective personalized image captioning. By leveraging verifiable rewards, curated data compositions, and tailored instruction templates, our post- tuned MILM achieves strong performance personalized captioning tasks across both single and multi-concept settings. Furthermore, our experimental results demonstrate that RePIC equips MLLMs with more robust visual recognition abilities, enhanced personalized generation, and improved generalizability than existing SFT-based methods, highlighting the superiority of our approach in complex scenarios such as 4-concept image captioning cases not seen during post-training.
Limitations. In this work, the reproduced baseline models may not have undergone exhaustive hyperparameter tuning. Further, as this study primarily focuses evaluating the impact of RL on personalized image captioning, future work could extend this approach to other personalized tasks, such multi-turn conversations. Additionally, human-level preference scores due to the lack of GT captions. This would be improved by generating GT captions with larger models like GPT-4o and human refinement.
evaluated caption faithfulness only using
Future Work Reducing retrieval noise through reasoning or stronger retrieval models, and exploring alternatives to length regularization (e.g., self-correction test-time scaling) are potential directions for future research. While this work focuses on RL-based post-training on the image domain, future research could extend personalization to other modalities such as audio and video, aligned with the progress of models like Qwen-2.5 Omni [57].
Acknowledgments
The authors also gratefully acknowledge the support from the NVIDIA Academic Grant Program This research was supported by the National Research Foundation of Korea (NRF) through a grant funded by the Korean government (MSIT) [No. (022RIA3BIII 1311077720); the Institute of Information & Communications Technology Planning & Evaluation (ITTP) grant funded by the Korean government (MSIT) [No. RS2021-1121 1343], Artificial Intelligence Graduate School Program (Seoul National University),
No.RS-2025 20220022223554444402022222222244 Human-Centre Embodied AI Agents with Autonomous Decision-Making Mobile eXperience(MMX) Business, Samsung Electronics Co., Ltd.; the AI-Bio Research Grant through Seoul National University; and the BK21 FOUR program, Education and Research Program for Future ICT Pioneers Seoul National University, in 2025. This research was also conducted as part of the Sovereign AI Foundation Model Project (Data Track),
10

---

organized by the Ministry of Science and ICT (MSIT) and supported by the National Information Society Agency (NIA), Korer (2025-AI Data-wi43).
References
[1] Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14X1:24444.1219, 2024.
[2] Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvlm: Personal- izing vims for user-specific queries. In European Conference Computer Vision, pages 73-91. Springer, 2024.
[3] Peter Anderson. Basura Fernando, Mark Johnson. and Stephen Gould. Spice: Semantic propositional image caption evaluation. In Computer Netherlands, October 11-14 2016, Proceedings. Part 14, pages 382-398. Springer, 2016.
2016: 14th European Conference, Amsterdam, The
[4] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Siho Song, Kai Dang. Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13323,, 2025.
[5] Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarizaaion,, pages 65-72, 2005.
[6] Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1197-19,,, 2020.
[7] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.00000 08701, 2023.
[8] Tianheng Cheng, Lin Song. Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo world: Real-time open-vocabone vocabulary object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16901-16911, 2024.
[9] Jooyoung Choi, Chachun Shin, Ycongtak Oh, Heeseung Kim, and Sungroh Yoon. Style-friendly snr sampler for style-driven generation. arXiv preprint arXiv:2411.14799 2024.
[10] Yisol Choi, Sangkyung Kwak, Kyungmin I Lee, Hyungwon Choi, and Jinwoo Shin. Improving diffusion models for authentic virtual try-on in the wild. In European Conference Computer Vision, pages 206-235. Springer. 2024.
[11] Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, rl generalizes: A comparative study of foundation model post-training. arXiv preprint arXiv:2501.17161, 2025.
[12] Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, and Jingren Zhou. How abilities in large language models are affected by supervised fine-tuning data composition. arXiv preprint 2rXiv:2310.0544 2023.
[13] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
[14] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. arXiv preprint 25t.:2500.12945 2025.
[15] Haoran Hao, Jiaming Han, Changsheng Li, Yu-Feng Li, and Xiangyu Yue. Remember, retrieve and generate: Understanding infinite visual concepts as your personalized assistant. arXiv preprint arX3v:2410.13360. 2024.
[16] Junjie He. Yifeng Geng, and Liefeng Bo Uniportrait: A unified framework for single-and multi-human image personalization. arXiv preprint 2xtv:22088.55939 2024.
[17] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika. Dawn Song, and Jacob Stein- hardt. Measuring massive multitask language understanding, arXiv preprint 20X 2020.
11

---

[18] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning, arXiv preprint arXiv:2104.08718, 2021.
[19] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICT.R, 1(2):3, 2022.
[20] Wenxuan Huang, Bohan Jia, Zijie Zhai, Shaosheng Cao. Zheyu Ye, Fei Zhao, Yao Hu, and Shaohui Lin. Vision-rl: Incentivizing reasoning capability in multimodal large language models, arXiv preprint arXiv:2503.0m 0067:99990....9 2025.
[21] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint 2rX1::2410.21276. 2024.
[22] Sangwon Jang, Jachyeong Jo, Kimin Lcc, and Sung Ju Hwang. Identity decoupling for multi-subject personalization of text-to-image models. arXiv preprint arXiv:2404.04445 2024.
[23] Chanran Kim, Jeongin Lee, Shichang Joung, Bongmo Kim, and Yeul-Min Back. Instantfamily: Masked attention for zero-shot multi-id image generation. arXiv preprint (e0i::24404.992277 2024.
[24] Black Forest Labs. Flux. https: / wtw.t.tcooooooo..... balaack-frrest-laaee labs ffwx, 2024.
[25] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li. Wei Li, Zejun Ma, and Chunyuan Li. Llava-nextinnterleave Tackling multi-image, video, and 3d large multimodal models, arXiv preprint arXiv:2407.07895, 2024.
[26] Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan, A diversity-promoting objective function for neural conversation models. arXiv preprint 150.0:55,, 2015.
[27] Shengzhi Li. Rongyu Lin, and Shichao Pei. Multi-modal preference alignment remedies degradation of visual instruction tuning on language models. arXiv preprint arXiv:2402.1088: 2024.
[28] Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oud, pages 74-81, 2004.
[29] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[30] Haotian Liu, Chunyuan Li, Yuheng Li. and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 36296-26306 2024.
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892 34916, 2023.
[32] Ziche Liu, Rui Ke, Yajiao Liu, Feng Jiang, and Haizhou Li. Take the essenec and discard the dross: A rethinking on data selection for fine-tuning large language models. arXiv preprint arXiv:2406.1444 14/15, 2024.
[33] Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision. pages 3730-3738, 2015.
[34] Ziyu Liu, Zeyi Sun, Yuhang 7ang, Xiaoyi Dong, Yuhang Cao, Haodong Duan, Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning. arXiv preprint
2025.
[35] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings the IEEE conference computer vision and pattern recognition, pages 11 20, 2016.
[36] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Feii Hannaneh Hajishirzi, Luke Zetllemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint Siii:2500.11995 2025.
[37] Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jac Lec. Yo'llava Your personalized language and vision assistant. arXiv preprint arXiv:2406 09400, 2024.
[38] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002.
12

---

[39] Yuang Peng. Yuxin Cui, Haomiao Tang, Zekun Qi. Runpei Dong, Jing Bai, Chumrui Han, Zheng Ge. Xiangyu Zhang, and Shu-Tao Xia. Dreambench++4: A human-aligned benchmark for personalized image generation. arXiv preprint urXiv:2406.1665 2024.
[40] Renjie Pi, Jianshu Zhang, Tianyang Han, Jipeng Zhang. Rui Pan, and Tong Zhang. Personalized visual instruction tuning. arXiv preprint #rXiv:2410007113 2024.
[41] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprini arXiv:3307.01955. 2023.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8788 8763. PmLR, 2021.
[43] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-22510, 2023.
[44] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017.
[45] Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference computer vision, pages 8430-8439, 2019.
[46] Zhihong Shao, Peiyi Wang. Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint 240.::22022.33000 2024.
[47] Haozhan Shen, Peng Liu, Jingcheng Li, Chunxin Fang, Yibo Ma, Jiajia Liao, Qiaoli Shen, Zilun Zhang, Kangjia Zhao, Qianqian Zhang, et al. Vlm-rl: A stable and generalizable r1-style large vision-language model. arXiv preprint 2Xii:5504.006155 2025.
[48] Ming Shen. Rethinking data selection for supervised fine-tuning. arXiv preprint arXiv:2402.06094 2024.
[49] Huajic Tam, Yuheng Ji, Xiaoshuai Hao, Minglan Lin, Pengwei Wang, Zhongyuan Wang, and Shanghang Zhang. Reason-rft: Reinforcement fine-tuming for visual reasoning. arXiv preprint arXiv:2503.20752. 2025.
[50] Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for diffusion transformer. arXiv preprint 1rXiv:2411.1009. 3, 2024.
[51] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean Bean-Baptiste Alayrac. Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[52] Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju. Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based on gemini research and technology- arXiv preprint arXiv:2403.0820 2024.
[53] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[54] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du. Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot leamers. arXiv preprint 11X00:2109.01652. 2021.
[55] Colin White, Samuel Dooley, Manley Roberts, Arka Pal, Ben Feuer, Siddhartha Jain, Ravid Shwartz-Zii Neel Jain, Khalid Saifullah, Siddartha Naidu, et al. Livebench; A challenging, contamination- free llm benchmark. arXiv preprint 2r0iv:2406. 19314, 2024.
[56] Jiazheng Xu, Xiao Liu, Yuchen Wu. Yuxuan Tong, Qinkai Li, Ming Ding, Jie Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:15903- 151155993331555,,.. 2023.
13

---

[57] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He. Shuai Bai, Keqin Chen. Jialin Wang, Yang Fan, Kai Dang, al. Qwen2. S-ommi technical report. arXiv preprint arXiv:2503.2021:: 2025.
[58] An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, al. Qwen3 technical report. arXiv preprint arXiv:2505. 003888 2025.
[59] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pages 69 85. Springer, 2016.
[60] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at seale. arXiv preprint Xrx:: 14476, 2025.
[61] Tianyu Yuan Yao, Haoye Zhang, Taiwen He. Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai- Tao Zheng. Maosong Sun, et al. Rlhf-v: Towards trustworthy milms via behavior alignment from fine- fine-graine grained correctional human feedback. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 13807-138 16, 2024.
[62] Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang, Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint arXiv:2504. 13837, 2025.
[63] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXi:: 09675, 2019.
[64] Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, Zhangchi Feng. and Yongqiang Ma. Llamafactory: Unified efficient tine-tuning 100+ language models. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), Bangkok, Thailand, 2024, Association for Computational Linguistics.
[65] Yaoming Zhu Sidi Lu, Lei Zheng, Jiaxian Guo, Weinan Zhang. Jun Wang, and Yong Yu. Texygen: benchmarking platform for text generation models. In The 4/st international ACM SIGIR conference on research & development in information refrieval, pages 1097-1100, 2018.
14

---

Referense Imsgge
Query Imuaee
SD Ope
reeferenoe lufo
CHO Amazon
-lecup-
Chief
Meta.
CEO of TESLA.
CEO META.
CEO OPENAI.
CEO Google.
NVIDAA.
detaille pensonaizedd cuption image.
A) The query image features Meta), coundar> (CBO of (Goolle), <elos> like setting. wearing lactical gour, suggesting collaburative effört stralogic meeting. Their expressions range from serious smiling, indicatingg mix of professicealism and camaraderic umong these
of inflonntial tech lenders, incloding Sjell> (CEO of Amazun), "jemson> (CEO NVIDIA), <lecunt (Chief Al Scientist TFSI smark (CFO MITA), suanro (CRO OpenAT), They standing together
Figure A.1: Visualization of more complex and diverse our proposed RePIC. The generated query image is sourced from InstantFamily [23].
image captioning result using
Real
Genereted
detailed pensonalized caption
A) The query image shows scustomer]> wearing sounfit 1329s The black t-shin features a white Soussy logo on the frent.
southt
Virtual Try- On Personalization
@) a persomalized caprion image.
Aj this photo, we sce threz dattnguishal individials: =A', profossor at the Univensti Montraal, <Bs, Chier Al Scientist at Meta; and <C>, a Nobel laureate physies 2004, They eaptural in cosual pursuiss.
holding mug, symbolizing their camanadeste shared
Multi-Huuann/Subyyyl Personalization
Q) a exption image
A) strking sayleI> sculpture festuring golden nstroneut dysamic pose, scemingly mid-jump scowboard, The figur's and thick, metaliie with moltes appeacance duat crips downward, the surcal aesherik. plece.
astronaufss light, adding depth
Style-Driven Personalization
Figure A.2: Visualization of RePIC results on various personalized image captioning tasks.
A Additional Qualitative Results
A.1 More Challenging Multi-Concept Setting
In this section, we further present the results of our method in the 7-concept setting, as illustrated in Figure 1. Notably, our approach generates faithful descriptions of the query image while accurately referencing the corresponding reference images and their associated information. In Figure A.2, we demonstrate that our proposed method can faithfully perform image captioning synthetic images across various tasks, including virtual try-on [10], multi-human or subject personalization [16], and style-driven personalization [9], which is compatible with the ttate-of-the-an personalized image generation benchmarks. These results highlight the superiority of our approach in handling diverse personalized image captioning tasks with MILMM
A.2 Visualizations on Image Captioning Quality
In Figure A.3, we present qualitative comparisons of image captioning quality in both single and multi-concept settings, highlighting the effecttiveness of our proposed method. Note that the RAP-
15

---

patsibast
Provide cuption of the given image.
lnfo: The unage ouraina Fiste Pop clasater wifh blz loolle Hadk shin, like join, and wlite meskers tie holding and and in Mo right hand
RAP-LLAV-211K
mole
<boy, firke__rr pop" standing fromt of'a picture of the empere saae huilding <hey, hor__iinkoorggg wearing yellow sunglasses,
<boy _unko, pape holding
captioning
<byy finko pope ready play poker!
Ours-KK
-boy farko pupl is staning in tabic summadddd by ministure tses includnng blse tnsck yellow
beeated "hilippond"
Correct captioning
Give a persoralized caption of image.
batabase
image shows
bulling
dog on neut named buby wrapped
key Kooping posctalle a cania Thi bay
sleeping peseefully
is sasural by black sing. dog lookeas buby with cuntuss expresior, baby an fiong rom with ponerned rag and white couch in the background.
umappet
Wroong captioning
weaad blck anp
RAP-Quen-210K
dog- looking camee
expression.
sibee degs
Ours-KK
light boons
Ia this beattwanning wome, -Bull dop the French Buildog and <Buby Q duare moment togrther <hhull dag has fhoor, fillbly Qu is blanket adomed stmospheee. Correct captioning
plytii mohil
tiem BBaly wrapped
while
dog' conforarlly book crainng
Figure A.3: Examples of generated captions single and 2-concept personalized image captioning tasks.
hikong: selephant>
bxfo: selephimii jay planh elephumt with lange and feet. The ribbed and have lishter trim. The feet ribbed with cream colored trim.
Provide cuption given image.
Ours <elephanc sits on white wooden mocking chair in room with handwood floors and a beige wall.
Ours wio length regularization
Solephun site whice worken mocking chair.
lungth regularization, weing prampt template
celephazes.
Q) Give personalized caption of the given image.
Ours
selephani> is plush claphant tay with large cans and fiet. The cans are ribbed and have lighte aas trim. The foet are ribbod with cream colored trim. Ther snother salephanc> nest to is which has a tig thet says "Lamm lys". Vieral peropption
Ours wio length regulartzation
selephanc
snother celeplurzs.
Ours lungth rigularizatinn who using detail prompt template
celephanc- is soft plush my that is smig on bod.
Figure A.4: Visualization of qualitative results for additional components used for our methods.
LLaVA often merely duplicates the retrieval information or generates visual hallucinations without considering vision perception, and RAP-QU RAP-Qwen severely fails caption correctly. In contrast, only the proposed approach faithfully and concretely describes the given query image.
A.3 Effects of Additional Components for Preserving Captioning Quality
In Figure A.4, we present additional ablation results evaluating the impact of applying length regularization and incorporating detailed prompts in the training dataset. The results indicate that in- corporating length regularization and detailed prompts effectively mitigates generating uninformative captions for the query images.
16

---

Personal Database
<boo-
nnfo | sheo a well-groomed, medium--soed Shila Inu with thick, cinramon-colored cost, croum sccents, alert eyer, and a black coller.
Please give cagtion image
1) <ttinke template
<thnk- imsage shows dag appears is be Saita heae os is plenical eiucaciersiio is the pointed ous, compatt bedy, facial Rasuess dog thick, cistansoo colored coot crean accents, mutcless the discription of -ber. doy's oupresioon and postare alo dlign wits the deses grioa provided oithink-
wellarrweed, aeditue-rtee Skela duverpttis
nag, looking huppy dog's consaneecodoond
oleady suotle
1) <abeerve> templete
soherees The imege shows Stita les fing yeng a white dugggy my indrons, The dog has thick, cieramencoskrrd cout crem accerss aerund it chre and posss It. care am porkel up, household
thiently exprosdon
tongur alightly out The hackgroodd includo wooder floor and some
key asi blue object. colberves
"osswer d anllaaraannt asatioon coaas acceatie
ios ousst haine a shanee out intion, Wistaiia cissunaancnttoo coot and Maxd codar doy's perked up, ond fllendly expeessiu winh tongue slightly our, bockgrood some homnchold here vyy Moo etject. stasserr
wooden
3) No template
<lap siting on white mag its lengue out, looking huppy and relased.
Figure A.5: Visualizations of output responses with and without the use of reasoning templates.
A.4 Effect of Reasoning Templates
We consider the post-tuned model trained with reasoning templates such <think> and <observe> to verify the effectiveness of visual reasoning in personalized tasks, which has become a prevalent choice [34, 20, 47] for MLLM post-training with RL. However, in Figure A.5, we observe that using reasoning templates often results in longer outputs that fail to faithfully describe the query image. In contrast, omitting templates leads to more concise yet accurate and faithful image descriptions. Furthermore, as demonstrated in our main analysis, reasoning templates negatively impact personal grounding performance in personalized image captioning tasks.
A.5 Further Limitations of RePIC
We illustrate the limitations of our RePIC model on the personalized image captioning task in Figure A. In the first row, RePIC incorrectly captions the image with blue jeans, despite no such item being present. A similar issue is observed in the second row, where the model references a polka-dotted dress that does not appear in the query image. These examples show a limitation of RePIC in generating accurate personalized captions, primarily due to insufficient fine- lineggrined visual perception. For instance, struggles when objects are not visibly present (e.g., no blue jeans appear) or when the reference and query images differ significantly (e.g., back view vs. front view), making it difficult to recognize them as the same person or the same object. We expect that these limitations can be mitigated either by constructing high-quality database for each concept-avoiding the use personal information based solely on the visual appearance of the image, and ensuring the reference image clearly shows front view of the object-or by leveraging an MLLM equipped with an advanced vision encoder and a more powerful backbone LLM, such as Qwen-3 [58].
We further acknowledge the inherent limitations of our current personalized retrieval pipeline, particularly in corner cases where relevant concepts in the database are absent from the query image. While our evaluation setting focused solely on object-centric benchmarks and did not account for such cases, we anticipate that addressing them will an important direction for future work. is expected to be particularly important in personalization scenarios that use video as input data. In this regard, we believe robustness could be enhanced by incorporating a verifiable reward that penalizes the model for generating responses based on incorrect or irrelevant information. We also conjecture that such scenarios more clearly expose the limitations of RePIC in handling real-world complexity.
B Additional Experimental Configurations
B.1 Experimental Details
Our implementation is based on the open-source codebase. To train our model, we set LoRA rank as 64, LoRA alpha as 128, and use the number of generations per prompt as 8. The base model we used
17

---

O) Live nevoralizzi carien of dis treaet
Oun Fall young man and warer, dito and cit, regorively squating ofetto's
lnfo youy run
pakke moot <it new ing grea bolien and <To worning wakee tuble wit Moojooos
Wrong ceptioning
Quea DeVV die TT aquai in dven of'a muck.
hitlvd cTs
tof young cocn wnth
Nerds wastty
RAP-Qwas onter and -it viting frant
blue
cxption maes
Namerihh
Ctax indoar seting suir
togeher droudd
Indo : zan weateg
casa sils write sakin and block shorts sis combbnally on unat undl doo. keeingg
whio sckiteed chems
Eylish Miie pottoctionall with hener and white sinrchis was baikee The come enjoy eg Wrong captioning
pevide atronphowy,
compury
faoor
Indo
young soras
eating is chirr santing
Nee polta feer
RAP-LLaYA: dii chair souro
boom cmneckie
Figure A.6: Examples illustrating additional limitations of RePIC in 2-concept scenario.
Converat Upper
23
In Vot Lak
Cd KXSMILL
Figure A.7: Datasets used for training and evaluation. Note that the Subject200K+ dataset (a) was used for training, while all real datasets (b) to (1) were used only for evaluation.
Qwen2. 5-VL- Instrnnt-tr a vision-language model capable of processing images, text, and videos simultaneously, with enhanced mnstructionfollowing capabilities. Following the FLAN [54] paradigm, the pretrained MLLM on a large corpus was further fine-tuned using instruction datasets. We adopted this backbone because it is open-source and supports multi-image understanding, crucial feature for MLLM personalization. We chose the 7B variant as a suitable backbone MLLM for our experiments, and the frozen copy remains fixed throughout training to perform a reference policy while GRPO.
B.2
Used Datasets for Evaluation
The data configuration used for both training and evaluation in our experiments is detailed in Figure A.7. Notably, the Subject200K+ dataset was used exclusively for post-training and was not included in the evaluation. All other real-image benchmarks were used for evaluation purposes. In Figure A.8, we present the configuration of our curated DreamBooth [43] database used for single-concept captioning evaluation in our experiment.
B.3 Used Data Templates
In Figure A.9, we illustrate the data and instructions for verifiable rewards of OCT, VLT, and ICT used for post-training. Table A. shows the two reasoning templates of using <observe> and <think> tokens used for our ablation studies involving special tokens. These templates were appended directly before each captioning query.
18

---

Proposed DremmDooth Databuse
himsee "buckpuckk>
Nams:: <dogg>
<hackpucke- is red backpack with multiple colorful putches attached to it.
<dogá> is dog with distinctive primarily white with putches of gray and black.
hitoug <cberry bowl>
Namig: <hear plushic> <bear plushic brown plush teddy bear with light blue bow on its head.
Info: : <berry bowl> while buw] with black gri puttens and the words "Ren Appétit written gold cursive
Namme:: <backpuck dog-
Nams::
Info : <hackpackk dong- features large, cartoonish face of dog front. The dog's brown and tongue sticking
tifo <can> is beer can with the label "Transetlaaric design features luage with surfer in background.
lighter
side,
Figure A.8: Visualization of the DreamBooth database constructed in this work.
OCT
didd is top canner
fall
Negetive
Please railly wietker the dijett in tiee picures ile As object sorsideed Be
some ifits consitency masniieed Zeeate vafarion it lilliig pass Answer with
Malti-ICT
Multi-SCT
1B- soage?
<
Preside
www.el.werriiooooo weeri--oeedd were--eeddiggg aprias Olee sccrovady
ormigy in image
Singl-ICT
OH (4f)) TIK In Aoloe 15, (1g00) Provide herel ciption of Image
Duscrioc Beinnge dehil givan
Qie Do pewooutttedd capiion of longl
Visualization of data templates used for MLLM post-training, including examples of Figure A.9:
OCT, VLT, single-ICT, and multi-ICT.
B.4 Dataset Compositions
For generating <name>, we use a random name generator to sample human or object names in an on-the-fly manner. To be specific, use the faker library to generate multilingual terms (e.g.. person and object names) in French, Korean, Italian, Chinese, and English in an on-the-fly manner.
To construct multi-concepy concept crop images, first select multi iitiiiimmee training samples (approximately 5% of training data) containing multiple crops from RAP-MLLM. and then manually select only those examples with non-overlapping crops and clearly visible identities. These cropped samples
19

---

(3) Dataset Compositions
(b) Instruction Compositions
(e) Datasct Sensitivity
Object300
Roundding
1.0
17%
Bax
Acarwa (eeurds
Visual Perception
0.8
48%
Bubjou220K++
38%,
Mi-t-ica-tayy 4%%
Heeiiyy instucdin 35%
Singleident by Grounding
Meeio viuuiiio 89%
Duta (561
10%
27%
Figure A.10: (a) Dataset composition, (b) instruction composition, and (c) the sensitivity to the proportion of identity grounding instructions within the overall training set.
Table A.I: Used reasoning templates
<think> Reasoning Template:
First output the thinking process <think> </think> tags and then output the final answer in <answer> </answer> tags.
<observation> Reasoning Template:
First, observe carefully and enclose the observation process in <observe> </ob- serve> tags and then output the final answer in <answer> </answer> tags.
used without applying additional data augmentation, such color jittering rotation. Further, we use the Subject200K+ [50] dataset, which provides pairwise high-quality synthetic images with realistic and diverse lighting and pose variations. By incorporating such synthetic dataset, intended to enhance the MLLM's visual perception capabilities for handling more complex and realistic applications, such personalized Al-generated image captioning.
Figure A. illustrates how we construct show that the dataset is composed of COCO, Objects365, CelebA, and Subject200K (b) visualizes the instruction composition, which includes OCT, VITT, single-ICT, and multi-ICT. We note that approximately 31% of the total training data is composed of single and multi-ICT samples. In (c), we highlight the dataset sensitivity for convergence. We observe that the amount of ICT instruction in training data is too high, the RL training often fails. This demonstrates that while RL is inherently data-efficient, is sensitive to data quality. Despite this, we are the first to empirically identify stable "sweet spot" in data and instruction composition and reward design that effectively balances and stabilizes the training process, which we rigorously validate in our experiments through both in-distribution and cut-of-diitribution experiments. Overall, our findings highlight the importance of a well-structured instruction dataset for effective RL-based post- post-training in MLLM personalization.
dataset for personal grounding. In (a), we
B.5 Details On Retrieval Setting
Following the previous work [15], for a retrieval setting, we first utilize a database of image, text pairs representing user-specific concepts. Given database images, each image first processed using pre-trained CLIP [42] encoder to obtain visual embeddings. Then, for given query image and its corresponding textual instructions, YOLO-World [8] is employed detect regions of interest. Thus, cropped images are encoded into embeddings, and by computing Euclidean distances between these embeddings and pre-stored embeddings, the most relevant reference images are retrieved from the database. Our database is composed of key-value pairs for all concepts, where each concept is associated with an image and corresponding textual information. Then, similar to skip-retrieval, the reference image is retrieved from the database. A detector first generates region proposals from the
20

---

query image, and then reference images are retrieved with their texts by selecting the top-K K samples from the database with high CLIP image cosine similarities.
More specifically, a detector generates ROIs for the query image relevant to the predefined category texts specified for each concept, like "person", "dog", "toy", and so on. Then, the CLIP similarities are computed between multiple ROIs captured from the query image and the reference images in the database. Specifically, for each ROI generated by the detector, we extract its visual embedding using the CLIP image encoder and compute cosine similarity against the image embeddings of candidate reference images, also obtained using the same CLIP model. The top-K most similar reference images and their texts are then selected for MLLM demonstration. Note, we do not use reference texts in the database for the retrieval. Once the reference images and texts are retrieved, they are prepended the query prompt in an in-context learning (ICL) fashion. Importantly. no example answers for the query image are provided; MILLMs generate responses solely based on the retrieved references (images and texts) and the query image in personalized manner.
B.6 Details on Length Regularization
The output length regularization was applied exclusively to the ICT reward, not to OCT or VLT. Further, the ICT reward is granted only when the output exceeds the cutoff length. As discussed in Figure A.4, we observed that naively maximizing the ICT reward could lead the model to exploit trivial shortcuts, such repeatedly generating phrases like "This is <name>." Since such outputs still receive a full reward of I, this results in a reward hacking issue. To address this, we introduced a regularization strategy: if the generated output is shorter than predefined cutoff' length, the reward is set to 0, even if the name is correctly included. We empirically set this minimum length to 100, which corresponds
the lower bound of text lengths in our database. Figure A.17(b) presents qualitative ablation across different thresholds, and we found that a value of 100 offers stable and meaningful reward signals during training. Additionally, to better encourage the model to produce more descriptive captions, we incorporated detailed prompts into the training data (e.g., "Describe the image in detail"), which promotes longer and more informative outputs. Importantly, the model's captioning and instruction GRPO loss.
datruction-folloiing following abilities are preserved through the KL divergence term in the
C Additional Analyses
C.1 Theorical Analysis of GRPO with PPO
We provide a theoretical comparison between PPO and GRPO explain why GRPO is suitable algorithm for RL training in our work. Proximal Policy Optimization (PPO) [44] is policy-based RL algorithm that stabilizes policy updates by employing a clipping mechanism, as represented in the equation below.
PPO (8) E( [min (re(0)A,, clip(r. (1)p((())).11.+()4))))
(A.1)
(0) Tray( s the probability ratio between the new policy Ty and the old policy Fref- e is a small hyperparameter that controls the clipping range. However, PPO heavily relies on absolute reward values, making sensitive to noise or suboptimal design, and its use of a separate value function typically as large as the policy model, for advantage estimation adds significant memory and computational overhead.
[s.
In contrast, GRPO is formulated as group-level extension of PPO, incorporating per-token policy ratios, clipped advantages, and explicit KL regularization, described in Eq. (1) of our paper. Since rewards are applied per sample, GRPO performs better with sparse reward data and stably learns from relatively better responses rather than relying on absolute reward values.
In our work, GRPO offers the following advantages:
Effectively handling multiple identity references: For each query, the model generates multiple responses and takes the most preferred responses at group level.
21

---

Table A.2: Single-concepl copy-and- analysis on wrong textual demonstrations.
Models
BLEU (4)
ROUGE-L (4)
METEOR (L)
SPICE (1)
BERTScore (1)
RAP-LLaVA
0.415
0.897
0.544
0.510
01.113
RAP-Qwen
0.190
0.834
0.361
0.375
0.447
Zero-Sho Shot
0.129
0.677
0.276
0.261
0.427
Ours
0.079
0.578
0.213
0.140
0.340
2. Preserving caption quality during personalization: Instead of relying clipping mech- anism, GRPO explicitly applies KL. divergence regularization, which helps maintain instruction-fllowing capabilities throughout the personalization process.
3. Robust training under sparse reward conditions: These strengths are partieularly evident in single- and multi-concept ICT tasks during post-training, In the early stages, when the MLLM lacks personal grounding ability, rewards tend be sparse and hard learn. However, we observed that as training progresses, the personal grounding ability begins to emerge, and the reward signal becomes richer and more informative.
Therefore, from those perspectives, adopt the GRPO algorithm suitable strategy for our RL-based post-training because of its stability and effectiveness, even when training with a small amount of data (2K).
C.2 Additional Technical Depth of GRPO
We provide additional explanations follows to enhance the clarity of using GRPO. In MLLM post-training using GRPO, the vision encoder remains frozen, and only the backbone LLM is fine-tuned. This ensures that the visual representation (extracted by the frozen encoder) stays stable, while GRPO updates only the language model's parameters based on RL signals.
How are group responses generated?: For each training instance, the model generates a group of G responses fo G by sampling from the reference policy ma. In work. group responses consist of personalized captions for ICT, predicted bounding boxes for VLT, and binary yes/no responses for OCT. These responses are then converted into verifiable rewards to calculate group-relative advantages.
2.
How are group-relative advantages normalized, and what are their merits?: For given set of scalar verifiable rewards composed of ICT, OCT, and VLT, the rewards are grouped and normalized together to calculate the advantage. The formulation of the group advantage provides stable reward signal, even in low-resource or noisy reward settings. In our work, this merit becomes prominent in single and multi-ICT rewards during post-training. In the early stages, the initial MLLM lacks personal grounding capability, resulting in sparse rewards. However, as training progresses, the rewards become richer, and we observe the emergence of personal grounding ability (see Figure 12). We pose that this emergence encouraged by the GRPO on learning from relatively better responses rather than relying on absolute rewards, thereby improving sample efficiency and training stability.
3. How is reference policy established?: GRPO, the reference policy typically established frozen copy of the initial pretrained MLLM, denoted Tref, and remains fixed throughout training. This frozen policy serves as stable anchor point for regularizing the current policy To via explicit KL divergence penalty, which helps prevent policy drift.
C.3 Copy-And-Passe Analysis
We assume that current MLLMs often overlook visual information and rely solely on textual cues without a thorough understanding, To address this issue, we conducted the following experiments. The metrics employed are commonly used in image captioning evaluation, such as BLEU [38], ROUGE-L [28], METEOR [5], SPICE [3], BERTScore [63], and measure the similarity between the generated output and the reference text, skip-retrieved from the database. In this context, higher score indicates greater similarity between the two sentences, which implies a higher degree of copy-and-pace paste behavior.
22

---

Table A.3: Molti-concept copy aopyy-and-pasae analysis on wrong textual demonstrations.
Models
BLEU (1)
ROUGE-L.
METEOR ())
SPICE ())
BERTScore ())
RAP-LLAVA
0.143
0.649
0.284
0.335
0.404
RAP-Qwen
0.012
0.259
0.110
0.025
0.195
Qwen-2.5 VL
0.008
0.080
0.028
0.014
0.179
Ours
0.015
0.240
0.093
0.034
0.188
Table A.4: Comparison of text diversity under different query prompt designs.
Query Prompt
Models
Sclf-BLEU (L)
Distinct-1 (f)
Distinct-2 (1)
RAP-LLaVA
0.444
0.486
0.677
w/o detailed
RAP-Qwen
0.521
0.383
0.588
prompt
Zero-Shot
0.409
0.613
0.542
Ours
0.271
0.539
0.803
RAP-LLaVA
0.324
0.483
0.730
w/ detailed
RAP-Que RAP-Qwen
0.137
0,523
0.815
prompt
Zero-Shot
0.359
0.542
0.782
Ours
0.128
0.624
0.850
In Table A.2, we investigate the extent of copy-and-pase behavior of MLLMs. In detail, skip-retrieval of reference texts, which corresponds to the setting where wrong textual demonstrations intentionally provided with the reference image. To elaborate, Table A.2 evaluates the extent to which the output of RePIC overlaps with the incorrect information given in the reference texts in a single-concept setting. In other words, assesses whether RePIC copies the misleading text verbatim rather than grounding its description in the actual visual content when the demonstration contains information that contradicts the image. As a notable result, despite being fine-tuned on large-scale datasets, SFT-based methods are more vulnerable to the copy-and-pans problem than the zero-shot baseline, In contrast, our RL-based method consistently achieves significantly better performance across all metrics. This suggests that, unlike other SFT-based approaches that often produce templated responses without thoroughly recognizing the query image, our proposed RePIC method enables the post-tuned MLLM to generate personalized captions grounded in proper visual understanding of the query image.
In Table A.3, we present the copy-and-paste results for a multi-concept setting using the RAP-MLLM dataset. In this experiment, we feed incorrect textual demonstrations for both concepts. Our method achieves the second-bes bbstt performance after the zero-shot baseline and shows consistently lower copy- and-paste behavior compared to other SFT-based models. It is important to note that these metrics do not reflect image captioning quality. Upon analyzing the outputs, we observed that the zero-shot baseline, despite producing the lowest overlap scores, frequently fails generate meaningful captions and instead outputs only its names or irrelevant responses in this specific scenario.
To further examine copy-and-paste behavior without considering the textual attack scenario. in Table A.4, we measured the textual diversity between output responses generated for each concept. Here, Self-BLEU [65] treats each caption hypothesis and the others as references, computing BLEU scores accordingly. Distinct-n [26] calculates the ratio of unique n-grams to the total number of generated tokens, offering an intuitive measure of lexical diversity. this context, low diversity, indicated by high S-B and low D scores, suggests that the generated captions for given concept similar each other, implying copy-and-pasee behavior. Conversely, lower S-B and higher D scores indicate greater diversity in generated captions for each concept. Note that the "detailed prompt" refers to appending the phrase "outppt the response without duplicating the given information directly to the system prompt. As shown in Table A.4, our proposed method consistently produces more diverse personalized captions compared to other methods in all settings. These results further reinforce our main claims that RePIC demonstrates strong robustness in distin- guishing between objects and effectively avoids simply replicating the provided information when generating personalized captions.
23

---

General Caploning Quarlyy
General Caplioning Quality
Our
Zero Shot
Ours
Zaro Shot
Our
RAP-Qwen
Oura
RAP-Q
Our-
RAPA
Our
RAPILLWAA
20
65
00
02
100
20
40
00
50
100
Preference [S]
Praference (14
(n) Single-Concept
(b) Multi-Concept
Figure A.11: Additional preference evaluation using Gemini 1iciini-2.0-0 -2.0-Flash single and multi-coneeee personalized image captioning tasks.
6-6-
0.
10
A3
10 110)
0.0
90
25
40
50
80
90
40
05
40
0M
100
(a)
(b) Single
ICT
Figure A.12: Distributions of mean verifiable rewards during training for each task: (a) OCT, (b-c) ICT, and (d) VLT.
C.4 Additional Preference Evaluation Results
We acknowledge the potential bias in ChatGPT-style preference scoring, where responses that are overly polite or "sweet" may favored. To address this issue, we enhance the evaluation Figure in our paper by incorporating results from multiple proprietary MILMs (e.g., Gemini 2. Flash) to validate our superior performance. This multi-mode evaluation strategy mitigates single-model bias and provides a more reliable assessment of overall caption quality. As shown in Figure A. the results are consistent with the preference rankings measured by GPT-4o further reinforcing RePIC's priority in both single- and multi-concept settings.
C.5 How Efficiently Does RL. Maximize Rewards During Post- Post-Training?
Figure A.12 illustrates how efficiently our proposed method achieves personalization of the model. To analyze this, we divide the sections with the criterion of seen data during training into bins and count number responses with a verifiable reward of within each bin. These counts are then normalized by the total number of responses that include both rewards of 0 and I, which we call this score as normalized occurrence. The results show clear upward trend performance across both OCT, single and multi-ICT, once the proportion of seen data exceeds 50%. Here, the total number of seen data is 2K. Notably, ICTs both begin with a low occurrence rate of approximately 0.2 but show sharp emergence towards 1.0 once the seen data surpasses 50% (i.e., 1K samples). These results suggest that our method effectively guides MLLM personalization in a data-efficient and effective manner, armed with carefully designed verifiable rewards, data construction, and instruction compositions. Note, VLT shows relatively stable performance regardless the amount of seen data.
C.6
Can Length Regularization Reward Guides To Prolong Output Completions?
Figure A.13 presents ablation results on output completion lengths of the image captioning task across the evaluation datasets. all cases, applying our length regularization proves as simple yet effective strategy for increasing output lengths, consistently yielding longer completions, surpassing those generated by both zero-shot and SFT (i.e., RAP-Qwen) baselines, which often generate uninformative captions such as 'This is <name>?.
24

---

20
36
No
(8) YoLLaVA
MyVLM
DreamBooth
(d) RAP-MLLM
Figure A.13: Ablation studies on output length distributions of image captioning across single and multi-concept evaluation datasets.
Oxppet Compaten Lanehh
Oupuu Conpiation Lingh
600
(b)
Dowee on IrM
Oxper Conalttin Langjh
600
Qurid pla
400
400
400
200
200
200
10
20
40
50
70
00
100
30
50
80
80
90
100
10
20
40
50
70
80
100
Sean Data (%)
Soon Data (%)
Scan Data (%)
Figure A.14: Visualization of results: (a) measured output response length (e.g. between <answer> and </answer> tokens), (b) output length measured within the reasoning template (e.g. between <think> and </think> tokens), and (c) ablation studies.
C.7 Does Reasoning Template Matter for Personalization?
We further conduct an experiment to investigate whether the reasoning templates have a meaningful impact on personalization. Specifically, we examine the effects of using special tokens and measure the output completion length when no reasoning template is used. Interestingly, as shown in Figure A. A.14, ablation results reveal the following: (a) Even without reasoning template, the model is capable of producing sufficiently long and informative answer responses. (b) In contrast, when using a reasoning template, we observe similar tendency toward overthinking [36], which is a decline in accuracy as the average reasoning time increases, where the model focuses primarily on the reasoning process at the expense of informative answers. Thus, in the same context as our experiments in the main paper, for post-training with RL, eliminating the reasoning template contributes to enhancing the personal grounding.
C.8 Does RePIC Enhance The General Image Captioning Ability of MLLM?
In this section, we compare the captioning performance of our proposed method with the zero-shot baseline on a general image captioning task. The evaluation does not consider both skip-retrieval and retrieval settings, as it focuses solely captioning a single query image using general prompts without any reference images.
As result, in Figure A.15, our method consistently generates more faithful and accurate descriptions for the image compared to the zero-shot model under general query settings. Figure A. we further compare results using detailed query prompts. In this case, the results of both our method and the zero-shot model show nearly equivalent performance caption generation. This suggests that RL-based post-training does not enhance model's ability to perform detailed image captioning beyond what the zero-shot model can already achieve. Rather, our RL-based post-training method reinforces the frequency of more faithful and preferable captions in the output under general query prompt settings. These observations align with the results reported in concurrent studies [62, 14], and quantitative results for the preference evaluations with GPT4o across the MyVLM, YoLLaVa Va, and DreamBooth datasets are presented in Figure 1.
Importantly, these results also demonstrate that our RePIC does not degrade the original model's general captioning capabilities after post-tuning. Unlike SFT approaches, our GRPO-based RL training maintains the model's generalization ability. This is achieved by applying KL-divergence regularization between the reference and target models during training, ensuring that the target model remains close to the reference. Thus, by maximizing verifiable reward while preserving instruction- followin ability through KI-divergence, RePIC generates the preferable personalized image captions without compromising the original model's s zero-shot capabilities.
25

---

Q) Cla you prociie caplion this plano!
Oura
cheerful uniuted dog with a bis unile exprossive cyes os grazy lawa, sumounded by colorful toys, incliding a purple pia and wiloww duis. scene casies wornth and playfilnao.
Zeco Shot
cheertul inmoottd dog engoys some playtime with colortul toys in gracy backyant setting
YeLLaaVA
Q) Give caption of the image.
Ours
platti quikky inside famuiin a smill faunin with alasses sitting the datittond looking out through the wisthlield intotioe vehicle includo moder stcoring wheel with Kis loges and outsile dhow mis greennry testured
Zam-Shut Sucring adventure quirky companion behind whell!
MyVLM
Give caption image.
Ours
actling trditional clas eecompaneed cup filled with wooden The evekes and trdition.
Zaro-Shot
actingg Saatring tradttional caa teapot accompanied delicase rastic wooden
DreomBooch Dataset
Provide caption the given
Ours
vibrut
stee cunot peschted
The include vallon booliground showcases
adding beauty
Shot-Shot
colarul perois perchal malling showceenng viton agrine lads yrezn badkinup.
RAP-MILMM Dataset
Figure A.15: Visualization of image captioning results for general query prompts.
Q) Describe the image in detail.
Oar
The imaee features person standing oundowes, smilng warnty at the cumera, They are wealing alialt bluz with small loga on flats and claces, The anividaal is dressed austite sbott dleved button up shur with relosed fit, anz looket sasallt diaedd their left ann The background showcoses uhmm nnd fhow libely sahin, with grean telisgo auggesting gooten park seming. The lighting appears naland, indicating dayline, and the evenall atmoophere is cheetful and screne.
ZaresShut
The image Ecalonos stending onderrs amilng the camuna They waring light blue cap with small lopa the foott and indivitral dressed short buttoug tncken draned falt background liss with chere homerrs liesls savn and mitias arggreting gander and relaxed
appears of'a lightweigtt fibric
indicates (Lanali possibly
given
The overall attoosphire cheecfil
Q) Deseribe insaee detail.
Oan
The mays depice on wnimo-tyle lo anme fosturing ton cismation On the left, there aa lines white. faffl crzaton nacemhin polar hezr sciall black eyes and pestle snile The creatue's dar appears and detailed, with mis white and light any tones. On the right. there connn with pirk har and areen evss Shte aas cheectil and content expression, with slighs bish on her checks Her hair styled is loob and ale wassing school unilerm that includes e. brows, jacket with pold sccenss white shirt, and and The badkspound suggeta andoor aetting, posibly daasromm oc sinikk envimament with woodes panclinu and fiendy.
with fiieolls expurssion. characteried
visbiie bchind the choractere The mecall atmoqqhinn the
Zere-Shof
mage feateres chametors from anime-style iltummtion the them dutty whine her traedly exprestion
bear's appears detaled,
darker shade brght, Her bax
styleil in with has
Site chee.ful unifiers with pancling. posilly expossione.
background sugerso onhuncin
RAP-M
Figure A.16: Visualization of image captioning results for detail query prompts.
Table A.5: Visualization of Recall scores (%) for 2-concept personal grounding.
2-Concept
AVG
Models
Skip-Ret.
Ret.
Ours-2K woo multi-ICT
43.9
42.7
43.3
Ours-2K w/ multi-ITT
98.8
92.7
95.8
26

---

(a) General Captoning
(b) Detail Captioning Quality
kyya
Cura
ZaroShol Ours
You LOA
Oura
Zero-hot
Ours
Zaro-Suu
Dreenthon
Our-
Zoro Bhot Ours
Zaro Zaro Shoc
20
40
60
80
100
20
40 Prefenence 1%
80
100
Pheterence [4]
Figure A.17: Quantitative results of preference evaluations for the single-image captioning task without reference images, using (a) general query prompts and (b) detailed query prompts. Note that RePIC outperforms the zero-shot model in (a), and achieves comparable results in (b).
2.0
1.2
Doud Pronge
Saggin Tooeg
1.5
tumeel
1.0
0.0
0.0
10
60
50
100
10
40
20
100
Dutta O1)
Sima
Datte (50)
(4) Ki--everreence dynamics
(b) Accumacy reward dynomics
Figure A.18: Visualization of KL-divergence and accuracy reward plots on the seen data.
C.9 Why Multi-ICT is Necessary?
We present the recall scores in the 2-concept settings to verify the need to contain multi-ICT in the training data. As shown in Table A.5, models trained only with single-ICT fail perform well in multi-concept setting. This highlights the necessity of our proposed multi-ICT for improving multi-concept personal grounding performance.
C.10 Sensitiveness of the Quality of Synthetic Data
To assess the generalizability of our method to other synthetic data variants, we incorporated the DreamBench++ dataset while keeping the data composition and hyperparameters consistent. Unlike the FLUUX-baeed Subject200K++ used in our main experiments, DreamBencher contains images from earlier diffusion models (e.g.. SDXL [41]). As shown in Table A.6, our method achieved superior performance in both S.R and R settings. We attribute this to the higher fidelity of Subject200K++ which underwent extensive quality filtering, whereas DreamBench++ lacked sufficient refinement. We conjecture that such image fidelity highly influences our OCT reward maximization during training. Consequently, these results underscore the importance of using high-quality synthetic datasets for effective RL training.
Table A.6: Additional results of varying synthetic data in the 2-concept image captioning task. Models
Skip-Retrieval
Retrieval
Pre.
Recall
F1
Pre.
Recall
F1
Ours w/ Subjec1200K++ [50]
100
98.8
99.4
98.7
92.7
95.6
Ours w/ DreamBench++ [39]
100
89.0
94.2
97.8
80.5
88.3
C.11 Analysis on Hyperparameter Sensitivity
Figure A. 18 presents the results of various ablation studies. In (a), we compare three settings: using only simple prompts, incorporating detailed prompts, and further applying length regularization based
27

---

on verifiable reward function. This reward assigns value of only when the output response length exceeds a predefined cutoff. We the cutoff length to 100, as the average length personal information in our database is approximately 100 tokens tokens-roughly equivalent to at least one complete sentence. To encourage more informative image captions, we regularize the model to generate outputs of at least this length. In (b), we investigate how the expected reward changes with different values of the KL-divergence regularization weight BKL. We also observe that the convergence behavior is influenced by the cutoff length used for length regularization. Our results indicate that the combination of (KL. 0.04 and a cutoff length of 100 yields the best performance.
D Used Templates
D.1 Evaluation Templates
In Table A.7, we present the evaluation prompts used for personalized image captioning.
Table A.7: Prompts used for evaluating the personalized image captioning experiments.
Single-Concept Caption Template:
Give personalized caption of this image.
Give caption of the image.
Can you provide personalized caption for this photo?
Provide a caption of given image.
Multi-Concept Caption Template:
Give personalized caption of this image.
Give personal caption of the image.
Provide a personalized caption of the given image.
D.2 Preference Evaluation Templates
The template used for our preference evaluation is shown below. Rather than favoring captions that merely duplicate retrieved content, we instructed the model to evaluate preferred captions that convey meaningful and accurate information to satisfy the following criteria:
1. Reference Similarity:
Measures how closely the generated caption matches the retrieved reference sentence. higher similarity indicates potential redundancy, and thus a lower preference score is assigned.
2. Captioning Faithfulness: Assesses how accurately the generated caption describes the visual content of the input image.
28

---

Preference Evaluation Template
Retrieval-based Preference Evaluation: You are an evaluation expert. Your task to determine which answer best describes the given image accurately. Carefully analyze the options and select the most appropriate one your final choice.
Input: <Image>
The name of the object in this image is: (Name). The additional information for the given image is: {Info].
The preferable caption is one that is not merely duplication of the given information but provides a meaningful and accurate description. Which one is more preferable caption to the (Name}?
Options:
A: {string1}
B: [string2]
Output the final answer by choosing of the options with single alphabet. Answer: A, B
D.3 Instruction Templates
We further present the system prompts used for OCT and ICTs. In the following, in Tables A.8, A.9, and A.10, we present the full instruction templates used for OCT, ICT, and VLT, respectively, Note, we augment the instructions using CPT-4o.
System Prompt for OCT
As evaluation expert, your task is to verify whether the object identified <name> in the first image is also present in the second image. Answer with yes or no. [Question}
System Prompt for Single-ICT
You are captioning expert. Your task is to generate accurate caption for the second image while referencing the first image. Both images contain the same object. The object the first image is named <name>. {Question)
System Prompt for multi-ICT
You a captioning expert. Your task generate accurate caption for the last query image while referencing the given reference images. The reference images each contain an object, named respectively as <name1>, <name2>, (Question}
These are additional information about the given images except the last image: <name <name2>, and <name3>. {Question}
Each object in the images not including the last image has name: <name1>, <name2>. [Question}
Below is additional information about the object all images except the last one: <name1>, <name2>. [Question}
29

---

Table A.8: Instruction templates used for OCT in training data.
Object Consistency Tuning (OCT) Template:
Please verify whether the objects in these pictures are the same. An object is considered the same if its consistency is maintained despite variations in lighting or pose.
Is <name> visible in this picture?
Is <name> in this image?
Do you see <name> in the photo?
Is <name> present in this photograph?
Can you identify if <name> is captured in this picture?
Is <name> depicted in this image?
Does the picture feature <name>?
Can you confirm if <name> appears in this photo?
Is <name> included in this shot?
Is <name> shown in this image?
Can you tell if <name> is part of this photograph?
Is there any sign of <name> in this picture?
Can you detect <name> in the photo?
Is <name> captured in this image?
Do you recognize <name> in this picture?
Table A.9: Instruction templates used for VLT training data.
Visual Localization Tuning (VLT) Template:
Please provide the bounding box coordinate of the region this sentence describes: <name>.
Give <name> 's bounding box in the image.
Describe <name>'s 's position in the image.
Please provide the coordinates of the bounding box for <name> in the given image.
Specify the rectangular boundaries of <name> in the image.
Give <name>'s position in the following image.
Please provide <name>'s 's bounding coordinates in the image.
Indicate the bounding box for <name> in the image.
Show the bounding box for <name> in the picture.
Specify <name>'s bounding box in the photograph.
Mark <name>'s 's bounding box within the image.
30

---

Table A.10: Instruction templates used for single and multi-ICT in training data.
Identity Consistency Tuning (ICT) Template:
Give caption of the image.
Give personalized caption of this image.
Provide a general caption of the image.
Summarize the visual content of the image.
Create detail caption of the image.
Offer a rich and clear interpretation of the image.
Describe the image in detail.
Render summary of the photo.
Provide a caption of the given image.
Can you provide personalized caption of this photo?
Could you describe this image faithfully?
Generate detailed and accurate description of the image.
Write caption that reflects the contents and context of the image.
Compose meaningful caption that truly represents the image.
Describe the image in personalized and context-aware manner.
Provide a natural- -atural-sounding caption that accurately conveys what is the image.
Craft caption that authentically describes the scene in the image.
Create caption that captures the essence of the image.
Write caption that reflects what's 's visually happening in the photo.
Generate human-like description that accurately represents the image.
Describe this image as you were explaining it to a friend.
Produce relevant and truthful caption based on the image.
Give caption that matches the visual elements in the image.
Summarize the visual content of this image in natural way.
Write an image-groundee caption that remains faithful to the content.
Provide a descriptive sentence that corresponds closely to the image.
31

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the
relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation. While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The paper clearly articulates its contributions and scope both the abstract
and introduction. maintaining consistent and comprehensive narrative throughout the manuscript.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
32

---

Justification: Our paper discusses the study's limitations and outlines future research direc- tions address them.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, c.g., if the approach was only tested few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof's
Question: For each theoretical result, does paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper introduces novel post-training scheme and demonstrates its effec- tiveness through extensive empirical evaluations rather than theoretical analysis.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our paper provides the details for reproducing the main experiments related to the proposed personalized image captioning approach. Additionally, we will release the complete code for both training and inference.
33

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-soneee open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: The checkpoint of the pre-trained model is available on HuggingFace Yeongtak/ Boongtak/RePIC_DUUO RePIICQwen2 5VL_7B. The corresponding training and inference codes are released as open-source on https: htt///gittttt com/ oyt9306/ PPIII.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
34

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our paper presents detailed information on the experimental setup, including data splits, hyperparameters, and design choices, which are described in both the main text and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: In Figure report the mean and standard deviation of rewards across different random seeds to demonstrate the stability of our RL training, independent of the data curriculum used during post-training.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
35

---

Justification: We report the types and number of GPUs used in our experiments, along with the software packages employed for training.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't oodn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / http///errrp.pppee http/pbbli/Bthicccuuudeeeeeeee
Answer: [Yes]
Justification: This research fully adheres to the NeurlPS Code of Ethics. ethical guidelines and standards have been rigorously observed throughout the study.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our paper discusses the potential broader impacts of the proposed research in the conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploymen of technologics that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
36

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our model does not present such risks; therefore, safeguards for responsible data or model release were not deemed necessary.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, we have properly credited the original creators all assets used in the paper, including publicly available models and packages, with accurate and appropriate citations.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswithcode thcode. com datasets
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not include new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
37

---

The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This study does not involve any research involving human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We affirm that no large language model (LLM) used for any critical, original, or non-standard component of the core methodology. LLMs were utilized solely for general writing, editing, formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: for what should or should not be described.
220
38


Reviews:
[{"review_id": "xg3zS8I93Y", "reviewer": "Reviewer_v1W9", "summary": "This paper proposes RePIC, an RL-based post-training framework for personalizing multi-modal language models (MLLMs) in image captioning. By integrating object consistency, visual localization, and identity consistency rewards, the method addresses the data inefficiency of supervised fine-tuning, achieving superior performance in multi-concept scenarios with only 2K training samples.", "questions": "Please see the points listed under Weaknesses", "limitations": "A further consideration is the model’s performance under fine-grained visual discrepancies. When reference and query images diverge notably, the model occasionally produces descriptions that do not accurately reflect the input, hinting at limitations in the visual encoder’s robustness to complex scene variations.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. This paper introduces an innovative reinforcement learning framework for post-training personalized image captioning in  MLLMs, marking a step forward beyond standard SFT approaches. The method is particularly compelling in how it sidesteps data dependency by leveraging well-defined, verifiable reward signals—object consistency, visual grounding, and identity alignment.\n2. The framework performs strongly, especially in multi-concept scenarios. For example, it achieves a 71.0% F1 score on 4-concept skip-retrieval tasks using only 2K samples, rivalling or surpassing SFT models trained on 210K samples.\n3. The paper is well-organized and easy to follow.\n\n**Weaknesses:**\n1. That said, there are a few concerns around robustness and clarity that should be addressed. The framework shows sensitivity to prompt template choices (e.g., the <observe> tag degrades performance by ~10% F1), and high ratios of ICT instructions (>31%) lead to training collapse. This suggests that the integration of components may not yet be fully stable or resilient to variations in input structure.\n2.  The lack of transparency in the dataset preparation. Key aspects such as how <name> entities are labeled (manually or automatically), how multi-concept crops are selected, or whether data augmentation is applied are not clearly specified, which limits reproducibility and hinders broader adoption.\n3. While GRPO intuitively appears to be a suitable optimization strategy for handling multiple identity references, the paper lacks a thorough empirical or theoretical comparison with alternative RL methods such as PPO. As a result, the specific advantages of GRPO remain somewhat speculative rather than well-substantiated.", "was_revised": true, "final_justification": "My question has been answered, and I'll keep my score.", "rebuttal": "We thank the reviewer for your thorough reading of our paper and for providing valuable and constructive feedback. We have addressed each concern below and will incorporate all necessary revisions in the final camera-ready version.\n\n# Strength\n\nWe sincerely appreciate the reviewer’s positive feedback highlighting our innovative RL framework for personalizing MLLMs, its effectiveness in addressing limitations of prior data-centric SFT methods, and its strong performance in multi-concept personalization scenarios.\n\n---\n# Weaknesses\n\n**W1) Prompt sensitivity and training instability Issues**\n\nWe appreciate the reviewer’s insight regarding the inherent instability of RL training and its sensitivity to the quality of the training dataset. RL is widely recognized as a challenging optimization paradigm [1], requiring careful reward design and delicate data curation, especially when compared to the relative simplicity of SFT.\n\n- **(1) Prompt sensitivity mitigation**: We observed that when guiding MLLMs to include \\<name\\> in the response, the model tends to generate generic captions without mentioning the name in the evaluation setting, if detailed prompts are not included in the training dataset. This phenomenon is especially prominent when the generated sentence becomes longer. We attribute this behavior to insufficient exploration of the model during RL training. To address this issue, we used diverse and difficult training prompts (see Tables A.4 to A.6) to prevent shortcut learning and to ensure stable personal grounding performance during evaluation.\n\n- **(2) Training instability mitigation**: In our setup, training stability is further challenged by the need to simultaneously:  \n(1) preserve instruction-following capability via a KL penalty with the frozen LLM, and  \n(2) maximize all different verifiable rewards (OCT, ICT, VLT).  \nDespite this, we are the first to empirically identify a stable “sweet spot” in data and instruction composition and reward design that effectively balances and stabilizes the training process, which we rigorously validate in our experiments through both in-distribution and out-of-distribution experiments. We will include this important point in the Limitation section.\n\n---\n\n**W2) Data Preparation Detail**\n\n- **(1) How \\<object\\> entities are labeled**: As described in Appendix B.4, we use the faker library to generate multilingual terms (e.g., person and object names) in French, Korean, Italian, Chinese, and English in an on-the-fly manner. \n\n- **(2) How multi-concept crops are selected**: For multi-concept crop images, we first select multi-image training samples (approximately 5% of training data) containing multiple crops from RAP-MLLM [2], and then manually select only those examples with non-overlapping crops and clearly visible identities. These cropped samples are used without applying additional data augmentation, such as color jittering, flip, or rotation. \n\n- **(3) Is data augmentation applied?**: Rather than applying data augmentation, we use the Subject200K+ [3] dataset, which provides pairwise high-quality synthetic images with realistic and diverse lighting and pose variations. By incorporating such a synthetic dataset, we intended to enhance the MLLM’s visual perception capabilities for handling more complex and realistic applications, such as personalized AI-generated image captioning (e.g., Figure 1, A.1-2). We will clarify these implementation details in the main paper.\n\n---\n\n**W3) Empirical or theoretical comparison of GRPO with PPO**\n\nWe provide a theoretical comparison between PPO and GRPO to explain why GRPO is a suitable algorithm for RL training in our work. Proximal Policy Optimization (PPO) [4] is a policy-based RL algorithm that stabilizes policy updates by employing a clipping mechanism, as represented in the equation below (We use the same notations in Eq. (1) of our paper). \n\n$$\nL^{PPO}(\\theta) = E_{(s, a) \\sim \\pi_{ref}} \\left[ \\min \\left( r_t(\\theta) \\hat{A}_t, \\, \\text{clip}(r_t(\\theta), 1 - \\epsilon, 1 + \\epsilon) \\hat{A}_t \\right) \\right]\n$$\n\nwhere $r_t(\\theta) = \\frac{\\pi_\\theta(o_{t} \\mid q, o_{<t})}{\\pi_{ref}(o_{t} \\mid q, o_{<t})}$ represents the probability ratio between the current and reference policies, $\\hat{A}_t$ is an advantage at time step t computed using the GAE (Generalized Advantage Estimation), and $\\epsilon$ is a small hyperparameter controlling the clipping range. However, PPO heavily relies on absolute reward values, making it sensitive to noise or suboptimal design, and its use of a separate value function typically as large as the policy model, for advantage estimation adds significant memory and computational overhead.\n\nIn contrast, GRPO is formulated as a group-level extension of PPO, incorporating per-token policy ratios, clipped advantages, and explicit KL regularization, as described in Eq. (1) of our paper. Since rewards are applied per sample, GRPO performs better with sparse rewards and stably learns from relatively better responses rather than relying on absolute rewards.\n\nIn our work, GRPO offers the following advantages:\n- **(1) Effectively handling multiple identity references**: By maximizing our proposed OCT and VLT rewards, the MLLM gains enhanced visual perception and localization capabilities. Armed with these abilities, the model learns to generate faithful captions accurately grounded in multiple identities present within the query image, even without additional visual grounding signals, while maximizing single or multi-ICT rewards.\n\n- **(2) Preserving caption quality during personalization**: Instead of relying on a clipping mechanism, GRPO explicitly applies KL divergence regularization, which helps maintain instruction-following capabilities throughout the personalization process (see lines 126–128 in the Appendix).\n\n- **(3) Robust training under sparse reward conditions**: These strengths are particularly evident in single- and multi-concept ICT tasks during post-training. In the early stages, when the MLLM lacks personal grounding ability, rewards tend to be sparse and hard to learn. However, we observed that as training progresses, the personal grounding ability begins to emerge, and the reward signal becomes richer and more informative (see Figure A.11).\n\nTherefore, from those perspectives, we adopt the GRPO algorithm as a suitable strategy for our RL-based post-training because of its stability and data efficiency, even when training with a small amount of data (2K). For a more in-depth technical explanation of GRPO, please refer to the authors’ response to Reviewer 3m5o [W1]. For a detailed theoretical comparison between GRPO and PPO, please see Section 4.1.1 of [1].\n\n---\n\n**Limitations**\n\n\nAs the reviewer rightly noted, our model’s performance degrades when the reference and query images differ significantly (see Figure A.6 and Section A.5). This limitation stems from the MLLM’s difficulty in understanding high-level visual discrepancies. We agree with the reviewer that a stronger visual encoder that enables fine-grained visual perception and understanding could help mitigate this issue. Additionally, as noted in our response to Reviewer 3m5o [L1], we pose that further enhancing the visual encoder to capture nuanced personal styles and visually contextual understanding could enable the MLLM to better handle more diverse and complex personalization scenarios.\n\n---\n# References \n\n[1] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint arXiv:2402.03300 (2024).\n\n[2] Hao, Haoran, et al. \"Remember, retrieve and generate: Understanding infinite visual concepts as your personalized assistant.\", The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.\n\n[3] Tan, Zhenxiong, et al. \"Ominicontrol: Minimal and universal control for diffusion transformer.\", The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.\n\n[4] Schulman, John, et al. \"Proximal policy optimization algorithms.\" arXiv preprint arXiv:1707.06347 (2017)."}, {"review_id": "gNn16HbHAM", "reviewer": "Reviewer_9oCi", "summary": "This paper introduces RePIC, an RL-based post-training framework designed to enhance the personalized image captioning abilities of MLLMs. It highlights the limitations of existing SFT-based methods in handling complex scenarios like multi-concept image captioning. RePIC incorporates three key verifiable reward components: Object Consistency, Visual Localization, and Identity Consistency, to boost the model's visual recognition and personalized generation capabilities. Experimental results demonstrate that RePIC achieves comparable performances in single-concept personalized image captioning tasks, and surpasses existing approaches in multi-concept personalized image captioning tasks, even with limited training data.", "questions": "Please refer to the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths：\n1. The paper is well-written and easy to follow.\n2. The framework is illustrated with clear diagrams that effectively aid comprehension.\n\nWeaknesses：\n1. Lack of Copy-Paste Analysis：The baseline achieves high performance in single-concept evaluation. This might result from the copy-paste phenomenon in MLLMs, where the model merely duplicates concept from reference concept without truly understanding the image. The paper doesn’t seem to analyze this aspect.\n2. Insufficient Evaluation Metrics : The evaluation metrics only focus on whether the reference concept is correctly mentioned, which simplifies the personalized image captioning task into a retrieval problem. They fail to reflect the overall quality of the generated captions. Though Fig 4 presents two cases with satisfactory results, there’s a lack of objective quantitative metrics like BLEU, CIDEr or any other reference-free caption evaluation metrics to assess caption quality after post-training.\n3. Incomplete Ablation Study: The ablation study in Table 4 doesn’t include results from the base model (without any post-training) and models using only one type of reward respectively.", "was_revised": true, "final_justification": "I appreciate the authors’ considerable effort in addressing my concerns. The rebuttal has fully resolved my doubts and, crucially, demonstrated that the paper does not simply apply GRPO to personalized captioning as a trend-chasing exercise, but rather provides solid experimental evidence of GRPO’s effectiveness for this task. Consequently, I am raising my score to 4. I encourage the authors to incorporate the additional experiments and analyses from the rebuttal into the revised version.", "rebuttal": "We sincerely thank the reviewer for carefully reading our paper and providing valuable suggestions for improvement. The authors are very pleased that your insightful and professional comments have significantly enhanced the quality of the current manuscript. We have carefully considered your feedback and made every effort to reflect it faithfully. We will incorporate the following changes into the final camera-ready version.\n\n---\n# Weaknesses\n\nWe have conducted extensive and quantitative evaluations to address the various weaknesses the reviewer pointed out. Please note that, due to space limitations, detailed descriptions of the evaluation metrics used will be provided in the Appendix. For single-concept evaluation, we use YoLLaVA dataset, and the reference captions are obtained via skip-retrieval from the database.\n\n**W1) Copy-and-Paste Analysis**\n\nWe strongly agree with the reviewer’s concern that current MLLMs often overlook visual information and rely solely on textual cues without adequate understanding. To address this issue, we conducted the following experiments.\n\n**Table R.1**: Results of reference-based copy-and-paste analysis via measuring metric scores.\n\n| **Models**        | **BLEU (↓)** | **ROUGE-L (↓)** | **METEOR (↓)** | **SPICE (↓)** | **BERTScore (↓)** |\n|-------------------|--------------|------------------|----------------|----------------|-------------------|\n| RAP-LLaVA-210K    | 0.415        | 0.897            | 0.544          | 0.510          | 0.713             |\n| RAP-Qwen-210K     | 0.190        | 0.834            | 0.361          | 0.375          | 0.447             |\n| Qwen-2.5 VL       | 0.129        | 0.677            | 0.276          | 0.261          | 0.427             |\n| **Ours-Full-2K**  | **0.079**    | **0.578**        | **0.213**      | **0.140**      | **0.340**         |\n\nIn Table R.1, we investigate the extent of copy-and-paste behavior of MLLMs. The metrics employed are commonly used in image captioning evaluation and measure the similarity between the generated output and the reference text, skip-retrieved from the database. In this context, a higher score indicates greater similarity between the two sentences, which implies a higher degree of copy-and-paste behavior. As a notable result, despite being fine-tuned on large-scale datasets, SFT-based methods are more vulnerable to the copy-and-paste problem than the zero-shot baseline. In contrast, our RL-based method consistently achieves significantly better performance across all metrics. This suggests that, unlike other SFT-based approaches that often produce templated responses without thoroughly recognizing the query image, our proposed RePIC method enables the post-tuned MLLM to generate personalized captions grounded in proper visual understanding of the query image.\n\n**Table R.2**: Results of reference-free copy-and-paste analysis via measuring text diversities.\n\n| **Prompt**           | **Models**        | **S-B (↓)** | **D-1 (↑)** | **D-2 (↑)** |\n|----------------------|-------------------|----------------|-------------|-------------|\n| **w/o detail prompt**     | RAP-LLaVA-210K    | 0.444          | 0.486       | 0.677       |\n|                     | RAP-Qwen-210K     | 0.521          | 0.383       | 0.588       |\n|                      | Qwen-2.5 VL       | 0.409          | **0.613**       | 0.542       |\n|                      | **Ours-Full-2K**  | **0.271**      | 0.539   | **0.803**   |\n| **w/ detail prompt**      | RAP-LLaVA-210K    | 0.324          | 0.483       | 0.730       |\n|                      | RAP-Qwen-210K     | 0.137          | 0.523       | 0.815       |\n|                      | Qwen-2.5 VL       | 0.359          | 0.542       | 0.782       |\n|                      | **Ours-Full-2K**  | **0.128**      | **0.624**   | **0.850**   |\n(S-B : Self-BLEU, D : Distinct-n)\n\nTo further examine copy-and-paste behavior, we measured the textual diversity between output responses generated for each concept. Here, Self-BLEU (S-B) [1] treats each caption as a hypothesis and the others as references, computing BLEU scores accordingly. Distinct-n (D) [2] calculates the ratio of unique n-grams to the total number of generated tokens, offering an intuitive measure of lexical diversity. In this context, low diversity, indicated by high S-B and low D scores, suggests that the generated captions for a given concept are similar to each other, implying copy-and-paste behavior. Conversely, lower S-B and higher D scores indicate greater diversity in generated captions for each concept. Note that the “detailed prompt” refers to appending the phrase “output the response without duplicating the given information” directly to the system prompt. As shown in Table R.2, our proposed method consistently produces more diverse personalized captions compared to other methods in all settings.\n\nIn summary, these results further reinforce our main claims (lines 255–256) that RePIC demonstrates strong robustness in distinguishing between objects and effectively avoids simply replicating the provided information when generating personalized captions.\n\n**W2) Image Caption Quality Measure**\n\nWe conduct both reference-based and reference-free evaluations of image captioning quality on the YoLLaVA dataset. It is important to note that the employed metrics do not assess the degree of personalization achieved by the MLLM; rather, they measure only the overall quality of the generated captions for the given query images.\n\n**Table R.3**: Image caption quality evaluation results with reference captions.\n\n| **Types**        | **Metrics**       | **RAP-LLaVA** | **RAP-Qwen** | **Zero-Shot** | **Ours**     |\n|------------------|-------------------|---------------|--------------|---------------|--------------|\n| Reference-based  | BLEU (×10⁻²)      | 0.260         | 0.170        | 0.210         | **0.290**    |\n|                  | CIDEr             | 0.193         | 0.185        | **0.208**     | 0.194        |\n|                  | METEOR            | 0.242         | 0.267        | 0.271         | **0.321**    |\n|                  | SPICE             | **0.104**     | 0.084        | 0.083         | 0.086        |\n|                  | BERTScore         | **0.683**     | 0.567        | 0.523         | 0.668        |\n\nTable R.3 presents a metric-based comparison between the generated captions and GT captions (not reference text in the database) for the query image. Since no GT captions are available for the used evaluation dataset, we generated them using GPT-4o by varying the seed three times. As a result, ours show relatively similar results trends, with minor differences in the top 1–2 rankings. In detail, our proposed method achieves the best performance in BLEU and METEOR, second-best in CIDEr, SPICE, and BERTScore.\n\n**Table R.4**: Image caption quality evaluation results without reference captions.\n\n| **Types**        | **Metrics**         | **RAP-LLaVA** | **RAP-Qwen** | **Zero-Shot** | **Ours**     |\n|------------------|---------------------|---------------|--------------|---------------|--------------|\n| Reference-free   | CLIPScore (↑)       | 0.332         | 0.316        | 0.323         | **0.339**    |\n|                  | ImageReward (↑)     | -0.094        | 0.087        | **0.287**     | 0.130        |\n\nTable R.4 presents the results of image-text alignment evaluation without reference captions. CLIPScore measures the cosine similarity between image and text embeddings, while ImageReward [3] is a general-purpose reward model aligned with human preferences that evaluates image-text alignment quality. On the YoLLaVA dataset, our proposed method achieved the highest CLIPScore and the second-best ImageReward score. \n\nTo sum up, these metric-based single-concept personalized captioning quality evaluation results indicate that our RePIC performs comparably to and is occasionally even preferred over zero-shot or SFT-based methods, even without considering personal grounding ability.\n\n**W3) Additional Ablation Studies**\n\n**Table R.5**: Additional ablation results of 2-concept image captioning.\n\n| **Models**         | **S.R** | Pre. | Recall | F1   | **R** | Pre. | Recall | F1   |\n|--------------------|---------|----------------|--------|------|-------|--------|--------|------|\n| Zero-Shot          | | 100                     | 75.0   | 85.7 | | 98.1                | 64.0   | 77.5 |\n| Ours only ICT      | | 100                     | 29.9   | 46.0 | | 98.0                | 25.0   | 39.5 |\n| Ours only OCT      | | 100                     | 12.8   | 22.7 | | 97.4                | 16.5   | 28.3 |\n| Ours only VLT      | | 100                     | 17.7   | 30.1 | | **98.9**                | 18.3   | 29.9 |\n| Ours \t\t         | | **100**                     | **98.8**   | **99.4** | | 98.7                | **92.7**   | **95.6** |\n(S.R : Skip-retrieval, R : Retrieval)\n\nWe include the results of (1) zero-shot and (2) a post-tuned MLLM trained using only a single type of reward in Table R.5. All cases trained with only one type of reward show consistently low performance in personal grounding. Among them, maximizing ICT rewards yields relatively better results. Nonetheless, the results highlight that combining all proposed rewards, designed to reinforce both visual recognition and personal grounding, is essential for the success of our RePIC.\n\n---\n# References \n\n[1] Zhu, Yaoming, et al. \"Texygen: A benchmarking platform for text generation models.\", The 41st international ACM SIGIR conference on research & development in information retrieval, 2018.\n\n[2] Li, Jiwei, et al. \"A diversity-promoting objective function for neural conversation models.\" Annual Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics, 2016.\n\n[3] Xu, Jiazheng, et al. \"Imagereward: Learning and evaluating human preferences for text-to-image generation.\" Advances in Neural Information Processing Systems, 2023."}, {"review_id": "s0iZoZsl2M", "reviewer": "Reviewer_3m5o", "summary": "This paper introduces RePIC, an innovative reinforcement learning (RL)-based post-training framework designed to enhance Multi-modal Large Language Models (MLLMs) for personalized image captioning, especially in multi-concept scenarios. The authors observe that existing supervised fine-tuning (SFT) methods struggle with faithful descriptions in complex real-world settings due to data limitations. To address this data-centric issue, RePIC leverages the Group Relative Policy Optimization (GRPO) algorithm with verifiable reward mechanisms (e.g., Object Consistency Tuning (OCT), Visual Localization Tuning (VLT), Identity Consistency Tuning (ICT)), eliminating the need for an auxiliary reward model. Experimental results clearly demonstrate RePIC's significant outperformance over existing SFT baselines in both visual recognition and personalized generation, particularly excelling in the challenging multi-concept image captioning task.", "questions": "1. How sensitive is the performance to the quality and diversity of the synthetic images used for OCT?\n\n2. For Multi-ICT, how does the model handle potential ambiguities or varying relevance when associating multiple mentioned names with objects in the query image?", "limitations": "1. The demonstrated personalization primarily focuses on identifying and describing specific named objects, and its extendability to more nuanced personal styles or contextual understanding remains to be fully explored.\n\n2. The direct use of GRPO without a special design for the task of this paper.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1）This work introduces the first RL-based post-training framework for personalized MLLM image captioning, effectively overcoming data acquisition bottlenecks inherent in traditional SFT methods.\n\n2）Effective Solution for Multi-Concept Captioning: The paper successfully identifies and addresses the challenge of acquiring high-quality data for multi-concept image captioning, providing an RL-based solution that mitigates this limitation.\n\nWeakNess\n\n1) Limited Technical Depth on GRPO: The paper's explanation of GRPO's specific adaptation within the MLLM context, such as how the group of responses is generated or the reference policy established, could be more detailed.\n\n2) The paper mentions applying \"output length regularization,\" but more specifics on its implementation or impact on model behavior would be beneficial.", "was_revised": true, "final_justification": "The authors have done a commendable job addressing the concerns regarding the limited technical depth of GRPO and the implementation of output length regularization. The explanations provided are both rigorous and informative:  thoughtful treatment of reward hacking in ICT through output length regularization and descriptive prompts.\n\nThese clarifications significantly improve the technical clarity of the method and demonstrate a strong grasp of RL training nuances in the context of MLLM personalization.", "rebuttal": "We sincerely appreciate the reviewer’s thorough reading of our paper and the thoughtful, professional review. Reflecting on the reviewer’s comments and incorporating revisions has helped us clarify ambiguous parts of our paper and gain valuable insight into how to better highlight our contributions.\n\n# Strengths\n\nWe are thrilled to introduce the first RL-based post-training framework for personalized MLLM image captioning. We are also pleased that our method effectively addresses the data acquisition bottleneck of SFT-based approaches. Furthermore, we appreciate the reviewer’s recognition that our paper successfully identifies and tackles the challenges of multi-concept captioning tasks.\n\n---\n# Weaknesses\n\n**W1) Limited technical depth of GRPO**\n\nWe appreciate the reviewer’s feedback on the limited technical depth in our current description of the GRPO. Accordingly, we provide additional explanations as follows to enhance the clarity of using GRPO.\n\nIn MLLM post‑training using GRPO, the vision encoder remains frozen, and only the backbone LLM is fine‑tuned. This ensures that the visual representation (extracted by the frozen encoder) stays stable, while GRPO updates only the language model’s parameters based on RL signals.\n\n- **(1): How are group responses generated?**: For each training instance, the model generates a group of G responses $\\{o_i \\}^G (i=1, \\cdots, G) $ by sampling from the reference policy $\\pi_{\\theta}$.  In our work, $\\{o_i \\}^G$ can be constructed with a group of personalized captions for ICT, or predicted bounding boxes for VLT, or binary yes/no responses for OCT. These responses are then converted into verifiable rewards to calculate group-relative advantages.\n\n- **(2) How are group-relative advantages normalized, and what are their merits?**: For a given set of scalar verifiable rewards, the rewards are grouped and normalized together to calculate the advantage. Given a group of rewards $r = \\{r_1, r_2, \\cdots, r_G\\}$ obtained from each generated response, the group-relative advantage is defined as $\\hat{A}_i = \\frac{r_i - \\mu}{\\sigma}$, where $\\mu$ and $\\sigma$ represent the mean and standard deviation of the rewards across the group. This formulation provides a stable reward signal, even in sparse reward settings. In our work, this merit becomes prominent in single and multi-ICT rewards during post-training. In the early stages, the initial MLLM lacks personal grounding capability, resulting in sparse rewards. However, as training progresses, the rewards become richer, and we observe the emergence of personal grounding ability (see Figure A.11). We pose that this emergence is encouraged by the GRPO on learning from relatively better responses rather than relying on absolute rewards, thereby improving sample efficiency and training stability.\n\n- **(3) How is reference policy established?**: In Eq (1) of GRPO, the reference policy $\\pi_\\theta$ is typically established as a frozen copy of the initial pretrained MLLM, denoted as $\\pi_{ref}$, and remains fixed throughout training. This frozen policy serves as a stable anchor point for regularizing the current policy $\\pi_\\theta$ via an explicit KL divergence penalty, which helps prevent policy drift.  \n\nWe will clarify these technical aspects and incorporate the additional explanations of GRPO in the camera-ready version.\n\n**W2) Details on output length regularization**\n\nWe acknowledge that our previous explanation lacked clarity, and we provide a more detailed description. The output length regularization was applied exclusively to the ICT reward, not to OCT or VLT. Further, the ICT reward is granted only when the output exceeds the cutoff length. As discussed in Appendix C.2, we observed that naively maximizing the ICT reward could lead the model to exploit trivial shortcuts, such as repeatedly generating phrases like “This is \\<name\\>.” Since such outputs still receive a full reward of 1, this results in a reward hacking issue. To address this, we introduced a regularization strategy: if the generated output is shorter than a predefined cutoff length, the reward is set to 0, even if the name is correctly included. We empirically set this minimum length to 100, which corresponds to the lower bound of text lengths in our database. Figure A.17(b) presents a qualitative ablation across different thresholds, and we found that a value of 100 offers stable and meaningful reward signals during training. Additionally, to better encourage the model to produce more descriptive captions, we incorporated detailed prompts into the training data (e.g., “Describe the image in detail”), which promotes longer and more informative outputs. Importantly, the model’s captioning and instruction-following abilities are preserved through the KL divergence term in the GRPO loss. Consequently, as illustrated in Figure 3(b), the combination of detailed prompting and output length regularization synergistically enables the MLLM to generate the most desirable personalized captions across a variety of scenarios.\n\n---\n# Questions\n\n**Q1) Sensitiveness of the quality of synthetic data**\n\n**Table R.1**: Additional results of varying synthetic data in 2-concept image captioning task.\n\n| **Models**         | **S.R** | Pre. | Recall | F1   | **R** | Pre. | Recall | F1   |\n|--------------------|---------|----------------|--------|------|-------|--------|--------|------|\n| Ours w/ Subject200K++ [1]    | | **100**                     | **98.8**   | **99.4** | | **98.7**                | **92.7**   | **95.6** |\n| Ours w/ DreamBench++ [2]     | | 100                     | 89.0   | 94.2 | | 97.8                | 80.5  | 88.3 |\n(S.R : Skip-retrieval, R : Retrieval)\n\nTo assess the generalizability of our method to other synthetic data variants, we incorporated the DreamBench++ dataset while keeping the data composition and hyperparameters consistent. Unlike the FLUX-based Subject200K++ used in our main experiments, DreamBench++ contains images from earlier diffusion models (e.g., SDXL). As shown in Table R.1, our method achieved superior performance in both S.R and R settings. We attribute this to the higher fidelity of Subject200K++, which underwent extensive quality filtering, whereas DreamBench++ lacked sufficient refinement. We conjecture that such image fidelity highly influences our OCT reward maximization during training. Consequently, these results underscore the importance of using high-quality synthetic datasets for effective RL training.\n\n**Q2) How model handle the potential ambiguities or varying relevances?**\n\nWe sincerely appreciate the insightful question. During training, Multi-ICT assigns different GT identities to manually selected, non-overlapping crops within a real image (i.e., COCO), and trains the model by rewarding the frequency of those identities appearing in the generated caption (please see Eq.(5) and Figure A.9); thus, we do not consider any potential ambiguity. On the other hand, at inference time, ambiguities may arise when the assigned reference name or textual information is incorrect or when the query image differs significantly from the reference. To test the robustness of our model, we design a visual attack scenario as described in Table 3, demonstrating that RePIC reveals robust performance against such attacks. Additionally, Figure A.4 in the Appendix shows that even when the same object (i.e., elephant) appears twice, the model can distinguish between the instances using position or contextual clues. \n\n---\n# Limitations\n\n**L1) Extendability of RePIC on style and contextual understanding**\n\nWe fully agree with the reviewer’s comment. In our paper, we only consider scenarios where visually explicit and object-centric images are provided for the personalization task. Although we briefly present a qualitative example of style personalization in Figure A.2, to the best of our knowledge, research on this domain remains very limited. We consider such extendable cases to fall under implicit personalization, where factors such as visual style, surrounding environment, background, and user intent must be interpreted contextually. We conjecture that if the visual encoder is equipped with the ability to perform such contextual understanding, it could enable more realistic and diverse personalization scenarios. This also opens up opportunities to explore new post-training methods tailored for these tasks. We will include this point in the future work section.\n\n**L2) Direct use of GRPO**\n\nWe acknowledge that the novelty of RePIC may be somewhat limited due to its direct use of GRPO for post-training. However, following concurrent work, Visual-RFT [3] that uses GRPO for MLLM post-training emphasizes that careful and verifiable reward design is both important and challenging. Reviewer v1W9 also acknowledged that our work presents an innovative RL framework and leverages well-defined, verifiable reward signals to reduce data dependency. Furthermore, while current RL-based approaches have shown some effectiveness in specific tasks like math or visual recognition, our work is the first to explore RL in the context of MLLM personalization, demonstrating that it can successfully achieve generalizable captioning performance even in OOD scenarios not seen during training. For future work, we hope to explore more specialized algorithmic designs of GRPO tailored for personalization tasks targeting more complex scenarios, such as those involving 4 or more concepts.\n\n---\n# References \n\n[1] Tan, Zhenxiong, et al. \"Ominicontrol: Minimal and universal control for diffusion transformer.\", The IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025.\n\n[2] Peng, Yuang, et al. \"Dreambench++: A human-aligned benchmark for personalized image generation.\", International Conference on Learning Representations, 2025.\n\n[3] Liu, Ziyu, et al. \"Visual-rft: Visual reinforcement fine-tuning.\", International Conference on Computer Vision, 2025."}, {"review_id": "VKOMWiPJD4", "reviewer": "Reviewer_MnDT", "summary": "- The authors aim at identifying persons or objects by their name/identifier in images when answering a question in the sense of visual question answering. This is facilitated by presenting other images with these persons/objects as examples at inference times, either by explicit examples or by retrieving them from a source. They call this setting personalized image captioning.\n\n- The authors present a method to retrain models (not using the term finetuning here to avoid confusion with the term supervised fine-tuning) using GRPO from one of the papers of the deepseek-AI group of researchers and compare this against supervised finetuning. \nThey define three types of rewards for use with GRPO supported by corresponding questions to an image. \nObject consistency aims to train the model to identify an object in the target image, visual localization aims at achieving the output of bounding box coordinates with a high IoU. Identity consistency aims at the model outputting the correct identifiers for objects given by examples. \n\n- They evaluate multiple aspects.\n\n  - Firstly it is precision, recall and F1 for outputting the correct identifiers in single and multi-concept settings, which is a central goal of the setup of personalized image captioning. They compare here against a supervised finetuning baseline with a large dataset and a small dataset comparable to the one used for GRPO. They compare it with 1,2 and 4 identifiers present. \n  - Secondly they measure chat-GPT4o preference ratings. \n  - Thirdly they measure robust to providing examples of wrong objects not present in the target image.\n  - Finally they perform an ablation on the three reward generating functions.", "questions": "- see questions in the weaknesses section:\n- what is the base model used to sample outputs as reference policy ? the untuned Qwen baseline ? The model used in  training from the previous epoch ?\n- it is not fully clear what the difference is between skip-retrieval and retrieval mode: \nskip-retrieval presents examples of images with identifiers, query images, and example answers for the query images ? does skip-retrieval show in all of its examples only the images with the correct identifiers  (correct = present in the query image)? \n[...]\n- appendix:  \"YOLO-World [1] is employed to detect regions of interest.\" \nhow do you achieve robustness against regions of interest which show no identifier of interest ? \n\naddditionally:\n- in the wrong GT experiments in Table 3: are all identifiers wrong or only a part of it ? \n\nlastly:\n- what is the impact or importance of or potential use cases for personalization ? To be fair, the reviewer would **not** **lower** the rating if your answer would not convince the reviewer. There is no risk of losing rating in this question. It is okay to do research on topics off the beaten path. Nevertheless the question still goes around in the mind of the reviewer.", "limitations": "- minor:\nthe chatgpt preference evaluation might be biases, because the provided captions might be more sweet and overly polite, and chatgpt has a tendency to prefer a conspicuously sweet style when it generates language. That should be added in the limitation section or in the appendix.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- Meaningful evaluation regarding precision, recall and F1 for outputting the correct identifiers. \n- They show that the GRPO-based algorithm performs better in the 4-identifier setting and also when SFT does not have a much larger dataset, which brings evidence for the usefulness of the proposed approach.\n- experiments with providing wrong identifiers to test robustness\n- ablation study validates the usefulness of all three reward components\n- evaluation templates are in the supplement\n\nWeaknesses: \n\n- unclear time indices t in equation 1. This is surprising because the rewards might not be properly evaluated until an answer has been fully generated. Note also that the GRPO paper does not use time indices for that reason. \n\n- A few unclear implementation choices, namely:\n- what is the base model used to sample outputs as reference policy ? the untuned Qwen baseline ? The model used in training from the previous epoch, the model lagging N=? epochs or ?\n- it is not fully clear what the difference is between skip-retrieval and retrieval mode: \nskip-retrieval presents examples of images with identifiers, query images, and example answers for the query images ? does skip-retrieval show in all of its examples only the images with the correct identifiers (correct = present in the query image for which the answer is evaluated)? \n\n\n- appendix:  \"YOLO-World [1] is employed to detect regions of interest.\" \nhow do you achieve robustness against regions of interest in the query image which show no identifier/concept of interest ? \n\n- the chatgpt preference evaluation is considered to be of low impact, because the provided captions might be more sweet and overly polite, and chatgpt has a tendency to prefer a conspicuously sweet style when it generates language. That should be added in the limitation section.\n\n- minor: readability: a link to details of the retrieval in the retrieval mode in appendix B.5 should be made in the main paper. \n- minor: readability: a link to details of the evaluation templates in appendix D should be made in the main paper. \n\n- minor: it would be interesting to see in 2/4 concepts how the systems behave when only a part of the identifiers is wrong.\n\nA slightly lower score on originality because it is a straightforward application of GRPO and question templates.\n\nOverall the reviewer is not sure to what extent personalized image captioning might have an impact, but the experiments are well-done (justifying a higher overall quality rating). \nThe proposed method works well with a small sample size and in settings with more identifiers.", "was_revised": true, "final_justification": "The authors addressed most of my concerns. Some remain, regarding chatGPT as evaluator, the limits in novelty and the usefulness of this setup. The reviewer keeps this score, which is, nevertheless, a positive one.", "rebuttal": "We sincerely thank the reviewer for the thoughtful and constructive comments on our paper. We have made every effort to respond to your feedback and faithfully reflect your suggestions. We will revise the manuscript to incorporate the following contents into the final camera-ready version.\n\n---\n\n# Strengths\n\nWe greatly appreciate that the reviewer recognized the key strengths of our work, particularly its meaningful evaluation results, the RePIC’s ability to perform well even in OOD scenarios such as the 4-identifiers (a.k.a., multi-concepts), and the validation of our three reward components through robustness testing and ablation studies. \n\n---\n# Weaknesses\n\n**W1) Time index of GRPO in Eq (1)**\n\nIn our formulation, $\\pi$ denotes an autoregressive LLM that predicts the next token based on the query $q$ and all previous tokens up to timestep (a.k.a., token position) $t-1$, i.e., it can model the output $o_{t}$. While many LLM-reward frameworks assign a reward only to the final token, complicating per-token value network learning, the pioneering work of DeepSeekMath [1] (Eq. (3)) and the DAPO paper [2] (Sec. 2.2) clarify that both PPO and GRPO can define and compute token-level advantages at each timestep $t$. \n\n**W2) Definition of Base model**\n\nThe model we used is Qwen2.5-VL-Instruct-7B, a vision-language model capable of processing images, text, and videos simultaneously, with enhanced instruction-following capabilities. Following the FLAN [1] paradigm, the pretrained MLLM on a large corpus was further fine-tuned using instruction datasets. We adopted this backbone because it is open-source and supports multi-image understanding, a crucial feature for MLLM personalization. We chose the 7B variant as a suitable backbone MLLM for our experiments, and the frozen copy remains fixed throughout training to perform as a reference policy while GRPO. We will add this explanation to the Appendix.\n\n**W3) Differences of skip-retrieval and retrieval settings**\n\nWe apologize to the reviewer for the insufficient explanation of the skip-retrieval and retrieval settings. Although Figure 2 and lines 184–188 in the main paper, and Appendix B.5 of our paper briefly describe these settings, we clarify the differences more explicitly below:\n\nFirst, our database is composed of key-value pairs for all concepts, where each concept is associated with an image and corresponding textual information. Then:\n\n**1. Skip-retrieval**: In this setting, the reference samples are directly retrieved from the database by performing key-value matching based on concept names in the evaluation dataset. \n\n**2. Retrieval**: Similar to skip-retrieval, the reference image is retrieved from the database. However, as detailed in Appendix B.5, a detector first generates region proposals from the query image, and then reference images are retrieved with their texts by selecting the top-K samples from the database with high CLIP image cosine similarities.\n\nOnce the reference images and texts are retrieved, they are prepended to the query prompt in an in-context learning (ICL) fashion. Importantly, no example answers for the query image are provided; MLLMs generate responses solely based on the retrieved references (images and texts) and the query image in a personalized manner. We encourage the reviewer to see Figure 2(b) in our paper for better understanding. We will incorporate this clarification into the camera-ready version of the paper.\n\n**W4) Robustness of selecting regions of interest**\n\nWe sincerely appreciate the reviewer’s insightful question. As the reviewer pointed out, when performing region proposals using the YOLO-World detector, ambiguous ROIs that do not contain any identifiable concept or entity result in what we refer to as retrieval noise. Such cases impose an upper bound on the performance, and MLLM becomes inherently fragile to generate personalized captioning when the retrieved demonstrations contain content irrelevant to the query. \n\nWe identify two primary causes for this issue: (1) the limited visual perception capabilities of the open-vocabulary object detection model, and (2) the absence of any relevant concept in the query image itself. We interpret the issue raised by the reviewer as corresponding to case (2). Although both scenarios present more complex challenges, our evaluation datasets do not account for the second case, as they consist primarily of object-centric images in which the target concept is clearly visible. We believe that robustness could be further improved by designing a verifiable reward that penalizes the model for generating responses based on incorrect or irrelevant information. We will incorporate this clarification into the revised manuscript.\n\n**W5) Inclusion of ChatGPT's limitations**\n\nWe strongly agree with the review's opinion that ChatGPT often favors responses that are overly polite or “sweet.” To mitigate such concern, in our evaluation, we employed GPT-4o, the most capable model available, and included instructions in the system prompt to prioritize accurate and meaningful captions. However, we acknowledge that such GPT-based evaluation inherently carries intrinsic biases, including a tendency to reward politeness in output responses. We will add this limitation to our paper to clarify the potential influence of GPT-based preference evaluations. Additionally, in response to Reviewer 9oCi’s [W2], we evaluated the quality of the generated captions not only using the ChatGPT preference score but also through both reference-based and reference-free metrics. We kindly refer the reviewer to those results.\n\n**W6) Enhancing the readability of the paper**\n\nWe sincerely thank the reviewer for their careful reading of our paper. As you pointed out, this is an important aspect we had not fully considered. To enhance the reader’s understanding, we will provide additional links in our paper referencing Appendix B.5 and D in the camera-ready version, possibly in the Experiments section, would be proper.\n\n**W7) More interesting results of 2/4 concept settings**\n\nWe agree with the reviewer’s observation. In the revised manuscript, we will include visualizations of such cases, including instances of retrieval noise where some given identifiers are incorrect. We attribute these cases to the limited visual understanding of CLIP. To better illustrate these realistic scenarios, we will provide additional qualitative examples in the paper.\n\n**W8) Straightforward application of GRPO and question templates**\n\nWhile multiple factors contribute to stabilizing RL, careful reward design and data quality are among the most critical components [2]. As noted in our response to Reviewer 3m5o [L2], designing task-specific verifiable rewards is challenging, especially for downstream applications such as math. In our work, our key contributions include proposing novel verifiable rewards to guide MLLM personalization effectively. Additionally, we address the difficulty of delicately constructing datasets for MLLM personalization and identifying a “sweet spot” for stable RL training. Our results demonstrate that these efforts lead to robust performance in both single and multi-concept captioning. In future work, we aim to explore algorithmic improvements for more challenging tasks. For better understanding, please refer to the authors' responses to the reviewer v1W9 [W3], reviewer 3m5o [W1]. \n\n---\n# Questions\n\nWe address the following two major questions, as most questions have been covered in our responses to the weaknesses section.\n\n**Q1) Experimental details of Table 3** \n\nIn Table 3, all of the reference images contain incorrect visual concepts, while the corresponding textual information is accurate. Specifically, in this visual attack setting, we intentionally feed reference images that depict concepts different from those in the query image. This experiment is designed to evaluate whether the MLLM relies solely on textual cues and disregards the visual understanding when generating responses.\n\n**Q2) Impact or importance or potential use cases of personalization** \n\nWe sincerely appreciate the reviewer’s honest and insightful question. To the best of our knowledge, MLLM personalization is still an emerging research area, and various concurrent studies are beginning to demonstrate its practical potential for the tasks of personalized image captioning, VQA, and personalized conversations, and so on. In particular, the ability to handle multi-concept inputs is becoming more essential in MLLM personalization. For example, consider a photo featuring you and your friends. In such cases, a personal multimodal agent should be able to accurately recognize and distinguish you from others to generate an appropriate personalized caption for the image.\n\nIn our paper, we aimed to demonstrate the advantage of RePIC in more realistic scenarios by showing its generalizable performance even in complex group photos involving 2, 4, or even 7 individuals (see Figure A.1), including both real and AI-generated images. Furthermore, as noted in the conclusion, extending this research to other modalities such as video or audio could further enhance its real-world applicability. Lastly, exploring methods for extracting information from historical data and updating the database, as well as designing both long-term and short-term memory modules for MLLM personalization, would be valuable directions for future research. We hope this additional context helps clarify our perspective and supports the reviewer’s understanding.\n\n---\n\n# References \n\n[1] Wei, Jason, et al. \"Finetuned language models are zero-shot learners.\", International Conference on Learning Representations, 2022.\n\n[2] Shao, Zhihong, et al. \"Deepseekmath: Pushing the limits of mathematical reasoning in open language models.\" arXiv preprint \narXiv:2402.03300 (2024).\n\n[3] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025)."}]

Author rebuttal:

Meta-review: This paper introduces RePIC, an RL-based post-training framework designed to enhance the personalized image captioning abilities of MLLMs. After rebuttal, it received scores of 4444. All the reviewers are generally positive about the paper, commenting that the proposed method is the first RL-based method for the personalized image captioning tasks, with well-defined verifiable rewards. Experimental results are solid, and the paper is well-organized and easy to follow. The authors have also done a nice job during rebuttal to address the reviewers' concerns. Therefore, the AC would like to recommend acceptance of the paper.
Decision: Accept (poster)