Personalized Visual Instruction Tuning
Authors: Renjie Pi
Jianshu Zhang
Tianyang Han
Jipeng Zhang
Rui Pan
Tong Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: large language model
multimodal large language model
Primary area: datasets and benchmarks

Abstract: Recent advancements in multimodal large language models (MLLMs) have demonstrated significant progress; however, these models exhibit a notable limitation, which we refer to as "face blindness." Specifically, they can engage in general conversations but fail to conduct personalized dialogues targeting at specific individuals. This deficiency hinders the application of MLLMs in personalized settings, such as tailored visual assistants on mobile devices, or domestic robots that need to recognize members of the family. In this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel data curation and training framework designed to enable MLLMs to identify target individuals within an image and engage in personalized and coherent dialogues. Our approach involves the development of a sophisticated pipeline that autonomously generates training data containing personalized conversations. This pipeline leverages the capabilities of various visual experts, image generation models, and (multi-modal) large language models. To evaluate the personalized potential of MLLMs, we present a benchmark called P-Bench, which encompasses various question types with different levels of difficulty. The experiments demonstrate a substantial personalized performance enhancement after fine-tuning with our curated dataset.

Published as a conference paper at ICLR 2025
PERSONALIZED VISUAL INSTRUCTION TUNING
Renjie pil: Jianshu Zhang': Tianyang Han', Jipeng Zhang', Rui Pan", Tong Zhang?
'The Hong Kong University of Science and Technology UUniversity of Illimois Arbana-Champae {rpi, justt hk, jianshu. zhang@whu edu. cn, 2310484 ruip40i 11 ioinois edu, tongzhange:.ongzaagg-g
connect. eononettgeconneet...
poly...k,
ABSTRACT
Recent advancements in multimodal large language models (MLLMs) have demon- strated significant progress; however, these models exhibit notable limitation, which we refer to as "face blindness". Specifically, they can engage in general conversations but fail to conduct personalized dialogues targeting specific indi- viduals. This deficiency hinders the application of MILMSs in personalized settings, such as tailored visual assistants on mobile devices, or domestic robots that need to recognize members of the family. In this paper, we introduce Personalized Visual Instruction Tuning (PVIT), a novel data curation and training framework designed to enable MLLMs to identify target individuals within an image and engage personalized and coherent dialogues. Our approach involves the de- velopment of a sophisticated pipeline that autonomously generates training data containing personalized conversations. This pipeline leverages the capabilities various visual experts, image generation models, and (multi-modal) large language models. To evaluate the personalized potential of MLLMs, we present benchmark called P-Bench, which encompasses various question types with different levels of difficulty. The experiments demonstrate a substantial personalized performance enhancement after fine-tuning with our curated dataset.
INTRODUCTION
The advent of Large Language Models (LLMs) has sigmificantly advanced AI, transforming natural language processing and understanding (Geng Liu. 2023 OpenAl 2023 Touvron et al. 2023 Scao
et al. 2022) Chowdhery et al.) 2022: Taori et al. 2023) Chiang et al. 2022). These models, trained
on extensive text corpora, possess substantial world knowledge, excelling in various tasks. Progress in LLMs has led to rapid enhancements in Multimodal Large Language Models (MLLMs) (Liu et al.
2023a Zhu et al., 2023: Dai et al. 2023 Li et al.) 2023b OpenAl 2023 Bai et al., 2023). MLLMS leverage pretrained visual encoders (e.g.. vision transformers) to process images, incorporating them as token embeddings alongside text token embeddings. These models expand LLM capabilities engage in conversations based image inputs, offering diverse applications like autonomous
driving (Ding et al. 2023) and medical assistants (Li et al., 2023a).
Despite the success of MLLMs, their effectiveness is limited in general purpose conversations, and drastically fail at personalized conversations targeting at specific individuals. For example, given a photograph of a girl named Lisa, and an image with Lisa inside of it, the state-of-the-an MLLMs are not able to recognize her and provide corresponding information. This deficiency prohibits the use of the MLLMs personalized use cases, such as your personal AI assistent deployed on the mobile phone, a domestic robot that needs to recognize and serve your family members, smart home system that requires individualized interactions to cater to the specific needs of different residents.
Equal Contribution. Code and data are available at the following links:
https: //gi t hub. com/ sterzhang /PIIT
https: / //huggi i ng aaccc 2o /at//aaaaseeeeee Stattts/tteerehngeeeee 3M
The code and data are released under MIT and apache2.0 licenses, respectively.

---

Published as a conference paper at ICLR 2025
To empower MLLMs with personalization capability (denoted as P-MLLM), previous endeavors propose to augment the MLLM with external heads and vocabularies, which are trained to identify specific individuals within a scene using few personalized training data (Alaluf et al., 2024 Nguyen et al. 2024). Although these approaches demonstrate good performances, they suffer from the following weaknesses: 1) they require additional training for each newly introduced individual, which is inflexible and unpractical for real-life scenarios, since the person of interest may frequently change; 2) it can not be guaranteed that we can always collect the training data for the person of interest. Therefore, it is undoubtedly more promising if the personalization capability of the MLLM is able to generalize, rather than being limited on few pre-defined individuals.
Owing to the auto-regressive training paradigm adopted for current state-of-the-ar LLMs and MLLMs,, they possess the ability to generate responses depending on a given prefix. This ability is also referred to as in-context learning capability (Wei et al. 2023), which enables the model to produce different outputs during inference by adjusting the prefix without further training. Therefore, one intuitive and practical approach is provide the information of individuals to the MLLM in its prefix. In this way, the MLLM is expected to provide answers for different input individuals during inference. However, MLLMs struggle to effectively comprehend such personalized inputs. This may be due to the fact that these MILMS fine-tuned with limited multimodal data and lack of personalized data, which together hinder their ability to develop in-context understanding for multimodal inputs.
experimental results indicate that this capability is challenging, and current
To address this challenge, we propose Personalized Visual Instruction Tuning (PVTT), novel training paradigm that enables MLLMs to perceive personalized inputs as in-context prefixes. Specif- <personal image, personal introduction> pair, which is
ically, each individual is represented provided to the MLLM as a multi-modal prefix. We further introduce personalized wrapper tokens to group the visual and textual information of each individual, thereby eliminating ambiguity when multiple individuals are involved. During training, the MLLM is optimized to answer questions related to target individuals within the prefixes. Once trained, the MI.LM is able to fully utilize its in-context learning capability, and generalizes to arbitrary individuals without requiring additional fine-tuning or modifications to the model architecture.
In our proposed PVIT paradigm, the most critical barrier is the absence of large-scale, high-quality training data. To address this challenge, we design an automatic framework to synthesize personalized training data, operating in three phases. First, visual expert models extract individual visual concepts from scene images (Visual Concept Curation). Next, we utilize the MILM to convert these visual concepts into both individual-level and scene-level textual descriptions, which are fused create a coherent representation (Dual-Level Textual Information Extraction and Fusion). Finally, LLMSs generate diverse personalized QA pairs using reasoning and instruction-followe Dataset Generation). To evaluate the personalization capability of MLLMs, we further created a benchmark termed P-Bench, which assesses personalization capability from multiple perspectives. The results indicate that the ability of current SOTA MLLMs to perceive personalized concepts is limited, which can be significantly boosted after training with our proposed PVIT.
capabilities (PVIT
To summarize, we make the following contributions in this paper:
Inspired by in-context learning capabilities of LLMs, we propose Personalized Visual Instruction Tuning (PVIT), a new training paradigm that equips MLLMs conduct personalized conversation for any arbitrary individuals with no addition training at inference.
We meticulously design an automatic data annotation framework to curate high-quality personalized training data, and synthesize a large scale dataset enhance the MLLM's capability to conduct personalized conversations.
We curate P-Bench, novel benchmark for evaluating the personalization capability of MLLMs. We demonstrate that the MLLM trained with our curated dataset demonstrates significantly improved performances.
2
RELATED WORK
Multi-Modal Large Language Model. Recent advancements in large language models (LLMs) have significantly improved language comprehension and generation, achieving hear-human profi-
2

---

Published as a conference paper at ICLR 2025
Dual-Leve Info Visual Concept
Curation
Person Extraction
Face Detestor
Generater
Extraction & Fus inn
Personalized Multimodal Profix
Scane Image Drigina
PVIT Dataset Generation
anwenal people, prontrantyy
girl She hos blonnd Rair
girl
with piril, ribbenn in her
whitee shirt and red tie He
childne sdafis suppeating
public event or gathering.
group
wh dress light
dissped over heir right
ribbon Shere seeeiing
Eoquarr aeciiino She gontty.
sequest
vidde
and
Adv
Augeanted
This & <namer,
This is snomeo.
Name
Deseription Frendorn GA
Personalized Q&&A
with smile
appears
Adv-Fare
TNSs is cromes.
Individual DB
hatppl Choice GA
about
Hollatic Information
Information
Dencribe
What color dlothes Is loc wear in??
W sat la wearing?
esa aweater
Payk dress
PVIT Dataset
image
Ratt has bionde hair styled She smilling
a wearing white
while looking at man in dark suR.
and pirk cress.
Answer Category
Sonyy Lde not know who Sote
Sony Idon's know who
E. Lon'' know
Answerable
Origina Crap r Auguanded
Scrry. Cats does not appeor this pictura.
Co not SAe in this image
notinn
Unaroweruble
Adw-Norw
AdueFFuce
The Personalized Visual Instruction Tuning (PVIT) framework consists of three phases. In the visual concept curation phase, we extract individuals and their faces from images, then augment them with different poses and angles. During the dual-level textual information extraction and fusion phase, MLLMs first generate both holistic information and personal information, then integrate them to get more detailed and contextually accurate information. the PVIT dataset generation phase, LLMs create QA pair templates based on the extracted textual information, which are filled with selected names to construct training data.
Figure 1:
ciency across various tasks (Brown et al. 2020 Scao et al. 2022 Chowdhery et al. 2022 Smith
al. 2022, Hoffmann et al.) 2022] Ouyang et al.
2022 Touvron et al.
2023 Bai et al. 2022). This
success has spurred interest in Vision- language interaction, leading to multi-modal large language
models (MILMM)) (Liu et al. 2023a Li et al. 2023b Dai et al. 2023 Zhu et al. 2023 Dai et al.
2023 OpenAI 2023 Bai et al. 2023 Su et al. 2023 Gao et al. 2023 Pi et al. 2023), which excel in dialogue based on visual inputs. However, they struggle with personalized conversations, such as dialogues about specific individuals like girl named Kitty, limiting their use in personalized AI assistants, tailored recommendations. or customized therapy. Addressing this limitation is crucial for enabling personalized Al interactions.
Model Personalization In model personalization, the main objective is to tailor a model to learn new user-specific concepts. Considerable research has been conducted in the area of text-to-image (12i) generation personalization (Gal et al. 2023 Ruiz et al. 2022 Alaluf et al., 2023 Arar et al.) 2023 Voynov et al., 2023, Ye et al.) 2023 Wang et al. 2024). The predominant method for t2i personalization involves fine-tuned the word embeddings using few examples to capture the nuances of the target concept. Alternatively, some research has concentrated on personalizing image captioning models (Chunscong Park et al. 2017: Zeng et al. 2019 Wang al. 2023 Park et al. 2018 Zeng jtt al., 2019] [Shuster et al. 2019). These personalized captioning methods aim to produce captions in particular writing style. In contrast, our goal is to enable the model to integrate new user-specific concept into a personalized textual description response that aligns with given image.
Personalized MILMs
Compared to text-to-image generation and traditional image captioning, the personalization of Multimodal Large Language Models (MLLMs) remains an under-explored area. Recent approaches propose introducing new tunable parameters for each new individual and conduct tuning using a few training samples. Specifically, My VLM (Alaluf et al. (224) adopts a Q-former- style architecture, incorporating learnable heads to extract specific concepts and append them to the visual features. On the other hand, Yo" LLaVA (Nguyen et al. 2024) utilizes LLa architec- ture, proposing to directly incorporate new concepts as additional tokens in the LLM's 's vocabulary. Despite these approaches demonstrating promising performance in personalized conversation, they require additional training and parameters for each newly introduced individual. This requirement makes the paradigm less practical in real life since new individuals are likely to be incorporated from
3

---

Published as a conference paper at ICLR 2025
time time. In addition, it is often not possible collect training samples associated with each individual. These limitations necessitate the model to generalize on new concepts on the fly.
PROBLEM FORMULATION
Unlike previous methods for personalized multimodal large language models (P-MLLMs), which require adding trainable parameters and fine-tuning for each new individual, our approach treats personalization as an in-context learning task. This climinates the need for fine-tuning on each new individual. More specifically, first provide the list of multi-modal prefixes to the MLLM, which consists of <personal image, personal introduction> pairs. Then, we present the scene image and a personalized query conduct conversation targeting at specific individuals. Conditioned on the multi-modal prefixes and personalized query, the MLLM is expected to produce responses specific to the individual of interest accordingly.
We observe that this capability is nontrivial, which is not possessed by the majority of state-of-the-art MLLMs. As shown in Figure 2 the MLLM tends to produce general responses, even when the user instruction is targeting at a specific individual. This issue persists even in MLLMs trained on interleaved image and text data. We presume that this limitation due to the lack of training, rather than deficiency in the visual encoder.
Therefore, we propose empower the MILM with the personalization capability via Per- sonalized Visual Instruction Tuming (PVTT). Specifically, we first collect training dataset where r² 9² and 1 stand for the 1th target response, user IaJkaz k=1 i=1
D = ( o
K
Pa
query, and the scene image provided to the MLLM, respectively. On the other hand, { & K' represent the K" multimodal prefixes containing images of input individuals and their personalized introduction, such names. The optimization problem be formulated as follows:
km1
N
L
L(D) = - N
log
I.
I. (T Px ph 1
K
(1)
During training, we minimize the auto-regressive loss to generate personalized responses based on the query, scene image, and multimodal prefixes. This approach fully harnesses the in in-context learning capabilities of MLLMs, allowing them to adapt to new individuals without additional training.
Personalized Wrapper Tokens Naively interleaving personalized images and introduction in the multimodal prefixes may introduce ambiguity, as personalized introduction (e.g., a name) could be mistakenly associated with either the preceding or following person's image. This ambiguity complicates training and confuses the MLLM during inference. To resolve this issue, we introduce two special tokens into the MLLM's vocabulary: <| person_start }} and <) person_end })- These tokens serve as wrappers to clearly enclose each individual's information, structured as follows: person start |) {photo} intro) person_em end I). This design ensures that the information belonging to each individual is properly grouped, improving the model's 's learning process.
The main challenge of the in-context learning paradigm lies in the lack of personalized training data, as existing multimodal instruction-tuning datasets focus on general conversations. To address this, we propose an automated framework for annotating multimodal conversation data with personalized inputs. leveraging LLMs, MLLMs, image generation models, and vision experts. Details of this framework are discussed in the next section.
DATA CONSTRUCTION
To equip the MLLM with personalized conversation capabilities, we developed a data generation framework that synthesizes various types of personalized training data. The framework begins by utilizing a set of images, which serve as scene images for the training data. It first extracts visual concepts of the individuals from these images using vision expert models. Then, it converts the visual information contained in the images into textual descriptions using MLLMs. Finally, it utilized the I.LM's reasoning and instruction following capability to create high-quality and diverse QA pairs. As
4

---

Published as a conference paper at ICLR 2025
illustrated in Figure the framework operates in three main phases: Visual Concept Curation, 2) Dual-Level Textual Information Extraction and Fusion, and 3) PVIT Dataset Generation. Below, we describe the procedure for each phase in detail.
4.1 VISUAL CONCEPT CURATION
In this phase, based on the scene images, we devise strategies for collecting images of individuals. First, we apply person identification accurately locate the individuals within the images. Next, we perform person augmentation to generate images of the same individuals in various contexts and different perspectives. These images will later serve as the foundation for creating both visual and textual information of the input individuals, as well personalized QA pairs.
Person Identification For each image, we apply an open-vocabulary object detector, such as GroundingDino (Liu al. 2023b), localize individuals by providing the image and the text prompt "a person". Next, for each detected person, we use face detector (Geitgey) (2166) to identify and localize the corresponding face, as the face is the most distinctive feature for identification. The images of the individuals along their faces are stored for later stages. Individuals without a detected face are excluded during this process.
Person Augmentation After the previous step, we derive list of <person, face pairs for each image. Each person in the scene image can be referenced by their corresponding face. However, in practice, the face used as reference to individual often differs from the scene image. To introduce more variation in human faces and enhance the capability of MLLMs to recognize individuals, we adopt the identity preserving image generator PhotoMaker (Li et al. 2023c) augment the person, which produces images of the same individual from different perspectives and contexts based on an input face. These augmented images can then be leveraged as reference to the individuals original scene images.
4.2 DUAL-LEVEL TEXTUAL INFORMATION EXTRACTION AND FUSION
To construct personalized conversations for specific individuals in the subsequent phase, it is essential not only derive the characteristics each person the scene image, but also capture how they are interacting with the surrounding context, which will serve as the basis for creating conversations that accurately reflect each individual's 's role and behavior in the scene image. To achieve this, we employ dual-level information extraction and fusion approach.
Personal Information Extraction Since current MLLMs are unable to directly provide specific features of designated person in the scene image containing each individual by providing their cropped images, which are extracted from the previous phase, to the MILM to create personal information. Since the cropped images contain only one person, the descriptions generated by the MLLM will only focus on the characteristies of this individual, which capture more fine- greiggr personalized details.
than person, focus
Holistic Information Extraction As shown in Figure although the personal features of the girl, such as "single braid", can be captured through the previous step, without the holistic context, would be impossible to extract the information "the girl is looking at the man". Therefore, We utilize the existing descriptive capabilities of MILMs to provide a holistic information of the scene image. Specifically, we emphasize describing the main characters in the image, such as the holistic information of describing "a man" and "a girl" in Figure I anchors" that facilitate the subsequent fusion of personal information and holistic information.
This approach aims to offer more "feature
Dual-Level Information Fusion After obtaining holistic information that provides contextual knowledge and personalized information that captures individual characteristics, we attempt to link these two pieces of information. This is done by matching the personal information with the descriptions characters in the holistic information, which results a fused description that describes how specific individual interacts with the context (demonstrated in Appendix (3). This dual-level fused information serves as the foundation for generating personalized conversations that more detailed and contextually accurate.
5

---

Published as a conference paper at ICLR 2025
4.3 PVIT DATASET GENERATION
With the visual concepts and the textual information associated with each individual extracted in the previous two phases, we can now construct the Personalized Visual Instruction Tuning (PVIT) dataset. The PVIT dataset primarily consists of three components: Personalized Multimodal Prefix, Scene Image, and Personalized QA.
4.3.1 PERSONALIZED MULTIMODAL PREFIX
In our in-context learning formulation of the personalization task, each input individual is represented by a multimodal prefix, which is the combination of personal image and a personalized introduction. The personal image can be drived from the previous stage, which can either be the headshot cropped from the original image, or generated photo from Photomaker (Li et al. 2023c). We design strategies to diversify the coverage of personalized introduction.
Name Swapping We introduce <nnme>> as placeholder to be replaced with actual names during the construction of the training dataset. Specifically, we collect a list of names (around 600 names) using ChatGPT. Then, we randomly select a name and swap it with the placeholder. It is important to note that this process can be repeated multiple times, augmenting the training data with diverse names. This approach not only enhances the model's generalization ability and robustness in adapting to new individuals, but also helps prevent overfitting by avoiding the association of a specific person with fixed name during training. The effectiveness of this technique verified in Section 6.2.3
Personal Pronoun To better align with the way users refer others everyday conversations, we not only introduce individuals by their names but also handle situations involving personal pronouns (e.g.. you, him). For example, if question contains "my dad." the response should adapt by using "your dad." To handle such cases effectively, we introduce training data that includes examples of personal pronouns, ensuring the model can appropriately adjust its responses based on context.
Adversarial Introduction To ensure the model truly learns recognize individuals accurately, solely considering the cases where the questions are answerable is insufficient. The model must also learn to handle challenging or misleading scenarios. We found that even SOTA MILMS often generate responses completely ignoring the input individuals provided in the prefix. For instance, even when the person interest (e.g., a girl named Lisa) is not present in the image, model may still respond questions about her and mistakenly identify other individuals her. To address this issue, we introduce adversarial inputs designed to challenge the model's ' ability to correctly handle unanswerable questions. Specifically, generate the following types of adversarial inputs:
Adversarial name mapping: When choosing the person of interest to construct the query (e.g., Lisa). we make sure that the person is not provided at all in the personalized multimodal prefixes. For this type of query, the model should respond with "Sorry, do not know who Lisa is", as the person was never introduced.
Adversarial image mapping: When constructing the multimodal prefixes, we randomly select images of person excluding those that are in the scene image. The person of interest is one of the individual contained in the prefixes. In this case, the scene image does not contain this person of interest. The model should then respond with "Sorry, I cannot see Lisa in the image", demonstrating its ability to correctly identify missing individuals.
4.3.2 SCENE IMAGE
In designing scene images containing specific individuals, our goal to enable the model to accurately recognize the target individuals provided in the prefix. We design two types of scene images for this purpose. The first type is the original, complete image, with the full context containing additional elements besides the person. This allows subsequent Q&A to be more comprehensive, focusing on both the specific features of individuals and their broader interactions with the environment. Additionally, enhance the model's 's ability to recognize specific person in an image containing multiple individuals, we also concatenate cropped images of individuals to create composite images, which elevates the MILM's capability in more challenging scenarios.

---

Published as a conference paper at ICLR 2025
4.3.3 PERSONALIZED QA
The dual-level information extracted from the previous stage transform visual information into texts, enabling us to utilize the LLM's advanced reasoning capabilities to create personalized conversations. Provided with such information, we meticulously design prompts and in-context examples to generate data for the following tasks using the LLM:
Personalized Description: We create conversations where the user queries for the descrip- tion of specific individuals, rather than the entire image.
Personalized Free-form QA: We create free-form QA pairs that queries for information or characteristics related to specific individuals, such as appearance and behavior, which aims at enabling the MLLM conduct personalized multi-round conversation.
Personalized Multi-choice QA We create personalized QA pairs in multi-choice format, where the choices include the ground truth answer, as well some confounding alternatives.
Compared with general purpose image captioning and VQA, the personalized conversation presents novel challenges for current MLLMs, since they not only need to recognize the individuals of interest in the scene image, but also properly incorporate them into the generated response.
In total, we create 3M training instances of personalized conversation, which we term PVTT-3M, The curated dataset encompasses diverse types of data and difficulty levels. We associate the detailed information and statistics of PVIT-3M in Section Of the Appendix..
5 EVALUATION USING P-BENCH
Although numerous benchmarks have been proposed to assess the effectiveness of MLLMs, none have been specifically designed to evaluate their personalization capabilities. To address this gap, we present a high-quality benchmark manually checked by human, P-Bench, aimed at thoroughly evaluating the personalization potential of MLLMs. We design both multiple choice (MC) questions and personalized image description queries for evaluation. In this section, we provide detailed overview of the problem types and evaluation metrics of P-Bench. Detailed statistics and curation process of the benchmark are presented in Section E
5.1 MULTIPLE-CIOOCC (MC) QUESTIONS
We design positive (answerable) and adversarial (unanswerable) questions to examine the MLLM's ability to correctly associate the target individual with the corresponding person in the scene image.
A Answerable Questions include the following types: 1) Crop: the input individual is represented by its original face cropped from the image; 2) Aug-In: using Photomaker (Li et al. 2023c), we generate an augmented photo of individuals based the original cropped face; 3) Aug-Sc-2 and Aug-Sc-3: we concat two or three different cropped images of individuals into single image to replace the original scene image, increasing the difficulty of accurately recognizing the individual.
Unanswerable Questions, on the other hand, include: 1) Adv-name: the question pertains to person who is not included in the list of input individuals, meaning the MLLM lacks knowledge of this person; 2) Adv-image: the individual mentioned in the question does not appear in the scene image, meaning the MLLM cannot visually identify this person.
Evaluation Metries For MC questions, we adopt accuracy for the evaluation metric. To gain a deeper understanding of the MLLM's capabilities and limitations, we separately evaluate each of the three types in MC questions. In addition, to further study the MLLM's 's ability to differentiate different individuals, we also separately evaluate the accuracy of images containing different numbers of people.
5.2 DESCRIPTIVE QUESTIONS
For this type of questions, we query for the descriptions of specific individuals, rather than general descriptions. We also design both positive and adversarial description questions. Specifically, the
7

---

Published as a conference paper at ICLR 2025
<Person Joay i whet
giv of picuus?
<Person image of Chandier.
Phoelie. <image- Descn be image for mis.
picuue pormays
The image shows pesss with auprissd exprestion waaring black ahiet The buckground plen dtrky whih pur foous persons
desription of Jony and Chandlar
image?
depicts
standing together
fectures, contiulsity the eyes and moun, which are wide open expresstvd gestune.
stint the side point undomeath looking roprosdion Chandior on the other hand is sitting 2 cance with expression on on ti free Both them sitiig aeal behrre them.
the image Ioey is eiting in chal: weating red
an inder: wetting Both commen have lang bndd hat The collur and has hands diapped In fron The womon the right weading an open wide suggesting the might be shouting or eepenorng supprie loc Has eyno
wevting
There
dressed in cosumm that
nusebblo lege, fuu's whth turley, The surbey clocured by the costume. The person gotuiin with the hands poesely conversction appears to be in mésicomeenatioon Jany svening and pointing towards the person in
room with poster
opems effect her expression the bacig cund. there room cursans portallyy crown, allowing some natural lights intn the mom. like beterd it there a plant The cveeaal amriarcee of suggets casuel donnsstic setting.
hair weering dork broon polo ot and bive yoars. le io diuped has ecpresiion nectral and looking dignity his left. Chandier is man with short's coved button up shint light colorid pants. saaned wooden sanoe, with nands looking direclly at camana
with
scated wooden carooc
right, thene is paace
colored shirt
cyts are wilee face mouth
to the right of the turkey costume. there another person who a pertally vuible pansen is ligttcolored chirt
Mis
Istaring convesation.
image depiets two woreen roding indooss shede Phoebo wonan with long, stright blonde hols is laige. iound pandont She hay surtrise exp repiiiiion shouting The woman the lett long, brasse ful and westiig a blck cool With nil disvissed cogression fer loco. with her moush nown eveuting crying from women background The mor appeers to be living toam or a
mouth
fumiture wablee The everal atmosphere suggedt a dromntt intence moment
<Peranno This
neachhet
<Persono This Devid in headsnce. <image- What does Doiid ock in
cimagas Denoribe Loc this imag
is jacket and a block shitt, he scanding in frone gray
the
the pettre David agpsers to attendig a scecal event or occasion the background
indoor ponsidy might have given the lihting and ambiance.
is young mann He is wwering prominently
displaydd en the dhod, The sut abo factuno has black logie, brim and yellow Fermn loge the froms. Jeso
David worting formal black junado with whits has bou boutomnign with purple Rower He aso nant of tim.
of
Miss espressio
<Person-This Chen. this Image?
thang
image, there fnstures The hai with harratyle that outwares vanous onscions, giving it fentured and
<Persono Thi is Kunikun.
<image= creatioe.
neutral expresion appearance face and appears indeors, suppertec by the background which seams even, hig highlghting the person's facal featunes without cearing harsh shacows.
rimage Descrbe image.
The MVP is awarded the best player in sports game league corned holding which shows booketblll it's not poosible to cktammine who the
image
Xu is man who wearing brace jacket and red and white ctrped the standing in bandmark in Paris France. Xu smiling and looking direcly camera..
My knowledge doeort decalss obout Zhang.
Kunkun's suppenders are real MPP. holding holding ball.
Sonry Xu does not feetuee in this photo.
Figure 2: Qualitative examples of P-LLaVA results: Each example includes the user's query, input individual photos, and the scene image. The current MLLMS fail to recognize the person of interest and conduct personalized conversations, whereas our model, after training with PVIT, enables coherent and accurate personalized dialogues. Examples illustrate both answerable and unanswerable scenarios. For answerable cases, inputs involve single or multiple individuals, and our model incorporates names from the prefix for personalized responses. In unanswerable cases, current MLLMs provide incorrect answers, while the our model appropriately refuses and explains the reason.
8

---

Published as a conference paper at ICLR 2025
Table 1: MC questions P-Bench. P-LLaVA trained with PVIT significantly outperforms other MLLMs across various question types. Remarkably, P-LLaVA demonstrates strong performances on challenging answerable tasks and unanswerable queries, where the other MLLMs drastically fail.
Answerable Aug-Se-2
Unanswerable Adv-Name
MLLM
Crop
Aug-In
Aug-So-3
Avg
Adv-Img
Avg
Qwen-VL-P2 VL-B- et 2023
66.58
72.92
60.87
50.57
62.74
1.11
20.62
10.87
VILAA155-B
57.54
48.39
55.91
38.29
50.03
36.36
3.23
19.79
LLaVA OneVision 7B La el ale 2024
84.82
82.17
77.53
78.31
80.71
24.12
1.07
12.60
IntermVI-Chat-V V.52266B
62.99
56.84
57.14
51.02
57.00
53.33
9.64
31.49
Deepseck- VL- 7b-hhha (Lu crale 2024
89.72
24.79
71.43
66.56
63.13
4.12
0.00
2,06
mPLUG--WW2 Ye
72.29
50.63
63.62
0,00
5.00
P-LLaVA (Ours)
95.42
97.23
96.48
97.62
96.69
99.43
100,00
99.72
Table 2: Evaluation personalized description capability on P-Bench. The P-LLaVA tuned with our
PVIT is able to accurately recognize and describe the person of interest, even for challenging cases where multiple individuals are contained in the image.
Answerable Cnt=3
Unanswerable Ady-Name
MLLM
Cnt=1
Cni=2
Cnt>==4
Avg
Adv-Img
Avg
Owen-VL-78 VL-7B Bai et al.
2025
76,20
72.38
69.59
63.04
70.30
0.00
15.00
7,50
VII.AI.5
(an ara 40243
74.90
74.58
67.29
70.02
71.70
10.00
20.00
15.00
LLaVA-OneVisioor
74.54
71.08
76.13
63.54
71.32
10.00
12.50
Intern Chat V1.5 3-2 Chen el al.
2124
82.64
75.94
67.12
71.28
74.25
15.00
20.00
17.50
Deepseek-VL. Brnseek--LL-b-chaa
al 2024
82.15
77.86
76.06
76.44
78.13
0,00
2,50
mPLUG-OW12 et
80.38
79.11
77.77
72.68
77.49
0.00
5,00
2.50
P-LJaVA (Ours)
85.24
83.10
83.14
78.71
82.55
100.00
100.00
100.00
positive questions involves different number of people in the scene images. As the number of people grows the scene image, becomes more challenging for the MILM to correctly recognize the person of interest and produce accurate descriptions.
Evaluation Metrics For descriptions, we perform evaluations using different strategies for an- swerable and unanswerable questions. For answerable ones, adopt LongClip (Zhang al. 2024) to evaluate the similarity between the image of the target individual and the MLLM-generated descriptions. For unanswerable ones, we calculate the percentage that the MLLM rejects to respond.
EXPERIMENTS
In this section, we demonstrate the effectiveness of our proposed PVTT on the constructed P-Bench. We first showcase the results of the PVIT-tuned LLaVA (Liu et al.) 2023a), which demonstrates significantly higher performances than the SOTA MLLMs that support multi-image inputs. Then, we conduct ablation study on each of the components of our constructed data demonstrate their contributions towards the final performance. We provide the detailed training configurations in Section of the Appendix.
6.1 MAIN RESULTS P-BENCH
We compare the personalization capability of P-LLaVA trained with our PVIT with other SOTA MLLMs on our constructed P-Bench. We conduct evaluations using both the MC questions and personalized descriptions in Table [ Table 2 respectively. We observe the following phenomena for current SOTA MLLMs:
1) The performances of SOTA MLLMs significantly lower with more complex inputs (i.c., Aug-In In, Aug-Sc-2 and Aug-Sc-3 for MC questions, and scene images that contain more people for description questions), which indicates their limited capability and robustness in recognizing input individuals for the scene images.
2) All the MLLMs drastically fail for unanswerable questions. They still tend to answer the questions that are not answerable by mistakenly treating other people in the image as the person of interest. This because the MILMs have never been trained reject replying to such unanswerable questions.

---

Published as a conference paper at ICLR 2025
3) After fine-tuning LLaVA (Liu et al. 2023a) with our PVIT, we observe significant performance boosts for all question types in P-Bench. Specifically, we observe performance enhancement for both positive and unanswerable questions. Notably, the performance more complex scene images is boosted even more significantly. The results verifies the effectiveness of our propose tuning strategy in improving the model's personalization capability.
6.2 ABLATION STUDY
Table 3: The performances on MC questions with scene images containing different numbers of people.
Table 4: Face augmentation boosts the
MIs capahilities to recoginize indi- viduals, while adversarial samples are
MLLM
Cat=1
Cni=2
Cnt=3
Cnt> =4
Qwen-VL-7BB
74.67
58.71
37.32
critical for enabling MILM reject an- swering the unanswerable questions.
VILA1 (Lirera ai AIZAB
64.31
47.02
44.61
38.13
L.VA--Oneissiii-77 17 2024
88.72
83.23
80.37
76.31
IntemVL-Chat-WV 1.52-6BB-B--.
Chen ef 2024
56.21
52.28
43.57
38.31
Data
Crop
Augment
Unanswerahle
Deepscek-V VI.-Th-ohar
84.03
68.71
76.52
74.90
Full
96.43
95.32
99.78
mPLUG- OWW2 Ye ar 2014
78.90
63.23
77.43
60.51
Aug
94.93
88.43
98.73
P-LLaVA (Ours)
98.71
95.03
95.32
wo Adv
95.92
92.48
1.12
6.2.1 PERFORMANCE FOR VAIROUS NUMBER OF PERSON
demonstrate the MLLMs' personalization performances MC questions with scene images containing different numbers of people. The SOTA MLLMs showcase deteriorated performances scene images containing more people due to the challenge of accurately recognizing the specific individual of interest. On the other hand, our trained P-LLaVA remains highly accurate on such challenging cases, which verifies its capability to accurately recognize person of interest.
In Table
6.2.2 DATA ABLATION
We examine the effectiveness of each data component in our curated dataset in Table We observe the following: 1) Face augmentation using PhotoMaker increases the diversity of input individuals, which effectively boosts the MLLM's capabilities to recoginize individuals; 2) Adversarial samples are critical for enabling MLLM to reject answering queries that are unanswerable. Without adversarial samples, the accuracy for rejecting answering unanswerable questions rapidly drops to near zero.
6.2.3 IMPACT OF DATA SCALE AND NAME REPETITION
Accuaac
In the figure on the right, we illustrate the evaluation accuracy after training with various amounts of data. Specifically, the horizontal axis indicates the number of data units, and each unit contains 8000 samples. We observe clear performance boost when scaling up the training dataset. Furthermore, we find that with the same data templates, by repeatedly constructing data using different names, the performance of MILM able to be further enhanced, which verifies that the diversity of names used during training makes the personalization capability more robust and generalizable to new individuals.
Sepeat
925
Sapuel
90.0
23.5
41
05
35
88 110
7 CONCLUSION
In this work, we introduce Personalized Visual Instruction Tuning (PVIT), novel formulation a training framework that enables personalized conversations targeting specific individuals. To achieve this, We first develop an automatic pipeline that generates training data with individuals in diverse visual and conversational contexts. Then, we adopt the data to finetune the MLLM, which significantly improves MLLM's personalized dialogue capabilities, as demonstrated through the P- Bench benchmark. We hope that our work will promote the advancement in personalized applications, such visual assistants and domestic robots, enhancing user-centric multimodal interactions.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Yuval Alaluf, Elad Richardson, Gal Metzer, and Daniel Cohen-Or. A neural space-time representation for text-to-image personalization. 2023.
Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or Or. Myvim: Personalizing vlms for user-specific queries. 2024. URL httth::///xxx hatgg/bb//2000
14599
Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H. Bermano. Domain-agnostic tuning-encoder for fast personalization of text-to-image models, 2023.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond, 2023.
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with 2022.
reinforcement learning from human feedback. arXiv preprint arXiv:2204.05885
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901. 2020.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual- linguissii tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24185-24198, 2024.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chathot impressing gpt-4 with 90%* chatgpt quality, March 2023. |RLhttpss
//tmsys..rg//lleeee../. http://looo/2222-332 -02 30-vicuna/
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023:: 2022.
Cesc Chunseong Park, Byeongchang Kim, and Gunhee Kim. Attend to you: Personalized image captioning with context sequence memory networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 895-903, 2017.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose purpose languuel models with instruction tuning, 2023.
Xinpeng Ding, Jianhua Han, Hang Xu, Wei Zhang, and Xiaomeng Li. Hilm-d: Towards high- resolution understanding in multimodal large language models for autonomous driving. arXiv preprint arXiv:2309.0 10XX:22309.018800 186,, 2023.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-or. An image is worth one word: Personalizing text-to-image generation using textual inversion. The Eleventh International Conference on Learning Representations, 2023.
URL http:://.ppaaeeeeeeeeessooooootd
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, Hongsheng Li, and Yu Qiao. Llama-adapter v2: Parameter-efficient visual instruction model, 2023.
Adam Geitgey. Machine learning is fun! part 4: Modern face recognition with deep learning. Medium. Medium Corporation, 24:2016, 2016.
Xinyang Geng and Hao Liu. Openllama: An open reproduction of Ilama, May 2023. URL https 7/gi Liggthub... www com/open m- www.ercheeeeeeeeeeeeeeeeeeeeeeee
11

---

Published as a conference paper at ICLR 2025
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint
2022.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome:
Connecting language and vision using crowdsourced dense image annotations, 2016.
URL https: hittps://arxiv org/ abs/ 602. 07332
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. |
//axxiv.org/abs/// 24wp://bb/242..... 03326
Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training large language-and-vision assistant for biomedicine in one day, 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023b.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding, 2023c. URL httpt::///axxxi org/ abs//ab// 2312. 04461
Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22699,, 2024a.
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015, URL
Weifeng Lin, Xinyu Wei, Ruichuan An, Peng Gao, Bocheng Zou, Yulin Luo, Siyuan Huang. Shang- hang Zhang, and Hongsheng Li. Draw-and-understann:: Leveraging visual prompts to enable milms to comprehend what you want, 2024b. URL
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023a.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, 2023b.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, al. Deepseek-vi: towards real-world languae understanding. arXiv preprint ar 2xXh::2403005225 2024.
Meta. The llama herd of models, 2024.
Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo'llava Your personalized language and vision assistant, 2024.
09400
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35: 27730-27744, 2022.
Cesc Chunseong Park, Bycongchang Kim, and Gunhee Kim. Towards personalized image captioning via multimodal memory networks. IEEE transactions on pattern analysis and machine intelligence, 1104),99--1012, 2018.
12

---

Published as a conference paper at ICLR 2025
Renjie Pi, Jiahui Gao, Shizhe Diao, Rui Pan, Hanze Dong, Jipeng Zhang, Lewei Yao, Jianhua Han, Hang Xu, Lingpeng Kong, and Tong Zhang. Detgpt: Detect what you need via reasoning, 2023.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models, 2016. URL BRLLhttp::// /tttpp://arxxv..oo /aaaaiiiiiiv..... /ttp///aii...or/a////////// 04870
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. 2022.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellic Pavlick, Suzana Ilie. Daniel Hesslow, Roman Castagne, Alexandra Sasha Luccioni, François Yvon, Matthias Galle, et al. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:221 1.05100, 2022.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li. and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In 2019 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 8429-8438 8438, 2019. doi: 10. 10.1109/ICC. 2019.00852.
Kurt Shuster, Samuel Humeau, Hexiang Hu, Antoine Bordes, and Jason Weston. Engaging image captioning via personality. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12516-12526, 2019.
Shaden Smith, Mostofa Patwary, Brandon Norick, Patrick LeGresley, Samyam Rajbhandari, Jared Casper, Zhun Liu, Shrimai Prabhumoye, George Zerveas, Vijay Korthikanti, et al. Using deepspeed and megatron train megatron-turing nlg 530b, a large-scale generative language model. arXiv preprint ar arXiv:2201.1 1990, 2022.
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction-follow them all, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An cestruction-folloiing Ilama model. 2023.
https://githisn com/ tttptttttsttttaataeeeataa
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023.
Andrey Voynov, Qinghao Chu, Daniel Cohen-Or, and Kfir Aberman, p+: Extended textual condi- tioning in text-to-image generation. arXiv preprint 2rXiv:2303309922, 2023.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: sero-shot identity- preserving generation in seconds. arXiv preprini arXiv:2401.075 2024.
Xuan Wang, Guanhong Wang, Wenhao Chai, Jiayu Zhou, and Gaoang Wang. User-aware prefix- tuning is a good learner for personalized image captioning. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pp. 384-395. Springer, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoe Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning large language models, 2023. URL
11903
Jianwei Yang, Hao Zhang, Feng Ii, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, 2023. URL https://axxx
org/ htg/abb/2310. 2310
11441
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-ow12: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
13

---

Published as a conference paper at ICLR 2025
Wenhuan Zeng, Abulikemu Abuduweili, Lei Li, and Pengcheng Yang. Automatic generation of personalized comment based on user profile. arXiv preprint drXiv:1907.103711 2019.
Beichen Zhang, Pan Zhang, Xiaoyi Dong, Yuhang Zang, and Jiaqi Wang. Long-clip: Unlocking the long-text capability of clip, 2024. URLhttps: //atxpiv:////.....//// /ttaaaivvv.o// org/ abs./403. 153788
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models, 2023.
A ETHICS CLARIFICATION
We would like clarify that all images used in the construction of dataset and benchmark were collected from publicly available datasets, including Visual Genome (Krishna et al. 2016).
COCO (Lnn al. 2015). Object365 (Shao al. 2019). and Flickr30k (Plummer et al. 2016).
These datasets are all licensed under the Creative Commons Attribution 4.0 (CC BY 4.0) license. Additionally, all textual information (e.g., names, hobbies, and character traits) was synthesized by large language model (LLM) through prompting and randomly paired with the images of individuals. As result, all the textual information fictional and does not contain any real personal information of the individuals in the images, there no actual correspondences between the images and the textual information. Therefore, our curated dataset does not involve any leakage of personal information. We emphasize that the goal of our dataset is to enable the multimodal large language model (MLLM) to recognize and refer to individual information within its context, rather than to memorize personal details of specific people.
B LIMITATION
In our work, we propose novel in-context learning paradigm for the personalization of MILMs, which equips the model with personalization ability using data-centric approach, where we meticu- lously design an automatic pipeline for creating personalized conversation data. However, in our work, we did not take into account the bottleneck the vision encoder's ability for producing accurate personalized representations. We assume this is one of the causes for the gap between our trained model with the performance GPT4o. This a parallel direction to our proposed method, which we leave future work to further explore.
C DETAILS OF PVIT-3M
We elaborate the detailed statistics of our constructed PVIT-3M in Figure 3 In addition, we provide explanations different categories in Table [ The images used in our data construction are all collected from public datasets, including Visual Genome (Krishna et al. 2016), COCO (Lin et al. 2015), Object365 (Shao et al. 2019) and flickr30k (Plummer et al. 2016).
D MODEL CHOICES
Throughout paper, we used L.L.LAAa.... VA- (Liu et al. 2023a
base MILMM for P-LLaVA.
For our proposed personalized data construction pipeline, we adopt InternVL.2226B (Chen et al.) 2024) for dual level textual information extraction. For textual information fusion and QA generation, we adopt 1.MM..1--B-inae.
(Touvron et al. 2023). To crop out the people from images, we
adopt GrounndingDino (Liu al. 2023b).
E DETAILS OF P-BENCH
Directly annotating the data manually introduces tedious human labour. To boost the annotation efficiency, we adopt an annotation procedure the one adopted in LLaMA3 (Metal 2024). Specifically, first feed the images containing people to GPT4o, and instruct it to design QAs for each individual. Then, we manually check the validity of the designed QAs, and match them the corresponding individuals.
Benchmark Data Curation
annotation pipeline: we follow a similar
14

---

Published as a conference paper at ICLR 2025
Choics 136,th)
Mad dn rame chickes (44.74)
crop chap es (80
eag inoholee 18.84)
Multi-Choice Choice
aageinden tmruge cholne (16.
add image choice (125
adk cholca (1225))
Multi PVIT-3M
egg do- choice (16 Ski
aug-aro (1733)
Pareean D pescrpption Feeo-oon /orm
description description
descciption (175.44
Ausiatte descrppion
chonsiption dic)
oug so S dosol liption 196, 7M4
deseription
(100.0K)
Frea-form (14-455)
peroonalard (56
irnage
Personllieediing Q4 (58.24)
orop (114 14.11)
Figure 3: Statistics of PVIT-3M, large scale personalized instruct tuning dataset. Left: Data Distribution within Each Category. The outer cirele shows the distribution of all data categories and the inner circle shows the distribution of data subsets. Right: The detailed quantities of datasets.
Keyword Crop
Explanation
Uses cropped faces in the personalized prefix.
Uses generated photos based on the cropped faces in the personalized prefix.
Aug-In
Involves concatenating different cropped individuals into one image to replace the original scene image. The number "x" following "Aug-Sc" indicates the number of individuals concatenated.
Aug-Sc-x
Indicates an unanswerable scenario where the name in the prefix is different from the name mentioned in the QA.
Adv-Name
Indicates an unanswerable scenario where the individual photo in the prefix is present in the scene image.
Adv-Image
Pronoun Description
Replaces specific names with pronouns in the description. Asks the MLLM to give a description of "x" individuals while providing augmented scene image that concats "x" persons.
x-person description
The query is changed general one, such "describe this image". while requiring the model to include certain individual's name from the prefix in the overall description of the image.
Hosnnallzed-Hooistte Description
Asks reasoning- questions involving the specific indi- vidual.
Personalized Reasoning
Generates humorous and creative responses involving the specific individual.
Personalized Witty QA
Table 5: Explanation of PVIT-3M Dataset Categories.
Benchmark Statistics P-Bench contains two main categories: Multiple Choice (MC) and Descrip- tion. The statistics for each category are detailed below, along with the total counts.
MULTIPLE CHOICE (MC) CATEGORY
Type Statistics: The MC category includes six types of samples, totaling 915 samples:
Aug-Se-2: 100 samples
Aug-Se-3: 100 samples
Adv- Image: 100 samples
Adv-Name: 100 samples
Aug: 100 samples
Crop: 415 samples
15

---

Published as a conference paper at ICLR 2025
Number of People in Scene Image: Total of 415 images:
Image containing person: 221
Images containing 2 people: 85
Images containing 3 people: 48
Images containing or more people: 61
DESCRIPTION CATEGORY
Type Statisties: The Description category includes total of 100 samples, divided as follows:
Answerable (Aug-In): 60 samples
Unanswerable: 40 samples, consisting of:
Adv-Image: 20 samples
Adv-Name: 20 samples
Number of People in Scene Image: Total of 60 images:
Images containing person: 14
Images containing 2 people: 31
Images containing people:
Images containing 4 or more people:
Holistic Information Generation Prompt
<image>>
Provide detailed description this image, with special emphasis on the main character, including their appearance, expressions, actions, and any distinguishing features.
Table The prompt for generating holistic information.
Personal Information Generation Prompt
<image>>
Describe person this image. Focus on this person's 's main features. Remember, ** Nop include any background information. Additionally, in your response, you should use <name>> to refer to the person you describe when you mention the person's name first time. Again, you must contain <name>> in your response.
Table 7:
The prompt for generating specific person information. <image>> represents the cropped photo of individual from the scene image. Here, we ask the MILM to use the placeholder <name>> to refer the individual when describing.
DETAILED PROMPTS FOR DATA GENERATION
We provide the detailed prompts for each phase of data generation. In Table 6 and Table 7 we showcase the prompts provided to the MLLM, which are used for generating the holistic information and the personal information, respectively. In Table we demonstrate the prompt used for integrating the dual-level information into a single description. which is provided to the LLM along with the holistic and personalized information. In Table demonstrate the prompt to LLM that is used for creating the Colti-Choice QA. For dual-level information fusion and Multi-Choice QA generation, incorporate manually designed in-context examples guide the LLM.
16

---

Published as a conference paper at ICLR 2025
Dual-Ievel Information Fusion Prompt
TASK DESCRIPTION
Given the Person Information of person and the Holistic Information of the whole image, you need to put the placeholder <Chame >n the Holistic Information to represent the person described in Person Information. You should reply starting with "Combined Description.".
# EXAMPLE
Person Information: <name>> is sitring in relaxed posture. <name>> is wearing dark blue T-shirt and light blue jeans. On his left wrist, he has watch. <name>> smiling and appears be a jovial mood. holding blue object his right hand, which looks like piece of cloth or a towel. The background is simple blue wall, and there a light-colored blanket or couch behind him.
The image captures heartwarming moment between a man and young boy. The man, wearing glasses and blue shirt, is sitting on a couch, holding a blue balloon his hand. He is smiling and looking at the boy, who is seated next to him. The boy, wearing a green shirt, is also smiling and looking at the man. The hackground of the image features blue wall, adding the overall warmth the scene. The man't position the left and the boy's the right create a balanced composition. The blue balloon held by the man adds playful element the image. The smiles on their faces suggest a moment of joy and connection between the two.
Holistic Information:
Integrated Information: The image captures heartwarming moment between <name>> and young boy. <name", wearing glasses and blue shirt, sitting on a couch, holding a blue balloon his hand. He smiling and looking at the boy. who is seated next to him. The boy, wearing a green shirt, is also smiling and looking at <name>> The background of the image features a blue wall, adding the overall warmth of the scene. <name>> 's position on the left and the boy's on the right create balanced composition. The blue balloon held by <name> adds playful element the image. smiles on their faces suggest a moment of joy and connection between the two.
# TASK
Person Information: {Person Information)
Holistic Information: (Holistic Information] Integrated Information:
Table 8: The prompt for fusing person information and holistic information. The (Person Informa- tion] and (Holistic Information] are placeholders that will be replaced by the actual content be processed.
G TRAINING DETAILS
In Table 10 we illustrate the detailed hyper-parameters used when fine-tuning the MLLM with our PVIT-3M. We wish to note that we start tuning from the checkpoint of LLa7A- Liu et al. (2023a). train the MLLM with subset of our PVIT-3M with 1M samples. The entire training is conducted on 8 A100 GPUs with 80GB memory, which lasted for 30 hours.
H PERSONALIZED ATTRIBUTE CREATION
To create personal attributes, we adopt strategy similar to name swapping described in Sec- tion 1 Specifically, we first predefine a set of categories for attributes. In our work, we mainly consider the following: birthday, hobbies, family, education, favorite things and personality. Then, for each category, we prompt the LLM to generate a list of attributes. The prompts are provided in Table II These attributes constructing the QA pairs. We show the results of trained MLLM in Figure 4 which demonstrates that model can correctly associate the personal attributes with their photos.
dynamically combined with the person's 's images and names when
17

---

Published as a conference paper at ICLR 2025
Multi-Choice QA Generation Prompt
TASK DESCRIPTION
Now you need generate multiple-chot choice questions based Information. You should pay particular attention to the characteristics mentioned in the description that describe this person, and use these characteristics to create questions and possible answers.
# RESPONSE FORMAT
Your response must strictly follow the format below: .... "J. "answer": "..."]]
[[" [["question", "...", "choices": ["...",
# ATTENTION
Please ensure that all references the person in your questions and answers are replaced with the placeholder <name >
2. Only generate multiple-choice questions about the individual.
3. Ensure that each set of choices has clear distinctions and no overlap avoid multiple correct answers.
# EXAMPLE
Information: In the photo, <name>> wearing a white shirt blue jeans. She standing beside a man a blue T-shirt and has her hands on her hips. She is also wearing a black hag.
Generated MC: ["qquestion": "What color shirt is <name"> wearing?". "choices": ["Red". "White", "Blue". " "Black" J. "answer": "White") "J !"question", "What color are <name>> jeans?". "choices": ["Black", " "Green", "Blue", "Yellow"], "answer": "Blue"), ("quextion". "What is <name> doing with her hands?", "choices": ["Holding a bag", "Hands on her hips". "Waving", "In her pockets") J, "answer": "Hands her hips"], "qquestio"": "What accessory is <name> wearing?", "choices": ["A hat", "A searf", black bag". "Sunglasses" J, "answer": "A black bag"] "1
#TASK
Information: (Information)
Generated MC:
Table 9: Prompt for generating Multi-Choice QA.
RELATIONSHIP WITH VISUAL PROMPTING
Recently, visual prompting has garnered a lot of attention. Set-of-marks Yang et al. 2023) first observed that after associating each object with a mask and tag, GPT4-V able to associate each region according to the user input. Inspired by this finding, Draw-And-Unndrrttand (Lin et al. 2024b) designs visual prompt encoder and that can encode various visual prompts (points, bounding boxes, and free-form shape), which helps the MLLM better understand fine-grained features in the image. This line of work also has the potential to improve MILM's s personalized capability, which is in parallel with our proposed in-context learning paradigm and can be applied in conjunction: the former enhances fine-grat -ee-grai understanding of various objects in the same image, while the latter helps understanding the relationship between different images (e.g., personal image and scene image).
FAILURE CASES
In Figure we demonstrate the failure cases of our P-LLaVA. The leftmost and middle figures demonstrate P-LLaVA's capability recognize person of interest various ages. They show that the model performs well when the age difference relatively small but struggles to identify the individual as the age gap widens. The rightmost figure reveals another limitation: P-LLaVA
18

---

Published as a conference paper at ICLR 2025
Table 10: Hyper-parameters used for PVTT.
Parameter
Value
--loreennble
True
--lora.r
128
-loraaalpha
-mm-proojector.rr
le-4
-deepspeed
//aczipta/2erc2.jeen
--version
--vision tower
gpenaL/cl.l--vtt-aarg----- 3
-mn.rrojector.cypg
mlp2x.qela
-2
im start end
False
-mm.use_im.paccc.cooe
False
image aspect -.groug_ry.modaally..eeee
True
--bf16
Truc
--zumm train epochs
16
--per_devi ceueva Lbatchhusia
-_gaadient aacadiient.ccumlled sions
eva_luerron.strateee
--save_stree -saavestrratey
steps
-sava-steps
50000
-avv__tttal.immi
-ceannnngrrte
2e-4
weight decay
-wamup__ratio
0.03
1r_scheduler..ype
cogine
logging steps tf32
True
-modal max length
4096
gradient.checckoonniii ing
datallades muatiirrr_uwxoocrr
1azy preprocess
True
Attribute Generation Prompt
You are tasked with creating detailed and unique attributes for the general category attributes: <category>>, For this category:
1. Break it down into diverse and non-overlapping attributes or aspects.
2. Ensure the attributes well-defined, specific, and meaningful.
Table 11: The prompt for generating a list of attributes for a category. <category>> represents the category of attributes, such as birthday, hobbies, and personality.
has difficulty distinguishing the person of interest from others who closely resemble them, such as impersonators present in the scene. These failures could stem from challenges that inherently difficult, even for humans-for instance, when the age gap is substantial or when the impersonator or sibling bears strong resemblance to the person of interest.
19

---

Published as a conference paper at ICLR 2025
<Person>
This is Lec.
<Person>
This Lec.
<
Person
This is Chan,
information: lovas car racing outgaing, loves
loves car racing, outgoing, loves Siamese born Öctober 16, 1997.
Information:
Information: loves the chick, statistical analysis specialist, married with Monica.
Siamese cets, was born October 16, 1997.
David.
David.
Person
Joey.
Information: loves art, singer, was barn on July 11, 1969, humorous.
Information: loves art, a singer, was born on July 11, 1969, humorous.
Information: loves eating sandwiches outgoing. an actor.
<Person> This Kun.
<Person>
This Kun.
Person
This Joey.
Information:
basketball player,
Information::
basketball player,
information: prefers sky blue,
enjoys electronic dance music, has two siblings, loves cats.
enjoys electronic dance music, has two siblings, loves cats.
musician, has two siblings.
cimages
<image>
<image>
What the date of birth of the shown in this image?
What is the date birth of the this image?
What does the
this image enjoy eating free time?
The person the image was born on October 16, 1997.
The Individiall the image identifled as according the information provided, their date of birth is July 11, 1989.
The person image is wearing and appears be smiling, which could indicate he onjoys
foods.
This image shaws David, who was born on 11, 1969.
This image shows Joey, who loves eating sandwiches.
am sorry.
person.
Figure 4: P-LLaVA's ability to associate the person of interest with his/her attributes. Left: P-LLaVA successfully recognizes the person the scene image with his attributes (birthday). Middle: P-LLaVA VA rejects to respond when the person's 's information is not provided to it. Right: P-LLaVA can correctly identify the person of interest even if there are multiple people with the same name. The red square and green square contain the responses from the original MLLM and P-LLaVA, respectively.
<Person> This is Steph. <image> What color is Steph wearing this image?
PPrrsnn> This is Steph. <image> What color Steph wearing in this image?
PPersnn> This Jack. <image> Describe in this image.
A. Red, B, Black:; C. Grey, D.
Red, B. Black; C. Grey; D.
Brown:;
Jack is a person with fair complexion and short, light-colored hair. He is wearing red tie. He appears to be speaking and gesturing with his hand raised, pointing forward.
E. don't know Steph;; Steph not this image.
E. don't know Steph; F. Steph is not this image.
C. Gray.
F. Steph is not in this image.
Figure 5: The leftmost and middle figures illustrate P-LLaVA's ability to recognize a person of interest different ages. These figures indicate that the model can successfully identify the person of interest when the age gap is relatively small. However, when the age gap becomes significant, the model fails recognize the individual. The rightmost figure highlights another limitation: P-LLa VA struggles to differentiate between the person of interest and individuals who closely resemble them, such as impersonators in the scene.
20


Reviews:
[{"review_id": "hqXWFoMfy9", "reviewer": "Reviewer_YLzK", "summary": "This paper tackles personalization issue in MLLM. Current MLLM struggle to recognize and engage with specific people in images, making them less useful for personal AI assistants. The authors propose Personalized Visual Instruction Tuning (PVIT), which lets MLLMs handle personalized conversations by introducing people as a multimodal prefix without needing extra training. They created a streamlined data pipeline to automatically generate large-scale, personalized datasets and a new benchmark, P-Bench, to measure personalization abilities.", "questions": "1. Does it work well on non-human objects? \n2. For the MLLM used for comparisons, what are the settings? For example, some models are designed for one image input, how do you use it in multi-image input setting?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed Personalized Visual Instruction Tuning (PVIT) MLLMs conduct personalized conversation based on a general MLLM for personalized context without further tuning during test time.\n2. The benchmark P-Bench of around 1500 samples could be used on relevant tasks.\n3. Good qualitative and quantitative results.\n4. The proposal method is easy to follow and the presentation is clear.", "weaknesses": "1. In other work like Yo'llava, the personalization including various types of subjects, while in this work, it is limited to human face, which is different to the claim in the paper such as in the introduction.\n2. For evaluation comparisons, many actual SOTA MLLMs are not included such as GPT-4. I tried those models and found out they perform pretty good in the task. For example, for GPT4o are correct on several examples in Figure 2. \n3. No failure cases analysis. I do not think such method could resolve personalization MLLM perfectly, but it lacks the detailed analysis.\n4. Some proposed mechanisms are already commonly used in previous methods. For example, Personalized Wrapper Tokens basically denote the added information with special token.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SqZolFdB3f", "reviewer": "Reviewer_dSk8", "summary": "This paper introduces Personalized Visual Instruction Tuning (PVIT), a framework that enables MLLM to engage in personalized conversations with specific individuals. PVIT addresses the “face blindness” limitation of MLLMs by training them to recognize individuals within images and generate contextually accurate responses.\n\nNOTE: \n\nPost-rebuttal: After the first rebuttal round, I have decided to raise my score from 5 to 6 since the author's responses have cleared my most concerns.", "questions": "1.  Line 277: More advanced knowledge, such as character, hobbies, and professions can be easily\nextended and will be considered in future work. -> About How? Like character would be quite abstract to describe.\n2. How about the temporal changes, like inputting a child's image while asking for the same person with quite different ages?\n3. How about the twins with similar names and similar faces?\n4. What happened to the situations where two different with similar faces yet different names or similar names yet different faces?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. PVIT presents a novel approach to MLLM personalization by leveraging in-context learning capabilities, avoiding the need for additional training for each individual.\n2. A benchmark is proposed for future research on this direction.\n3. The paper is well-written and clearly explains the PVIT framework, data generation process, and experimental setup.", "weaknesses": "1. Some typos like Line 191-192 Two special tokens are mentioned, however, only one is described\n2. The current implementation focuses primarily on names and faces.\n3. The paper primarily focuses on individuals present in the training data. Some investigation on the zero-shot ability would help to understand this framework's ability.\n4. This problem is an interesting setting, but the proposed methods is not that novel.\n5. Current experiments are not that enough, more experiments as described in questions are welcome.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "El4wlPkZ3B", "reviewer": "Reviewer_C8KK", "summary": "This paper introduces Personalized Visual Instruction Tuning (PVIT), a novel framework that enables multimodal large language models (MLLMs) to engage in personalized, coherent dialogues tailored to specific individuals within images. The PVIT framework addresses the limitations of existing MLLMs, which often fail to recognize individuals and deliver contextually accurate, personalized responses. The paper proposes a comprehensive pipeline for personalized multimodal data construction. The experiments demonstrate that models fine-tuned with PVIT show substantial improvements in personalization tasks.", "questions": "1. Which version of LLaVA is finetuned as P-LLaVA?\n\n2. What models are used in dual-level textual information extraction and fusion?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. PVIT’s multi-phase pipeline is designed to generate personalized training data. By integrating visual expert models, image generation tools, and large language models, the authors ensure a rich and diverse dataset that enhances the personalization capabilities of MLLMs.\n\n2. The authors contribute P-Bench, a novel benchmark designed to evaluate personalization capabilities in MLLMs.\n\n3. Experiments demonstrate that PVIT improves MLLMs’ accuracy in handling personalized queries across various tasks. The ablation studies further validate the impact of component designs in the pipeline, highlighting the framework’s effectiveness.", "weaknesses": "1. Reliance on Pretrained MLLMs and Potential Hallucinations: The pipeline relies heavily on pretrained multimodal large language models (MLLMs) for tasks such as image captioning and textual information generation. However, these models are known to sometimes produce hallucinated details. Without mechanisms to detect or filter out such inaccuracies, the generated data could introduce noise into the training process, leading to degraded model performance or unpredictable responses. To strengthen the pipeline's reliability, the authors should provide examples of failed cases and outline a strategy for identifying and filtering out unreasonable or erroneous data.\n\n2. Ethical and Privacy Concerns in Human Image Sourcing: The paper does not mention the origin or sourcing of human images used for training and testing, which is critical from both ethical and privacy standpoints. Without a clear explanation of how these images were obtained and the measures taken to ensure consent and privacy, the approach may risk ethical violations. The authors should explain more about the original metadata collection.\n\n3. The limitation section, which is required, is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nQ4o0metjo", "reviewer": "Reviewer_ZV1c", "summary": "This paper tries to tackle the \"face blindness\" issue of the multimodal large language models. While, it is quite questionable whether you need the model to handle \"face blindness\" directly, or because it is safety and ethics issues, the model needs some mechanism to handle this issue safely. This is an important question for the motivation of this paper.\n\nTechnically, this paper presents a method to collect high-quality data for personalized conversations, then proposes Personalized\nVisual Instruction Tuning (PVIT) to enable the MLLMs (LLaVA in the paper) for personalized dialogues. This paper also constructs a P-Bench for evaluation purpose. Empirical experiments on P-Bench show that the P-LLaVA can do well in the personalized setting.", "questions": "1. As suggested, one potential major issue is about the ethics issues about the people topic, which needs more discuss. And also need safety evaluation for the data, benchmark and models.\n\n2. One question is - after you adapt the LLaVA model to the personalized setting, how its (P-LLaVA) performance on the original general benchmarks.\n\n3. And one more question is - just curious, how is the performance of GPT-4o or GPT-4v, Gemini or these frontier models on the proposed P-Bench? How they handle this people/face issue.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Technical Strengths:\n\n1. This paper proposes a way to curate the personalized instruction conversations, and also introduces a P-Bench for evaluation purpose.\n\n2. It finetuned LLaVA in the personalized setting, and demonstrates proposing results.", "weaknesses": "1. One major issue is there might lack a section to discuss the potential Ethics issues since it involves people in the dataset and models. \n\n2. Methodologically, the data curation is okay, the model training part is incremental.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)