Yo'LLaVA: Your Personalized Language and Vision Assistant
Authors: Thao Nguyen
Haotian Liu
Yuheng Li
Mu Cai
Utkarsh Ojha
Yong Jae Lee
Venue: NeurIPS 2024 poster (2024)
Keywords: personalization
multimodal models
Primary area: machine_vision
TL;DR: We introduce Yo'LLaVA -- a framework to embed personalized subjects (e.g., your pet) into a comprehensible prompt for Large Multimodal Models (e.g., LLaVA)

Abstract: Large Multimodal Models (LMMs) have shown remarkable capabilities across a variety of tasks (e.g., image captioning, visual question answering).
While broad, their knowledge remains generic (e.g., recognizing a dog), and they are unable to handle personalized subjects (e.g., recognizing a user's pet dog).

Human reasoning, in contrast, typically operates within the context of specific subjects in our surroundings. For example, one might ask, "What should I buy for *my dog*'s birthday?"; as opposed to a generic inquiry about "What should I buy for *a dog*'s birthday?".
Similarly, when looking at a friend's image, the interest lies in seeing their activities (e.g., "*my friend* is holding a cat"), rather than merely observing generic human actions (e.g., "*a man* is holding a cat").

In this paper, we introduce the novel task of personalizing LMMs, so that they can have conversations about a specific subject. We propose Yo'LLaVA, which learns to embed a personalized subject into a set of latent tokens given a handful of example images of the subject.  Our qualitative and quantitative analyses reveal that Yo'LLaVA can learn the concept more efficiently using fewer tokens and more effectively encode the visual attributes compared to strong prompting baselines (e.g., LLaVA).

Yo'LLaVA: Your Personalized Language and Vision Assistant
Thao Nguyen
Haotian Liu
Yuheng Li Mu Cai
Utkarsh Ojha
Yong Jae Lee
University of Masconsin-Madison
https://thaoshibee wwwbee.ggttu........ee ttt Bahub.ioYYoLLaVA.
Porsonalized Concepts
Personalized Text Conversation
Personalized Visual Conversation
Hoy, can your
photo?
LL-VA
<bop
<A>
Allt 4bo-. What recommend?
thinking about buying birtaday
soery, not sure whst you rederring Could you plaase provide more
personalizo 2D
It's always gond
LLavA
Yo' LLVV Yes, <boo is this photo He bring held by woman who is wearing
clacily question?
intoreats and profonmese the presee sou boying for...
Micley Mouse headhand.
Yo' LVV
cocsonaation
<bo- a cate dog! For a birthday gift, you could consider getting tho toy, line fitdh tay oe pussle toy, to keep tham estertained and mentally stimolatted. You could also consider taking sbor on fun cuting, like hike or trip to the park,
What Kāo doing?
LLAVA indivishale
image,
appear home setting...
Training Images
Input Image
holding smiling
Yo'LLaVA
Figure I: Given just a few images of a novel subject (c.g., a dog named <bo>), Yo'LLaVA learns to facilitate textual/visual conversations centered around that subject.
Abstract
Large Multimodal Models (LMMs) have shown remarkable capabilities across variety of tasks (e.g., image captioning, visual question answering). While broad, their knowledge remains generic (e.g., recognizing a dog), and they are unable to handle personalized subjects (e.g.. recognizing user's pet dog). Human reasoning, contrast, typically operates within the context of specific subjects surroundings. For example, might ask, "What should buy for my dog's s birthday?"; ; as opposed to generic inquiry about "What should buy for a dog s birthday?". Similarly, when looking friend's 's image, the interest lies in seeing their activities (e.g., "my friend is holding a cat"), rather than merely observing generic human actions (e.g., "a is holding cat"). In this paper, we introduce the novel task of personalizing LMMs, so that they can have conversations about specific subject. We propose Yo'L. LLaVA, which learns embed personalized subject into a set of latent tokens given a handful of example images of the subject. Our qualitative and quantitative analyses reveal that Yo' LL learn the concept more efficiently using fewer tokens and more effectively encode the visual attributes compared strong prompting baselines (e.g.. LLaVA).
1
Introduction
Consider the following questions: "What is <bo> doing in this photo?" or "I'm thinking about buying birthday gift for <bo>, What do you recommend?"! While simple, existing Large Multimodal Models (LMMs) [1-4] are not designed to answer such personalized questions. For example, while these models can use their broad knowledge to categorize objects and people an image (e.g., Fig.
Herr <bo> is user's pet dog.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Prompting
Learnable Prompt (Ours)
<aks> wellow-white plush shaped like a dog. Can you check if <ake> in this photo?
<aka> is <token, ><ooken>> ... <tokeng>.
Can you check <ake> in this photo?
Table I: Prompting vs. Learnable Prompt (Yo Yo)L,aVA), Instead of using an implicit text-based prompt (left), we personalize LLMs for subject (e.g., <sks>) by employing a learnable prompt (right).
(Right), "There are two individuals who appear in a home setting...", they cannot recognize those objects as specific subjects known to the user nor provide any personalized details (e.g.. "The man in the image is your friend <A>, and he is holding a cat."), without access to additional context.
Personalized AI assistants would be useful for a wide range of applications including health and wellness, education and learning, entertainment, etc. In particular, the way that individuals interact with and perceive modern Al systems can vary widely, underscoring the need for such systems to adapt to user-specific concepts and contexts [5. 6]. LMMs by default lack personalization primarily due to the nature of their training data (e.g., [7-9]), which predominantly consists of common and generic concepts (e.g., person, dog, bicycle), without personalized concepts (e.g., a person named <A>). Unfortunately, gathering training dataset that is personalized at scale can be difficult due to privacy concerns and also because the number of images for each subject might be limited (e.g.. a user is only willing to share 4-5 images of a person named <A>).
In this paper, we introduce Lo'LLaVA a novel personalized LMM built upon the state-of-the-ar LLaVA framework [2. 10]. Given just a handful of images of a personalized concept (e.g.. a personal stuffed animal). Yo' VAA learns to embed the concept into few special tokens (e.g.. <aks>), and can then answer questions about it when prompted. While one could try to describe the personalized visual concept using language (e.g.. "My stuffed animal named <sks> is a yellow-white plush shaped like a dog"), textual descriptions can often be vague and may not capture all visual details (c.g.. <sks> has a unique appearance that resembles a Shiba Inu) [11-14]. In these cases, learning a visual representation of the personalized concept can be much more precise.
There two key challenges for learning personalized LMM. First, when personalizing an LMM, we want to ensure that its broad pre-trained knowledge is unaffected (i.e., there is no catastrophic forgetting [15, 16]). To this end, we freeze nearly all the LMM's pre-trained weights, and intro- duce a set of learnalle input tokens [17, 18, 12, 19]: one special token <sks> and k latent tokens The special token acts as an identifier for the personalized con- cept, so that the user and model can refer to it in the input and output, respectively, while the latent tokens help to capture <ske>'s 's relevant visual details. The only pre-trained weights that we train are the output weights for the special token. In this way, the model can acquire new personalized knowledge through the learnable tokens, while retaining all of its prior knowledge in its original weights. This design has the added benefit of being fast to train and lightweight to store.
<token)><too <tokeng>........ ... <tokeng>.
The second challenge is enabling the LMM to capture fine-grained visual details. For example, when learning about a personalized subject e.g., <A>, we want the model to learn to recognize and distinguish <A> from other objects of the same category in meaningful way; e.g., that <A> is an Asian man who wears black glasses, has short black hair, etc., and is visually distinct from other Asian men who have similar features. To this end, we perform hard negative mining [20-23] to gather negative examples that are visually similar but not identical to the personalized concept. We then train the model with a large set of questions (e.g.. "Is <A> in this photo?") with both positive and negative image samples (e.g., "Yes" and "No"). In this way, the model learns to embed the fine-gran -raiind visual attributes of the personalized concept into the learnable tokens.
Contributions. In summary, our main contributions are:
We introduce a novel task of personalizing LMMs, enabling them to adapt to and answer to user-specific concepts. Officient.fummeoor Efficient frumework without forgetling: We introduce Yo'LLaVA - personalized LMM that efficiently learns personalized concepts with only a handful of images of each concept, while retaining broad pre-trained knowledge.
Personalized Large Multimodal Models:
Training dataset: We create novel dataset specifically designed to explore the task of personal- izing LMMs, providing a solid foundation for both training and evaluation.
Open source: We will publicly release the training and evaluation data for the personalized concept modeling task, as well as our code and models.

---

2 Related Work
Large Multimodal Models. In recent years, we have witnessed the emergence of large language models (LLMs) [1,24, 25,3], with significantly improved general meassion-answering and reasoning capabilities. These advancements have been further extended and we now have systems capable of language understanding as well as visual perception, i.e., Large Multimodal Models (LMMs) [26, 2, 4, 10]. These LMMs represent a groundbreaking frontier, enabling models to process and reason with input images alongside text, with applications spanning various domains such as embodied Al and robotics. However, while these models can showcase their general knowledge in many ways (e.g., recognizing and writing about a famous person or a dog breed in given image), they are not designed to handle personalized queries (e.g., recognizing you or your dog). In this work, we propose a method to extend the existing general purpose knowledge of such a LMM model to some new, personalized knowledge important for user so that a tailored, personalized experience can be given to that users (e.g., answering questions that relate to your dog).
Carraeter-Efffcien Fine-Tuning. Traditionally, fine-tuning has been the standard approach to adapt trained models to new tasks or concepts. However, in the era of LLMs/LMMSs fine-tuning these models can be extremely costly in both compute and memory requirements. To overcome this limitation, Parameter-Efficien Fine-Tuning (PEFT) methods has been introduced to adapt these models for various downstream tasks with only a small number of trainable parameters. There are two main directions: (1) Introducing additional trainable parameters into existing layers of the model (e.g.. LoRA [27], ALaMA-Adapte [28]). (2) Soft prompt tuning: learning prompts (c.g., text prompts) that can guide the models to adapt to new tasks or datasets. The latter concept is inspired by the ability of prompt engineering, which leverages task task-specific instructions (prompts) to enhance model abilities without modifying model parameters. Soft prompt tuning has shown impressive results in various tasks (e.g., agent tool calling [18]) and the concept has been extended to other domains (e.g.. recovering prompts from generated images [12]. learning image edits [14]). In this paper, we leverage the idea of soft prompt tuning to learn personalized concepts within the context of LMMs.
Personalizing Multimodal Models. In the context of image generation, personalization often refers to the task of enabling models recreate pixel-level visual details of a given subject [29, 13, 30]. Proposed methods often optimize either or both of the following: (1) token(s) for a specific concept (e.g.. (1. 30I or (2) the part/entire of image generation model (e.g.. [29]). In contrast, in the NLP community, personalization usually involves making LLMss behave in specific way (e.g., adopting a humorous or informal tone) [31, 32] or enabling LLMs to provide personalized responses (e.g.. recommending movies for specific user [33].. The main approaches include (1) prompting (e.g.. modifying system prompts for specific persona "You are a humorous person") or (2) information retrieval (e.g.. referring to users' saved metadata during communication). In the context of LMMs, however, personalization has been understudied. Personalizing a LLM requires extracting information not only from text (c.g., "<bo> is Shiba Inu"), but also from visual inputs (e.g.. "This is a photo of <bo>"). To the best of our knowledge, our paper is a pioneer in the personalization task for L.Mss. A concurrent work tackling the same problem is VLM [34]: but it relies external modules to recognize subjects, and is therefore not a completely integrated system. We position our work in the in- between image understanding and personalized conversation: After personalization, the LMM can not only recognize visual aspects of the subject, but also retain reasoning abilities about that subject (e.g.. "<bo> is Shiba Inu, so he may be very alert and loyal"). We also aim to have lightweight, complete system in which no external modules are involved, relying solely on the LMM itself.
3 Lo'LLaVA: Personalizing LMMs to Understand User-Specific Concepts
Given handful of images of person subject _ 1 (c.g.. 5 images of your plush toy called <sks>) without any textual labels or captions, our goal is to embed this subject into a pre-trained LMM (in our case, LLavA [2, 10, 35]), so that both the user and model can communicate using an identifier (e.g., <sks>) for that subject, while also retaining the broad pre-trained knowledge.
After being personalized, our method (Yo'LLaVA) can: (1) recognize the subject in new images during testing (e.g.. Yo'LLaVA can determine whether <sks> in a photo or not); (2) support visual question answering about the subject (e.g., given new photo, one can ask about <aks>'s 's location); and (3) support text-only conversations without any test-time reference images about the subject (e.g.. ask questions about intrinsic attributes of <aks> like its color, shape, etc.).
3

---

We start by detailing how we represent the subject as learnable concept for LLaVA in Sec. 3.1. We then discuss our methods enable Lo'LLaVA's to recognize the subject with hard negative example mining in Sec. 3.2, followed by discussion on enhancing understanding through hard negative examples enhancement in Sec. 3.2.
3.1
Personalizing the Subject as a Learnable Prompt
Prompting is straightforward and natural way to steer multimodal models. For example, when presented with image, if one wishes to ask an LMM whether their personal toy (e.g., called <sks>) is in that image, one might begin by pro- viding a personalized description (c.g.. "<sks> is yellow-white plush shaped like dog", Ta- ble 1, Left). However, manually crafting such prompts be cumbersome and often imprac- tical, as it can require an excessive number of words (tokens) accurately capture the subject. Crucially, describing all (subtle) visual details of a subject with words can be extremely chal- lenging, if not, impossible (e.g., describing how your friend looks different from any other person). Inspired by recent research in which shows that learning soft prompts can be more effective and efficient alternative [18, 14], we propose to represent personalized description for subject as learnable prompt for LMMs (Table 1, Right). This approach lightweight, requires updating only a few parameters (new tokens and corresponding output weights), while leaving the core parameters (e.g., image encoder, all layers except the output layer of the language model) untouched.
Yes, <ake> indeed
the photo.
Vocabulary
tokons
Large Language Model
<ake>
Vision Encoder & &rroectorr* +
Text Tokenizer
<token,> <token,>
laystan. prempt]
<token,>>
<ake> <token,><tokenp> <token,>.
can you recogniza <ake> in this phote?
Figure 2: Training pipeline.
Specifically, given set of images ,?,...,,"? 1 of a subject (e.g., called <sks>), we define personal- ized soft-prompt for the subject as:
"<aks> is stokeni><tokens ... <token.>."
Here, <sks> is a newly added vocabulary token that serves as an identifier for the subject, allowing both the user and the model reference this subject when asking or answering questions. The tokens (<token;>) Vi=1 are soft tokens that are learned to embed visual details about the subject, Since <sks> new entry to the token vocabulary, we expand the final classifier head matrix of the language model W from C x N to C' x (N 11), where C is the hidden feature dimension and N is the original vocabulary size. our Yo' LL framework, the trainable parameters are:
0
<sks>> <token
<tokeng>, W(,N+1))-
Herein, we train the k + newly added input tokens and the final classifier head matrix W associated with the identifier token <sks> only. Except from this, all other components of the pre-trained LLaVA [10] are frozen (i.c., vision encoder, vision projector, and language model).
To help the model learn the new visual concept, we generate conversational training data triplets (T', X ( X:), where I is an input image, X is the question, and X the corresponding answer (Details on dataset creation are in Sec. 3.2 and 3.3). We use the standard masked language modeling loss to compute the probability the target responses X, for each conversation length by:
p P(X.TT) = II p0 X.,s)) 1=1
where 0 is the trainable parameters, and Xa. are the instruction and answer tokens in all turns before the current prediction token 2y, respectively.
3.2 Enhancing Recognition with Hard Negative Mining
The most basic and essential ability for a personalized LMM its capability recognize personal- ized subject (e.g.. <sks>). A direct approach to achieve this is by creating visual recognition question and answer templates for training images. These questions can be as simple as asking whether <sks> is in the photo. However, training with only positive examples (or in another words, only images of <sks>) can lead to an undesirable shorteut, where the model learns to always answer "Yes" " for any question relating to the subject regardless of whether the subject is actually in the photo; rather than

---

Positive Examples
Basic question answering
Question: What type of object is <sks>? (Note: No image)
Answer: <sks> is a stuffed animal.
2 Positive recognition task
Question: <image> Can you see if <aks> in this photo?
Answer: Yes, <ake> is in the photo.
Negative Examples
Negative recognition task
Question: <image> Can you check if <aks> this photo?
Answer: have analyzed the image.
and can confirm that <sks> is not present in the photo.
Table 2:
Example of training dataset for subject <sks>.
learn the necessary visual attributes recognize the subject. To overcome this, we randomly sample 100 images from LAION [7] to serve as negative examples (images that do not contain <sks>). Training with a mixture of positive and negative examples helps the model understand the visual attributes of the subject (e.g., <sks> is a stuffed animal), but it can also lead to over-generalization. For example, if <sks> is a yellow dog-shaped plush, the model can overgeneralize and assume that all yellow stuffed animals are <sks>, which is undesirable. The challenge remains in how to improve the model's ability to distinguish more fine-grained features of the subject that can help differentiate it from visually similar ones (e.g., other similar yellow stuffed animals).
To overcome this, we employ hard negative mining [20-23]. If the subject <sks> is a stuffed animal, the hard negative examples would be other stuffed animals that are not identi- cal to the subject (Fig. 3, more examples of hard negatives can be found in Appendix 1). By exposing the model to a diverse range of visually similar but non-identical objects, we encourage it to learn more discriminative features and prevent over- seeer-eneraliztion.. We retrieve the negative exam- ples from LAION [36]. Specifically, for each training image I. (i 11 .... n), we retrieve the top m images with highest CLIP image embedding similarity [37]. Finally, the negative example data are: 100 easy and n x m hard negative examples for the subject <sks>
<ake?
Positive Images
Image Retrieval
Synthesis
not
<ake>
Negative Images
Conversations
Figure 3: Training data creation.
To enable the model recognize subjects in an image, we pair training images with recognition question-answar answer template. This process involves asking whether a specific subject (e.g.. <sks>) is present in the photo. In particular, during training, each positive and negative image is randomly paired with one of the question-answer templates (Details in Appendix F). Answer templates are sampled based on the type of input image (Positive vs. Negative). Essentially, all question-answer pairs are framed as binary classifications, with Yes/No questions determining if the subject (e.g.. <sks>) visible in the photo (See Type and QA in Table 2).
3.3 Learning to Engage in Natural Conversations about the Subject
So far, Yo'LLaVA is capable of recognizing a subject in a new image. However, learning with recognition data alone does not enable the model to communicate with users about anything beyond recognition. For example, while the model may correctly answer whether <sks> is present in an image, it might struggle with other questions (e.g.. "Describe <sks> in detail", see Tab. 7).
Thus, we next aim to create more generic conversations for training (c.g., visual question answering). These conversations focus on the subject's 's visual characteristics, as opposed to the recognition abilitics used in the previous recognition conversations.
For this, we use a template consisting of 10 manually written, basic questions related to intrinsic attributes, divided into two main categories: human-related questions (e.g., "What is the hair color of this person?") and subject-related questions (e.g., "What color is this subject?"). We exclude complex or nuanced questions that may not generalize to all (e.g., "What is the tail color of this toy?"). We show specific example in the Type QA in Table 2 (Please refer to Sec. C for details). For each image 7', we employ LLaVA [10] to generate an answer for each template question, forming conversation with triplet (I', X X
5

---

LLaVA vs. Yo'LLaVA
<bo?t
Visual Conversation
Question:
Can you see if <T> is this photo?
Hey, can you see <T> and <bo> in this photo? Could you write a cute and detailed Facebook caption it?
GPT-4V [26]
Sorry, I can't help with identifying or making
Yes, can see them in the photo!
+ text
assumptions about people in images.
Here's a cute Facebook caption: "Sunstiine, smiles, and Shiha snuggles! Couldn't I have asked for a hetter day at the park with my favorite furry friend! ' * #DogLover #ayr #Shibalnu #Best'urg lrrr''''e''''ddd" l'riend"
prompting:
LLaVA [2]:
I'm unable to see any individuals in the photo you've ve provided... [omitted]
In the photo, I see person and a dog. The person is sitting on the grass... [omitted]
Yo'LLaVA (Ours):
<T> is the photo, and she stunding another person. They both appear to be having a good time, possibly at a party or event, <T> is smiling and making a peace sign with her hand, which suggests positive and fun atmosphere.
to
Yes, I can see <T> and <bo> the photo. They sitting on the grass in park, enjoying a beautiful day. <T> is holding <bo?, and they both seem to be having a great time together.
As for the Facebook caption. here's cute and detailed one: "Spending the day in the park with my furry best friend! <bo> is the cutest thing I' I've ever seen, and I'm so grateful for our adventures together. <bo>, you' 'e the best boy! y * #ParkDay #DogLove #Furyyriend"
Text conversation
(Note: No input image are given)
Question:
How would you deseribe <T> to a blind person?
Can you describe <bo>'s car in detail?
GPT-
<T> is someone with long, dark hair that cas- cades down their back. They often choose out- fits that are both fashionable and enzy, priori- tizing comfort without sacrificing style.
<bo>'s car is likely upright, triangular, and slightly rounded at the tips, characteristic of the Shiba Inu breed. It would he covered in the same thick, cinnamon colored fur as rest his coat, possibly with some cream- colored accents around the edges.
4V [26]
+ text
prompting:
LLaVA [2]:
I'm sorry, but I'm not sure what you are refer- ring to with <T> Could you please provide more context clarify your question?
I'm sorry. but I'm not sure what you're 'e referring to. Could you please provide more context or clarify your question?
Yo'LLaVA (Ours):
<T> is a person with warm and friendly smile. She has long dark hair and is wearing glasses. She also wearing a white top with and patterned scarf around her neck. Her overall appearance cheerful and approachable.
<bo>'s car is small. round, and has dark brown color. It is located on the side of its head and appears to be perked up, indicating that it might he alert or attentive to something in its surroundings. The car is not overly large, but it is distinctive feature of <bo>'s appearance.
Table 3: Examples of Yo'LLaVA's abilities with a learned concept (c.g., a dog named <bo>). Yo'LLaVa can recognize and answer questions about that concept.
A conventional approach would involve training Yo'LL 'LLLAV directly with the triplet (T', X X However, this approach does not effectively facilitate the learning of personalized prompts (i.e., embed new visual knowledge into them), as the model is provided with extra information (the reference image I') already sufficient to answer the question. For example, if presented with photo of a stuffed animal <sks> and asked "What color is it?", LLaVA [10] would be able correctly answer the question without knowing or understanding the visual attributes of <sks>; i.e., it can simply use the input image to answer the question. Thus, to encourage the model to distill the visual attributes of the subject into the learnable personalized prompts, we exclude I' during training, which results in training solely with (X X (i.e., we omit image I in Eq 1 in practice). In this way, Lo correctly learns to embed the relevant visual information of the subject into the soft prompts, and can answer various questions about the visual attributes of the subject, even without any reference image as we shown in our text-only QA experiments (Table 7).
Experimental Setup
Training.
Unless stated otherwise, we use 5 images and I = 16 tokens to learn the subject. Each conversation is single-turn (one question and one answer). We use AdamW [38] with a 0.001 learning rate and -.L [10] as the base model. Training images include ~200 negative images per subject (~100 hard negatives from retrieval and 100 easy negatives randomly sampled). We train

---

Human
LLaVA [2]
GPT-4V [1]
<ake> is a male Asian model with
<aks> is wearing a
<eks> is a fashionahle individual with short, styled, platinum blonde hair, often seen in modern, stylish outfits.
black jucket with a skele-
white hair.
ton design on front.
Table Example generated descriptions for a subject, which can be used in place of the image as references (In this case, person).
each subject for up to 15 epochs, saving the best checkpoint based on recognition accuracy on the train set. All experiments are conducted on single A6000 GPU.
Dataset, We collect a new dataset of 40 subjects: Person (10), Pets (5), Landmarks (5), Objects (15), and Fiction Characters (5). The dataset is divided into train and test splits. The number of images per subject varies from 10-20 images. Please refer to Appendix C. for more details about our dataset.
Baselines. We choose Vanilla LL [2] as our main baseline. We consider two main variants of LLaVA: Naive LLaVA. which is simply LLaVA itself without any personalized information; and LLaVA + Personalized Description, where LLaVA is assisted with personalized descriptions about the subject. We employ two methods to acquire personalized descriptions: (1) Human- Human-written. written: We manually write a description for each subject (see Table 4, "Human"), mimicking a real scenario where a user describes a personalized subject to LMMs. (2) Automated description: We first prompt LLaVA to generate captions for all training images of this subject. We provide two ways to use these captions: (a) We concatenate all captions together resulting in a long, rich description for the subject; (b) We prompt LLaVA to summarize these captions into a brief personalized description (See Table 4 "LLaVA"). These automated descriptions correspond to "LLavV + Prompt, Text" with 11.3 (long) and ~6 (summarized) tokens in Table 5, respectively.
To expand our evaluation of prompting, we extend our analysis to GPT-4V, a leading proprictary multimodal chatbot. We use the same methodology to generate brief personalized descriptions (Table 4, "GPT-4V"). Additionally, as GPT-4V supports multi-image conversations (a feature not supported by LLaVA), we also experiment with personalized image prompting. Specifically, present training image(s) of the subject together with an introduction text (e.g., "You are seeing photo(s) of a subject named <sks>"). These experiments correspond to "GPT-4V + Prompt, Image" with ~ 1k (given image) and ~5k tokens (given 5 images), respectively (Table 5). Since images convey more information than text, we hypothesize that personalized image prompts represent the upper bound for prompting effectiveness. Notably, due to GPT-4V's closed-source nature, our approach cannot be directly integrated, making this comparison purely for reference.
5 Results
We showcase Yo' VA''ss performance across two primary tasks: (1) Recognition Ability and (2) Question Answering. The first task evaluates Yo'LLaVA's 's ability in recognizing personalized subject within a test image, while the second assesses the model's 's capacity to have natural conversations (i.e., refer and respond to queries) about a personalized subject.
5.1 Recognition Ability
First, we evaluate the model's ability to recognize personalized subject <sks>, We have 40 subjects, each with 5 to 10 test images containing the corresponding subject. For each subject, all of its test images serve as positive test images, while test images from the remaining 39 categories serve as negative test images. In total, there are 333 positive and 13,320 negative testing samples in our experiment.
During testing, we show photo to the model and ask, "Can you see if <sks> is in this photo? Answer with a single word or phrase." The ground-trutt truth response is "Yes" for photos containing <sks?, and "No" for others. We report the accuracy for positive and negative test images in Table 5. Given the imbalanced test set, we calculate the weighted accuracy: weighted = 0.5 accuracy positive + 0.5 accuracy negative.
Table 5 shows results. As expected, the Vanilla LLaVA baseline cannot recognize the personalized subject, an ability not present in it, and we empirically notice that it always answers "No, it is not <ske>" (thus resulting in 0.5 accuracy). When we prompt it with a short personalized description (whether self-ggnnraatd or crafted by human), LLaVA achieves decent accuracy (i.e., 0.819-0.823 with 16 tokens). On the other hand, overly lengthy descriptions negatively impact its performance (i.c., 0.650 with 1.3k tokens), likely due to too much side information that may not be helpful (e.g..
7

---

details about the background), In contrast, Yo'LLaVA displays clear advantages with trainable tokens, achieving the highest accuracy (i.e.. 0.924) with the roughly same amount of tokens.
We also present results using GPT-4V with both text and image prompting. Results indicates that Yo'LLaVA is better than GPT-4V with text prompting (i.e., 0.924 vs. 0.838-0.841). In terms of image prompting, GPT-4V performance improves with more reference images of the subject. Yo" YLALL with just 16 tokens outperforms GPT-4V with single image prompting (~~lk tokens). Also, is worth noting that Yo'LLa even only with 16 tokens, yields almost comparable results with GPT-4V using 5k tokens (5 images as reference); see Fig. 4. We anticipate that integrating Yo' Vo' with GPT-4V could significantly reduce the number of tokens used while maintaining performance; however, we could not try this since GPT-4V is a closed source framework.
Ours
LLaVA
LLavA []] + Pronpt
GPT-4V U1+ + Prompe
Accuracy Number Prompting Tokens
Type
Learnable
0
Text
Human
Text
Hummn
Image
Weightes Accuracy
0.925
tokens
16
<16
~1k
0.900
Recognition Accuracy
Positive
0.949
0.000
0.734
0,520
0.740
0.697
0.696
0.809
0.851
0.875
Weighted
Negaiive
0.924
0.898
0.500
1.000
0.903
0.19
0.980
0.650
0.903
0.974
0.985
0.992
0.501
0.998
0.850
Le'L (Ours)
GPT-4V Prompting
Question Answering
0.825
LLaVA Prompting
Visual 0.929
0.899
0.913
0.883
0.925
0.932
0.936
0.866
0.857
Text
0,883
0.659
0.503
0,663
0.790
0.770
0.798
0.982
0,987
101
10?
10
Number Prompting Takens
Table 5: Comparisons of Yo LL with LLaVA [2]; GPT-
Figure 4: Number of Prompting To-
4V results with personalized prompt presented as reference.
kens vs. Recognition Ability.
5.2 Question and Answering
To evaluate the performance of the model on a question-answering task, we develop new benchmarks for both visual and text-only question answering. For the visual component, we present a photo of a <sks> subject and pose questions about it (e.g.. "Where is <sks>?"). For the text-only component, we focus on questions concerning the intrinsic visual features of <sks> (e.g., "Is <sks> a dog or a cat?"). All questions are structured as multiple-choice options (A or B). In total, we create 571 questions; 17 1/400 visual/text-only questions. Examples are given in Appendix H. We report the accuracy of correctly answering the questions in Tab. 5.
Lo'LLaVA is the leading method in visual question answering (i.e., 0.929), followed by LLaVA [2] with a human-drafted personalized prompt. Overall, it is evident that if given an image, LMMs can use the presented information to answer questions accurately (e.g., given photo of dog, they can correctly identify the color of the dog's coat without knowing that the dog is named <bo>). For text-only question answering, where we do not have a test image and we directly ask questions about the subject, results indicate that text prompt (even by human) may not capture as many details as trainable prompt, as evidenced by YoA still being the leading method (i.e., accuracy of 0.883) compared with both LLaVA and GPT-4V. When given image(s) as prompt, GPT-4V can answer all the intrinsic questions very well (i.e., 0.982-0.90 0.97). But this is expected, because all the information can be found in the given image. However, it is worth noting that using an image as personalized prompt requires at least k tokens, while Lo'LLaV VA uses only 16 tokens!
5.3 Comparison with My VLM
We compare our method with the concurrent work of My [34] using their dataset, which consists of 29 different objects, and exactly follow their experimental protocol.
Requirements
Supported Cony.
Accuracy
Recall
Captions
External Recognizer
Visual
Text
Positive
Negutive
Weighted
Average
MyVLM [34]
yes
yes
96.6
90.9
93.8
96.0
Ours
no
no
97.0 (40.4)
95.7 (477))
96.4 (22
100.0 (14.0)
Table 6:
Ours MyVLM [34] following the experiment settings in [34]. Yo'LLaVA (Ours) demonstrates advantages over MyVLM without relying on external recognition modules.
To evaluate a model's s ability to recognize the personalized subject in an image, we utilize the same accuracy metric as VLM. If the subject is in the photo, we set the ground truth as "Yes"; otherwise, "No." We prompt Yo' LLLVV with "Can you see if <sks> this photo? Answer with a single word or phrase."
8

---

Ablation Study: Effectiveness of Token Length and Number of Images
Negative Example
Posiivv Example
Accuracy
1.0
1.0
1.0
0.8
0.8
0.8
0.4
0.4
0.4
0.2
0.2
0.2
0.0
0.0
0.0
4
16
20
28 32
0
12
16
24
28
8
12
32 16
20
24
28
Taken Length
Token Length
Token Length
1.0
1.0
1.0
Accuracy
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.0
0.0
0.0
Number of Images
Number of Images
Number of Images
Figure 5: Ablation studies on the number of trainable tokens and training images. Overall, the model's ability to recognize personalized subjects increases as these parameters increase.
Acc.
Describe <bo> in great detail. (Note: Yo LL'LLVV is not trained on this question)
Vanilla LLaVA
0.500
X I'm sonry, but I'm not sure what you'l re asking for... [omitted]
Recognition
0.707 (+0.207)
X I'm sorry, but I'm not sure what you re asking for... [omitted]
Conversation data
0.754 (40.04))
/ <bo> is a Shiba Inu dog with a distinctive fox-like appearance... [omitted]
Retricval Negative
0.914 (+0.160)
<bo> is a Shiba Inu dog with a distinctive fox-like appcarance... [omitted]
Table 7: Ablation study on dataset creation. To visualize the question-answwriin answering ability, qualitative example of the personalized model for <bo> is shown (Training photos provided in Fig. 1).
We compare to the reported numbers for MyVLM, which evaluates their concept heads (external face/object recognizers). We also evaluate whether the trained LMM can generate caption containing the subject's identifier e.g., <sks>. Following Vy.L.M we measure "recall", which is whether <sks> appears least once in the generated caption for its image. Table 6 shows the results. Our approach shows clear advantages for both metrics compared to MyVLM,, despite being simpler and not relying on an external recognizer (e.g., face recognizer).
6 Ablation Studies
Number of trainable tokens. We set the number of training images per subject at n = 10 and vary the number of trainable tokens & from 0 to 36. With & = 0, training is limited to the identifier token (e.g.. <sks>). As shown in Fig. 5 (first row), training just this token yields a low accuracy of 24% in recognizing the personalized subject. Overall, the number of latent tokens increases beyond == 8, the model's ability to recognize personalized objects generally improves for both positive and negative examples. To balance accuracy (higher is better) and token length (lower is more efficient), we select & = 16 for our final model, which yields 91 %% accuracy in this ablation study.
Number of images. Next, set the number of trainable tokens to k = 16 and vary the number of training images from n = to n = 10. Fig. 5 (second row) shows that the model's 's recognition ability improves gradually with more photos. We opt for 5 in final version of Yo' LLaVA, it is the minimum number of training images required to achieve 90+% accuracy in this ablation study.
Dataset creation. Finally, we conduct an ablation study on dataset creation. Table 7 presents the weighted accuracy for the recognition task and qualitative example of personalized model <bo> to demonstrate the model's capability in supporting question answering. Vanilla I.LaVA [10] cannot perform either text conversation or recognition (it always answers "No" to all test images, 50%). After training solely the recognition task (i.c.. determining whether <sks> is in given photo), LLaVA can recognize the subject to some extent (i.e., 70%), however, it still cannot perform text conversation tasks. After training with both synthesized conversation and recognition data, both recognition accuracy and conversation ability improve (i.e., 75%). Finally, with the introduction of retrieved hard negative examples (Yo' LLL VA), the accuracy significantly boosted to 91 %.

---

7 Conclusion
We introduced the novel task of personalizing LLMss which requires learning visual attributes of given subject (e.g., a dog named <bo>) from only handful of images, and then recognizing that subject in new images and performing question answering about that subject when prompted. To tackle this problem, we proposed Yo' Yo',LaVA, where personalized subject is represented by learnable prompts with an identifier (e.g., <sks>), and series of k latent tokens (e.g., <token 1>). Experiments showed that Yo" LLaVA can learn the concept more efficiently using fewer tokens and more effectively by capturing more visual attributes compared to strong prompting baselines (e.g.. GPT-4 and LLaVA)). A promising future direction involves integrating Yo" LLaVA with users' metadata (e.g., linking personalized concepts about dog named <bo> to its medical records or preferences) for enhanced personalization in real-world applications.
Acknowledgements. This work was supported in part by NSF IIS2404180, Adobe Data Science award, Microsoft Accelerate Foundation Models Research Program, and Institute of Information & communications Technology Planning & Evaluation (IIP) grants funded by the Korea government (MSIT) (No. 2022-0-00871, Development of AI Autonomy and Knowledge Enhancement for AI Agent Collaboration) and (No. R8S2022--01882238, Development of Large Korean Language Model Technology for Efficient Pre-training).
References
[1] OpenAL. Gpt-4 technical report. arXiv, 2023.
[2] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023.
[3] Gemini Team. Gemini: A family of highly capable multimodal models. In arXiv, 2024.
[4] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision-language understanding with advanced large language models. In arXiv, 2023.
[5] Hannah Rose Kirk, Bertie Vidgen, Paul Röttger, and Scott A Hale. Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback. In arXiv, 2023.
[6] Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hemanndez-Orally Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong. Scán Ó hEigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Yoshua Bengio, Danqi Chen, Samuel Albanie, Tegan Maharaj. Jakob Foerster, Florian Tramer, He He, Atoosa Kasirzadeh, Yejin Choi, and David Krueger. Foundational challenges in assuring alignment and safety of large language models. In arXiv, 2024.
[7] Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk. Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtred 400 million image-text pairs. In arXiv, 2021.
[8] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014.
[9] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In CVPR, 2021.
[10] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lee. Improved baselines with visual instruction tuning. In arXiv. arXiv:2310.03744, 2023.
[11] Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Han- naneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. Prompt waywardness: The curious case of discretized interpretation of continuous prompts. In ACI. 2022.
10

---

[12] Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-basee discrete optimization for prompt tuning and discovery. In NewrIPS, 2023.
[13] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion, arXiv, 2022.
[14] Thao Nguyen, Yuheng Li, Utkarsh Ojha, and Yong Jae Lee. Visual instruction inversion: Image editing via visual prompting. In NeurIPS, 2023.
[15] Cheolhyoung Lee, Kyunghyun Cho, and Wanmo Kang. Mixout: Effective regularization to finetune large-scale pretrained language models. In ICLR, 2020.
[16] Yuexiang Zhai, Shengbang Tong, Xiao Li, Mu Cai, Qing Qu, Yong Jae Lee, and Yi Ma. Investigating the catastrophic forgetting in multimodal large language models. In CPAL, 2024.
[17] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In ACL, 2021.
[18] Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu. ToolkenGPT: Augmenting frozen language models with massive tools via tool embeddings. In NeurlPS, 2023.
[19] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022.
[20] Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. CVPR, 2015.
[21] Ben Harwood, Vijay Kumar B G, Gustavo Carneiro, Ian Reid, and Tom Drummond. Smart mining for deep metric learning. arXiv, 2017.
[22] Chao- Yuan Wu, R. Manmatha, Alexander J. Smola, and Philipp Krähenbühl. Sampling matters in deep embedding learning. arXiv, 2018
[23] Yumin Suh, Bohyung Han, Wonsik Kim, and Kyoung Mu Lee. Stochastic class-based hard example mining for deep metric learning. In CVPR, 2019.
[24] Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint 2:3x::23033.6199, 2023.
[25] Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b. In arXiv, 2023.
[26] OpenAl Team. Gpt-4 technical report. 2024.
[27] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
[28) Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16/99, 2023.
[29] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning sext-to-image diffusion models for subject-driven generation. arXiv preprint 2208. /2242, 2022.
[30] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun-Yan Zhu. Multi- concept customization of text-to-image diffusion. In CVPR, 2023.
11

---

[31] Qian Liu, Yihong Chen, Bei Chen, Jian Jian-Guang Lou, Zixuan Chen, Bin Zhou, and Dongmei Zhang. You impress me: Dialogue generation via mutual persona perception. In ACL, 2020.
[32] Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. Personalizing dialogue agents: have a dog, do you have pets too? In Iryna Gurevych and Yusuke Miyao, editors, ACL, 2018.
[33] Baolin Peng, Michel Galley, Pengcheng He, Chris Brockett, Lars Liden, Elnaz Nouri, Zhou Yu, Bill Dolan, and Jianfeng Gao. Godel: Large-scale pre-training for goal-directed dialog. In arXiv, 2022.
[34] Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvlm: Personalizing vlms for user-specific queries. In arXiv, 2024.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Leava-next: Improved reasoning, ocr, and world knowledge. 2024.
[36] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open lcale dataset for training next generation image-text models. arXiv preprint vrXiv:2210.08402 2022.
[37] Romain Beaumont. Clip retrieval: Easily compute clip embeddings and build a clip retrieval system with them, 2022.
[38] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXivv1412.6980, 2014.
[39] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. 2023.
[40] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. 2021. If you use this software, please cite it as below.
[41] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. EMNLP, 2023.
[42] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang. Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? +xXiv:2307.0628/ 2023.
12

---

A Broader Impact
The broader impact of YoA, personalized visual assistant, has potential benefits and risks associated with its deployment and release. Some considerations are unique due to its visual nature, while others share similarities with existing instruction-folloiing following LLMs (e.g., Alpaca, Vicuna, etc.). As Yo' 'LaV is built upon LLaVA [2], it inherits some of the issues associated with LLMs and vision encoders (e.g., LLaMA [24], Vicuna [39], and CLIP [40]). In the following, we outline both the risks and mitigation strategies in place for the release of this model.
Hallucination. Similar to LLMs, Lo might generate outputs that aren't grounded in facts or input data (Tab. 15). This raises concerns about inferences made, especially in critical applications (e.g, medical).
Biases. Bias can be transferred from the base models to Yo'LLaVA, both from the vision encoder (CLIP) and the language decoder (LLaMA/Vieuna), This may lead to biased outcomes or unfair representations of diverse content.
Evaluation complexities. Assessing the performance of Yo'LLaVA challenging as it involves both language and visual tasks.
B Catastrophic Forgetting
Catastrophic forgetting is typically referred to as a scenario in which a neural network (e.g.. LMMs) completely substantially forgets previously acquired knowledge after being trained task. To evaluate the extent of catastrophic forgetting in Yo'LLaVA, we assessed both the Original -LaVA-11--133 [10] and Yo' YoVALaVA using common benchmarks for multimodal models: POPE [41], MMBench [42], and I.L.V--Wild [2]. The results are presented in Table 8. As shown, Yo'LLaVA maintains almost identical performance compared to the Original LLaVA, which is expected, as the all the core weighted of the models are frozen.
POPE [41]
MMBench [42]
LLaVA- [2]
Benchmark
rand
pop
adv
Original LLaVA [2]
0.87
0.87
0.86
0.68
72.3
Yo'LLaVA
0.86
0.86
0.85
0.68
72.3
Table 8: Catastrophic forgetting evaluation. Overall, Yo'LLaVA retain nearly identical perforamance with Vanilla LLaVA [10], while offers ability perform personalized conversation.
C
Dataset
visualization of our dataset can be found Table 2.
For person, we collect micro-influencers (e.g., TikTokers) or personal acquaintances. All pets are personal pets. Landmarks are local landmarks. Objects are obtained from Amazon's product reviews. Fictional characters are either from movies released in 2023 (c.g., <characterA>) or supporting characters (c.g., <characterC>), All subjects undergo evaluation via LLaVA [10] with the question "Do you know this person/subject?" to ensure LLaVA lacks prior knowledge of the subject.
13

---

D Additional Qualitative Results
We provide additional qualitative in Tab, dog named <butin>), Tab. 10 (a person named <Y>), Tab. 11 (a person named <T>), Tab. 12 (a cat named <mam>), Tab. 13 (a fictional character named <characterC>), and Tab. (a fictional character named <characterE>).
Vo'LLaVA
<butin>:
Visual Conversation
Question:
Can you see if <butin> is in this photo? What <butin> doing?
Hey, do you notice anything new about this photo of <butin>?
Yo'LLaVA:
Yes, can confirm that <butin> is in- deed in this photo. <butin> is laying the floor, looking up at the camera with somewhat surprised or curious expression.
Yes, do notice something new about <butin>, In this photo, <butin> is wear- ing black shirt and has hairstyle that in- cludes a bang. The hairstyle gives <butin> unique and adorable appearance.
Text conversation
(Note: No input image are given) Hey, can <butin> live in cold enviro-
Question:
Can you write short poem for <butin>?
ment?
YoVLLaVA:
Yes, <butin> can live in cold environ- ments. The dog the photo appears be a Siberian Husky, which is breed that is well-adapted to cold climates. They have thick double vides insulation against the cold, and they are also known for their ability to withstand harsh weather conditions.
In world of endless beauty. dog named <butin> stands to me. With eyes that sparkle like the sun, And coal that's pure and white snow.
that pro-
He's a sight to behold, this furry friend, A true companion until the very end. Through thick and thin, he'll always be, A loyal pup. so full of glee.
So here's to <butin>, the star the show, A pup that brings joy wherever he goes. He's gift from above, shining light, pup that brings day into night.
Table 9: Examples of Yo' 'LLV''ss abilities with learned concept <butin>.
14

---

Yo'LLaVA
Visual Conversation
Question:
Can you see if <Y> in this photo? If yes, what <Y> doing?
Is <Y> in this photo? If yes, describe his location and the scene.
Yo'LLaVA:
Yes, can confirm that <Y> is indeed in the photo. He is standing next to another person, holding a bottle of champagne in one hand and potted plant in the other. They both appear to be in room with whiteboard in the background.
can confirm that <Y> is indeed in the photo. He is standing in the center, wearing a navy blue sweater with white and red stripe pattern. He is smiling and appears to be enjoying the moment. The table in front of him is filled with various dishes, including whole roasted turkey. bowls, and plates of food. There are vases with flowers on the table, adding a festive touch the scene.
Visual Conversation
Question:
If <Y> here, what do you think that he would do?
I'm thinking about buying <y> this. Do you think that he will like it?
Yo'LLaVA:
If <Y> was here, I think he would enjoy the serene view of the lake and the city in the distance. He might take moment relax under the trees perhaps go for walk along the shore take in the peaceful surroundings.
While can't predict the future, it's certainly possible that <Y> will like the plush toy you't re considering buy- ing for him. It appears to be a cute and cuddly grey and white stuffed animal, which many people find appeal- ing. However, personal tastes and preferences vary, it's always good idea consider <Y>'s interests and whether this toy aligns with them.
Text conversation - (Note: No input image are given) Question:
What is <Y> typically wear?
What race do you think <Y> is?
Yo'LLaVA:
<Y> typically wearing gray hoodie and black pants.
<Y> appears to be of East Asian descent, but without not appropriate make sumptions about someone's race.
information,
Questions:
How old is <Y>?
Can you tell me some attributes of <Y>?
Yo'LLaVA:
<Y> appears to young adult. It's difficult to determine their exact age, but they seem to be in their late teens or early twenties.
a person with several attributes. From the image, can observe that he a strong and confident posture, as he stands with his shoulders back and head held high.
He has a friendly smile on his face, which suggests that he is approachable and enjoys engaging with oth- ers. His attire, which includes gray hoodie and blue jeans, is casual and comfortable, indicating laid-back personality. Additionally, his glasses give him in- tellectual appearance.
Overall, <Y> seems to be a person with a positive and outgoing demeanor.
Table 10: Examples of Lo'LLs abilities with a learned concept <Y>,
15

---

Yo'LLaVA
Visual Conversation
Question:
Is <T> in this photo? If yes, describe her outfit and location.
This is an Al generated artwork of <T>, Do you think that this is good illustration of <T>, Why why not?
Yo'LLaVA:
Yes, <T>is in this photo. She wearing a white top and is standing in the back row, second from the left.
Yes, believe illustration captures 's ap- pearance quite well. The character in the image has long dark hair, glasses, and is wearing a cozy sweater, which are all features that match the de- scription of <T> The character is also lying bed with dog and cat, which adds to the warm and comfortable atmosphere that I imagine <T> would enjoy. Overall, think the illustration does good job of representing <T>,
Visual Conversation
Question:
What is <T> doing in this photo?
Suggest Instagram caption for this photo of <T> For this photo of <T>, an Instagram caption could be: "Exploring the beauty of nature with my fa- vorite person. The tranquility of the woods and the company of good friends make for the perfect day.
Yo'LLaVA:
<T> is standing in front of a poster board, likely at conference or exhi- bition. She appears to be posing for photo or perhaps presenting her work.
#FriendshipGoals #NatureLoverss #Peace- fulPlaces".
Text conversation - (Note: No input image given) Question:
composing brief limerick for <T>?
Could you assist
<T> was Marvel character, based <T>'s appearance, who would it <T> be?
LoVLLaVA:
Of course. I'd be happy to help you com- pose a limerick for <T>! Here's 's possi- ble one:
Based on the photo you' 've provided, if <T> was a Marvel character, she might be similar to Wanda Maximoff, also known as Scarlet Witch. Both characters have similar style with their hair and attire, and they also have similar intensity in their expressions. However, it's important to note that this is just an observation hased the photo and not definitive answer.
There once was girl named <T> Whose smile was as bright star
She wore helmet steel And rode unicom school Where studied both hard far.
Table 1: Examples of Yo'LLaVA abilities with a learned concept <T>,
16

---

LLaVA Yo'LLaVA
Visual Conversation
Question:
Can you see if <man> is in this photo? If yes, please explain what is <mam> doing.
Is <mam> in this photo?
Yo'LLaVA:
Yes. can see <mam> in the photo. The cat is lying on desk next to a computer monitor, and appears be looking directly era. The cat's position and gaze give the impression that it attentively observing the camera or the person taking the photo.
have scanned the image, and can confirm that <mam> is not present the photo. The image shows two cats, one sitting and one standing, with plain background.
the
Visual Conversation
Question:
Is <mam> in this photo? Can you explain his expression?
What are the similarities and differences between this cat and <mam>? Explain your answer.
Yo'LLaVA:
<mam> is indeed in this photo. His expression appears to he one of mild annoyance discomfort, possibly due the water the sensation of heing bathed. Cats often have somewhat displeased look on their faces when they are being bathed, as they are not natural swimmers and may not enjoy experience.
The similarities between this cat and <eam> are quite striking. Both cats have similar color pattern with mix gray and white fur. Their have distinetive "M" shape their foreheads.
pointed upwards, and they
However, there are also some differences between the two cats. The cat in the photo appears to be larger and more muscular than <mam>. The cut in the photo also prominent ruff around neck, which less pronounced <mam?, Furthernore, the pronounced "M" shape on its forehead, while <mam²'s "M" is less defined.
more
the photo has
conversation - (Note: input image are given)
Question:
What does typically wear?
Describe <mam> appearance great details
Yo'LLaVA:
<mam> cat, and typically cats collars with tags bells, and sometimes they may wear sweaters other clothing, especially during colder weather. lowever, in the image you provided. <mam> is not wearing any clothing.
<mam> a gray white cat with distinctive striped pattern its fur. The stripes vertical and the length of its body, creating striking contrast against its gray fur. <man> has round head and large, expressive eyes that are a mix of gray and white. Its ears are pointed and perked up. indicating that alert and attentive to surroundings. <mam>'s flufly appearance. Overall, <mam> has very cute and charming look that is sure melt anyone's 's heart.
is long bushy, adding
Question:
Is <mam>> dog toy?
Can you identify the breed of <mam>?
Yo'LLaVA:
<mam>
domestic shorthair This is common type of domestic cat that has short coat and muscular, sturdy build. They are known for their friendly and adaptable nature, making them popular pets.
<mam> appears
Table 12: Examples of Yo'LLaVA's abilities with a learned concept <mam?,
17

---

Yo'LLaVA
<characterC>:
Visual Conversation
Question:
What is <character doing in this photo?
Among these mascots, which do you think is the most similar <characterC>?
YoALLaVA:
In the photo, <characterC> is sit- ting the floor next black cat. They both appear to a room with a pallerned carpet. <characterC> is wearing pink bow on its collar, while the cat has a red collar. They seem to be interact- ing with each other, possibly playing or just resting logether.
Among these mascots, the one that most closely resembles <characterC> is the white cat mascot. has similar color scheme and a cute, friendly appearance that is characteristic of <characterC>.
> Text conversation
(Note: No input image are given)
Question:
What the most unique thing about <characterC>?
Based on its appearance, what do you think the personality of that <characterC> is like?
Yo
The most unique thing about <characterC> is that she is a car- toon character with distinctive pink bow on her head.
<characterC> appears to be friendly and approachable character, given the warm and inviting expression on its face. The bow on its head suggests and perhaps playful or whimsical person- ality. The fact that is sitting in room with piano and book also implics that may have an appreciation literature. However, it's important to note that these observations are based solely visual information provided and may not fully capture the true personality of the character.
of charm
music
Table 13: Examples of Yo' abilities with learned concept <characterC>.
18

---

Yo'LLaVA
<characterB>;
Visual Conversation
Question:
Is <characterE> in this photo? Please explain his actions.
Can you describe the person that <characterE> is looking at?
Yo'LLaVA:
Yes. <characterE> is in this photo. He is sitting the ground next to the older gentleman, who appears to be in contemplative sad mood. <characterE> seems be looking up at the older gentleman with a con- cerned or curious expression, possi- bly trynng cheer him up or under- stand what's wrong.
<characterE> is looking at per- son wearing yellow hat and back- pack.
Text conversation (Note: No input image given)
Question: If <characterE> was a real dog, Can you guess <characterE'ss what breed do you think it is?
personality based on his appear- ance?
Yo'LLaVA:
If <characterE> was real dog, would guess that is Golden Re- triever.
his appearance, <characterE> seems be friendly and approachable character. The fact that he is smiling and waving suggests that he is in good mood and wants to engage with others.
Based
Table 14: Examples of Lo''Ls abilities with learned concept <characterE>.
19

---

E Limitations
Our approach is not without limitations. First, the accuracy in recognizing very fine-grained details is unsatisfactory, especially when the facial features of subject are relatively small in the image (see Table 15). Second, due traits inherited from language models, Yo'LLaVA may generate hallucinations provide ungrounded information like person's 's birthday, Table 15).
Limitation of Yo'LLaVA
<T>
Visual Conversation
[false negative example]
[false positive example] Is this a photo of <T>? Explain your answer.
Question:
Can you check if <T> is in this photo? Explain your answer.
YoLLLaVA:
can confirm that <T> is present in the photo. The person in the im- age is young woman standing to bicycle grassy area. There no individuals named <T> in this scene.
This is a photo of <T>, I can tell because she is wearing white shirt with the letter "A's" on it, and she has it. Additionally, she
distinctive hair bun with wearing glasses and necklace, which also visible in the photo. These details, along with her unique hairstyle, confirm that the photo is of <T>. [this not photo <T>]
[<T> is indeed the photo]
Text conversation (Note: No input image are given)
Question:
Do you know <T>'s 's birthday?
Can you describe <T> in detail? Try be as specific possible.
YoVLLaVA:
Yes, know <T> birthday. It's on December 25th.
<T> is a person captured a photo. Here are the details I
can observe:
[model assume person's birthday]
1. <T> wearing a light-colored top with a darker pattern
or design on it. The top appears to be casual and comfort- able.
2. <T> is wearing glasses with a dark frame. The glasses simple design and do not have any visible logos or branding.
3. <T> has long, dark hair that falls over the shoulders.
The hair has a natural, wavy texture.
4. <T> is smiling and looking directly at the camera. The expression on <T>'s face is friendly and upprouchable.
S. <T> is wearing a watch on left wrist. The watch
dark band and a small, round face.
6. <T> is wearing a pink cap with a white logo on the
front. The cap is casual and seems to a branded item,
possibly from sportswear or lifestyle brand.
7. <T> is wearing a dark-colored jacket or sweater over the top. The jacket has a collar and appears to be made of soft fabric.
Please note that this description is based on the visible elements in photo and does not include any speculative or imaginary content.
[visual detail is incorrect, <T> does not wear any watch
Table 15: Limitation of Yo" LLaVA.
20

---

Recognition Questions Template
We present templates for positive and negative question answering Tables 16 and 17, respectively. Given a training image we will randomly assign it template question for answering, forming training recognition conversation corresponding to either the positive negative type. There are total of 30 positive templates and 30 negative templates.
Index
Question
Answer
1
Is <sks> in this photo?
Yes, <sks> is in this photo. Yes, <aks> appears in this picture.
2
Can you tell if <aks> appears in this pic- ture?
3
Could you check whether <sks> is in the image?
<sks> is indeed in the image.
4
Do you see <sks> anywhere in this snap- shot?
<sks> visible this snapshot.
5
there chance <sks> could be in this photo?
Absolutely, <sks> is in this photo.
6
Would you happen to know if <sks> is shown in this photograph?
<aks> is shown in this photograph.
7
Can you see <sks> in this photo?
Yes, can confirm that <sks> is indeed in the photo.
8
Have you spotted <sks> this photo?
have spotted <sks> this photo.
9
Is that <aks> in the photo there?
That is <sks> in the photo.
10
Is <sks> in this image?
Yes, <sks> is in this image.
Am seeing <sks> this picture?
Yes, you are seeing <aks> this picture.
12
Does this photo include <sks>?
This photo includes <sks>
13
Is <sks> featured in this photo?
<sks> is featured in this photo.
Can you point out <aks> this photo?
Certainly, <sks> right here
photo.
15
Is <sks> visible in this photo? Check <sks> this photo for me, will you?
<aks> visible in this photo.
After checking, yes, <aks> this photo,
17
Hey AI, can you tell if you <sks> in this photo?
Yes, can confirm that <aks> is indeed in the photo.
18
Do you recognize <sks> in this photo? I'm looking for <aks>, <sks> in this photo?
do recognize <sks> in this photo. Yes, <aks> in this photo.
19
20
Can you see if <sks> is in this photo? This photo, does have <sks>? Could you confirm if this is <aks> the photo?
Yes, <sks> is indeed in the photo. This photo does have <sks>. can confirm that this is <sks> in the photo.
21
22
23
Any chance that <ske> might be this photo?
Yes, <sks> definitely in this photo.
24
Can you recognize if <aks> is in this photo?
Yes, <sks> is indeed in the photo.
25
I think I see <sks>, is it so?
You're correct, that is <sks> you see in the photo.
26
Has <sks> been captured in this photo? <sks>'s 's in this photo, right?
<sks> been captured this photo.
27
Yes, <sks>'s 's in this photo.
28
Is <ske> present in this particular photo?
<ske> is present in this particular photo.
29
Hey AI, can you tell me you recognize
Yes, can see <sks> the photo.
<aks> in this photo?
30
Can you if <aks> is in this photo?
Yes, <sks> in this photo.
Table 16: Example of positive recognition question answering.
21

---

Index
Question
Answer
1
Is <sks> in this photo?
No, <aks> is not in this photo.
2
Can you tell if <sks> appears in this pic- ture?
No, <aks> is not this picture.
3
Could you check whether <sks> is in the image?
<sks> is not in the image.
4
Do you see <aks> anywhere in this snap- shot?
<aks> does not appear in this snapshot.
5
Is there chance <aks> could be this photo?
There's no indication that <ska> in this photo.
6
Would you happen know if <aks> is shown this photograph?
<sks> not shown in the photograph.
7
Can you recognize if <sks> is in this photo?
have analyzed the image, and <sks> is not present in the photo.
8
Have you spotted <aks> this photo? Is that <sks> in the photo there?
have not spotted <sks> in this photo. That is not <sks> in the photo.
9
10
Can you check if <aks> is in this photo?
I've analyzed the photo, and <aks> is
present the image you provided.
11
Am seeing <aks> in this picture?
You are not seeing <sks> in this picture.
12
Does this photo include <sks>?
This photo does not include <sks>.
13
Is <sks> featured in this photo?
<sks> is not featured in this photo.
14
Can you point out <aks> in this photo?
can't I point out <sks> because <sks> isn' in the photo.
15
Is <ake> visible in this photo? Check if <ska> is in this photo for me, will you?
<aks> is not visible in this photo.
16
After checking, <sks> not in this photo.
17
Can you see <aks> in this photo?
I' analyzed the image, and <aks> is not present in the photo.
18
Do you recogmize <aks> in this photo? I'm looking for <aks>, is <sks> in this photo?
do not recognize <aks> this photo. <aks> is not in this photo.
19
20
there any sign of <ake> this photo?
There is no sign <aks> this photo. This photo does not have <sks>, confirm that this is <aks> in the photo.
21
This photo, does it have <sks>? Could you confirm if this is <ska> the photo?
22
23
Can you see if <sks> is in this photo?
I've searched the image, and <sks> is not present the photo.
24
Is <sks> part of the group in this photo?
<sks> is not part of the group in this photo.
25
I think I see <aks?, it so?
You do not <sks>, as <sks> is not in the photo.
26
Has <aks> been captured in this photo?
<ska> has not been captured in this photo.
27
<aks>'s 's this photo, right?
No, <sks>'s not in this photo.
28
Is <sks> present in this particular photo?
<ska> is not present in this particular photo.
29
I can't II find <aks>, is <aks> in the photo?
You can't find <sks> because <ske> is not in the photo.
30
<sks> this image?
No, <sks> is not this image.
Table 17: Example of negative recognition question answering.
22

---

G
Conversation Data Synthesis Question Template
The list of question templates used to briefly describe the image content for humans and objects is shown in Tables 18 and 19, respectively.
We chose to Riley reference the person because it is gender-neutral name. For other types of subjects (e.g., objects, pets), we refer to them as this subject.
After receiving the answer, replace Riley and this subject with the identifier (e.g., <sks>> to form the training conversation.
What is Riley color?
What color Riley eyes?
What is Riley height?
What is Riley skin tone?
How would you describe Riley hairstyle?
Does Riley wear glasses or any accessories?
How would you describe Riley's attire?
Does Riley have any distinctive facial features?
What is Riley overall build or physique?
What is Riley general expression demeanor?
Table 18: List 10 questions used for conversation synthesis for human.
What color is this subject?
What shape docs this subject have?
What is the overall vibe of this subject?
What material is this subject made of?
What size is this subject?
Does this subject have any patterns markings?
What type of object this subject?
Does this subject have any distinetive features details?
What's 's this subject general texture like?
How would you describe this subject overall appearance?
Table 19: List of 10 questions used for conversation synthesis for objects and animals.
23

---

H
Multiple choices questions answering
We present snapshot of multtppe--hooce question answering for dog named <bo> in Table 20.
Telt-only Question-Answering (No image given input) [due limited space, only fraction of the questions shown here]
Question 1: Is <bo> dog or cat?
A. A dog
B. cat
Correct Answer: A
Question 2: What is <bo>'s breed?
A. Shiba Inu
B. Corgi
Correct Answer: A
Question What is dominant color <bo>'s 's coat?
A. White
B. Orange
Correct Answer: B
Question 4: Does <bo> have tail?
No
B. Yes
Correct Answer: B
Question S: Is <bo> double coated?
Yes
B. No
Correct Answer: A
Visual Question-Answering
[due to limited space, only fffrccionn of the questions are shown here]
Question 1: What is the expression <bo>'s face?
A. Happy
Angry
Correct Answer: A
Question 2: What type of flooring is <bo> sitting on?
A. Hardwood
Carpet
Correct Answer: B
Table 20: Example of multiple choice question answering.
24

---

Subject
Personalized Description
<bo>
<bo? a well-groomed. medium-sized Shiba Inu with thick, cinnaroon-colored coat, cream accents, alert eyes, and a black collar.
<butin>
<butin> is fluffy, cream-colored Siberian Husky with striking blue eyes, black and white fur patters, and a playful demeanor.
<churchC>
cchurchC>>> is lowering, multi-tiered, ornate pagoda, surrounded by lush greenery and symbolizing Vietnamese culture.
<C>
<C> is blonde with braids, often wears trendy outfits, has bird tattoo, and white lace choker.
<D>
<D> is fashionable individual with short, styled, platinum blonde hair, often seen in modern, stylish outfits.
<characterE>
<characterE> is a 3D animated golden retriever with large expressive eyes, fluffy golden fur, and red collar.
<<>
<K> is a long-haired individual often seen in casual or traditional attire, usually sporting black wrist accessory.
<mam>
<mam> is fluffy grey tabby cat with wide-set yellow eyes, round face, and a plush coat.
</>
<A> is person often seen in casual attire, including striped polo shirts, sweaters, and t-shirts. They have short, dark hair and sometimes wear glasses. They are frequently pictured in both indoor and outdoor settings, with activities ranging from working on a laptop to possibly cycling outdoors. They are occasionally seen wearing a black helmet or a blue baseball cap.
<P>
<P> is bald individual with full, red beard often wearing a black cap and various T-shirts.
<objectM>
<objectM> is light pink, pig- themee ceramic mug with protruding cars, snout, and apple-shaped lid,
<objectX>
<objectX> is light grey, ceramic, cat-shaped mug with simple facial features and pointy cars.
<object
<objectG> is a large, tan and white, plush corgi toy with floppy ears, black paw details, and peaceful expression.
<objectF>
<objectF> is large, round, plush Shiba Inu toy, light brown on lop, white below, with a cheerful face.
<T>
<T> is person with long, dark hair, often seen wearing stylish, comfortable outfits.
<churchD>
<churchD> is an East Asian style stone pagoda with tiers, red characters, and a verdant
surrounding.
<churchE>
<churchE> is an ancient, towering brick structure with intricate carvings, arched entrance, and signs of weathering.
<N>
<N> is woman with long, dark hair, often elegantly dressed in various colors, adorned with jewelry.
<characterD>
<characterD> is a small, white, anthropomorphic mouse/cat, with large pink ears, pink nose, long black eyelashes, and a red or purple bow. She often has dreamy, hopeful, annoyed expression.
<V>
<V> is a short-haired, wlasing-wearing individual often seen in formal attire. acces- sorized with jewelry.
<character B>
<character B> is translucent, Pixar-style animated character with gradient purple- blue skin, expressive eyes, and wears patterned purple clothing.
<W>
<W> is curly- haired individual often seen in casual attire. such as blue-striped shirts, dark blue button-ups, or graphic tees, and often engaging in lively activities.
<t>
<Y> is short-hairee individual, often seen in dark clothing, with tattoo on his left forearm.
Table 21: Examples of GPT-4V's generated personalized text descriptions
25

---

I Visualization of Retrieved Negative Examples
We visualize the top-100 hard example images retrieved from LAION [7] for dog named <bo> (Table 22), stuffed animal <objectF> (Table 23), cat named <mam> (Table 24), and a person named <A> (Table 25).
Positive:
Negative Examples:
Table 22: Top 100 hard negative examples for <bo>.
26

---

Positive:
Negative Examples:
H1
Table 23: Top 100 hard negative examples for <objectF>.
27

---

Positive:
Negative Examples:
Table 24: Top 100 hard negative examples for <mam>,
28

---

Positive:
Negative Examples:
Table 25: Top 100 hard negative examples for <A>
29

---

Person
<)>>
<K>
<A>
<V>
<T>
<D>
<W>
<P>
<y>
<C>
Pets
<bo>
<mam>
<henry>
<mydieu>
<butin>
Landmarks
<churchA>
<churchB>
<churchC>
<chur
<churchE>
Objects
<objectA>
<objectB>
<objectC>
<objectD>
<objectE>
<objectF>>
<objectG>
<objectH
<objectI>
<object.J>
<objectK>
<objectL>>
<objectM>
<objectN>
<object0>>
Fiction Characters
<characterA>
<characterB>
<characterC>
<characterD>
<characterE>
Table 26: Dataset.
30

---

GPT-4(V)/LLA VA Prompts
We have listed all the prompts used to invoke GPT-4(V) [1] in Table 27.
For LLaVA, we employ the same prompts for Image Captioning, Caption Summarization, and Personalized Text Prompting. Since LLaVA does not support conversations involving multiple images, we are unable to conduct experiments on Personalized Image Prompting with LLaVA.
And example of personalized text description for selected number of subjects with GPT-4V are given Tab. 21.
Image Captioning.
You are an visual assistant, and you are seeing single image of <aks>, normally the main object or person in the image. Describe the photo in detail.
Caption Summarization.
You are given descriptions of the same subject named <sks> in different photos (e.g., the same cat).
Your job is to write a hrief description of the subject's appearance. Your description should be expressive enough that you use this caption to recognize the ohject in another photo.
You can use maximum ignored. Start your answer with: "<sks> is..."
words. Any extra words will be
Personalized Text Prompting.
<aks> SINSERT PERSONALIZED DESCRIPTION of <sks>). [INSERT QUESTION!
Personalized Image Prompting.
You seeing photo(s) of subject named <sks>,
Use the given photo(s) question about <sks>,
[INSERT QUESTION]
Table 27: GPT-4V Prompts
K Visualizing Learned Tokens
It would be interesting to what information is embedded in latent tokens <token,>. To visualize this, we directly ask LL'LA to Explain <token;>". The results for cat named <mam> and a dog named <bo> are presented in Tab. 28 (train with h = 16 tokens).
Interestingly, each token seems embed complete sentences about the visual aspects of the training images and the visual characteristics of the subjects (c.g., "<mam> is a cat with a gray and white coat"). We empirically found that while there is duplicate information between these tokens (c.g.. "<bo> is dog"", each token appears to carry distinctive features that other tokens do not have.
In Tab. 28, trim the duplicated information and only show the distinctive information that the corresponding tokens carry. This demonstrates that learnable prompts (Yo'l Vo) not only learn to embed the visual attributes of the subjects into soft-prompts, but also try to distinguish the visual features of the subject. We leave the task of fully explaining and exploring the meaning of these tokens for future research.
31

---

Token
<man>:
<bo>:
<tokeng?
<mam> uppeurs to be a cat with a gray and white coat its cars are perked up. which could indicate curiosity alert- ness. The cat's 's fur looks soft and well- groomed, and it has a calm demcanor
<bo> is dog with brown and white coat ... <bo> is looking directly at the camera with happy expression his facc. His cars are perked up and his tail is wagging, which suggests that he is in good mood...
<tooken>> >
The cat's 's tail is wrapped around its body... ...
It appears to be in relaxed and happy state, as indi- cated by its open mouth and wagging tail. ...
<tokeny>
<mam> appears to be a domestic
The dog appears to be happy and relaxed, as indicated by open mouth and wagging tail ... The dog't position the rug suggests that comfortable and ease in its environment...
shorthair cat...
<tokeny>
The cat's position the counter- top and its attentive gaze suggest that it might be interested in something hap- pening... ...
... His ears perked up and his tail wagging, which suggests that be is in good mood...
<tokeng>
It appears that you have attempted to type a sentence, but there are some cr- rors and incomplete phrases.... ...
The image shows a dog named <bo>, who appeurs to breed with distinctive facial structure, such as a brachycephalic breed like a Pug or a Shih Tzu...
<tokens?
<mam> appears to looking directly the camera with somewhat curious atlentive expression...
The dog's 's fur is mix brown and white, and it has a distinctive black patch over one eye...
<tokeng>
The seems be curious or
... The dog's coat is mix of brown and white, and seems to be in relaxed and happy state...
attentive...
<tokeny>
<mam> <mam> <nam?...
<bo> is looking directly the camera with happy expression on his face. His cars are perked up and his tail is wagging, which suggests that he is in a good mood...
<tokeng>
The cat's 's attentive posture and the fact that he is looking directly at the camera ...
<bo> is looking directly at the camera with happy ex- pression on his facc. The room appears to be a cozy and comfortable living space.
<tokeny>
It seems like a comfortable space <mam> to relax and observe his sur- roundings... ...
The overall expression on the dog's ' faaee is one of contentment and friendliness...
<token1u>
...
The cal has distinctive striped pal- on its face and ears, which com- mon in some breeds like the Ragdoll...
Overall, the image captures a pleasant moment of dog enjoying time indoors....
<token]1>
The cat's 's expression and the overall
Iii tail is curled up
setting suggest a moment of calm and curiosity...
<token12>
<mam> is gray and white cat with
The dog's 's fur is mix of brown and white, and it has a distinctive black patch over its eye.
fluffy coat...
<tokenig>
<nan> appcars to be looking di- rectly at the camera with a somewhat curious or attentive expression....
Hc is sitting on a white rug and appears to be looking directly at the camera with a relaxed expression....
<tokenjy>
... It seoms like a comfortable space ... the rug provides a comfortahle spot for <bo> to <nam> relax and observe his surroundings...
on...
<taken1s>
The has fluffy coat with mix of gray and white fur, and his eyes wide open...
...
... breed with distinctive facial structure, such as brachycephalic face, which is characterized by short snout and broad, flat forehead .. its front paws tucked under its body and its tail curled up beside it... The dog's ' fur looks welo-groomed and shiny, suggesting that is well taken care
Table 28: We ask Yo'LLaVA to explain each learned token by prompting it with "Explain <token,>".
32

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We propose novel task of personalization for Large Multimodal Models and a framework called Yo' VL to achieve this.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: discuss limitation Sec. and provide qualitative results for that in Appendix E.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess..
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
33

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
We do not present any theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Code and data will be open-sourced. Details about experiments setting can be found in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
34

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code and data will be released to public upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co) publcc/guuides/ooeebbbbssssss oon.eeesuuumssssooooooooyyo Policy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https: httllss.cc/ubblccguuuiieee htapliii/guude//Cddeeeieei
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Details can be found in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
35

---

The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question:
Does the paper report error bars suitably and correctly defined or other appropriate
information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Fig. 5 reported variance in accuracy across different categories.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
OK to report I-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Details can be found in Sec. 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
36

---

Question:
Does the research conducted in the paper conform, in every respect, with the
NeurIPS Code of Ethics https: attps://neurigg httpge///arrpp...o/pb.i///eaaaee
Answer: [Yes]
Justification:
We make sure to preserve anonymity (e.g., all figures paper are blurred
faces).
Guidelines: The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Yes, we discuss Broader Impacts in Sec.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Yes, we discuss this in Sec, A.
Guidelines:
The answer NA means that the paper poses such risks.
37

---

Released models that have high risk for misuse or dual-use use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cited original paper whenever their dataset/ code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. cot..coo/aaaeeeoeeeoooo comddatasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should provided.
If this information is available online, the authors encouraged to reach out the asset's 's creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Yes, we describe how to collect dataset in Sec. C.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
38

---

14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
39


Reviews:
[{"review_id": "lQmNj7TbiN", "reviewer": "Reviewer_PZy9", "summary": "The paper introduces Yo'LLaVA, a novel approach to personalizing Large Multimodal Models (LMMs) to handle user-specific concepts and contexts. The proposed method embeds a personalized subject into a set of latent tokens given a handful of example images, enabling personalized textual and visual conversations. The paper includes qualitative and quantitative analyses demonstrating the efficiency and effectiveness of Yo'LLaVA in comparison to existing prompting baselines.", "questions": "Although only limited number of images are required, the process of curating the conversationing data and retrieving hard negatives should be executed for each personalized subject, which incurs a lot of human efforts.", "limitations": "See weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The task of personalizing LMMs is novel and addresses a significant gap in current LMM capabilities, which are primarily designed for generic tasks. The proposed method has wide-ranging applications, including personalized assistants in health, education, and entertainment.\n2. The approach of using learnable prompts to embed personalized concepts is innovative and well-justified. The method ensures the retention of broad pre-trained knowledge, which is crucial for maintaining the overall functionality of the model while adding personalized capabilities.\n3. The paper provides both qualitative and quantitative analyses to validate the effectiveness of Yo'LLaVA. The comparisons with strong prompting baselines (e.g., LLaVA) highlight the improvements in terms of efficiency and the ability to encode visual attributes.", "weaknesses": "1. While the paper provides promising results, the evaluation is somewhat limited to specific tasks. A broader evaluation across more diverse tasks and real-world scenarios would strengthen the claims.\n2. The paper does not thoroughly address the scalability of the proposed method. How well does Yo'LLaVA handle a large number of personalized subjects or frequent updates to personalized concepts? Discussing and testing the scalability of the approach would be beneficial.\n3. The method proposed in this paper effectively addresses the issue of personalizing LMMs and highlights its advantages over providing a language description for a given object. However, what are the benefits of this approach compared to directly providing image of the object to the LMM, and then using a CoT to first prompt LMM to give detailed descriptions of the subject, and bind these descriptions with the speical token like <bo> in the paper. Besides,  I have also tried the proposed personalized setting on GPT4-o, and it seems that GPT4-o can directly handle this problem.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **While the paper provides promising results, the evaluation is somewhat limited to specific tasks. A broader evaluation across more diverse tasks and real-world scenarios would strengthen the claims.**\n\nWe agree that a broader evaluation across more diverse tasks would strengthen the paper.\n\nHowever, at this moment, we are not aware of any benchmark/ datasets for this personalization yet (e.g., personalized datasets in healthcare, business).\nThus, we include two main types of tasks: Recognition (Sec. 5.1) and Question and Answering (Sec. 5.2), which are the foundational tasks of personalization (similar to MyVLM [1], in which introduce a novel dataset with 29 concepts, while our dataset has 40 concepts).\n\nWe have also made an effort to test the model with some examples of real-world scenarios (e.g., Fig. 1, suggesting a gift for a pet’s birthday; or Table 3, Left, writing a Facebook caption). More examples can be found in Appendix, Table 9-19.\n\n> **The paper does not thoroughly address the scalability of the proposed method. How well does Yo'LLaVA handle a large number of personalized subjects or frequent updates to personalized concepts? Discussing and testing the scalability of the approach would be beneficial.**\n\nIn Table 3 (Right), we indeed show examples of handling 2 subjects at a time (a person named <T> and her dog named <bo>). In the Rebuttal PDF, we show additional qualitative results of handling 3 subjects (a person named <T>, her dog named <bo>, and her cat named <mam>). If users want to update the personalized subject, the re-train process would take roughly 40 minutes.\n\nA further study on scalability of Yo’LLaVA would heavily rely on the availability of a large scale dataset/ benchmark for personalization. At this moment, we are not aware of any such available dataset/ benchmark for this task yet.\nTo collect and label this dataset would be challenging, as (1) it depends on privacy (e.g., as stated in Line 31 “user is only willing to share 4-5 images of a person named <A>”); and (2) multiple subjects should be related (e.g., different subjects appear in the same photo like <T> and her pets <bo> and <mam>).\nWe anticipate that future research will address these challenges!\n\n> **The method proposed in this paper effectively addresses the issue of personalizing LMMs and highlights its advantages over providing a language description for a given object. However, what are the benefits of this approach compared to directly providing image of the object to the LMM, and then using a CoT to first prompt LMM to give detailed descriptions of the subject, and bind these descriptions with the speical token like <bo> in the paper. Besides, I have also tried the proposed personalized setting on GPT4-o, and it seems that GPT4-o can directly handle this problem.**\n\nThanks for your suggestion to use Chain of Thought (CoT) to improve the detail of descriptions about a subject. In limited rebuttal time, we have quickly tested LLaVA + CoT for more detailed descriptions, and recognition accuracy increased roughly 5% (from 0.819 to 0.852 (Table 5)). While increasing, this is still behind Yo’LLaVA (0.924) and Image Prompt (0.901-0.925).\n\nThis is expected as discrete text will always contain some ambiguity and may not fully capture the visual details (e.g., for a dog named <bo>, the coat color might be described as “yellow,” but visually, yellow can have many different shades). The strongest baseline is directly providing the image prompt. We indeed tried this in the main paper with GPT-4V, as GPT-4V is currently a leading proprietary multimodal model. The results shown in Table 5 demonstrate that Yo’LLaVA can achieve similar performance to GPT-4V with 5k tokens, which is much more computationally efficient!\n\nWith text-only response models (e.g., LLaVA, GPT-4V), Yo’LLaVA can help to retain the recognition accuracy while using much less tokens (e.g., increasing from 0.822 vs 0.924).\nWe anticipate that when shifting to another modality (e.g., image/text generation models like Chameleon [2]), Yo’LLaVA also helps with image generation quality.\nWe provide early results from our experiments integrating Yo’LLaVA and Chameleon in the Rebuttal PDF (Fig. 1). As shown, the optimized prompt captures much more precise visual details about the subject compared to the plain text description (e.g., Yo’LLaVA + Chameleon captures more details about a dog named <bo>)!\n\nThese early results show that not only Yo’LLaVA can be adapted to other multimodal models (e.g., Chameleon), it can also provide clear advantages with clear text personalized prompts.\n\n> **Although only limited number of images are required, the process of curating the conversationing data and retrieving hard negatives should be executed for each personalized subject, which incurs a lot of human efforts.**\n\nAll the training data is generated automatically, so no manual labeling is needed!\n\nFor the conversation training data, we employ LLaVA to generate answers for each template question (Lines 215-216). The template questions are fixed and universal for all subjects (a complete list of these questions can be found in Appendix G, Tables 18-19). For the negative images, for each positive image, we retrieve the top m images with the highest CLIP image embedding similarity (Lines 191-192). This process is fully automated and does not involve any human supervision!\n\n*Reference:*\n\n*[1] Alaluf et. al., MyVLM: Personalizing VLMs for User-Specific Queries, ECCV, 2024*\n\n*[2] Chameleon Team, Chameleon: Mixed-Modal Early-Fusion Foundation Models, arXiv, 2024*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFM1BanuCq", "reviewer": "Reviewer_VkWL", "summary": "The paper attempts to personalize LLM's by adding personal details like dog etc. The overall idea is to add the corresponding tokens in the LLM and fine-tune the last output layer for the newly added tokens in the embedding space. This results in the ability to do personalized question answering and recognition.", "questions": "Please answer the weaknesses above.\n\nW1 above requires justification on why the method is more than providing image context in the system prompt. It is understandable that current LLaVA architecture does not allow that. But, will this method still be useful when such an architecture is open sourced? Or provide justifications on why multi-image conversation model will not be released anytime soon.\n\nPlease answer other weaknesses as well that are crucial for the paper.", "limitations": "The authors have discussed the limitations and weaknesses and it correctly reflects the issues.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The paper is nicely written with all the details covered with figures that improve the understanding.\n\n2. The overall idea is good, and personalization is an important trait that current LMMs are missing.\n\n3. The proposed approach is simple, and the model is easily able to learn new personalized concepts.", "weaknesses": "1. The overall paper relies on the fact that LLaVA-like architecture cannot do multi-image conversations (L253) and the only available model is GPT-V at present, which is not open source. I believe this is a strong assumption. Given the strong performance of GPT-V, it is also likely that a LLaVA-like open source model that supports multi-image conversation will achieve good performance when the image is given in the system prompt and then asked to recognize or do QA (similar to the GPT-V experimental setup).\n\n2. The current architecture scheme does not look scalable. To add one concept, the model has to add (k+1) tokens. A person can have typically 10 personalized concepts that they might want to add, so the token space increased fast. Given the nature of the problem, it is essential to discuss what happens when there are multiple concepts that are to be learned.\n\n3. (L307) It is true that GPT-V requires more tokens compared to your method, but the less token requirement is a direct consequence of training with the new token, whereas GPT-V is zero-shot. Hence, I do not find this paragraph appealing.\n\n4. The recognition metrics is unclear to me. Given the four categories True Positive (TP), FP, FN, and TN, it is good to report F1 score and other scores that consider class imbalance. Is the average of positive and negative accuracy a standard metric?\n\n5. (Fig. 4) The GPT-V graph keeps increasing with increase of prompting tokens. Is it possible to go one order more to see the saturation in the performance? The current trend makes it appear that it can easily go above Yo'LLaVA's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **The overall paper relies on the fact that LLaVA-like architecture cannot do multi-image conversations (L253) and the only available model is GPT-V at present**... [omitted]\n\nYo’LLaVA can learn personalized subjects more efficiently using fewer tokens and more effectively encode visual attributes compared to strong prompting baselines (e.g., GPT-4V + Image prompting).\n\nIt is evident that Yo’LLaVA, with just 17 tokens, outperforms GPT-4V with single image prompting (1k+ tokens) (Tab. 5). Notably, Yo’LLaVA, even with only 17 tokens, yields results almost comparable to GPT-4V using 5 images (5k+ tokens); see Fig. 4 (Lines 286-288).\nUnfortunately, GPT-4V is a closed-source model, so we cannot integrate Yo’LLaVA with GPT-4V to reduce the number of tokens used while retaining performance.\n\nRecently, Meta AI released Chameleon [1] - a LMM that supports multiple images/text as inputs and images/text as outputs. We have tested Yo'LLaVA with Chameleon, and it has demonstrated superior results in image generation! (See Rebuttal PDF, Fig. 1). These early results show that not only can Yo’LLaVA be adapted to other multimodal models (e.g., Chameleon), but it can also provide clear advantages!\n\n> **The current architecture scheme does not look scalable**... [omitted]\n\nYo’LLaVA is much more token-efficient while achieving similar performance with Image Prompting.\n\nAs noted by the reviewer (in above comment about Image prompting vs. Yo’LLaVA), consider a scenario where we need to personalize 10 subjects with two options: (1) Yo’LLaVA and (2) Image Prompting.\n* Using Yo’LLaVA: We would add 10 (subject) x (16+1) (token/subject) = 170 tokens\n* Using Image Prompting: We would add 1 (reference image) x 10 (subject) x 1024 (token/image) = 10,240 tokens (we use 1024 tokens here which is the number of tokens for an image used in Chameleon [1], since the number of tokens is unknown for GPT-4V).\nThis clearly illustrates the significant token efficiency of Yo’LLaVA!\n\n> **less token requirement is a direct consequence of training with the new token, whereas GPT-V is zero-shot**... [omitted]\n\nAs stated in the main paper, GPT-4V + Image Prompting baseline is meant to show accuracy and efficiency tradeoff of a SOTA proprietary multi-image understanding model with zero-shot inference time personalization (Line 257 “Since images convey more information than text, we hypothesize that personalized image prompts represent upper bound for prompting effectiveness.”).\n  \nWe agree that requiring fewer tokens is a direct result of training new tokens. However, the primary goal of this paper is to introduce a new problem setting: Personalizing LMMs. Our work is the first to demonstrate its feasibility: we proposed a new novel approach and showed promising results: Yo’LLaVA is much more token-efficient (17) while achieving similar performance with image prompting (5k~)!\n\nWe anticipate that integrating Yo’LLaVA with GPT-4V could significantly reduce the number of tokens used while retaining performance; however, we could not try this since GPT-4V is a closed-source framework (Line 289-290).\n\n> **The recognition metrics is unclear to me**... [omitted]\n\nWe have 40 subjects, each with 5-10 test images containing the corresponding subject.\nFor each subject, all of its test images serve as positive test images, while test images from the remaining 39 categories serve as negative test images.\n\nDuring testing, we show a photo to the model and ask, “Can you see if <sks> is in this photo? Answer with a single word or phrase.” The ground-truth response is “Yes” for photos containing <sks>, and “No” for others. An extension of Tab. 5 with F1 score is:\n\n| | Ours | LLaVA | LLaVA + Prompt | GPT-4V + Prompt |  |\n|------------------|-------|-------|-------------------|--------------------|-------|\n| Type | Learnable | Text | Human | Text | Human | Image |\n| # tokens  | 16    | 0     | 16 | 1.3k | 1k    | 5k   |\n| F1 | 0.93  | 0.00  | 0.80 | 0.48 | 0.81  | 0.81 | 0.89 | 0.92 |\n\nAs explained in Line 269-274, we purposely report weighted accuracy = 0.5 ∗ accuracy positive + 0.5 ∗ accuracy negative because:  \n* In personalization setting, the ability to recognize the positive class (it is <sks>) and not-sks (it is not <sks>) is equally important\n* The positive/ negative class are heavily imbalanced\n\nWe also report accuracy for each positive and negative class.\n\n> **It appear that GPT-4V can easily go above Yo'LLaVA's performance** ... [omitted]\n\nOur setting involves visual personalization, which typically requires 3-5 images per user, as initially proposed by Textual Inversion/ Dreambooth [2]. In our experiment, we provided 5 images/subject to GPT-4V, which is similar to the number used to train Yo’LLaVA. As shown in Tab. 5, we achieved comparable performance while using significantly fewer tokens (17 vs. 5k). Unfortunately, we cannot extend to an order of magnitude more images (e.g., ~10) as suggested by the reviewer, just to see if GPT-4V will surpass Yo’LLaVA in performance, this is because:\n\n-   As stated in Line 258, comparison to GPT-4V is solely for reference as it’s closed-source framework.\n-   With 5 images, GPT-4V already surpassed Yo’LLaVA in terms of recognition accuracy (0.925 vs. 0.924; Line 287); however, use significantly more tokens (5000+ vs. 17!)\n-   The best setting would be to directly integrate Yo’LLaVA with GPT-4V for both performance and token-efficiency – Unfortunately, we cannot do that as GPT-4V is closed-sourced! (Line 288).\n\nAlso, there is no existing dataset to support this (e.g., dataset for personalization that have 10+ imgs/subject), and we believe this is not a practical setting. If the reviewer is aware of any such dataset, we would much appreciate the reference.\n\n*Reference:*\n\n*[1] Chameleon Team, Chameleon: Mixed-Modal Early-Fusion Foundation Models, arXiv, 2024*\n\n*[2] Ruiz et al, DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation, CVPR, 2023*", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RqkQpeyy5g", "reviewer": "Reviewer_ddkC", "summary": "This paper proposes a new task of adapting a LLaVA model on personal images on specific instances, e.g., a specific pet dog, and answer visual questions about the instance. The authors proposed a finetuning pipeline to learn identity tokens, and retrain the original LLaVA ability while being able accept the new identity tokens. Training and evaluation are done on a small datasets the authors collected. Experiments show the proposed method outperform LLaVA and GPT4o baselines, as well as an concurrent work.", "questions": "Overall this paper works on an important new problem with a valid method. The method and evaluation all makes full sense to me. I only have clarification or discussion questions. I believe a large number of audience would be interested in this topic.", "limitations": "Yes.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The task of personalized VLM is important and interesting. This paper setup the task and (small scale) datasets, and provide a valid first attempt to this task, with good results.\n\n- The method of representing object identity by a few visual tokens, and train them on recognition tasks makes full sense to me. The method description is clear, and the effectiveness is shown in both the qualitative example and ablation studies.\n\n- Quantitative results are strong and sufficient. The authors ablated the necessary design choices, and compared to concurrent works with similar goal on their benchmark.\n\n- It is nice that the authors also verified the original LLaVA abilities are retrained (Table 8).", "weaknesses": "- From reading the paper, it is unclear to me if we need to train a separate model for each instance, or we can train a single model on N instances together using N*16 learned identity tokens. If it is the later, does the model has a number of objects limit that it can learn together?\n\n- One limitation might be we always need to finetune the model to adapt to new instances (rather than in-context learning). However the finetuning cost seems to be low enough (1 A6000 GPU). More discussion on the finetuning cost (e.g., wall clock time) in the rebuttal is appreciated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **From reading the paper, it is unclear to me if we need to train a separate model for each instance, or we can train a single model on N instances together using N*16 learned identity tokens. If it is the later, does the model has a number of objects limit that it can learn together?**\n\nAs stated in Sec. 3.1 (Lines 152-157), we train (1) the newly added tokens (e.g., $<sks>$) and (2) the final classifier head matrix associated with these newly added tokens.\nDuring training, we train for each subject separately; then at test time, we can easily adapt the model to handle N subjects by concatenating their personalized prompts together.\n\nFor example, in Table 2 (left), we have two personalized subjects: a person named $<T>$ (subject 1) and a dog named $<bo>$ (subject 2). The personalized prompt would be: “$<T> is <token 1_1>...<token1_{16}>. <bo> is <token2_1>...<token2_{16}>.$”\n\nExamples with 3 personalized subjects can be found in the Rebuttal PDF file (Fig. 2).\n\n> **More discussion on the finetuning cost (e.g., wall clock time) in the rebuttal is appreciated.**\n\nThe total time for learning a new subject with LLaVA-1.5-13B would be around 40 minutes.\nFor a new subject with 3-5 images, the time required to create the training data is roughly 3 minutes (1-2 minutes for conversation creation and 1 minute for hard negative retrieval). The optimization process, including training new tokens, takes approximately 34 minutes for 15 epochs (measured on an A6000 machine).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xRW063lehJ", "reviewer": "Reviewer_wZcX", "summary": "This paper studies personalization for lage multimodal models (LMMs). More specifically, how can a model understand that \"my dog\" refers to the dog of the user. The proposed model, Yo'LLaVA, learns latent tokens to encode personalized subjects using a handful of images of for each personalized concept. \n\nRegarding the methodology, the assumption is to have access of a small number of images (O(10)) of a person or a subject without any text label. This setting is realistic. In order to learn personalized concepts, the authors use one special token for the personalized concept (e.g., <sks>) and then k latent tokenss are used to capture relevant visual details. The personalized concept token is a new token, and the latent tokens soft ones.\n\nDuring training, all parameters are frozen except the k+1 new tokens and the final classifier head. To capture fine-grained details in the latent tokens, the authors propose to use hard negative minig to gather negative examples that are visually similar but not identical to the personalized concept. More specifically, they generate conversational training data triplets (image, question, answer) using a template-based approach (10 manually written conversations). Negative samples are retrieved from LAION with the top m images with the highest CLIP image embedding similarity. Overall, they are n (training images) x m (hard negative images) + 100 (easy images) generated samples.\n\nIn the experiments, <10 images and 16 latent tokens are used to learn the personalization of a subject/person. A new dataset of 40 subjects is collected and used. The authors show the effectiveness of their approach on LLaVA-1.5-13B. The tasks are recognition ability and question answering. The proposed method significantly outperforms the baseline. I really appreciate that there is a comparison with a concurrent work MyVLM; this is definitely a plus that highlights the effectiveness of the proposed method. Finally, the ablation studies support the modeling decision. To close the loop, the authors could conduct a small human evaluation.\n\nOverall, this paper is very well written, novel, and results are strong.", "questions": "Could you elaborate more regarding the computational time needed for your method?", "limitations": "Yes", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "+ Novelty\n+ Strong performance\n+ Comparison with concurrent work", "weaknesses": "- While I acknowledge that personalized datasets don't exist, it would be nice to have datasets in other domains.\n- Lack of a small human evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **While I acknowledge that personalized datasets don't exist, it would be nice to have datasets in other domains.**\n\nThanks for your suggestion. Our current dataset includes humans, pets, and objects for personalization. We anticipate that future research will introduce more datasets in other domains!\n\n> **Could you elaborate more regarding the computational time needed for your method?**\n\nThe total time for learning a new subject with LLaVA-1.5-13B would be around 40 minutes.\nFor a new subject with 3-5 images, the time required to create the training data is roughly 3 minutes (1-2 minutes for conversation creation and 1 minute for hard negative retrieval). The optimization process, including training new tokens, takes approximately 34 minutes for 15 epochs (measured on an A6000 machine).\n\n> **Lack of small human evaluation.**\n\nThank you for your suggestions! Due to the limited time available during the rebuttal phase, unfortunately, we cannot include a human evaluation, as it would involve privacy concerns and obtaining permissions (e.g., from a person named <A>). We will continue to investigate this and report if it becomes feasible.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We introduce the *novel task of personalizing LMMs* and present *Yo'LLaVA* -- a framework to embed personalized subjects (e.g., your pet) into a comprehensible prompt for LLaVA.

We are encouraged by positive feedback from reviewers on our paper!
- **Originality**: “novel” (#PZy9, #wZcX), “good” (#VkWL), “important and interesting” (#ddkC)
- **Significance**: “addresses a significant gap in current LMM capabilities” (#PZy9), “an important trait that currently LMMs are missing” (#VkWL), “a valid first attempt to this task” (#ddkC).
- **Technicality**: “innovative and well-justified” (#PZy9), “easily able to learn new personalized concepts” (#VkWL), “makes full sense” (#ddkC)
- **Clarity**: All reviewers think the paper presentation is *“excellent”*!

---

We thank reviewers for their time and effort in reviewing our paper.
Answers to individual reviewers are addressed below each review.
Please let us know if you have any additional questions or concerns!

Meta-review: The reviewers find the problem of personalization of VLMs being studied in this paper to be relevant and important, the proposed method to be innovative, simple and well justified, and the experimental results to be strong. The reviewers had raised some concerns, but the rebuttal successfully addressed most of them and all reviewers recommend acceptance. The authors are recommended to improve the final paper version by following the reviewer recommendations.
Decision: Accept (poster)