IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model
Authors: Yatai Ji
Shilong Zhang
Jie Wu
Peize Sun
Weifeng Chen
Xuefeng Xiao
Sidi Yang
Yujiu Yang
Ping Luo
Venue: ICLR 2025 Poster (2025)
Keywords: large vision-language model; ID recognition
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose an ID-aware LVLM, IDA-VLM to recognize instance IDs across diverse scenes, towards understanding complex visual inputs, such as movies. We construct a new benchmark, MM-ID, to examine LVLMs on instance IDs recognition.

Abstract: The rapid advancement of Large Vision-Language models (LVLMs) has demonstrated a spectrum of emergent capabilities. Nevertheless, current models only focus on the visual content of a single scenario, while their ability to associate instances across different scenes has not yet been explored, which is essential for understanding complex visual content, such as movies with multiple characters and intricate plots. Towards movie understanding, a critical initial step for LVLMs is to unleash the potential of character identities memory and recognition across multiple visual scenarios. To achieve the goal, we propose visual instruction tuning with ID reference and develop an ID-Aware Large Vision-Language Model, IDA-VLM. Furthermore, our research introduces a novel benchmark MM-ID, to examine LVLMs on instance IDs memory and recognition across four dimensions: matching, location, question-answering, and captioning. Our findings highlight the limitations of existing LVLMs in recognizing and associating instance identities with ID reference. This paper paves the way for future artificial intelligence systems to possess multi-identity visual inputs, thereby facilitating the comprehension of complex visual narratives like movies.

Published as a conference paper at ICLR 2025
IDA VLM: TOWARDS MOVIE UNDERSTANDING VIA ID AWARE LARGE VISION-LANGUAGE MODEL
Yatai Ji Shilong Zhang', Jie Wu', Peize Sun', Weifeng Chen', Xuefeng Xiao², Sidi Yang", Yujiu Yang" Ping Luo The University of Hong Kong, Tsinghua University, BByteDance
ABSTRACT
The rapid advancement of Large Aision-Language models (LVLMs) has demon- strated spectrum of emergent capabilities. Nevertheless, current models only focus on the visual content of single scenario, while their ability to associate instances across different scenes has not yet been explored, which is essential for understanding complex visual content, such movies with multiple characters and intricate plots. Towards movie understanding, a critical initial step for LVLMs is unleash the potential of character identities memory and recognition across multiple visual scenarios. To achieve the goal, we propose visual instruction tun- ing with ID reference and develop an ID-Aware Large Vision- LangunnL Model, IDA-VLM. Furthermore, our research introduces novel benchmark MM-ID, to examine LVLMS on instance IDs memory and recognition across four dimensions: matching, location, the limitations of existing LVLMs in recognizing and associating instance identi- ties with ID reference. This paper paves the way for future artificial intelligence systems to possess multi-identity visual inputs, thereby facilitating the compre- hension of complex visual narratives like movies.
and captioning. Our findings highlight
INTRODUCTION
Our real world contains a wide variety of information, such as texts, images, sounds, etc. Towards multimodal comprehension (Guo et al. [2019] Lu et al., 2023). inspired by the success of Large
Language Models (LLMs) [OpenAl]|00) 2023a vicuna 2023 Touvron et al. 2023 Meta 2024 Google 2023a), there is surging interest in Large
Models (LVLMs), such as LLa VA (Liu al.) 2023a). Otter (Li et al.] 2023a), GPT-4V (OpenAl 2023b), Gemini (Google 2023b), and others. In the quest for Artificial General Intelligence (AGI), LVL.Ms serve as pivotal milestones, enhancing machines' capabilities in multimodal perception, reasoning, and knowledge.
Typical LVLMs incorporate visual encoders with LLMs, such as LLaVA (Liu et al.) 2023a), MiniGPT4 (Zhu et al.) (002). These models project an image's visual features into the embed- ding space of language models and employ visual instruction tuning, allowing users to complete a variety of visual tasks through language instructions, as depicted in Figure Ma). However, pure language interaction makes VLMs hard to receive precise Region-of Sfggion-ofline In-enterest references from users, thus hindering their capability to focus on specific regions of an image. To address this issue, a series of works (Chen et al. 2023b 7h2nng
al. 2023 Cai al. 2023 Chen al.) 2023a] /hha al., 20236] Rasheed et al. 2023) attempts to allow adding Rol references to language instructions and then use region-level data to align these RoI references the image. As shown in Figure these excellent works usually support using position coordinates or visual prompts as the Rol reference, which provides a more flexible interactive experience within specific areas.
Previous LVLMs have demonstrated versatile capabilities for visual understanding from the image level to ofgionn-ofIIteeeeee However, these models can only process the visual input of a single scenario, and their ability to associate instances across different scenes has not yet been explored, which requires the model to memorize the instance identity and recognize in different scenes. This ability, which we call 'ID awareness", is a fundamental human visual competency essential for
Corresponding author.

---

Published as a conference paper at ICLR 2025
men is sitting at table on the left
The man at cregionI> is sitting at the table on the left
Chorlie
David is sittngg at the table
David Is
UM
LUM
Please describe what the people in the picture are doing
Please describe what people picture doing?
Please describe what the people the picture are
(a) Visual Instruction Tuning
(b) Visual Instruction Tuning with Ro! Reference
[c] Visual Instruction Tuning ID Reference
Comparison of different Visual Instruction Tuning Formats. For visual instruction tuning with ID) reference, we arrange the names and images of each character as references. The model should be able to recognize the correct character identity and then answer the user's instruc- tions.
Figure 1:
comprehending complex multi-identity visual inputs, such as movies and animations. In the context of movie understanding, viewers should remember the name and appearance of each character, then recognize them across disparate scenes correctly, linking instances across diverse images for accurate narrative interpretation.
To achieve ID awareness in LVLMs like humans, as shown in Figure [c), we propose visual in- struction tuning with ID reference, which allows users utilize the name and reference image of the character (ID image) to define the identity in the prompt, and raise questions about test im- ages. However, there no off- ffff-teeshher visual tuning data with ID reference, we meticulously craft a dual-stage instruction tuning dataset based existing datasets. The initial phase leverages annotations datasets such as VCR (Zellers et al., 2019), Flickr30k (Plummer et al. 2017), and RefCOCO (Kazemzadeh et al., 2014) with our data configuration strategies, to tutor the model on associating the instances of ID images with test images, where ID references are extracted from the test image. The subsequent phase utilizes MovieNet (Huang et al. 2020) generate Q&A and caption instruction tuning data with GPT-4V (OpenAl (023b). The second stage data con- tains ID references of higher quality, which is more similar to real movie understanding. These tuning data unleash the potential of LVLMs to memorize and recognize instance identities in dif- ferent scenes. Based the visual instruction tuning with ID reference, we develop ID-aware LVLM, called IDA-VLM, suitable for multi-identity visual comprehension. Our model learns the fine-grained identity information from ID references and generalizes recognize characters from different scenes of test images. We also introduce specialized component, termed ID-Former, to enhance the model's 's capability recognizing character identities.
To measure ID recognition capability for movie understanding, we propose new benchmark, namely MM-ID. Our MM-ID aims at challenging LVLMs to remember and recognize instance iden- tities, further understanding complex visual scenes. MM-ID dissects LVLMs' competency across four progressively complex levels: matching, location, question-answeringg and captioning. MM- ID comprises collection of 585 diverse testing samples, whose instance identities source from the characters of movies, the roles of animations, and individualized animals and objects. We test both open-source models and closed-source APIs, which exposes their apparent deficiencies in identity memory and recognition, and the results are unsatisfactory even for GPT-4V. The evaluation results only reveal the current limitations of LVLMs instance recognition with ID reference but also highlight our benchmark's significance. In contrast, -DA-LLM achieves the best performance on MM-ID, which demonstrates the effectiveness of our instruction tuning strategies and model design.
In nutshell, our contributions are summarized as follows:
This paper is the first attempt to investigate ID awareness of LVLMs for complex multi- identity visual recognition and comprehension, which is critical challenge towards movie understanding. We propose visual instruction tuning with reference and construct cor- responding tuning datasets.
2

---

Published as a conference paper at ICLR 2025
We develop an ID-aware LVLM, IDA-VLM recognize character identities and under- stand visual content in an end-to-end manner. We adopt a dtal-stage fine-tuning method to unleash the potential of LVLMs in identity memory and recognition across diverse scenes.
We propose MM-ID, a novel benchmark to examine LVLMs on identity memory and recog- nition. We evaluate representative LVLMs on our benchmark, uncovering the limitations of existing LVLMS in contextual identity recognition. Furthermore, our IDA-VLM can serve as reference and inspirational baseline model on MM-ID.
RELATED WORK
2.1 LARGE VISION- MODELS
Conventional multimodal models consist of uni-modal encoders and cross-modal fusion encoders. Relying on dision-language pre-training (Tan & Bansal, 2019| Lu et al.) (2199] Dou et al., [022] Wang et al. 2021 et al., 2023b), they have shown an impressive mooss semantic alignment ability (Ji et al., 2023a Tu et al. )023). which brings substantial advances on various downstream tasks (Goyal et al. 2019 Plummer et al., 2017: Lin et al., 2014). However, due to the limitations of model size and training data scale, the performance of vision-language pre-training models is unsatisfied in open-ended scenarios.
Nowadays, Large language model (LLM) has exhibited remarkable abilities to understand. reason, and generate texts. Large Vision-Language Model (LVLM) (Chen et al.) [202cc] Dai et al. 2023 Dong al. 2024) incorporates visual encoder and LLM, aligning visual features to the embedding space of L.M. Leveraging strong generalization and emergent capability of LLM, LVLM realizes free multimodal interactions with human. Flamingo (Alayrac et al. 2022) pioneering work on extending LLMS to vision-language pretraining by inserting additional cross-attention layers for vi- sual input. BLIP-2 (Li et al., 2023d) proposes Q-former to map the visual features to the hidden space of language models. To date, various works have shown encouraging progress with instruc- tion tuning, including MiniGPT-4 (Zhu al., 2023), LL (Liu et al., 2023a), Otter (Li et al.) 2023a), which demonstrate impressive results natural instructiionfoollwwinn and visual reasoning capabilities. The perceptual capabilities of LVL.Ms are evolving towards fine- enn-grained understanding. Numerous models (Zhang et al. [203 Liu ct al. 2023a Chen et al. 2023b Wang et al. 2023) focus on region-leve -lveel understanding, using visual instruction tuming with Rol reference, so that users could ask questions about specific instances within the content. Moreover, there are some LVLMs equiped
with Stable Diffusion, which can produce multimodal outputs (Koh et al. 2023 Ge et al. 2023] Sun al. 2023 Li al. 2023ch. LVLMs hold the potential perform wider array of functions and to understand more intricate visual information. For instance, movie is considered as one of the most intricate mediums for conveying visual information. This paper researches on recognizing characters across disparate scenes with ID reference, towards movie content understanding.
2.2 MULTIMODAL BENCHMARK
Previous evaluation for multimodal models measures some specific abilities, such as VQAv2 Goyal et al., 2019). GQA (Hudson & Manning, 2019) for visual question answering, Ref- COCO (Kazemzadeh et al.) 2014) for visual grounding, Visual7W (Zhu et al. 2016) for PointQA, VCR (Zellers et al.) 2019) for commonsense reasoning in an image, ChartMimic (Shi et al. 2024) for chart understanding. Recently, there is a surging interest in developing comprehensive bench- marks for evaluating LVLMs. MMBench (Liu et al., 20235) consists of multiple-choice questions and introduces a CircularEval strategy for evaluation framework. LLaVA-Bench (Liu al. 2023a) employs GPT-4 to assess responses from both GPT-4 and the model under evaluation, then pro- vides scores and explanations for the answers. MM-Vet (Yu et al., (023) assesses LVL.Ms across six fundamental visual- linguistic capabilities, utilizing GPT-4-based evaluator for open-ended re- sponses. MME (Fu et al.) 2023) tests for perception and cognition competencies across 14 distinct subtasks, which requires models to provide simple 'yes' 'no' answers. SEED-Bench (Li et al.) 20236) covers 12 evaluation dimensions and adopts an answer ranking strategy to evaluate LVLMs via multiple-choice questions. POPE ( al. 2023e) is a dedicated benchmark for assessing object hallucination. Existing benchmarks evaluate multifaceted performance of LVLMs, yet they predom- inantly focus visual scenarios that lack character identities and intricate plots.
3

---

Published as a conference paper at ICLR 2025
User Instruction
Julka is <ID-Imgos, Jack is <ID-img1>, Give description the following images: <Tett-mmgtb-Teest/mm-- Prest-mm2><<fsstlling
Images
ID-Former
Test Images
Tokenization Embedding
Tobedding
Reploce
Cina
chireggs
ID-Former
Chees
LLM
Ferkirt
Goerie
cheblegh
Jack and Julla seated side by side in vehicle, looking out at desolate landscape. in a subsequent scane,
Figure 2: IDA-VLM an end-to-end vision- lssounlanguage model for processing instructions that con- tain references to specific instances. We introduce II) reference as using correlating character reference image and corresponding name to characterize an identity, exemplified as Julia is <ID>- Im. During tokenization and conversion to embeddings, the embedding of <ID-Img(i} and <Test-Img in the instruction are replaced with the ID and Test image embeddings respectively. simple yet effective image feature projector termed ID-Former proposed to enhance the ID identification ability. As the output in the figure, IDA-VLM can memorize these character IDs, rec- ognize them in test images, and respond to user instructions with the correct ID references.
METHOD
Movie understanding represents a considerable departure from conventional video understanding, which typically limits to a single scenario or activity. To understand complex visual input, for example, movie or animation, the model need memorize and recognize character identities, linking characters in distinct scenes, with each scene represented by an image in our setting. As shown in Figure we simplify movie understanding as given ID images of certain characters and keyframes from the movie or animation served as test images, the model completes instruction tasks. The ID images, test images, and instruction text are sent to the model together for inference an interleaved format of images and text. The model need to memorize instance identities in ID references and recognize them in test images for providing responses.
3.1 MODEL ARCHITECTURE
In this paper, adopt Qwen-VL-Chat VL-Chat (Bai al. 2023) as baseline model. In order to adapt the model to the task of instance ID recognition, we employ a dual-stage visual instruction tuning with ID reference. The architecture of IDA-VLM comprises three components: visual encoder, ID-Former consisting of cross-attention mechanisms, and a subsequent large language model.
As shown in Figure 2 the ID-Former is designed to project visual features into the input semantic space of LLM and contribute to recognizing instance identities. This achieved by two cross- attention modules. The first one interacts learnable queries with the visual features through cross- attention, effectively compressing the visual semantics into a shorter, fixed-length feature encoding. The second cross-attention utilizes queries of ID images to modulate test image embeddings, acti- vating identity information of test images.
3.2 FIRST-STAGE TUNING
In the first phase, we harness the off-the-shelf annotations available in existing datasets along with proposed data configuration strategies, reducing the cost for extra annotations about ID recog- nition. Specifically, we utilize the public datasets containing instance spatial information, including Visual Commonsense Reasoning (VCR) (Zellers al. 2019), RefCOCO (Kazemzadeh et al.) 2014). and Flickr30k (Plummer et al., 2017) datasets, to construct visual instruction tuning data with ID reference, which contains three instruction tasks: question -anseiinnnn (Q&A), location, captioning.
VCR dataset consists of questions that require commonsense reasoning to answer, as well as choos- ing the reasons. These questions ask about particular people or objects, which have location anno-
4

---

Published as a conference paper at ICLR 2025
Grop. augmentction
First-stage
Noming
VCR
V. ter [otsen)) letteg: i (poeen)) bohadppesoc]]
How bile Seiling
anopped
RefCoco
coa ebbccaatc wilh
Ghandee 4377
reference
Flickr30k
Ciptae::
Auge hakkng and tadgpoodd
babkiag boudh dhe bacigroana voncees
Captax
Gecond-stage
SARI PLS, TRX SOROO (MN 24,
Katn CDB, 754, 500)
Kan (101,
Katn 128, 224,
Maxin 1966 100, 1860, (01)
Mone (147
Moria (501, (001 171
You helyfu and precise
for continces images accanding the fallow ng rulea User give the mage sine (width, height) some inapes each mace, user same charscter pestione You sheuld desebe the chaned the characters storiss and intercctoos Fenoghoor
deseription
Kain und Mone maigaate (hrough comn due Filled stuckens. They then ergage hand and justell enplon ournclling Mome Mome appsors focused on cais to milared flowerbed, appreciening Hewarre
inages gegancce
GPT4-Vision
tuning ID reference
Templates
Figure First-stage and second-stage stage instruction tuning data construction pipelines.
tations. crop sub-images of the individuals and assign them identifiable labels, such as person names. By replacing the people in the questions and answers with these labels, gain new set of QA data that necessitates ID recognition. RefCOCO (and variants RefCOCO+ and RefCOCOg) is dataset targeted visual localization based on descriptions. For the items to be localized, create ID images after cropping them from the original images to perform localization tasks. Flickr30k, a dataset comprising image descriptions, includes coordinates for the people objects mentioned in the descriptions. We extract the areas associated with individuals to produce ID images and provide names to generate corresponding descriptions. Additionally, and cleaning operations, such as filtering out samples with only one person or those with sub-images of inappropriate sizes. The annotation pipeline shown the upper part of Figure
conduct some data augmentation
The essential purpose of this stage is to train the model on its ability to build relationships between multiple images, using the images to identify and accurately locate or describe objects within test images. However, the instance II) images are cropped from the original images, which reduces the difficulty of recognition.
3.3
SECOND-STAGE TUNING
The second-stage fine-tuning data is based on the MovieNet (Huang et al., 2020) dataset. The original annotations in the MovieNet dataset encompass the names the characters present each movie shot and their coordinate location information. In the MovieNet, we select pictures that contain only single character to serve as ID images, while those containing multiple characters used test images. This approach allows us to naturally develop dataset for location task. Moreover, we compile ID images featuring the same character to construct data for matching task.
For Q&A and caption tasks, adopt GPT-4V convert annotations from the MovieNet dataset into the question format that refers to specific roles. Specifically, as illustrated the lower part of Figure feed test images along with their character location information into GPT-4V, and encourage the model to generate captions or question-answer pairs via prompt engineering. The detailed prompts used for generating descriptions question-answer pairs are shown Appendix
5

---

Published as a conference paper at ICLR 2025
Pacan right intte roles Whic daner?
Images
oight
Locaion
Quaritc
aggi cingla, Mayyuge
Made culoil
Craangll (/s lang cingl,
Heage giggt Werls hobling soneel?
Cradia Cavi
the dingle
rk
Chafle stancigg and spowe to venation with Derld who seeted The outsowr ring perora baciground
Ciption
Quaticc
teitob Juck drgtt deccptione bllwing www.----ggg-i---eeee be2---mmh--mmm--din---
Mode puluu:
sapoaes ineraciing somating samiec
soans varte aharding not
Fredly, Jich
witle Jule merain
Figure 4: Data samples of MM-ID. Each sample consists of ID images of each character, and test images containing multiple characters. Our model can memorize the identity information ID reference and generalizes to recognize characters from different scenes.
Finally, we integrate images, test images and results GPT-4V into conversation templates, producing second-stage instruction tuning data.
After completing data construction process, we train our model with next-token prediction loss in two stages. The first stage tuning data contains approximately 80,000 samples, while the second one comprises around 60,000 samples. Moreover, to ensure that the model would not forget its previously learned knowledge during ID recognition training, integrate instruction tuning data from LLaVA and ShareGPT4V with the aforementioned data, training them simultaneously.
MM-ID
4.1 PROBLEM DEFINITION
To measure ID recognition capability of models, we propose new benchmark, MM-ID. We appraise the capability of models to recognize IDs across four incrementally complex levels. The first sub- task investigate is matching: given an instance image. which could feature a person, an animal, or a building, the model must choose an image from four options that contains the same instance. The second sub-task involves localizing the instance within a test image based on the ID image, and providing coordinates of the bounding box, with the challenge coming from numerous similar distractor objects or individuals present in the test image. The third and fourth sub-tasks -uetassin---stnonnn- answering and caption generation-requin the model to recoginize the identities in the test images, then either respond to related questions or produce a comprehensive description. These represents the most difficult level of tasks, as the model should not only recognize each instance's identity in the image but also provide appropriate answers based on information related to their names, including states, actions, locations, and more. The samples of sub-tasks shown in Figure

---

Published as a conference paper at ICLR 2025
4.2 DATA ANNOTATION AND STATISTICS
In the construction of MM-ID benchmark, we initially collect images that meet the specified condi- tions. To evaluate the model's capability of recognizing instance-level information, the images used for testing need to contain multiple characters. Furthermore, each character requires distinctiveness, such as different actions, clothing attributes, etc., facilitate the design of questions. Our data sources primarily encompass three types: shot images of movie scenes from the MovieNet dataset, animated images with download links collected from the internet, and pictures of buildings, vehi- cles, and animals gathered from object re-identification (Re-ID) datasets (Ye et al. [024] Jiao al.)
2023). In each example, there exists ID images corresponding to each character and one or more test images for inquiry.
Following preparing images of MM-ID, we annotate questions and standard answers manually. For description and localization sub-tasks, answering task, annotators pose questions regarding specific characters' actions, attributes, clothing colors, positions, relationships, emotions, etc., following detailed annotation documents and guid- ance examples. The question design ensures that the model can only provide correct answer when accurately recognizes the characters. After question annotation, lance in refinement and balancing, increasing the diversity of expression and grammatical structure. Finally, the correct answers labeled manually by other four annotators, providing accurate and detailed descriptions for caption task and correct answers for Q&A task. Everyone participates in discussions and makes peer reviews for each other. The instructions for question annotations and quantitative evaluation of inter-annotator agreement are shown in Appendix
GPT-4 to generate question templates. In question-
request other individuals' assis-
Sub-task
Ques Num
Img Num
Source
Proportion (50
Matching
67
335
MovieNet, Re-ID datasets
Location
123
360
MovieNet, animations
Q&A
154
507
MovieNet, animations
Caption
241
1110
MovieNet, animations
Figure 5: MM-ID data statistics. (a) Sample number and data sources of four sub-tasks. (b) The proportion of each perspective of Q&A.
Totally, we collect and annotate 585 samples. The data composition is shown Figure 5 (a). The "Q&A" and "Caption" sub-tasks for scoring, it is costly to have too many evaluation samples and we select the most representative samples for MM-ID. As shown in Figure 5 (b), Questions of "Q&A" task comprehensively cov- ers different perspectives of characters (actions, states, attributes, positions, emotions), which can sufficiently evaluate the recognition capability of LVLMs.
main parts of MM-ID benchmark. Because
GPT-4
4.3 EVALUATION STRATEGY
Quantitative evaluation for open-domain LVLMs has always been challenging. Our MM-ID incor- porates four sub-tasks, each presenting its unique answer format. For matching task, which resem- bles multiple-choice questions, accuracy is used metric. In localization task, compare the predicted bounding box to the actual bounding box, computing the Intersection over Union (loU) metric measure aceuracy. When IoU exceeds threshold of 0.5, we consider the model recognizes instance identity accurately.
For question-answerte and caption generation tasks, due the open-ended nature of the generated responses, it is challenging to employ rule-based evaluations. Hence, inspired by MM-Vet and LLaVA-Bench, GPT-4 used to correct answers, we design prompts to guide GPT-4 to focus on correspondence of character names and their states actions, scoring the accuracy of characters' descriptions within model predictions.
the results. Under the condition of provided questions and
version: gpt-4-1 106-preview

---

Published as a conference paper at ICLR 2025
Table I: The comparison results MM-ID. Notably, The scores of 'Q&A' and 'Caption' eval- uated with GPT-4 absolute scoring strategy. indicates the model can't an't complete corresponding instruction.
Model
Matching
Location
Q&A
Caption
Open-source Models
MMICL Zhao et al.
2023a
3.53
3.18
SEED (G et al. 2023)
3.19
3.58
LLa VIsion LL el al.
2024
3.91
2.86
Qwen-VL-Chal (aa al. 2023)
0.504
3.63
2.65
IntenLM-XCCnposerz Dong et al.
2024
0.106
3.44
2.93
Closed-source APIs
Qwen-VL-Plus (Bai et al.
2023
0.313
0.187
3.87
3.79
Qwen- VL-Max (Bal et al.
2023
0,224
0.301
4.64
4.23
Gemini-pro (Google) 2023b)
0.687
0.081
4.97
4.04
GPT-4V (OpenAl 2023b)
0.627
0.244
4.77
4.67
IDA-VLM
0.716
0.821
5.71
5.15
It is found that outputs of GPT-4 still exist variance, utilize GPT-4 score the predictions by 5 times and report average scores.
We both absolute and relative scoring strategies. Absolute scoring rates model's prediction di- rectly against the correct answer on ten-point scale, while relative scoring pits two models' results against each other, providing more immediate comparative insights into the models' performance with a fraction. The prompts used for GPT-4 scoring are shown in Appendix [ Besides using GPT-4 scoring, we also employ conventional caption metries like METEOR and CTDEr.
EXPERIMENTS
5.1 QUANTITATIVE RESULTS
We use Qwen-VL.-Chal for model initialization, which 9.6B parameters. During IDA-VLLM training. we set learning rate le-5 for the first stage and 5e-6 for the second stage. The model is trained for epochs both the first and second stages. The batch size with gradient accumulation set to 128. The visual encoder is fixed, while ID-Former and LLM are fine-tuned.
We compare IDA- VLM with other open-source LVLMs (Zhao et al, 2023a Ge et al. 2023 Bai et al. 2023 Dong et al. 2024) and closed-source APIs (Bai et al.) 2023 Google 2023b OpenAl 2023b) on MM-ID. To meet the requirement of ID recognition, the models selected for testing should support multiple images input. Because they are not fine-tuned for this task, the prompt crucial for guiding models on ID awareness. We design appropriate prompts to guide them to respond according character names. The details can be found Appendix D
As illustrated in Table I our model achieves the best performance on MM-ID. According to the quantitative results, Gemini-pro exhibits best performance on matching and Q&A sub-tasks among previous models. In the location sub-task, Qwen- VL-Chat gains the highest score, even higher than other closed-source APIs. As for the caption sub-task. GPT-4V surpasses other models. Overall, our model outperforms previous I.L.Ms by significant margin on four sub-tasks. The results demonstrate visual instruction tuning with ID reference unleashes the potential of LVLMM on ID awareness, which can not be realized only by prompt engineering.
As shown Table 2 we calculate relative scores using GPT-4 to provide further demonstration on the 'Q&A' and "Caption' sub-tasks. We compare the predictions from various closed-source APIs against those from our model respectively. All relative scores are below 1, signifying that model outperforms its counterparts. Among the closed-source APIs evaluated, GPT-4V exhibits comparatively superior results. In Table 3 our model also gains best performance on traditional metrics of caption task.
8

---

Published as a conference paper at ICLR 2025
Using relative scoring strategy to com- pare previous LVLMs with our model. Each re- sult is the ratio of the first model's ' score to that the second model's, after feeding the predictions of two models into GPT-4 for comparison.
Table 2:
Table 3: METEOR and CIDEr metries evaluation on 'Caption' sub-task. Both metrics take into account the semantic similarity of the captions.
Model
METEOR
CIDEr
Model
Q&A
Caption
Qwen- Qwen-VL-Chal VL-Chal
0.052
0.031
GPT-4V IDA-VLM
0,925
0.793
-L-Puu
0.119
0.140
Gemini-pro/ / IDA-VLM
0.892
0.704
wen- wen-VLLMMas VL-Max
0.134
0.083
IDA-VLM
0.765
0.396
Gemini-pro
0.121
0.185
Qwen- VL-Plus IDA-VLM,
0.796
0.544
GPT-4V
0.144
0.078
Qwen- VL-Max IDA-VLM
0.918
0.636
IDA-VLM
0.191
0.515
Table 4: Performance of VLM on standard benchmarks. "w/o LLaVA data" means finetuning
with only constructed ID reference datasets.
MME
SEED-Bench
Model
Perception
Cogmition
Image
Video
Qwen- VL-Chat IDA-VLM (w/o LLaVA data) IDA-VLM,
1441.4
387.5
0.650
0.397
1167.5
349.6
0.441
0.311
1484.6
383.6
0.647
0.412
To assess the basic multimodal capabilities, we also evaluate IDA-VLM with the baseline model VL-Chat on standard benchmarks. As shown in Table the perception metric MME gets improved, which due to the ID-level recognition learning boosts fine-grained perception ability. On SEED-Bench ing ability. Comparing the last two rows, the inclusion of LLaVA and ShareGPT4V data greatly contributes preserving the original capabilities.
Qwen-VL-Chat
training samples containing multiple test images enhance video understand-
5.2 QUALITATIVE COMPARISON
As shown in Figure 6 qualitatively compare the results of IDA-VLM with other models. In the first case, GPT-4V does not exhibit accurate localization ability, and Qwen-VI.-Cha mistakenly recognizes Leizi as another similar person. In the second case, the model should determine which person walks in front and match the person with the given characters. Our model gives the right answer 'Ariadne' while GPT-4V and Gemini-pro recognizes inaccurately. In the last case, models are asked to provide description for a movie scene with three characters. GPT-4V describes that David and Elise sit table, ignoring Charlie standing. Gemini-pro only gives a generic caption for the test image without recognizing characters' identities. In contrast, our model gives the most reasonable description the scene. present more qualitative examples in Appendix B
5.3 ABLATION STUDIES
Effect of Dual-stage instruction tuning. We report the separate effect of each instruction tuning stage in Table 5 When the second stage of tuning is removed, there is significant drop in per- formance, suggesting that the second stage has more substantial impact than the first. During the second stage, the ID images for model learning. When the first and second stages of tuning are combined, our model achieves its optimal performance. It is worth noting that the accuracy of 'Matching' gets lower when adding the first stage tuning, because the first stage tuning data can't improve matching ability.
independent of the test images, which enhances the quality of data
Table 5: The ablation study about separate stage instruction tuning."
Model
Matching
Location
Q&A
Caption
w/o first stage tuning w/o second stage tuning
0.746
0.813
5.43
5.08
0.715
4.08
3.86
Ours
0.716
0.821
5.80
5.19

---

Published as a conference paper at ICLR 2025
Instruction: Leizi <ingOs Melgin Heizi Chuchun the image:
Idantify Leizi groundingg
<inggo
Cyven VL-Chat
Cluis
Gobb simg@r Ariadne <ingl, the image <ing2), walking in front?
Single
Gepiti Golo
ok
Instruction: Chorlie cimg@s, <iggts, Elise is <img2s the image <ing3r, description.
<ingl>
<ing!
<img2>
<ing5
SPT-40
Don
with Elnc baned preeded.
patentialy corveration
ano meal legether.
fie The wearing and fi. The wonon wearing Mackdress dreds. They boking of the mas the dilt.
by oppaare
facing They dring pornens
We present visualizations of selected samples from MM-ID, corresponding to location, Q&& and caption sub-tasks. We showcase results of some representative models and ours (IDA- VLM). The detailed prompts for other models to complete instructions are shown Appendix D
Figure 6:
Table 6: The ablation study about ID-Former.
Model
Matching
Location
Q&A
Caption
w/o ID-Former
0.701
0.803
5.44
5.13
Ours
0.716
0.821
5.71
5.15
Effeet of ID-Former. We use ID-Former project visual features of ID images and test images to the semantic space of LLM. As depicted in Table 0 substituting the ID-Former with a standard query former leads reduction in all sub-task metrics. The ID-Former enhances the interaction between features of ID images and test images, thereby activating ID information within the test images.
The ablation study for the mixing rate of LLaVA and ShareGPT4V instruction tuning data are de- picted in Appendix
CONCLUSION
In this paper, focus the capability to recognize and link instances across various scenes, which is significant for understanding complex visual narratives, such as movies. We propose visual instruction tuning with ID reference, which unleashes the potential of LVLM in ID recognition and develop an ID-aware LVLM, IDA.VLM. The model memorizes the name and appearance of each character in ID reference, then recognizes them across disparate scenes correctly, and understands visual narrative in an end-to-end manner. To thoroughly assess ID awareness of LVLMs, novel benchmark named MM-ID has been introduced, which consists of four sub-tasks. Our model can serve as reference and inspirational baseline, which achieves best performance among previous models and closed-soure, APIs. Conclusively, this research contributes to broadening the horizons for future Al systems to efficiently understand multi multi-identity visual content.
Because the sample of the first stage tuning has only one text image, present the 'QA' and 'caption' results sub-set of MM-ID, which has test image in each sample.
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGMENTS
This work was partly supported by the National Key Research and Development Program of China (No. 2024YFB2808003, the research fund of Tsinghua University Tencent Joint Laboratory for Internet Innovation Technology. This paper is partially supported by the National Key R&D Program China No.2022ZD0161000 and the General Research Fund of Hong Kong No. 1720622 and 17209324.
LEGAL STATEMENT
This work utilized the MovieNet dataset, which is publicly available under open-source license for academic research. For animation images from the internet, we provided web urls of the original content rather than redistributing the images directly, to respect potential copyright restrictions. The authors acknowledge full responsibility for the usage of all datasets in this research and have made best efforts to comply with relevant terms of use and copyright laws. The authors are responsible for any legal violations of data rights.
REFERENCES
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NeurIPS, 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. CoRR, abs/2308.11999 2023.
Mu Cai, Haotian Liu, Siva Karthik Mustikovela, Gregory Meyer, Yuning Chai, Dennis Park, and Yong Jae Lee. Making large multimodal models understand arbitrary visual prompts. CoRR, abs/2312.0070 2023.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krish- namoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhoseiny, Minigpt-v2: large language model unified interface for langunel multi-task learning. CoRR, abs/2310.09478, 2023a.
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. CoRR, abs/2306. 15195, 2023b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. CoRR, abs/231 11.293,, 2023c.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision- language models with instruction tuning. In NeurIPS, 2023.
Xiaoyi Dong, Pan Zhang, Yuhang Zang. Yuhang Cao, Bin Wang. Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Qiao, Dahua Lin, and Jiaqi Wang. Internin-xcomposer2.. Mastering free-form text-image composition and comprehension in vision-language large model. CoRR, abs/2401.16420,, 2024.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang. Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. An empirical study of training end-to-end vision-and-langgage transformers. In CVPR, pp. 181 18145-18155. IEEE, 2022.

---

Published as a conference paper at ICLR 2025
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang. Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. CoRR, htss2236.133994 2023.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang, and Ying Shan. Making Ilama SEE and draw with SEED tokenizer. CoRR, abs/2310.01218, 2023.
Google. Google bard.
2023a.
Google. Gemini-pro.
2023b.
Yash Goyal, Tejas Khot, Aishwarya Agrawal, Douglas Summers-Ster Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role image understanding in visual ques- tion answering. Int. J. Comput. Vis., 127(4):398 -14,, 2019.
Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep multimodal representation learning: A survey, IEEE Access, 7:63373-63394 2019.
Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In ECCV (4), volume 12349 of Lecture Notes in Computer Science, pp- 709-727. Springer, 2020.
Drew A. Hudson and Christopher D. Manning. GQA: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019.
Yatai Ji, Rongcheng Tu, Jie Jiang, Weijie Kong. Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang, and Wei Liu. Seeing what you miss: Vision-language pre-training with semantic comple- tion learning. In CVPR, pp. 6789-6798 IEEE, 2023a.
Yatai Ji, Junjie Wang. Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang. Jiaxing Zhang, Tetsuya Sakai, and Yujuu Yang. MAP: multimodal incertainty-awar lasounlanguuae pre-training model. In CVPR, pp. 13262-23277 IEEE, 2023b.
Bingliang Jiao, Lingqiao Liu, Liying Gao, Ruiqi Wu, Guosheng Lin, Peng Wang, and Yanning Zhang. Toward re-identifying any animal. In NeurIPS, 2023.
Sahar Kazemzadeh, Vicente Ordonez. Mark Matten, and Tamara L. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, pp. 787-798. ACL, 2014.
Jing Yu Koh, Daniel Fried, and Russ Salakhutdinov. Generating images with multimodal language models. In NeurIPS, 2023.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. CoRR, abs/2305.03726, 2023a.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. CoRR, ab$/2408.03326. 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Reed-bench: Bench- marking multimodal Ilms with generative comprehension. CoRR, abs/2307.16125, 2023b.
Huayang Li, Siheng Li, Deng Cai, Longyue Wang, Lemao Liu, Taro Watanabe, Yujiu Yang, and Shuming Shi. Textbind: Multi-turn interleaved multimodal mnstruction-folloiinn in the wild. CoRR, 055/2309.086333 2023c.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2:
bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, volume 202 of Proceedings of Machine Learning Research, pp. 19730-19742. PMLR, 2023d.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination large vision-language models. In EMNLP, pp. 292-305. Association for Computational Linguistics, 2023e.
12

---

Published as a conference paper at ICLR 2025
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In Proc. of ECCV, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023a.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player? CoRR, abs/2207.00281,, 2023b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnoosiiiticc visiolin- guistic representations for vision-and-language tasks. In NeurlPS, pp. 13-23, 2019.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. UNIFIED-IO: A unified model for vision, language, and multi-modal tasks. In ICLR. Open- Review.net, 2023.
Meta. Llama-3. http:://aimetu..oombbgggeeeeaeed 2024.
OpenAl. Chatgpt. https:/opeeaa..co/bllg//ccaaag 2023a.
2023a.
OpenAL. Gpt-4v(ision) system card. http::/cdd..opeeeetee
2023b.
Bryan A. Plummer, Liwei Wang, Chris M. Cervantes, Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. Int. Comput. Vis., 123(1):74 2017.
Hanoona Abdul Rasheed, Muhammad Maaz, Sahal Shaji Mullappilly, Abdelrahman Shaker, Salman H. Khan, Hisham Cholakkal, Rao Muhammad Anwer, Erix Xing, Hing-Hsuunn Yang, and Fahad Shahbaz Khan. Glamm: Pixel grounding large multimodal model. CoRR, abs/2311,03356. 2023.
Chufan Shi, Cheng Yang, Yaxin Liu, Bo Shui, Junjie Wang, Mohan Jing, Linran Xu, Xinyu Zhu, Siheng Li, Yuxiang Zhang, Gongye Liu, Xiaomei Nie, Deng Cai, and Yujiu Yang. Chart- mimic: Evaluating Imm's 's cross-modal reasoning capability via chart-to-code generation. CoRR, 099124 2024.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. CoRR, 022222077.5222, 2023.
Hao Tan and Mohit Bansal. LXMERT: learning cross-modality encoder representations from trans- formers. In EMNLP/JJNNPP (I), pp. 5099-5110. Association for Computational Linguistics, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. CoRR, abs/2302. ,
Rong-Cheng Tu, Yatai Ji, Jie Jiang, Weijie Kong, Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang, and Wei Liu. Global and local semantic completion learning for vision-language pre- training. CoRR, has/2306.00996 2023.
vicuna. Vieuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. https://vicuna.mmsy..oyy 2023.
Junjie Wang, Yatai Ji, Jiaqi Sun, Yujiu Yang. and Tetsuya Sakai. MIRTT: learning multimodal interaction representations from trilinear transformers for visual question answering. In EMNLP Association for Computational Linguistics, 2021.
(Findings). pp.
Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou. Yu Qiao, and Jifeng Dai. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In NeurlPS, 2023.
13

---

Published as a conference paper at ICLR 2025
Mang Ye, Shuoyi Chen, Chenyue Li, Wei-Shi Zheng, David Crandall, and Bo Du. Transformer for object re-identification: A survey. CoRR, abs/2401.06960, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. CoRR, abs/2308.02499 2023.
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. CVPR, pp. 6720-6731. Computer Vision Foundation / IEEE, 2019.
Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest CoRR, abs/2307.03601.. 2023.
Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, and Baobao Chang. MMICL: empowering vision-language model with multi-modal in-context learning. CoRR, abs/2309.07915,, 2023a.
Liang Zhao, En Yu, Zheng Ge, Jinrong Yang. Haoran Wei, Hongyu Zhou, Jianjian Sun, Yuang Peng, Runpei Dong, Chunrui Han, and Xiangyu Zhang. Chatspot: Bootstrapping multimodal lims precise referring instruction tuning. CoRR, abs/2307.09477 2023b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. CoRR, abs/2304.10592 2023.
Yuke Zhu, Oliver Groth, Michael S. Bernstein, and Li Fei-Fei. Visual7w: Grounded question an- swering in images. In CVPR, pp. 1995-5004 IEEE Computer Society, 2016.
14


Reviews:
[{"review_id": "CzTiyZePfE", "reviewer": "Reviewer_BH7G", "summary": "This paper addresses the challenge of movie understanding in VLMs by developing mechanisms to associate instances across different scenes. The authors propose ID-Former, integrated within their ID-Aware VLM architecture, and implement visual instruction tuning with ID references. To evaluate instance ID memory and recognition capabilities across multiple visual scenarios systematically, they introduce a new benchmark called MM-ID.", "questions": "1. The potential scoring variance inherent in subjective assessments with wide scoring ranges by GPT-4. The paper uses GPT-4 to score predictions based on a ten-point scale. Although it adopts the average of 5 times predictions, it would be nice to know whether GPT-4 can consistently provide reasonable explanations for its scoring and whether it can discriminate adjacent score points on the ten-point scale. \n\n2. Regarding Table 5, the observed degradation in ID matching performance with first-stage tuning calls for investigation into potential mitigation strategies to maintain performance at the same level as the \"w/o first stage tuning\" configuration.\n\nI am willing to change the score if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The framework architecture is conceptually straightforward.\n2) The training dataset construction methodology leverages existing open-source databases with replicable procedures.\n3) The MM-ID benchmark provides comprehensive evaluation metrics across multiple aspects of ID-related movie understanding tasks.", "weaknesses": "1. The ID-Former is the essential innovative module in the framework. The authors are encouraged to provide the architectural details and hyperparameter configurations of the ID-Former.\n\n2. The integration of LLaVA and ShareGPT4V data. 1) The selection criteria for the 10\\% subset of LLaVA data remain unclear. 2) The paper would benefit from an ablation study showing \"IDA-VLM (w/ LLaVA data only)\" to isolate the impact of the newly constructed tuning data.\n\n3. The experimental setup primarily addresses scenarios where reference ID count equals or is less than the character count in test images. However, real movie scenes often involve more main and important supporting characters than visible in a single frame. Usually, it is unlikely to accurately know the characters of interest for each requested frame beforehand. Therefore, the model's performance in scenarios with excess reference IDs is worth an investigation.\n\n4. The absence of a systematic evaluation of robustness to ID variations. Particularly, it would be interesting to know its performance regarding environmental factors (lighting, viewing angles) and character appearance changes (clothing, age progression), and whether there is a degradation under significant reference-test image discrepancies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UrPgNngp9G", "reviewer": "Reviewer_nMGR", "summary": "This paper mainly focuses on VQA tasks with multiple images containing the same character identities. To reach a higher performance, the authors construct specific datasets from both (1) multi-image reasoning datasets like VCR, and (2) movie shot datasets, and utilize a certain templates to train such abilities. To evaluate such abilities, the authors propose a new benchmark MM-1D with four question types and 585 samples.", "questions": "My main concern is if this fixed template can cover the wide range of real-world VQA samples. Please see weakness for details.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The motivation that identifies same characters in different images make sense, and it is important for VLMs.\n\nThe new benchmark MM-ID is delicately curated, with statics and detailed process to collect these data.", "weaknesses": "All questions in stage-2 training and MM-ID testing share a similar template as `sombody is <image>. somebody is <image> ...`. It requires the question to have single-character images and be formatted in a certain template. This is too narrow for real world usage and can hardly represent real-world performance.\n\nMoreover, does the data used in stage-2 share the similar template. If so, how can I tell if the increase in performance is due to efficient training on such template?\n\nThe same problems also appears in the model structure in Fig. 2. It seems that the model needs to treat ID images and other images differently in the ID-Former. Moreover, I am not a little confused of the three cross-attn in ID-Former. I suppose there would be multiple ID images and multiple test images, how can I do cross attention taking multiple images as queries and keys at the same time. Please elaborate more on the model structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NplClNuG3L", "reviewer": "Reviewer_JxCK", "summary": "In this paper, the authors propose IDA-VLM, which is designed to conduct visual instruction tuning with ID reference. Furthermore, the authors also construct a benchmark named MM-ID to evaluate the ability of ID memory and recognition in existing  VLMs.", "questions": "See Weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The proposed visual instruction tuning with ID reference is interesting and has the potential to be applied to movie understanding.\n2.The paper is well-written and is easy to understand.", "weaknesses": "Problems:\n1. The authors wrote: “We crop sub-images of the individuals and assign them identifiable labels, such as person names.” in lIne243. I would like to know how to assign each individual a person's name. Is the name randomly generated via Large language models e.g. GPT4?\n\n2. Why the results of Qwen-VL-Chat are not included in Table2 and Table3. Qwen-VL-Chat should be a baseline since the proposed IDA-VLM is built based on it.\n\n3. More models should be included in Table1. There are a lot of popular open-sourced VLMs e.g. LLaVa. These models should be included in Table1 to enhance the benchmark. and facilate future reserch.\n\n4. In the Table4, more standard benchmarks should be included. Only evaluating IDA-VLM on two benchmarks is not enough. The authors should include more benchmarks e.g TextVQA and MMBench for a more comprehensive evaluation.\n\n\nIssues:\n1.The font in the figures is too small, which makes it hard for readers to read. It is highly recommended to increase the font size in figures.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5qY8hkn36W", "reviewer": "Reviewer_1ujz", "summary": "The paper investigate a novel capability for large vision-language models (LVLMs) to recognize and distinguish multiple characters across different visual scenes. The authors introduce a benchmark, MM-ID, to evaluate this ability in four areas: matching, location, question-answering, and captioning. To improve LVLM performance on these tasks, they propose a tuning recipe and develop the IDA-VLM model.", "questions": "- As is described in 'weaknesses', are there other ways to construct localization samples, such as using 3D datasets or leveraging base models like SAM2 and human detection models to construct samples from videos?\n- When there are multiple ID images and multiple test images, how does ID-Former determine which ID image each test image should attend to? Figure 2 only illustrates the one-to-one case.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The motivation is well-grounded. The benchmark also evaluate the model’s ability to understand prompts that involve multiple images interleaved with text.\n- The task of location based on reference character images is innovative. It is different from other locating tasks, such as text-to-roi grounding.", "weaknesses": "When constructing localization samples using RefCOCO, the ID images are cropped and flipped from the original images (this is inferred from the figure, as the text does not mention the flipping). This means that the pattern of the ID images is identical to the region of interest (ROI) in the original image. This does not align with the task defined in the paper, which aims to understand character features and locate the character in another image in a different state. The RefCOCO-based data does not require understanding of character features but simply finding identical content.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)