IDA-VLM: Towards Movie Understanding via ID-Aware Large Vision-Language Model

## Abstract

Abstract The rapid advancement of Large Vision-Language models (LVLMs) has demonstrated a spectrum of emergent capabilities. Nevertheless, current models only focus on the visual content of a single scenario, while their ability to associate instances across different scenes has not yet been explored, which is essential for understanding complex visual content, such as movies with multiple characters and intricate plots. Towards movie understanding, a critical initial step for LVLMs is to unleash the potential of character identities memory and recognition across multiple visual scenarios. To achieve the goal, we propose visual instruction tuning with ID reference and develop an ID - A ware Large V ision- L anguage M odel, IDA-VLM. Furthermore, our research introduces a novel benchmark MM-ID, to examine LVLMs on instance IDs memory and recognition across four dimensions: matching, location, question-answering, and captioning. Our findings highlight the limitations of existing LVLMs in recognizing and associating instance identities with ID reference. This paper paves the way for future artificial intelligence systems to possess multi-identity visual inputs, thereby facilitating the comprehension of complex visual narratives like movies.
Codes and models are available at https://github.com/jiyt17/IDA-VLM .

## 1 Introduction

Our real world contains a wide variety of information, such as texts, images, sounds, etc.
Towards multimodal comprehension Guo et al. (); Lu et al. (), inspired by the success of Large Language Models (LLMs) OpenAI (); vicuna (); Touvron et al. (); Meta (); Google (), there is a surging interest in Large Vision-Language Models (LVLMs), such as LLaVA Liu et al. (), Otter Li et al. (), GPT-4V OpenAI (), Gemini Google (), and others. In the quest for Artificial General Intelligence (AGI), LVLMs serve as pivotal milestones, enhancing machines’ capabilities in multimodal perception, reasoning, and knowledge.

Figure: Figure 1: Comparison of different Visual Instruction Tuning Formats. For visual instruction tuning with ID reference, we arrange the names and images of each character as references. The model should be able to recognize the correct character identity and then answer the user’s instructions.
Refer to caption: x1.png

Typical LVLMs incorporate visual encoders with LLMs, such as LLaVA Liu et al. (), MiniGPT4 Zhu et al. (). These models project an image’s visual features into the embedding space of language models and employ visual instruction tuning, allowing users to complete a variety of visual tasks through language instructions, as depicted in Figure 1(a). However, pure language interaction makes VLMs hard to receive precise Region-of-Interest references from users, thus hindering their capability to focus on specific regions of an image. To address this issue, a series of works Chen et al. (); Zhang et al. (); Cai et al. (); Chen et al. (); Zhao et al. (); Rasheed et al. () attempts to allow adding RoI references to language instructions and then use region-level data to align these RoI references to the image. As shown in Figure 1(b), these excellent works usually support using position coordinates or visual prompts as the RoI reference, which provides a more flexible interactive experience within specific areas.

Previous LVLMs have demonstrated versatile capabilities for visual understanding from the image level to Region-of-Interest. However, these models can only process the visual input of a single scenario, and their ability to associate instances across different scenes has not yet been explored, which requires the model to memorize the instance identity and recognize it in different scenes. This ability, which we call ‘ID awareness’, is a fundamental human visual competency essential for comprehending complex multi-identity visual inputs, such as movies and animations. In the context of movie understanding, viewers should remember the name and appearance of each character, then recognize them across disparate scenes correctly, linking instances across diverse images for accurate narrative interpretation.

To achieve ID awareness in LVLMs like humans, as shown in Figure 1(c), we propose visual instruction tuning with ID reference, which allows users to utilize the name and a reference image of the character (ID image) to define the identity in the prompt, and raise questions about test images.
However, there is no off-the-shelf visual tuning data with ID reference, so we meticulously craft a dual-stage instruction tuning dataset based on existing datasets.
The initial phase leverages annotations in datasets such as VCR Zellers et al. (), Flickr30k Plummer et al. (), and RefCOCO Kazemzadeh et al. () with our data configuration strategies, to tutor the model on associating the instances of ID images with test images, where ID references are extracted from the test image. The subsequent phase utilizes MovieNet Huang et al. ()
to generate Q&A and caption instruction tuning data with GPT-4V OpenAI (). The second stage data contains ID references of higher quality, which is more similar to real movie understanding.
These tuning data unleash the potential of LVLMs to memorize and recognize instance identities in different scenes.
Based on the visual instruction tuning with ID reference, we develop an ID-aware LVLM, called IDA-VLM, suitable for multi-identity visual comprehension.
Our model learns the fine-grained identity information from ID references and generalizes to recognize characters from different scenes of test images.
We also introduce a specialized component, termed ID-Former, to enhance the model’s capability of recognizing character identities.

To measure ID recognition capability for movie understanding, we propose a new benchmark, namely MM-ID.
Our MM-ID aims at challenging LVLMs to remember and recognize instance identities, further understanding complex visual scenes. MM-ID dissects LVLMs’ competency across four progressively complex levels: matching, location, question-answering, and captioning. MM-ID comprises a collection of 585 diverse testing samples, whose instance identities source from the characters of movies, the roles of animations, and individualized animals and objects. We test both open-source models and closed-source APIs, which exposes their apparent deficiencies in identity memory and recognition, and the results are unsatisfactory even for GPT-4V. The evaluation results not only reveal the current limitations of LVLMs in instance recognition with ID reference but also highlight our benchmark’s significance. In contrast, IDA-VLM achieves the best performance on MM-ID, which demonstrates the effectiveness of our instruction tuning strategies and model design.

In a nutshell, our contributions are summarized as follows:

- •
This paper is the first attempt to investigate ID awareness of LVLMs for complex multi-identity visual comprehension, like movies. We propose visual instruction tuning with ID reference and construct corresponding tuning datasets.
- •
We develop an ID-aware LVLM, IDA-VLM to recognize character identities and adopt a dual-stage fine-tuning method to unleash the potential of LVLMs in identity memory and recognition across diverse scenes.
- •
We propose MM-ID, a novel benchmark to examine LVLMs on identity memory and recognition. We evaluate representative LVLMs on our benchmark, uncovering the limitations of existing LVLMs in contextual identity recognition. Furthermore, our model achieves State-of-The-Art (SoTA) performance on MM-ID.

## 2 Related Work

### 2.1 Large Vision-Language Models

Conventional multimodal models consist of uni-modal encoders and cross-modal fusion encoders. Relying on vision-language pre-training Tan and Bansal (); Lu et al. (); Dou et al. (); Wang et al. (); Ji et al. (), they have shown an impressive cross-modal semantic alignment ability Ji et al. (); Tu et al. (), which brings substantial advances on various downstream tasks Goyal et al. (); Plummer et al. (); Lin et al. (). However, due to the limitations of model size and training data scale, the performance of vision-language pre-training models is unsatisfied in open-ended scenarios.

Nowadays, Large language model (LLM) has exhibited remarkable abilities to understand, reason, and generate texts. Large Vision-Language Model (LVLM) Chen et al. (); Dai et al. (); Dong et al. () incorporates visual encoder and LLM, aligning visual features to the embedding space of LLM. Leveraging strong generalization and emergent capability of LLM, LVLM realizes free multimodal interactions with human.
Flamingo Alayrac et al. () is a pioneering work on extending LLMs to vision-language pretraining by inserting additional cross-attention layers for visual input.
BLIP-2 Li et al. () proposes Q-former to map the visual features to the hidden space of language models.
To date, various works have shown encouraging progress with instruction tuning, including MiniGPT-4 Zhu et al. (), LLaVA Liu et al. (), Otter Li et al. (), which demonstrate impressive results on natural instruction-following and visual reasoning capabilities.
The perceptual capabilities of LVLMs are evolving towards fine-grained understanding.
Numerous models Zhang et al. (); Liu et al. (); Chen et al. (); Wang et al. () focus on region-level understanding, using visual instruction tuning with RoI reference, so that users could ask questions about specific instances within the content.
Moreover, there are some LVLMs equiped with Stable Diffusion, which can produce multimodal outputs Koh et al. (); Ge et al. (); Sun et al. (); Li et al. ().
LVLMs hold the potential to perform a wider array of functions and to understand more intricate visual information.
For instance, movie is considered as one of the most intricate mediums for conveying visual information.
This paper researches on movie understanding with ID reference, recognizing characters across disparate scenes.

### 2.2 Multimodal Benchmark

Previous evaluation for multimodal models measures some specific abilities, such as VQAv2 Goyal et al. (), GQA Hudson and Manning () for visual question answering, RefCOCO Kazemzadeh et al. () for visual grounding, Visual7W Zhu et al. () for PointQA, VCR Zellers et al. () for commonsense reasoning in an image.
Recently, there is a surging interest in developing comprehensive benchmarks for evaluating LVLMs.
MMBench Liu et al. () consists of multiple-choice questions and introduces a CircularEval strategy for evaluation framework.
LLaVA-Bench Liu et al. () employs GPT-4 to assess responses from both GPT-4 and the model under evaluation, then provides scores and explanations for the answers.
MM-Vet Yu et al. () assesses LVLMs across six fundamental visual-linguistic capabilities, utilizing a GPT-4-based evaluator for open-ended responses.
MME Fu et al. () tests for perception and cognition competencies across 14 distinct subtasks, which requires models to provide simple ‘yes’ or ‘no’ answers.
SEED-Bench Li et al. () covers 12 evaluation dimensions and adopts an answer ranking strategy to evaluate LVLMs via multiple-choice questions.
POPE Li et al. () is a dedicated benchmark for assessing object hallucination.
Existing benchmarks evaluate multifaceted performance of LVLMs, yet they predominantly focus on visual scenarios that lack character identities and intricate plots.

## 3 Method

To understand complex visual input, for example, a movie or animation, the model need memorize and recognize character identities, linking characters in distinct scenes.
As shown in Figure 2, we simplify movie understanding as given ID images of certain characters and keyframes from the movie or animation served as test images, the model completes instruction tasks. The ID images, test images, and instruction text are sent to the model together for inference in an interleaved format of images and text.
The model need to memorize instance identities in ID references and recognize them in test images for providing responses.

Figure: Figure 2: IDA-VLM is an end-to-end vision-language model for processing instructions that contain references to specific instances. We introduce ID reference as using a correlating character reference image and corresponding name to characterize an identity, exemplified as Julia is <ID-Img{i}>. During tokenization and conversion to embeddings, the embedding of <ID-Img{i}> and <Test-Img{i}> in the instruction are replaced with the ID and Test image embeddings respectively. A simple yet effective image feature projector termed ID-Former is proposed to enhance the ID identification ability. As the output in the figure, IDA-VLM can memorize these character IDs, recognize them in test images, and respond to user instructions with the correct ID references.
Refer to caption: figures/framework.png

### 3.1 Model Architecture

In this paper, we adopt Qwen-VL-chat Bai et al. () as our baseline model. In order to adapt the model to the task of instance ID recognition, we employ a dual-stage visual instruction tuning with ID reference.
The architecture of IDA-VLM comprises three components: a visual encoder, ID-Former consisting of cross-attention mechanisms, and a subsequent large language model.

As shown in Figure 2, the ID-Former is designed to project visual features into the input semantic space of LLM and contribute to recognizing instance identities. This is achieved by two cross-attention modules. The first one interacts learnable queries with the visual features through cross-attention, effectively compressing the visual semantics into a shorter, fixed-length feature encoding. The second cross-attention utilizes queries of ID images to modulate test image embeddings, activating identity information of test images.

### 3.2 First-stage Tuning

In the first phase, we harness the off-the-shelf annotations available in existing datasets along with our proposed data configuration strategies, reducing the cost for extra annotations about ID recognition. Specifically, we utilize the public datasets containing instance spatial information, including Visual Commonsense Reasoning (VCR) Zellers et al. (), RefCOCO Kazemzadeh et al. (), and Flickr30k Plummer et al. () datasets, to construct visual instruction tuning data with ID reference, which contains three instruction tasks: question-answering (Q&A), location, captioning.

VCR dataset consists of questions that require commonsense reasoning to answer, as well as choosing the reasons. These questions ask about particular people or objects, which have location annotations. We crop sub-images of the individuals and assign them identifiable labels, such as person names. By replacing the people in the questions and answers with these labels, we gain a new set of QA data that necessitates ID recognition.
RefCOCO (and variants RefCOCO+ and RefCOCOg) is a dataset targeted on visual localization based on descriptions. For the items to be localized, we created ID images after cropping them from the original images to perform localization tasks. Flickr30k, a dataset comprising image descriptions, includes coordinates for the people or objects mentioned in the descriptions. We extracted the areas associated with individuals to produce ID images and provided names to generate corresponding descriptions.
Additionally, we conduct some data augmentation and cleaning operations, such as filtering out samples with only one person or those with sub-images of inappropriate sizes.
The annotation pipeline is shown in the upper part of Figure 3.

The essential purpose of this stage is to train the model on its ability to build relationships between multiple images, using the ID images to identify and accurately locate or describe objects within test images.
However, the instance ID images are cropped from the original images, which reduces the difficulty of recognition.

Figure: Figure 3: First-stage and second-stage instruction tuning data construction pipelines.
Refer to caption: x2.png

### 3.3 Second-stage Tuning

The second-stage fine-tuning data is based on the MovieNet Huang et al. () dataset. The original annotations in the MovieNet dataset encompass the names of the characters present in each movie shot and their coordinate location information. In the MovieNet, we select pictures that contain only a single character to serve as ID images, while those containing multiple characters are used as test images. This approach allows us to naturally develop a dataset for location task. Moreover, we compile ID images featuring the same character to construct data for matching task.

For Q&A and caption tasks, we adopt GPT-4V to convert annotations from the MovieNet dataset into the question format that refers to specific roles. Specifically, as illustrated in the lower part of Figure 3, we feed test images along with their character location information into GPT-4V, and encourage the model to generate captions or question-answer pairs via prompt engineering. The detailed prompts used for generating descriptions or question-answer pairs are shown in Appendix A. Finally, we integrate ID images, test images and results of GPT-4V into conversation templates, producing second-stage instruction tuning data.

After completing data construction process, we train our model with next-token prediction loss in two stages.
The first stage tuning data contains approximately 200,000 samples, while the second one comprises around 60,000 samples.
Moreover, to ensure that the model would not forget its previously learned knowledge during ID recognition training, we integrate instruction tuning data from LLaVA with the aforementioned data, training them simultaneously.

## 4 MM-ID

### 4.1 Problem Definition

Figure: Figure 4: Data samples of MM-ID(^1^11The urls of animation images are shown in [Github](https://github.com/jiyt17/IDA-VLM).). Each sample consists of ID images of each character, and test images containing multiple characters. Our model can memorize the identity information in ID reference and generalizes to recognize characters from different scenes.
Refer to caption: x3.png

To measure ID recognition capability of models, we propose a new benchmark, MM-ID. We appraise the capability of models to recognize IDs across four incrementally complex levels. The first sub-task we investigate is matching: given an instance image, which could feature a person, an animal, or a building, the model must choose an image from four options that contains the same instance.
The second sub-task involves localizing the instance within a test image based on the ID image, and providing coordinates of the bounding box, with the challenge coming from numerous similar distractor objects or individuals present in the test image. The third and fourth sub-tasks—question-answering and caption generation—require the model to recoginize the identities in the test images, then either respond to related questions or produce a comprehensive description. These represents the most difficult level of tasks, as the model should not only recognize each instance’s identity in the image but also provide appropriate answers based on information related to their names, including states, actions, locations, and more.
The samples of sub-tasks are shown in Figure 1.

### 4.2 Data Collection and Statistics

In the construction of MM-ID benchmark, we initially collect images that meet the specified conditions. To evaluate the model’s capability of recognizing instance-level information, the images used for testing need to contain multiple characters. Furthermore, each character requires distinctiveness, such as different actions, clothing attributes, etc., to facilitate the design of questions. Our data sources primarily encompass three types: shot images of movie scenes from the MovieNet dataset, animated images with download links collected from the internet, and pictures of buildings, vehicles, and animals gathered from object re-identification (Re-ID) datasets Ye et al. (); Jiao et al. (). In each example, there exists ID images corresponding to each character and one or more test images.

Following preparing images of MM-ID, we annotate questions and standard answers manually. For every subtask, we use GPT-4 to generate templates, such as instructions for descriptions, localization. In question-answering tasks, MM-ID poses questions regarding specific characters’ actions, attributes, clothing colors, positions, relationships, emotions, etc. After question annotation, we request other individuals’ assistance in refinement and balancing, increasing the diversity of expression and grammatical structure. Finally, the correct answers are labeled manually by other annotators, providing accurate and detailed descriptions for caption task and correct answers for Q&A task.
Totally, we collect and annotate 585 samples. The data composition is shown in Appendix B.

### 4.3 Evaluation Strategy

Quantitative evaluation for open-domain LVLMs has always been challenging. Our MM-ID incorporates four sub-tasks, each presenting its unique answer format.
For matching task, which resembles multiple-choice questions, accuracy is used as a metric.
In localization task, we compare the predicted bounding box to the actual bounding box, computing the Intersection over Union (IoU) metric to measure accuracy. When IoU exceeds a threshold of 0.5, we consider the model recognizes instance identity accurately.

For question-answering and caption generation tasks, due to the open-ended nature of the generated responses, it is challenging to employ rule-based evaluations. Hence, GPT-4 is used to score the results. Under the condition of provided questions and correct answers, we design prompts to guide GPT-4 to focus on scoring the accuracy of character roles, states, and activities within predictions. We use both absolute and relative scoring strategies. Absolute scoring rates a model’s prediction directly against the correct answer on a ten-point scale, while relative scoring pits two models’ results against each other, providing more immediate comparative insights into the models’ performance with a fraction. The prompts used for GPT-4 scoring are shown in Appendix B.

## 5 Experiments

### 5.1 Quantitative Results

We compare IDA-VLM with other open-source LVLMs Zhao et al. (); Ge et al. (); Bai et al. (); Dong et al. () and closed-source APIs Bai et al. (); Google (); OpenAI () on MM-ID. To meet the requirement of ID recognition, the models selected for testing should support multiple images input. For each model, we design appropriate prompts to guide them to respond according to character names. The prompts and training settings of our model can be found in Appendix C.

As illustrated in Table 1, our model achieves the best performance on MM-ID. According to the quantitative results, Gemini-pro exhibits best performance on matching and Q&A sub-tasks among previous models. In the location sub-task, QwenVL-Chat gains the highest score, even higher than other closed-source APIs. As for the caption sub-task, GPT-4V surpasses other models. Overall, our model outperforms previous LVLMs by a significant margin on four sub-tasks.

Because the absolute scoring of GPT-4 is not entirely objective, we calculate relative scores using GPT-4 to provide a further demonstration on the ‘Q&A’ and ‘Caption’ sub-tasks. We compare the predictions from various closed-source APIs against those from our model respectively. All relative scores are below 1, signifying that our model outperforms its counterparts. Among the closed-source APIs evaluated, GPT-4V exhibits comparatively superior results.

Table: Table 1: The comparison results on MM-ID. Notably, The scores of ‘Q&A’ and ‘Caption’ are evaluated with GPT-4 absolute scoring strategy. ‘-’ indicates the model can’t complete corresponding instruction.

Table: Table 2: Using relative scoring strategy to compare previous LVLMs with our model. Each result is the ratio of the first model’s score to that of the second model’s, after feeding the predictions of two models into GPT-4 for comparison.

Figure: Figure 5: We present visualizations of selected samples from MM-ID, corresponding to Q&A and caption sub-tasks. We showcase responses of GPT-4V, Gemini-pro and ours (IDA-VLM).
Refer to caption: x4.png

### 5.2 Qualitative Comparison

As shown in Figure 5, we qualitatively compare the results of IDA-VLM with GPT-4V and Gemini-pro. In the first case, the model should determine which person walks in front and match the person with the given characters. Our model gives the right answer ‘Ariadne’ while GPT-4V and Gemini-pro recognizes inaccurately. In the second case, models are asked to provide a description for a movie scene with three characters. GPT-4V describes that David and Elise sit at a table, ignoring Charlie standing. Gemini-pro only gives a generic caption for the test image without recognizing characters’ identities within the context. In contrast, our model gives the most reasonable description of the scene.
We present more qualitative examples in Appendix D.

### 5.3 Ablation Studies

Effect of Dual-stage instruction tuning.
We report the separate effect of each instruction tuning stage in Table 1. When the second stage of tuning is removed, there is a significant drop in performance, suggesting that the second stage has a more substantial impact than the first. During the second stage, the ID images are independent of the test images, which enhances the quality of data for model learning. When the first and second stages of tuning are combined, our model achieves its optimal performance. It is worth noting that the accuracy of ‘Matching’ gets lower when adding the first stage tuning, because the first stage tuning data can’t improve matching ability.

Table: Table 3: The ablation study about separate stage instruction tuning.(^1^11Because the sample of the first stage tuning has only one text image, we present the ‘QA’ and ‘caption’ results on a sub-set of MM-ID, which has one test image in each sample.)

Table: Table 4: The ablation study about ID-Former.

Effect of ID-Former.
We use ID-Former to project visual features of ID images and test images to the semantic space of LLM. As depicted in Table 4, substituting the ID-Former with a standard query former leads to a reduction in all sub-task metrics. The ID-Former enhances the interaction between features of ID images and test images, thereby activating ID information within the test images.

The ablation study for the mixing rate of LLaVA instruction tuning data are depicted in Appendix A.

## 6 Conclusion

In this paper, we focus on the capability to recognize and link instances across various scenes, which is significant for understanding complex visual narratives, such as movies. We propose visual instruction tuning with ID reference, which unleashes the potential of LVLM in ID recognition, and develop an ID-aware LVLM, IDA-VLM.
The model memorizes the name and appearance of each character in ID reference, then recognize them across disparate scenes correctly, linking characters across diverse images for accurate narrative interpretation.
To thoroughly assess the performance of instance recognition with ID reference, a novel benchmark named MM-ID has been introduced, which consists of four sub-tasks. Our model achieves best performance among previous models and closed-source APIs.
Conclusively, this research contributes to broadening the horizons for future AI systems to efficiently understand multi-identity visual content.

## 7 Limitations

As we employ QwenVL-chat as the baseline model for our research, our proposed model inherits certain limitations thereof. One such constraint is the upper limit on the number of input images. Within a sample, the sum count of ID images and test images can not surpass eight. Consequently, we opt to use keyframes as test images to effectively perform comprehension of movie segments. By integrating our visual instruction tuning with ID reference, if we fortify the baseline, which is anticipated to enhance overall performance.

Our model focuses on the capability to recognize and link instance IDs across various scenes for accurate narrative interpretation. It should be acknowledged that during the visual instruction tuning with ID reference, there might be a slight reduction in the original capabilities of the model. However, our objective is to develop an ID-aware LVLM, rather than a versatile LVLM.

## 8 Ethical Considerations

Our method is related to person and instance ID, so we pay much attention when collecting data.
Our tuning data and benchmark data mainly come from MovieNet, which is a open-source movie dataset, containing numerous shot images of actors, necessitating careful consideration about the copyright.
Moreover, there exists the potential risk that our model or dataset may be utilized by others to engage in activities concerning a particular individual.

## Appendix A Visual Instruction Tuning Data Construction

We construct dual-stage instruction tuning data with ID reference. The first stage data is constructed with predefined rules, cropping specific instances from test images, while the second stage data is generated by GPT-4V. Specifically, GPT-4V is utilized to produce four kinds of tuning datasets: single-image caption, multi-image caption, question-answer pair for a single image, and question-answer pair for multiple images. We present prompts for producing data in Table 5 and Table 6.
The constructed tuning data scale of two stages are shown in Table 7. We will produce more data and open-source them.

**Table 5: Prompts for GPT-4v to annotate ‘Caption’ instruction tuning data based on MovieNet.**
| Description generation for a single test image |  |
| --- | --- |
| You are a helpful and precise assistant for providing a description for an appropriate image. |  |
| User will give an image and the image size (width, height). Then user will give some character names and their position in the image. The position is expressed with bounding box, which is the person’s left-top corner coordinates and right-bottom corner coordinates (left, top, right, bottom). |  |
| Firstly, you need to judge if it is appropriate. An appropriate image should be clear, should be easy for you to give a caption, the people in it should be easy to recognize. |  |
| If the image is not appropriate, you should answer ‘no’, if it is appropriate, you should give an accurate description of the image with given character names according to the following rules. |  |
| Different characters will be split by ‘\n’, you must remember the right people in the right position. Maybe there are some other people without name in the image, your caption need to contain them if necessary, but the main subject should be about characters with names. |  |
| The description should be accurate and brief. The answer should be less than 60 words. Please pay more attention to people’s action. Your answer should be only about the visual content, don’t include your own speculation. |  |
| Then you should give an accurate description of the image with given character names. |  |
| Description generation for multiple test images |  |
| You are a helpful and precise assistant for providing a description for some continuous images. You can treat it as video caption generation. You need give an overall story description for these images according to the following rules. |  |
| User will give the image size (width, height) and some images. For each image, user will give some character names and their positions in the image. The position is expressed with bounding box, which is the person left-top corner coordinates and right-bottom corner coordinates (left, top, right, bottom). |  |
| Different characters will be split by ’\n’, you must remember the right people in the right position. Maybe there are some other people without name in the image, your caption need contain them if necessary, but the main subject should be about characters with names. |  |
| The description should be accurate and brief. The answer should be less than 60 words. Please pay more attention to people’s action. Your answer should be in accordance with temporal order of the images. Your description should be coherent and have some logical connections. Your answer should be only about the visual content, don’t include your own speculation. |  |
| You should describe the changes in the characters’ states and interactions throughout the entire images sequence as much as possible, avoiding fragmented descriptions for each individual image. Especially refrain from using phrases like ’in the image 1’ and so on. |  |
| Then you should give an accurate and brief description of the images with given character names. |  |
| A good example: ’Carol watches as Hal, in a white tank top, looks at his shirt. As she approaches, they engage in a close conversation, and eventually, Hal looks at Carol while gesturing, continuing their discussion.’ |  |

**Table 6: Prompts for GPT-4v to annotate ‘Q&A’ instruction tuning data based on MovieNet.**
| Q&A generation for a single test image |  |
| --- | --- |
| You are a helpful and precise assistant for providing a question-answer pair of an image with given character names. |  |
| User will give an image and the image size (width, height). Then user will give some character names and their position in the image. The position is expressed with bounding box, which is the person left-top corner coordinates and right-bottom corner coordinates (left, top, right, bottom). |  |
| Firstly, you need to judge if it is appropriate. An appropriate image should be clear, should be easy for you to give a caption, the people in it should be easy to recognize. |  |
| If the image is not appropriate, you should answer ’no’, if it is appropriate, you should give a question-answer pair of the image with given character names according to following rules. |  |
| Different characters will be split by ’\n’, you must remember the right people in the right position. |  |
| Then you should give a pair of question and corresponding answer about the image with given character names. The question and answer should be split by ’\n’. |  |
| The question asks about the given character, including character actions, character attributes (clothes, expression, etc), character locations, relative relationship between characters, etc. Only include questions that have definite answers. Some examples of question templates: What is xxx doing? What color is xxx’s clothes? |  |
| The question and answer should be accurate and brief. The answer should be strictly correspond to the question and be less than 30 words. |  |
| Q&A generation for multiple test images |  |
| You are a helpful and precise assistant for providing a question-answer pair for some continuous images with given character names. You can treat it as video queation answering generation. You need give an overall question and answer for these images according to the following rules. |  |
| User will give the image size (width, height) and some images. For each image, user will give some character names and their positions in the image. The position is expressed with bounding box, which is the person left-top corner coordinates and right-bottom corner coordinates (left, top, right, bottom). |  |
| Different characters will be split by ’\n’, you must remember the right people in the right position. |  |
| Then you should give a pair of question and corresponding answer about the images with given character names. The question and answer should be split by ’\n’. |  |
| The question asks about one of or some given characters, including character actions, character attributes (clothes, expression, etc), relative relationship between characters, etc. Only include questions that have definite answers. |  |
| The question and answer should be accurate and brief. The answer should be strictly correspond to the question and be less than 30 words. |  |
| You should focus on the changes in the characters’ states, actions or interactions throughout the entire images sequence as much as possible, avoiding fragmented question for each individual image. Especially refrain from using phrases like ’in the image 1’ and so on. |  |
| A good example: ’What is Timmy doing?\nTimmy walks into the room, then has a conversation with another man, finally they hug each other excitedly.’ |  |

To preserve the innate proficiency of the baseline model, we integrate the instruction tuning dataset from LLaVA with our compiled dataset. As depicted in Table 8, we conduct an ablation study on the proportion of LLaVA dataset. The capability for ‘Matching’ reaches its best in the absence of LLaVA dataset, suggesting LLaVA data has no benefit for ‘Matching’. The location score is highest at a 20% inclusion rate of LLaVA data. In contrast, ‘Q&A’ and ‘Caption’ scores reach their optimal levels when the inclusion rate is set at 10%. This could imply that a judicious amount of LLaVA data serves to fortify foundational multimodal and text generation competencies. However, an excessive infusion of LLaVA data seems to hinder the learning of ID recognition. Therefore, we select 10% as the mixing rate of LLaVA data.

**Table 7: ID recognition instruct-tuning dataset.**
| Stage | tuning data | data scale |
| --- | --- | --- |
| The first stage | VCR | 60k |
| RefCoco | 60k |  |
| Flickr30k | 60k |  |
| The second stage | Matching | 10k |
| Location | 10k |  |
| Single-image Q&A | 10k |  |
| Single-image Caption | 10k |  |
| Multi-image Q&A | 10k |  |
| Multi-image Caption | 10k |  |

**Table 8: The ablation study about instruction tuning data of LLaVA.**
| Model | Matching | Location | Q&A | Caption |
| --- | --- | --- | --- | --- |
| Ours (w/o LLaVA data) | 0.746 | 0.797 | 5.27 | 4.98 |
| Ours (10% LLaVA data) | 0.716 | 0.821 | 5.71 | 5.15 |
| Ours (20% LLaVA data) | 0.716 | 0.829 | 5.67 | 5.11 |

## Appendix B MM-ID

The data composition of MM-ID is shown in Table 9.
During MM-ID construction, we annotate questions and standard answers manually. The instructions for annotators are shown in Table 10.

It is a challenge to evaluate accuracy of the model on ‘Q&A’ and ‘Caption’ sub-tasks. We utilize GPT-4 to score the predictions under the condition of provided questions and correct answers. We propose GPT-4 absolute and relative scoring strategies with designed prompts, as depicted in Table 11.

The calculate process of relative score is as follows: GPT-4 gives two scores (10 points) for predictions from different models respectively. We compute mean value of scores for each model, then calculate the ratio of the average scores as the final relative score.

**Table 9: MM-ID data statistic.**
| Sub-task | Question Num | Image Num | Source |
| --- | --- | --- | --- |
| Matching | 67 | 335 | MovieNet, Re-id datasets |
| Location | 123 | 360 | MovieNet, animations |
| Q&A | 154 | 507 | MovieNet, animations |
| Caption | 241 | 1110 | MovieNet, animations |

**Table 10: The instructions for Annotators.**
| Design Questions: The questions are about specific characters in an image. These questions require the model to correctly identify a specific person to provide an answer, such as inquiring about someone’s condition, actions, positions, etc. Consider more angles for asking questions.<br>Annotate Answers for Q&A and Descriptive Types of Questions: Answers to Q&A types of questions need to be accurate based on the character. Descriptive questions only require a correct description, as detailed as possible. Besides what each person is doing, include some simple background descriptions as well. |
| --- |

**Table 11: The prompts for evaluating with GPT-4 on MM-ID.**
| Absolute Score |  |
| --- | --- |
| You are a helpful and precise assistant for evaluating answers. We would like to request your feedback on the quality of an AI assistant’s answer according to the given question and ground truth. The question, answer of AI assistant and ground truth will be signed by ‘question’, ‘prediction’ and ‘GT’. You need to judge whether the overall meanings of prediction and ground truth answer are consistent or not. Please pay more attention to the correspondence of character names and their states or actions. Please rate the helpfulness, relevance, accuracy of the responses. You should give an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Please first output a single line containing only one value indicating the score for Assistant answer. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment. |  |
| Relative Score |  |
| We would like to request your feedback on the performance of two AI assistants in response to the user question according to the given ground truth. The question, ground truth answer and predictions of two AI assistants will be signed by ’question’, ’GT’, ’prediction 1’ and ’prediction 2’. Please rate the helpfulness, relevance, accuracy, level of details of their responses. Each assistant receives an overall score on a scale of 1 to 10, where a higher score indicates better overall performance. Please pay more attention to the correspondence of character names and their states or actions. Please first output a single line containing only two values indicating the scores for Prediction 1 and 2, respectively. The two scores are separated by a space. In the subsequent line, please provide a comprehensive explanation of your evaluation, avoiding any potential bias and ensuring that the order in which the responses were presented does not affect your judgment. |  |

## Appendix C Experiment Settings

We use QwenVL-chat for model initialization, which has 9.6B parameters. In IDA-VLM training, we set learning rate as 1e-5 for the first stage and 5e-6 for the second stage. The model is trained for 5 epochs in both the first and second stages. The batch size with gradient accumulation is set to 128. The visual encoder is fixed, while ID-Former and LLM are fine-tuned. It costs approximately 1 day training on 8$\times$A100-SXM-80GB for both the first stage and second stage tuning.

To promote the evaluation of models on MM-ID tasks, we craft specific prompts to instruct models in accurately identifying character IDs. The prompts tailored for closed-source APIs are detailed in Table 12. and prompts for open-source models are designed similarly.

**Table 12: The prompts for closed-source APIs testing on MM-ID.**
| Matching, Q&A, Caption (GPT-4V, Gemini-pro, QwenVL-Plus, QwenVL-Max) |  |
| --- | --- |
| You are a helpful and precise assistant for providing a answer to the question. You need recognize instance identity to answer questions about reference characters or give a caption with character names (for multiple continuous images). You must provide an exact answer. |  |
| Location (GPT-4V, Gemini-pro) |  |
| You need to give coordinates of bounding box of some given characters or objects. The answer form should be ‘bbox: [x1, y1, x2, y2].’, where x1 is left side of bounding box, y1 is upper side, x2 is right side, y2 is bottom side, they are all integers. |  |
| Location (QwenVL-Plus, QwenVL-Max) |  |
| You need to give coordinates of bounding box of one given character or object. The answer form should only be ’<ref>xxx</ref><box>(x1,y1),(x2,y2)</box>.’, where x1 is left side of bounding box, y1 is upper side, x2 is right side, y2 is bottom side, they are all integers. |  |

## Appendix D More Visualization

In this section, we present more qualitative results of IDA-VLM, as shown in Figure 6, Figure 7, Figure 8, Figure 9.

Figure: Figure 6: Samples of Matching sub-task.
Refer to caption: x5.png

Figure: Figure 7: Samples of Location sub-task.
Refer to caption: x6.png

Figure: Figure 8: Samples of Q&A sub-task.
Refer to caption: x7.png

Figure: Figure 9: Samples of Caption sub-task.
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The potential scoring variance inherent in subjective assessments with wide scoring ranges by GPT-4. The paper uses GPT-4 to score predictions based on a ten-point scale. Although it adopts the average of 5 times predictions, it would be nice to know whether GPT-4 can consistently provide reasonable explanations for its scoring and whether it can discriminate adjacent score points on the ten-point scale. 

2. Regarding Table 5, the observed degradation in ID matching performance with first-stage tuning calls for investigation into potential mitigation strategies to maintain performance at the same level as the "w/o first stage tuning" configuration.

I am willing to change the score if my concerns are addressed.
soundness: 3
strengths: 1) The framework architecture is conceptually straightforward.
2) The training dataset construction methodology leverages existing open-source databases with replicable procedures.
3) The MM-ID benchmark provides comprehensive evaluation metrics across multiple aspects of ID-related movie understanding tasks.
summary: This paper addresses the challenge of movie understanding in VLMs by developing mechanisms to associate instances across different scenes. The authors propose ID-Former, integrated within their ID-Aware VLM architecture, and implement visual instruction tuning with ID references. To evaluate instance ID memory and recognition capabilities across multiple visual scenarios systematically, they introduce a new benchmark called MM-ID.
weaknesses: 1. The ID-Former is the essential innovative module in the framework. The authors are encouraged to provide the architectural details and hyperparameter configurations of the ID-Former.

2. The integration of LLaVA and ShareGPT4V data. 1) The selection criteria for the 10\% subset of LLaVA data remain unclear. 2) The paper would benefit from an ablation study showing "IDA-VLM (w/ LLaVA data only)" to isolate the impact of the newly constructed tuning data.

3. The experimental setup primarily addresses scenarios where reference ID count equals or is less than the character count in test images. However, real movie scenes often involve more main and important supporting characters than visible in a single frame. Usually, it is unlikely to accurately know the characters of interest for each requested frame beforehand. Therefore, the model's performance in scenarios with excess reference IDs is worth an investigation.

4. The absence of a systematic evaluation of robustness to ID variations. Particularly, it would be interesting to know its performance regarding environmental factors (lighting, viewing angles) and character appearance changes (clothing, age progression), and whether there is a degradation under significant reference-test image discrepancies.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: My main concern is if this fixed template can cover the wide range of real-world VQA samples. Please see weakness for details.
soundness: 2
strengths: The motivation that identifies same characters in different images make sense, and it is important for VLMs.

The new benchmark MM-ID is delicately curated, with statics and detailed process to collect these data.
summary: This paper mainly focuses on VQA tasks with multiple images containing the same character identities. To reach a higher performance, the authors construct specific datasets from both (1) multi-image reasoning datasets like VCR, and (2) movie shot datasets, and utilize a certain templates to train such abilities. To evaluate such abilities, the authors propose a new benchmark MM-1D with four question types and 585 samples.
weaknesses: All questions in stage-2 training and MM-ID testing share a similar template as `sombody is <image>. somebody is <image> ...`. It requires the question to have single-character images and be formatted in a certain template. This is too narrow for real world usage and can hardly represent real-world performance.

Moreover, does the data used in stage-2 share the similar template. If so, how can I tell if the increase in performance is due to efficient training on such template?

The same problems also appears in the model structure in Fig. 2. It seems that the model needs to treat ID images and other images differently in the ID-Former. Moreover, I am not a little confused of the three cross-attn in ID-Former. I suppose there would be multiple ID images and multiple test images, how can I do cross attention taking multiple images as queries and keys at the same time. Please elaborate more on the model structure.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See Weakness
soundness: 3
strengths: 1.The proposed visual instruction tuning with ID reference is interesting and has the potential to be applied to movie understanding.
2.The paper is well-written and is easy to understand.
summary: In this paper, the authors propose IDA-VLM, which is designed to conduct visual instruction tuning with ID reference. Furthermore, the authors also construct a benchmark named MM-ID to evaluate the ability of ID memory and recognition in existing  VLMs.
weaknesses: Problems:
1. The authors wrote: “We crop sub-images of the individuals and assign them identifiable labels, such as person names.” in lIne243. I would like to know how to assign each individual a person's name. Is the name randomly generated via Large language models e.g. GPT4?

2. Why the results of Qwen-VL-Chat are not included in Table2 and Table3. Qwen-VL-Chat should be a baseline since the proposed IDA-VLM is built based on it.

3. More models should be included in Table1. There are a lot of popular open-sourced VLMs e.g. LLaVa. These models should be included in Table1 to enhance the benchmark. and facilate future reserch.

4. In the Table4, more standard benchmarks should be included. Only evaluating IDA-VLM on two benchmarks is not enough. The authors should include more benchmarks e.g TextVQA and MMBench for a more comprehensive evaluation.


Issues:
1.The font in the figures is too small, which makes it hard for readers to read. It is highly recommended to increase the font size in figures.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - As is described in 'weaknesses', are there other ways to construct localization samples, such as using 3D datasets or leveraging base models like SAM2 and human detection models to construct samples from videos?
- When there are multiple ID images and multiple test images, how does ID-Former determine which ID image each test image should attend to? Figure 2 only illustrates the one-to-one case.
soundness: 4
strengths: - The motivation is well-grounded. The benchmark also evaluate the model’s ability to understand prompts that involve multiple images interleaved with text.
- The task of location based on reference character images is innovative. It is different from other locating tasks, such as text-to-roi grounding.
summary: The paper investigate a novel capability for large vision-language models (LVLMs) to recognize and distinguish multiple characters across different visual scenes. The authors introduce a benchmark, MM-ID, to evaluate this ability in four areas: matching, location, question-answering, and captioning. To improve LVLM performance on these tasks, they propose a tuning recipe and develop the IDA-VLM model.
weaknesses: When constructing localization samples using RefCOCO, the ID images are cropped and flipped from the original images (this is inferred from the figure, as the text does not mention the flipping). This means that the pattern of the ID images is identical to the region of interest (ROI) in the original image. This does not align with the task defined in the paper, which aims to understand character features and locate the character in another image in a different state. The RefCOCO-based data does not require understanding of character features but simply finding identical content.

Meta-review: This paper explores understanding complex visual content, such as movies with multiple characters and intricate plots. It proposes a visual instruction tuning approach for Vision-Language Models (VLMs) that incorporates identity awareness.

The reviewers praise the paper for addressing an important topic, its well-motivated and interesting method, the value of its benchmark, and clear paper writing. In the end, Most primary concerns were resolved during the rebuttal. All four reviewers are in favor of accepting the submission. The remaining issues include clarifying model parameters and the straightforward nature of the method.

The AC thinks that being straightforward yet effective does not diminish the paper's contribution. The AC recommends accepting this submission.

Decision: Accept (Poster)