INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning
Authors: Wujian Peng
Lingchen Meng
Yitong Chen
Yiweng Xie
Yang Liu
Tao Gui
Hang Xu
Xipeng Qiu
Zuxuan Wu
Yu-Gang Jiang
Venue: NeurIPS 2025 poster ()
Keywords: multimodal models
instruction tuning
fine-grained understanding
Primary area: deep_learning

Abstract: Large Multimodal Models (LMMs) have made significant breakthroughs with the advancement of instruction tuning. However, while existing models can understand images and videos at a holistic level, they still struggle with instance-level understanding that requires a more fine-grained comprehension and alignment. Instance-level understanding is crucial for LMMs, as it focuses on the specific elements that we are most interested in. Excitingly, existing works find that the state-of-the-art LMMs exhibit strong instance understanding capabilities when provided with explicit visual cues. Motivated by this, we proposed Inst-IT, a solution to enhance LMMs in Instance understanding via explicit visual prompt Instruction Tuning for instance guidance. Inst-IT consists of a benchmark to diagnose multimodal instance-level understanding, a large-scale instruction-tuning dataset, and a continuous instruction-tuning training paradigm to effectively enhance spatial-temporal instance understanding capabilities of existing LMMs. Experimental results show that, enhanced by Inst-IT, our models not only achieve outstanding performance on Inst-IT-Bench and other instance understanding benchmarks, but also demonstrate significant improvements across various generic image and video understanding benchmarks. This highlights that our method not only boosts instance-level understanding but also strengthens the overall capabilities of generic image and video comprehension.

INST-IT: Boosting Instance Understanding via Explicit Visual Prompt Instruction Tuning
Wujian Pengl.2
Lingehen Meng'* Yitong Chen Yiweng Xie Yang Liu
Tao Gui
Hang Xu
Xipeng Qiu Zuxuan Wu Yu-Gang Jiang'
Institute of Trustworthy Embodied AI, Fudan University Shanghai Innovation Institute Huawei Noah's Ark Lab
https:/ git.gg.tttubii.
Abstract
Large Multimodal Models (LMMs) have made significant breakthroughs with the advancement of instruction tuning. However, while existing models can under- stand images and videos at holistic level, they still struggle with instance-level understanding that requires a more fine- ane-grrined comprehension and alignment. Instance- understanding is crucial for LMMs, as it focuses on the specific elements that we are most interested late-of-the-arr art LMMs exhibit strong instance understanding capabilities when pro- vided with explicit visual cues. Motivated by this, we proposed INST-IT, a solution to enhance LMMs in Instance understanding via explicit visual prompt Instruction Tuning for instance guidance. INST-IT consists of a benchmark to diagnose multi- modal instance-lewe understanding, large-scale instruction tunnnng dataset, and a continuous instruction seaectiiontttnnnggg training paradigm to effectively enhance spatial- temporal instance understanding capabilities of existing LMMs. Experimental results show that, enhanced by INST-IT, our models not only achieve outstanding performance INST-IT Bench and other instance understanding benchmarks, but also demonstrate significant improvements across various generic image and video understanding benchmarks. This highlights that our method not only boosts instance-level understanding but also strengthens the overall capabilities of generic image and video comprehension.
Excitingly, existing works find that the
Introduction
Recently, Large Multimodal Models (LMMs) have seen remarkable advancements. A key break- through is visual instruction tuning [45, 17], enabling models to follow any type of user instructions. This paves the way to building general-purpose multimodal assistants capable of handling wide range of real-world tasks [32]. Inspired by this initial work, numerous follow-up studies have emerged both image-language [43, 60, 106, 9, 13] and video-language [51, 22, 99, 84, 82] model- ing. However, although they understand images videos at holistic level. they still struggle to comprehend -sstifc--peecffcc content that the users are most interested in, as illustrated in Fig. I (a).
Instance-level understanding involves comprehending the attributes of individual instances within an image video, as well the relationships and interactions between them. This requires models to exhibit nuanced comprehension and fine-ge mmn-igaieed alignment. Instance understanding has been long- standing pursuit of the community with extensive efforts devoted to object detection [73, 71, 23, 58), instance segmentation [74, 29, 59]. and object tracking [19, 76]. This capability is essential for
Equal contributions; Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Frame
Frame
Frame
Frame
Xx-3
What is the difference about the man and the woman between
and
Instancces
The difference that woman's white. matching man's shirt.
Image
dividuull instances
mege
walking
direction in @ but in O the woman walking in the opposite direction.
Entire mage
co
Temporal changes
X.
In O the man and woman are walking
V!
together, in I the woman standing near
Inst-I7
Video level
the car, and the man no longer visible..
Sescitttion you
Open anded queskion-gnsaeer gait
(a)
Voddo
ShareGPT4Vicde
Our Model
(b)
Figure I: (a) LMMs struggle with instance understanding, failing to capture the nuanced details of instances specified in user queries, (b) Our instance-centric data annotation pipeline, providing multi-level annotations for individual instances in images and videos.
real-world applications, where users pay more attention to the instances that they are interested in. In the era of LMMs, although there have been some attempts in exploring multimodal instance understanding [24, 5, 101, 105, 102], they are primarily limited in the image domain, leaving the videos under-explored. Compared to images, understanding instances in videos is considerably more challenging, as it requires not only capturing their spatial information but also temporal dynamics. Driven by this, we aim to advance the multimodal instance understanding in both images and videos. To this end, we focus on three aspects: instruction-tuning dataset, evaluation benchmark, and training recipe.
Existing multimodal benchmarks and datasets primarily provide coarse- coarse-grained knowledge for images and videos, lacking tine-graine annotations for individual instances. To address this, introduce an automated pipeline to generate detailed instance-specinn annotations. As illustrated in Fig. (b), leverage GPT-4o [61] to produce multi-level annotations, including instance-level descriptions, image-level captions, temporal dynamics, video-level summaries, and open-ended question-answer pairs. To fully unleash the capability of GPT-4o for more accurate annotations, we systematically design task prompts and employ set-of-marks visual prompts [88] to highlight instances in the visual inputs. Powered by this pipeline, we construct INST-IT Dataset, an instance-grounded multimodal dataset comprising 51k images and 21k videos, 207k image-level captions, 135k temporal dynamics descriptions, 2 1 video-level captions, and 335k open-ended question-answe pairs. Furthermore. we carefully design the INST-IT Bench to diagnose the instance-level understanding capabilities of LMMs, and perform rigorous manual verification and refinement to ensure its data quality.
Building on INST-IT Dataset, we propose a continuous instruction tuning recipe that effectively integrates instance understanding datasets with general Instruction-tuninn -uningnnn data. We augment images and videos with visual prompts, and convert the fine-gras anneggrined annotations from INST-IT Dataset into instruction tuning format, emphasizing the model's spatiotemporal understanding of individual instances. Experimental results show that our enhanced models achieve strong instance understanding performance not only INST-IT Bench, but also demonstrate consistent improvements on other instance understanding benchmarks e.g. RefCOCOg [53] and ViP-Bench [5]. We also investigate the models' general comprehension capabilities on widely used generic benchmarks. The results reveal significant improvements over the bascline, achieving 4.4% and 13.5% gains on A12D [28] and CharQA [54] image benchmarks, as well as 7.8% and 11.8% improvements on Egoschema [52] and NEXT-QA [85] video benchmarks, respectively. This highlights the effectiveness of INST-IT in boosting instance understanding while strengthening general comprehension in both images and videos. Our contributions are three-fold:
1. We construct the INST-IT Dataset, the first instance-grounde instruction-tuning dataset that includes both images and videos, featuring explicit instance-level visual prompts and fine-grained annotations grounded on individual instances.

---

2. We introduce the INST-IT Bench, a numan--vriifen benchmark specifically designed to evaluate the instance-level understanding capabiliiiies of LMMs on both images and videos.
3. We propose a continuous instruction tuning recipe, which leverages our instance-level dataset alongside general data, effectively enhancing models in instance understanding while consistently improving general comprehension in both images and videos.
2
INST-IT
To address the scarcity of instance-grounder data, we propose an automated pipeline to generate detailed annotations for both images and videos, with particular emphasis on instances of interest (Sec. 2.1). Based on this, we build a large-scale instance- grouuddd multimodal dataset (Sec. 2.2), and carefully design an instance-centric evaluation benchmark (Sec. 2.3). Furthermore, we propose continuous instruction-tuning recipe (Sec. 2.4) to enhance LMMs in instance understanding.
2.1 Instance-centric annotation pipeline
Overview. We propose automated pipeline to generate annotations grounded individual instances. As in Fig. (b), we annotate each frame sequentially, aggregate frame-level annotations into a comprehensive video-level description, and generate open-ended question-answer pairs.
Visual prompting. Directly processing the raw visual inputs suffers from hallucinations and distrac- tion. To mitigate this issue, augment the images and videos with visual prompts to highlight the instances. Specifically, we use set-of-marks (SoMs) visual prompt [88], which overlays a numerical ID on each instance. We find this method highly effective in guiding GPT-4o to provide annotations focused on individual instances. For more details, please refer to Sec. A.1.
Frame-level annotation. We annotate video frames sequentially. At timestamp t, we provide GPT-4o with the current frame X,, the previous frame X,.1, and a tailored task prompt P!. We then obtain a frame-level annotation Y/² ,3/, img 1/1 dis encompassing three aspects, where y/" represents the captions for individual instances, V/ Vim is a caption for the entire image, and y: describes the temporal differences from the previous frame:
Y G X..).
Video-level summary. After obtaining annotations for each frame, we aggregate them into a caption for the entire video Yvid capturing detailed spatiotemporal information of individual instances:
yndd GPT(p [X/ YY....,,.......... ,Y&I).
where poid is the task prompt designed for video-level summary and N is the total number of frames.
Open-ended question-answer pairs. We also prompt GPT-4o with the task prompt prt to create M open-ended QA pairs y0==((g;, a.)) ).M M focusing on instance understanding:
yo GPT(P"M (Y/ Y/1)
Following these steps, each video is enriched with multi caalii-raaularity annotations that incorporate instance-specifie information. As illustrated in Fig. 2, these annotations include the following aspects:
N frame-level annotations, each contains detailed descriptions individual instances, the entire image, and the temporal dynamics between adjacent frames.
A comprehensive description covering the entire video.
M open-ended question-answer pairs that focused on individual instances or their relationships.
Additional information about the design of each task prompt is provided in Sec. A.2.
2.2 INST-IT Dataset
Instruction tuning plays crucial role in multimodal training; however, the lack of instance-level datascts hinders the advancement of instance understanding. Using the data annotation pipeline
3

---

Frame
Frame
Frame
Sample
under paviloon dressed light patterned and parts, bending forward towards El a small chil blue denim overalls. extands Ell a bright yellow ball, towards El, who standing nearby. seemingly reaching for the ball, At
At
cark outfit visible
Captions:
Image-level
wearing light patterned
Bl is standing upright, visible waist wearing dark pants and standing on a surface.
has left the frame, and
small child with short hair, wearing
has shifted upright position... has taken possession []
blue denim overal.
adjusted,
the
bright yellow ball being held by !l
and El, small child denim overalls, holding El yellow object, its right hand. scene appears to in partially outdoor.... angle elevated, showing the scene standing focusing on interaction between B. Bl. and Ei...
surrounding gressy At O. facing away from the camera, slighlly bent forward, with...
Temporal Change Captions:
(b) Entire video description
no longer visible.
Q: object did towards 3 first
moved
bending
&: E held bright yollow towardsEl.
upright position and...
frames and
has changed direction and to have moved further away from is the grasp of El, suggesting has possession....
viible
Q What happens from frame frame
A [] moved from bending an upright, and less visible.
Q ...?
(a) Frame-level
(c) Question-answer
Figure 2: Visualization of data structure in INST-IT Dataset. For each video, we provide (a) N frame-level annotations, (b) video-level description, and (c) M open-ended question-answer pairs. complete example data can be found Sec. C.3.
described in Sec. 2.1, create a large-scale instruction-tuning dataset, the INST-IT Dataset. To the best of our knowledge, this is the first instruction fine-tuning dataset that provides multi-level fine-grained annotations centric on individual instances in both images and videos.
Data sources. We utilize five video instance segmentation datasets (BRUST [3, 18], UVO [83]. OVIS [65], LVVIS [79] and Youtube [89]) and two object tracking datasets (BenSMOT [38], VidOR (75)) as our video sources, as they provide annotations instance locations, which is useful in SoM visual prompting [88]. For the image source, select the SA-1B [29] dataset due to its diversity and abundance of instance objects. In total, we collect 51k images and 21k videos. More details can be found in Sec. C.I.
Statisties. average, each video includes one video-level annotation, 7.3 frame-level annotations, and 15.6 open-ended QA pairs. Images regarded as single-frame videos without temporal changes. In total, INST-IT Dataset includes 2 vidcos and 51k images, alongside 21k vidco-level captions, 207k frame-level captions, 836k instance-level captions, 135k temporal descriptions, and 335k open-ended QA pairs. More statistical analyses are provided Sec. C.2.
Data quality. We employ three strategies to ensure the data quality: (1) High-qualins visual prompts, we use manually annotated labels in segmentation and tracking tasks SoMs reduce noise. (2) Specialized prompt design, we introduce multi-level prompt engineering at the instance, image, two-frame, and video levels mitigate long-term inconsistencies. (3) Diversity filtering, we filter out samples with few instances to enhance diversity and domain coverage. We randomly select 500 data samples and invite voluntcers to independently rate each sample with a score ranging from I to 5 (higher is better). The mean of scores and average time spent per sample are in Tab. 10. The average score 4.49+0.05. indicating the satisfactory quality of our data. We use the maximum score difference (maxan) among volunteers assess rating consistency. 19.8% of samples have max and 78.6% max < 1, showing high agreements on the ratings of different volunteers.
Comparison with existing instruction tuning datasets. Tab. 1 (left) compares INST-IT Dataset with other datasets. Prior video datasets, e.g. ShareGPT4Video [10] and L.-VA-Vide [104], focus on holistic understanding without instance-level annotations. While VIP-LLaVA [5] offers instance annotations for images, does not include any video data. contrast, INST-IT Dataset encompasses both images and videos with multi-level, fine-grained annotations grounded on individual instances.

---

Table 1:
Comparison of INST-IT with existing datasets and instance understanding benchmarks. Left: Instruction tuning datasets. Right: Instance understanding benchmarks. IMG and VID indicate whether the data contains images or videos, respectively. INST denotes the availability of instance- level annotations. OE and MC indicate open-ended and multiple-choice QA.
IMG
VID
INST
IMG
VID
TASK
ShadeGPT4Video [10]
RefCOCO [27]
caption
VL Video [104] ViP-LLaVA-Dait [5]
RefCOCOg [53]
caption OE
V
ViP-Bench [5]
INST-IT Dataset
V
INST-IT Bench
OE&MC
2.3 INST-IT Bench
Existing benchmarks primarily focus on global understanding, failing to provide more in-depth insights into the instance-level comprehension. We present the INST-IT Bench, specifically designed diagnose multimodal instance-level understanding in both images and videos.
Construction process. To prevent data leakage, we use videos from the test split, ensuring no overlap with ITST-IT Dataset. We apply the pipeline in Sec. 1.1 to generate 20 open- open-ended QA pairs for each image and video. Then, we manually review these QA pairs to ensure their accuracy, diversity, and difficulty. Overly simple questions We also refine the questions and answers, making necessary rephrasing to ensure correctness. After this rigorous checking process, each sample retains average of 3.7 carefully polished QA pairs. In addition, we generate three hard negative for each question to construct a multiple-choice QA data with four options. More details are provided in Sec. B.1.
removed ensure the remaining ones instances-centric.
Statistics. INST-IT Bench comprises 1,000 QA pairs for 338 images and 1,000 QA pairs for 206 videos. Each QA pair supports two evaluation formats, i.e. open-ended and multiple-choice.
Metries. For open-ended QAs, we leverage GPT-4o to evaluate the response from a model based on its similarity to the ground-truth answer. For multiple-choice QAs, we calculate the average accuracy across all samples. More details about the metric calculations can be found in Sec. B.2.
Comparison with existing instance understanding benchmarks. Tab. 1 (right) highlights the main differences between INST-IT Bench and existing instance understanding benchmarks such as RefCOCO [27], RefCOCOg [53] and ViP-Bench [5]: (1) its inclusion of evaluation data for both images and videos, pioneered the evaluation in video LMMs; and (2) it supports both open-ended and multiple-choice formats, enabling comprehensive evaluation.
2.4 Instruction tuning with explicit visual prompt
Architecture. We adopt the widely-used LLa VA-NNXT [44] architecture evaluate the effectiveness of our INST-IT. We train our model under an image-video joint training pipeline, where we mix our INST-IT Dataset with the open-source LLLVA-NeXT-DDNS
[48]. For single-image samples, we follow the original AnyRes paradigm [44] to split and encode sub-images according to the aspect ratio. For video and multi-image data, we batch the samples together, encode them, and flatten them into a sequence. Additionally, we apply 2 x 2 spatial pooling to reduce the number of visual tokens the video inputs. More details are in Sec. 3.
Converting INST-IT Dataset into instruction tuning format. INST-ITT Dataset provides annotations multiple levels of granularity. For the instance- and image-level captions in Fig. 2(a), we use single frame as input and structure the task as a two-turn dialogue: the model is first prompted to describe all individual instances, followed by holistic description of the entire scene. To capture temporal dynamics, we use temporal captions from Fig. 2(a), asking the model to describe the differences between two consecutive frames. The video-level description in Fig. 2(b) is treated as a captioning task, where the model is instructed to generate a summary based on all video frames. For the open-ended QA pairs in Fig. 2(c), we organize them into a multi-turn conversation, with the model answering one question per turn. In total, we construct 243k instruction tuning samples in the form of single-turn and multi-turn dialogues. All images and video frames are augmented with SoM visual prompts to explicitly provide instance-level guidance.
5

---

Table 2: Results on INST-IT Bench. We conduct evaluations on INST-IT Bench, including state-of- the-art open-source image models, video models, and cutting-edge proprietary models. #IT indicates the number of training samples used during the instruction-tunie stage. N/A indicates that the number is unknown. OE and MC represent open-ended and multiple-choice evaluations, respectively.
Image
Video
Model
LLM
#IT
OE Q&A
MC Q&A
OE Q&A
MC Q&A
Random Guess
N/A
25.0
25.0
GPT-4o [61]
N/A
84.8
65.5
81.0
Gemini-1.5 1.5-flash [72]
N/A
65.3
79.5
$7.9
75.8
Open-soure image models
LLaVA-1.5 [43]
Vicuna- 7B
665K
41.6
32.1
ViP-LLaVA [5]
Vicuna-7B
~v -12M
42.1
29.2
SoM-LLaVA [86]
Vicuna-77 7B
695K
45.1
40.0
L.aVA-NeXT [44]
Vicuna-78
765K
46.0
42.4
Open-source video models
LLaVA-NeXT-Video [103]
Vicuna-7B
860K
46.5
39.5
25.8
24.8
ShareGPT4Video [10]
Llama3-8B
~1.0
43.2
48.7
16.1
LLaVA-OV (SI) [31]
Qwen2-7B
7
61.8
31.4
36.4
LLaVA-OV [31]
Qwen2-7B
~
48.0
71.7
33.2
45.6
LLaVA- Video [104]
Qwen2-7B
~ ~7.4M
45.1
67.0
34.1
53.2
InternVI V1,2 [13]
Internl .22.57BB
N/A
58.6
66.5
39.8
45.5
Qwen2-VL- VL-Instrct [82]
Qwen2-7B
N/A
48.3
64.9
38.2
59.4
[82]
Qwen2-72B
N/A
55.5
74.7
45.5
74.6
Our models
LLaVA-NeXT-INST-II
Vicuna-7B
920K
68.6
63.0
49.3
42.1
-EXT--NETTIISS-IIT
Qwen2-7B
920K
75.3
45.7
53.3
3 Experiments
3.1 Implementation details
We use LLaVA-NeXT [44] as our baseline due its widespread adoption. In the default configuration, Vicuna-1.5-7B [16] serves as the language model with CLIP- ViT-336 [67] as the vision encoder. We utilize the AdamW [49] with cosine learning rate schedule for optimization. During the vision- language alignment stage, we use the LCS-558K dataset [43]. and for the supervised fine-tuning stage, we leverage the open-source L.aVA--NeX--DAM image into up to sub- sub-images based on its aspect ratio following the AnyRes [44] approach, and then concatenate the global image with these sub-images. For multiple images and video inputs, we skip the AnyRes procedure and encode every single image. Additionally, we apply 2 x 2 spatial pooling to reduce the number of visual tokens video inputs. We limit the maximum number of frames to 32 and the context length of LLMs 6K due to GPU memory constraints. To enhance instance-level understanding with our INST-IT Dataset, we combine INST-IT Dataset with VA-Nett in additional continuous supervised fine-tuning stage. In this stage, freeze the first 12 layers of the vision encoder to miti mitigate gate potential distribution shifts caused by visually prompted images. Furthermore, use Qwen2- 7B [87] with SigLIP-S0400R
[48]. For single images, we split the original
[97] for improved performance in main experiment, and Qwen2-1.5B with VITP-VVT-36 for efficiency in our ablation study. We use 8x H100 for all experiments. The image- video joint training stage takes approximately 20 hours when using Vicuna-7B 7B as the language model and hours using Qwen2-78 with SigLIP-SO400M-
3.2 Main experiments
Results on INST-IT Bench. We conduct extensive evaluations on INST-IT Bench. The results in Tab. 2 show that with instruction tuning using INST-IT Dataset, our models achieve a significant improvement of nearly 20% on average score, validating the effectiveness of INST-IT. Moreover, although ViP-LLa VA [5] utilizes visual prompts for instruction tuning, it shows minor improvement over its baseline, iee. VA-1.5 [43], possibly due to overfitting to its training data. contrast, our model demonstrates consistent improvements
other instance understanding benchmarks, such as ViP-Bench [5] and RefCOCOg [53) (Sec. 3.3), as well as on general- pprrpsse evaluation sets like A12D and Egoschema (will be discussed in the following sections). This suggests that the model

---

Table 3: Main results on image benchmarks.
A12D[28]
MMMU[95]
POPE[37]
GQA[26] ChartQA[54] (val)
Method
LLM
Vision Encoder
(test)
(val)
(test F!)
(test)
LLaVA-1.5 [43]
Vicuna-7B CLIP-ViT-Large
54.8
35.3
85.9
62.0
18.2
DeepStack-L [60]
Vicuna-7B CLIP-ViT-Large
35.7
86.7
63.1
21.0
DeepStack-L-HD [60]
Vicuna-7B CLIP-ViT-Lage
35.6
86.5
65.2
56.3
VII.A [42]
Vicuna-7B CI.I--ViT-.arge
85.5
62.3
LLvVA-OV (SI) [31]
Qwen2-7B SigLIP-SO4000
81.6
47.3
78.8
LLaVA-OV [31]
Qwen2-7B SigLIP-SO40OM
81.4
48.8
80.0
Dwvn2--LL-Isstee [82]
Qwen2-7B DEN-CLIP-H
83.0
54.1
83.0
LLaVA-NeXT [44] (baseline)
Vicuna-7B CLIP-ViT-Large
66.6
35.1
86.4
64.2
54.8
LLaVA-NeXT-INSS (ours)
Vicuna-7B CLIP-ViT-Lare
71.0
14.4
12.4
87.2
10.8
65.9
11.7
68.3
113.5
LLAVA-NeXT-INST-IT (ours)
Qwen2-7B SigLIP-SO400
78.7 112.1
42.7
17.6
87.6 10.2
65.5 11.
72.8 188.0
Table 4: Main results on video benchmarks. We report the average of MCQA, Y/N and CM in TempCompass for determinism results." indicates results reproduced by us.
ANerQA[94]
FguSchema[52] NEXTQA[85]
VideoMME((3) Tempaampas( (46
Method
LLM
Vision Encoder
(oe)
(subset)
(mc)
(w/o subs)
avg)
DeepStack-L [60]
Vicuna-7B
CTIP-NiT-Laree
49.3
38.4
61.0
Video ChaesPTTTTTTTSS
Vicuna-7B 7B
CLIP-VIT-Lagy ViT-Large
35.2
47.3
VideoLLaMA2 [14]
Vicuna-7B 7B
CLip-NTT-Lare
50.2
51.7
ALaVA.Next-Vidle [103]
Vicuna-7B
CLIP ELIP-NIT Large
53.5
43.9
46.5
IntemVVL2[13] [13)
IntemLM-7B
54.0
LLaVA-OV [31]
Qwen2-78
SigLIP-
56.6
60.1
79.4
58.2
69.4
LLaVA- Video [104]
Qwen2-7B
SigLIP: S0400M
56.5
57.3
83.2
63.3
QwenZ-VP VL-Inntruct [82]
Qwen2-7B
DEN WEN-CLIP-H
66.7
63.1
72.9
LLxVA-NeXT [44] (baseline)
Vicuna-7B
CLIP- Largee
LLaVA-NeXT-NNS--I (ours)
Vicuna-7B
CLIP-NIT LiP--e
53.7 J0.1
57.8 17.8
70.2 1118
44.3 18.1
59.8 13.0
LLaA--NeTTIIss--II (ours)
Qwen2-7B
SigLIP-S0400
55.2 71.4
50.4 10.4
73.0 (14.6
$4.0 117.8
63.9 17.1
trained with INST-IT generalizes well other tasks. Qwen2VL-72B does not show substantial improvements over its smaller 7B model, indicating that simply scaling up the model size canmot address the challenges in instance understanding. Similarly, by comparing the amount of instruction tuning data used by each model, we observe that large-scale coarse- coarse-grained annotations do not lead to essential improvements either. This highlights the importance of instance-specifie annotated data.
Results on generic benchmarks. To evaluate general understanding capabilities, we assess our mod- els several widely used image and video benchmarks using the L.Mss-Evll [100]. To ensure fair comparison with other models, we primarily report results from their original papers or reproduced results in previous studies. On generic image benchmarks, shown in Tab. INST-IT consistently outperforms our direct baseline model, i.e. LLaVA-NeXT. The improvement in AI2D, a benchmark that requires grounding and referring understanding capability, is particularly clear. This suggests that INST-IT effectively boosts the model in fine-grained understanding. Furthermore, when utilizing a more advanced language model and vision encoder, our method achieves performance comparable large-scale SFT LMMs, such as LLaVA-OV and Qwen2-VI--Intnuu while requiring significantly less computational and data cost. For video understanding benchmarks in Tab. 4, INST-IT signifi- cantly outperforms both LLa VA-NeXT and LLaVA-NceT--iuu These consistent improvements demonstrate that enhancing instance-level understanding through explicit visual prompted instruction tuning is an effective strategy for improving generic spatiotemporal understanding capabilities.
3.3 Evaluation on other instance-understanddng benchmarks
To assess whether our model has learned generalizable instance understanding capability, we con- ducted evaluations out-of-domain instance understanding benchmarks in zero-shot manner.
a region-level understanding benchmark that closely aligns with the objectives of INST-IT. As shown in Tab. 5, our model exhibits strong generalization performance. In particular, our INST-IT with Vicuna-7B achieves performance comparable to ViP-LLaVA when using rectangular bounding boxes as visual prompts and even surpasses ViP-LLaVA when employing human-style visual prompts. Notably, our model performs as a generalist under zero-shot evaluation, whereas ViP-LLaVA benefits from in-domain tuning, since it is tine-tuned on the dataset of ViP-Bench.
ViP-Bench [5]
7

---

Table 5: Results on ViP-Bench. We perform evaluation with our INST-IT models without fine-tuning.
Synthesized visual prompts Math
Visual prompts from human
Model
Rec
OCR
Know
Rel
Lang
All
Rec
OCR
Know
Math
Rel
Lang
All
hi [1]
58.1
69.8
59.5
71.0
61.4
51.9
60.7
56.9
69.7
63.7
80.6
61.1
45.6
59.9
GoT- [1]
53.2
50.3
55.6
67.7
57.5
57.5
52.8
51.7
50.3
59.3
60.3
55.0
43.8
51.4
Instruc1BLIP-7B (17]
36.9
16.3
34.2
22.3
26.8
7.5
31.7
38.9
17
35.4
9.7
29.3
17.5
33.3
Shikra-7B [8]
40.2
28.0
3.5
18.9
20.6
33.7
GPT4RÓI-78 [101]
35.6
16.7
29.7
13.8
35.1
Kosmos-2 [63]
29.5
14.2
18.5
9.7
21.9
26,9
LLaVA-1.5-7B [43]
50.8
12.4
49.2
6.5
51.8
23.8
41.6
49.1
13.0
42.9
9.7
50.0
27.5
40.2
Qwen-VL-Chat [4]
43.0
30.4
40.2
9.7
25.7
28.7
39.2
48.7
22.1
41.2
6.5
48.2
25.0
41.7
ViP-LLaVA-7B [5]
54.8
18.8
52.9
9.7
53.9
42.5
45.5
55.3
17.6
45.9
8.1
44.6
33.1
46.8
LLaVA-NeXT-INST-III Vicuna---
51.3
23.7
54.2
12.9
64.3
46.2
45.1
55.0
21.3
52.5
16.1
57.5
40.6
48.2
58.9
24.5
48.5
12.9
48.2
46.3
50.5
57.7
22.5
53.2
19.4
53.6
45.0
49.0
RefCOCOg [53] is referring expression comprehension benchmark, with fewer labeling errors than its counterpart RefCOCO [27]. We evaluate our -AAA-NAT--NNS--I--VEEE benchmark and observe clear improvement of 10.8% over the baseline LLaVA-NeXT L----NeeT-ELI M-XT-VVucn--8 (63.0% vs. 52.2%). This further confirms that our approach effectively enhances the model in instance understanding, rather than simply overfitting to our INST-IT data format.
7B model on this
3.4 Ablation study
We use Qwen2-1.55 [87] as the language model and CLIP- VIT-L-3- [68] as the vision encoder for ablation experiments. We first conduct ablation on the training recipe investigate how to effectively integrate INST-IT Dataset with existing academic SFT datasets [48] for balanced improvement. Next, we perform a detailed analysis of the impact of each component in our INST-IT Dataset.
Effectiveness of our continuous instruction-tunine paradigm. As shown in Tab. 6, directly mixing the video split of IT Dataset with -LaVA-Nex--DATI leads to significant improvements video benchmarks. However, the performance on generic image understanding slightly declines. We believe this due two main reasons: (1) the increased ratio of video data may suppress image understanding; (2) visually prompted images may introduce a distribution shift from natural images, To address these issues, we propose a continuous SFT paradigm based on single-image models and freeze the first 12 layers of the vision encoder to preserve realistic low-level features. Our model achieves balanced performance across both image and video benchmarks with this training approach.
Detailed dataset combination. As illustrated in Fig. 2, INST-IT Dataset contains fine-grained annotations multi-level. To investigate the effectiveness of each component in INST-IT Dataset, we conduct an extensive ablation by progressively adding data components. As shown in Tab. 7, the instance-level and image-level frame captions are essential for improving instance understanding in images. Mcanwhile, temporal differences, along with video-level descriptions and QA, sig- nificantly enhance video instance understanding. Finally, incorporating the image component of INST-IT Dataset enables our model to achieve the most balanced performance across generic image and video understanding benchmarks, as well as our INST-IT Bench.
4 Related Work
Large multimodal models. Recently, significant progress has been witnessed in LMMs [91]. BLIP-
2 [34] and Flamingo [2] leverage visual re-samplers to integrate image features language inputs by extracting fixed number of visual tokens. LLaVA [45] and its follow-ups [43, 31, 42, 57, 98, 60, 11] achieve remarkable success by connecting vision and language through a simple projection module. Additionally, researchers are extending LMMs' capabilities to temporal understanding by incorporat- ing multi-frame inputs [41, 82, 104] or explicit temporal modules [39, 25) However, existing LMMs struggle with instance-level understanding and often fail to accurately follow instructions to ground specific instances. We emphasize the importance of instance understanding and enhance through instruction fine-tuning with explicit visual prompts.

---

Table 6: Ablation on data training recipe. L.N. denotes LLaVA-NeXT-DDat while INST-IT img and INST-IT vid refer to the image and video subsets of INST-IT. INST-IT-I and INST-IT-V indicate the multi-choice splits of the image and video part of our INST-IT Bench, respectively.
A12D
POPE
GQA INST-IT-I
Next-QA VideoMME INST-IT-V
CL Tune Enc Data Combination
(test)
(test FI) (val)
(mc)
(mc)
(w/o subt)
(mc)
All
L.N.
61.1
86.9
61.4
45.3
56.6
45.7
31.3
All
L.N. & INST-IT vid
60.7
86.1
61.2
60.7
59.7
47.1
43.0
V All
L.N. & INST-IT vid
62.3
86.7
62.9
61.8
62.4
46.7
44.4
V None
L.N. & INST-IT rid
63.1
62.5
60.2
47.2
44.3
V Last 12
L.N. & INST-IT
63.2
87.0
62.5
60.1
63.3
47.2
44.0
None
L.N. & INST-IT Song vidd
63.0
87.0
62.7
58.6
59.8
46.7
41.6
V Last 12
L.N. & INST-IT img-uidd
63.0
87.2
62.7
59.6
64.3
46.6
43.7
Table Ablation on detailed data combination. The dataset combination in line #3 corresponds to the video part of INST-IT Dataset, while line #4 represents the complete INST-IT Dataset by incorporating the image part into line #3.
AI2D
MMMU
POPE (F1)
GQA (val)
INST-IT-I
Next-QA (mc)
VideoMMME
INST-IT-V
Data Combination
(test)
(val)
(mc)
(w/o subt)
(me)
LLaVA-NeXT
35.9
86.9
61.4
45.3
56.6
31.3
+ inst-cap & img-cap
63.0
35.1
86.1
62.7
58.9
62.4
46.0
33.8
+ temporal diff
63.0
35.6
87.1
62.7
59.6
64.2
45.6
36.9
vidco-description & qu
63.2
34.9
87.0
62.5
60.1
63.3
47.2
44.0
+ INST-IT Dataset king
36.1
87.2
62.7
59.6
64.3
46.6
43.7
Multimodal datasets and benchmarks. With the rapid progress LMMs, numerous instruction- tuning datasets have been developed. boxes, and image-level captions to generate diverse visual instruction tuning data. Follow-up studies more powerful models to generate synthetic data [9, 81, and improve the annotation pipeline [36, 10, 104]. Simultaneously. various benchmarks are proposed to evaluate LMMs across different aspects [21, 35, 40], such comprehensive understanding [30], OCR [54, 56, 55, 78], temporal understanding [20, 52, 85, 6, 46, 47], and instruction-followie
[45] leverages object categories, bounding
[66]. However, they focus more image video-level understanding and lack fine-grained emphasis on specific instances. We emphasize the importance of instance understanding in both images and videos, and propose the INST-IT Bench to evaluate the instance understanding LMMs and create the INST-IT Dataset, providing detailed instance-level annotations to enhance instance understanding.
Multimodal instance understanding. Understanding individual instances is central focus in computer vision community, with key tasks like object detection [73, 71, 12], instance segmenta- tion [74, 29], and object tracking [19, 50, 90]. In the of LMMs,, instance understanding gains increasing attention. SPEC [62], ARO[96], and Winoground [77] reveal that CLIP [68] struggle to
understand instances. To address this, KOSMOS-2 [64], Ferret [92], GLaMM [69] and Shikra [8] encode instance information in textual form. In parallel, SoM-LLaVA [86], RegionGPT [24],
GPT4ROI [101], MG-LLaVA [105], OMG-LLaVA [102], and ViP-LLaVA [5], explores the use of visual prompting to guide models in focusing on specific instances. SoM-LLaVA [86] and Ely- sium [80] are closely related ours. SoM-LLa VA [86] asks models to list the instances images, finding this effective in enhancing model comprehension. However, it is limited to the image domain. Elysium [80] focuses on object understanding videos but employs relatively simplistic instance annotations. In contrast, we focus on both images and videos and provide multi-level fine- fine-grained annotations for instances, aiming advance multimodal models in understanding the spatiotemporal dynamics of individual instances.
Conclusion
Instance understanding that detects, segments, and reasons nuanced relationships among objects has long been the goal of computer vision research, yet limited effort has been made to equip LMMs

---

with such capabilities. We introduced INST-IT Bench, a carefully curated benchmark for evaluating multimodal instance understanding abilities. Extensive evaluations for a wide range of models demonstrate the limitations of current models for understanding at the instance level. To mitigate this issue, we collected INST-IT' Dataset, the first tatructtion-uuiig dataset with explicit instance-level visual prompts and annotations. Based on INST-IT Dataset, we proposed INST-IT, continuous finetuning framework that excels in instance understanding and general comprehension.
Acknowledgement This work was supported in part by the National Natural Science Foundation of China (Grant 62472098) and the Science and Technology Commission of Shanghai Municipality (No. 24311103100).
References
[1] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, Altman, Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 087774 2023. 8
[2] J.-B. Alayrae, J. Donahue, Luc. A. Miech, I. Barr. Y. Hasson, K. Lene, A. Mensch, Millican. M. Reynolds, et al. Flamingo: a visual language model for few-shot learning. In NewrIPS, 2022. 8
[3] A. Athar, J. Luiten, P. Voigtlaender, T. Khurana, A. Dave, B. Leibe, and D. Ramanan, Burst: A benchmark for unifying object recognition, segmentation and tracking in video. In WACY, 2023. 4, 19
[4] J. Bai, S. Bai, S. Yang, S. Wang, S. Tan, P. Wang. J. Lin, C. Zhou, and J. Zhou. Qwen-vi: A versa- tile vision-language model for understanding, localization, reading, and beyond. arXiv preprint arXiv:2308. 12966, 1(2):, 2023. 8
[5] M. Cai, H. Liu, S. K. Mustikovela, G. P. Meyer, Y. Chai, D. Park, and Y. J. Lee, Making large multimodal models understand arbitrary visual prompts. In CVPR, 2024. 2, 2,4, 5, 6, 7, 8,9
[6] M. Cai, R. Tan. J. Zhang. B. Zou. K. Zhang. F. Yao, F, Zhu, J. Gu. Y. Zhong, Y. Shang, et al. Temporal- bench: Benchmarking fine-grained temporal understanding for multimodal video models. arXiv preprint arXiv:2410. 108/8, 2024. 9
[7] G. H. Chen, S. Chen, Zhang, Chen, X. Wo, Zhang, Chen, J. Li, Wan, and B. Wang, Allava: Harnessing synthssyt data for a lite vision-languse model, 2024, 9
[8] K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint (tt::2306.119933 2023. 8,9
[9] L. Chen, J. Li, X. Dong. P. Zhang, C. He, J. Wang. F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. In ECCV, 2024. 1,9
[10] I. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y. Zang, 7. Chen, H. Duan, B. Lin, 7. Tang. I. Yuan, Y. Qiao, D. Lin, F. Zhao, and J. Wang. Sharegpt4video: Improving video understanding and generation with better captions. In NeurlPS, 2024. 4, 5, 6,
[11] Y. Chen,, L. Meng, W. Peng. Z. Wu, and Y.-G. Jiang. Comp: Continual multimodal pre-training for vision foundation models. ArXiv, 2025, 8
[12] Y. Chen, W. Yao, I. Meng, S. Wu, Z. Wu, and Y.-G. Jiang. Comprehensive multi-modal prototypes are simple and effective classifiers for vast-vocabulary object detection. In AAAI, 2025. 9
[13] Chen, J. Wu, W. Wang, W. Su, G. Chen, Xing, M. Zhong, Q Zhang, X. Zhu, L. Lu, B. Li, Luo, T. Lu, Y. Qiao, and J. Dai. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. arXiv preprint arXiv:2312. 14238, 2023. 1, 6,7
[14] Z. Cheng, S. Leng, H. Zhang, Y. Xin, X. Li, G. Chen, Y. Zhu, W. Zhang, Z. Luo. D. Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint 2xxx:2406000776 2024.
[15] C.-H. Chiang and H.-y. Lec. Can large language models be an alternative to human evaluations? In ACL, 2023. 18
[16] W.L.. Chiang, Z, Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang. L. Zheng, S. Zhuang. Y. Zhuang. J. E. Gonzalez, I. Stoica, and E, P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. 6
10

---

[17] W. Dai. J. Li, D. Li, A. M. H. Tiong. J. Zhao, W. Wang. B. A. Li, P. Fung, and S. C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. In NewrlPS, 2023. 1,8
[18] A. Dave, T. Khurana, P. Tokmakov, C. Schmid, and D. Ramanan. Tao: A large-scale henchmark for tracking any object. In ECCV, 2020.
[19] P. Dendorfer, A. Osep, A. Milan, K. Schindler, D. Cremers, I. D. Reid, S. Roth, and L. Leal-Taixe. Motchallenge; A benehmark for single-camera multiple target tracking. IJC,, 129, 2020. 1,9
[20] Fu. Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang. C. Zhou, Y. Shen, M. Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms video analysis. arXiv preprint arXiv:2405.21075, 2024. 7, 9
[21] C. Fu, Y.-F, Zhang. S. Yin, Li, X. Fang, S. Zhao, H. Duan, X. Sun, Z. Liu, L. Wang, et al. Mme-survey: A comprehensive survey on evaluation of multimodal llss. arXiv preprint arXiv:241 2024. 9
[22] Fu, R. Wang, Y. Fu, D. Pani Paudel, X. Huang, and L. Van Gool. Objectrelator: Enabling cross-view object relation understanding in ego-centric and exo-centric videos. In ICCV, 2025. 1
[23] Y. Fu, Y. Wang. Y. Pan, L. Huai, X. Qiu, Z. Shangguan, T. Liu, Y. Fu, L. Van Gool, and X. Jiang. Cross-domain few-shot object detection via enhanced open-set object detector. In ECCV, 2024. 1
[24] Q. Guo, S. D. Mello, H. Yin, W. Byeon, K. C. Cheung, Y. Yu, P. Luo, and S. Liu. Regiongpt: Towards region understanding vision language model. In CVPR, 2024. 2,
[25] B. Hc, H. Li, Y. K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S.-N. Lim. Ma-Imm: Memory- augmented large multimodal model for long-term video understanding. In CVPR, 2024. 8
[26] D. A. Hudson and C. D. Manning, Gqa: A new dataset for real-world visual reasoning and compositional question answering, In CVPR, 2019. 7
[27] S. Kazemzadeh, V. Ordonez, M. Matten, and T. Berg. Referitgame: Referring to objects in photographs of natural scenes. In EMNLP, 2014. 5,8
[28] A. Kembhavi, M. Salvato, E. Kolve, M. Seo, H. Hajishizzi, and A. Farhadi. A diagram is worth a dozen images. In ECCV, 2016. 2,7
[29] A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-.. Lo, al. Segment anything. In ICCV, 2023. 1, 4, 9, 18, 19
[30] B. Li, R. Wang. G. Wang. Y. Ge. Y. Ge, and Y. Shan, Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[31] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, Y. Li, Z. Liu, and C. Li. Llava-onevision: 2024. 6, 7, 8
Easy visual lask transfer. arXiv preprint arXiv:2408.0333
[32] Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, Gao, et al. Multimodal foundation models: From specialists to general-purposs assistants. FTCGV, 2024.
[33] Li, H. Zhang, P. Sun, X. Zou, S. Liu, J. Yang. C. L. L. Zhang, and J. Gao. Semantic-sam: Segment and recognize anything at any granularity. In ECCV, 2024, 19
[34] J. L., D. Li, S. Savarese, and Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In JCML, 2023.
[35] J. Li, W. Lu, H. Fei, M. Luo, M. Dai, M. Xia, Y. Jin, Z. Gan, D. Qi C. Fu, et al. A survey on benchmarks of multimodal large language models. arXiv preprint 2rXiv:2408.08633 2024.
[36] X. Li, F. Zhang, H. Diao, Y. Wang, X. Wang, and L.-Y. Duan. Densefusion- Im: Merging vision experts for comprehensive multimodal perception. In NewrlPS, 2024.
[37] Y. Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J.RR. Wen. Evaluating object hallucination in large vision- language models. arXiv preprint arXiv:2305. 10355, 2023. 7
[38] Y. Li, Q. Li, H. Wang, X. Ma, Yao, S. Dong, H. Fan, and L. Zhang. Beyond mot: Semantic multi-object tracking. In ECCV, 2024. 4, 19
[39] Y. Li, C. Wang, and J. Jia. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024. 8

---

[40] Z. X. Wu, H. Du, H. Nghiem, and G. Shi. Benchmark evaluations, applications, and challenges of large vision language models: A survey. arXiv preprint urXiv:2501.02189, 2025. 9
[41] B. Lin, B. Zhu, Y. Ye, M. Ning. P. Jin, and L. Yuan. Video-Ilava: Learning united visual representation by alignment before projection. In EMNLP, 2024. 8
[42] Lin, H. Yin, W. Ping. P. Molchanov, M. Shoeybi, and S. Han. Vila: On pre-training for visual language models. In CVPR, 2024. 7, 8
[43] H. Liu, C. Li, Y. Li, and J. Lee. Improved baselines with visual instruction tuning. In CVPR, 2024. 1, 6, 7,8
[44] H. Liu, C. L.. Li, B. Li. Y. Zhang, S. Shen, and Y. J. Lee. Llava-next: Improved reasoning, ocr, and world knowledge, 2024. 5, 6,
[45] H. Liu, C. Li, Q. Wu, and Y. Lec. Visual instruction tuning. In NeurlPS, 2023. 1, 8,
[46] Y. Liu, S. Li, Y. Liu, Y. Wang. S. Ren, L. Li. S. Chen, X. Sun, and L. Hou. Tempcompass: Do video Ilms really understand videos? arXiv preprint arXiv:2403.0047 2024. 7,9
[47] Y. Liu, Z. Ma, Z. Qi. Y. Wu, Shan, and C. W. Chen. Et beneh: Towards open-ended event-level video-language understanding. In NearlPS, 2024. 9
[48] Imms lab. Llava-next-data, 2024. 5, 6, 8
[49] I. Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05100 2017.
[50] W. Luo, J. Xing, A. Milan, X. Zhang, W. Liu, and T.K.. Kim. Multiple object tracking: A literature review, AI, 2021.
[51] M. Maaz, H. Rasheed, S. Khan, and F. S. Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In ACI, 2024. 1,7
[52] K. Mangalam, R. Akshulakov, and J. Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. In NeurlPS, 2023. 7,7,9
[53] J. Mao, J. Huang, A. Toshev, O. Camburu, A. L. Yuille, and K. Murphy. Generation and comprchension of unambiguous object descriptions. In CVPR, 2016. 6,
[54] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and F. Hoque Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint 22214:2203.10244, 2022. 7.9
[55] M. Mathew, V. Bagal, R. Tito, D. Karatzas, E, Valveny, and C. Jawahar. Infographicvga. In WACV, 2022.
[56] M. Mathew, D. Karatzas, and C. Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021.
[57] B. McKinzie, Z. Gan, J.PP. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, F. Peng. A. Belyi, al. Mml: methods, analysis and insights from multimodal 11m pre-training. In ECCV, 2025. 8
[58] L. Meng. X. Dai. Y. Chen, P. Zhang, D. Chen, M. Liu, J. Wang, Z. Wu, L. Yuan, and Y.-G. Jiang. Detection hub: Unifying object detection datascts via query adaptation on language embedding. In CVPR, 2023. 1
[59] L. Meng, S. Lan, H. Li, J. M. Álvarez, Z. Wu, and Y.-G. Jiang. correspondence for in-context segmentation. In ECCV, 2024.
Segie: Unleashing the emergent
[60] L. Meng. J. Yang, R. Tian, X. Dai, Z. Wu, J. Gao, and Y.-G. Jiang. Deepstack: Deeply stacking visual tokens surprisingly simple and effective for Imms. NeurIPS, 2024, 1, 7,8
[61] OpenAL. GPT-4o system card, 2024. 2,
[62] W. Peng, S. Xie, Z. You, S. Lan, and Z. Wu, Synthesize diagnose and optimize: Towards fine- tne-grained vision -anguage understanding. In CVPR, 2024. 9
[63] Z. Peng. W. Wang, L. Dong, Y. Hao, S. Huang, S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint ar3iv:2306. 14824, 2023. 8
[64] Z. Peng, W. Wang, L. Dong, Y. Hao, S. Huang. S. Ma, and F. Wei. Kosmos-2: Grounding multimodal large language models to the world. ArXiv, abs/2306. 00822306..4824, 2023. 9
12

---

[65] J. Qi, Y. Gao, Y. Hu, X. Wang. X. Liu, X. Bai. S. Belongie, A. Yuille. P. H. Torr, and S. Bai. Occluded video instance segmentation: A benchmark. IJC,, 2022. 4, 19
[66] Y. Qian. H. Ye, J.PP. Fauconnier, P. Grasch, Y. Yang, and Z. Gan, Mia-bench: Towards better instruction following evaluation of multimodal llms. arXiv preprint arXiv:440..01500 2024. 9
[67] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021. 6
[68] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transfcrable visual models from natural language supervision. In ICML, 2021. 8,
[69] H. Rasheed, M. Maaz, S. Shaji, A. Shaker, S. Khan, H. Cholakkal, R. M. Anwer, E, Xing, M.-H. Yang, and F. S. Khan. Glamm: Pixel grounding large multimodal model. In CVPR, 2024. 9
[70] N. Ravi, V. Gabeur, Y.-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.000714 2024. 20
[71] J. Redmon. You only look once: Unified, real-time object detection. In CVPR, 2016. 1,9
[72] M. Reid, N. Savinov, D. Teplyashin, D. Lepikhin, T. P. Lillicrap, J.-B. Alayrac, R. Soricut, A. Lazaridou, O. Firat, J. Schrittieser. I. Antonoglou, R. Anil, S. Borgeaud, A. M. Dai, Millican. Dyer. M. Glaese, T. Sottiaux, B. Lee, F. Viola, M. Reynolds. Y. Xu, J. Molloy. J. Chen, M. Isard, P. Barham, T. Hennigan, R. Mcllroy, M. Johnson, J. Schalkwyk. E. Collins, E. Rutherford, E. Moreira, K. W. Ayoub, M. Goel, C. Meyer, G. Thomton, Z. Yang, H. Michalewski, Z. Abbas, and Nathan Schucher. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. ArXiv preprint, 2024. 6
[73] S. Ren, K. He, R. B. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. TPAMI, 2015. 1,9
[74] O. Ronneberger, P. Fischer, and T. Brox. U-net: Convolutional networks for biomedical image segmenta- tion. ArXiv, 2015. 1.9
[75] X. Shang, D. Di, J. Xiao, Y. Cao, X. Yang, and T.-S. Chua. Annotating objects and relations in user- generated videos. In JCMR, 2019. 4, 19
[76] Y. Tan, Z. Wu, Y. Fu, Z. Zhou, G. Sun, E. Zamfi, C. Ma. D. P. Paudel, L. Van Gool, and R. Timofte. Xtrack: Multimodal training boosts rgb-x video object trackers. ArXiv, 2024.
[77] T. Thrush, R. Jiang, M. Bartolo, A. Singh, A. Williams, D. Kiela, and C. Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022. 9
[78] R. Tito, D. Karatzas, and E. Valveny. Hierarchical multimodal transformers for multipage doovqa. Puttern Recognition, 2023. 9
[79] H. Wang, C. Yan, K. Chen, X. Jiang, X. Tang, Hu, G. Kang, W. Xie, and E. Gavves. Ov-vis: video instance segmentation. IJV,, 2024, 4, 19
[80] H. Wang, Y. Ye, Y. Wang, Y. Nie, and Huang. Elysium: Exploring object-level perception in videos via mllm. In ECCV, 2024.
[81] J. Wang, L. Meng, Z. Weng, B. He, Z. Wu, and Y.G.. Jiang. To see is believe: Prompting gpt-4v for better visual instruction tuning, arXiv preprint arXiv:2311.072 2023, 9
[82] P. Wang, S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and Lin. Qwen2-vl: Enhancing vision- -aaigun- model's 's perception the world at any resolution. arXiv preprint arXiv:2409. /2191, 2024. 1, 6, 7, 8
[83] W. Wang. M. Feiszli, H. Wang. and D. Tran. Unidentified video objects: A benchmark for dense. worn-world segmentation. In ICCV, 2021. 4, 19
[84] Z. Wu, Z. Weng, W. Peng, X. Yang, A. Li, L. S. Davis, and Y.-G. Jiang. Building open- open-vocabulary video clip model with better architectures, optimization and data. TPAMI, 2024. 1
[85] J. Xiao, X. Shang, A. Yao, and T.-S. Chua. Next-qa: Next phase of question-answering to explaining temporal actions. In CVPR, 2021. 2, 7,9
[86] A. Yan, Z. Yang, J. Wu, W. Zhu. J. Yang. L. Li. K. Lin, J. Wang, J. McAuley, J. Gao, et al. List items one by one: A new data source and learning paradigm for multimodal llms. In COLM, 2024. 6,
13

---

[87] A. Yang. B. Yang. B. Hui, B. Zheng, B. Yu. C. Zhou, C. Li, C. Li, D. Liu, F. Huang, G. Dong, H. Wei. H. Lin, J. Tang, Wang. Yang. J. Tu, J. Zhang, J. Ma, J. Xu. J. Zhou, J. Bai, J. He, Lin, K. Dang, Lu, K.-Y. Chen, K. Yang, M. Li, M. Xue, Ni, P. Zhang, Wang, R. Peng. R. Men, R. Gao, R. Lin, S. Wang, S. Bai, S. Tan, T. Zhu, T. Li, T. Liu, W. Ge, X. Deng, X. Zhou, X. Ren, X. Zhang, X. Wei,
X. Ren, Y. Fan, Y. Yao, Y. Zhang, Y. Wan, Y. Chu, Z. Cui, Z. Zhang, and Z.-W. Fan. Qwen2 technical report. ArXiv, abs/2407.10677 2024. 6, 8
[88] J. Yang, H. Zhang, F. Li, X. Zou, C. Li, and J. Gao. Sct-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint trXiv:2310.11441 2023, 2, 3, 4, 15, 19
[89] L. Yang, Y. Fan, and N. Xu. Video instance segmentation. In ICCV, 2019. 4, 19
[90] A. Yilmaz, O. Javed, and M. Shah. Object tracking: A survey. CSUR, 2006. 9
[91] Yin, Fu. Zhao, K. Li, X. Sun. T. Xu, and E. Chen, survey on multimodal large language models. National Science Review, 2024. 8
[92] H. You, Zhang, Z. Gan, X. Du, B. Zhang, Z. Wang, L. Cao, S.-E, Chang, and Yang. Ferret: Refer and ground anything anywhere at any granularity. In ICLR, 2024. 9
[93] W. Yu, Z. Yang. L. Li, J. Wang. Lin, Z. Liu, X. Wang, and L. Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In ICML, 2024. 18
[94] Yu, Xu, Yu, T. Yu, Z. Zhao, Zhuang, and D. Tao, Activitynet-qa: dataset for understanding complex web videos via question answering. In AAAI, 2019.
[95] X. Yue, Ni, K. Zhang, T. Zheng. R. Liu, G. Zhang. S. Stevens, D. Jiang, W. Ren, Y. Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.7 7
[96] M. Yuksekgonul, F. Bianchi, P. Kalluri, D. Jurafsky, and J. Y. Zou. When and why vision-language models behave like bags-of-wordd and what to do about it? In ICLR, 2023. 9
[97] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023. 6
[98] H. Zhang, M. Gao, Z. Gan, P. Dufter, N. Wenzel, F. Huang, D. Shah, X. Du, B. Zhang, Y. Li, et al. Mml.
5: Methods, analysis & insights from multimodal Il fine-tuning, arXiv preprint 2009 2024. 8
[99] H. Zhang, X. Li, and L. Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding, In EMNLP, 2023. 1
[100] K. Zhang, B. Li, P. Zhang, F. Pu, J. A. Cahyono, K. Hu, S. Liu, Y. Zhang, J. Yang, C. Li, al. Lmms-eval: Reality check the evaluation of large multimodal models. arXiv preprint arXiv:2407.12772, 2024.
[101] S. Zhang, P. Sun, S. Chen, M. Xiao, W. Shao, W. Zhang, Y. Liu, K. Chen, and P. Luo. Gpt4roi: Instruction tuning large language model region-of-interest arXiv preprint arXiv:2307.03601, 2023. 8.
[102] T. Zhang, X. Li, H. Fei, H. Yuan, S. Wu, S. Ji, C. L. Chen, and S. Yan. Omg-Ilava: Bridging image-level, object-level, pixel-level reasoning and understanding. In NeurlPS, 2024. 2, 9.
[103] Y. Zhang, B. Li, h. Liu, Y.j. Lee, L. Gui, D. Fu, J. Feng, Z. Liu, and C. Li. Llava-next: A strong zero-shot video understanding model, 2024. 6.
[104] Y. Zhang, J. Wu, W. Li, B. Li, 7. Ma, 7. Liu, and C. Li. Video instruction tuning with synthetic data. arXiv preprint 240.222400.02713,, 2024, 4,5, 6, 8,9
[105] X. Zhao, X. Li, H. Duan, H. Huang, Y. Li, K. Chen, and H. Yang. Mg-llava: Towards multi- gulti-granullrrity visual instruction tuning. ArXiv, 2024. 2,9
[106] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. ArXiv, 2023. 1
14

---

Appendix
In Sec. A, we outline additional implementation details of the GPT-4o-assisted data annotation pipeline,
In Sec. B, present further information about the instance understanding benchmark, INST- IT Bench.
In Sec. C, share more details about the instruction fine-tuning dataset, INST-IT Dataset.
In Sec. D, we provide more discussions on failure cases and real-world applications.
A
Data Annotation Pipeline
A.1 Set-of-Marks Visual Prompting
Performing instance-level annotations challenging, and we adopt the SoM visual prompting technique [88] to address this. Specifically, illustrated in Fig. 3, overlay numeric ID at the center of each instance and maintain the same ID for a given instance across all frames. This simple augmentation
explicitly guide GPT-4o focus more effectively the instances of interest, enabling finer-grained and more accurate annotations. Furthermore, segmentation masks necessary calculate the center coordinates of each instance. Details how these masks obtained are provided Sec. C.I.
B
Figure 3; Set-of-Marks visual prompting on the original videos. Each instance is assigned unique numeric ID, which remains consistent across all frames.
A.2 Prompting GPT-4o
Task prompt templates. Prompt engineering is crucial for enabling GPT-4o to accomplish specific tasks. In this section, we present the task prompts that we designed to prompt GPT-4o for data annotation:
The task prompt pf for frame-level annotation, Fig. 5.
The task prompt puid for video-level annotation, Fig. 6.
The task prompt pao for open-ended question-answer pairs generating, Fig.
GPT-4o API version. During the annotation process, we use the GPT--4--022-088-06 API and leverage its structured output functionality to facilitate output parsing, enabling the model respond in predefined ISON format.
More Details about INST-IT Bench
B.1 Negative Options Generation
We use the ground-truth from open-ended QA as the positive option and additionally craft three negative options, forming multiple-choiee question with four options. To create hard negatives,
15

---

Task Description
are expert evaluator tasked scoring accuracy of responses to open-ended questions. You will be provided with set of questions, each with corresponding ground-truth answer, as well as responses from a tester. Your job is to assess the accuracy of each response provide score between and
Guldelines:
Score Range: Your score for each test item must be between and 1. A higher score means more correctness. Choose from the following:
(completely incorrect), 0.1, 0.2,
0.5, 0.6, 0.7, 0.8, 1.0
icompletely correct)
for each test item, consider the question, the ground-truth answer, and the tester's response together to determine correctness.
answers may referenced using format [ID] (e.g.. [1]. [2]). Ensure that any objects referenced in the tester's response match correctly with the ground-truth answer.
Objects questions
Time points may be indirated with <timestamps (e.q., <1>), and timestappa-sent timestamp> Q.,
time intervals
<3>-<5>). Verify that the tester's response includes accurate time expressions.
# Input Format:
input
items scored, where item
includes:
id: the unique identifier for the test item;
question;
ground-truth answer for the question;
response
Now, let's begin the evaluation, here are the input test items:
<samples to be scoreds...
Figure 4: GPT-4o-based open-ended question answering correctness assessment. The underlined parts in the figure are included only when evaluating the video split, while the italicized parts will be replaced by the actual sample for scoring.
Frame-level Annotation
Description:
You are an expert video analysis, skilled at detecting dynamic changes between consecutive video frames, In this task, you are given two consecutive frames. Each image contains objects with unique numeric IDs treforned to "marked objects"), Your task to:
Provide object-level descriptions for marked objects the current. frame. image description
entire current Identify any dynemic changes or differences between the current frame and the previous frame.
and
# Guidelines for Object-level Descriptions:
Describe each marked ubject's
current frame. focus sitributes like shape, lextual, object
confident, specify calleoory
object belongs description, fee,
Guidelines Image-level Descriptions:
Mentton all marked objects the current frame, focusing behaviors, movements, states, positions, other dynamic information.
Describe the interactions between the objects, as wall as the bockground, enviranment, perspective, and anglo of the shot.
* Guidelines for Describe the Temporal Changes:
abject, any changes background. environment, Reasonebly infer the causes of the changes, trends, and possible impacts.
Highlight
actions, position, well interactions relatianships.
angles scene transitions.
# Constraints:
Accuracy ertical: if marked object too small or abscured, and you cannat confidently Identify skip without attempting describe Describe
direction, object position from Frame of interaction is, simply saying interacting with [21", you should say '[1] catching [2]"
Specify
Onject Referring Forma: When rafor to singla object, use the format: (11) when listing multiple objects, use the farmtt '[1] [2]'
# Structured Output:
Your output should have three sections:
Object-level Descriptions:
provide comprehensive description appearance. comprehenstve description entire image, capturing many details possiala. Temporal Changes: Outline any changes and differences compared to the previous trame, highlighting important transtions or events.
marked objects
image-lavel Description: Provide
# Input:
will
<the
Figure 5: Frame-level annotation task prompt, the italicized part are placeholders for the actual inputs.
16

---

capturing dynamic information for Video-level Annotation Task Prompt
# Task Description:
expert summarizinn content. Given a sequence these descriptions into en accurale, coleesive summery of the entire video.
by-frame descriptions
aggregete
# Guidelines and Rules:
Base your description solely on the input to ensure accuracy, avoid inferring any unmentiooned content.
Please nate that the description of single frame may contain some inaccuracies. You need to use the overall context fuuther correct these
errors, ensuring accurecy and coreistency.
Use chronelogical order: organize your summary according to the timest amp of the frames, follow these conventions: for specific moments,
write < -timestampp, e.a. <3>; for time intervals: write <start timestamps timestamps, during <5=-c7>
Referencing objects by ID: in your response, use the some [IO] format provided in the input reference onjects: for one object: [I] [e.g.. [8]
multiple objects: [ID]]
Output Requirements:
Your ouppuu should a dense, detailled and accurate description entire video, summarizing main objects, key events, and various
spetiel end event-related details.
# Input Format:
Each frame's descriptton includies four
chronologicel 2. Instance-level description: lists primary objects the frame using the format "[object ID]: object description"
Timestamp:
3. Freme-level description: offers comgrehersive view of the frame's content, covering main ubjects, object relationehips, and the
background or environment details.
Temporal change description: highlights key changes underslanding the video's progression.
capturing dynamic information essential
# Input Frame-leval Annotations:
Timestamp: <1>; Instance-level description i. Freme-level description: Temporal changes: None, this the first frame.
Timestamp: <Z>; instacesdevel description... Frame-lovel description: Temporal changes:...
Figure 6: Video-level annotation task prompt, the italicized part are placeholders for the actual inputs.
Open-Ended Generation Task Prompt
Task Description:
You are an expert in video content enelysis. this task. you will receive textual descriptions of individual video fremes, Your task generate high-quality and contextually coherent questions and accurate answers based on the content of the video.
# Guidelines:
Avoid speculative questions; ensure all questions can be answered from the frame descriptions.
Diversify types questions
whyt
The number of question-enswer pairs should between to 20, Uhis degends on how much valusble information contained in the video.
Be creative and nexiblle forming questions and answers, and avoid redundant or overly simple questions.
Use the frame timestamps express time in the video: for specific moment, use <timestamp> 9.g., at <3bx; for a time interval, use timestamps,. reter specific object: you can
<start
<5>> Don't
the
In input.
same tormat your output
specific objects:
for single objest, write [ID] (e.g.. '(8)") for multiple objects, use '[DDL] [ID2) ... such as "[3) (5]".
Output:
The output is list 10 to high-quality, context-aware question-answor answer pairs about the video's content.
Input Format:
cansists
by descriptions,
includes:
1. Timestamp: the chronological position of frame video.
2. Prome-laval description: offers a comprehersive view of the frame's contert, covering main objects, object relationships, and the background
environment detalls.
Temporal description: highlights key changes understanding the video's progression.
since
# Input Frame-level Annotations:
Timestamp:: <I>; Arame-lovel descrittio: i Temporal changes: None, as this is the first frame.
Timestamp: <2>; Fram-lleve descoption:... Temporal changes:
Figure 7: Open-ended question-answer pairs generation task prompt, the italicized part are placeholders for the actual inputs.
17

---

Open-Ended
the Open Ended
incividuals longer
action does score captured
reaching ground, indicating they are searching for picking up something.
from
and
Multiplle QA
Multiple-Choice
Q: Which individuals are no longer visible the later frames of video?
What action take score captured
the video were the man and the woman who were standing next to the
dusls who were longer visible
A. the scene captured at <100, [2] petting the horse.
8. [2] seen holding white cleth at scene captured at <10>
individuals
visible the
Petting the horse
[3] and
disappears from the frame at <5o, and (5) diseppears from the frame at <c>,
D.
reaching
indicating
they are searching for picking up something.
[3]
Figure 8: A data example from INST-IT Bench. Each test sample includes both open-ended QA and multiple-choice QA, focusing on specific instances or the relationships and interactions between instances.
first have the model answer the open-ended questions and use GPT-4o to score the correctness the responses. If the score is lower than 0.4, consider it difficult negative answer and include it as one of the negative options. Finally, we randomly shuffle the four options to ensure that the correct one appears in each position with equal probability.
B.2 LLM-based Evaluator for Open-Ended QA
Recent studies [93, 15] suggest that LLMs can serve as effective evaluators. Building on this, we GPT-4o to assess the accuracy of open-ended question answering. Specifically, GPT-4o assigns a score between 0 and based on three key factors: the question, the ground-truth answer, and the model prediction. Given that INST-IT Bench prioritizes instance-level understanding, attention to the accuracy of instance ID references. Furthermore, for the video split of INST-IT Bench, emphasize the correctness of timestamps to ensure temporal correctness. The task prompt for GPT-4o is illustrated in Fig. 4.
pay special
B.3
Data Example
To provide clearer understanding of INST-IT Bench, we present data example in Fig. 8. Each question includes both open-ended and multiple-choice formats, focusing on specific instances or exploring the relationships and interactions between multiple instances. This design highlights the significant distinction from other benchmarks, emphasizing fine-grained understanding at the instance level.
C More Details about INST-IT Dataset
C.1 Data Collection and Processing
Collection. We select five instance segmentation datasets and two multi-object tracking datasets as sources of video data. To prevent data leakage, we only used the training splits of these datasets, leaving their test and validation splits untouched. Additionally, we use the SA-1B [29] dataset as source of image data and only utilize the first ten officially provided data splits. For each split,
18

---

Table 8: Data sources. We use seven video datasets and one image dataset as our data sources. We show their annotation formats, the splits we used, and the number of samples from each dataset.
Dataset Name
Ann. Type
Split
Sample Num.
Video Instance Segmentation
BRUST [3]
mask
training
500
UVO [83]
mask
training
5,135
OVIS [65]
mask
training
599
LVVIS [79]
mask
training
3,057
Youtube [89]
mask
training
2,897
Video Object Tracking
BenSMOT [38]
box
training
2,261
VidOR [75]
box
training
6,969
Image
SA-1B [65]
none
1-10
51,101
only use the first 50% of its images. In total, we collect 21,418 videos and 51,101 images. Tab. 8 provides detailed statistics on our data sources.
Processing. When constructing SoM [88] visual prompts, we need to obtain the mask annotations for each instance to determine the location the numeric IDs. For the video instance segmentation datasets [3, 83, 65, 79, 89], the instance masks are already provided and can be used directly. For multi-object tracking datasets [38, 75], we prompt SAM [29] with their bounding box annotations to
generate instance masks. For images in the SA-1B dataset [29]. we employ Semantic-SAN [33] to segment the instances and obtain their masks.
C.2 Statistics Analysis.
Number of instances. The key characteristic of INST-IT Dataset is its specific focus individual instances in images and videos, which provides a more fine- amne-grained description of the visual inputs. We visualize the distribution of the number of instances in each sample in Fig. 9. For the video split, each sample has an average of 3.7 instances, with total of 79,709 instances. For the image split, each sample contains average of 6.9 instances, totaling 351,495 instances. Across the entire dataset, each sample includes an average of 5.9 instances, adding up to 431,204 instances in total, We measure the scene complicity by the number of instances in each sample. Specifically, 31 % of the samples contain < 3 instances (simple), 39% have between 3 to 8 instances (medium), and the remaining 30% contain instances (hard).
Dataset diversity. We visualize the object categories in INST-IT Dataset in Fig. 10, highlighting its diverse range. The objects include humans, animals, plants, vehicles, landmarks, etc. covering domains like daily life, egocentric perspectives, sports, transportation, etc. The rich diversity of INST-IT Dataset ensures its applicability real-world scenarios and enhances its transferability to different domains.
Text captions. INST-IT Dataset contains multi-level textual descriptions of visual content, covering instances, frames, temporal changes, and video-level annotations. We conduct statistical analysis on these text annotations, including the number of each type of text, and their average length. As shown in Tab, 9, the average length of INST-IT Dataset is 49.1 words per caption, with video-level averaging 323.2 words, highlighting its richness of details. We also present the results of lexical analysis in Tab. 9. The instance-level captions contain a rich variety of nouns and adjectives, indicating that they primarily describe the objects' categories and attributes. The captions of temporal changes include high volume of verbs and adverbs, suggesting that they capture dynamic information.
Human evaluation of data quality We invited three volunteers to rate each sample on a scale from to 5, with higher scores indicating better quality. Tab. 10 presents the scores of different types of annotations, along with the average time spent by each volunteer to evaluate each sample. The average score across all types is 4.49 10.05. indicating that the data in INST-IT Dataset of satisfactory quality.
19

---

Video Sp8t
Image S2R:
AB Sample
Dontes
0.20
fish
wa
0.15
a
Tope
perss
son
0.10
D
dog
bird
car
0.05
bottle
0.00
sky
Number
street
Stone
Figure 9: The distribution of the number of instances per sample in INST-IT Dataset. We separately present the distribution for the video split, image split, and the entire dataset.
Figure 10: Analysis of object categories in
INST-IT Dataset, which shows a diverse range of types spanning multiple domains.
Table 9: Statistical and lexical analysis of INST-IT Dataset. We present the results for each annotation level as well as the entire dataset.
Cuption Type
#Caption
#Char./Cap.
#Word/Cup.
#Sen//Cap.
Nouns
Adj.
Adv.
Verb.
Prep.
Instance-level
836,524
102.1
24.3
1.5
26.5%
13.3%
2.3%
12.3%
10.7%
Frame-level
207.662
458.0
106.5
5.7
25.2%
10.5%
2.6%
14.9%
11.5%
Temporal-change
135,143
306.6
67.7
3.7
21.2%
10.0%
6.0%
16.4%
10.8%
Video-level
21,372
1441.8
342.2
14.3
24.8%
10.6%
3.6%
13.2%
11.8%
All
1,200,701
210.5
49.1
2.7
25.0%
11.4%
3.1%
14.0%
11.1%
C.3 Data example.
In this section, provide complete video data sample from INST-IT Dataset to offer clearer understanding of its content and format. In all annotations, we use the format [ID] to refer to instances and <timestam>>> to refer timestamps. We present the frame-level annotations in Tab. 11. We can see that each frame-level annotation Yf consists of three parts: instance-level descriptions y' image-level descriptions ymm and temporal differences ydty- Additionally, each video is accompanied by series of open-ended question question-answer answer pairs Y*8, most of which center on specific instances or their relationships, illustrated in Tab. 12. Furthermore, we generate a dense video-level caption vid summarizing the entire video in chronological order, as shown in Tab. 13.
D
More discussions.
D.1 Failure cases.
We manually inspect the dataset and model to identify the failure cases. We find that occasional failures occur scenarios where instances are severely occluded, the image is blurry, or instances are excessively small or crowded. These challenges are common among LMMs, and future research can further investigate them.
D.2 Real-world applications.
In real-world applications, users can interactively prompt models like SAM2 [70] to automatically track instances of interest and generate SoMs. Additionally, our model also supports inputs without SoMs, allowing users to specify particular instances using textual descriptions. In the first scenario, our INST-IT introduces only a marginal overhead for generating SoMs, while in the second case, incurs no extra cost compared to the base model.
E Limitations and broader impacts.
Limitations. Our current experiments are conducted on 7B and 1.5B models due to the computation cost. Morcover, our current data pipeline is automated but constrained by the overhead of GPT-4o.
20

---

Table 10: Human evaluation on the quality of INST-IT Dataset.
Instance Caption
Image Caption
Temppral Caption
Video Caption
QA Pairs
Score (1) Time (s)
4.66+0.12
=..8 12.4
4.48 11.9
4.34
4..1
7.3
31.0
10.6
We can further scale the model size and scale the dataset using model-in-the-loop approach and improve the model through multi-round instruction tuning with self-synthesized data. We leave this direction for future work.
Broader impacts. This paper proposes an enhancement instance-level understanding capabilities in large multimodal models, enabling them to better assist users by answering questions about the content of interest. However, similar to existing large multimodal models, this approach also faces potential risks, such as issues related to fairness and bias. Future work can address this issue through approaches such as data filtering and validation.
21

---

Table 11: INST-IT Dataset frame-level annotations. For the ease of visualization, we only demonstrate the first three frames. Please zoom to view the instance ID labels.
Frame
Instance-level captions
Image-level captions
Temporal differences
[1] [2] [3] are standing closcly to- gether in an indoor setting. [1] is the left side wearing formal, light gray suit with white shirt. [2], the middle, wearing sleeveless white lace dress, hold- ing something in their hand. [3] is the right side in dark floral- pattemed dress with long. wavy hair. They appear to a room with wooden paneling and some framed art on the wall. null
I: Wearing light gray suit with white shirt, standing indoors. Wear- ing sleeveless white lace dress, holding an object in the hand. 3: Wearing dark floral-patterned dress with long wavy hair.
timestamp<11 <1>
[1] has moved closer to [2] and is now in contact with [2]'s hand. [2] has turned slightly towards [1] compared to the previ- ous frame. [3] remains in similar position, but the expression suggests more engagement with the scene. [5) and [6] have appeared in the frame: [5] visible the back- ground holding [6]. The table with a houquet of flowers is now visible, in- dicating shift in camera angle slightly to include more the right side of the room.
The seene appears to he in an of- fice setting with wonden table at the foreground. [1] is standing to the left. facing (2), and appears holding (2]'s finger or hand. [2] stands slightly to the right, return- ing focus with [1]. [3] is to the right of [2), slightly in the back- ground, smiling and looking for- ward. A bouquet white flowers lies on the table near [2]. [5] is partially visible in the background the right, seated and wearing red, [6] is a cellphone held by (5). Background shows a wooden wall and a reflection in a window.
I: A person wearing gray suit with white shirt, short hair, 2: A per- son in a white, sleeveless dress with long dark hair. 3: A person wearing dark floral dress with long dark hair. 5: A person wearing red, partially vis- ible in the background. 6: A small black cellphone held in hand.
timestamp=(2>>
I: Wearing a grey suit, standing beside [2] and slightly turned towards them. 2: Wearing a white, sleeveless dress with flo- ral textures. Holding a bouquet of white flow- ers. 3: Wearing dark patterned dress, standing slightly behind [2]. 4: Partially visible, wearing dark clothing, located at the edge of the left side of the frame. 5: Seated, wearing red outfit. Hold- ing a white object ahove their head, possibly ob- scuring their face.
The scene shows [1] [2] [3) near a wooden conference table in pro- fessional setting. possibly an office. [I] wears a grey suit and is stand- ing to the left, engaged with [2] who is wearing white dress and holding flowers. [3], who is in pattemedd dress, stands closely be- hind [2]. The newly appeared [4] is seated to the far left, partially visi- ble at the edge of the frame. [5] is seated the right side, holding object above their head, possibly obscuring their face. The room has wooden walls and a framed picture hanging on the wall.
Object [5] has lifted an object above their head, possibly piece of paper. Object [4] has appeared in the scene, seated on the left side of the frame, which was not visible car- lier. The positions of ob- jects [1]. (2), and [3] re- main unchanged, as does the background and set- ting of the no significant movement noticed terms of cam- era angle or position for objects (1) (2) (3).
timestamp-(3>
Overall,
22

---

Table 12: INST-IT Dataset Open-ended question-answer pairs.
Question
Answer
What change occurs with [1J's expression be- tween <10>> and the previous frame?
[1] changes from smiling a neutral expres- sion.
What activity are [1] and [2] involved at <11>?
and [2] are engaged in a kiss.
What is the overall mood during <11> as suggested by [3]'s actions?
A celebratory or joyous event.
What interaction occurs between [1] and [2] at <5>?
[1] holds [2]'s hand, suggesting an intimate gesture or exchange, likely a ring.
Who joins [1] and [2] in the frame at <7>?
[4] appears in the frame, joining [1] and [2].
What changes in the group's composition be- tween <7> and <8>?
[3] reappears, and [4] is no longer visible.
What element is seen throughout the frames <1> to <12>?
The scene is in an indoor setting with wooden paneling and framed art.
What type of event is likely taking place based on the atmosphere in <4> and <6>>
A formal event, possibly wedding official gathering.
What new elements are introduced in the scene at <2>?
holds a cellphone in the background, par- tially visible.
What the mood and lighting like at <6>?
The mood is formal and celebratory, with bright lighting enhancing this atmosphere.
What new background element appears at <7>?
There a map or blueprint on the wall.
What is notable about [5]'s actions at <3>?
[5] lifting an object above their head, possi- bly piece of paper.
What is the setting like in <3>?
The group is gathered near a wooden confer- ence table in formal setting.
How are [1] and [2] interacting <8>>
They engaged in conversation commu- nication, indicated by body language and fo-
What does [I]'s expression suggest at <12>?
[1] speaks or smiles, suggesting engagement with [2] others.
What shift occurs in the focus of the camera
The camera focuses more individuals standing together, reducing focus on the fore- ground objects.
between <5> and <6>?
What are [3] and [4] doing at <9>?
They clapping their hands in celebration.
What decorative element visible at <2>?
A bouquet of flowers lies the table near [2].
How has the posture of [1] and [2] changed by <6>?
[1] and [2] face slightly outward, suggesting pose for a photograph audience.
What overall physical change occurs between [1] and [2] from <10> to <1>>?
There's 's noticeable increase in their physi- cal interaction, enhancing emotional engage- ment.
23

---

Table 13: INST-IT Dataset video-level caption.
Video-level caption
The video appears document formal celebratory event indoors, possibly ceremony such as wedding or official gathering, occurring in room with wooden paneling and art or framed pictures on the wall. At the beginming, during < 1>, [1] is wearing light gray suit and stands with [2] sleeveless white lace dress, and [3] in a dark floral-patterned dress. The three are intimate focused setting. The progression between <2> and <3> involves subtle changes posture and interaction. [1] moves closer to [2]. appearing to hold hands engage in an exchange, possibly involving ring, as indicated by a bouquet of flowers. [3] remains supportive and smiling, while (5), in red, momentarily holds an object above their head, before disappearing from view by <4>.In frames <5> <7>, [1] and [2] maintain a close interaction, suggestive of a significant moment such as an exchange of vows or rings.
close together, suggesting
They are closely observed by [3], who stands smiling nearby, while [1] and [2] occasionally adjust their positions, facing each other initially and then turning outward, which may signal transitioning from an intimate moment to posing for a photo. By <7>, [4] joins, dressed darker attire, emphasizing the formal setting as [3] is no longer visible. Through <8> and <9>, the group dynamics change slightly with the absence of [4] and [3] entering the scene again. [1] and [2] appear to engage in warm interaction as [3] supports them, clapping, alongside the visible hands of [4] indicating applause, marking a cheerful tone. Finally, during <10> intimate conclusion
<12>, the focus shifts as [1] and [2] first engage in kiss, underscoring
their ceremony. They later stand apart slightly at the center, with [1] smiling or speaking, and [2] leaning towards [1] suggestively content. Throughout, the consistent joyous mood is accentuated by [3]'s s ongoing clapping and expression of joy, emphasizing shared celebration and approval from the audience captured.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes. We clearly and accurately state the contribution and scope in the abstract and introduction (Sec. 1).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Yes, we discussed the limitations in the appendix, please see Sec. E for more details.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we clearly describe the data construction process in Sec. 2 and provide the necessary details about the model training in Sec. 3. The codes, models, dataset, and benchmark will be fully open-sourced.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will release all the codes, data, and models once the blind review period is finished. We will also provide a clear instructions to reproduce our experimental results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
onooolooooolll icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc /gwpi/eg///////eeeeee /Cae//ooeubbiiic sttsSuuumisiiooii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Yes, we specified the data combination, hyperparameters setting, model architecture, and type of optimizer in Sec. 3.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars would be too computationally expensive to report. We claim that gains in our experimental results are consistent and significant.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we mentioned the runtime and device configurations in Sec. 3.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attps://neurs http:///irrrppppppppiiic//////// csGuidelines!
Answer:
[Yes]
Justification:
We conduct the research strictly following the NeurIPS Code of Ethies.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discussed the broader impacts in Sec, E.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Yes, we credited them in appropriate ways, and respected their license.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi wwwode.hco....c. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
30

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We describe the usage of LLMs in Sec. and Sec. A.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy (https: : http///pps..c//oeeeeeeeeeeeeeeee /LLM) for what should or should not be described.
31


Reviews:
[{"review_id": "dc71D8gqBs", "reviewer": "Reviewer_67yH", "summary": "This paper proposes INST-IT, a framework for instance-level understanding via explicit visual prompt Instruction Tuning. It introduces both the INST-IT benchmark and a large-scale training dataset. Experiments demonstrate that training on this dataset—combined with LLaVA-NeXT-Data—significantly improves performance on the INST-IT benchmark, as well as general video and image understanding benchmarks. These improvements are observed when following the authors' proposed training recipe, data combination strategy, and layer tuning scheme.", "questions": "Question.\n\n1. Regarding Video-MME, could the authors provide a breakdown of performance across different axes such as *Overall*, *Medium*, and *Long* video segments? This would help clarify whether the proposed INST-IT dataset improves long video understanding.\n2. For the INST-IT training dataset, could the authors report statistics on the duration of the video examples included? This would help better understand the dataset composition.\n3. An open-ended question: many examples in the benchmark ask about the *exact ordering of frames*, such as “Who disappears from the frame at `<idx>`?”. However, many models apply temporal compression or frame subsampling before input, which makes precise frame index questions potentially unfair. Moreover, this may not be an optimal way to construct training data. Do the authors think it would be better to use **timestamps** or **relative temporal markers** (e.g., \"beginning\", \"middle\", \"end\") instead of raw frame indices in both training and evaluation?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is well-written and easy to follow.\n- The proposed INST-IT training dataset and benchmark represent a substantial contribution to the community.\n- Experimental results offer valuable insights—for instance, that high-quality instance-level understanding data can also enhance generic image and video comprehension.\n- The dataset description is thorough, detailing a multi-level, instance-centric annotation pipeline and quality filtering procedures.\n- The experimental evaluation is comprehensive, and the ablation studies provide useful guidance on balancing image and video data.\n\n**Weaknesses**\n\nWhile the paper is comprehensive and shows improved results, it seems more appropriate for the *Datasets and Benchmarks* track rather than the main NeurIPS track. The primary contribution is a new dataset, with relatively limited methodological innovation. Although the dataset appears highly useful, the work may fall short of the level of technical novelty typically expected for the main conference track.", "was_revised": true, "final_justification": "The authors' rebuttal addressed my questions. The paper’s primary contribution lies in the dataset itself, with limited technical novelty and contribution in the method or model architecture. Therefore, I will maintain a borderline acceptance.", "rebuttal": "Dear reviewer 67yH,\n\nWe sincerely appreciate your valuable feedback and insightful comments, which are greatly helpful in improving our work. We will address your concerns as follows.\n\n---\n> [W1] Track appropriateness and primary contribution\n\nWe appreciate the reviewer's feedback and are highly encouraged by the recognition of our dataset as highly useful and our paper is comprehensive. \n\nWhile the dataset is indeed an important part of INST-IT, we would like to clarify that our contributions go beyond data creation. \nSpecifically, we\n\n* **Demonstrate the effectiveness of explicit visual-prompted instruction tuning**, showing that directly training with images and videos with visual prompts can effectively enhance the model’s fine-grained spatiotemporal understanding capabilities.\n* **Desgin a training recipe that enhances the model’s instance-level understanding without compromising its general capabilities**, which includes a detailed data combination strategy and an investigation of tunable layers within the vision encoder.\n* **Emphasize the importance of instance-level understanding**, a challenge that has been relatively underexplored compared to holistic scene-level understanding. Our work identifies key limitations in existing models and proposes INST-IT as a solution to this challenge.\n\nTherefore, our work not only introduces a novel dataset but also highlights the importance of instance-level understanding and proposes the use of explicit visual-prompted instruction tuning along with an effective training recipe to address this challenge.\n\n> [Q1] Breakdown of Video-MME performanceg\n\nThank you for you suggestion. We present the performance breakdown across different axes of Video-MME in the table below.\n\n| Model | Short | Medium | Long | Overall |\n|-------|-------|--------|------|---------|\n| Baseline| 41.9 | 35.8 | **30.8** | 36.2 |\n| LLaVA-Next-Inst-It-Vicuna-7B | 53.1 (+11.2) | 42.2 (+6.4) | **38.0 (+7.2)** | 44.3 (+8.1) |\n| LLaVA-Next-Inst-It-Qwen2-7B | 63.0 (+21.1) | 51.3 (+15.5) | **45.7 (+14.9)** | 54.0 (+17.8) |\n\nThe results demonstrate that the INST-IT Dataset effectively enhances the performance on VideoMME, \nincluding the long video understanding capabilities. When employing Vicuna-7B and Qwen2-7B as LLM backbones, we achieved improvements of 7.2% and 14.9% respectively on the Long segment of VideoMME. \n\n> [Q2] Statistics on the duration of the video examples \n\nSpecifically, the INST-IT dataset contains 21,000 video examples, with an average duration of 18.4 seconds and a maximum of 234 seconds. Among them, 5,580 videos are longer than 30 seconds, and 1,483 videos are longer than 60 seconds.\n\nWe would like to respectly note that the primary goal of INST-IT is to support instance-level understanding, which prioritizes the richness and precision of fine-grained annotations over the length of the videos. Moreover, since we use a sliding window over adjacent frames for temporal dynamic annotation, the data pipeline of INST-IT is also well-suited to long videos, we leave this extention for future works. Thank you for your suggestion. We will add the statistics on video durations to the revised manuscript. \n\n> [Q3] Temporal referencing method in both training and evaluation\n\nThank you for the great question. In fact, timestamps and frame indices can be converted between each other. We believe that using timestamps is a more suitable choice for long video understanding, especially when leveraging LLMs with extended context lengths (e.g., 128k in Seed1.5-VL`[1]`), while frame indices are more appropriate for short video understanding and LLMs with relatively shorter context lengths (e.g., 16k/32k in LLaVA-NeXT`[2]`).\nSince our current dataset and benchmark focus on relatively short videos, we adopt frame indices for reference. We plan to explore long video understanding in future work.\n\n\n---\n\nMany thanks to Reviewer 67yH for the professional and valuable reviews. We have try our best to address each of your concerns, and hope you can feel satisfied with our responses~ Please let us know if you have any further questions. Looking forward to hearing from you!\n\nBest regards,\n\nThe authors\n\n**References**\n1. Seed1.5-VL Technical Report, Arxiv 2025\n2. LLaVA-NeXT: Improved reasoning, OCR, and world knowledge"}, {"review_id": "YkxtgwZpcw", "reviewer": "Reviewer_SMig", "summary": "This paper addresses the limitation of Large Multimodal Models (LMMs) in instance-level understanding, which requires fine-grained comprehension of specific objects within images and videos. The authors propose INST-IT, a comprehensive visual instruction-tuning dataset annotated by GPT-4o with Set-of-Marks (SoM) visual prompts, resulting a large-scale instruction-tuning dataset with 51k images and 21k videos. The work also introduces INST-IT Bench for evaluating instance-level understanding capabilities. Experiments show that training on INST-IT can improve the performance on image understanding and video understanding QA, proving the quality of the dataset.", "questions": "1. How much effects does SoM have on the data quality? Are there any experiments proving it?\n2. For videos, how does the method ensure temporal consistency of instance IDs across frames, especially when instances appear and disappear?", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1. This paper focus on the speicfic instance understanding task and curated instruction-tuning dataset to improve it. Understand specific instances rather than just holistic scene understanding is practically important for real-world applications.\n2. The use of GPT-4o with SoM visual prompts to generate multi-level annotations (instance-level, frame-level, temporal dynamics, video-level) is creative and appears effective.\n3. Thorough evaluation on both image understanding benchmarks and video QA benchmarks, with consistent improved performance after adding INST-IT as the benchmark.\n\n## Weaknesses\n1. The paper's method novelty is dataset itself, with little analysis of potential VLM architecture exploration (directly use LLaVA-Next). \n2. Also the new dataset is also curated by GPT-4o, thus making the whole method like a distillation from GPT-4o. It's unknown whether the SoM can improve the data curation quality, or the whole performance improvement simply benefit from the GPT-4o's ability.", "was_revised": true, "final_justification": "Thanks for the rebuttal, it solves some of my concerns.", "rebuttal": "Dear reviewer SMig,\n\nThank you for your time and effort, which are greatly appreciated in helping us improve our work. We will address your concerns one by one.\n\n---\n\n> [W1] Method novelty\n\nWhile the dataset is indeed one of our key contributions, our work goes beyond data collection in several important aspects. Specifically, we\n(1) investigate the effectiveness of instruction tuning with explicit visual prompts for enhancing fine-grained spatiotemporal understanding; and\n(2) propose a continuous instruction tuning paradigm that includes carefully designed data combination strategies to balance general capabilities and instance-level understanding, as well as a tuning scheme tailored for the visual encoder.\n\nRather than modifying the model architecture itself, our research focuses more on developing a model-agnostic training framework aimed at enhancing multimodal instance understanding. \n\n\n> [W2, Q1] The effectiveness of SoM in improving data quality\n\nThank you for raising this question. In fact, SoMs visual prompts play a critial role in instance-level annotation by marking numerical IDs on each instance in the frames and use the corresponding ID in to reference them. This significantly enhances GPT-4o’s grounding ability, enabling it accurately focus on the specific instance of interest. \n    \nWithout SoMs, GPT-4o struggles to produce fine-grained annotations that meet the requirements, the key challenges are:\n* GPT-4o struggles to precisely locate target instances, especially in crowded scenes or when occlusion occurs.\n* In video frames lacking instance ID prompts, GPT-4o struglle to recognize an object as the same one after it disappears and reappears.\n    \nBy providing a straightforward referencing (direct use of instance IDs) and explicit spatiotemporal grounding, SoMs effectively enable fine-grained spatiotemporal annotation at the instance level, thereby improving annotation quality.\n    \nTo further investigate, we replace the SoMs in the INST-IT Dataset with language descriptions, to simulate model training without visual prompts. As shown in the table below, the model trained with SoM annotations outperforms the one without SoMs, confirming its effectiveness.\n    \n|                                          | MMMU | POPE | RefCOCOg(CIDEr) | INST-IT-I |\n|----------------------------------------------|------|------|-----------|--------------------|\n|  /wo SoM         | 36.7 | 87.0 | 60.5      | 49.5               |\n| /w SoM           | 37.8 | 87.5 | 62.0      | 59.6               |\n\n\n> [Q2] How to ensure the temporal consistency of instance IDs across frames\n\nMaintaining the consistent instance IDs across frames is indeed a crucial concern. \nTo ensure high data quality, we use video segmentation and multi-object tracking datasets, in which the IDs are **manually annotated**. \nThis guarantees the temporal consistency, even when instances disappear or reappear. \n\nIn addition, recent advances in video instance segmentation `[1][2][3]` have shown strong performance in ID matching across frames, especially when instances disapper and reappear. These models can serve as promising tools to annotate instance IDs automatically, and we leave this exploration for future work.\n\n---\n\nThank you once again for your valuable feedback and thoughtful comments. We hope that our responses can address your concerns. If you have any additional questions, please feel free to reach out to us. If you feel your concerns are resolved, we kindly hope that you can consider reevaluating our work. We look forward to your reply!\n\nBest regards,\n\nThe authors\n\n**References**\n1. Matching Anything by Segmenting Anything, CVPR2024\n2. SAMURAI: Adapting Segment Anything Model for Zero-Shot Visual Tracking with Motion-Aware Memory, Arxiv 2024\n3. SeC: Advancing Complex Video Object Segmentation via Progressive Concept Construction, Arxiv 2025"}, {"review_id": "N003LkL2dM", "reviewer": "Reviewer_42ms", "summary": "This paper addresses a significant limitation in current MLMMs: their difficulty with \"instance-level understanding,\" which requires comprehending the specific details, attributes, and relationships of individual objects within images and videos, as opposed to understanding the scene holistically. To tackle this, the authors introduce INST-IT, a comprehensive solution comprising a dataset, a benchmark, and the trained models. Experiments show that models enhanced with INST-IT achieve state-of-the-art performance on the INST-IT Bench and other instance-understanding benchmarks like RefCOCOg and ViP-Bench.", "questions": "Generally, I think this paper is good, and please refer to the weaknesses part.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The instance-level understanding is a missing puzzle for existing MLLMs datasets and benchmarks.\n2. The dataset/benchmark curation pipeline is technically sound.\n3. The paper provides extensive experiments and ablation studies.\n\nWeakness:\n\nI think it's not that natural for users to ask questions by giving a lot of visual masks. Instead, I think it's more common to use language to specify different people or objects in the video. I can understand this might be harder to build benchmarks or datasets, but It would be great if the author could discuss this.", "was_revised": true, "final_justification": "I think this work contributes a good instance-level dataset for MLLM. I will vote for accept this paper.", "rebuttal": "Dear reviewer 42ms,\n\nWe sincerely appreciate your valuable feedback and insightful comments, which are greatly helpful in improving our work. We are highly encouraged that you thought our paper is good. We will address your concerns as follows.\n\n---\n\n> [W1] Regarding how to refer to objects in videos, visual prompts or language descriptions?\n\nGreat question! For instance-level understanding, how to inform the model of the instances we are interested in is an important design consideration. The main approaches include 1) visual prompting and 2) language referring. Each method has its own advantages and limitations.\n\n* Using visual prompts allows for explicitly and accurately marking the region of interest in the frame, which can more directly guide the model's attention to the specified area. However, the labor required to annotate visual prompts is indeed an issue that needs to be considered.\n* Using language to refer to specific instances is more natural and aligns better with human communication habits. However, due to the potential ambiguity of language, such references might become unclear when there are many objects. This also places higher demands on the model’s grounding ability to locate the region referred to by language within the visual scene.\n\nAdditionally, in terms of user experience, with the rapid advancement of interactive segmentation paradigms such as Segmentation Anything (SAM `[1,2]`) and SEEM `[3]`, users can now directly generate object masks by simplely clicking or swiping, offering a natural and intuitive interaction to work with visual prompts.\n\nIt is worth noting that although INST-IT is trained using explicit visual prompts, our model also exhibit improved understanding of language-based references. For instance, in the example shown on the left side of Figure 1 in the main paper, we directly use terms such as “man” and “woman” for instance reference, and our model demonstrates better comprehension compared to other models. We speculate that this improvement may stem from the grounded instance-level annotations in INST-IT, which implicitly enhance the model’s ability to interpret referring expressions and perform visual grounding.\n    \nThank you for bringing this up. We will include a discussion on this point in the revised version.\n\n---\n\nWe sincerely thank you once again for your insightful questions and are glad to have a meaningful discussion with you. We have shared our thoughts on instance referring in videos, and hope to engage in further discussion with you:-) Looking forward to your reply!\n\n\nBest regards,\n\nThe authors\n\n**References**\n1. Segment Anything\n2. SAM 2: Segment Anything in Images and Videos\n3. Segment Everything Everywhere All at Once"}, {"review_id": "MtaviSFPjj", "reviewer": "Reviewer_no3B", "summary": "This paper presents INST-IT, a comprehensive framework aimed at enhancing instance-level understanding in Large Multimodal Models through explicit visual prompting and instruction tuning. The authors propose a large-scale dataset (INST-IT Dataset) that incorporates fine-grained, instance-specific annotations for both images and videos, generated through an automated pipeline using GPT-4o to highlight target instances. Besides, they construct a benchmark (INST-IT Bench) designed to evaluate instance-level understanding in both open-ended and multiple-choice formats. The training strategy, based on continuous instruction tuning, blends general instruction data with instance-level supervision, demonstrating significant improvements not only on the proposed benchmarks but also on a variety of generic image and video understanding tasks. The results show that their method is broadly effective across domains.", "questions": "1. Does the use of GPT-4o for large-scale annotation introduce any detectable biases or hallucinations in the model’s final responses? How was this risk assessed or mitigated during data generation and training?\n\n2. Could you provide results or analysis comparing your approach against stronger and more recent baseline models? \n\n3. Could the authors clarify the typical duration of the videos in the dataset and describe the specific frame sampling strategy used? Additionally, is there a risk of missing important or key frames when sampling at fixed intervals?\n\n4. Could the authors specify the total cost of annotating the entire dataset?", "limitations": "See above.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The paper addresses a pressing and relatively underexplored limitation in current LMMs, specifically their poor instance-level understanding, and offers a practical and well-justified solution. \n\n2. The dataset construction is large-scale, multi-granular, and thoughtfully designed with visual prompts, covering both images and videos, which adds notable novelty and utility to the field. \n\n3. The benchmark (INST-IT Bench) is carefully built and provides both open\u0002ended and multiple-choice QA formats, which allow for more robust evaluation.\n\n**Weaknesses**：\n\n1. The baselines used for comparison are not comprehensive and are relatively outdated. Including stronger and more recent models such as the InternVL series and the Qwen2.5-VL series would strengthen the validity and credibility of the performance claims.\n\n2. The human evaluation uses 500 randomly sampled data points, which is helpful but relatively limited given the dataset's scale in the tens of thousands. Expanding the size of the human-reviewed subset would strengthen the credibility and representativeness of the quality assessment.\n\n3. The use of visual prompting is effective but not fundamentally novel. Some parts of the methodology may be seen as incremental extensions.\n\n4. The potential computational overhead during inference when using visual prompts (e.g., SoMs) is not quantitatively assessed, which may limit the method’s practicality in latency-sensitive applications.", "was_revised": true, "final_justification": "While I appreciate the authors' detailed rebuttal, I remain unconvinced that the technical novelty and contributions are sufficient for acceptance. I will therefore keep my original score.", "rebuttal": "Dear reviewer no3B,\n\nThank you for your time and thoughtful comments. We are encouraged by your recognition of our contributions as well-justified and novel. Below, we address your concerns in detail.\n\n---\n\n> [W1, Q2] Including stronger and more recent baseline models.\n\nThank you for the valuable suggestion. We have conducted additional experiments using the recent Qwen2.5-VL-3B-Instruct model to further evaluate the effectiveness and generalizability of our INST-IT. The results are presented in the table below. \n\nAs shown, integrating INST-IT into Qwen2.5-VL consistently improves performance across a range of benchmarks. Specifically, we observe gains on general image understanding (MMMU, GQA), general video understanding (NeXTQA, VISBench), and particularly clear improvements on instance-level understanding benchmarks (RefCOCOg, INST-IT-I, INST-IT-V). These results are consistent with our previous findings on LLaVA-NeXT, further validating the effectiveness and cross-model applicability of INST-IT.\n\n\n|            | MMMU | GQA | NeXTQA | VISBench | RefCOCOg(CIDEr) | INST-IT-I | INST-IT-V |\n|------------|------|-----|--------|----------|----------|-----------|-----------|\n| Qwen2.5VL*  | 46.4 | 60.0 | 75.9 | 27.9 | 62.6 | 65.7 |  52.1  |\n| Qwen2.5VL+INST-IT | 48.4 (+2.0) | 63.3 (+3.3) | 77.4 (+1.5) | 32.9 (+5.0) | 69.0 (+6.4) | 81.3 (+15.6) |  68.6 (+16.5)  |\n\n\\* We used the lmms-eval `[1]` toolkit for model evaluation and re-evaluated Qwen2.5VL-3B-Instruct using the same inference parameters and runtime configurations as our model.\n\n> [W2] Expanding the size of the human-reviewed subset.\n\nThank you. We have further expanded the human study by including an additional 200 data points. \nThe scores and the time spent reviewing each sample are shown in the table below. Consistent with the findings from the initial subset of 500 samples, the results indicate that the annotation quality in INST-IT remains satisfactory. However, since manual evaluation is very labor-intensive, we were only able to add 200 additional samples during the rebuttal phrase. We will further scale up the human study in the revised version to improve the representativeness of the evaluated subset.\n\n|                | Instance Caption | Image Caption | Temporal Caption | Video Caption | QA Pairs |\n| :------------- | :--------------- | :------------ | :--------------- | :------------ | :------- |\n| Score (↑)      |       4.61       |     4.55      |      4.42        |      4.29     |    4.30  |\n| Time (s)       |       8.4        |     13.5      |      14.0        |     40.0      |    11.9  |\n> [W3] The use of visual prompting is effective but not fundamentally novel.\n\nWhile visual prompts have been used in previous work for referring to specific objects during model inference `[2]` or for training image understanding models `[3]`, to the best of our knowledge, INST-IT is the first work to directly perform grounded instruction tuning using explicit visually prompted **images and videos**, and empirically demonstrated that this approach can effectively enhance the model's **spatiotemporal** fine-grained understanding capabilities.\n\n> [W4]The potential computational overhead when using visual prompts.\n \nThe SAM2 series models `[4]` can be used for automated generation of SoMs visual prompts. The table below lists the officially reported `[4]` inference speeds of different model sizes (measured on an A100 using `Torch 2.5.1` and `CUDA 12.4`).\n\n| Model                 | Size (M) | Speed (Frame Per Second, FPS) |\n|----------------------|----------|--------------|\n| sam2_hiera_tiny      | 38.9     | 91.5         |\n| sam2_hiera_small     | 46       | 85.6         |\n| sam2_hiera_base_plus | 80.8     | 64.8         |\n| sam2_hiera_large     | 224.4    | 39.7         |\n\nCompared to LMMs with billions of parameters, the overhead introduced by SoMs construction is almost negligible. Users can choose video segmentation models of different sizes based on the latency requirements of their applications. Thank you for your valuable suggestion, and we will include this discussion in the revised version.\n    \n> [Q1] Discussion on the potential bias and hallucination introduced by GPT-4o.\n\nThank you for raising this important concern which is indeed a common challenge faced by all works foucused synthetic data.\n\nTo address this, we first conducted small-scale trial annotation (hundreds of samples), followed by manual review to assess annotation quality and identify common hallucination or bias patterns. Based on these observations, we refined the task prompts and optimized the annotation pipeline. Large-scale annotation was only launched once we were satisfied with the output quality.\n\nDuring initial trials, we observed two main issues: (1) When scenes contained many instance objects, the model occasionally confused their attributes or relationships; (2) For longer videos, it exhibited temporal hallucinations and struggled to maintain accurate spatiotemporal grounding of specific objects.\n\nTo mitigate these issues, we introduced several strategies:\n* Rather than directly annotating raw videos or images, we inserted SoM visual prompts and used instance IDs to guide the model’s attention. This significantly improved GPT-4o’s spatial and temporal grounding, even in complex scenes.\n* For long video sequences, we adopted a sliding-window approach with adjacent frames and explicitly prompted the model to focus on local temporal dynamics, which effectively reduced long-range hallucinations.\n* During training, we carefully balanced general data with INST-IT to prevent overfitting to a narrow distribution. Furthermore, we froze the first 12 layers of the vision encoder to avoid biasing the model toward visually prompted inputs.\n\nThese efforts substantially alleviated the hallucination risk. Performance gains on both general benchmarks and instance-level understanding tasks further demonstrate the this claim.\n\n\n> [Q3] Typical duration of videos and frame sampling strategy; risk of missing important frames.\n\n* **Video duration.**\nSpecifically, the INST-IT dataset contains 21,000 video examples, with an average duration of 18.4 seconds and a maximum of 234 seconds. Among them, 5,580 videos are longer than 30 seconds, and 1,483 videos are longer than 60 seconds.\nWe would like to respectly note that the primary goal of INST-IT is to support instance-level understanding, which prioritizes the richness and precision of fine-grained annotations over the length of the videos. \n\n* **Frame sampling strategy.**\nFor videos sourced from VidOR, keyframes are determined based on the original human annotations; for videos from other sources, we adopt a fixed-interval frame sampling strategy. \nIndeed, applying a uniform sampling interval across all videos may risk missing critical frames. To mitigate this, we manually inspected a random subset of videos from each source. Based on differences in visual information density and video characteristics, we then assigned different sampling intervals tailored to each source. We present the annotation intervals (annotated frames per second) for different sources in the table below. The average annotation frequency in INST-IT is **0.74 FPS**, which is significantly denser than that of previous works such as LLaVA-Hound (0.008 FPS) `[5]` and ShareGPT4Video (0.15 FPS) `[6]`. This higher density substantially reduces the likelihood of missing critical frames.\n\n|     | BenSMOT | LVVIS | OVIS | BRUST | UVO | VidOR | YoutubeVIS-2021 | Avg |\n| :---: | :-------: | :-----: | :----: | :----: | :---: | :----: | :----: | :----: |\n| **FPS** |  1/6   |  1  |  1/2 |  1/2  |  1 |  - |  4/5 | **0.74** |\n\n> [Q4] The total cost of annotating the entire dataset.\n\nWe use `GPT-4o` API to provide detailed and dense annotations for 51K images and 21K videos, and the total API cost is approximately $6000.\n\n---\n\nThank you again for your valuable feedback and professional comments. We have done our best to address your concerns and hope our responses are satisfactory. Please feel free to reach out to us if you have any further questions. If you feel they are resolved, we would sincerely appreciate your re-evaluation of our work~ Looking forward to hearing from you!\n\nBest regards,\n\nThe authors\n\n**References**\n1. LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models\n2. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V, Arxiv 2023\n3. ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts, CVPR 2024\n4. SAM 2: Segment Anything in Images and Videos, Arxiv 2024\n5. Direct Preference Optimization of Video Large Multimodal Models from Language Model Reward, Arxiv 2024\n6. ShareGPT4Video: Improving Video Understanding and Generation with Better Captions, NeurIPS 2024"}]

Author rebuttal:

Meta-review: The final ratings are mixed (two "Borderline Accept," two "Borderline Reject").
After a careful review of the paper, rebuttal, and discussions, the AC leans toward acceptance.

The AC agrees with reviewers that the paper addresses an under-explored yet important limitation in current VLLMs: fine-grained, instance-level understanding. The core contribution, the INST-IT dataset and benchmark, was consistently recognized as a high-quality and valuable resource.

However, initial reviews raised valid concerns. The most significant issues were:
1) Methodological Novelty: Several reviewers (67yH, SMig, no3B) noted that the primary contribution is a dataset and training recipe rather than a novel model or algorithm.
2) Data Curation Effectiveness: Reviewer (SMig) questioned whether performance gains were simply from distilling knowledge from GPT-4o, asking for more evidence on the specific contribution of the Set-of-Marks (SoM) visual prompts.

The authors' rebuttal effectively addressed these points. They justified their contribution by framing it as a complete, model-agnostic training paradigm and benchmark (Inst-IT), validated it with an additional baseline (Qwen2.5-VL-3B-Instruct). Also, they quantified the impact of SoM visual prompts with an ablation study, demonstrating their value beyond simply using GPT-4o.

While the concern regarding methodological novelty is valid, the AC believes that the paper already makes a valuable contribution by identifying a key problem, creating a high-quality benchmark to diagnose it in current state-of-the-art VLMs, and demonstrating a successful training paradigm that boosts performance across various models. The authors' rebuttal and additional experiments have addressed the paper's initial weaknesses. Therefore, the AC recommends this paper for acceptance.
Decision: Accept (poster)