SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models
Authors: Ye Sun
Hao Zhang
Henghui Ding
Tiehua Zhang
Xingjun Ma
Yu-Gang Jiang
Venue: NeurIPS 2025 poster ()
Keywords: SAMA
Video Large Multimodal Model
Referring
Grounding
Segmentation
Refer and Ground
Primary area: deep_learning

Abstract: Achieving fine-grained spatio-temporal understanding in videos remains a major challenge for current Video Large Multimodal Models (Video LMMs). Addressing this challenge requires mastering two core capabilities: video referring understanding, which captures the semantics of video regions, and video grounding, which segments object regions based on natural language descriptions.
However, most existing approaches tackle these tasks in isolation, limiting progress toward unified, referentially grounded video interaction. We identify a key bottleneck in the lack of high-quality, unified video instruction data and a comprehensive benchmark for evaluating referentially grounded video chat.
To address these challenges, we contribute in three core aspects: dataset, model, and benchmark.
First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically curated to enable joint learning of video referring understanding, grounding, and multi-turn video chat.
Second, we propose the SAMA model, which incorporates a versatile spatio-temporal context aggregator and a Segment Anything Model to jointly enhance fine-grained video comprehension and precise grounding capabilities.
Finally, we establish SAMA-Bench, a meticulously designed benchmark consisting of 5,067 questions from 522 videos, to comprehensively evaluate the integrated capabilities of Video LMMs in multi-turn, spatio-temporal referring understanding and grounded dialogue.
Extensive experiments and benchmarking results show that SAMA not only achieves strong performance on SAMA-Bench but also sets a new state-of-the-art on general grounding benchmarks, while maintaining highly competitive performance on standard visual understanding benchmarks.

SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models
Ye Sun', Hao Zhang?, Henghui Ding'!
Tiehua Zhang?, Xingjun Ma Yu-Gang Jiang'
Fudan University, HKUST, Tongji University
https://github. www.coo/////aae //n//23//y2a/aa..
Abstract
Achieving fine-grained spatio-temporal understanding in videos remains major challenge for current Video Large Multimodal Models (Video LMMs). Addressing this challenge requires mastering two core capabilities: video referring understand- ing, which captures the semantics video regions, and video grounding, which segments object regions based natural language descriptions. However, most existing approaches tackle these tasks isolation, limiting progress toward unified,
referentially grounded video interaction. We identify key bottleneck in the lack of high-quality, unified video instruction data and a comprehensive benchmark for evaluating referentially grounded video chat. To address these challenges, we contribute in three core aspects: dataset, model, and benchmark. First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically curated enable joint learning of video referring understanding, grounding, and multi-turn video chat. Second, propose the SAMA model, which incorporates versa- tile spatio-tempora context aggregator and Segment Anything Model to jointly enhance fine- fine-graindd video comprehension and precise grounding capabilities. Finally, we establish SAMA -Bench, a meticulously designed benchmark consisting of 5,067 questions from 522 videos, to comprehensively evaluate the integrated capabilities of Video LMMs in multi-tur, ppatio-temporal referring understanding and grounded dialogue. Extensive experiments and benchmarking results show that SAMA not only achieves strong performance SAMA-Bend' but also sets new state-of-the-arr on general grounding benchmarks, while maintaining highly competitive performance standard visual understanding benchmarks.
Introduction
Recent years have witnessed remarkable progress in Large Multimodal Models (LMMs) [46, 57], driving significant advances in general vision-language understanding [37, 75, 31, 133. However, extending these capabilities effectively to the video domain remains critical and open research challenge [35, 69, 42, 32]. Unlike static images, videos introduce additional complexity through temporal dynamics, requiring models not only to perform holistic scene comprehension but also to achieve fine- aene-grained alignment between language and temporally evolving objects and actions within a continuous spatio-temport context [2]. While current Video LMMs [35, 69, 42, 32, 30, 33] demonstrate strong performance in global scene understanding. they often fall short in fine-grained apatio-temporal reasoning. This limitation is particularly evident when models are required to interpret references to specific entities and deliver precisely grounded information amid complex and dynamic sntivites-ccpaailiiie that are essential for applications such as detailed event forensics, interactive robotic assistance, and nuanced content analysis.
CCoresponding authors: henghui. gmail.com, xingjunma @ cundan.edu.cn
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Video Referential Grounded Chat
Video Referential Captioning
Video Grounded Description
Te How doea sabject- interact with coblects and cobject-?
What can you
Describe detail
the sobjects in
Interteaved segmentation masks.
The person holdy the lonile pomegnanati their hands, suggesting they to the
person, The haysk amoll uppears
Agrnnn loyaat baing paddied by:a
with bullis standing
shone a7740 paople LnwEEEa
lighteeeight, rounded it monteerrred through the witer the podiles
wesden platfirr,
beading black shar and
SAMA
SAMA
SAMA
Video Referring Methods Folled Grounding
Video Grounding Methads Failed Captioning
General MLLMss
Falled Grounding
The person briefly holds a small cytindricall object, sets down, and them picka the fruit.
The video capture youthiul and energetic scene from traditional builfig ghiingg or bull-chasing event, Ukklyy in setting....
Gemini
Sure, it is [SEG).
Artemis
Video
GLaMM
(c) Figure 1: Comparisons with prior specialized and general MILMMs Our SAMA model excels in
(a)
(b)
multiple fine-grained video understanding tasks that require both referring and grounding capabilities: complex video referential grounded chat, video referential captioning, and video grounded description.
Enabling fine-grained video understanding hinges on mastering two fundamental and deeply intercon- nected capabilities: video referring understanding the ability to comprehend semantics associated with designated video regions| [48, 68, 20] and video grounding the capacity to accurately seg- ment specific video regions based on natural language descriptions [12, 9, 67, 45, 1, 64]. In the image domain, substantial progress has been made with multimodal large language models (MLLMs) such as Ferret [65], Shikra [3]. GPT4Rol [72], LLaVA-Grounding [70]. and CogVLM [59], which integrate object-level representations into MILMs to enable coordinate output or segmentation masks for fine-grain tne-grained image comprehension. However, analogous advances in the video domain remain limited and face unique challenges. For video referring understanding, recent works like Artemis [48], VideoRefer [68], and OmniRGPT [20] demonstrate reasonable regional understanding by leveraging object prompts. Meanwhile, in video grounding, specialized LLM-based models such as VISA [61] [, VideoLISA [1]. and VideoGLaMM [45] have been developed to segment target objects based on textual descriptions. Yet, a critical limitation persists: most of these efforts treat referring under- standing and grounding as distinct tasks, neglecting their intrinsic interdependence namely, that both fundamentally require precise spatio-temporal alignment with semantic cues. This separation prevents current Video LMMs from achieving unified, human-like capability to both comprehend nuanced object details and fluidly interpret dynamic scenes, as shown in Figure 1.
We identify key bottleneck hindering further progress: the lack of high-quality, unified vidco instruc- tion data specifically tailored to enable the joint learning of referring understanding and grounding, along with the absence of comprehensive benchmark for evaluating integrated performance [68]. While existing video understanding datasets typically focus on isolated tasks such as action recog- nition captioning [22, 27], and foundational grounding datasets offer valuable segmentation or localization annotations for visual entities [9, 11, 53, 44, 74, 10, 14], a critical gap remains: the lack of resources that combine these components support advanced referential grounded video dialogue. Specifically, current datasets fall short in providing: 1) rich, multi-turn enferring-based video dialogues; 2) precise spatio-temporal grounding of objects and actions mentioned in those dialogues; and 3) explicitly structured joint supervision to facilitate the simultaneous learning of referring understanding and grounding within an interactive framework.
To address these challenges, we contribute in 3 key aspects: dataset, model, and benchmark.
We present SAMA-239K, a carge-scale video instruction-tuninn dataset specifically curated to facilitate the joint learning of video referring understanding and grounding. To construct this dataset, we developed -LLe- automated annotation pipeline that first enriches video data by generating accurate segmentation masks using HQ-SAM [24]. Building upon these grounding annotations and leveraging advanced prompt engineering with Gemini-1.5 Pro [55], we generate 67K detailed oeject-level descriptions and 172K multi-turn conversational QA pairs to support training. Unlike previous single-turn video QA datasets, SAMA-239K. is designed to emulate natural human
Dataset.

---

interaction, evolving from basic observations to complex reasoning and contextual inference, while explicitly linking conversational entities to their corresponding spatio- masks.
Model. We propose SAMA, a novel Video LMM architecture that seamlessly integrates fine-grained spatial and temporal understanding. SAMA introduces versatile spatio-temporal aggregator to capture video-level temporal dynamics, alongside a powerful SAM2 [50] for extracting frame-level pixel-level semanties. This unified design enables the model to fully exploit the rich supervision in SAMA-239K, supporting end-to-end learning across video referring understanding, video grounding, and multi-turn referential video dialogue within single cohesive framework.
Benchmark. We introduce HAMA-Bench, a comprehensive benchmark comprising 522 diverse videos curated from four established datasets, totaling 5,067 challenging questions. It consists of two main components: SAMA-Bench which evaluates referential grounded dialogue through a broad range of question types involving object behaviors, interactions, attributes, and spatial relationships; and SAMA-Bench which assesses video referring region captioning by requiring models to generate detailed captions for specified object categories, including persons, animals, tools, and vehicles.
Evaluation. Extensive benchmarking and experiments show that our SAMA model outperforms existing grounding LMMs on SAMA-Bench and also achieves strong results established grounding benchmarks. To the best of our knowledge, SAMA is the first Video LMM to successfully unify fine une-grained referential understanding and grounded dialogue, achieving state-of-the- art performance on both image and video referring segmentation tasks.
2 Related Work
Multimodal Large Language Models. The remarkable capabilities of LLMs [46, 57, 41] across wide range of language-centric tasks have reshaped the landscape of artificial intelligence (AI), establishing a powerful foundation for extending reasoning abilities to visual understanding. In response, research has rapidly converged on the development of image-based MI..Ms. Pioneering works such as LLaVA [37]. MiniGPT-4 [75], and InstructBLIIP [8] have demonstrated impressive visual understanding by effectively integrating visual inputs with LLMs. Building upon these successes in processing static images, attention has increasingly shifted toward Video LMMs [33, 35, 69, 42, 30J. To capture the rich spatio-temporal information in videos, Video LMMs typically extract sequential visual features from frames using pre-trained vision backbones. These features are often processed via direct token concatenation [S, 35], and then interleaved with textual embeddings as input to the LLM to generate responses. Despite promising results on tasks requiring holistic understanding- such as general video question answering and captioning-exisiingg Video LMMs still face notable challenges in fine-grained, spatio-temporal object-level grounding and understanding [65, 3, 59, 70, 15].
MLLMs for Referring and Grounding. Enabling fine-grained interaction in multimodal systems requires MILMs accurately refer to and ground specific entities within visual inputs. In the image domain, notable progress has been made. One major direction, represented by models such as Kosmos-2 [47], Shikra [3], Ferret [65], GPT4Rol [72], and CogVLM [59], equips MLLMS with the ability to process and generate spatial location information, typically in the form of bounding boxes or point coordinates expressed numerically or as discrete tokens. Another active direction integrates segmentation capabilities, enabling models to produce pixel-level masks aligned with textual references. This is often achieved by connecting LLMS with segmentation models like SAM [26] through specialized integration mechanisms [70, 28, 49, 56].
Extending referring and grounding capabilities to the video domain introduces new challenges, particularly due to temporal complexity and the need for consistent object tracking. Recent works like Artemis [48], VideoRefer [68], Omni-RGPT [20], and DAM [34] have advanced fine-grained video understanding through region-level descriptions conditioned on prompts. However, they do not support dynamic segmentation mask generation within conversational outputs. In contrast, VideoGLaMM [45] achieves strong referring segmentation in videos but focuses on grounded descriptions, lacking support for interactive, fine-grained video dialogue. This highlights a key gap: the lack of models that jointly support precise object-level grounding and nuanced referring in interactive video conversations. We bridge this gap through improved datasets, model designs, and benchmarks, achieving rich, grounded, and interactive video dialogue.
3

---

3 SAMA
3.1
SAMA-239K Data Creation
As illustrated in Figure 2, our data construction lever- ages sophisticated L.M-- based automated annotation pipeline, centered around Gemini-1.5 Pro [55], which systematieally transforms existing video referring seg- mentation or grounding datasets into rich, grounded conversational data.
Video Segrentation Ootsact
Pronet
cecy lovel Descriptions
La Yolie Momartee in
Assstcted Video
Video Grounday Dahoset
Gemini
Reeforential Greasded Dilleges
"ehet
Objjctt
"ayote witter dock der (MU'T wan to
Object-level Description Generation. Our initial aim is to obtain precise, object-centric textual de- scriptions. For ground- ing datasets that only offer Figure 2: Data creation pipeline of bounding boxes but lack pixel-Ievel masks, we first employ HQ-SAM [24] to generate high-quality pseudo masks from the existing box information. Inspired by SoM [62]. we then leverage segmenta- tion mask information to highlight objects of interest within sampled video frames using distinctly colored bounding boxes. These visually cued frames, along with relevant categorical or action information for the highlighted objects, are provided to Gemini to generate fine-grained, descriptive captions for each identified object instance.
Woall "sagt gaeen
and teroghet This oddo- with
Data Preparation
Automatic Data Generation
Best viewed in zoom.
Referential Grounded Dialogue Generation. Building upon the rich, localized descriptions, we further task Gemini with creating multi-turn question-anae answer pairs. These dialogues are designed to emulate natural human conversational patterns, initiating with basic observational questions about one or more objects and progressively advancing to more complex reasoning that might involve inter-object relationships or inferential steps. A critical instruction is that the generated answers must explicitly incorporate grounding information, linking back to the previously established spatio- temporal masks of the referred entities.
Finally, through an automated filtering process ensure the correctness of the generated data format, our proposed SAMMA-239K has 172,296 referential grounded video-QA triplets along with 67,005 object-level descriptions in total.
3.2 SAMA-239K Data Characteristics
Pratio-Temmorra Dynamies. For understanding complex object motion and achieving precise, temporally-consiste grounding, select training data from referring video object segmentation datasets, primarily MeViS [9] and Ref- YouTube- VOS [53], which provide video sequences with dense annotations of objects undergoing significant motion changes.
Large Vocabulary Knowledge. To ensure SAMA comprehends broad range of objects and their diverse textual descriptions, we incorporate LV-VIS [58], video instance segmentation dataset that contributes a wide vocabulary of object categories, vital for open-world generalization.
Complex Scene Robustness. Real-world video understanding demands robustness against complex environmental conditions. To cultivate this, we incorporate data from SAV [50], which contributes challenging scenarios featuring occlusions and motion blur. Furthermore, we utilize video grounding dataset VidSTG [74] to ensure the model maintain consistent tracking and contextual comprchension over long temporal horizons.
3.3
SAMA Model
Here, we present the architecture of SAMA, which builds upon Sa2VA [67] and introduces novel contributions to synergize fine-grained frame-level information with long-range temporal context.

---

Tpe-he punda-ipe|SEC) elings opo-the other pandacip-[SEG] and coacinues to cat Brom ago-thte persea-/p-|SECC" offerings
Large Language Model
Video Prsme-level
Peey
Spatial- Temporri
Language Embesiiing lang
SAM2
Decoder
Fabj
Context Aggregator
Object
Visual
Embeddinng
Feature
Sliding Window
Tokenizer
Spatial Ageregator
Temporal Aggregator
Object Pronuee
Frames
SAM2 Encader
Attention
Atteation
What huppews after -a cobjects eats some of the food offered objoct
Visual Encoder
Self
DDTm Spatial Query
Questinn
Zemporal Query
Figure 3: Model architecture of our SAMA for referential grounded video understanding.
3.3.1
Overall Architecture
The overall architecture of the SAMA model is illustrated in Figure 3. Conventional Video LMMs typically process video input via two main strategies: (1) concatenating tokenized features from sparsely sampled keyframes [5, 67, 35], or (2) aggressively pooling merging features from individual frames into a very small number of tokens to accommodate longer sequences [33, 19, 51, 71]. We argue that both approaches have inherent limitations: the former often overlooks critical long-range temporal dynamics, while the latter tends to lose the fine-grained visual detail necessary for accurate referring segmentation and detailed object understanding.
SAMA is designed overcome these limitations by effectively integrating both rich, frame-level visual information and robust, long-range temporal context. As depicted, a video input first passes through a Visual Encoder (e.g., a pre-trained VIT [5]) to extract per-frame visual features F Frames E N PxxxxD, where N is the number of sampled frames, P is the number of patches per frame, and D feature dimension. A subset of these frame features, typically from keyframes, key C Frumess are directly flattened and projected to serve as fine-grained visual tokens for the LLM. Concurrently, features from a more extensive set of frames Flong C frames (potentially by proposed Versatile Spatial-Temporaal Cemporal-Contexx Aggregator (detailed in Sec. 3.3.2) 3) to derive compact yet informative representation of long-range temporal dynamics, denoted as Fagg-
frames) are processed
The language input is processed by a tokenizer to produce language embeddings Euang For referring understanding tasks involving specific objects, we incorporate object embeddings Evsy into E Specifically, Eoby is obtained by performing mask pooling on the visual features F Fkey of the relevant frame using a bounding box prompt for the target object. Eobj then inserted into the language embeddings Eunn at the position corresponding the object reference.
Finally, FREY. F aggg. and [Erang: Eabj] concatenated and fed into a large language model 4PIMM generate the textual response. For grounding tasks, if the LLM outputs a special [SEC] token, the hidden state corresponding this token HSEG is passed the decoder of a pre-trained SAM2 model [50] to produce the final segmentation mask Morj for the referred object.
3.3.2 Versatile
Aggregator
To efficiently capture both spatial details and temporal evolution from a potentially large number of video frames F longg we introduce three-stage aggregator module.
5

---

Spatial Aggregator. Given Ny frames selected for long-range oangrannge temporal modeling, where each long R. the initial goal of the spatial aggregator is to condense the spatial information from each frame while retaining essential semantics, thereby reducing compu- tational load for subsequent temporal modeling. Inspired by Q-Former [31], for each frame i, we employ set of Kş learnable spatial queries Qs E RKxxDD_ These queries interact with the frame's
frame has visual features F
patch features F( through cross attention mechanisms, followed by feed-forward networks (FFN),
long
to produce a compact representation g E RKxxDD;
Z = SpatialAgg( Qs, Flogg)
(I)
In our implementation, we use Kg = 32 tokens per frame. This results in a sequence of spatially (1)
aggregated tokens Zş = |Z
S g (2)
N RsN
Temporal Aggregator. To capture comprehensive spatio-temporal information relevant to the input query, the sequence of spatially aggregated tokens Zs is processed by the temporal aggregator. We employ sliding window approach: tokens from Zg corresponding to window of Wi frames are iteratively fed into temporal Q-Former structure. The temporal aggregator takes Kr learnable temporal queries Qr E R De and interacts with the windowed spatial tokens Z. window. Crucially, the textual question embedding Equestion (and any associated object embeddings E,nj if present within the question context) is also injected into this temporal Q-Former by concatenation with the temporal queries, which guides the aggregation process to focus on temporal dynamics relevant to the query:
Zyjindow TemporalAgg' Qr: Zs windoww Equestio:
Eobil).
The outputs from each sliding window are then concatenated to yield a query-awar spatio-temporal representation Zy R finat where K final is the resulting number of temporal tokens.
Context Aggregator. Finally, to ensure that the spatio-temporal information Zr effectively aligned and integrated with the original fine-grained visual features F Flong we employ a context aggregator. This module uses the frame features longg as queries, and the learned query-aware temporal features ZT keys and values in an attention mechanism, allowing the model to selectively enhance the frame representations with relevant long-range temporal context. Subsequently, we apply adaptive average pooling the temporalyy enhanced frame features, compressing each frame into a single token. These tokens are then projected through a linear layer to align with the language embedding space of the LLM. The final aggregated context feature F Fagy is obtained through the following process:
F agg Mean(Sollmax W x (ZrWK)//VC) x Z, W w', F E NR
(3)
where W W WV, w denote the linear projection layer. F agg combined with the fine- ine-grained keyframe features and language features are fed into the LLM to generate the target response.
3.4 SAMA-Bench
To evaluate the integrated referen- tial understanding and grounded di- alogue capabilities of Video LMMs,
Behavior Questions
we propose SAMA-Bench, constructed using the same annotation pipeline as SAMA-239K. SAMA-Bench. com- prises 5,067 questions synthesized from 522 videos across four public validation datasets: MeViS [9], Ref- YouTube- VOS [53], LVVIS [58], and VidSTG [74]. It includes two sub-benchmarks:
Questions
Context
7%
9%
SAMA Bench-G
26-8
7%
8%
Bench-C
SAMA
34%
vehicle
tool
animal
furniture
Spettal Relatiosship
8%
appliance
clathing
17%
22%
Interoction Questions
12%
19%
food
Attribtt
Questions
(a) Figure Data characteristics of SAMA-Bench.
Question types in Bend-G
(b) Categary list Banc-C iexch
SAMA-Bench 9 (3,036 questions) and DAMA-Bencht (2,031 questions). The distributions of ques- tion types and object categories are shown in Figure 4.

---

Table I: Performance comparisons SAMA-Bench The best results boldfaced, and second-best results are underlined. "_" denotes that the model does not support the specified output format. Entries in gray represent that the original model is incapable of performing the task. Values in red show SAMA's performance change relative to corresponding Sa2VA variants.
Method
B
mloU
Recall
METEOR
CIDEr
CLAIR
METEOR
CIDE
CLAIR
Genoralar Medels
LitenLL2.5000 [S]
0.14
0.33
DA7
0,00
0.31
Cemini-2.0 Fladh [54)
0.11
0.34
0.53
0.11
0.16
0.52
Genini Pro
0.48
0.13
0.27
Specialisr Madel
Imagedeeel models
GlaMM [43] SAM2 [50]
0.28
0.04
0.04
0.03
0.16
0.04
0.02
0.33
Shikra [2] SAME (50)
0.27
0.26
0.08
0.35
0.32
0.04
0.01
0.29
Ferne-7h (63) SAM2 (50)
0.64
0.44
0.14
0.21
0.37
0,10
0.12
0.31
SAM2 [50]
0.43
0.20
0.39
0.11
0.10
Raloi-dend awodids
Sa2VA.11B (67)
0.07
Sa0VA.4B (57)
0.55
0.25
0.05
0.02
0.19
0.00
0.00
0.07
SaCVA-88 (67)
0.64
0.17
0.02
0.02
0:30
0.00
0.00
0.13
SAMA-1I
0.67 (0.501)
0.53 (0.467)
0.16 (3.96))
0.56 (8.48))
0.53 (0,231)
0.14 (000073
0.31 00,2915
0:45 (8.197)
SAMA-4B
0.62 (1:147)
0,58 00.977
0.17 (8.121)
0.65
0.57 (0.1871
0.11 (6.137)
0.50 (1,311
SAMA-BB
0.69
0.32
(0.37)
SAMA-Bench
This sub-benchmark evaluates a model's ability to conduct fine- anee-graiedd, referential. and grounded visual dialogue. It focuses on region-level understanding and responses that require precise spatio-temporal grounding. As illustrated in Figure (a). question types include object behaviors, interactions, attributes, spatial relationships, and more, requiring identification and reasoning over grounded entities. Following GLaMM [49, 45], we assess grounding performance using mloU and Recall, and dialogue quality using METEOR, CIDEr, and CLAIR.
SAMA-Bench This nub-benchmark focuses on generating accurate descriptions for specified spatio-temporal video regions. As shown in Figure 4 (b), the benchmark spans diverse object categories, including person, animal, tool, vehicle, furniture, appliance, clothing, and food. Given first-frame prompt, models are required to generate detailed captions for target region. Description quality is evaluated using METEOR, CIDEr, and CLAIR.
Experiments
4.1 Experimental Setup
Training Datasets. SAMA is trained on a diverse set of image/video QA and referring segmenta- tion/grounding datasets, including LLa 665K [38]. the ChatUniVi [21] dataset, image- based interring/roundiigg data (refCOCO/+/g [23, 43], GRand-F [49]), and video referring segmentation datasets Yref-YouTTuee-OS. [53], Me [9], and ReVOS [61]). Critically, training is enhanced with our proposed SAMA-239K dataset, comprising 239K instances of referential grounded dialogue and object-level descriptions.
Implementation Details. We implement SAMA leveraging the XTuner [7] codebase. Model training and inference is conducted on NVIDIA A.00 GPUs (80GB). During the instruction tuning phase, we make the parameters of our peatialttemporl-coonte
aggregator and the decoder of the SAM2 model [50] trainable to learn spatial-tempen information and inject referential grounded video chat capability into the base model. The model weights are initialized using the pre-trained Sa2VA [67] to benefit from its existing grounding capabilities. The initial learning rate is set 4e-5. The maximum sequence length for the LLM is configured to 8.192.
Evaluation Metrics. For SAMA -Bench, grounding performance on SAMA-Bench is measured using mloU and Recall. Text generation quality on both SAMA-Bench' and SAMA-Bench is evaluated using METEOR, CIDEr, and CLAIR For standard image/video referring segmentation tasks, we adopt widely used metrics such as cloU and J&F. Chat performance is assessed with VLMEvallit [16].
4.2 Main Results
Performance Comparisons on SA.AABennh. Table presents a comprehensive performance comparison on SAMA-Bench, encompassing both the SAMA-Bench and BAMA-Bench" sub- benchmarks. To ensure fair evaluation across diverse model types, tailored protocols were applied. For lmaeellevel specialist models (c.g., GLaMM [49], Shikra [3], Ferret [65]) that accept coordinate
7

---

Table 2: Performance comparisons referring segmentation in images and videos. Bold and underlined values indicate the best and second-bes results, respectively. Red highlights SAMA'S performance difference from corresponding Sa2VA variants.
Method
Image Segmentation
Video Segmentation
[.)] Rot3000ggg
Image-level models
LISA-7B [28]
74.1
62.4
66.4
PixcLMM-7B [52]
73.0
66.3
69.3
GLaMM-7B [49]
79.5
72.6
74.2
28.9
HLVAA-G-7R [70]
77.1
71.5
GSVA-13B [60]
79.2
70.3
75.7
OMG-LLaVA-7R [7:]
78.0
69.1
72.9
29.9
Video-leevel
models models
VideoCLaMM [45]
45.15
69.5
VISA-13B [6]]
72.4
65.5
70.4
VSAe [1]
73.8
63.4
68.3
44.4
68.8
63.7
47.5
Sa2VA-4B
82.4
79.7
73.7
54.1
Sa2VA-8B [67]
82.6
78.0
80.2
32.2
51.5
75.9
72.3
$7.6
SAMA-IB
82.5 (0.11)
77.9
80.3
(0.61)
32.6 (1.61)
48.3 (1.91)
74.1 (0.41)
71.5
58.8 (4.71)
SAMA-SB
82.7 (0.17)
78.1 (0.11)
80.6 (0.31)
31.7 (0.5L)
53.7 (2.21)
77.3
72.8 (0.51)
59.1 (1.51)
Table Performance comparisons on chat benchmarks in images and videos. Bold and underlined values indicate the best and second-best results, respectively.
Image Chat
Video Chat
Method
MMF (17)
MMilench
SEFD-Bench
AID
MMStar
SQA tH1 (40)
Villo-MEE (18]
Generalist MLLMs
LLAVA-I.5-133 [36]
1531
70.1
Video-LLaVA-77B [35]
60.9
39.9
IJaMA-VID-7B [33]
1521
65.
59.9
MPLUG-OW13-8B [63]
77.6
$3.5
VL2-88 [6]
81.7
76.2
54.0
MLLMs with segmentation capability
Image-level models
PixelLM-7B [52]
309/135
17.4
GLaMM-7B [49]
36.8
OMG-LLaVA-7B [73)
1177/235
47.9
56.5
Viveo-level models
Sa2VA-8B [67]
1690/610
84.4
76.5
82.7
62.4
97.4
54.3
SAMA-8B
1639/621
80.8
76.2
79.4
60.1
95.3
51.8
inputs but not direct video, we adopted multi-step evaluation. First, these models queried with questions containing target object coordinates from the initial video frame. Their coordinate-inclusse textual responses were then processed by Gemini to validate object correspondence. Finally, these val- idated coordinates prompted SAM2 [50] to generate segmentation masks for the video. For generalist and video models lacking explicit coordinate input (e.g., Gemini, IntervL,, Sa2VA), visual cues like colored bounding boxes were overlaid on frames for querying their referential capabilities. The results in Table highlight two key findings: (1) SAMA consistently outperforms all baselines on both sub- bub-benchmarks, Notably, SAMA- 8B achieves an mloU of 0.70 and a Recall of 0.55, significantly surpassing Sa2VA. Shikra, and GLaMM, and exceeding the strong Ferret- + SAM2 baseline by 6% mloU and 12% Recall. (2) SAMA also demonstrates strong text generation capabilities. It achieves the highest METEOR and CIDEr scores among all models, even outperforming Gemini-1.5 Pro, and significantly surpassing Ferret. These results underscore SAMA's superior apatio-tempora reasoning and fine-grained grounding abilities.
Performance Comparisons on Referring Segmentation in Images and Videos. The results in Table 2 demonstrate SAMA's robust and consistent performance across both image and video referring segmentation tasks. (1) SAMA achieves superior image segmentation performance. Specifically, SAMA-8B obtains top scores on RefCOCO (82.7), RefCOCO+ (78.1). and RefCOCOg (80.6). outperforming all previous models, including GLaMM-7B and VISA-13B. Furthermore, on the GCG benchmark, SAMA-4B reaches 32.6, surpassing its Sa2VA-4B variant by a notable margin of +1.6. (2) SAMA sets new state-of-the-a results video segmentation. On video benchmarks, SAMA-8B achieves 53.7 on MeViS, 77.3 on Ref-DAVISI7, and 59.1 on RevOS- outperforrngg Sa2VA-8B by 2.2, 1. 1.4, and 1.5 points, respectively. These consistent improvements highlight SAMA's enhanced grounding ability.

---

Table 4: Ablation studies spatial-eemporallcomteex aggregator and SAMA-239K dataset.
Sepmoentation
METBOR
(9)
(2)
[6]]
SAMA-4B
0
0.55
0.17
0.65
0.57
55.4
74.0
2/o S2C
0.66 13
0,53 (00211)
0.15 (8821)
DU63 (UA21)
0,50 (RU77))
57.9-2.51
74.5 (8.51)
58.3 (0.3)
v/o 547A-259K
4.46 21
0.10 (145L)
0.04 (8.13))
0.22 (RASU)
0.22 (8,35))
(RA7)
74.200.21
$5.6 (3.2))
Object Prompts
Prediction results from SAMA
How does the movement of <abject> change over time? ? Is there anything else in the scene besides <object> and <object>>
SAMA-08
Pemre-33445400
2 Foiled Object Segmentation and Association
@ Foiled Temporal Reasoning
The movement (484, 427, 641, 472] is interealingly dynamic. shows skatsboardar in mid-air, performing tricks off concrete wall. No, the only objects in the scene are (482. 63, 652, 458) and (484, 427, 641, 472), There are no other items present.
Ferret
Referentiol Grounded Dialogue
Video Spotio- Temporal Reasoning
ape slateboardsip-BEG initally at
moving spothe
personsiip- (BEG) jumpa lands directly
spothe personslpo sgothe skataboarddjpe background, streetighttsp
spobuldingecpp sp2a
SAMA
Pecul
Ceer
CLAMR
Figure 5: Visual comparisons between our SAMA and Ferret. Best viewed with zoom.
Performance Comparisons on Chat Benchmarks in Images and Videos To evaluate SAMA's general conversational ability beyond SAMA- -Bncc, we test it on standard image and video chat benchmarks. As shown in Table 3, SAMA performs robustly overall, with only slight drop in some general chat metrics compared to Sa2VA. These results warrant contextual interpretation. First, SAMA's primary strength lies in unifying fine-grained referring, grounding, and dialogue within videos- capabilitiis that most grounding-focuse image-level grounding models like GLaMM-7B, SAMA-8B achieves significantly higher scores on shared benchmarks such as MME (1665/593 vs. 14/9), reflecting its superior dialogue ability. Second, although SAMA is mainly optimized for grounding, it still performs competitively on general conversation tasks, achieving 80.8 on MMBench, 76.2 on SEED-Bench, and 51.8 on Video-MME, which is comparable to generalist video LMMs not tailored for grounding.
models lack. For instance, compared to
4.3 Ablation Studies
Here, we conduct ablation studies from both model and data perspectives to thoroughly evaluate the effectiveness of the SAMA architecture and the SAMA-239K dataset. As shown in Table 4, removing the
aggregator (w/o STC) from SAMA-4B leads to performance drops across all metrics SAMA-Bench', ", with mloU decreasing from 0.69 to 0.66 and CLAIR from 0.57 0.50. More critically, training SAMA-4B without the SAMA-2339 dataset (w/o SAMA-239H results in a near-total collapse of grounded dialogue capabilities, with Recall plummeting O. 0.10 and CIDEr to 0.02. This highlights the dataset's essential role in supporting robust referring understanding and grounded chat. Figure 5 further illustrates that SAMA, trained on SAMA-239K. exhibits substantially stronger referential understanding and temporally grounded dialogue than Ferret. However, neither the STC module nor the SAMA-239K dataset consistently improves performance on MeViS (val_u) and Ref-DAVIS. We recognize that building universally effective components and datasets for all downstream tasks is inherently challenging and plan to address these limitations future work.
5 Conclusion
In this work, we introduced a comprehensive solution to equip Video Large Multimodal Models with fine-grained referential understanding and grounded dialogue capabilities. Our contributions are threefold: 1) SAMA-2330 239K, a large-scale video instruction dataset designed to unify referential under- standing and grounding; 2) SAMA Model, a novel Video LMM architecture that supports detailed video comprehension and segmentation; and 3) SAMA-Rench, curated benchmark for evaluating integrated referential understanding and grounded dialogue videos. Extensive experiments and analysis validate the effectiveness of dataset, model, and benchmark. We hope work provides

---

useful step toward advancing fineggrained grounding and interactive understanding in video-based multimodal systems.
Limitations and Future Work. SAMA has several limitations that offer opportunities for future work. Currently, it only handles box-based prompts, and expanding to other input formats like points masks could improve its flexibility. While strong visual grounding, its performance general dialogue tasks still lags behind specialized conversational models, suggesting room for better reasoning integration. Like prior work, we observe that joint training grounding and dialogue can cause interference, pointing to a need for more robust multi-task learning approaches. Finally, SAMA-239K is currently limited to short clips; scaling longer videos would better support research on complex, long-range temporal understanding.
Acknowledgements
This work is in part supported by National Key R&D Program of China (Grant No. 20222D0160103) National Natural Science Foundation of China (Grant No. 62276067), and National Natural Science Foundation of China (Grant No. 62472104).
The computations in this research were performed using the CFFF platform of Fudan University.
References
[1] Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language instructed reasoning segmentation in videos. In NeurlPS, 2024.
[2] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon. Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" video" in video-language understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2917-2927, 2022.
[3] Keqin Chen, Zhao Zhang. Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's 's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
[4] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan. Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision laiiunl models? arXiv preprint A4KY::2403.2333300 2024.
[5] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundarics of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.052771 2024.
[6] Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404. 16821, 2024.
[7] XTuner Contributors. Xtuner: A toolkit for efficiently fine-tuning 1lm. https: http:://ggtuubb.oo//teeennt
xtuner, 2023.
[8] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards gencral-purpose vision-language models with instruction tuning. arXiv preprint 2.Xiv:2305.06500 2023.
[9] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2694-2703, 2703, 2023.
[10] Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. MOSE: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEFF/CVF international conference on computer vision, pages 20224-20234, 2023.
[11] Henghui Ding, Chang Liu, Shuting He. Kaining Ying, Xudong Jiang, Chen Change Loy, and Yu-Gang Jiang. MeViS: A multi-modal dataset for referring motion expression video segmentation. TPAMI, 2025.
[12] Henghui Ding, Chang Liu, Suchen Wang, and Xudong Jiang. Vision-language transformer and query generation for referring segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16321-16330, 2021.
[13] Henghui Ding, Song Tang, Shuting He, Chang Liu, Zuxuan Wu, and Yu- Gang Jiang. Multimodal referring segmentation: A survey, arXiv preprint 2508 00265, 2025.
10

---

[14] Henghui Ding, Kaining Ying, Chang Liu, Shuting He, Xudong Jiang, Yu-Gang Jiang, Philip HS Torr, and Song Bai. MOSEv2: A more challenging dataset for video object segmentation in complex scenes. arXiv preprint arxiv:2508.00000 05630, 2025.
[15] Dexin Duan, Peilin Liu, Bingwei Hui, and Fei Wen. Brain-inspired online adaptation for remote sensing with spiking neural network. IEEE Transactions on Geoscience and Remote Sensing, 2025.
[16] Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. In ACMMM, 2024.
[17] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang. Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306.11 13394, 2023.
[18] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075. 2024.
[19] Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: Memory-augmented augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 13504- 13514, 2024.
[20] Miran Heo, Min-Hung Chen, De-An Huang. Sifei Liu, Subhashree Radhakrishnan, Seon Joo Kim, Yu- Chiang Frank Wang. and Ryo Hachiuma. Omni-rgpt: Unifying image and video region-level understanding via token marks. arXiv preprint arXiv:250I. 083326, 2025.
[21] Peng Jin, Ryucchi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, 2024.
[22] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier. Sudheendra Vijyanarasimman. Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action vidco datasct. arXiv preprint wwxiv:705.0691 77750055500 2017.
[23] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring objects in photographs of natural scenes. In EMNLP, 2014.
[24] Lei Ke, Mingqiao Yc, Martin Danelljan, Yu- Yu-Wing Tai, Chi-Keung Tang, Fisher Yu, et al. Segment anything in high quality. Advances in Neural Information Processing Systems, 36:29914- 14-29934, 2023.
[25] Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. In ACCV, 2018.
[26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. ICCV, 2023.
[27] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pages 706-715, 2017.
[28] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In CVPR, 2024.
[29] Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang. Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench: Benchmarking multimodal large language models. In CVPR, 2024.
[30] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chumyuan Li. Llaa--ncttinnterleav: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07895 2024.
[31] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre- pre-training with frozen image encoders and large language models. In ICML, 2023.
[32] Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang, Yi Liu, 7un Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-22206 2024.
[33] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[34] Long Lian, Yifan Ding, Yunhao Ge. Sifei Liu, Hanzi Mao. Boyi Li, Marco Pavone, Ming- Yu Liu, Trevor Darrell, Adam Yala, et al. Describe anything: Detailed localized image and video captioning. arXiv preprint 250v 16072, 2025.
[35] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.1012 2023.
[36] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.
11

---

[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-33916, 2023.
[38] Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan Zou, Jianwei Yang, Hang Su, Jun Zhu, et al. Llava-plus: Learning to use tools for creating multimodal agents. In ECCV, 2024.
[39] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang. Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In ECCV, 2024.
[40] Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurlPS, 2022.
[41] Xingjun Ma, Yifeng Gao, Yixu Wang, Ruofan Wang. Xin Wang, Ye Sun, Yifan Ding, Hengyuan Xu, Yunhao Chen, Yunhao Zhao, et al. Safety at scale: comprehensive survey of large model and agent safety, Foundations and Trends@ Privacy and Security, 8(3-4):254 469, 2025.
[42] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video aidoo-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.0544 2023.
[43] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy.
Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11 10-20, 2016.
[44] Jiaxu Miao, Xiaohan Wang, Yu Wu, Wei Li, Xu Zhang, Yunchao Wei, and Yi Yang. Large-scale video panoptic segmentation in the wild: A benchmark. CVPR, 2022.
[45] Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, and Salman Khan. Videoglamm: large multimodal model for pixel-level visual grounding in videos. arXiv preprint
2tti:2241..0422,22000
[46] OpenAL. GPT-4 technical report. httpa://arxiv... wttp://w...rr/a///333.3333 0bbba//33300774,, 2023.
[47] Zhiliang Peng, Wenhui Wang. Li Dong, Yaru Hao, Shaohan Huang. Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv preprint arXiv:2306. 14824, 2023.
[48] Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, and Yunjie Tian. Artemis: Towards referential understanding in complex videos. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[49] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Ahdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming- Husgn Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13009-13018, 2024.
[50] Nikhila Ravi, Valentin Gabeur, Than-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.0077 2024.
[51] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Hou. Timechat: A time- sensitive multimodal large language model for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313- 14323, 2024.
[52] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojic Jin. Pixellm: Pixel reasoning with large multimodal model. In CVPR, 2024.
[53] Seonguk Seo, Joon- YoungY Lee, and Bohyung Han. URVOS: Unified referring video object segmentation network with a large-scale benchmark. In ECCV, 2020.
[54] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Saan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2.12.. 1805, 2023.
[55] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer. Damien Vincent. Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint 403.552403.05530, 2024.
[56] Yunjie Tian, Tianren Ma, Lingxi Xic, and Qixiang Yc, Chatterbox: Multimodal referring and grounding with chain-of-questions. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7401-7409, 2025.
[57] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux. Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23X 2023.
12

---

[58]
Haochen Wang. Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie. and Efstratios
Gavves. Towards open-vocabulary video instance segmentation. In proceedings of the IEEE/CVF international conference on computer vision, pages 4057-4066, 2023.
[59]
Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang. Lei
Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475-122 117759111149,,,, 2024.
[60] Zhuofan Xia, Dongehen Han, Yizeng Han, Xuran Pan. Shiji Song, and Gao Huang. Gsva: Generalized segmentation via multimodal large language models. In CVPR, 2024.
[61] Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. In European Conference Computer Vision, pages 98-115. Springer, 2024.
[62] Jianwei Yang. Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting unleashes extraordinary visual grounding in gpt-4v, arXiv preprint arXiv:2310. 11411, 2023.
[63] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mPLUG-OWI3: Towards long image-sequence understanding in multi-modal large language models. arXiv preprint, 2024.
[64] Kaining Ying, Henghui Ding, Guangquan Jie, and Yu-Gang Jiang. Towards omnimodal expressions and reasoning in referring audio-visual segmentation. arXiv preprint arXiv:2507.22886. 2025.
[65] Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. ICLR, 2024.
[66] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In ECCV, 2016.
[67] Haobo Yuan. Xiangtai Li. Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji, Yunhai Tong, Lu Qi. Jiashi Feng, and Ming-Hsuan Yang. Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos. arXiv preprint arXiv:250J.04M0
2025.
[68] Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. Videorefer suite: Advancing spatial-temporal object understanding with video Ilm. arXiv preprint arXiv:2501. .05599, 2024.
[69] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-uus tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858, 2023.
[70] Hao Zhang, Hongyang Li. Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao. Chunyuan Li. Jainwei Yang, al. Llava-grounding: Grounded visual chat with large multimodal models. In ECCV, 2024.
[71] Shaolei Zhang, Qingkai Fang. Zhe Yang, and Yang Feng. Llava- mini: Efficient image and video large multimodal models with one vision token. arXiv preprint arXiv:2501.0366 2025.
[72] Shilong Zhang, Peize Sun, Shoufa Chen, Min Xiao, Wenqi Shao, Wenwei Zhang, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on region-of-interest arXiv preprint or 2330:2330,, 2023.
[73] Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji, Change Loy Chen, and Shuicheng Yan. Omg-llava: Bridging image-level, object-level, pixel-level reasoning and understanding. In NeurlPS, 2024.
[74] 7hu Zhang, Zhou 7hao, Yang Zhao, Qi Wang, Huasheng Liu, and Lianli Gao. Where does it exist: Spatio-temporal video grounding for multi-form sentences. In Proceedings the IEFFJCVF Conference on Computer Vision and Puttern Recognition, pages 10668-10677, 2020.
[75] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vasion-language understanding with advanced large language models. arXiv preprint arXiv:2304.1059)
2023.
13

---

A Appendix
A.1 Broader Impacts
Our SAMA imposes several positive broader impacts. 1) SAMA introduces a unified framework that enhances the capabilities of Video Large Multimodal Models (LMMSs) by bridging fine-grained referential understanding and grounded, multi-turn dialogue- nwwo areas that have traditionally been addressed separately. This integration may encourage shift in video understanding research toward more contexl-aware multimodal systems that unify perception and interaction. 2) SAMA- 239K provides a high-quality dataset designed to support joint learning of video grounding and dialogue. It offers a valuable resource for advancing interactive video reasoning and object-centric video comprehension, and may also facilitate research grounding alignment and conversational multi-object tracking. 3) The SAMA model serves as an early baseline for combining referring understanding with pixel-Ievel grounding video. It demonstrates that detailed visual grounding and referential dialogue can be effectively optimized within a single cohesive architecture, potentially guiding future designs for complex video-language tasks. 4) SAMA-Bench addresses a key evaluation gap by introducing the first benchmark tailored to jointly assess referential video understanding and grounded dialogue. We believe that the SAMA suite-dataset, model, and benchmark- can help steer the development of future video LMMs and foster progress in multimodal human-Al interaction, where spatial-temporal grounding and communication are tightly coupled.
Ethical Considerations. SAMA-239K is constructed entirely from publicly available research datasets, ensuring no private or personally identifiable data are included. The released data, model, and code will be provided under a research-only license that explicitly prohibits malicious or unethical uses, such as surveillance, disinformation, or privacy violations. We encourage responsible use of SAMA to advance transparent and beneficial research in multimodal video understanding.
A.2
Data Curation
Table 5: SAMA-239K for referential grounded video chat training.
Dataset
Video Clips
Q&& Pairs
Object-level Deseriptions
MeViS [9]
1,087
10,257
3,550
Ref- Youtube- VOS [53]
1,976
15,622
4,929
LV-VIS [58]
2.595
30,943
14,268
SAV [50]
2,867
34,729
12,262
VidSTG [74]
6,618
80,745
31.996
239K
15,143
172,296
67,005
SAMA-239K. The details of SAMA-239K are summarized in Table 5. To construct the SAMA- 239K dataset, curated and transformed several existing video understanding datasets into unified referentially grounded video question-ansee answering format. As illustrated in Figure 2, our automatic data generation pipeline integrates both video segmentation and video grounding datasets. For segmentation datasets with ground-truth masks, we extract object coordinates directly from the masks. For grounding datasets lacking masks, employ HQ-SAM to generate pseudo masks. In both cases, we visualize each object by drawing a uniquely colored bounding box and assigning it a distinct tag. These annotated video frames, along with structured object information and tailored prompts, are fed into Gemini-1.5 Pro [55] to generate fine-grained object-level descriptions and multi-turn referential grounded dialogues. Below, we outline the detailed data curation process for the five primary sources contributing to our dataset: MeViS [9], Ref- You'Tbe VOS [53], LV-VIS [58], SAV [50], and VidSTG [74]. MeViS and Youbee VOS are large-scale referring video object segmentation datasets with rich pixel-level mask annotations aligned with textual expressions. For the SAV dataset, we adopt object-level referring expressions released by Sa2VA [67]. To ensure visual diversity and avoid trivial cases, we discard videos with only single object. For the remaining videos, we uniformly sample 16 frames and use segmentation masks to identify referred objects. Each object is visualized with a uniquely colored bounding box, establishing a one-to-one mapping between object and color. These annotated frames, along with referring expressions and color-object mappings, are integrated into prompt templates and fed to Gemini, which then generates fine fine-grained descriptions and multi-turn dialogues that emulate natural human interaction from basic observations to complex reasoning about object dynamics and relationships. LV-VIS [58] is large-vocabulas vidco instance
14

---

segmentation dataset with pixel-level masks and category labels. For each object, its category label included in the prompt guide Gemini in generating category-awar descriptions. All other steps- iteps-including frame sampling, bounding box visualization, and prompt construction--follee the same procedure. VidSTG [74] a video grounding dataset that provides bounding box annotations but lacks mask information. We uniformly sample each video at 4-frame intervals to extract JPEG frames. The frame-level bounding boxes are then used as input to HQ-SAM [24] to generate pseudo masks. The resulting data is subsequently processed following the same pipeline as the other datasets to produce training samples.
Table 6: SAMA ReMA-Bench for video referential grounded chat evaluation.
Datasel
Video Clips
Q&A Pairs
MeViS [9]
41
244
Ref-Youtube- VOS [53]
131
756
LV-VIS [58]
150
1,019
VidSTG [74]
200
1,019
SAMA-BencH:
522
3,038
Table 7: BAMA-Bench" for video referential captioning evaluation.
Dataset
Video Clips
Q&A Pairs
MeViS [9]
41
117
Ref-Youtube- VOS [53]
131
350
LV-VIS [58]
150
589
VidSTG [74]
200
975
SAMA R
522
2,031
SAMA-Bench. SAMA-Bench is constructed using the same annotation pipeline as SAMA-239K, followed by rigorous automatic filtering and manual verification remove low-quality or ambiguous samples. To ensure comprehensive evaluation across diverse video understanding scenarios, we randomly select subset of validation videos from four datasets: MeViS, Ref- YouTueeee VOS, LV- VIS, and VidSTG. As shown in Table 6, 3 consists of 3,038 video referential grounded chat questions, with 244 from ViS,S, 756 from Ref- YouTueeue VOS, 1,019 from LV-VIS, and 1,019 from VidSTG. Similarly, Table 7 summarizes SAMA- " which comprises 2,031 video referential captioning questions, sourced from the same set of videos: 117 from McViS, 350 from Ref- YouTube- VOS, 589 from LV-VIS, and 975 from VidSTG.
Table 8: Model performance under different training data proportions.
Training Data
MME
SEED
A12D
MMStar
SQAtest
MMBench
Video-MME
50% mix665k
1409
56.2
46.5
39.3
75.1
43.6
39,5
100% mix665k
1451
65.7
57.8
44.5
78.4
55.8
41.3
A.3 More Analysis
Improving Dialogue Performance. We investigate two approaches to improve SAMA's perfor- mance on general dialogue. We first validated the impact of expanding conversational training data. Keeping all other factors constant during training, we adjusted the proportion of the LLaVA-v1.55 mix665k dataset used in our instruction-tumie tuning mixture. The results in Table show clear, positive correlation between the amount of dialogue data and SAMA's performance across various image chat benchmarks, confirming this is direct and effective improvement strategy. Second, we hypothesize that a discrepancy in output granularity between general dialogue tasks and visual grounding can cause mutual interference during joint training and thus hurt dialogue performance. To probe this, we adjust the next-token prediction loss weight and observe its impact on chat results. As shown Table 9, we found that increasing the loss weight to 1.5 led to improved performance on most chat benchmarks. This implies that effective strategy is to re-balance the optimization objective to focus more on the holistic, dialogue-relevant output embeddings, rather than over-emphasizing coounning-sppciffe tokens.
15

---

Table 9: Model performance under different loss weight configurations.
Weight
Benchmurk
MMBBnch
100
611
51.7
169
493
1451
05.7
64.5
78.4
41.3
1.25
66-4
52.2
152
51.7
1424
61.4
52.5
42.8
76.4
49.7
90.2
1.0
66.9
515
15.0
520
1479
06.0
60.0
45.7
50.0
56.2
40.0
Table 10: Model performance under different prompt configurations.
SANA-Bench"
SAMA-Bench
Method
mloU
Recall
METEOR
CIDEr
METEOR
CIDEr
Point (1)
0.60
0.44
0.14
0.46
0.11
(2)
0.24
Point (4)
0.62
0.46
0.14
0.49
0.13
0.29
Point (8)
0.64
0.48
0.14
0.48
0.13
0.30
Box
0.66
0.49
0.15
0.50
0.13
0.30
Mask
0.65
0.50
0.14
0.46
0.13
0.31
Prompt Sensitivity. To validate SAMA's flexibility, conducted new experiments where extended the instruction tuning to include a mixture of prompt formats. We then evaluated the model across diverse set of inputs, including bounding boxes, masks, and varying number of points (1, 2, 4, and 8). The results in Table 10 demonstrate that SAMA maintains strong and consistent performance across all formats.
Table 11: Model performance SAMA Bench-G VidSTG subset.
Method
mloU
Recall
METEOR
CIDEr
55.8
36.3
SAMA-IB
61.2
45.3
15.5
42.6
Long-Vite Video Evaluation. To demonstrate SAMA's long-video capabilities, we evaluated it on the challenging VidSTG subset of SAMA Bench-G. As shown in Table 11, the results are twofold: first, SAMA-IB achieves satisfactory performance on this long-video-centric benchmark, confirming its ability to handle minute-level videos. Second, our model outperforms the powerful image- image-domain model, Ferret-13++SMM2 underscoring the unique challenges of referring and grounding in the video domain.
A.4 Visualization
Figure and Figure illustrate the results of SAMA on the video referential grounded chat task. Figure illustrates the results of SAMA on the video grounded description task.
Figure illustrates the results of SAMA on the video referential captioning task.
16

---

Object Prompts
ih the spetial relationship between the cyclists sobjects, cobjects, and sobject=? Response with interleaved segmentation masks.
<>> The eyc list the white jersey </p> [SEG) is riding behind <p> the cyclist the black jersey </p> The ey the yellow jersey <ip> [SEG) also riding behind <p> the cycllstin the black jersey </p> [SEG].
[SEG].
SAMA
Object Prompts
Video Prediction results from SAMA
What the cobject> doing throughout the video? Response with Interleaved segmentation masks.
The adult [SEG] assisting <p> the child [SEG] performing gymnastics routine. He
<p> <pp> [SEG) standing position
higher, preparing a swing on the rings.
SAMA
Object Prompts
Video Prediction results from SAMA
How does the person with <object> Interact with cobject>? Response with Interleaved segmentation masks.
The person <pp> [SEG] with <pp the bracelet <pp> [SEG] holding baby </p>> [SEG), and they feeding the giraffe <ip> [SEG] with <p> leaf <tp> [SEG].
SAMA
Figure 6: Visualizations of SAMA on the video referential grounded chat task. Best viewed in zoom.
17

---

Object Prompts
Video Prediction results from SAMA
What sobject> doing with sobject? Response with interleaved segmentation masks.
The person <pp> [SEG] Is holding
the joystick </p>> [SEG] and demonstrating its use.
SAMA
Object Prompts
Video Prediction results from SAMA
Besides <objects, sobjects, and cobjects, what else can be seen in the video? Response with interleaved segmentation
In addition to the hockey stick <ip> [SEG), the glove
[SEG], and <p> the puck <pp> video hackey goal, white <p> <ip>> <p> <pp>
[SEG].
SAMA
Object Prompts
is cobject> doing relation to cobject>? Response with interleaved segmentation
masks.
<>> The person the blue shirt <pp>
[SEG] Is serving the tannis ball to <p> the man in the dark blue
shirt
[SEG].
SAMA
Figure 7: Visualizations of SAMA on the video referential grounded chat task. Best viewed in zoom.
18

---

riscribe the detail segmentation
person
<pp>>
[SEG] is interacting with racing wheel <ip> [SEG] desk.
keyboard
<pp>
<p>
<ip>
on <p> computer monitor
<pp> [SEG]
visible in the background.
SAMA
The video shows group of people interacting with a bull in an outdoor pen. <p> A man in pink shirt <ip> [SEG]
and <p> man in a black shirt <ip> [SEG] are standingg an wooden platform, with <p>> man a gray jacket <ip> [S holding a red flag. <p> man in a black shirt <ip>: [G] end <p> man a white shirt <ip> [SEG) are also present
brown bull <pp> [SEG] seen moving around the pen.
SAMA
nighttime scene <p> A man <ip> [SEG] is standing near the car, possibly directing <p> Another man <pp> [SEG] Is
car <pp> [SEG] driving road.
<p>
standing further back, observing the crowd of people [G be seen the background, watching the
SAMA
child </p> [SEG] is playing with <p> yellow ball <pp> [SEG]and blue ball [SEG].
<p> </p>> [SEG] stands nearby, observing. <po- toy walker <pp> [SEG] also visible
background.
SAMA
Figure Visualizations of SAMA on the video grounded description task. Best viewed in zoom.
19

---

you region <object>
A brown horse is wading through shallow river. carrying rider, who wearing black shirt and blue jeans. The horse is moving slowly appears enjoying water.
red bicycle with white frame is being ridden by
tag cycllng outfit. The bikke
SAMA
SAMA
person wearing a red shirt and light-colored pants is holding fencing foil. They engaged in fencing match another person.
person playing guitar stage. guitar color, passibly white or beige. The person is wearing dark-colored shirt and light-colored pants.
SAMA
SAMA
pair skis is visible, with left slightly ahead of the right. The skis are angled slightly inward, suggesting skier making turn.
racket held player on side the court: The racket is in mation, suggesting the player is actively engaged in the game.
SAMA
SAMA
person wearing a yellow helmet and a black life is paddling green kayak. They are using double-bladed paddle navigate river.
The paddle houble-bladed, black and white color, and is being used by the kayaker navigate the river.
SAMA
SAMA
Figure 9: Visualizations of SAMA on the video referential captioning task. Best viewed in zoom.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The contributions of this paper are clearly presented in the abstract and introduction.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We note that our method still has limitations and room for improvement in our experiments.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model cell-speciiiaatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer:
[NA]
Justification:
This paper does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification:
Information related the experiment is detailed in the experiment setup
section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Source code will be released to foster the community.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //niss... wttpssccccppcccccccetce.ceccccct http/gudddes/oooee /Cooeeuuoiiiseeeeeee ssisnnolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
All relevant settings are described in the experimental setup section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
The experimental results in this paper require further error analysis to be reported.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe the training settings for each experiment in the experimental section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: /ete://nneuupp/e cc wwcpppcppc/eeeeecccceecceceeeeee wtt.llii///hiiicuideeeeeeeeee
Answer: [Yes]
Justification: This research complied fully with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We explore some possible societal impacts of our method in the conclusion and appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper proposes a new task, data and model, so there no safety risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Relevant sources used in this paper are appropriately cited.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com aamdaatseess has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[Yes]
Justification:
Our code, data and model will be released foster the community.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The experiments in this paper does not include these contents.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The experiments in this article did involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
26

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper clearly describes how to use a large language model to generate our training data.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: :/ttss//ae /ttprr/er.pp.///// for what should or should be described.
27


Reviews:
[{"review_id": "AWAONYefxO", "reviewer": "Reviewer_tQRK", "summary": "This paper introduces SAMA, a framework for fine-grained spatial-temporal understanding and grounded dialogue in videos.\n\n- Dataset: SAMA-239K, a large-scale dataset with 15K videos, 172K referential grounded video-QA pairs, and 67K object-level descriptions. It includes video referring understanding, grounding, and multi-turn video chat. ​\n- Model: The SAMA model integrates a spatial-temporal context aggregator and SAM2 for precise grounding. ​\n- Benchmark: SAMA-Bench is composed of 522 videos with 5,067 questions.", "questions": "- Can SAMA be extended to handle other input formats, such as points or masks, beyond box-based prompts? ​ \n- What specific steps can be taken to improve SAMA's performance on general dialogue tasks? Could integrating specialized reasoning modules or leveraging additional conversational datasets help? ​\n- What are the main challenges in scaling SAMA to longer videos? Are there computational or architectural limitations that need to be addressed? ​\n- How does interference between grounding and dialogue tasks manifest during training? ​ Are there specific techniques or architectures that could mitigate this issue? ​\n- Are there any safeguards or ethical considerations for the release of SAMA, particularly given its potential for misuse in surveillance or disinformation? ​", "limitations": "- SAMA currently handles only box-based prompts, limiting its flexibility. ​ Expanding to other input formats like points or masks could improve usability. ​\n- SAMA-239K is limited to short video clips, and scaling to longer videos would better support research on complex, long-range temporal understanding. ​", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper introduces SAMA, a unified framework that integrates fine-grained referential understanding and grounded dialogue capabilities in videos, addressing a critical gap in video-based multimodal systems. ​\n- The proposed SAMA-239K dataset is a significant contribution, offering a large-scale, high-quality resource for joint learning of video referring understanding and grounding. ​\n- SAMA-Bench provides a well-designed evaluation framework for assessing integrated referential understanding and grounded dialogue capabilities, filling a key evaluation gap in the field. ​\n- The introduction of the spatial-temporal-context aggregator and integration with SAM2 enhances SAMA's ability to capture both fine-grained spatial details and long-range temporal dynamics, making it robust for complex video understanding tasks. ​\n\nWeaknesses:\n- SAMA currently supports only box-based prompts, which restricts its flexibility and applicability in diverse scenarios. ​ Expanding to other input formats like points or masks would improve usability. ​\n- While strong in visual grounding, SAMA's performance on general dialogue tasks lags behind specialized conversational models, indicating a need for better reasoning integration. ​\n- SAMA-239K is limited to short video clips, which constrains its applicability to long-range temporal understanding. ​ Scaling to longer videos would enhance its utility for complex scenarios. ​", "was_revised": true, "final_justification": "After reading the rebuttal, my concerns have been addressed. I raised my score and thanks for all the efforts.", "rebuttal": "We sincerely thank Reviewer tQRK for the insightful feedback and for recognizing our work's contributions.\n\n**Q1: Can SAMA be extended to handle other input formats, such as points or masks, beyond box-based prompts?**\n\n**A1:** Yes, absolutely. Our framework is inherently flexible and not limited to box prompts. Initially, we focused on box prompts for two practical reasons: (1)  Point prompts can be ambiguous (e.g., clicking on a person’s shirt leaves the model unsure whether the query targets the person or the clothing). (2) Mask prompts are hard for users to supply and typically require an extra segmentation module, raising the bar for real‑world use. Box prompts offer a good balance of user-friendliness and precision.\n\nTo validate SAMA's flexibility, we conducted new experiments where we extended the instruction tuning to include a mixture of prompt formats. We then evaluated the model across a diverse set of inputs, including bounding boxes, masks, and a varying number of points (1, 2, 4, and 8). The results demonstrate that SAMA maintains strong and consistent performance across all formats.\n|Prompt|SAMA Bench-G| | | |SAMA Bench-C| |\n|:----|:----|:----|:----|:----|:----|:----|\n| |mIoU|Recall|METEOR|CIDEr|METEOR|CIDEr|\n|Point(1)|0.60|0.44|0.14|0.46|0.11|0.20|\n|Point(2)|0.61|0.45|0.14|0.48|0.12|0.24|\n|Point(4)|0.62|0.46|0.14|0.49|0.13|0.29|\n|Point(8)|0.64|0.48|0.14|0.48|0.13|0.30|\n|Box|0.66|0.49|0.15|0.50|0.13|0.30|\n|Mask|0.65|0.50|0.14|0.46|0.13|0.31|\n\n**Q2: What specific steps can be taken to improve SAMA's performance on general dialogue tasks?**\n\n**A2:** Before answering this question, please allow us to briefly clarify the primary goal of our work. \n\nSAMA's core objective was not merely to advance general dialogue skills, but to address a more fundamental, long-standing gap in video understanding: *to prove that video referring understanding and video grounding—two highly related but historically separate tasks—could finally be unified to create a more powerful and versatile vision system.*\n\nPrior to our work, this unified task was undefined: no large-scale dataset existed for joint training, no model possessed this integrated capability, and no benchmark was available for evaluation. SAMA is the first comprehensive solution designed to fill this critical research vacuum, providing the definition, data, model, and benchmark for this new field. Our experiments confirm its success, demonstrating that SAMA achieves sota referential grounded video chat capabilities while maintaining highly competitive general dialogue performance.\n\nAs for this question, we believe that SAMA's performance on general dialogue tasks can be enhanced via following approaches:\n* Expanding the Training Data: For the sake of a fair comparison, our SAMA was instruction-tuned on a data mixture identical to Sa2VA, which includes only 100k video‑dialogue samples from ChatUniVi. Therefore, a simple and effective method is to enrich this mixture with a larger proportion of high-quality, general conversational datasets.\n* Integrating Advanced Reasoning Methods: Recent studies [a] have shown that reinforcement learning can boost LLM performance on vision-related tasks. A promising path for SAMA is to first convert the training data to a Chain-of-Thought (CoT) format, which would teach the model to articulate its intermediate reasoning steps. Following this, we can apply a two-phase refinement process: initially, supervised fine-tuning on the CoT data would build a solid reasoning foundation, and subsequently, reinforcement learning (e.g., DPO or GRPO) could be used to optimize the model's decision-making pathways, further improving its robustness and accuracy.\n\n**Q3: SAMA-239K is limited to short video clips.**\n\n**A3:** We thank the reviewer for this question. \n* In fact, SAMA-239K is not limited to very short clips. SAMA-239K consists of 15,143 videos totaling 76.82 hours. The average video length is 18.3 seconds, with the longest video lasting 179.8 seconds. All questions in SAMA-239K are annotated by the state-of-the-art video model, Gemini, to ensure quality. **In comparison, SAMA‑239K offers 14 × more videos than VISA [b], 6 × more questions than VideoGLaMM [c], and 10.3 × the total duration of MOSE [d].** Importantly, SAMA-239K is the first dataset that uniquely supports both video referential dialogue and precise grounding. \n* More importantly, we provide a reproducible LLM-assisted annotation pipeline. This empowers the community to effortlessly generate referentially grounded dialogue data for much longer videos. *We will release all the annotation code, lowering the barrier for future research in this crucial direction.*\n\nWe genuinely believe the release of SAMA will significantly advance research in joint learning for video referring understanding and grounding within the community.\n\n**Q4: What are the main challenges in scaling SAMA to longer videos?**\n\n**A4:** As we mentioned in Q3, SAMA is already capable of processing videos of up to several minutes, thanks to our training on SAMA-239K.\nHowever, scaling SAMA to much longer, even hour-level videos presents two primary challenges. We outline these challenges below, along with promising solutions to overcome them:\n\n* Our current approach samples five keyframes to capture fine-grained visual detail. While effective for short videos, this fixed number may be insufficient to represent the full semantic content of a very long video. A potential solution is to develop an adaptive keyframe selection strategy, which dynamically identifies the most informative or visually diverse frames based on the video's content.\n* SAMA currently samples N frames for long-range context, converting them into N tokens for the LLM. For extremely long videos, a large N could increase computational load. To mitigate this, we could employ techniques such as token clustering to condense the temporal context, or integrate memory mechanisms to efficiently manage information over extended periods.\n\n**Q5: How does interference between grounding and dialogue tasks manifest during training? Are there specific techniques or architectures that could mitigate this issue?**\n\n**A5:** This interference is indeed a common challenge for segmentation-capable LMMs. Models like Sa2VA, PixelLM, and LLaVA-Grounding all exhibit a decline in conversational ability after fine-tuning on grounding data, while others like LISA, VideoLISA, and VideoGLaMM even show a much worse performance. We hypothesize this interference stems from two primary sources:\n\n* Discrepancy in Output Granularity: General dialogue tasks require the LLM to generate *diverse and rich output embeddings*. In contrast, the typical response in segmentation training data is a terse `\"Sure, it is [SEG]\"`. This provides no new general dialogue knowledge and contributes little to the optimization of conversational output embeddings.\n* Imbalance in Special Token Distribution: Grounding datasets are saturated with special tokens (e.g., `<p>`,  `</p>`, `[SEG]`). Crucially, these tokens do not exist in the LLM's pre-training vocabulary and thus lack the rich, generalized representations of regular word embeddings. Their frequent appearance during fine-tuning also causes the model to over-specialize on these \"weak\" tokens, biasing its generation towards a rigid, non-conversational style.\n\nTo explore a potential solution, we conducted a new experiment on SAMA-1B where we up-weighted the token-level cross-entropy loss for next-token prediction. We found that increasing this loss weight to 1.5 led to improved performance on most chat benchmarks. This implies that an effective mitigation strategy is to re-balance the optimization objective to focus more on the holistic, dialogue‑relevant output embeddings, rather than over-emphasizing grounding-specific tokens.\n| |SAMA Bench-G| | | |Chat Benchmark| | | | | | |\n|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|:----|\n|Loss weight|mIoU|Recall|METEOR|CIDEr|MME|SEED|AI2D|MMStar|SQAtest|MMBench|Video-MME|\n|1.00|67.1|51.7|14.9|49.7|1451|65.7|57.8|44.5|78.4|55.8|41.3|\n|1.25|66.4|52.2|15.2|51.7|1424|61.4|52.5|42.8|76.4|49.7|39.2|\n|1.50|66.9|51.5|15.0|52.0|1479|66.0|60.0|45.7|80.0|54.2|40.0|\n\nSimultaneously developing grounding skills while preserving chat capabilities is a non-trivial challenge, involving a delicate interplay of data curation and training strategies. SAMA strikes a good balance: it successfully introduces the field's first referentially grounded dialogue and achieves state-of-the-art visual grounding, all while retaining highly competitive chat performance. We will discuss this trade-off in our revision to inspire future work.\n\n**Q6: Are there any safeguards or ethical considerations for the release of SAMA?**\n\n**A6:** Thanks for this question. We take ethical considerations seriously. \n\n* Data Sourcing: SAMA-239K is built upon publicly available academic datasets created for research purposes. We do not use any private or sensitive data.\n* Model Release: We will release our data, model, and code under a research-oriented license that explicitly prohibits use in malicious applications, such as surveillance, creating disinformation, or violating privacy.\n* Broader Impact Statement: We will include a comprehensive \"Broader Impact and Ethical Considerations\" section, discussing potential misuse and outlining our mitigation strategies to encourage responsible research and development by the community.\n\n >[a] Visual-RFT: Visual Reinforcement Fine-Tuning. arxiv 2025\n\n >[b] VISA: Reasoning Video Object Segmentation via Large Language Models. ECCV 2024\n\n >[c] VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos. CVPR 2025\n\n >[d] MOSE: A New Dataset for Video Object Segmentation in Complex Scenes. ICCV 2023"}, {"review_id": "6vW0Vl87lR", "reviewer": "Reviewer_Dj2a", "summary": "The paper addresses the challenge that current Video LMMs struggle with fine-grained spatio-temporal understanding, as they typically handle video referring (understanding regions) and grounding (segmenting objects) as separate tasks.\n\nTo address this, the authors present: a new dataset, SAMA-239K, for joint training, a model named SAMA, and a benchmark, SAMA-Bench.\n\nThe SAMA model architecture integrates a spatio-temporal context aggregator with SAM2 to enhance video comprehension and precise object grounding jointly.\n\nThe SAMA-239K dataset, created using an automated pipeline with Gemini-1.5 Pro, contains 239K instances of referential grounded dialogue and object descriptions to enable unified learning.\n\nExperiments demonstrate that the SAMA model not only achieves descent performance on the new SAMA-Bench but also performs well on existing image and video referring segmentation benchmarks", "questions": "1. Please answer the questions and concerns in the weakness.\n\n2. Could you provide a more detailed analysis of the \"interference\" phenomenon between grounding and conversational skill?  Is it a case of catastrophic forgetting of general conversational skills, or does the model learn to prioritize grounding-specific token sequences over more detailed dialogue? \n\n3. What is your hypothesis for why the dataset and model failed to provide a benefit on MeViS and Ref-DAVIS? Do these benchmarks have distinct data characteristics that are not well-represented in your training data?", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n* The paper is well written and easy to follow. The figure visualization makes it easy to comprehend the method and the purposes of the dataset.\n* The paper presents a pipeline to address the problem of referentially grounded video chat, comprising a new dataset for training and testing, as well as a proposed model. The release of a new dataset may benefit the research community.\n* Extensive experiments and a thorough ablation study support the paper's claim. The study confirms that both the spatio-temporal aggregator and the SAMA-239K dataset are essential for the model's high performance.\n\nWeaknesses\n\n1. Method.  \n(a) Given that this is more of a dataset paper, the proposed method is not novel; the way of disintangling spatio and temporal representation mechanisms was explored previously and is widely used in designing video models.  \n(b) The authors acknowledge that jointly training for grounding and dialogue can cause \"interference\" between the tasks, indicating a need for more robust multi-task learning strategies. This implies that the goal of solving referring and grounding simultaneously has not yet been achieved.  \n(b) The SAMA model's flexibility is currently limited, as it only supports box-based object prompts and cannot handle other formats, such as points or masks.  \n(c) The current model only operates on the objects in the videos, neglecting the action as a whole, which limits the potential for action understanding in videos.\n\n\n2. Experiment.  \n(a) In Table 1, why is the Sa2Va performance so low? The METEROR and CIDEr score reaches 0 in benchC. Any particular reason for that?  \n(b) In Table 2, the ablation study reveals that the architectural and data contributions did not consistently improve performance across all tasks; specifically, on the MeViS and Ref-DAVIS benchmarks, their impact was not uniformly positive.  \n(c) In Table 3, the model's specialization in grounding tasks comes at a cost to its general conversational abilities. SAMA's performance shows a slight drop in some general chat metrics compared to Sa2VA. Showing a clear trade-off across different aspects.", "was_revised": true, "final_justification": "Thanks for the detailed rebuttal. I appreciate you running new experiments—the results showing flexibility with different prompts were very helpful, and your analysis of the task interference was insightful.\n\nWhile the paper is definitely stronger now, I'm holding my score steady because some of my original concerns remain. I agree the dataset is a valuable contribution, but the model's technical novelty is still a weak point. More importantly, your own explanation for the performance on MeViS and Ref-DAVIS confirms that the model has trouble generalizing to key existing benchmarks.\n\nBecause of these limitations, the work still feels like a borderline case to me. It's a solid contribution, but the issues with novelty and generalization prevent me from raising my score at this time.", "rebuttal": "We sincerely appreciate your insightful, constructive and encouraging reviews.\n\n**Q1: Given that this is more of a dataset paper, the proposed method is not novel.**\n\n**A1:** Please allow us to clarify the core novelty of our work. While some architectural components are inspired by prior work, our primary contribution is not an isolated component, but a holistic, foundational solution to a new and challenging problem.\n\nOur core objective was to address a fundamental, long-standing gap in video understanding: *proving that video referring understanding and video grounding—two historically separate but deeply related tasks—could finally be unified to create a more powerful and versatile vision system.*\n\nPrior to our work, this unified task was uncharted territory: no large-scale dataset existed for joint training, no model possessed this integrated capability, and no benchmark was available for evaluation. Our work is the first comprehensive solution designed to build this field from the ground up:\n\n* **We defined the task and created the foundational resources:** Our SAMA-239K dataset and SAMA-Bench are the first of their kind, providing the essential cornerstone for all future research in this direction.\n* **We presented the first strong model:** Our SAMA serves as the crucial first baseline, validating that this unified task is not just a concept, but an achievable reality.\n* **We will fully open-source our work:** To accelerate progress, we will release all our assets—data, benchmark, code, and pretrained weights.\n\n*We genuinely argue that establishing a new research paradigm, creating the entire ecosystem of resources to support it, and presenting the first strong, reproducible baseline is a foundational and valuable contribution to the community.*\n\n**Q2: The authors acknowledge that joint training for grounding and dialogue can cause \"interference\" between the tasks, indicating a need for more robust multi-task learning strategies. This implies that the goal of solving referring and grounding simultaneously has not yet been achieved.**\n\n**A2:** We would like to clarify that SAMA has indeed successfully solved referring and grounding simultaneously. We have rigorously validated SAMA's unified capabilities across multiple benchmarks:\n\n* Validation on SAMA-Bench: This benchmark was specifically designed to evaluate this integrated skill.\n  - SAMA-Bench-G (3,038 questions) explicitly requires the model to answer a referring question while simultaneously providing segmentation masks for the objects in the answer, testing both skills at once.\n  - SAMA-Bench-C (2,031 questions) tests fine-grained region understanding by requiring captions for referred objects.\n\n  Through rigorous testing on both benchmarks, SAMA demonstrates state-of-the-art performance in unified referring and grounding.\n* Standard Grounding Benchmarks: Extensive evaluations on established visual grounding benchmarks, such as MeViS and ReVOS, also confirm that SAMA possesses a strong and generalizable grounding ability.\n* Furthermore, our qualitative results provide visual evidence of SAMA's unified referring and grounding capabilities.\n  - In Figure 4, SAMA not only answers the main query but also accurately identifies and grounds other objects in the background, a task where the strong Ferret baseline fails.\n  - In Figure 5 (second example), SAMA correctly analyzes the evolving actions of the man throughout the video.\n  - In Figure 7 (second example), SAMA precisely segments multiple objects in a complex scene and provides a correct behavioral description of the bull.\n  - Similarly, Figure 8 showcases SAMA's ability to describe the changing behaviors of various objects, proving its understanding is rooted in video dynamics.\n\n**Q3: The SAMA model's flexibility is currently limited, as it only supports box-based object prompts and cannot handle other formats, such as points or masks.**\n\n**A3:** Please kindly see our response to Reviewer tQRK’s Q1 due to page limitations. We have already conducted new experiments showing that our SAMA framework can be easily extended to handle point and mask prompts, demonstrating its inherent flexibility. We will include these results in the final version.\n\n**Q4: The current model only operates on the objects in the videos, neglecting the action as a whole, which limits the potential for action understanding in videos.**\n\n**A4:** SAMA already handles both. SAMA is trained on a broad data spectrum (lines 249–254), combining ChatUniV’s general video‑dialogue dataset with our SAMA‑239K object‑centric dataset. This allows the model to capture both local object details and global action semantics. To our knowledge, SAMA is the first video multimodal large model that simultaneously supports general video understanding, video referential dialogue, and video grounding.\n\n**Q5: In Table 1, why is the Sa2Va performance so low on Bench-C?**\n\n**A5:** Actually, Sa2VA does not natively support video region-level captioning. During our Bench-C evaluation, we first marked the target object in the initial frame with a colored bounding box (e.g., red) and queried the Sa2VA with questions such as `\"Provide me with a description of the object within the red box.\"` However, we found that Sa2VA generally failed to associate the textual instruction with the visual cue, whereas Gemini responded correctly. Additionally, since Sa2VA is fine-tuned on large-scale segmentation data, it often responds with `\"Sure, it is [SEG]\"` instead of a descriptive text, resulting in low scores. \n\nBeing trained on our SAMA-239K dataset, SAMA successfully addressed this limitation in Sa2VA and demonstrated significant improvements.\n\n**Q6: Could you provide a more detailed analysis of the \"interference\" phenomenon between grounding and conversational skill? Is it a case of catastrophic forgetting of general conversational skills, or does the model learn to prioritize grounding-specific token sequences over more detailed dialogue?**\n\n**A6:** We have provided a comprehensive analysis of this phenomenon in our response to Reviewer tQRK's Q5, which we kindly refer you to for the full details due to page limitations.\n\nTo summarize, we agree with your assessment that the model learns to prioritize grounding-specific token sequences. As detailed in our response to Reviewer tQRK's Q5, this interference stems from discrepancies in output granularity and biases towards under-trained special tokens. Our new experiments show that this can be mitigated to some extent by re-balancing the optimization objective.\n\nFurthermore, we would like to add a crucial point of context: while there is a slight trade-off compared to dialogue-only models, SAMA's conversational ability remains highly competitive and significantly outperforms prior segmentation-capable models like OMG-LLaVA and PixelLM. Most previous reasoning segmentation models, such as LISA, VideoGLaMM, and VideoLISA, lack conversational abilities entirely.\n\nThis demonstrates that for a minor trade-off in conversational skill, SAMA gains two powerful, previously non-existent capabilities in the video domain: unified referential understanding and precise visual grounding. We believe this represents a valuable and worthwhile advancement for the field.\n\n**Q7: What is your hypothesis for why the dataset and model failed to provide a benefit on MeViS and Ref-DAVIS? Do these benchmarks have distinct data characteristics that are not well-represented in your training data?**\n\n**A7:** Thank you for this highly insightful question. The performance variations on MeViS and Ref-DAVIS likely stem from a subtle but crucial misalignment between the linguistic structure of these benchmarks and our grounded dialogue task, as well as a mismatch in feature granularity.\n* MeViS and Ref-DAVIS frequently use actional phrases as referring expressions (e.g., `\"a dog running in the garden\"`), where the grounding target is the entire action-object entity. However, such phrases are less common in natural conversation. When adapted to our interactive, dialogue-oriented format, the same scene would be structured as:\n\n  - `Q: What is <object> doing?`\n\n  - `A: <p>The dog</p>[SEG] is running in the garden.`\n\n  In this conversational format, the `[SEG]` token is associated with the nominal reference `\"The dog\"`. This creates a discrepancy: the model is trained to ground the noun, whereas the benchmark evaluates its ability to ground the entire actional phrase. This misalignment is likely a primary reason for the lack of performance gain. We believe this can be addressed by refining our data generation prompts to produce grounded phrases that are more aligned with the referring segmentation style (e.g., `A: <p>The dog running in the garden</p>[SEG] is white.`).\n* On the other hand, our STC aggregator is designed to capture coarse-grained, long-range temporal context. However, benchmarks like MeViS and Ref-DAVIS demand the capture of extremely fine-grained, subtle motion details to distinguish objects. Consequently, the coarse-grained features captured by the STC may not provide a direct performance boost for these highly specialized tasks. A promising solution is to implement an adaptive keyframe sampling strategy, which would allow the model to dynamically select more frames from action-intensive moments, thus enriching the fine-grained visual input."}, {"review_id": "nq3SmRj4cF", "reviewer": "Reviewer_Ewj9", "summary": "Summary:\nThe authors introduce a new dataset, method and benchmark for multi-turn referential grounded video chat for LMMs, i.e., the task of answering questions about explicit objects and their spatio-temporal changes/interactions, in a form of a dialog. For the dataset and the benchmark, the authors build upon existing data with object-level annotations, enriched with corresponsing dialogs, derived using Gemini 1.5 pro. To solve the task, the authors propose a LMM able to handle frame-level semantic details and combine it with long-form context, to solve the task. The model achieves SoTA-level performance on the benchmark.", "questions": "Questions:\n- What’s the duration of the videos in the dataset?\n- Could the authors be more specific about “Object-level Description Generation” in Section 3.1? It’s not clear if that part is done on images or videos, and what Gemini1.5 is applied to exactly.\n- It’s not very clear from the description how the explicit object handling is performed (ll179-183). Specifically, what does it mean to “performing mask pooling on the visual features F_{key} of the relevant frame using a bounding box prompt for the target object.”? Also, what do the authors mean by “inserting” object features into the language embeddings “at the position corresponding to the object reference”? Does it mean the language embedding is replaced by the visual one, or augmented in some way?\n- In line 184, what is F_{agg}? It wan’t defined previously.\n- How are the frames selected for long-range modelling and how some frames become the key frames?\n- Judging by Table 1, the difference between 1B, 4B and 8B models is quite small. Why would that be the case?\n- What are the resource requirements for training and inference of SAMA?\n\n\n\nSuggestions:\nPerhaps annotation diagram in Figure 2 with symbols (eg, F_{long}, E_{obj}, …) could make following the paper easier.", "limitations": "Discussed in the paper", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- It’s very important to establish a benchmark for grounded video understanding\n- The provided training set is highly valuable for the community\n- The design of the long-context aggregation effective and efficient: processing the individual frames with Q-Former, then sliding window with another Q-former to aggregate along the temporal dimension\n- Great results on video segmentation.\n\nWeaknesses:\n- At no point the authors mention using human manual involvement in the data annotation process (i.e., it’s automated by ML models). While this is acceptable for the training dataset, the benchmark samples should be annotated or at least filtered/reviewd manually.\n- The authors compare to Gemini-1.5 Pro that is a generalist model and show similar performance. While one could claim that Gemini does not produce segmentations directly, one could use the bounding boxes produced by the model to derive object representations, that could be then segmented. Thus the value of this method is not very clear.", "was_revised": true, "final_justification": "I thank the authors for answering my questions. This clarifies a lot. I raise my score to 5.", "rebuttal": "We sincerely appreciate Reviewer Ewj9 for their insightful feedback and for recognizing our contributions.\n\n**Q1: Human manual involvement in the data annotation process.**\n\n**A1:** Our SAMA-Bench undergoes a rigorous two-stage data quality assurance process: automated filtering and manual verification. In the automated filtering stage, we employ a suite of custom scripts to programmatically clean the data. This process ensures all special markers (e.g., `<p>, </p>, [SEG]`) are correctly formatted, removes invalid tags, and verifies that each `[SEG]` token corresponds to a valid segmentation mask. In the manual verification phase, we visualize the masks and manually check their alignment with the question-answer pairs, removing any problematic data. Through these thorough checks and processes, we ensure that SAMA-Bench evaluation yields stable and reliable results without errors.\n\n**Q2: The authors compare to Gemini-1.5 Pro that is a generalist model and show similar performance. While one could claim that Gemini does not produce segmentations directly, one could use the bounding boxes produced by the model to derive object representations, that could be then segmented. Thus the value of this method is not very clear.**\n\n**A2:** While such a pipeline is possible, our SAMA model provides a solution that is superior in three key aspects: end-to-end solution, performance, and accessibility.\n* End-to-end solution: The mentioned approach in the question is a complex, multi-step pipeline requiring heavy prompt engineering and heavy post-processing to get right object-associated information. In contrast, SAMA is a single, end-to-end model that seamlessly generates masks within dialogue, which is a more robust and elegant design.\n* Performance: We have already tested this pipeline. As shown in Table 1, the \"Ferret-13B + SAM2\" baseline, which follows this detect-then-segment logic, is outperformed by our SAMA model in grounding accuracy (0.64 vs. 0.70 mIoU, 0.43 vs. 0.55 Recall).\n* Accessibility: Gemini-1.5 Pro is a closed-source model with a costly API. In contrast, our work provides a fully open-source and reproducible set of resources (model, data, and code), empowering the community to advance this research without relying on a black box.\n\n**Q3: What’s the duration of the videos in the dataset?**\n\n**A3:** SAMA-239K consists of 15,143 videos totaling 76.82 hours. The average video length is 18.3 seconds, with the longest video lasting 179.8 seconds. All questions in SAMA-239K are annotated by the state-of-the-art video model, Gemini-1.5 Pro, to ensure quality. **In comparison, SAMA-239K contains 14 × more videos than the VISA [a] dataset, 6 × more questions than the VideoGLaMM [b] dataset, and 10.3 × the total duration of MOSE [c] dataset. The number of questions in SAMA-Bench is 11 × greater than that in ReasonVOS [d].** Importantly, SAMA uniquely supports both referential dialogue and precise grounding. \n\nWe genuinely believe the release of SAMA will significantly advance research in joint learning for video referring understanding and grounding within the community.\n\n**Q4: Could the authors be more specific about “Object-level Description Generation” in Section 3.1?**\n\n**A4:** We thank the reviewer for this question.\n\nAs detailed in Appendix A.1 (L523-536), our pipeline for generating these descriptions is as follows: For a given video, we first uniformly sample 16 frames. Within these frames, each target object is visually cued with a uniquely colored bounding box. These visually annotated frames are then provided to Gemini-1.5 Pro, which is prompted to generate a detailed, object-level description based on the cued region. \n\n*We will release the complete annotation code and execution scripts for every dataset, enabling researchers to reproduce our annotation pipeline effortlessly.*\n\n**Q5: It’s not very clear from the description how the explicit object handling is performed (ll179-183). Specifically, what does it mean to “performing mask pooling on the visual features F_{key} of the relevant frame using a bounding box prompt for the target object.”? Also, what do the authors mean by “inserting” object features into the language embeddings “at the position corresponding to the object reference”? Does it mean the language embedding is replaced by the visual one, or augmented in some way?**\n\n**A5:** Thank you for this precise question. We are happy to provide a step-by-step explanation for our explicit object handling mechanism.\n* On \"performing mask pooling... using a bounding box prompt\": This describes how we extract a visual feature vector for a specified object region.\n  - First, a bounding box prompt (e.g., `[x_min, y_min, x_max, y_max]`) is converted into a 2D binary mask of shape `H x W`. The pixel values inside the box are set to 1, and all values outside are 0.\n  - Let the visual features of a keyframe (`F_key`) have the shape `H x W x C`.\n  - \"Mask pooling\" then refers to masked average pooling: we multiply the visual features `F_key` with this binary mask and then compute the average of the features only in the masked (non-zero) region.\n  - This process yields a single feature vector `E_obj` of shape `1 x C`, which serves as the compact visual representation of the target object.\n* On \"inserting object features...\": This describes how we inject the object's visual representation `E_obj` into the language input embedding. It is a replacement, not an augmentation.\n  - Consider the input text: `\"What is the <object> doing?\"`.\n  - The text is first processed by a standard tokenizer, which converts the string into a sequence of token embeddings. The special placeholder `<object>` becomes a specific token embedding in this sequence.\n  - We then replace the embedding of this `<object>` placeholder token directly with the `E_obj` vector we extracted in the previous step.\n  - The final sequence of embeddings, now containing the visual information of the object, is fed into the Large Language Model.\n\nThis simple and direct replacement mechanism effectively grounds the textual reference `<object>` to its precise visual features. We will add this detailed explanation to the final version.\n\n**Q6: In line 184, what is F_{agg}? It wan’t defined previously.**\n\n**A6:** `F_agg` is already defined on line178 as the \"compact yet informative representation of long-range temporal dynamics\" derived from our STC aggregator. We will ensure this is crystal clear in the revision.\n\n**Q7: Frame selection for long-range and keyframes.**\n\n**A7:** We adopt a simple yet effective frame sampling strategy. We treat the first five frames of each clip as key frames​. We uniformly sample 32 frames over the full video for long‑range modeling​. Our experiments confirm that this simple method is effective enough to produce satisfactory results.\n\n**Q8: Small performance difference between 1B, 4B, and 8B models.**\n\n**A8:** We think the small performance gap between model sizes stems directly from our parameter-efficient training strategy. In our training, we freeze the core parameters of both the vision encoder and the LLM, and only train our proposed STC aggregator and the SAM2 mask decoder. This means the number of trainable parameters is small and consistent across all model variants.\n\nThis deliberate design choice has several key advantages:\n\n* Efficiency: It makes the training process highly memory-efficient, allowing even the 8B model to be trained on standard hardware without risk of OOM.\n* Effective Knowledge Injection: This strategy enables our lightweight STC module to effectively align its learned temporal features with the powerful, frozen representations of any LLM, acting as a \"plug-and-play\" component for grounding capabilities.\n* Preservation of LLM Abilities: By only updating a small fraction of parameters, we inject the new, fine-grained grounding capability while preserving the vast knowledge of the pretrained LLM.\n\nTherefore, since the performance gains are driven by the same set of newly trained parameters in all three models, it is reasonable that the final performance differences are small. We will explore more parameter-efficient training strategies to enhance the model's performance in future work.\n\n**Q9: Resource requirements and Figure 2 suggestion.**\n\n**A9:** Thank you for this question and valuable suggestion. \n\nSAMA's training and inference were performed on 8 x A100 GPUs. The 1B, 4B, and 8B models took approximately 48 hours, 55 hours, and 82 hours to train, respectively. \n\nWe also appreciate the suggestion for Figure 2 and will add the relevant symbols to the diagram in the revised version.\n\n > [a] VISA: Reasoning Video Object Segmentation via Large Language Models. ECCV 2024\n\n > [b] VideoGLaMM: A Large Multimodal Model for Pixel-Level Visual Grounding in Videos. CVPR 2025\n\n > [c] MOSE: A New Dataset for Video Object Segmentation in Complex Scenes. ICCV 2023\n\n > [d] One Token to Seg Them All: Language Instructed Reasoning Segmentation in Videos. NeurIPS 2024."}, {"review_id": "qdew3ZhrhX", "reviewer": "Reviewer_dpKE", "summary": "This paper proposes a multimodal video understanding model that combines a STC aggregator with region-level alignment and a newly collected instruction-following dataset annotated via Gemini. The model uses advanced visual modules (e.g., InternVL 2.5 and SAM2) and introduces a unified mechanism for referential video reasoning. Experimental results on the proposed benchmark and other benches show performance improvements over existing baselines.", "questions": "**1. Data Verification**: Were the Gemini-generated annotations subject to any human review or post-filtering? If not, how is annotation quality ensured?\n\n**2. Data Leakage Risk**: Since VIDSTG and some other datasets were re-annotated via SAM, how can you confirm that the validation samples (or more specifically, the objects involved in validation questions) did not appear in the training set?\n\n**3. Benchmark Generalization**: To what extent is the benchmark task distribution influenced by the collected dataset? Could the benchmark favor models trained on your specific data distribution?\n\n**4. Unify Module Impact**: Can you provide evidence (ablation or analysis) that the unify mechanism leads to improved reasoning or comprehension?", "limitations": "1. There is a risk of unintentional overlap or leakage between training and evaluation sets, especially when reusing public datasets with new annotations.\n\n2. (minor) Reliance on unverified Gemini annotations introduces uncertainty into the dataset quality. Model behavior may be driven by annotation artifacts rather than true understanding.\n\n3. (minor) The approach depends heavily on strong pretrained components (e.g., SAM2, InternVL-2.5), which may limit scalability or deployment in less resource-rich environments.\n\nWe hope the authors can carefully think about these issues to make their paper a technically sound and useful submission.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strengths:\n\n**1. Timely Task**: The focus on multimodal video understanding with open-ended, instruction-following capability is an important and emerging direction.\n\n**2. Dataset Scale**: The authors construct a large-scale, instruction-annotated dataset using Gemini and include re-annotated samples from existing datasets.\n\n**3. Strong Empirical Results**: The model achieves high performance on the proposed benchmark, outperforming prior baselines.\n\n\n## Weaknesses:\n\n**1. Limited Architectural Novelty**: \nThe model reuses familiar design elements such as RoI Align and hidden-state-as-query mechanisms, which are well-established in prior work. Although the proposed STC module bears resemblance to Q-Former, the paper does not offer sufficient discussion to clarify or justify its novelty. Moreover, the training objective and inference cost of STC are not described—Is it trained end-to-end? What are the training dynamics? Without this information, it's difficult to assess the model’s practicality or stability. As it stands, the architecture appears to be an incremental combination of known components.\n\n**2. Dataset/Benchmark Problem**: The proposed SAMA dataset and benchmark consolidate multiple sources with auto-generated annotations, but the resulting task setup resembles existing benchmarks for referential comprehension in static images. While the authors claim SAMA *supports joint learning of video referring understanding and grounding*, the improvements on grounding benchmarks (Table 2) are marginal. Additionally, results on general video understanding tasks (Table 3) are unconvincing: SAMA even degrades performance on video-chat, indicating that its contribution to broader video understanding remains unclear and needs further validation.\n\n**3. Underspecified Training Protocol**: Important training details are omitted, such as the training stages, and whether training was single- or multi-pass over the SFT data. This lack of clarity raises concerns about reproducibility and the risk of overfitting—especially given the Q-Former and SAM-decoder-style architecture, which is known to be prone to overfitting on limited data.\n\n**4. Qualitative Examples Issue**: (minor) Some qualitative examples (e.g., Figure 1a), intended to illustrate the model’s reasoning ability, rely mainly on static visual cues. These outputs could likely be produced by a strong visual encoder alone, rather than reflecting meaningful referential reasoning.", "was_revised": true, "final_justification": "The author's response has resolved my concerns about data curation, training protocol, and comparison with similar model designs. Although the novelty issue is not fully resolved, this shall be considered as a minor problem for the LMM community.", "rebuttal": "We sincerely appreciate Reviewer dpKE for their valuable feedback.\n\n**Q1: Limited Architectural Novelty.**\n\n**A1:** We thank the reviewer for this question.\n* First, to address your specific questions about the architecture: The STC module is trained end-to-end. Its output features are concatenated with other visual and text embeddings as input to the LLM. Therefore, it shares the same final optimization objectives: a token-level cross-entropy loss for next-token prediction and a pixel-level cross-entropy loss for segmentation mask generation. For training resources, please kindly refer to our response to Reviewer Ewj9 Q9.\n\n* Please allow us to clarify that our contribution is *not to propose a new, isolated technique for a pre-existing problem, but rather to leverage existing technologies to define and solve a new problem.*\n\n  Our primary objective was to address a fundamental, long-standing gap in video understanding: proving that video referring understanding and video grounding—two historically separate but deeply related tasks—could finally be unified to create a more powerful and versatile vision system.\n\n  Prior to our work, this unified task was uncharted territory: no large-scale dataset existed for joint training, no model possessed this integrated capability, and no benchmark was available for evaluation. Our work is the first comprehensive solution designed to build this field from the ground up:\n  * **We defined the task and created the foundational resources:** the SAMA-239K dataset and SAMA-Bench.\n  * **We presented the first strong baseline:** our SAMA model, which is purposefully tailored to prove this unified task is solvable.\n  * **We will fully open-source our work to accelerate community progress.**\n\n  *We genuinely argue that establishing a new research paradigm, creating the entire ecosystem of resources to support it, and presenting the first strong, reproducible baseline is a foundational and valuable contribution to the community.*\n\n**Q2: Dataset/Benchmark Problem.**\n\n**A2:** Please allow us to clarify the significance of our results and contributions.\n* **Task Setup:** While our task involves referential comprehension, it is fundamentally video-centric and demands spatio-temporal reasoning far beyond static image analysis. Our qualitative results in Figures 4-8 provide clear evidence. \n* **Grounding Improvements:** On challenging video segmentation benchmarks (Table 2), SAMA-8B establishes a new state-of-the-art, outperforming the Sa2VA with gains of +2.2 on MeViS, +1.4 on Ref-DAVIS17, and +1.5 on ReVOS. We wish to emphasize that a +2.2 gain on MeViS is significant, as this benchmark requires segmenting moving objects based on complex descriptions in dynamic environments, making it a highly challenging task. The difficulty of MeViS is further highlighted by the marginal 0.1-point gain of VISA-13B (ECCV'24) over the smaller VideoLISA-3.8B (NeurIPS'24).\n* **General Dialogue Performance:** The slight drop in chat performance compared to Sa2VA is a trade-off and a common challenge across segmentation-capable LMMs (e.g., OMG-LLaVA, LLaVA-Grounding, VideoLISA). Fine-tuning on segmentation data typically causes a decline in general dialogue capability (please see our response to Reviewer tQRK’s Q5 for further details regarding this common problem). However, it is crucial to note that while SAMA's dialogue performance is not the absolute best, it remains highly competitive and significantly surpasses that of many other segmentation-capable models, such as OMG-LLaVA and PixelLM.\n\n**Q3: Underspecified Training Protocol.**\n\n**A3:** We thank the reviewer for this question.\n* SAMA is developed through a single-stage instruction tuning process. We freeze the core parameters of the large vision encoder and the LLM. Only our proposed STC aggregator and the SAM2 mask decoder are made trainable. This allows our lightweight STC module to efficiently learn how to align spatio-temporal video features with the LLM's representation space. \n* The risk of overfitting is minimal. The number of trainable parameters is small, while our training data is a massive and diverse mixture (L249-254). This large data-to-parameter ratio ensures the model learns generalizable skills rather than memorizing the training set. \n\n**Q4: Qualitative Examples Issue.**\n\n**A4:** We thank the reviewer and agree that Figure 1(a) was a simplified example. Our full capabilities in handling complex, dynamic scenes beyond static cues are better demonstrated in Figures 4-8:\n\n* Figure 4: SAMA not only answers the main query but also accurately identifies and grounds other objects, a task where the strong Ferret baseline fails.\n* Figure 5 (second example): SAMA correctly analyzes the evolving actions of the man throughout the video.\n* Figure 7 (second example): SAMA precisely segments multiple objects in a complex scene and provides a correct behavioral description of the bull.\n* Figure 8 showcases SAMA's ability to describe the changing behaviors of various objects.\n\n**Q5: Data Verification.**\n\n**A5:** Our data underwent a rigorous verification process to ensure the quality, including automated filtering and manual verification. Please kindly refer to our detailed response to Reviewer Ewj9 Q1 due to page limitations. To summarize:\nDue to its large scale, SAMA-239K underwent a comprehensive automated post-filtering process.\nSAMA-Bench was subjected to both automated filtering and a manual human review to ensure the quality. \n\n**Q6: Data Leakage Risk.**\n\n**A6:** We can state with **absolute certainty that there is no data leakage**. We strictly adhered to the official `train/val` splits of all source datasets. Our SAMA-239K training data was built exclusively from the `train` splits, while our SAMA-Bench was built exclusively from the `val` splits, ensuring a complete and clean separation. Moreover, the validation set is carefully checked and cleaned by human annotators to ensure high-quality annotations and avoid data leakage.\n\n**Q7: Benchmark Generalization**\n\n**A7:** We can confirm the generalization of SAMA-Bench through three key facts:\n* SAMA-Bench is curated from the validation splits of four diverse public datasets (MeViS, Ref-YouTube-VOS, VidSTG, LV-VIS.), ensuring a rich variety of scenes, objects, and queries. This multi-source design makes it both the largest benchmark of its kind (e.g., 5× more questions than VideoRefer [CVPR'25], 16.5× more than VideoGLaMM [CVPR'25]) and the first to jointly evaluate referential understanding and grounding.\n* The benchmark does not unfairly favor our model. As shown in Table 1, strong baselines that were not trained on SAMA-239K (such as Ferret-13B+SAM2 and Gemini-1.5 Pro) still achieve competitive results.\n* Our model, trained on SAMA-239K, achieves state-of-the-art results on multiple external benchmarks for both image and video grounding (Table 2). This strong cross-benchmark performance would be impossible if the model had merely overfitted to our data distribution.\n\n**Q8: Unify Module Impact.**\n\n**A8:** Thanks for this question. The impact is indeed profound, but it manifests as *enabling a fundamentally new capability, rather than incrementally improving existing reasoning skills.*\n\nThe primary impact of our unify mechanism is that it transforms a standard Video LMM into a model that can seamlessly perform referential dialogue and precise grounding. Before being trained with our SAMA-239K, a strong baseline like Sa2VA, despite its high general video chat scores, completely lacks this capability. Our work takes the model's ability from zero to one on this new, crucial task. This leap in capability, we argue, is the most significant form of \"improved comprehension\" one can achieve.\n\nThis contribution is particularly vital for the video domain. While the image domain has mature unified models (e.g., Ferret, Shikra, CogVLM), this has been a long-standing gap for video, primarily due to the lack of a formal task definition and the necessary data. Our work provides the first complete solution (task definition, dataset, and benchmark) to fill this gap.\n\nFinally, while our main goal was to pioneer this new capability, we achieved it while maintaining a highly competitive level of general dialogue, far surpassing previous grounding models (e.g., PixelLM, OMG-LLaVA). We believe that perfectly balancing the injection of new skills with the preservation of all existing ones is a non-trivial challenge, and we have discussed this trade-off in our paper to inspire future work.\n\n**Q9: (minor) Reliance on unverified Gemini annotations introduces uncertainty into the dataset quality.**\n\n**A9:** \nWe would like to note that using LLMs for large-scale annotation is a standard practice in this field, employed by numerous leading works such as GLaMM, VideoGLaMM, Artemis, Osprey, VideoRefer, Ferret, LLaVA-Grounding and Sa2VA.\n\nConstructing SAMA-239K was a non-trivial effort and we implemented a rigorous pipeline to ensure dataset quality. Our process included: (a) multiple rounds of iterative prompt engineering to ensure rich, correctly formatted outputs; (b) using the state-of-the-art Gemini 1.5 Pro for all annotations; (c) curating data from diverse sources to avoid dataset-specific biases; and (d) a critical two-stage data verification process.\n\n**Q10: (minor) On the reliance on strong pretrained components.**\n\n**A10:** We thank the reviewer for this question. Our reliance on strong pretrained models is a deliberate choice, consistent with the standard paradigm in the LMM era. This allows us to focus our efforts on solving a new, unsolved research problem, rather than training a foundation model from scratch.\n\nCrucially, our primary contributions, the SAMA-229K dataset and the STC aggregator, are modular and backbone-agnostic. They provide a lasting ecosystem that empowers future research and can be readily adapted to more efficient backbones for deployment in resource-constrained environments."}]

Author rebuttal:

Meta-review: The post-rebuttal discussion resulted in a positive consensus among the reviewers. Three reviewers acknowledged that their concerns are fully resolved and consequently increased their ratings. The remaining reviewer acknowledged the paper's solid contribution on the dataset side, but still had reservations about the technical novelty of the proposed model. Despite this, the reviewer concluded that the paper is a borderline case but ultimately leans toward acceptance.

Based on these final ratings and discussions, the AC recommends acceptance of the paper.
Decision: Accept (poster)