Title: SAMA: Towards Multi-Turn Referential Grounded Video Chat with Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: SAMA

Video Large Multimodal Model

Referring

Grounding

Segmentation

Refer and Ground
Abstract: Achieving fine-grained spatio-temporal understanding in videos remains a major challenge for current Video Large Multimodal Models (Video LMMs). Addressing this challenge requires mastering two core capabilities: video referring understanding, which captures the semantics of video regions, and video grounding, which segments object regions based on natural language descriptions.
However, most existing approaches tackle these tasks in isolation, limiting progress toward unified, referentially grounded video interaction. We identify a key bottleneck in the lack of high-quality, unified video instruction data and a comprehensive benchmark for evaluating referentially grounded video chat.
To address these challenges, we contribute in three core aspects: dataset, model, and benchmark.
First, we introduce SAMA-239K, a large-scale dataset comprising 15K videos specifically curated to enable joint learning of video referring understanding, grounding, and multi-turn video chat.
Second, we propose the SAMA model, which incorporates a versatile spatio-temporal context aggregator and a Segment Anything Model to jointly enhance fine-grained video comprehension and precise grounding capabilities.
Finally, we establish SAMA-Bench, a meticulously designed benchmark consisting of 5,067 questions from 522 videos, to comprehensively evaluate the integrated capabilities of Video LMMs in multi-turn, spatio-temporal referring understanding and grounded dialogue.
Extensive experiments and benchmarking results show that SAMA not only achieves strong performance on SAMA-Bench but also sets a new state-of-the-art on general grounding benchmarks, while maintaining highly competitive performance on standard visual understanding benchmarks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a multimodal video understanding model that combines a STC aggregator with region-level alignment and a newly collected instruction-following dataset annotated via Gemini. The model uses advanced visual modules (e.g., InternVL 2.5 and SAM2) and introduces a unified mechanism for referential video reasoning. Experimental results on the proposed benchmark and other benches show performance improvements over existing baselines.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n**1. Timely Task**: The focus on multimodal video understanding with open-ended, instruction-following capability is an important and emerging direction.\n\n**2. Dataset Scale**: The authors construct a large-scale, instruction-annotated dataset using Gemini and include re-annotated samples from existing datasets.\n\n**3. Strong Empirical Results**: The model achieves high performance on the proposed benchmark, outperforming prior baselines.\n\n\n## Weaknesses:\n\n**1. Limited Architectural Novelty**: \nThe model reuses familiar design elements such as RoI Align and hidden-state-as-query mechanisms, which are well-established in prior work. Although the proposed STC module bears resemblance to Q-Former, the paper does not offer sufficient discussion to clarify or justify its novelty. Moreover, the training objective and inference cost of STC are not described—Is it trained end-to-end? What are the training dynamics? Without this information, it's difficult to assess the model’s practicality or stability. As it stands, the architecture appears to be an incremental combination of known components.\n\n**2. Dataset/Benchmark Problem**: The proposed SAMA dataset and benchmark consolidate multiple sources with auto-generated annotations, but the resulting task setup resembles existing benchmarks for referential comprehension in static images. While the authors claim SAMA *supports joint learning of video referring understanding and grounding*, the improvements on grounding benchmarks (Table 2) are marginal. Additionally, results on general video understanding tasks (Table 3) are unconvincing: SAMA even degrades performance on video-chat, indicating that its contribution to broader video understanding remains unclear and needs further validation.\n\n**3. Underspecified Training Protocol**: Important training details are omitted, such as the training stages, and whether training was single- or multi-pass over the SFT data. This lack of clarity raises concerns about reproducibility and the risk of overfitting—especially given the Q-Former and SAM-decoder-style architecture, which is known to be prone to overfitting on limited data.\n\n**4. Qualitative Examples Issue**: (minor) Some qualitative examples (e.g., Figure 1a), intended to illustrate the model’s reasoning ability, rely mainly on static visual cues. These outputs could likely be produced by a strong visual encoder alone, rather than reflecting meaningful referential reasoning.\n\nQUESTIONS:\n**1. Data Verification**: Were the Gemini-generated annotations subject to any human review or post-filtering? If not, how is annotation quality ensured?\n\n**2. Data Leakage Risk**: Since VIDSTG and some other datasets were re-annotated via SAM, how can you confirm that the validation samples (or more specifically, the objects involved in validation questions) did not appear in the training set?\n\n**3. Benchmark Generalization**: To what extent is the benchmark task distribution influenced by the collected dataset? Could the benchmark favor models trained on your specific data distribution?\n\n**4. Unify Module Impact**: Can you provide evidence (ablation or analysis) that the unify mechanism leads to improved reasoning or comprehension?\n\nLIMITATIONS:\n1. There is a risk of unintentional overlap or leakage between training and evaluation sets, especially when reusing public datasets with new annotations.\n\n2. (minor) Reliance on unverified Gemini annotations introduces uncertainty into the dataset quality. Model behavior may be driven by annotation artifacts rather than true understanding.\n\n3. (minor) The approach depends heavily on strong pretrained components (e.g., SAM2, InternVL-2.5), which may limit scalability or deployment in less resource-rich environments.\n\nWe hope the authors can carefully think about these issues to make their paper a technically sound and useful submission.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary:\nThe authors introduce a new dataset, method and benchmark for multi-turn referential grounded video chat for LMMs, i.e., the task of answering questions about explicit objects and their spatio-temporal changes/interactions, in a form of a dialog. For the dataset and the benchmark, the authors build upon existing data with object-level annotations, enriched with corresponsing dialogs, derived using Gemini 1.5 pro. To solve the task, the authors propose a LMM able to handle frame-level semantic details and combine it with long-form context, to solve the task. The model achieves SoTA-level performance on the benchmark.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- It’s very important to establish a benchmark for grounded video understanding\n- The provided training set is highly valuable for the community\n- The design of the long-context aggregation effective and efficient: processing the individual frames with Q-Former, then sliding window with another Q-former to aggregate along the temporal dimension\n- Great results on video segmentation.\n\nWeaknesses:\n- At no point the authors mention using human manual involvement in the data annotation process (i.e., it’s automated by ML models). While this is acceptable for the training dataset, the benchmark samples should be annotated or at least filtered/reviewd manually.\n- The authors compare to Gemini-1.5 Pro that is a generalist model and show similar performance. While one could claim that Gemini does not produce segmentations directly, one could use the bounding boxes produced by the model to derive object representations, that could be then segmented. Thus the value of this method is not very clear.\n\nQUESTIONS:\nQuestions:\n- What’s the duration of the videos in the dataset?\n- Could the authors be more specific about “Object-level Description Generation” in Section 3.1? It’s not clear if that part is done on images or videos, and what Gemini1.5 is applied to exactly.\n- It’s not very clear from the description how the explicit object handling is performed (ll179-183). Specifically, what does it mean to “performing mask pooling on the visual features F_{key} of the relevant frame using a bounding box prompt for the target object.”? Also, what do the authors mean by “inserting” object features into the language embeddings “at the position corresponding to the object reference”? Does it mean the language embedding is replaced by the visual one, or augmented in some way?\n- In line 184, what is F_{agg}? It wan’t defined previously.\n- How are the frames selected for long-range modelling and how some frames become the key frames?\n- Judging by Table 1, the difference between 1B, 4B and 8B models is quite small. Why would that be the case?\n- What are the resource requirements for training and inference of SAMA?\n\n\n\nSuggestions:\nPerhaps annotation diagram in Figure 2 with symbols (eg, F_{long}, E_{obj}, …) could make following the paper easier.\n\nLIMITATIONS:\nDiscussed in the paper", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge that current Video LMMs struggle with fine-grained spatio-temporal understanding, as they typically handle video referring (understanding regions) and grounding (segmenting objects) as separate tasks.\n\nTo address this, the authors present: a new dataset, SAMA-239K, for joint training, a model named SAMA, and a benchmark, SAMA-Bench.\n\nThe SAMA model architecture integrates a spatio-temporal context aggregator with SAM2 to enhance video comprehension and precise object grounding jointly.\n\nThe SAMA-239K dataset, created using an automated pipeline with Gemini-1.5 Pro, contains 239K instances of referential grounded dialogue and object descriptions to enable unified learning.\n\nExperiments demonstrate that the SAMA model not only achieves descent performance on the new SAMA-Bench but also performs well on existing image and video referring segmentation benchmarks\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n* The paper is well written and easy to follow. The figure visualization makes it easy to comprehend the method and the purposes of the dataset.\n* The paper presents a pipeline to address the problem of referentially grounded video chat, comprising a new dataset for training and testing, as well as a proposed model. The release of a new dataset may benefit the research community.\n* Extensive experiments and a thorough ablation study support the paper's claim. The study confirms that both the spatio-temporal aggregator and the SAMA-239K dataset are essential for the model's high performance.\n\nWeaknesses\n\n1. Method.  \n(a) Given that this is more of a dataset paper, the proposed method is not novel; the way of disintangling spatio and temporal representation mechanisms was explored previously and is widely used in designing video models.  \n(b) The authors acknowledge that jointly training for grounding and dialogue can cause \"interference\" between the tasks, indicating a need for more robust multi-task learning strategies. This implies that the goal of solving referring and grounding simultaneously has not yet been achieved.  \n(b) The SAMA model's flexibility is currently limited, as it only supports box-based object prompts and cannot handle other formats, such as points or masks.  \n(c) The current model only operates on the objects in the videos, neglecting the action as a whole, which limits the potential for action understanding in videos.\n\n\n2. Experiment.  \n(a) In Table 1, why is the Sa2Va performance so low? The METEROR and CIDEr score reaches 0 in benchC. Any particular reason for that?  \n(b) In Table 2, the ablation study reveals that the architectural and data contributions did not consistently improve performance across all tasks; specifically, on the MeViS and Ref-DAVIS benchmarks, their impact was not uniformly positive.  \n(c) In Table 3, the model's specialization in grounding tasks comes at a cost to its general conversational abilities. SAMA's performance shows a slight drop in some general chat metrics compared to Sa2VA. Showing a clear trade-off across different aspects.\n\nQUESTIONS:\n1. Please answer the questions and concerns in the weakness.\n\n2. Could you provide a more detailed analysis of the \"interference\" phenomenon between grounding and conversational skill?  Is it a case of catastrophic forgetting of general conversational skills, or does the model learn to prioritize grounding-specific token sequences over more detailed dialogue? \n\n3. What is your hypothesis for why the dataset and model failed to provide a benefit on MeViS and Ref-DAVIS? Do these benchmarks have distinct data characteristics that are not well-represented in your training data?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SAMA, a framework for fine-grained spatial-temporal understanding and grounded dialogue in videos.\n\n- Dataset: SAMA-239K, a large-scale dataset with 15K videos, 172K referential grounded video-QA pairs, and 67K object-level descriptions. It includes video referring understanding, grounding, and multi-turn video chat. ​\n- Model: The SAMA model integrates a spatial-temporal context aggregator and SAM2 for precise grounding. ​\n- Benchmark: SAMA-Bench is composed of 522 videos with 5,067 questions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper introduces SAMA, a unified framework that integrates fine-grained referential understanding and grounded dialogue capabilities in videos, addressing a critical gap in video-based multimodal systems. ​\n- The proposed SAMA-239K dataset is a significant contribution, offering a large-scale, high-quality resource for joint learning of video referring understanding and grounding. ​\n- SAMA-Bench provides a well-designed evaluation framework for assessing integrated referential understanding and grounded dialogue capabilities, filling a key evaluation gap in the field. ​\n- The introduction of the spatial-temporal-context aggregator and integration with SAM2 enhances SAMA's ability to capture both fine-grained spatial details and long-range temporal dynamics, making it robust for complex video understanding tasks. ​\n\nWeaknesses:\n- SAMA currently supports only box-based prompts, which restricts its flexibility and applicability in diverse scenarios. ​ Expanding to other input formats like points or masks would improve usability. ​\n- While strong in visual grounding, SAMA's performance on general dialogue tasks lags behind specialized conversational models, indicating a need for better reasoning integration. ​\n- SAMA-239K is limited to short video clips, which constrains its applicability to long-range temporal understanding. ​ Scaling to longer videos would enhance its utility for complex scenarios. ​\n\nQUESTIONS:\n- Can SAMA be extended to handle other input formats, such as points or masks, beyond box-based prompts? ​ \n- What specific steps can be taken to improve SAMA's performance on general dialogue tasks? Could integrating specialized reasoning modules or leveraging additional conversational datasets help? ​\n- What are the main challenges in scaling SAMA to longer videos? Are there computational or architectural limitations that need to be addressed? ​\n- How does interference between grounding and dialogue tasks manifest during training? ​ Are there specific techniques or architectures that could mitigate this issue? ​\n- Are there any safeguards or ethical considerations for the release of SAMA, particularly given its potential for misuse in surveillance or disinformation? ​\n\nLIMITATIONS:\n- SAMA currently handles only box-based prompts, limiting its flexibility. ​ Expanding to other input formats like points or masks could improve usability. ​\n- SAMA-239K is limited to short video clips, and scaling to longer videos would better support research on complex, long-range temporal understanding. ​", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-16"}