MomentSeg: Moment-Centric Sampling for Enhanced Video Pixel Understanding

## Abstract

Abstract Referring Video Object Segmentation (RefVOS) seeks to segment target objects in videos guided by natural language descriptions, demanding both temporal reasoning and fine-grained visual comprehension. Existing sampling strategies for LLM-based approaches typically rely on either handcrafted heuristics or external keyframe models. The former often overlooks essential temporal cues, while the latter increases system complexity. To address this, we propose a unified framework that jointly optimizes Temporal Sentence Grounding (TSG) and RefVOS, naturally incorporating key moment grounding capability. During training, we introduce a novel TSG paradigm that employs a dedicated [FIND] token for key moment identification through temporal token similarity matching, thereby avoiding the need for external timestamp encodings. For inference, we design a Moment-Centric Sampling (MCS) strategy that densely samples informative moments while sparsely sampling non-essential frames, preserving both motion details and global context. To further enhance tracking stability, we develop Bidirectional Anchor-updated Propagation (BAP), which leverages the most relevant moment as start point for high-quality mask initialization and dynamically updates at sampled points to mitigate accumulated errors.
Code and model will be available at: https://github.com/Dmmm1997/MomentSeg

## 1 Introduction

RefVOS has attracted considerable attention in recent years. The task involves localizing and segmenting target objects at the pixel level in videos based on natural language expressions (Gavrilyuk et al., 2018; Khoreva et al., 2019; Seo et al., 2020). It requires models to interpret temporal action semantics described in text and to maintain consistent tracking of the referred object across frames. Recent Transformer-based methods (Botach et al., 2022; Luo et al., 2023; He & Ding, 2024; Pan et al., 2025) mainly focus on improving temporal reasoning and inter-frame consistency. In contrast, large multimodal models (LMMs) (Bai et al., 2025; Chen et al., 2024e; Li et al., 2024a), pre-trained on massive video-text corpora, demonstrate stronger potential for advancing pixel-level video understanding tasks (Munasinghe et al., 2024; Lin et al., 2025; Yuan et al., 2025).

A central challenge in RefVOS lies in effectively sampling keyframes that are relevant to the linguistic expression. Given the variable and often large number of frames, feeding all frames into the model is both computationally prohibitive and redundant. Consequently, identifying and selecting semantically meaningful keyframes is worth exploring.
Existing approaches employ diverse strategies for keyframe selection, as summarized in Table [1](https://arxiv.org/html/2510.09274v1#S2.T1). For instance, Sa2VA (Yuan et al., 2025) simply selects the first K frames, disregarding content-specific cues such as object presence or action timing. VideoLISA (Bai et al., 2024) compresses temporal information and samples frames uniformly to retain coarse timing information but still overlooks referentially critical frames. In contrast, methods such as VISA (Yan et al., 2024), ViLLA (Zheng et al., 2024), VRS-HQ (Gong et al., 2025), and GLUS (Lin et al., 2025) depend on separately trained models to pinpoint the most relevant timestamps, then either use tracking networks (Cheng & Schwing, 2022) to generate mask trajectories or select key sampling frames to guide large-model segmentation.
While these methods enable the identification of keyframes, they are dependent on these additional models.

Building on prior explorations, we first pose a fundamental question: *Is keyframe selection necessary?*
From an intuitive perspective, failing to capture critical moments in a video risks missing the target object’s appearance or the described action, which directly hinders accurate segmentation. Conversely, precisely identifying and leveraging text-relevant key moments mitigates the misleading effects of irrelevant frames on LMMs while reducing computational redundancy.
From a quantitative perspective, as shown in Fig. [2](https://arxiv.org/html/2510.09274v1#S1.F2), we evaluate several sampling strategies across multiple RefVOS datasets. Two consistent patterns emerge: (1) random sampling exhibits high variance, highlighting the strong influence of sampling strategy; and (2) for scenarios involving motion-driven (e.g., MeVIS (Ding et al., 2023)) and complex semantic understanding (e.g., ReVOS (Yan et al., 2024)), uniform sampling often surpasses the firstK approach. In contrast, firstK performs better on datasets (Khoreva et al., 2019; Seo et al., 2020) where targets appear early in the video and descriptions rely less on temporal cues. Likewise, MeVIS, which includes numerous action-centric descriptions and scenes with multiple visually similar objects, poses a particular challenge: if sampled frames miss the described action while static cues match several candidates, the model struggles to disambiguate the target. This analysis underscores that keyframe selection is critical for RefVOS.

Figure: Figure 2: Impact of Sampling Strategies. We evaluate various sampling strategies for the RefVOS task on four datasets. Five independent runs of random sampling show high variance, highlighting the need for a robust sampling mechanism. Our proposed MCS consistently outperforms alternative methods across all datasets.
Refer to caption: figures/sample_std.png

Next, we consider the following question: *How can we effectively identify keyframes, and how can these keyframes enhance RefVOS performance?* Prior work offers two primary strategies: (1) Keyframe for tracking initialization. VISA (Yan et al., 2024) employs LLAMA-VID (Li et al., 2025b) to select the most critical frame as the tracking initialization timestamp, and applies XMem (Cheng & Schwing, 2022) for subsequent tracking. (2) Keyframes for LMM understanding. ViLLA (Zheng et al., 2024) leverages Grounded-VideoLLM (Wang et al., 2024a) to generate relevant temporal intervals and then samples frames inside and outside these intervals using handcrafted ratios, subsequently feeding these frames to LMM. VRS-HQ (Gong et al., 2025) further combines CLIP-based (Radford et al., 2021) vision-language similarity with SAM2 confidence scores to filter keyframes before passing them to the LMM.
While effective, all these methods depend on external models for keyframe identification, inevitably increasing overall system complexity.

In this paper, we propose a unified framework for the synchronized optimization of TSG and RefVOS. During training, as shown in Fig. [3](https://arxiv.org/html/2510.09274v1#S2.F3), we introduce a special [FIND] token for the TSG task, similar to the [SEG] token in RefVOS. This token computes similarity with sampled video frame tokens, offering two key advantages: First, for RefVOS, it eliminates the need for external models for keyframe selection, as the model natively identifies key moments. Second, it obviates the need for explicit timestamp encoding (Chen et al., 2024c) by leveraging a frame-level matching mechanism between [FIND] and temporal tokens.
During inference, as depicted in Fig. [4](https://arxiv.org/html/2510.09274v1#S3.F4), we propose a Moment-Centric Sampling (MCS). MCS leverages the similarity distribution from the [FIND] token to densely sample video moments highly relevant to the description, while sparsely sampling less critical frames. This approach effectively preserves crucial motion cues while maintaining a global temporal context. For the RefVOS task, consistent with Sa2VA (Yuan et al., 2025), a single [SEG] token is used to represent the referential object throughout the video. We further introduce a novel Bidirectional Anchor-updated Propagation (BAP) strategy, which is composed of two key components:
(1) Bidirectional Propagation: The central keyframes identified by MCS serve as starting points for propagation in both forward and backward temporal directions.
(2) Anchor-updated: At these sampled time points, the target mask is updated and the prior memory is adaptively refreshed, which mitigates cumulative tracking errors.

In summary, our key contributions are as follows:
(1) We propose MomentSeg, a unified framework that integrates temporal sentence grounding (TSG) and referring video object segmentation (RefVOS). For TSG, MomentSeg obviates explicit timestamp encoding by leveraging the similarity between the [FIND] token and temporal tokens. For RefVOS, our model inherently identifies text-relevant keyframes, thereby removing the dependency on external keyframe-selection models.
(2) We introduce Moment-Centric Sampling (MCS), a similarity-driven sampling strategy that densely samples frames around relevant moments and sparsely samples non-essential frames. MCS preserves salient motion cues while retaining global temporal context.
(3) We present Bidirectional Anchor-updated Propagation (BAP), which initiates mask propagation from query-relevant keyframes and adaptively updates the target mask at sampled temporal points. BAP enhances tracking robustness and mitigates cumulative propagation errors.
(4) MomentSeg achieves new SOTA performance, delivering a 5% improvement on the MeVIS and a 6% gain on the ReVOS.

## 2 Related Work

Referring Video Object Segmentation.
RefVOS aims to segment targets in a video conditioned on a given description. Existing approaches typically employ object queries to fuse expressions with visual features for referent identification (Wu et al., 2022; 2023; Luo et al., 2023; Tang et al., 2023; Yuan et al., 2024), while a line of work emphasizes motion aggregation to capture dynamic cues (Ding et al., 2023; He & Ding, 2024; Fang et al., 2025b). Concurrently, LMMs (Liu et al., 2024b) have been used to reason over compositional and complex expressions (Zhu et al., 2023; Yan et al., 2024; Bai et al., 2024; Munasinghe et al., 2024; Lin et al., 2025; Deng et al., 2025). To address these gaps, we propose MomentSeg, which performs keyframe selection natively, removing the need for external modules. We further introduce Bi-directional Anchor-updated Propagation (BAP), which propagates masks forward and backward from key moments and adaptively updates memory at sampled nodes, enhancing segmentation robustness and reducing cumulative errors.

**Table 1: Comparison of *keyframe models*, *sampling*, and *information* utilized in existing RefVOS MLLMs. “cont.” means continuous sampling. Unlike prior methods, our approach inherently supports keyframe selection while simultaneously leveraging both “Global” and “Essential” information during inference.**
| Method | External Keyframe Model | Training | Inference | [SEG] |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Sampling | Information | Sampling | Information |  |  |  |
| VideoLISA (Bai et al., 2024) | / | Uniform | Global | Uniform | Global | 1 |
| VISA (Yan et al., 2024) | LLaMA-VID (Li et al., 2025b) | Random | Random | Uniform + Cont. | Global + Local | 1 |
| ViLLa (Zheng et al., 2024) | G-VideoLLM (Wang et al., 2024a) | Uniform | Global | Cont. | Local | $N$ |
| GLUS (Lin et al., 2025) | Chat-UniVi (Jin et al., 2024) | Uniform + Cont. | Global + Local | Uniform + Cont. | Global + Local | $N$ |
| Sa2VA (Yuan et al., 2025) | / | Random | Random | FirstK | Local | 1 |
| VRS-HQ (Gong et al., 2025) | CLIP (Radford et al., 2021) | Uniform | Global | Selection | Essential | $N$ |
| MomentSeg (Ours) | Inherently | Uniform + Cont. | Global + Local | Selection | Global + Essential | 1 |

LLM-based RefVOS Sampling Strategies. Existing LLM-based RefVOS methods adopt diverse frame-sampling and keyframe-selection schemes (Table [1](https://arxiv.org/html/2510.09274v1#S2.T1)). VideoLISA (Bai et al., 2024) and Sa2VA (Yuan et al., 2025) adopt handcrafted sampling strategies, using uniform or firstK sampling to generate a global [SEG] token. In contrast, VISA (Yan et al., 2024), ViLLA (Zheng et al., 2024), and GLUS (Lin et al., 2025) rely on external models (e.g., LLaMA-VID (Li et al., 2025b), Grounded-VideoLLM (Wang et al., 2024a), Chat-UniVi (Jin et al., 2024)) for keyframe selection, while VRS-HQ (Gong et al., 2025) selects keyframes via CLIP-based vision–language similarity (Radford et al., 2021). Such dependencies increase pipeline complexity and limit adaptability. By comparison, MomentSeg performs keyframe selection natively and introduces Moment-Centric Sampling (MCS), which applies dense sampling around text-relevant moments and sparse sampling elsewhere, preserving salient motion cues while maintaining a compact global temporal context.

Timestamps Encoding in Temporal Sentence Grounding. TSG (Gao et al., 2017; Caba Heilbron et al., 2015; Zala et al., 2023; Wang et al., 2024a; Li et al., 2025a; Liu et al., 2025) aims to localize a video segment corresponding to a language query. A key challenge for LMM-based methods lies in developing an effective timestamp encoding strategy. Approaches such as TimeChat (Ren et al., 2024a), VTimeLLM (Huang et al., 2024a), and LITA (Huang et al., 2024b) leverage instruction tuning for plain text temporal grounding. Momentor (Qian et al., 2024a) injects explicit temporal position encodings into frame-level features to improve temporal localization, while VTG-LLM (Guo et al., 2024) incorporates absolute-time tokens. Vid2Seq (Yang et al., 2023) and Grounded-VideoLLM (Wang et al., 2024a) adopt discrete or relative temporal tokens to circumvent direct timestamp encoding. NumberIT (Wu et al., 2025) injects sequential frame indices into images for grounding. In contrast, MomentSeg avoids timestamp encoding by employing a specialized [FIND] token that performs frame-level temporal matching.

Figure: Figure 3: Training framework of the proposed MomentSeg model. In the TSG paradigm, we employ the Qwen2.5-VL (Bai et al., 2025) video encoder with low-resolution image inputs. We further introduce a [FIND] token trained under a contrastive learning scheme. In the RefVOS paradigm, both low-resolution uniformly sampled frames and high-resolution continuously sampled frames are used as inputs. Supervision is provided via the [SEG] token, with segmentation masks generated by the SAM2 decoder.
Refer to caption: x2.png

## 3 Methods

### 3.1 Preliminaries

Referring Video Object Segmentation:
Given an input video with $T$ frames $I_{1:T}\in\mathbb{R}^{T\times H\times W\times 3}$ and a referring language expression $R$, RefVOS aims to train a model $\phi_{\theta}$ that predicts binary segmentation masks ${M}_{1:T}$ for the referred object:

$$ $M_{1:T}=\phi_{\theta}(I_{1:T},R).$ (1) $$

LMMs-based RefVOS Paradigm:
Recent LMM-based RefVOS approaches build upon image-level referring segmentation models such as LISA (Lai et al., 2024), which employ a dedicated [SEG] token to represent the target and a segmentation decoder $\mathtt{Dec}$ to generate masks:

$$ $\texttt{[SEG]}=\mathtt{LMM}(I,R),\ \ \ M=\mathtt{Dec}(I,\texttt{[SEG]}).$ (2) $$

This paradigm extends to videos by adopting a multi-image formulation, where the LMM processes multiple frames $I_{1:N}$ and segmentation tokens decode masks for each frame:

$$ $\texttt{[SEG]}=\mathtt{LMM}(I_{1:N},R),\ \ \ M_{1:N}=\mathtt{Dec}(I_{1:N},\texttt{[SEG]}).$ (3) $$

This equation exemplifies a “One Token Seg All” paradigm (Bai et al., 2024), in which a single [SEG] token represents the referential object throughout the video. Here, $N$ denotes the number of frames provided to the LMM, which is typically smaller than the full sequence length $T$.

### 3.2 Model Architecture of MomentSeg

#### 3.2.1 Overall Pipeline

Training Pipeline:
Fig. [3](https://arxiv.org/html/2510.09274v1#S2.F3) presents the training paradigm of MomentSeg for both TSG and RefVOS. For RefVOS, we follow a scheme similar to Sa2VA (Yuan et al., 2025), while additionally incorporating low-resolution, uniformly sampled frames $I^{l}_{1:L}$ and high-resolution, densely sampled frames $I^{h}_{1:N}$ from temporal segments where the referential object is present. The sampling of these low-resolution frames is referred to as *Temporal Token Injecting* in this paper. Notably, the number of tokens introduced into the LMM remains minimal, at no more than 16 tokens per frame in our setting. The low-resolution frames $I^{l}_{1:L}$ supply global temporal context, whereas the high-resolution frames $I^{h}_{1:N}$ capture fine-grained spatial details. A SAM2 decoder is then applied to the high-resolution frames and is supervised as follows:

$$ $\texttt{[SEG]}^{h}=\mathtt{LMM}(I^{l}_{1:L},I^{h}_{1:N},R),\quad M_{1:N}=\mathtt{Dec}(I^{h}_{1:N},\texttt{[SEG]}^{h}).$ (4) $$

For TSG, we introduce a special token [FIND] to identify relevant frames through similarity matching with temporal features. This design removes the need for external timestamp encoding, as temporal correspondence is established directly via frame-level matching. Specifically, we first extract the temporal tokens corresponding to the LMM output. These tokens are projected into a feature space using an MLP layer and subsequently average-pooled to obtain frame representations $T^{v}\in\mathbb{R}^{L_{t}\times C}$. The [FIND] tokens, $T^{f}\in\mathbb{R}^{N_{f}\times C}$ (where $N_{f}$ denotes the number of [FIND] tokens in a sample under the multi-round dialogue paradigm), are mapped through the same MLP layer. We then compute the similarity matrix $\ell\in\mathbb{R}^{N_{f}\times L_{t}}$ between frame tokens and [FIND] tokens to identify text-relevant frames. The matching loss is defined as:

$$ $\displaystyle\ell_{ij}=\frac{(T^{f}_{i})^{\top}T^{v}_{j}}{\|T^{f}_{i}\|\,\|T^{v}_{j}\|\,\tau},\mathcal{L}_{find}=\frac{1}{|\Omega|}\sum_{(i,j)\in\Omega}\Big[-\lambda_{p}y_{ij}\log\sigma(\ell_{ij})-(1-y_{ij})\log(1-\sigma(\ell_{ij}))\Big],$ (5) $$

where $\sigma$ denotes the sigmoid function, $T^{v}_{i}\in\mathbb{R}^{C}$ is the $i$-th pooled frame token, $T^{f}_{j}\in\mathbb{R}^{C}$ is the $j$-th [FIND] token, $y_{ij}\in\{0,1\}$ is the match label, $\Omega$ is the set of valid pairs, and $\tau$ is the temperature factor, set to 0.07. The positive sample weight $\lambda_{p}$ is set to 2.0.
Notably, since Qwen2.5-VL compresses the temporal dimension during video encoding (with $L_{t}$ being half of $L$), we apply bilinear interpolation to the similarity distribution to ensure alignment with the original input length.

Figure: Figure 4: Inference framework of the proposed MomentSeg model. First, we use dense-sampled video frames to find the frame-query similarity distribution related to the description, then apply a Moment-Centric Sampling (MCS) to select key frames from the sequence. These key frames are input to the model, along with the RefVOS instructions, to perform the RefVOS task. Finally, we enhance segmentation robustness through Bidirectional Anchor-updated Propagation (BAP).
Refer to caption: x3.png

Inference Pipeline:
As shown in Fig. [4](https://arxiv.org/html/2510.09274v1#S3.F4), inference begins with densely sampled low-resolution frames $I^{l}_{1:L}$ to compute the frame-query similarity distribution relative to the description. Based on this distribution, the Moment-Centric Sampling (MCS) strategy selects key frames $I^{h}_{1:N}$. These key frames, together with RefVOS instructions, are then input to the model for segmentation. Finally, Bidirectional Anchor-updated Propagation (BAP) is applied to enhance segmentation robustness.

#### 3.2.2 Moment-Centric Sampling (MCS)

Figure: Algorithm 1 Moment-Centric Sampling (MCS)

MCS selectively samples frames most relevant to the text query while preserving global temporal context. It operates by analyzing the similarity distribution between video frames and the query, identifying high-relevance regions, and performing dense sampling around these critical moments while maintaining sparse coverage elsewhere.
Specifically, MCS first computes the similarity between each frame token $I^{l}_{1:L}$ and the [FIND] token $T^{f}$. To suppress noise, Gaussian smoothing is applied, producing a refined similarity distribution $\mathcal{S}\in\mathbb{R}^{T}$. The moment center is then identified via a sliding window that maximizes cumulative similarity within a window of size $w$:

$$ $i^{*}=\operatorname*{arg\,max}_{i=0\rightarrow{T-w}}\sum_{j=i}^{i+w-1}\mathcal{S}_{j},\quad c^{*}=i^{*}+\lfloor\frac{w}{2}\rfloor,$ (6) $$

where $i^{*}$ denotes the starting index of the optimal window and $c^{*}$ the corresponding moment center. This formulation ensures that the most semantically relevant temporal segment is captured within the sequence.
Given the identified moment center $c^{*}$ and the total number of required samples $K$, moment-centric sampling is performed as outlined in Algorithm [1](https://arxiv.org/html/2510.09274v1#alg1). The algorithm partitions the sequence into left and right regions relative to the center, allocates samples proportionally according to cumulative weights, and applies inverse CDF sampling to preserve the similarity distribution.
The InverseCDF procedure takes as input a sub-distribution $\mathcal{S}_{a:b}$ and a target sample size $k$, normalizes the weights to form probabilities $p_{i}=\mathcal{S}_{i}/{\sum_{j=a}^{b}\mathcal{S}_{j}}$, and computes the $F(i)=\sum_{j=a}^{i}p_{j}$. Then, for each uniform random variable $u_{m}\sim\mathcal{U}(0,1)$, the sampled index is given by:

$$ $i_{m}=\min\{\,i\in[a,b]\mid F(i)\geq u_{m}\,\},\qquad m=1,\ldots,k.$ (7) $$

This ensures that moments with higher similarity receive denser sampling while still maintaining representation across the entire temporal span. The key advantage of MCS lies in its ability to enhance the capacity to capture both fine-grained actions and their broader temporal dependencies.

#### 3.2.3 Bidirectional Anchor-updated Propagation (BAP)

To improve segmentation robustness, we propose BAP, which leverages key moments as anchors for bidirectional propagation. The primary goal is to provide SAM2 with a strong initialization while ensuring stable tracking through mask updates during propagation. Its two key features are: (1) SAM2 propagation starts from critical temporal anchors and proceeds bidirectionally (Initialization in Fig. [4](https://arxiv.org/html/2510.09274v1#S3.F4)); and (2) mask updates occur at sampled key moments with selective memory clearing (Update in Fig. [4](https://arxiv.org/html/2510.09274v1#S3.F4)). This design enables dynamic error correction, enhancing long-term tracking.
Specifically, the most relevant moment $c^{*}$ identified by MCS is selected as the starting point for bidirectional propagation. At each sampled key moment $\mathcal{I}$, a memory cleaning mechanism adaptively clears the cache based on the mask prediction score $S^{p}$ and the tracking score $S^{t}$, as determined by:

$$ $U_{i}=\begin{cases}1,&\text{if }\prod_{j=1}^{t}S^{t}_{j}<\lambda\cdot S^{p}_{i}\\ 0,&\text{otherwise}\end{cases}$ (8) $$

where $U_{i}$ is the binary indicator (1 for clearing memory, 0 for retention), $S^{t}_{j}$ is the tracking score at step $j$, $\prod_{j=1}^{t}S^{t}_{j}$ denotes the cumulative tracking confidence up to time $t$, $S^{p}_{i}$ is the prediction confidence at node $i$, and $\lambda$ is a sensitivity hyperparameter (set to 0.9). An update is triggered when accumulated confidence falls below the threshold determined by the current prediction quality, ensuring that memory is refreshed only when beneficial for tracking accuracy.

## 4 Experiments

### 4.1 Implementation Details

We construct our baseline by integrating Qwen2.5-VL (Bai et al., 2025) with SAM2 (Ravi et al., 2024). Following prior work (Yuan et al., 2025), segmentation masks are generated by decoding the hidden state of the [SEG] token using the SAM2 decoder. The LLM is fine-tuned with LoRA (Hu et al., 2021), with a maximum sequence length of 8,192 tokens. Training is conducted on 8 NVIDIA H20 GPUs (96GB memory each) and takes approximately 24 hours for the MomentSeg-3B model. For evaluation, we adopt lmms-eval (Zhang et al., 2024a) for both image and video chat tasks.

### 4.2 Main Results

Temporal Sentence Grounding Task.
We evaluate our proposed method, MomentSeg, on the Charades-STA and ActivityNet datasets. As shown in Table [2](https://arxiv.org/html/2510.09274v1#S4.T2), MomentSeg is compared with a range of advanced methods. The MomentSeg-3B model, despite its smaller scale, consistently outperforms most existing Video-LLMs. This strong performance validates the effectiveness of our proposed paradigm, which is underpinned by the novel mechanism where the [FIND] token interacts with temporal tokens via cosine similarity to achieve frame-level matching.

**Table 2: Comparison of MomentSeg with methods on temporal sentence grounding benchmarks.**
| Method | Charades-STA | ActivityNet-Grounding |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R@0.3 | R@0.5 | R@0.7 | mIoU | R@0.3 | R@0.5 | R@0.7 | mIoU |  |
| Video-LLaMA-7B (Zhang et al., 2023) | 25.2 | 10.6 | 3.4 | 16.8 | 21.9 | 10.8 | 4.9 | 16.5 |
| SeViLA-3B (Yu et al., 2023) | 27.0 | 15.0 | 5.8 | 18.3 | 31.6 | 19.0 | 10.1 | 23.0 |
| Video-ChatGPT-7B (Maaz et al., 2023) | 27.2 | 6.2 | 1.9 | 19.7 | 19.5 | 10.6 | 4.8 | 14.2 |
| VideoChat2-7B (Li et al., 2024b) | 38.0 | 14.3 | 3.8 | 24.6 | 40.8 | 27.8 | 9.3 | 27.9 |
| Momenter-7B (Qian et al., 2024b) | 42.6 | 26.6 | 11.6 | 28.5 | 42.9 | 23.0 | 12.4 | 29.3 |
| VTimeLLM-7B (Huang et al., 2024a) | 51.0 | 27.5 | 11.4 | 31.2 | 44.0 | 27.8 | 14.3 | 30.4 |
| TimeChat-7B (Ren et al., 2024a) | - | 32.2 | 13.4 | - | - | - | - | - |
| VTG-LLM-7B (Guo et al., 2025) | - | 33.8 | 15.7 | - | - | - | - | - |
| HawkEye-7B (Wang et al., 2024b) | 50.6 | 31.4 | 14.5 | 33.7 | 49.1 | 29.3 | 10.7 | 32.7 |
| Grounded-VideoLLM-4B (Wang et al., 2024a) | 54.2 | 36.4 | 19.7 | 36.8 | 46.2 | 30.3 | 19.0 | 36.1 |
| NumPro-LongVA-7B (Wu et al., 2025) | 63.8 | 42.0 | 20.6 | 41.4 | 55.6 | 37.5 | 20.6 | 38.8 |
| Qwen2.5-VL-7B (Bai et al., 2025) | - | - | - | 43.6 | - | - | - | - |
| MomentSeg-3B (Ours) | 76.1 | 58.2 | 25.8 | 50.2 | 67.5 | 44.7 | 23.2 | 45.4 |

**Table 3: Comparison of MomentSeg with methods on referring video object segmentation benchmarks.**
| Method | MeViS ($val^{u}$) | MeViS ($val$) | Ref-Youtube-VOS | Ref-DAVIS17 |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ |  |
| LISA-7B (Lai et al., 2024) | 43.2 | 39.9 | 46.5 | 37.2 | 35.1 | 39.4 | 53.9 | 53.4 | 54.3 | 64.8 | 62.2 | 67.3 |
| TrackGPT-7B (Zhu et al., 2023) | - | - | - | 40.1 | 37.6 | 42.6 | 56.4 | 55.3 | 57.4 | 63.2 | 59.4 | 67.0 |
| VideoLISA-3.8B (Bai et al., 2024) | 54.5 | 50.9 | 58.1 | 44.4 | 41.3 | 47.6 | 63.7 | 61.7 | 65.7 | 68.8 | 64.9 | 72.7 |
| VISA-7B (Yan et al., 2024) | - | - | - | 43.5 | 40.7 | 46.3 | 61.5 | 59.8 | 63.2 | 69.4 | 66.3 | 72.5 |
| ViLLa-6B (Zheng et al., 2024) | - | - | - | 49.4 | 46.5 | 52.3 | 67.5 | 64.6 | 70.4 | 74.3 | 70.6 | 78.0 |
| GLUS-7B (Lin et al., 2025) | 60.9 | - | - | 51.3 | 48.5 | 54.2 | 67.3 | 65.5 | 69.0 | - | - | - |
| InstructSeg-3B (Wei et al., 2025b) | - | - | - | - | - | - | 67.5 | 65.4 | 69.5 | 71.1 | 67.3 | 74.9 |
| Sa2VA-4B (Yuan et al., 2025) | - | - | - | 46.2 | - | - | 70.0 | - | - | 70.0 | - | - |
| Sa2VA-8B (Yuan et al., 2025) | - | - | - | 46.9 | - | - | 70.7 | - | - | 75.2 | - | - |
| Sa2VA-Qwen2.5-VL-3B | 57.5 | 53.6 | 61.3 | 50.0 | 46.9 | 53.1 | 71.0 | 68.8 | 73.1 | 74.4 | 70.1 | 78.7 |
| MomentSeg-3B (Ours) | 62.0 | 58.1 | 65.9 | 54.8 | 51.7 | 58.0 | 72.0 | 69.8 | 74.3 | 76.4 | 72.2 | 80.6 |
| MomentSeg-7B (Ours) | 62.6 | 58.7 | 66.5 | 57.1 | 53.9 | 60.2 | 72.3 | 70.1 | 74.5 | 77.4 | 73.2 | 81.7 |

**Table 4: Comparison of MomentSeg with methods on reasoning video object segmentation benchmarks.**
| Method | ReVOS | ReasonVOS |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Referring | Reasoning | Overall | $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ |  |  |  |  |  |  |  |
| $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}$&$\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ |  |  |  |  |
| TrackGPT-13B (Zhu et al., 2023) | 49.5 | 48.3 | 50.6 | 40.5 | 38.1 | 42.9 | 45.0 | 43.2 | 46.8 | - | - | - |
| VISA-13B (Yan et al., 2024) | 57.4 | 55.6 | 59.1 | 44.3 | 42.0 | 46.7 | 50.9 | 48.8 | 52.9 | - | - | - |
| VideoLISA-3.8B (Bai et al., 2024) | - | - | - | - | - | - | - | - | - | 47.5 | 45.1 | 49.9 |
| HyperSeg-3B (Wei et al., 2025a) | 58.5 | 56.0 | 60.9 | 53.0 | 50.2 | 55.8 | 55.7 | 53.1 | 58.4 | - | - | - |
| ViLLa-6B (Zheng et al., 2024) | - | - | - | - | - | - | 57.0 | 54.9 | 59.1 | 55.4 | - | - |
| GLUS-7B (Lin et al., 2025) | 58.3 | 56.0 | 60.7 | 51.4 | 48.8 | 53.9 | 54.9 | 52.4 | 57.3 | 49.9 | 47.5 | 52.4 |
| Sa2VA-4B (Yuan et al., 2025) | - | - | - | - | - | - | 53.2 | - | - | - | - | - |
| VRS-HQ-7B Gong et al. (2025) | 62.1 | 59.8 | 64.5 | 56.1 | 53.5 | 58.7 | 59.1 | 56.6 | 61.6 | - | - | - |
| Sa2VA-Qwen2.5-VL-3B | 61.5 | 59.0 | 64.1 | 55.9 | 53.0 | 58.8 | 58.7 | 56.0 | 61.4 | 49.6 | 46.9 | 52.4 |
| MomentSeg-3B (Ours) | 65.4 | 63.0 | 67.9 | 59.9 | 57.1 | 62.6 | 62.6 | 60.0 | 65.2 | 61.7 | 58.2 | 65.3 |
| MomentSeg-7B (Ours) | 67.4 | 64.9 | 69.8 | 62.8 | 59.8 | 65.7 | 65.1 | 62.3 | 67.8 | 62.7 | 59.2 | 66.1 |

Referring/Reasoning Video Segmentation Task.
For referring tasks (Table [3](https://arxiv.org/html/2510.09274v1#S4.T3)), MomentSeg-3B achieves $\mathcal{J}$&$\mathcal{F}$ scores of 62.0 on MeViS ($val^{u}$) and 54.8 on MeViS ($val$), surpassing GLUS-7B by 1.1 and 3.5 points, respectively. On Ref-Youtube-VOS and Ref-DAVIS17, it attains scores of 72.0 and 76.4, improving over Sa2VA-4B by 2.0 and 6.4 points. Scaling to 7B further enhances performance across benchmarks.
For reasoning tasks (Table [4](https://arxiv.org/html/2510.09274v1#S4.T4)), MomentSeg-3B obtains 62.6 on ReVOS, exceeding VRS-HQ-7B by 3.5 points. On ReasonVOS, it achieves 61.7, outperforming ViLLa-6B by 6.3 points, with consistent improvements at larger scales.

Referring/Reasoning Image Segmentation Task.
As shown in Table [5](https://arxiv.org/html/2510.09274v1#S4.T5), MomentSeg-3B achieves 82.1, 76.9, and 78.8 on the RefCOCO/+/g validation sets, outperforming Sa2VA-4B by 3.2, 5.2, and 4.7 points, respectively. For reasoning tasks, MomentSeg-7B attains 65.5 on the ReasonSeg test set, surpassing the previous method, LISA-7B, by 2.6 points. Furthermore, on the GCG benchmark, MomentSeg-7B achieves 67.8/67.9 mIoU (val/test), exceeding GLaMM-7B by 2.0/3.3 points.

**Table 5: Comparison of model results across image segmentation tasks. Both RefCOCO/+/g and ReasonSeg use the cIoU metric, while GCG employs the mIoU metric.**
| Method | RefCOCO | RefCOCO+ | RefCOCOg | ReasonSeg | GCG |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| val | testA | testB | val | testA | testB | val(U) | test(U) | val | test | val | test |  |
| PixelLM-7B (Ren et al., 2024b) | 73.0 | 76.5 | 68.2 | 66.3 | 71.7 | 58.3 | 69.3 | 70.5 | - | - | - | - |
| LISA-7B (Lai et al., 2024) | 74.9 | 79.1 | 72.3 | 65.1 | 70.8 | 58.1 | 67.9 | 70.6 | 61.3 | 62.9 | 62.0 | 61.7 |
| GLaMM-7B (Rasheed et al., 2024) | 79.5 | 83.2 | 76.9 | 72.6 | 78.7 | 64.6 | 74.2 | 74.9 |  |  | 65.8 | 64.6 |
| VisionLLMv2-7B (Wu et al., 2024) | 79.2 | 82.3 | 77.0 | 68.9 | 75.8 | 61.8 | 73.3 | 74.8 | 56.9 | 48.3 | 64.6 | 62.8 |
| OMG-LLaVA-7B (Zhang et al., 2024b) | 75.6 | 77.7 | 71.2 | 65.6 | 69.7 | 58.9 | 70.7 | 70.2 | - | - | 65.5 | 64.7 |
| LaSagnA-7B (Wei et al., 2024) | 76.8 | 78.7 | 73.8 | 66.4 | 70.6 | 60.1 | 70.6 | 71.9 | 48.8 | 47.2 | - | - |
| VISA-7B (Yan et al., 2024) | 72.4 | 75.5 | 68.1 | 59.8 | 64.8 | 53.1 | 65.5 | 66.4 | 52.7 | 57.8 | - | - |
| VRS-HQ-7B (Gong et al., 2025) | 73.5 | 77.5 | 69.5 | 61.7 | 67.6 | 54.3 | 66.7 | 67.5 | 51.8 | 52.9 | - | - |
| Sa2VA-4B (Yuan et al., 2025) | 78.9 | - | - | 71.7 | - | - | 74.1 | - | - | - | - | - |
| MomentSeg-3B (Ours) | 82.1 | 83.7 | 79.2 | 76.9 | 81.1 | 71.8 | 78.8 | 79.2 | 62.0 | 64.3 | 67.0 | 65.9 |
| MomentSeg-7B (Ours) | 82.6 | 85.1 | 80.2 | 78.2 | 81.9 | 72.3 | 80.1 | 80.1 | 63.3 | 65.5 | 67.8 | 67.9 |

### 4.3 Ablations Studies

Key Components Design.
To assess the effectiveness of individual components, we conduct ablation studies on six RefVOS datasets. As shown in Table [6](https://arxiv.org/html/2510.09274v1#S4.T6), starting from the baseline, introducing Temporal Token Injecting (TTI) consistently improves performance, with smaller gains on Ref-Youtube-VOS and Ref-DaVIS17 due to shorter videos and static expression. Incorporating MCS provides further improvements, particularly on motion-intensive datasets such as MeViS, where key frame sampling enhances action understanding. Finally, adding BAP strengthens tracking and error correction, with the largest benefits observed on reasoning-oriented benchmarks involving long video sequences, underscoring the importance of robust temporal tracking mechanisms.

Effectiveness of Moment-Centric Sampling.
Different sampling strategies affect performance in dataset-specific ways. MeViS requires action understanding in long videos where targets appear only in certain segments, Ref-DAVIS17 involves targets that are present throughout most of the video, and ReasonVOS emphasizes relational reasoning with targets often emerging mid-video. As shown in Table [8](https://arxiv.org/html/2510.09274v1#S4.T8), FirstK underperforms relative to Uniform on MeViS and ReasonVOS because early frames frequently miss target segments. Keyframe–based strategies alleviate this by prioritizing high-similarity moments: NearbyK samples adjacent frames, while TopK selects globally most similar frames. Among them, NearbyK proves more robust, underscoring the benefit of dense sampling around critical regions.
Building on this insight, our proposed Moment-Centric Sampling integrates dense sampling near key moments with sparse sampling elsewhere, thereby capturing both critical and global context and achieving the best overall performance across benchmarks.

Effectiveness of Bidirectional Anchor-updated Propagation.
Mask propagation often struggles with poor initialization when the target appears late in the video. BAP overcomes this by initializing at the most probable target moment, propagating bidirectionally, and applying adaptive memory updates to limit error accumulation.
Table [8](https://arxiv.org/html/2510.09274v1#S4.T8) shows ablations of its components. Moment-anchored Updating notably improves performance, particularly on long video scenarios, by correcting accumulated tracking errors, while Adaptive Memory Cleaning provides smaller but consistent improvements. Overall, Bidirectional Propagation strategy improves robustness by initializing from the most probable target moment without adding inference cost.

Effectiveness of Temporal Tokens for TSG and RefVOS.
We use temporal tokens in training for both tasks, with RefVOS additionally sampling frames for segmentation. We examine: (1) whether TTI in RefVOS establishes temporal action cues; (2) whether tokens can be omitted at inference to reduce overhead; and (3) whether jointly optimizing temporal tokens for RefVOS and TSG leads to mutual benefits.
For (1) and (2), Table [10](https://arxiv.org/html/2510.09274v1#S4.T10) shows consistent gains across datasets, including +2 points on ReasonVOS. Notably, inference without tokens achieves similar performance, so we omit them to avoid extra cost. More detailed analyses are in Appendix [E.1](https://arxiv.org/html/2510.09274v1#A5.SS1).
For (3), Table [10](https://arxiv.org/html/2510.09274v1#S4.T10) demonstrates that joint training improves both tasks, yielding a 1.2-point gain on TSG.
Jointly optimizing temporal tokens provides mutual benefits, and incorporating the video-chat data further boosts performance.

**Table 6: Effectiveness of the key components. We report the $\mathcal{J}$&$\mathcal{F}$ metric for all six datasets.**
| ID | TTI | MCS | BAP | MeViS ($val^{u}$) | MeViS ($val$) | Ref-DAVIS17 | Ref-Youtube-VOS | ReasonVOS | ReVOS (Overall) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1 |  |  |  | 57.0 | 51.3 | 75.5 | 70.1 | 54.1 | 58.3 |
| 2 | ✓ |  |  | 58.7( $\uparrow$1.7) | 52.3( $\uparrow$1.0) | 76.2( $\uparrow$0.7) | 70.3( $\uparrow$0.2) | 58.1( $\uparrow$4.0) | 59.2( $\uparrow$0.9) |
| 3 | ✓ | ✓ |  | 61.3( $\uparrow$2.6) | 53.9( $\uparrow$1.6) | 76.8( $\uparrow$0.6) | 70.8( $\uparrow$0.5) | 60.8( $\uparrow$2.7) | 60.4( $\uparrow$1.2) |
| 4 | ✓ | ✓ | ✓ | 62.0( $\uparrow$0.7) | 54.1( $\uparrow$0.2) | 76.4( $\downarrow$0.4) | 72.0( $\uparrow$1.2) | 61.7( $\uparrow$0.9) | 62.6( $\uparrow$2.2) |

**Table 7: Impact of moment-centric sampling.**
| Method | MeViS ($val^{u}$) | Ref-DAVIS17 | ReasonVOS |
| --- | --- | --- | --- |
| FirstK (Baseline) | 58.3 | 76.3 | 55.9 |
| Uniform | 59.6( $\uparrow$1.3) | 76.2( $\downarrow$0.1) | 57.1 ( $\uparrow$1.2) |
| KeyFrame (TopK) | 60.1 | 76.9 | 60.2 |
| KeyFrame (NearbyK) | 60.9 ( $\uparrow$0.8) | 77.1( $\uparrow$0.2) | 59.9( $\downarrow$0.3) |
| MCS | 61.3( $\uparrow$3.0) | 77.3( $\uparrow$1.0) | 60.8( $\uparrow$4.9) |

**Table 9: Effectiveness of temporal token injecting.**
| Method | MeViS ($val^{u}$) | Ref-DAVIS17 | ReasonVOS |
| --- | --- | --- | --- |
| *Training Process* |  |  |  |
| w/o Temporal Tokens | 59.1 | 73.3 | 53.1 |
| w. Temporal Tokens | 60.1 ( $\uparrow$1.0) | 74.2( $\uparrow$0.9) | 55.1( $\uparrow$2.0) |
| *Inference Process* |  |  |  |
| w/o Temporal Tokens | 60.1 | 74.2 | 55.1 |
| w. Temporal Tokens | 60.3 ( $\uparrow$0.2) | 73.2( $\downarrow$1.0) | 56.4( $\uparrow$1.3) |

## 5 Qualitative Results

As illustrated in Fig. [5](https://arxiv.org/html/2510.09274v1#S5.F5), we show qualitative results of MomentSeg on TSG and RefVOS tasks.
For the referring task (Fig. [5](https://arxiv.org/html/2510.09274v1#S5.F5)(a)), the model accurately localizes the described action and, through dense moment sampling with MCS, delivers precise segmentation.
For the reasoning task (Fig. [5](https://arxiv.org/html/2510.09274v1#S5.F5)(b)), MomentSeg identifies the temporal segment aligned with complex relational descriptions, highlighting its strong temporal reasoning.
Additional visualizations are provided in Appendix [F](https://arxiv.org/html/2510.09274v1#A6).

Figure: Figure 5: Qualitative results of MomentSeg on TSG and RefVOS tasks. The figure displays the input expression, frame-query similarity distribution, sampled frames, and the predicted segmentation masks for TSG and RefVOS. (a) illustrates an example for the referring task, while (b) presents an example for the reasoning task.
Refer to caption: figures/visualization.jpg

## 6 Conclusion

In this paper, we introduce MomentSeg, a unified framework that optimizes both TSG and RefVOS tasks simultaneously. Our approach eliminates the reliance on external models for keyframe selection, addressing the inherent complexity of previous methods. By leveraging a [FIND] token during training, MomentSeg identifies text-relevant key moments directly, bypassing the need for explicit timestamp encodings and enhancing temporal understanding. For inference, we propose Moment-Centric Sampling (MCS), which intelligently selects relevant frames while preserving essential motion cues and global temporal context. Additionally, our Bidirectional Anchor-updated Propagation (BAP) further improves tracking robustness by mitigating cumulative errors and handling occlusions effectively. Experiments on several RefVOS and TSG benchmarks demonstrate that MomentSeg outperforms existing methods, achieving SOTA results.

## 7 Ethics Statement

This work focuses on algorithmic analysis and enhancements to improve the capabilities of LMMs for the referring video object segmentation (RefVOS) task.
All datasets employed are publicly available and have undergone appropriate ethical review, ensuring lawful and responsible use.

## 8 Reproducibility Statement

To support reproducibility, we provide detailed implementation information and materials.
Section [4.1](https://arxiv.org/html/2510.09274v1#S4.SS1) describes the construction of the baseline models, while Appendix [B](https://arxiv.org/html/2510.09274v1#A2) presents training configurations, datasets, and evaluation metrics.
Appendix [C.2](https://arxiv.org/html/2510.09274v1#A3.SS2) further details the templates and procedures used to generate training data instructions.
The complete codebase and trained models will be released upon publication to facilitate community verification and replication.

## References

- Bai et al. (2025)
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai
Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun
Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng
Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang,
Haiyang Xu, and Junyang Lin.
Qwen2.5-vl technical report.
*arXiv preprint arXiv:2502.13923*, 2025.
- Bai et al. (2024)
Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Lei Liu,
Zheng Zhang, and Mike Zheng Shou.
One token to seg them all: Language instructed reasoning segmentation
in videos.
*NeurIPS*, 2024.
- Botach et al. (2022)
Adam Botach, Evgenii Zheltonozhskii, and Chaim Baskin.
End-to-end referring video object segmentation with multimodal
transformers.
In *CVPR*, 2022.
- Caba Heilbron et al. (2015)
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles.
Activitynet: A large-scale video benchmark for human activity
understanding.
In *CVPR*, pp. 961–970, 2015.
- Chen et al. (2024a)
Guo Chen, Yicheng Liu, Yifei Huang, Yuping He, Baoqi Pei, Jilan Xu, Yali Wang,
Tong Lu, and Limin Wang.
Cg-bench: Clue-grounded question answering benchmark for long video
understanding.
*arXiv preprint arXiv:2412.12075*, 2024a.
- Chen et al. (2024b)
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong
Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al.
Mmstar: Are we on the right way for evaluating large vision-language
models?
*arXiv preprint arXiv:2403.20330*, 2024b.
- Chen et al. (2024c)
Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma.
Timemarker: A versatile video-llm for long and short video
understanding with superior temporal localization ability.
*arXiv preprint arXiv:2411.18211*, 2024c.
- Chen et al. (2024d)
Wei Chen, Long Chen, and Yu Wu.
An efficient and effective transformer decoder-based framework for
multi-task visual grounding.
In *ECCV*, pp. 125–141. Springer, 2024d.
- Chen et al. (2024e)
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong,
Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al.
Internvl: Scaling up vision foundation models and aligning for
generic visual-linguistic tasks.
In *CVPR*, pp. 24185–24198, 2024e.
- Cheng & Schwing (2022)
Ho Kei Cheng and Alexander G Schwing.
Xmem: Long-term video object segmentation with an atkinson-shiffrin
memory model.
In *ECCV*, 2022.
- Cuttano et al. (2025)
Claudia Cuttano, Gabriele Trivigno, Gabriele Rosi, Carlo Masone, and Giuseppe
Averta.
Samwise: Infusing wisdom in sam2 for text-driven video segmentation.
In *CVPR*, pp. 3395–3405, 2025.
- Dai et al. (2024)
Ming Dai, Lingfeng Yang, Yihao Xu, Zhenhua Feng, and Wankou Yang.
Simvg: A simple framework for visual grounding with decoupled
multi-modal fusion.
*NeurIPS*, 37:121670–121698, 2024.
- Dai et al. (2025a)
Ming Dai, Wenxuan Cheng, Jiang-Jiang Liu, Lingfeng Yang, Zhenhua Feng, Wankou
Yang, and Jingdong Wang.
Improving generalized visual grounding with instance-aware joint
learning.
*TPAMI*, 2025a.
- Dai et al. (2025b)
Ming Dai, Wenxuan Cheng, Jiang-jiang Liu, Sen Yang, Wenxiao Cai, Yanpeng Sun,
and Wankou Yang.
Deris: Decoupling perception and cognition for enhanced referring
image segmentation through loopback synergy.
*ICCV*, 2025b.
- Dai et al. (2025c)
Ming Dai, Wenxuan Cheng, Jiedong Zhuang, Jiang-jiang Liu, Hongshen Zhao,
Zhenhua Feng, and Wankou Yang.
Propvg: End-to-end proposal-driven visual grounding with
multi-granularity discrimination.
*ICCV*, 2025c.
- Dai et al. (2025d)
Ming Dai, Jian Li, Jiedong Zhuang, Xian Zhang, and Wankou Yang.
Multi-task visual grounding with coarse-to-fine consistency
constraints.
In *AAAI*, volume 39, pp. 2618–2626, 2025d.
- Deng et al. (2025)
Andong Deng, Tongjia Chen, Shoubin Yu, Taojiannan Yang, Lincoln Spencer, Yapeng
Tian, Ajmal Saeed Mian, Mohit Bansal, and Chen Chen.
Motion-grounded video reasoning: Understanding and perceiving motion
at pixel level.
In *CVPR*, pp. 8625–8636, 2025.
- Ding et al. (2023)
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy.
Mevis: A large-scale benchmark for video segmentation with motion
expressions.
In *ICCV*, 2023.
- Fang et al. (2025a)
Hao Fang, Runmin Cong, Xiankai Lu, Xiaofei Zhou, Sam Kwong, and Wei Zhang.
Decoupled motion expression video segmentation.
In *CVPR*, pp. 13821–13831, 2025a.
- Fang et al. (2025b)
Hao Fang, Runmin Cong, Xiankai Lu, Xiaofei Zhou, Sam Kwong, and Wei Zhang.
Decoupled motion expression video segmentation.
In *CVPR*, pp. 13821–13831, 2025b.
- Fu et al. (2023)
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin,
Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji.
Mme: A comprehensive evaluation benchmark for multimodal large
language models.
*arXiv preprint arXiv:2306.13394*, 2023.
- Fu et al. (2025)
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan
Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al.
Video-mme: The first-ever comprehensive evaluation benchmark of
multi-modal llms in video analysis.
In *CVPR*, pp. 24108–24118, 2025.
- Gao et al. (2017)
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia.
Tall: Temporal activity localization via language query.
In *CVPR*, pp. 5267–5275, 2017.
- Gavrilyuk et al. (2018)
Kirill Gavrilyuk, Amir Ghodrati, Zhenyang Li, and Cees GM Snoek.
Actor and action video segmentation from a sentence.
In *CVPR*, pp. 5958–5966, 2018.
- Gong et al. (2025)
Sitong Gong, Yunzhi Zhuge, Lu Zhang, Zongxin Yang, Pingping Zhang, and Huchuan
Lu.
The devil is in temporal token: High quality video reasoning
segmentation.
In *CVPR*, pp. 29183–29192, 2025.
- Guo et al. (2024)
Yongxin Guo, Jingyu Liu, Mingda Li, Xiaoying Tang, Xi Chen, and Bo Zhao.
Vtg-llm: Integrating timestamp knowledge into video llms for enhanced
video temporal grounding.
*arXiv preprint arXiv:2405.13382*, 2024.
- Guo et al. (2025)
Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xiaoying Tang, Dianbo Sui,
Qingbin Liu, Xi Chen, and Kevin Zhao.
Vtg-llm: Integrating timestamp knowledge into video llms for enhanced
video temporal grounding.
In *Proceedings of the AAAI Conference on Artificial
Intelligence*, volume 39, pp. 3302–3310, 2025.
- He & Ding (2024)
Shuting He and Henghui Ding.
Decoupling static and hierarchical motion perception for referring
video segmentation.
In *CVPR*, 2024.
- Hu et al. (2021)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean
Wang, Lu Wang, and Weizhu Chen.
Lora: Low-rank adaptation of large language models.
*ICLR*, 2021.
- Huang et al. (2024a)
Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu.
Vtimellm: Empower llm to grasp video moments.
In *CVPR*, pp. 14271–14280, 2024a.
- Huang et al. (2024b)
De-An Huang, Shijia Liao, Subhashree Radhakrishnan, Hongxu Yin, Pavlo
Molchanov, Zhiding Yu, and Jan Kautz.
Lita: Language instructed temporal-localization assistant.
*arXiv preprint arXiv:2403.19046*, 2024b.
- Iashin & Rahtu (2020)
Vladimir Iashin and Esa Rahtu.
Multi-modal dense video captioning.
In *CVPRW*, pp. 958–959, 2020.
- Jin et al. (2024)
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan.
Chat-univi: Unified visual representation empowers large language
models with image and video understanding.
In *CVPR*, 2024.
- Kazemzadeh et al. (2014)
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg.
Referitgame: Referring to objects in photographs of natural scenes.
In *EMNLP*, 2014.
- Kembhavi et al. (2016)
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi,
and Ali Farhadi.
A diagram is worth a dozen images.
In *European Conference on Computer Vision*, pp. 235–251.
Springer, 2016.
- Khoreva et al. (2019)
Anna Khoreva, Anna Rohrbach, and Bernt Schiele.
Video object segmentation with language referring expressions.
In *ACCV*. Springer, 2019.
- Lai et al. (2024)
Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya
Jia.
Lisa: Reasoning segmentation via large language model.
In *CVPR*, 2024.
- Lei et al. (2021)
Jie Lei, Tamara L Berg, and Mohit Bansal.
Detecting moments and highlights in videos via natural language
queries.
*NeurIPS*, 34:11846–11858, 2021.
- Li et al. (2023)
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan.
Seed-bench: Benchmarking multimodal llms with generative
comprehension.
*arXiv preprint arXiv:2307.16125*, 2023.
- Li et al. (2025a)
Hongyu Li, Jinyu Chen, Ziyu Wei, Shaofei Huang, Tianrui Hui, Jialin Gao,
Xiaoming Wei, and Si Liu.
Llava-st: A multimodal large language model for fine-grained
spatial-temporal understanding.
In *CVPR*, pp. 8592–8603, 2025a.
- Li et al. (2024a)
Jian Li, Weiheng Lu, Hao Fei, Meng Luo, Ming Dai, Min Xia, Yizhang Jin, Zhenye
Gan, Ding Qi, Chaoyou Fu, et al.
A survey on benchmarks of multimodal large language models.
*arXiv preprint arXiv:2408.08632*, 2024a.
- Li et al. (2024b)
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan
Xu, Guo Chen, Ping Luo, et al.
Mvbench: A comprehensive multi-modal video understanding benchmark.
In *CVPR*, pp. 22195–22206, 2024b.
- Li et al. (2025b)
Yanwei Li, Chengyao Wang, and Jiaya Jia.
Llama-vid: An image is worth 2 tokens in large language models.
In *ECCV*, 2025b.
- Liang et al. (2025)
Tianming Liang, Kun-Yu Lin, Chaolei Tan, Jianguo Zhang, Wei-Shi Zheng, and
Jian-Fang Hu.
Referdino: Referring video object segmentation with visual grounding
foundations.
*ICCV*, 2025.
- Lin et al. (2025)
Lang Lin, Xueyang Yu, Ziqi Pang, and Yu-Xiong Wang.
Glus: Global-local reasoning unified into a single large language
model for video segmentation.
In *CVPR*, pp. 8658–8667, 2025.
- Liu et al. (2023)
Chang Liu, Henghui Ding, and Xudong Jiang.
Gres: Generalized referring expression segmentation.
In *CVPR*, pp. 23592–23601, 2023.
- Liu et al. (2024a)
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan
Zou, Jianwei Yang, Hang Su, Jun Zhu, et al.
Llava-plus: Learning to use tools for creating multimodal agents.
In *ECCV*, 2024a.
- Liu et al. (2024b)
Shilong Liu, Hao Cheng, Haotian Liu, Hao Zhang, Feng Li, Tianhe Ren, Xueyan
Zou, Jianwei Yang, Hang Su, Jun Zhu, et al.
Llava-plus: Learning to use tools for creating multimodal agents.
In *ECCV*, pp. 126–142. Springer, 2024b.
- Liu et al. (2024c)
Ye Liu, Zongyang Ma, Zhongang Qi, Yang Wu, Ying Shan, and Chang W Chen.
Et bench: Towards open-ended event-level video-language
understanding.
*NeurIPS*, 37:32076–32110, 2024c.
- Liu et al. (2025)
Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou.
Videomind: A chain-of-lora agent for long video reasoning.
*arXiv preprint arXiv:2503.13444*, 2025.
- Liu et al. (2024d)
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike
Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al.
Mmbench: Is your multi-modal model an all-around player?
In *ECCV*, 2024d.
- Luo et al. (2023)
Zhuoyan Luo, Yicheng Xiao, Yong Liu, Shuyan Li, Yitong Wang, Yansong Tang, Xiu
Li, and Yujiu Yang.
SOC: semantic-assisted object cluster for referring video object
segmentation.
In *NeurIPS*, 2023.
- Maaz et al. (2023)
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan.
Video-chatgpt: Towards detailed video understanding via large vision
and language models.
*arXiv preprint arXiv:2306.05424*, 2023.
- Munasinghe et al. (2024)
Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz
Khan, and Salman Khan.
Videoglamm: A large multimodal model for pixel-level visual grounding
in videos.
*arXiv preprint arXiv:2411.04923*, 2024.
- Pan et al. (2025)
Feiyu Pan, Hao Fang, Fangkai Li, Yanyu Xu, Yawei Li, Luca Benini, and Xiankai
Lu.
Semantic and sequential alignment for referring video object
segmentation.
In *CVPR*, pp. 19067–19076, 2025.
- Qian et al. (2024a)
Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting
Zhuang, and Siliang Tang.
Momentor: Advancing video large language model with fine-grained
temporal reasoning.
In *ICML*, 2024a.
- Qian et al. (2024b)
Long Qian, Juncheng Li, Yu Wu, Yaobo Ye, Hao Fei, Tat-Seng Chua, Yueting
Zhuang, and Siliang Tang.
Momentor: Advancing video large language model with fine-grained
temporal reasoning.
*arXiv preprint arXiv:2402.11435*, 2024b.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh,
Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark,
et al.
Learning transferable visual models from natural language
supervision.
In *ICML*, 2021.
- Rasheed et al. (2024)
Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan,
Hisham Cholakkal, Rao M. Anwer, Eric Xing, Ming-Hsuan Yang, and Fahad S.
Khan.
Glamm: Pixel grounding large multimodal model.
In *CVPR*, 2024.
- Ravi et al. (2024)
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali,
Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson,
Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan
Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer.
Sam 2: Segment anything in images and videos.
*arXiv preprint arXiv:2408.00714*, 2024.
- Ren et al. (2024a)
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou.
Timechat: A time-sensitive multimodal large language model for long
video understanding.
In *CVPR*, pp. 14313–14323, 2024a.
- Ren et al. (2024b)
Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng,
and Xiaojie Jin.
Pixellm: Pixel reasoning with large multimodal model.
In *CVPR*, 2024b.
- Rong et al. (2025)
Fu Rong, Meng Lan, Qian Zhang, and Lefei Zhang.
Mpg-sam 2: Adapting sam 2 with mask priors and global context for
referring video object segmentation.
*ICCV*, 2025.
- Seo et al. (2020)
Seonguk Seo, Joon-Young Lee, and Bohyung Han.
Urvos: Unified referring video object segmentation network with a
large-scale benchmark.
In *ECCV*, 2020.
- Tang et al. (2023)
Jiajin Tang, Ge Zheng, and Sibei Yang.
Temporal collection and distribution for referring video object
segmentation.
In *ICCV*, 2023.
- Wang et al. (2024a)
Haibo Wang, Zhiyang Xu, Yu Cheng, Shizhe Diao, Yufan Zhou, Yixin Cao, Qifan
Wang, Weifeng Ge, and Lifu Huang.
Grounded-videollm: Sharpening fine-grained temporal grounding in
video large language models.
*arXiv preprint arXiv:2410.03290*, 2024a.
- Wang et al. (2023)
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti
Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al.
Image as a foreign language: Beit pretraining for vision and
vision-language tasks.
In *CVPR*, pp. 19175–19186, 2023.
- Wang et al. (2024b)
Yueqian Wang, Xiaojun Meng, Jianxin Liang, Yuxuan Wang, Qun Liu, and Dongyan
Zhao.
Hawkeye: Training video-text llms for grounding text in videos.
*arXiv preprint arXiv:2403.10228*, 2024b.
- Wei et al. (2024)
Cong Wei, Haoxian Tan, Yujie Zhong, Yujiu Yang, and Lin Ma.
Lasagna: Language-based segmentation assistant for complex queries.
*arXiv preprint arXiv:2404.08506*, 2024.
- Wei et al. (2025a)
Cong Wei, Yujie Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu
Yang.
Hyperseg: Towards universal visual segmentation with large language
model.
*CVPR*, 2025a.
- Wei et al. (2025b)
Cong Wei, Yujie Zhong, Haoxian Tan, Yingsen Zeng, Yong Liu, Zheng Zhao, and
Yujiu Yang.
Instructseg: Unifying instructed visual segmentation with multi-modal
large language models.
*ICCV*, 2025b.
- Wu et al. (2023)
Dongming Wu, Tiancai Wang, Yuang Zhang, Xiangyu Zhang, and Jianbing Shen.
Onlinerefer: A simple online baseline for referring video object
segmentation.
In *ICCV*, 2023.
- Wu et al. (2022)
Jiannan Wu, Yi Jiang, Peize Sun, Zehuan Yuan, and Ping Luo.
Language as queries for referring video object segmentation.
In *CVPR*, 2022.
- Wu et al. (2024)
Jiannan Wu, Muyan Zhong, Sen Xing, Zeqiang Lai, Zhaoyang Liu, Zhe Chen, Wenhai
Wang, Xizhou Zhu, Lewei Lu, Tong Lu, et al.
Visionllm v2: An end-to-end generalist multimodal large language
model for hundreds of vision-language tasks.
*NeurIPS*, 37:69925–69975, 2024.
- Wu et al. (2025)
Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao,
Bernt Schiele, and Xu Yang.
Number it: Temporal grounding videos like flipping manga.
In *CVPR*, pp. 13754–13765, 2025.
- Xiao et al. (2024)
Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, and Changsheng Xu.
Oneref: Unified one-tower expression grounding and segmentation with
mask referring modeling.
*NeurIPS*, 37:139854–139885, 2024.
- Yan et al. (2024)
Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang,
Weidi Xie, and Efstratios Gavves.
Visa: Reasoning video object segmentation via large language models.
*ECCV*, 2024.
- Yang et al. (2023)
Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi
Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid.
Vid2seq: Large-scale pretraining of a visual language model for dense
video captioning.
In *CVPR*, pp. 10714–10726, 2023.
- Yu et al. (2016)
Lichen Yu, Patrick Poisson, Shan Yang, Alexander C Berg, and Tamara L Berg.
Modeling context in referring expressions.
In *ECCV*, 2016.
- Yu et al. (2023)
Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal.
Self-chained image-language model for video localization and question
answering.
*NeurIPS*, 36:76749–76771, 2023.
- Yuan et al. (2025)
Haobo Yuan, Xiangtai Li, Tao Zhang, Zilong Huang, Shilin Xu, Shunping Ji,
Yunhai Tong, Lu Qi, Jiashi Feng, and Ming-Hsuan Yang.
Sa2va: Marrying sam2 with llava for dense grounded understanding of
images and videos.
*arXiv preprint arXiv:2501.04001*, 2025.
- Yuan et al. (2024)
Linfeng Yuan, Miaojing Shi, Zijie Yue, and Qijun Chen.
Losh: Long-short text joint prediction network for referring video
object segmentation.
In *CVPR*, 2024.
- Yue et al. (2023)
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel
Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al.
Mmmu: A massive multi-discipline multimodal understanding and
reasoning benchmark for expert agi.
*arXiv preprint arXiv:2311.16502*, 2023.
- Zala et al. (2023)
Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Barlas Oguz, Yashar Mehdad,
and Mohit Bansal.
Hierarchical video-moment retrieval and step-captioning.
In *CVPR*, pp. 23056–23065, 2023.
- Zhang et al. (2023)
Hang Zhang, Xin Li, and Lidong Bing.
Video-llama: An instruction-tuned audio-visual language model for
video understanding.
*arXiv preprint arXiv:2306.02858*, 2023.
- Zhang et al. (2024a)
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui
Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu.
Lmms-eval: Reality check on the evaluation of large multimodal
models, 2024a.
URL [https://arxiv.org/abs/2407.12772](https://arxiv.org/abs/2407.12772).
- Zhang et al. (2024b)
Tao Zhang, Xiangtai Li, Hao Fei, Haobo Yuan, Shengqiong Wu, Shunping Ji,
Change Loy Chen, and Shuicheng Yan.
Omg-llava: Bridging image-level, object-level, pixel-level reasoning
and understanding.
In *NeurIPS*, 2024b.
- Zhang et al. (2024c)
Yuxuan Zhang, Tianheng Cheng, Lianghui Zhu, Rui Hu, Lei Liu, Heng Liu, Longjin
Ran, Xiaoxin Chen, Wenyu Liu, and Xinggang Wang.
Evf-sam: Early vision-language fusion for text-prompted segment
anything model.
*arXiv preprint arXiv:2406.20076*, 2024c.
- Zheng et al. (2024)
Rongkun Zheng, Lu Qi, Xi Chen, Yi Wang, Kun Wang, Yu Qiao, and Hengshuang Zhao.
Villa: Video reasoning segmentation with large language model.
*arXiv preprint arXiv:2407.14500*, 2024.
- Zhu et al. (2023)
Jiawen Zhu, Zhi-Qi Cheng, Jun-Yan He, Chenyang Li, Bin Luo, Huchuan Lu, Yifeng
Geng, and Xuansong Xie.
Tracking with human-intent reasoning.
*arXiv preprint arXiv:2312.17448*, 2023.

## Appendix

We provide an overview of the Appendix below. Each section includes a brief description of its contents.

- •
Appendix [A](https://arxiv.org/html/2510.09274v1#A1) — Use of Large Language Models (LLMs)
- •
Appendix [B](https://arxiv.org/html/2510.09274v1#A2) — Additional Implementation Details
–
Appendix [B.1](https://arxiv.org/html/2510.09274v1#A2.SS1) Dataset and Evaluation Metrics
–
Appendix [B.2](https://arxiv.org/html/2510.09274v1#A2.SS2) Training Details
- •
Appendix [C](https://arxiv.org/html/2510.09274v1#A3) — Additional Methods
–
Appendix [C.1](https://arxiv.org/html/2510.09274v1#A3.SS1) Temporal Token Injecting
–
Appendix [C.2](https://arxiv.org/html/2510.09274v1#A3.SS2) Instruction Details
- •
Appendix [D](https://arxiv.org/html/2510.09274v1#A4) — Additional Results
–
Appendix [D.1](https://arxiv.org/html/2510.09274v1#A4.SS1) Results on Image/Video QA Tasks
–
Appendix [D.2](https://arxiv.org/html/2510.09274v1#A4.SS2) Comparion Image/Video Segmentation with Non-LLM-based Methods.
- •
Appendix [E](https://arxiv.org/html/2510.09274v1#A5) — Additional Ablation Studies
–
Appendix [E.1](https://arxiv.org/html/2510.09274v1#A5.SS1) Analysis of Temporal Token Injecting
–
Appendix [E.2](https://arxiv.org/html/2510.09274v1#A5.SS2) [FIND] Matching Paradigm v.s. Plain-Text Generation
–
Appendix [E.3](https://arxiv.org/html/2510.09274v1#A5.SS3) Effect of Frame Numbers on RefVOS
–
Appendix [E.4](https://arxiv.org/html/2510.09274v1#A5.SS4) Effect of Post-processing Threshold for TSG
- •
Appendix [F](https://arxiv.org/html/2510.09274v1#A6) — Additional Qualitative Results
–
Appendix [F.1](https://arxiv.org/html/2510.09274v1#A6.SS1) Visualization Comparison with Sa2VA
–
Appendix [F.2](https://arxiv.org/html/2510.09274v1#A6.SS2) Referring Instruction Visualization
–
Appendix [F.3](https://arxiv.org/html/2510.09274v1#A6.SS3) Reasoning Instruction Visualization
–
Appendix [F.4](https://arxiv.org/html/2510.09274v1#A6.SS4) Badcase Analysis
–
Appendix [F.5](https://arxiv.org/html/2510.09274v1#A6.SS5) Limitations

## Appendix A Use of Large Language Models (LLMs)

A Large Language Model was employed solely to improve the manuscript’s language, including grammar, clarity, and readability.
No technical content, experimental design, analysis, or conclusions were produced or altered by the LLM.
All scientific ideas, methodologies, and results remain entirely the authors’ original work.

## Appendix B Additional Implementation Details

### B.1 Dataset and Evaluation Metrics

Datasets: We train our MomentSeg model on a comprehensive dataset encompassing five categories: temporal sentence grounding, image question answering (QA), video QA, image segmentation, and video segmentation. The detailed composition of the training set is provided in Table [11](https://arxiv.org/html/2510.09274v1#A2.T11). Our data curation largely follows the methodology of Sa2VA (Yuan et al., 2025), with notable additions including ReasonSeg (0.2K) (Lai et al., 2024) and two temporal sentence grounding datasets: Charades-STA (5.3K) (Gao et al., 2017) and ActivityNet Captions (10K) (Caba Heilbron et al., 2015).

Given that the Qwen2.5-VL (Bai et al., 2025) has already been extensively pre-trained on image and video QA data, we deliberately restrict the inclusion of such data to 665K samples from LLaVA 1.5 (Liu et al., 2024a) and 100K from ChatUniVi (Jin et al., 2024), thereby mitigating the risk of catastrophic forgetting of its foundational QA capabilities. For image-level referring segmentation, we incorporate 56K referring expression data (Kazemzadeh et al., 2014; Yu et al., 2016) and 214K grounding conversation generation data (Rasheed et al., 2024). For video-level referring segmentation, we utilize 5.8K existing referring VOS samples from Ref-YouTubeVOS (Seo et al., 2020), MeVIS (Ding et al., 2023), and ReVOS (Yan et al., 2024).

Evaluation Metrics: For RefVOS, unless specified otherwise, we use the following evaluation metrics: $\mathcal{J}$ (average Intersection over Union, IoU), $\mathcal{F}$ (boundary F-measure), and $\mathcal{J}$&$\mathcal{F}$ (the average of $\mathcal{J}$ and $\mathcal{F}$). For TSG, we report the Intersection over Union (IoU) between the predicted and ground truth timestamps, including Recall at IoU thresholds of {0.3, 0.5, 0.7}, as well as the mean IoU.

**Table 11: Composition of datasets from different tasks used during training.**
| Type | Datasets |
| --- | --- |
| Temporal Sentence Grounding | Charades-STA (5.3K), ActivityNet Captions (10K) |
| Image QA | LLaVA 1.5 (665K) |
| Image Segmentation | RefCOCO (17K), RefCOCO+ (17K), RefCOCOg (22K), Grand-f (214K), ReasonSeg (0.2K) |
| Video QA | ChatUniVi (100K) |
| Video Segmentation | Ref-YTVOS (3.5K), MeVIS (0.6K), ReVOS (1.7K), Ref-SAV (37K) |

### B.2 Training Details

Table [12](https://arxiv.org/html/2510.09274v1#A2.T12) summarizes the training configurations. Our model is trained on 8 H20 GPUs with a per-GPU batch size of 2, requiring approximately 24 hours for the 3B model. During training, we configure the maximum temporal frames to 50, TSG frames to 60, and video frames to 5. For inference, we increase the maximum TSG frames to 100, set the TSG threshold to 0.4, and use 8 video frames. The maximum pixel resolutions are configured as follows: video inputs at $16\times 16\times 28\times 28$, temporal inputs at $4\times 4\times 28\times 28$, TSG inputs at $8\times 8\times 28\times 28$, and image inputs at $20\times 20\times 28\times 28$. We employ the AdamW optimizer with $\beta_{1}=0.9$, $\beta_{2}=0.999$, and a weight decay of 0.05. The learning rate is set to $4\times 10^{-5}$ with a warmup ratio of 0.05.

**Table 12: Implementation details of training process.**
| Config | Value |
| --- | --- |
| max temporal frame num | 50 |
| max TSG frame num | 60 |
| video frame num | 5 |
| max video pixels | 16*16*28*28 |
| max temporal pixels | 4*4*28*28 |
| max TSG pixels | 8*8*28*28 |
| max image pixels | 20*20*28*28 |
| optimizer | AdamW |
| optimizer momentum | $\beta_{1},\beta_{2}=0.9,0.999$ |
| optimizer weight decay | 0.05 |
| learning rate | 4e-5 |
| LoRA rank | 128 |
| batch size | 2 |
| warmup ratio | 0.05 |

## Appendix C Additional Methods

### C.1 Temporal Token Injecting

Beyond the visual tokens extracted from selected video frames, we introduce temporal tokens to explicitly encode temporal dynamics. These tokens are meticulously designed to capture the relative temporal ordering of frames within the video sequence, thereby substantially bolstering the model’s capacity for temporal reasoning.
Our methodology draws conceptual parallels with GLUS (Lin et al., 2025), which integrates a finite set of global tokens, and VideoLISA (Bai et al., 2024), which condenses video frames into a singular token representation. A key distinction lies in that our temporal tokens are exclusively injected during the training phase. Furthermore, we leverage the Qwen2.5-VL’s video encoder mode to achieve significant computational efficiency. Specifically, these temporal tokens are generated from uniformly sampled video frames. Each sampled frame undergoes compression to a highly compact input dimension, with a maximum allocation of 16 tokens per individual image. The max count of uniformly sampled frames is restricted to 50. This entire process can be formally described by Eq. [4](https://arxiv.org/html/2510.09274v1#S3.E4).

### C.2 Instruction Details

We list the prompts used for different tasks during training in Table [13](https://arxiv.org/html/2510.09274v1#A3.T13). For the TSG task, the prompt consists of video, query, and answer components, where the video component is constructed using Qwen2.5-VL’s 
 placeholder. For the RefVOS task, the prompt consists of video, image, query, and answer components. Compared to TSG, RefVOS includes an additional image component, which is constructed using Qwen2.5-VL’s 
 placeholder. For the answer component, we use [FIND] and [SEG] to represent the answers for TSG and RefVOS tasks, respectively. We design multiple different prompts for each component and randomly sample them during training to improve the model’s robustness.

Table: Table 13: Prompts used for different tasks.

## Appendix D Additional Results

### D.1 Results on Image/Video QA Tasks.

Maintaining basic QA capabilities remains a significant challenge for LMMs designed for specific tasks. Table [14](https://arxiv.org/html/2510.09274v1#A4.T14) presents the performance of our method on image/video-based question answering tasks (Fu et al., 2023; Chen et al., 2024b; Kembhavi et al., 2016). Compared to other MLLMs with segmentation capabilities, our approach achieves the best accuracy trade-off between image chat and referential segmentation datasets. Additionally, our method excels on the Video-MME (Fu et al., 2025) video question answering benchmark, demonstrating its strong capabilities in multimodal understanding. Furthermore, we replicate the results for Qwen2.5-VL-3B (Bai et al., 2025) using the lmms-eval framework (Zhang et al., 2024a), ensuring consistency and fairness in evaluation. After incorporating segmentation capabilities into MomentSeg based on Qwen2.5-VL, there is no significant degradation in the underlying QA performance. In fact, MomentSeg outperforms on certain evaluation sets, such as MMbench (Liu et al., 2024d), SEED-Bench (Li et al., 2023), and MMMU (Yue et al., 2023), showcasing improvements in specific benchmarks.

**Table 14: Performance on image/video benchmarks with MLLMs possessing segmentation capabilities. The results for Qwen2.5-VL-3B (Bai et al., 2025) are replicated using the same lmms-eval (Zhang et al., 2024a) framework.**
| Method | Image QA | Video QA |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| MME | MMBench | SEED-Bench | AI2D | MMStar | MMMU | Video-MME |  |
| LISA-7B (Lai et al., 2024) | 1/1 | 0.4 | - | - | - | - | - |
| PixelLM-7B (Ren et al., 2024b) | 309/135 | 17.4 | - | - | - | - | - |
| LaSagnA-7B (Wei et al., 2024) | 0/0 | 0.0 | - | - | - | - | - |
| GLaMM-7B (Rasheed et al., 2024) | 14/9 | 36.8 | - | 28.2 | - | - | - |
| OMG-LLaVA-7B (Zhang et al., 2024b) | 1177/235 | 47.9 | 56.5 | 42.9 | - | - | - |
| Sa2VA-4B (Yuan et al., 2025) | 1553/540 | 76.8 | 72.6 | 79.9 | 53.7 | 46.2 | 50.4 |
| Sa2VA-8B (Yuan et al., 2025) | 1651/578 | 82.4 | 75.5 | 82.1 | 60.3 | 44.7 | 52.1 |
| Qwen2.5-VL-3B (Bai et al., 2025) | 1511/622 | 77.6 | 74.7 | 78.4 | 55.7 | 46.4 | 57.7 |
| MomentSeg-3B (Ours) | 1491/529 | 79.2 | 74.8 | 78.3 | 56.0 | 47.6 | 55.4 |
| MomentSeg-7B (Ours) | 1595/684 | 83.4 | 76.7 | 82.6 | 62.1 | 50.3 | 60.9 |

### D.2 Comparison of Image/Video Segmentation with Non-LLM-based Methods

Recent advances in small-scale models have achieved remarkable progress in traditional RIS and RVOS tasks. Several approaches (Dai et al., 2024; 2025c) leverage powerful vision-language understanding models such as BEiT-3 (Wang et al., 2023), substantially enhancing referring comprehension capabilities in both image (Xiao et al., 2024; Dai et al., 2025a) and video (Rong et al., 2025; Zhang et al., 2024c) domains. To provide a comprehensive comparative analysis, we additionally evaluate non-LLM-based methods. As shown in Table [15](https://arxiv.org/html/2510.09274v1#A4.T15), MomentSeg, as a general-purpose video and image segmentation model, demonstrates competitive performance across diverse tasks, particularly when compared to smaller-scale models. Notably, it excels on datasets requiring temporal action understanding, such as MeVIS, where it exhibits superior capabilities in modeling temporal dynamics.

**Table 15: Comparison of model results with non-LLMs-based methods**
| Method | RefCOCO | RefCOCO+ | RefCOCO | MeVIS | Ref-Youtube-VOS | Ref-DAVIS17 |
| --- | --- | --- | --- | --- | --- | --- |
| ReLA (Liu et al., 2023) | 73.8 | 66.0 | 65.0 | - | - | - |
| EEVG (Chen et al., 2024d) | 79.5 | 71.9 | 73.6 | - | - | - |
| C3VG (Dai et al., 2025d) | 81.4 | 77.1 | 76.3 | - | - | - |
| OneRef-L (Xiao et al., 2024) | 81.3 | 76.6 | 75.7 | - | - | - |
| DeRIS-B (Dai et al., 2025b) | 82.0 | 75.6 | 76.3 | - | - | - |
| SOC (Luo et al., 2023) | - | - | - | - | 67.3 | 65.8 |
| DsHmp (He & Ding, 2024) | - | - | - | 46.4 | 67.1 | 64.9 |
| DMVS (Fang et al., 2025a) | - | - | - | 48.6 | 64.3 | 65.2 |
| SAMWISE (Cuttano et al., 2025) | - | - | - | 48.3 | 67.2 | 68.5 |
| ReferDINO (Liang et al., 2025) | - | - | - | 49.3 | 69.3 | 68.9 |
| MPG-SAM2 (Rong et al., 2025) | - | - | - | 53.7 | 73.9 | 72.4 |
| MomentSeg-3B (Ours) | 82.1 | 76.9 | 78.8 | 54.8 | 72.0 | 76.4 |
| MomentSeg-7B (Ours) | 82.6 | 78.2 | 80.1 | 57.1 | 72.3 | 77.4 |

## Appendix E Additional Ablation Studies

### E.1 Analysis of Temporal Tokens Injecting.

To examine the effect of temporal tokens on model performance, we conduct ablation studies by varying their number during both training and inference.
As shown in Fig. [7](https://arxiv.org/html/2510.09274v1#A5.F7), we report results on three RefVOS datasets: MeVIS, Ref-DAVIS17, and ReasonVOS.
During training, increasing the number of temporal frames yields a steady performance gain on MeVIS, saturating around 50 frames.
Training cost grows proportionally with the number of temporal frames; thus we set it to 50 in all experiments.
At inference, as also illustrated in Fig. [7](https://arxiv.org/html/2510.09274v1#A5.F7), adding temporal frames has minimal impact on accuracy.
To improve efficiency, we therefore omit temporal frames during inference.

Figure: Figure 6: Effect of temporal numbers for training.
Refer to caption: figures/app_ab_videoframes_training.png

### E.2 [FIND] Matching Paradigm v.s. Plain-Text Generation

To compare the effectiveness of the [FIND] token approach, we conducted an ablation experiment that simultaneously supports both matching paradigm and plain-text generation. In prompt construction, we additionally include output with format of (start_ratio, end_ratio) as a 0-1 range. after the [FIND] instruction. The answer construction further adds The range is ({},{}) based on the [FIND] token, enabling the model to perform TSG tasks through both methods simultaneously.
The experimental results in Table [16](https://arxiv.org/html/2510.09274v1#A5.T16) demonstrate that the [FIND] token approach significantly outperforms the plain-text generation method across all metrics on both Charades-STA and ActivityNet-Grounding datasets. On Charades-STA, the [FIND] token achieves improvements of 10.4, 11.2, and 6.5 points in R@0.3, R@0.5, and R@0.7, respectively, with a substantial 9.3-point gain in mIoU. Similarly, on ActivityNet-Grounding, the [FIND] token shows consistent improvements, particularly in R@0.3 and R@0.5 metrics.
The superior performance of the [FIND] token can be attributed to its direct prediction mechanism. Unlike the plain-text generation approach, which requires the model to understand and generate explicit timestamp encoding information, the [FIND] token directly enables the model to predict text-relevant key positions through similarity-based localization. This approach significantly simplifies the learning difficulty by eliminating the need for complex temporal coordinate regression, allowing the model to focus on semantic alignment between textual queries and visual content rather than numerical timestamp generation.

**Table 16: Effect of post-process threshold $\theta$ on TSG.**
| Method | Charades-STA | ActivityNet-Grounding |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R@0.3 | R@0.5 | R@0.7 | mIoU | R@0.3 | R@0.5 | R@0.7 | mIoU |  |
| Plain-Text Generation | 64.9 | 45.4 | 25.8 | 41.3 | 65.1 | 48.6 | 27.9 | 46.0 |
| [FIND] Matching | 75.3 | 56.6 | 32.3 | 50.6 | 70.3 | 52.0 | 29.0 | 49.3 |

### E.3 Effect of Frame Numbers on RefVOS

In the RefVOS task, the number of sampled frames strongly affects the model’s ability to identify the referred target.
Table [17](https://arxiv.org/html/2510.09274v1#A5.T17) reports performance on three RefVOS datasets under different frame counts.
On MeViS, accuracy improves as the number of frames increases, reflecting its motion-driven nature where richer temporal context aids understanding.
In contrast, results on Ref-DAVIS17 and ReasonVOS fluctuate, and excessive frames can even reduce accuracy.
A moderate increase in frames can help, but too many introduce irrelevant segments that degrade segmentation.
Moreover, the gap between training and testing—training uses only five frames—reduces the proportion of tokenized text instructions at inference, causing the model to underutilize crucial textual cues.
This mismatch warrants further investigation.

**Table 17: Effect of frame number $N_{f}$ on RefVOS. Evaluated with $\mathcal{J}$&$\mathcal{F}$ metric.**
| $N_{f}$ | MeViS ($val^{u}$) | Ref-DAVIS17 | ReasonVOS |
| --- | --- | --- | --- |
| 5 | 61.4 | 77.1 | 59.2 |
| 8 | 62.0 | 76.3 | 61.7 |
| 10 | 61.9 | 76.4 | 60.3 |
| 12 | 62.5 | 76.8 | 59.9 |
| 15 | 62.6 | 76.0 | 59.7 |

### E.4 Effect of Post-processing Threshold for TSG

For the proposed MomentSeg, the post-processing threshold $\theta$ plays a crucial role in determining the accuracy of moment localization. Table [18](https://arxiv.org/html/2510.09274v1#A5.T18) demonstrates the impact of varying $\theta$ values on the performance of two TSG benchmarks, Charades-STA and ActivityNet-Grounding, evaluated using recall at different IoU thresholds (R@0.3, R@0.5, and R@0.7) and mean IoU (mIoU). We observe that for Charades-STA, increasing $\theta$ from 0.2 to 0.3 results in significant improvements across all metrics, particularly in R@0.3 and mIoU, where the best performance is achieved at $\theta=0.3$. Further increasing $\theta$ leads to a decline in some lower-precision metrics, while higher-precision metrics continue to improve, indicating that a higher threshold favors retaining high-confidence moments, resulting in a high-precision, low-recall state.
Similarly, for ActivityNet-Grounding, we observe that $\theta=0.4$ strikes a balance between recall and precision, providing an overall stable performance across the metrics. Therefore, $\theta=0.4$ is selected as the optimal threshold for this paper.

**Table 18: Effect of post-process threshold $\theta$ on TSG.**
| $\theta$ | Charades-STA | ActivityNet-Grounding |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| R@0.3 | R@0.5 | R@0.7 | mIoU | R@0.3 | R@0.5 | R@0.7 | mIoU |  |
| 0.2 | 75.4 | 49.8 | 21.9 | 47.5 | 66.1 | 41.0 | 20.9 | 44.9 |
| 0.3 | 78.9 | 55.9 | 25.3 | 50.7 | 65.6 | 45.6 | 23.8 | 45.1 |
| 0.4 | 76.1 | 58.2 | 25.8 | 50.2 | 67.5 | 44.7 | 23.2 | 45.4 |
| 0.5 | 72.9 | 55.4 | 31.1 | 49.7 | 62.2 | 42.1 | 22.4 | 42.7 |
| 0.6 | 71.8 | 54.6 | 27.9 | 48.4 | 59.2 | 39.5 | 21.7 | 41.2 |

## Appendix F Additional Qualitative Results

### F.1 Visualization Comparison with Sa2VA

We compare the visual results of our proposed MomentSeg with Sa2VA on the RefVOS task.
As shown in Fig. [8](https://arxiv.org/html/2510.09274v1#A6.F8), three representative examples are provided.
(a) Highlights a key drawback of Sa2VA’s FirstK sampling strategy: sampling only the early portion of a video fails to capture later object appearances, leading to missed segmentations. In contrast, our TSG module accurately localizes the action described in the text and performs dense sampling within that segment, enabling correct object segmentation.
(b) Shows a case where Sa2VA entirely misses the target. The object is small and requires precise sampling; early sampling prevents Sa2VA from observing the later ‘browsing phone’ action.
(c) Demonstrates that insufficient sampling of later key moments produces unreliable mask initialization and incomplete predictions for Sa2VA. Our method instead initializes at the most relevant frames and applies bidirectional propagation and updating, yielding more stable mask segmentation.

Figure: Figure 8: Additional visual comparison of MomentSeg and Sa2VA on the RefVOS task.
Refer to caption: figures/app_comparision_visualization.jpg

### F.2 Referring Instruction Visualization

In Fig. [9](https://arxiv.org/html/2510.09274v1#A6.F9), we present additional visualization results of MomentSeg-3B on the Referring VOS task. First, it is observed that the TSG inference stage accurately localizes the action segments described by the text. Moreover, dense sampling occurs in regions with sharp peaks in the similarity distribution, as shown in the 2nd and 5th samples. This sampling strategy effectively aids the model in understanding the action described by the text and allows for accurate tracking of the referred target during the subsequent BAP tracking process. Additionally, in complex scenarios, such as the 2nd, 3rd, and 4th samples, where multiple similar targets appear in the video, the model can still accurately segment the referred target by combining the understanding of the text description with action information.

Figure: Figure 9: The additional visualization of the MomentSeg on Referring VOS task.
Refer to caption: figures/app_referring_visualization.jpg

### F.3 Reasoning Instruction Visualization

In Fig. [10](https://arxiv.org/html/2510.09274v1#A6.F10), we show additional visualization results of MomentSeg-3B on the Reasoning VOS task. These examples are characterized by text descriptions that do not directly refer to a specific object but require indirect reasoning to identify the referred object. Most of these examples are in the form of questions. During this stage, it is observed that since the text does not describe much motion-related information, the TSG typically localizes the target’s segment in the video rather than focusing on the critical moment. This remains a bottleneck for current methods, as we use reasoning-related TSG data for training.

Figure: Figure 10: The additional visualization of the MomentSeg on Reasoning VOS task.
Refer to caption: figures/app_reasoning_visualization.jpg

### F.4 Badcase Analysis

In Fig. [11](https://arxiv.org/html/2510.09274v1#A6.F11), we show some failure cases of MomentSeg-3B on the RefVOS task. (a) shows that due to incorrect localization of key segments during the TSG stage, the sampled frames are not representative, and with interference from multiple targets, the model misidentifies the target as smaller, resulting in erroneous and fluctuating segmentation. (b) highlights a significant issue with current methods: the lack of encoding for frame-relative video timestamps prevents the model from understanding concepts like ”at the end of the video.” This leads to tracking drift and inaccurate segmentation results. (c) and (d) show cases with multiple referred objects where, due to the absence of instance-level supervision, the model performs poorly. This remains a major bottleneck in current methods.

### F.5 Limitations

MomentSeg still offers substantial room for improvement.
For example, for temporal grounding, the current training data remain limited and are confined to the TSG task.
Incorporating related tasks—such as Dense Video Captioning (Iashin & Rahtu, 2020), Video Highlight Detection (Lei et al., 2021), and Temporally Grounded Video Question Answering (Chen et al., 2024a; Liu et al., 2024c)—could enhance the model’s generalization ability.
Moreover, the number of sampled frames is currently fixed; future work could explore dynamically adapting the sampling rate to different video complexities, allowing more frames to assist understanding in scenes with intricate descriptions and thereby enabling more accurate target segmentation.

Figure: Figure 11: Additional visualization of MomentSeg failure cases.
Refer to caption: figures/app_badcase.jpg

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 3
strengths: 1.	The paper addresses a valuable research question: Is keyframe selection necessary?
2.	The experiments cover a wide range of tasks and datasets, with comprehensive baseline comparisons.
3.	The ablation studies are well-designed and clearly demonstrate the contribution of each component.
4.	The proposed method achieves competitive performance.
summary: This work proposes a unified framework that jointly optimizes temporal sentence grounding and segmentation. A novel [FIND] token enables key moment identification without external timestamps, while a Moment-Centric Sampling (MCS) strategy balances dense and sparse frame sampling for efficiency and context preservation. Additionally, a Bidirectional Anchor-updated Propagation (BAP) mechanism enhances tracking stability by adaptively updating masks during inference.
weaknesses: 1.	The FIND token plays a critical role in the proposed approach. However, the paper does not clearly explain how this token is supervised or optimized. Moreover, the semantic role of the FIND token within the model (e.g., whether it acts as a query, indicator, or moment aggregator) remains unclear.
2.	The Moment-Centric Sampling design appears rather complex, but the authors lacks justification for its necessity and detailed analysis of why this design is better.
3.	The overall framework is complicated with many hyperparameters, which increases the tuning cost.
4.	The inference efficiency is not adequately discussed. Considering the method includes multiple tokens and multiple steps, the inference cost may be relatively high.
5.	Figures and writing quality could be further polished for clarity and readability.

---

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See the weakness.
soundness: 2
strengths: 1. Clear Motivation: The work provides a clear and compelling motivation, effectively highlighting the inherent limitations of conventional fixed or simple keyframe sampling approaches when dealing with moment-centric video-language tasks.
2. Comprehensive Experiments and Strong Results: The experimental evaluation is notably comprehensive, providing ample technical detail and achieving excellent, state-of-the-art results across a diverse set of benchmark datasets.
3. Well-Structured and Accessible Methodology: The overall methodology and presentation of the paper are clear, logical, and easy to follow, making the proposed MomentSeg architecture and techniques accessible to the broader reader community.
summary: This paper proposes MomentSeg, a novel framework for referring video object segmentation (RVOS) that addresses the limitations of sparse and fixed sampling in existing methods. The core contribution is the Moment-Centric Sampling (MCS) mechanism, which intelligently selects frames most relevant to the referring expression to improve temporal grounding and visual context. Furthermore, the model employs a Bidirectional Anchor-updated Propagation (BAP) strategy to ensure robust and temporally consistent mask tracking across the selected video segment. The experimental results, validated across multiple challenging datasets, demonstrate that MomentSeg achieves state-of-the-art or highly competitive performance, confirming the effectiveness of the proposed sampling and propagation strategies.
weaknesses: 1. Incremental Gain of MCS: The core idea of "keyframe mining" appears somewhat incremental, and the reported empirical gains are small; Figure 2 suggests the impact of the sampling strategy is marginal (around 2 points), and Table 7 confirms that the gain provided by MCS over the base keyframe methods is minimal.
2. Similarity to Prior Grounding Work: The approach of using the [FIND] token to localize keyframes is highly similar to the established "streaming EOS prediction" mechanism found in related work, such as VideoLLM-Online, which diminishes the technical novelty of this specific component.
3. Limited Applicability to Online Scenarios: Key components of the proposed method, including Moment-Centric Sampling (MCS) and Bidirectional Anchor-updated Propagation (BAP), fundamentally require access to the entire video's context, severely limiting their applicability to real-world online or streaming environments.
4. Novelty of Bidirectional Propagation: While effective within this framework, the concept of bidirectional propagation for ensuring temporal consistency is not a novel technique in the broader Video Object Segmentation (VOS) literature and represents mainly an adaptation to the SAM model rather than a core innovation.
5. Missing Robustness Analysis: The paper lacks a deep-dive analysis on the method's robustness, particularly how BAP performs when anchor frames contain significant noise or the model encounters high degrees of occlusion, which are common challenges in propagation-based models.

[*] VideoLLM-online: Online Video Large Language Model for Streaming Video, CVPR2024.

---

confidence: 3
contribution: 2
initial_rating: 4
presentation: 4
questions: 1. Inference Cost
    - Could you provide a detailed breakdown of the inference latency and/or FLOPs for the full MomentSeg pipeline? How does it compare to other LMM-based methods that might use a simpler, single-pass sampling strategy?
soundness: 3
strengths: ## Strengths

1. **Comprehensive empirical evaluation**: The paper provides extensive experiments across multiple benchmarks including TSG (Charades-STA, ActivityNet), RefVOS (MeVIS, ReVOS, Ref-Youtube-VOS, Ref-DAVIS17), image segmentation (RefCOCO/+/g, ReasonSeg, GCG), and QA tasks. This breadth demonstrates the method's versatility.

2. **Consistent improvements across benchmarks**: The method shows meaningful gains over strong baselines, particularly +5% on MeVIS valu (62.0 vs 57.5) and +6% on ReVOS overall (62.6 vs 58.7) compared to Sa2VA baseline. The 3B model outperforms several 7B models on certain tasks.

3. **Thorough ablation studies**: Tables 6-10 provide detailed ablations of key components (TTI, MCS, BAP), different sampling strategies (Table 7), and joint training effects (Table 10). The analysis of sampling strategies in Fig. 2 effectively motivates the problem.

4. **Eliminates external keyframe selection models**: Unlike VISA, ViLLA, GLUS, and VRS-HQ which require separate models (LLaMA-VID, Grounded-VideoLLM, CLIP) for keyframe identification, this method learns keyframe selection end-to-end within the main model, reducing pipeline complexity.

5. **Well-motivated design**: The analysis showing that random sampling has high variance (Fig. 2) and that uniform sampling outperforms firstK on motion-driven datasets provides good empirical motivation for learned keyframe selection.

6. **Strong TSG results**: Table 2 shows impressive performance on temporal grounding (76.1 R@0.3, 58.2 R@0.5 on Charades-STA) despite using a smaller 3B model, outperforming methods like VTimeLLM-7B and HawkEye-7B.

7. **Detailed supplementary material**: The 16-page appendix includes additional experiments, visualizations, implementation details, and failure case analysis, demonstrating thoroughness.

8. **Clear presentation**: The paper is generally well-written with informative figures (especially Fig. 1, 3, 4) that effectively illustrate the approach. The distinction between training and inference pipelines is clearly explained.
summary: ## Summary
This paper proposes MomentSeg for joint TSG and RefVOS training with [FIND] token-based keyframe selection, moment-centric sampling (MCS), and bidirectional anchor-updated propagation (BAP). While empirical results show improvements, the work suffers from limited novelty, questionable experimental design, and theoretical gaps.
weaknesses: ### Concerns Regarding the Novelty of Proposed Components

While the paper achieves impressive empirical results, I have some concerns regarding the claimed novelty of the core technical components ([FIND] token, MCS, and BAP), which appear to be effective applications of well-established principles.

**1. On the [FIND] Token Paradigm**

The paper introduces the [FIND] token as part of a "novel TSG paradigm". However, the described mechanism—a learnable token that is matched against frame features using cosine similarity and trained with a contrastive loss—bears a strong resemblance to standard learnable queries used in attention-based models (e.g., DETR) and has been widely explored in video grounding literature.

* **Request for Clarification:** Could the authors elaborate on the specific technical distinctions that qualify this as a "novel paradigm" rather than a successful and well-executed application of existing query-key matching mechanisms for the task of temporal grounding?

**2. On Moment-Centric Sampling (MCS)**

The proposed MCS strategy samples frames based on the similarity distribution generated by the [FIND] token, effectively sampling more densely from regions of high relevance.

* **Observation:** This method appears to be a direct application of importance sampling. The use of Inverse CDF sampling, as detailed in Algorithm 1, is a textbook statistical method for drawing samples from an arbitrary probability distribution. While this is a sensible and effective choice, it may be an overstatement to present the sampling algorithm itself as a novel contribution.

**3. On Bidirectional Anchor-updated Propagation (BAP)**

The BAP mechanism is presented as a method to enhance tracking robustness.

* **Observation:** The key ideas of (1) initializing a tracker from a high-confidence "anchor" frame and (2) propagating both forward and backward in time are standard and widely-used techniques in the video object segmentation (VOS) field to prevent error accumulation.
* **Request for Clarification:** The ablation in Table 8 also shows that the full BAP (with Bidirectional Propagation) results in a slight performance drop on Ref-DAVIS17 (76.8 vs. 77.0) compared to just using "Moment-anchored Updating." This seems to slightly undermine the claim of its universal benefit. Could the authors clarify the specific novelty of BAP beyond combining these existing VOS techniques, and perhaps comment on this performance discrepancy?

**Summary of Concern:**

The primary contribution of this work appears to be the skillful combination and joint optimization of these components into a high-performing, unified framework for RefVOS and TSG. While the engineering is strong and the results are state-of-the-art, the paper would be more impactful if it more clearly distinguished its fundamental research contributions from the novel application of existing, well-established methods.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: 1. Efficiency Analysis Needed: No report of inference time or computational overhead-important for real-world deployment.
2. Comparative Positioning vs.Prior Work: The relation to window-based methods (MS-TCN++,ASFormer)should be analytically clarified-e.g.,how MomentSeg differs from dynamic temporal pooling or multi-scale convolution.
3. Generalization Study Beyond Benchmarks: ncluding transfer experiments (e.g.,from GTEA>Breakfast)could demonstrate robustness beyond in-distribution evaluations.
soundness: 3
strengths: 1. The moment-centric modeling formulation is conceptually sound and backed by a clear motivation:local semantic coherence is often more stable than frame-wise predictions.
2. The appendix adds significant methodological rigor,defining how temporal tokens and attention masks are constructed,and validating their influence.
3. The ablation studies are systematic and reproducible-exploring token number,temporal frame count,threshold sensitivity,and alternative output paradigms ([FIND]vs.text generation).
summary: This thesis addresses the shortcomings of existing large language model (LMM)-based approaches in the referential video target segmentation (RefVOS) task i.e., either relying on external keyframe models to increase the system complexity or adopting a manual sampling strategy to lose the critical timing cues i.e., proposes a unified framework MomentSeg to achieve the joint optimization of temporal sentence localization (TSG) and RefVOS to improve the video pixel-level comprehension. In the training phase, [FIND] token is introduced to identify text-related key frames by frame-level temporal tokn similarity matching without external timestamp encoding; in the inference phase, a moment-centered sampling (MCS) strategy is designed to densely sample high-relevance key moments and sparsely sample non-key frames, and a bi-directional anchor update propagation (BAP) is proposed to bi-directionally propagate the forward and backward temporal sequence from key frames At the same time, we propose the bidirectional anchor update propagation (BAP), which propagates the mask from the key frames to the forward and backward time sequences in a bi-directional manner and updates it adaptively to reduce the tracking accumulation error. 
Its core contributions include 1. Proposing MomentSeg unified framework, which realizes native keyframe selection by matching the similarity between [FIND] token and frame token during training, which eliminates the need of relying on external keyframe models and the need of explicit timestamp encoding, and simplifies the system design. 2. Design Moment Centered Sampling (MCS): Based on the similarity distribution of [FIND]tokn output, dense sampling at text-related key moments and sparse sampling in non-critical regions, taking into account the preservation of key motion details and global timing context maintenance. 3. Bidirectional Anchor Update Propagation (BAP) is proposed: taking the key frames recognized by MCS as the starting point, the mask is propagated forward and backward chronologically in a bidirectional manner, and the mask and memory are updated adaptively at the sampling points, which can effectively alleviate the cumulative tracking error and improve the robustness of long video segmentation.
weaknesses: 1. The moment definition remains implicit:moments are derived implicitly from attention maps,not through an explicit segmentation constraint This makes reproducibility somewhat architecture-dependent.
2. The causal or theoretical justification for why "moment aggregation"improves generalization is intuitive but lacks formal backing (e.g.,no information-theoretic analysis or invariance claim).
3. Temporal boundary precision is still limited;CRF helps smooth predictions but does not guarantee sharp semantic transition detection. 
4. Efficiency considerations are largely omitted (e.g.,FLOPs,latency),though essential for TAS scalability.


Decision: Reject