ReferEverything: Towards segmenting everything we can speak of in videos

## Abstract

Abstract We present REM, a framework for segmenting a wide range of concepts in video that can be described through natural language. Our method capitalizes on visual-language representations learned by video diffusion models on Internet-scale datasets. A key insight of our approach is preserving as much of the generative model’s original representation as possible, while fine-tuning it on narrow-domain Referral Object Segmentation datasets. As a result, our framework can accurately segment and track rare and unseen objects, despite being trained on object masks from a limited set of categories. Additionally, it can generalize to non-object dynamic concepts, such as waves crashing in the ocean, as demonstrated in our newly introduced benchmark for Referral Video Process Segmentation (Ref-VPS).
Our experiments show that REM performs on par with state-of-the-art approaches on in-domain datasets, like Ref-DAVIS, while outperforming them by up to 12 points in terms of region similarity on out-of-domain data, leveraging the power of Internet-scale pre-training. We include all of the video visualizations at https://miccooper9.github.io/projects/ReferEverything/ .

## 1 Introduction

One of the most remarkable features of natural language is its ability to describe human visual experience in all of its richness and complexity. Whether capturing fleeting moments, like raindrops rolling down the window, or smoke dissipating from a cigarette (see row 2 in Figure 1), or describing dynamic processes, such as a glass shattering or a whirlpool forming in the water (row 1 in Figure 1), if we can utter them, we can also accurately localize them in space and time. This universal mapping between the discrete, symbolic realm of language and the continuous, ever-changing visual world is developed through a lifetime of visual-linguistic interaction (Barsalou, ; Popham et al., ).

Figure: Figure 1: We present REM, a framework for segmenting a wide range of concepts in video that can be described through natural language by capitalizing on powerful visual-language representations learned by video diffusion models. REM generalizes with ease to challenging, dynamic concepts, such as raindrops or shattering glass, shown above. Video visualizations are available [here](https://miccooper9.github.io/projects/ReferEverything/#REM).
Refer to caption: x1.png

The corresponding problem in computer vision - Referral Video Segmentation (RVS) (Gavrilyuk et al., ; Hu et al., ), is defined as the task of segmenting a specific region in a video based on a natural language description.
However, virtually all existing benchmarks and methods focus on a specific subset of RVS - Referral Video Object Segmentation (RVOS) (Seo et al., ; Wu et al., ), where the goal is to track and segment the object referenced by a given expression.
Why has the field concentrated so narrowly on this task? Although multiple factors contribute, we argue that the primary reason lies in the data. Historically, RVOS datasets have been developed by adding referral expression annotations to existing object tracking benchmarks (Pont-Tuset et al., ; Xu et al., ), which are inherently object-centric and limited in scale.

At the same time, recent advances in Internet-scale datasets with billions of paired image- and video-language samples (Schuhmann et al., ; Bain et al., ) have opened new possibilities. These datasets have been used to train powerful denoising diffusion models (Rombach et al., ; Wang et al., ), and provide excellent representations of the natural visual-language space.
In the image domain, numerous studies have shown that re-purposing diffusion models can yield highly generalizable representations of object shapes (Zhao et al., ; Ozguroglu et al., ). Very recently, Zhu et al. () explored the application of video diffusion models for referral segmentation, but their approach exhibited limited generalization capabilities.

In this work, we introduce a novel approach to RVS that leverages large-scale video-language representations learned by diffusion models. Our method, shown in Figure 3, enables spatio-temporal localization of a wide range of concepts in video that can be described through natural language, beyond conventional object tracking. A key factor behind our approach’s success is preserving the rich representations learned by the generative model (see Figure 2). To this end, we retain the original model architecture and fine-tune it on existing referral image- and video-segmentation datasets, adjusting the output to generate object masks instead of Gaussian noise. As shown in Section 5.1, our model demonstrates competitive performance against state-of-the-art models on RVOS benchmarks. More significantly, it exhibits a much stronger generalization.

To quantify this effect, we report results on the open-world object tracking benchmark - BURST (Athar et al., ), and the non-object ‘Stuff’ segmentation dataset - VSPW (Miao et al., ), as well as collect a new benchmark that focuses on dynamic process in Section 4. We define the latter as temporally evolving events, where the subjects undergo continuous changes in state, shape, or appearance (see examples in Figure 1).
Our new benchmark, which we call Ref-VPS for Referral Video Process Segmentation, consists of 141 videos that are labeled with referral expressions and masks at 24 fps and span 39 unique concepts. Our experiments in Section 5.2 demonstrate that existing approaches fail to generalize to outside of the narrow training distribution, whereas our method effortlessly segments a wide spectrum of targets as shown in Figures 1 and 4.

Crucially, our approach exhibits a relative improvement of up to 32% compared to the very recent method of Zhu et al. (), which is also based on a video-diffusion representation. We investigate this in Section 5.3 and demonstrate that preserving as much of the representation learned during generative pre-training as possible is key to achieving the highest degree of generalization in referral video segmentation. We will release the code, models, and data for reproducing our results.

## 2 Related Work

Figure: Figure 2: Through Internet-scale pre-training, video diffusion models can generate realistic videos capturing the entire diversity of the dynamic visual world (generated samples shown above). We leverage their powerful visual-language representation for open-world referral video segmentation .
Refer to caption: x2.png

Referring Video Segmentation (RVS) involves segmenting specific regions in a video based on a natural language description (Gavrilyuk et al., ; Khoreva et al., ; Seo et al., ). Most benchmarks for this task were developed by adding referral expression annotations to existing Video Object Segmentation (VOS) datasets, such as DAVIS’17 (Pont-Tuset et al., ) or YouTube-VOS (Xu et al., ). Consequently, the role of language in these benchmarks is limited to providing an interface for user-initialized object tracking (Wu et al., ; Perazzi et al., ). While this specific task — Referral Video Object Segmentation (RVOS) — is valuable, it addresses only a narrow subset of the possible interactions between language and the space-time continuum of videos.
Equally important is the ability of RVS methods to segment video concepts beyond common object categories.
To address this gap, we introduce a new benchmark focused on segmenting dynamic processes, which we term Referral Video Process Segmentation (Ref-VPS).

Earlier RVOS approaches (Bellver et al., ; Ning et al., ; Hui et al., ) generally employed a *bottom-up* strategy: first, image-level methods (Rother et al., ; Ye et al., ; Carion et al., ; Plummer et al., ) were applied to obtain frame-level masks, followed by spatio-temporal reasoning, such as mask propagation (Seo et al., ), to refine the segmentation across frames. More recently, with the success of cross-attention-based methods  (Vaswani, ; Meinhardt et al., ; Zeng et al., ) in object segmentation and tracking, query-based architectures have been introduced to RVOS, leading to significant improvements, with ReferFormer (Wu et al., ) and MUTR (Yan et al., ) being notable examples. The limited scale of paired video-language data with segmentation annotations has always been a major limitation in RVOS, causing most methods to train jointly on video and image samples (Kazemzadeh et al., ; Jhuang et al., ). The latest approaches go even further and unify all object localization datasets and tasks in a single framework to maximize the amount of training data (Yan et al., ; Wu et al., ; Cheng et al., ). However, while these models excel in object tracking, they struggle to generalize to more dynamic concepts. In contrast, we demonstrate that generative video-language pre-training on Internet-scale data (Schuhmann et al., ; Bain et al., ) results in a universal mapping between the space of language and the ever-changing visual world.

Diffusion Models have become the de-fact standard for generative learning in computer vision (Sohl-Dickstein et al., ; Ho et al., ) and beyond (Chi et al., ). Among them, the Denoising Diffusion Probabilistic Model (DDPM) (Ho et al., ) leverages neural network components to model the denoising process and builds a weighted variational bound for optimization. Stable Diffusion (SD) (Rombach et al., ) shifts the denoising process into the latent space of a pre-trained autoencoder (Kingma & Welling, ), allowing for model scaling. Expanding from images to videos, diffusion models have seen success in text-to-video (T2V) generation (Wang et al., ; Chen et al., ; ; Zheng et al., ; Blattmann et al., ). In addition to the capacity to generate high-fidelity images based on text prompts, the T2V diffusion models implicitly learn the mapping from linguistic descriptions to video regions, providing an opportunity to repurpose them for RVOS. Among current T2V methods, ModelScope (Wang et al., ) and VideoCrafter (Chen et al., ; ) stand out for their open-source implementations, forming the backbone of our research.

Visual-language Pre-training for Perception: in addition to being highly effective in image and video generation, diffusion models have been shown to learn a strong representation of the natural image manifold. Several works have demonstrated that these representations can be re-purposed for classical computer vision problems, including semantic segmentation (Xu et al., ; Zhao et al., ; Zhang et al., ) and pixel-level correspondence (Tang et al., ), achieving an impressive degree of generalization. Others have shown that image diffusion models learn powerful representations of objects, enabling open-world novel view synthesis (Liu et al., ) and amodal segmentation (Ozguroglu et al., ).
Most recently, Zhu et al. () also leverages pretrained T2V models for RVOS, however, our analysis shows that their approach fails to fully capitalize on the universal visual-language mapping learned in generative pre-training.
In this work, we explore the application of video diffusion models to RVS,
demonstrating how to maintain a high-level generalizability during fine-tuning.

In a separate line of work, visual-language representations learned with contrastive objectives (Bao et al., ; Radford et al., ) have been adapted for referring image (Lai et al., ; Rasheed et al., ; You et al., ; Xu et al., ) and video segmentation (Zhou et al., ). However, their performance remains limited, compared to both generative models, as well as classical referral segmentation approaches.

## 3 Method

### 3.1 Learning the visual-language manifold via video denoising

Text-to-Video (T2V) diffusion models (Wang et al., ; Chen et al., ; Zheng et al., ) generate videos that align with a given language description, starting from Gaussian noise. The process can be formalized as:

$$ $\hat{x}=f_{\text{vdm}}(x_{T},c,T),$ (1) $$

where $\hat{x}$ is the generated video, $T$ denotes the maximum timestep specified by the video diffusion model $f_{\text{vdm}}$, $x_{T}$ is a sample drawn from a Gaussian distribution $\mathcal{N}(\mu_{T},\sigma_{T}^{2})$ predefined by the video diffusion model, and $c$ is the conditioning prompt. To reduce computational complexity, these models often perform denoising in the latent space (Rombach et al., ). Specifically, a pretrained Variational Autoencoder (VAE) (Kingma & Welling, ) is employed to map the video $x$ from pixel space into latent space, denoted as $\mathcal{E}(x)=z$, while a decoder reconstructs it from the latent space, $\mathcal{D}(z)\approx x$. Thus, the generation process becomes:

$$ $\hat{x}=\mathcal{D}\left(f_{\text{vdm}}(z_{T},c,T)\right).$ (2) $$

During training, rather than denoising from pure Gaussian latents, T2V models denoise from partially noisy video latents and optimize the following latent diffusion objective:

$$ $\min_{\theta}\mathbb{E}_{z\sim\mathcal{E}(x),t,\epsilon\sim\mathcal{N}(0,1)} \left\|\epsilon-\epsilon_{\theta}(z_{t},e_{c},t)\right\|_{2}^{2},$ (3) $$

where $\epsilon$ is the Gaussian noise added to the clean video latent, $z_{t}$ represents noisy video latent at timestep $t$ derived by the diffusion forward pass (Ho et al., ; Rombach et al., ), and $e_{c}$ is the conditional embedding generated from $c$ using a text encoder, such as CLIP (Radford et al., ). The denoising network $\epsilon_{\theta}(z_{t},e_{c},t)$, typically a U-Net (Ronneberger et al., ), is tasked with predicting the noise $\epsilon$. In this network, the conditional embedding $e_{c}$ interacts with the latent representations through cross-attention mechanisms, guiding the model to generate diverse, semantically accurate videos based on the provided text descriptions.

### 3.2 From language-conditioned denoising to referral video segmentation

Referral Video Segmentation (RVS) involves segmenting an entity in a video across spatial and temporal dimensions, guided by a natural language description. Formally, the task is defined as:

$$ $\hat{m}=f_{\text{RVS}}(x,c),$ (4) $$

where $f_{\text{RVS}}$ is the RVS model, $x$ represents a video sequence, $c$ denotes a referral text prompt, and $\hat{m}$ corresponds to the binary masks produced as output. This task aligns naturally with T2V models, which establish a robust mapping between the entities described in the text and the corresponding spatial-temporal regions in the video by optimizing the denoising objective in Equation 3.

Figure: Figure 3: The model architecture of Refer Everything with Diffusion Models (REM). Like a video diffusion model it is based on, our approach takes video frames with added noise and a language expression as input. Our key insight is preserving as much of the diffusion representation intact as possible by supervising segmentation masks in the latent space of the frozen VAE.
Refer to caption: x3.png

Several prior works have explored the alignment of diffusion models with referral segmentation (Zhao et al., ; Xu et al., ; Zhu et al., ), typically employing these models as feature extractors. Specifically, they adjust the input format of a referral segmentation model to match that of the denoising network $\epsilon_{\theta}$, and pass the resulting features to a task-specific decoder $f_{\text{dec}}$ (*e.g.*, a convolutional network) to predict the target masks:

$$ $\hat{m}=f_{\text{dec}}(\epsilon_{\theta}^{(n)}(z_{t},e_{c},t)),$ (5) $$

where $z_{t}$ is the noisy latent representation of the input images at timestep $t$, $e_{c}$ is a feature embedding of the referral expression $c$, and $\epsilon_{\theta}^{(n)}$ denotes the intermediate feature at the $n^{th}$ layer. In practice, $t$ is usually set to a small value (*e.g.*, 50), and $n$ is set to the last layer indexes to obtain the optimal performance. The entire model is then trained in a conventional discriminative learning setup.
However, replacing parts of the generative model with newly initialized layers can disrupt alignment between the model’s representation from pre-training and the new features learned on narrow-domain datasets, leading to a substantial loss of generalization capabilities.

In our approach, shown in Figure 3, we propose to instead preserve the architecture (and thus the representation) of the video diffusion model in its entirety. Specifically, rather than using intermediate features $\epsilon_{\theta}^{(n)}$, REM repurposes the whole denoising network $\epsilon_{\theta}$ (together with the VAE) by shifting its objective from predicting noise to predicting mask latents (shown on the right in Figure 3):

$$ $\hat{m}=\mathcal{D}(\epsilon_{\theta}(z_{t},e_{c},t)),$ (6) $$

where $\mathcal{D}$ denotes the (frozen) VAE decoder used to produce the actual binary segmentation masks from the predicted latents. That is, instead of learning the decoder network $f_{\text{dec}}$ from scratch, we reuse the VAE from the video diffusion model.
This subtle yet powerful modification allows the model to better preserve its representation learned on Internet-scale data during generative pretraining while adapting to the task of RVS.

Training and Optimization. During training, to encode the ground-truth segmentation masks with the VAE, we broadcast the single-channel mask into three channels by simply duplicating it (shown in the top right of Figure  3). For simplicity, we still denote this three-channel mask representation as $m$. The pretrained VAE can then map the mask sequence into the latent space via $\mathcal{E}(m)=z^{m}$ and decode the masks back from predicted latents via $\mathcal{D}(z^{m})\approx m$.
For the noisy latent $z_{t}$ and timestep $t$, we prioritize using latents that remain as clean as possible. Therefore, we always set the timestep to its minimum value, $t=0$. To train the model, we supervise the predicted mask latents using an $\mathcal{L}_{2}$ loss (shown in the center-right of Figure 3) by minimizing:

$$ $\min_{\theta}\mathbb{E}_{z^{m}\sim\mathcal{E}(m),t=0}\left\|z^{m}-\mathbf{ \epsilon}_{\theta}(z_{t},e_{c},t)\right\|_{2}^{2}.$ (7) $$

Model Inference. During inference, we follow Equation 6, with $t=0$, to decode the predicted mask latent and generate three-channel mask predictions. We then compute the single-channel masks by averaging the pixel values of the three channels and applying a constant threshold of 0.5 to binarize the result (as shown in the bottom right of Figure 3).

## 4 Benchmark design and collection

Existing datasets only allow to quantify the generalization of RVS models to rare object categories (Athar et al., ) or static ‘Stuff’ (Miao et al., ). To present a more comprehensive generalization evaluation, we now discuss our approach to collecting a new benchmark. As covering the entire spectrum of concepts that can be spoken of in videos would be extremely costly, we seek to identify the most salient subset of the problem that requires joint modeling of language and temporal dynamics. Specifically, we target dynamic processes, defined as temporally evolving events, where the subjects undergo continuous changes in state, shape, or appearance. Crucially, the subjects in this context are not limited to objects, but include all concepts that are spatio-temporally localizable in videos, such as light or fire. The key steps for collecting this new benchmark, which we call Referral Video Process Segmentation (Ref-VPS), include selecting representative videos and annotating them with referring expressions and segmentation masks.

### 4.1 Video Selection

To source the videos for our benchmark we require a large, public, and diverse database that is queriable with natural language and allows re-distribution of content for research purposes. Based on these requirements, we choose the [TikTok](https://www.tiktok.com) social media platform which has over 1 billion active users across the world and receives tens of millions of video uploads daily, capturing a wide range of dynamic visual content. TikTok’s policies generally allow for free redistribution of content, with individual users having the option to opt-out.

To search for videos that capture dynamic processes, as defined above, we first identify a non-exhaustive list of six broad and possibly overlapping concepts (*e.g.*, ‘object transformations’, or ‘entities with dynamic boundaries’, the full list together with definitions provided in Section A in the appendix). Then, for each concept we ask ChatGPT (OpenAI, ) to provide a list of concrete examples together with multiple text queries for search on TikTok (*e.g.*, ’a wax candle melting’ for ’object transformations’), resulting in 120 individual concepts. We retrieved over 1,000 samples based on these queries, however, a majority of the queries did not yield suitable videos because of the physical nature of the event or ambiguity of the search query not lending itself to being accurately captured on TikTok. After removing irrelevant videos, the retrieved set is reduced to 342 samples.

We then manually filter these videos based on the following criteria: (1) videos that do not feature significant dynamic changes of the subject (*e.g.*, mostly stationary clouds in the sky); (2) dynamic processes that occur too rapidly to allow for the labeling of a sufficient number of non-empty frames (*e.g.*, flashes of lightning); (3) video with frequent shot changes, which make it impossible to extract an interrupted clip capturing the event of interest. Additionally, for videos that represent compilations of similar events, we split them into individual clips and treat each one independently. The resulting dataset contains 141 video clips representing 39 dynamic process concepts. The entire dataset is intended for zero-shot evaluation, so we do not define any additional splits. Representative samples of the videos are shown in Figure 1.

### 4.2 Annotation collection and evaluation

To label the videos selected above, we begin by adjusting the temporal boundaries of each clip to focus on the event of interest and avoid shot changes. We also make sure that the event is captured in its entirety whenever possible, including some context before and after it. The clips are then exported at 24 FPS as frames. If a video contains irrelevant frames, such as the TikTok logo at the end, we crop the frames accordingly to remove the padding.

To collect referral expressions, we first manually identify the entity of interest in each clip. The selected entity is then labeled with referral expressions by two independent annotators. Each annotator provides two expressions for the target, resulting in a total of four expressions per clip, capturing different ways to describe the same phenomenon. Following the standard protocol (Khoreva et al., ; Seo et al., ), models are evaluated on all queries and the results are averaged.

Finally, we densely label the targets identified above with segmentation masks at 24 FPS. To this end, we employ a semi-automatic pipeline, capitalizing on the recently introduced SAM2 (Ravi et al., ) foundational model for interactive video segmentation. In particular, we provide positive and negative click annotation in the middle frame of a video first to ensure accurate boundary segmentation. SAM2 then automatically segments the entity of interest in the frame, as well as propagates the mask across the entire clip. We interactively improve segmentation quality by providing additional clicks as needed. In the end, we manually refine the masks in frames where SAM2 fails. We report the statistics of our benchmark in the appendix. For evaluation, we follow Tokmakov et al. () and only report region similarity $\mathcal{J}$ as contour accuracy $\mathcal{F}$ is often not well defined for the entities like smoke or light which are frequent in Ref-VPS.

## 5 Experiments

Datasets and Evaluation.
We evaluate our method on five benchmarks in total. Ref-YTB (Seo et al., ) and Ref-DAVIS (Khoreva et al., ) are standard RVOS benchmarks for evaluating our model’s performance on object tracking. The evaluation on Ref-YTB is done on the official challenge server, and Ref-DAVIS is evaluated using the official evaluation code. For evaluating generalization to rare objects and ‘Stuff’ categories, we use the BURST (Athar et al., ) and VSPW (Miao et al., ) datasets respectively.
Finally, we evaluate REM and the strongest baselines on our newly introduced Ref-VPS benchmark that focuses on dynamic process segmentation (detailed in Section 4). All these datasets except Ref-YTB are only used for evaluation (*i.e.*, the results are zero-shot).

For Ref-YTB (Seo et al., ) and Ref-DAVIS (Khoreva et al., ) we use the standard evaluation metrics - Region Similarity ($\mathcal{J}$), Contour accuracy ($\mathcal{F}$) and their mean ($\mathcal{J\&F}$). For all other evaluations we use the Region Similarity ($\mathcal{J}$) metric.

Implementation details. We build our method upon ModelScopeT2V (Wang et al., ). We adopt a two-stage training strategy following prior work (Zhu et al., ). In the first stage, we fine-tune only the spatial weights using image-text samples from Ref-COCO (Yu et al., ) for 1 epoch and then fine-tune all weights for 40 epoch using Ref-YTB (Seo et al., ) video-text training samples and 12k samples from Ref-COCO jointly. In the second stage, the image samples from Ref-COCO are converted to pseudo videos through augmentations following Wu et al. (). We freeze the CLIP encoder and VAE during training.

More details about the dataset, evaluation details on BURST and VSPW, and implementation are included in Section C in the appendix.

### 5.1 Referral Video Object Segmentation Results

In this section, we compare REM to the state of the art on the standard RVOS benchmarks. We report results on the validation set of Ref-DAVIS (Khoreva et al., ) and the test set of Ref-YTB (Seo et al., ) in Table 1. Our method outperforms state of the art in terms of $\mathcal{J}$ on Ref-DAVIS and is only second to UNINEXT (Yan et al., ) on Ref-YTB. Note that this approach is specifically designed for object segmentation and utilizes more than 10 datasets with localization annotations like bounding boxes and masks for training. In contrast, REM adopts an architecture of a video generation model and is only fine-tuned on one image- and one video-segmentation dataset. Despite this, our method is competitive with UNINEXT on standard RVOS benchmarks, and we will show next, outperforms it by up to 21 points out-of-domain in terms of $\mathcal{J}$.

Another notable observation is that REM also outperforms VD-IT (Zhu et al., ), which is built on top of the same video diffusion backbone of Wang et al. (), on both datasets. This result demonstrates the effectiveness of our approach to preserving the visual-language representations learned on the Internet data, which will become even more evident in out-of-domain evaluation.

**Table 1: Comparison to the state of the art on the validation set of the Ref-DAVIS and the test set of Ref-YTB benchmarks using the standard metrics. Our method performs on par with the strong UNINEXT approach, despite not being specifically designed for object localization and having access to only a fraction of the localization labels used by that method.**
| Method | Pretraining Data | Mask/Box Supervision | Ref-DAVIS | Ref-YTB |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ |  |  |  |
| Referformer (Wu et al., ) | ImageNet + Kinetics + SSv2 | Ref-COCO/+/g + Ref-YTB | 61.1 | 58.1 | 64.1 | 62.9 | 61.3 | 64.6 |
| MUTR (Yan et al., ) | ImageNet + Kinetics + SSv2 | Ref-YTB + AVS | 68.0 | 64.8 | 71.3 | 68.4 | 66.4 | 70.4 |
| VLMO-L (Zhou et al., ) | Unknown | Ref-COCO/+/g + Ref-YTB | 70.2 | 66.3 | 74.1 | 67.6 | 65.3 | 69.8 |
| UNINEXT (Yan et al., ) | Object365 | 10+ Image/Video datasets | 72.5 | 68.2 | 76.8 | 70.1 | 67.6 | 72.7 |
| VD-IT (Zhu et al., ) | LAION5B + WebVid | Ref-COCO/+/g + Ref-YTB | 69.4 | 66.2 | 72.6 | 66.5 | 64.4 | 68.5 |
| REM (Ours) | LAION5B + WebVid | Ref-COCO/+/g + Ref-YTB | 72.6 | 69.9 | 75.29 | 68.4 | 67.05 | 69.73 |

Figure: Figure 4: Qualitative results of REM and state of the art baselines on BURST, VSPW and Ref-VPS benchmarks. Our method demonstrates both superior coverage of rare, dynamic concepts and higher segmentation precision. Video comparisons are available [here](https://miccooper9.github.io/projects/ReferEverything/#dataset).
Refer to caption: x4.png

### 5.2 Out-of-domain generalization

We begin by performing a generalization study on existing open-world tracking BURST dataset (Athar et al., ) as well as on the ‘Stuff’ categories (Caesar et al., ) from VSPW (Miao et al., ) in Table 3. BURST is an open-world video object segmentation benchmark, whereas VSPW tests the ability to generalize to non-object categories. We report zero-shot evaluation results on the validation set of VSPW and combined validation and test sets of BURST and compare to the top performing methods from Table 1 that have public models.

Firstly, we observe that on both out-of-domain challenges our method outperforms all the baselines by significant margins. The improvements are especially noticeable on BURST, demonstrating that REM successfully preserves the strong object representation learned by Internet-scale pre-training of the diffusion backbone. In contrast, VD-IT loses this generalization capacity during fine-tuning and only performs on par with UNINEXT. On the ‘Stuff’ categories all the methods do relatively poorly, reflecting the challenge of generalizing to more amorphous ‘Stuff’. Here VD-IT maintains a lead over entirely object-centric UNINEXT but REM still outperforms both baselines.

**Table 2: $\mathcal{J}$ Comparison to state of the art on the ‘Stuff’ categories in the eval set of VSPW and on the joint val and test sets of BURST. Our approach demonstrates much stronger generalization, notably, outperforming VD-IT which is based on the same diffusion backbone.**
| Benchmark | MUTR | UNINEXT | VD-IT | REM (Ours) |
| --- | --- | --- | --- | --- |
| VSPW | 10.5 | 10.1 | 12.7 | 15.2 |
| BURST | 27.9 | 30.2 | 30.9 | 40.4 |

Finally, we compare REM to the top-performing RVS baselines on our Ref-VPS benchmark in Table 3. Here the differences between the methods are more pronounced, highlighting the value of Ref-VPS in assessing video-language understanding capabilities of neural representations. Our approach outperforms all baselines by up to 12 points in Region Similarity (32% relative improvement), and notably surpasses the top RVOS method, UNINEXT, by 21.2 points (74.8% relative improvement). While generative pre-training enhances VD-IT’s generalization ability compared to UNINEXT, it struggles to preserve its representations as effectively as REM.

A qualitative comparison of REM with VD-IT and UNINEXT is provided in Figure 4. Firstly, we can see that our approach can successfully track the sponge (which was never seen in training) in a challenging sequence from BURST, whereas other methods focus on foreground objects. In the second sequence from VSPW REM successfully generalizes to the non-object ‘wall’ category, whereas UNINEXT focuses on a nearby object and VD-IT fails entirely. The following examples from our Ref-VPS illustrate that both baselines exhibit object-centric bias, as in the examples with the lizard skin in row 3 and blue smoke in row 6. While VD-IT shows better generalization, it often simply segments the dominant region in the video (see rows 4 and 7 in Figure 4). In contrast, REM demonstrates both good coverage of rare concepts and high precision with respect to the language prompt. See more examples of highly dynamic sequences in Section B.1 in the appendix.

### 5.3 Ablation analysis

In this section, we analyze our proposed approach of transferring generative representations to the task of RVS. We report results on one representative RVOS benchmark (Ref-YTB) and on our new Ref-VPS. Note that for efficiency we fine-tune all the models on a subset of image and video data (12000 samples) so the numbers are lower than those reported in the previous section.

Generative pre-training. We begin by evaluating the effect of the generative pre-training strategy in Table 4. Firstly, we design a frame-level baseline that fine-tunes StableDiffusion (Blattmann et al., ) on every frame individually (row 1 in the table). While this variant has no temporal modeling capacity, its architecture is similar to UNINEXT (Yan et al., ) - the state-of-the-art approach for RVOS. Interestingly, it strongly under-performs compared to our best video-based variant not only on our Ref-VPS but also on the object-centric Ref-YTB benchmark. These results demonstrate that, despite the fact that images are the dominant data source in generative pre-training, fine-tuning StableDiffsuion for video generation is crucial for learning an effective representation for tracking.

Next, we compare several strategies for learning video diffusion models. We begin by studying two variants of the VideoCrafter model (Chen et al., ; ) (denoted as VideoCrafter-1 and VideoCrafter-2 in Table 4). They are both trained on 600M images from LAION (Schuhmann et al., ) and 10-20M Internet videos. However, VideoCrafter-2 is further tuned to increase the quality of the generated samples. Our findings indicate that this fine-tuning step leads to significant performance gains across both benchmarks. This suggests that improving the quality of video generation models can directly translate to enhanced performance in our video segmentation framework.

Finally, we evaluate ModelScope (Wang et al., ), which is trained on larger LAION 2B and a comparable amount of video samples (last row in Table 4). This model delivers performance comparable to the best version of VideoCrafter on the Ref-YTB benchmark, while demonstrating superior generalization to more challenging concepts in Ref-VPS. These results further highlight that both large-scale pre-training on image data as well as learning to model video-language interactions are crucial components for robust RVS representation learning.

**Table 4: Analysis of the effects of generative pre-training and discriminate fine-tuning strategies on Ref-YTB and Ref-VPS. The key to the success of REM is capitalizing on Internet-scale image and video pre-training and preserving as much of this representation as possible.**
| Backbone | Decoder | Ref-YTB | Ref-VPS |
| --- | --- | --- | --- |
| ($\mathcal{J}\&\mathcal{F}$) | $\mathcal{J}$ |  |  |
| Stable Diffusion 2.1 | Frozen VAE | 59.38 | 30.69 |
| VideoCrafter-1 | 59.10 | 29.22 |  |
| VideoCrafter-2 | 65.00 | 37.28 |  |
| ModelScope T2V | 64.57 | 39.05 |  |
| ModelScope T2V | CNN | 59.6 | 29.93 |

Fine-tuning strategy. We now ablate the effectiveness of our design decision to re-use a frozen VAE decoder for mask prediction, rather than replacing it with a dedicated mask prediction module, as was done in some of the prior work (Zhao et al., ; Zhu et al., ). To this end, we replace the VAE with a CNN mask decoder adopted from (Zhao et al., ) and train it jointly with the rest of the model (last row in Table 4). Removing the pre-trained VAE decoder has a moderate negative effect on performance on Ref-YTB, but, notably, destroys the model’s ability to generalize our challenging Ref-VPS benchmark. This result underscores the main message of our paper - preserving as much of the representation learned during generative pre-training is key for achieving generalization in referral video segmentation.

## 6 Discussion

In this paper, we proposed REM, a framework that capitalizes on Internet-scale video-language representations learned by diffusion models to segment a wide range of concepts in
video that can be described through natural language. Our key insight is that changing as little as possible in the representation is key to preserving its universal mapping between language and visual concepts during fine-tuning. We have evaluated the generalization capabilities of our approach on existing datasets for open-world object and stuff segmentation, and also collected Ref-VPS - a new benchmark for referral segmentation of dynamic processes in videos. Our extensive experimental evaluation demonstrates that, despite only being trained on object masks, REM successfully generalizes to unseen object and non-object concepts, outperforming all prior work by up to 12 points in terms of region similarity.

Despite REM’s impressive generalization abilities, the problem of RVS is far from being solved. In Figure C in the Appendix we visualize a few failure cases of our method. REM still exhibits some object-centric bias and struggles with extremely fast changes. Exploring ways to preserve even more of the representation learned during generative pre-training, *e.g.*, via low-rank adaptation (Hu et al., ) of the visual backbone, is a very promising direction to address some of these issues. In addition, note that REM is a generic framework where the backbone of  Wang et al. () can be replaced with a more advanced representation, tracing the progress of language-conditioned video generative models.

## Appendix A Dataset Details

During our dataset collection, we first identify a non-exhaustive list of six broad and possibly overlapping concepts. The list of the concepts and their definitions are included below:

- •
Temporal object changes: Concepts involving changes over time (*e.g.*, object deformation, melting)
- •
Motion Patterns: Concepts involving movement and displacement of non-object regions (*e.g.*, water ripples, flickering flames)
- •
Dynamic environmental changes: Changes in the environment that affect spatial regions over time (*e.g.*, clouds moving across the sky, waves rising )
- •
Interaction Sequences: Concepts involving interactions between objects (*e.g.*, bullet hitting glass, object collisions)
- •
Pattern evolution: Concepts where patterns or textures evolve or change dynamically (*e.g.*, changing patterns of smoke dispersion, fluctuating light levels)

The final dataset contains 141 video clips representing 39 dynamic process concepts. We report a comprehensive list of key statistics in Table A. Most of our samples are around 2.5 to 5 secs in length but can go up to more than 20 seconds. The distribution of our sample lengths is reported in Figure A.

## Appendix B Additional Experimental Evaluations

### B.1 Evaluation on Challenging Fight Scenes

Fight sequences in movies, television, and animated shows pose a unique set of challenges. Typically fight scenes are characterized by objects/characters undergoing severe and frequent occlusions and leaving the frame entirely, coupled with frequent pose changes of the camera. This leads to drastic changes in the appearance of the object and requires high levels of temporal and semantic consistency to accurately track, re-identify, and segment the referred entity. Our diffusion fine-tuning method excels in this domain of extremely challenging samples as illustrated in Figure B. It is clear to find that UNINEXT and VD-IT both fail whenever there is a large occlusion causing the referred entity to become invisible. Even though VD-IT uses frozen Video diffusion features, their method is unable to leverage the temporal consistency learned during Video Diffusion pre-training as well as our method.

### B.2 Failure Cases

A few representative failure cases of REM on Ref-VPS are shown in Figure C. Our method suffers from object-centric bias in the most challenging scenarios and struggles with extremely fast processes.

**Table A: Statistics of our Ref-VPS benchmark. Our dataset contains 141 video clips covering 39 concepts for dynamic processes.**
| Clips | 141 |
| --- | --- |
| FPS | 24 |
| Frames | 22831 |
| Concepts | 39 |
| Avg length (s) | 6.75 |
| Annotation FPS | 24 |
| Min-resolution | $712\times 576$ |
| Max-resolution | $1024\times 576$ |

Figure: Figure B: Qualitative comparison of REM with state-of-the-art baselines on dynamic and challenging fight scenes. The incorrectly labeled frames are outlined in red. Our method outperforms the other methods in handling frequent occlusions and POV changes. For a better illustration of the differences, please watch the full videos [here](https://miccooper9.github.io/projects/ReferEverything/#fight).
Refer to caption: x5.png

Figure: Figure C: Failure cases of REM on Ref-VPS. Our model still exhibits some object-centric bias and struggles with extremely dynamic entities such as lightning.
Refer to caption: x6.png

### B.3 Concept Coverage Plot on BURST Dataset

Figure: Figure D: Class-wise $\mathcal{J}$ scores (mIoU) concept coverage on BURST. As indicated by the arrows, Our method is robust on the long-tail region compared to other methods.
Refer to caption: x7.png

We additionally show the concept coverage on BURST (Athar et al., ) dataset among VD-IT (Zhu et al., ), MUTR (Yan et al., ), UNINEXT (Yan et al., ), and ours in Figure D. In general, our model has significantly better coverage of different object concepts with a performance improvement of at least 9.5 %. Moreover, our REM is more robust in the long-tail regions, indicating the promising generalization capacity of our method by repurposing video diffusion models. The performance improvement compared to VD-IT, which uses frozen video diffusion features, indicates that simply freezing the diffusion model as a backbone does not guarantee the transfer of the diffusion knowledge to downstream tasks and verifies the design of our approach.

## Appendix C Implementation Details

### C.1 Dataset Details

We report the details about the benchmarks we used in Table B.

**Table B: Details about the benchmarks we used for training and evaluation.**
| Benchmark | Type | Training Samples | Testing Samples |
| --- | --- | --- | --- |
| Ref-COCO (Yu et al., ) | Image | 320K | - |
| Ref-YTB (Seo et al., ) | Video | 12,913 | 2,096 |
| Ref-DAVIS (Khoreva et al., ) | Video | - | 90 |
| BURST (Athar et al., ) | Video | - | 2,049 |
| VSPW (Miao et al., ) | Video | - | 343 |

### C.2 Stuff Category Evaluation

Neither BURST (Athar et al., ) nor VSPW (Miao et al., ) contains referral text for the segmented entities. For them, we automatically generate referral expressions using only the category information of the mask entity as ”the $<$class$>$” (*e.g.*, ”the hat”). For VSPW we conduct our evaluation on the validation set which has 66 different stuff categories. In the case of BURST, we evaluate the combined validation and test set which contains 454 classes and a total of 2049 sequences.

### C.3 Baseline Models

In addition to Table 1 in the main paper, we report the comprehensive list of bounding/mask supervision used by all methods in Table C. We quote the results of all the baseline models on Ref-YTB and Ref-DAVIS from their original paper. For the evaluation of BURST, VSPW, and our dataset, we report the numbers by running the official checkpoints of MUTR(^1^11[https://github.com/OpenGVLab/MUTR](https://github.com/OpenGVLab/MUTR)), UNINEXT(^2^22[https://github.com/MasterBin-IIAU/UNINEXT](https://github.com/MasterBin-IIAU/UNINEXT)), and VD-IT(^3^33[https://github.com/buxiangzhiren/VD-IT](https://github.com/buxiangzhiren/VD-IT)).

**Table C: Comprehensive list of bounding/mask supervision used by all methods.**
| Method | Mask/Box annotations | Ref-Davis | Ref-YTB |
| --- | --- | --- | --- |
| ($\mathcal{J}\&\mathcal{F}$) | ($\mathcal{J}\&\mathcal{F}$) |  |  |
| Referformer | RefCOCO/g/+, Ref-Youtube-VOS | 61.1 | 62.9 |
| MUTR | Ref-Youtube-VOS, AVS | 68.0 | 68.4 |
| VLMO-L | RefCOCO/g/+, Ref-Youtube-VOS | 70.2 | 67.6 |
| UNINEXT | Objects365, COCO, RefCOCO/g/+, GOT-10K, LaSOT, TrackingNet, Youtube-VOS, BDD100K, VIS19, OVIS, Ref-Youtube-VOS | 72.5 | 70.1 |
| VD-IT | RefCOCO/g/+, Ref-Youtube-VOS | 69.4 | 66.5 |
| REM (Ours) | RefCOCO/g/+, Ref-Youtube-VOS | 72.6 | 68.4 |

### C.4 Training Details

We build our method upon ModelScopeT2V (Wang et al., ). We adopt a two-stage training strategy following prior work (Yan et al., ; Wu et al., ). In the first stage, we fine-tune only the spatial weights using image-text samples from Ref-COCO (Yu et al., ) for 1 epoch and then fine-tune all weights for 40 epoch using Ref-YTB (Seo et al., ) video-text samples and 12k samples from Ref-COCO jointly. In the second stage, the image samples from Ref-COCO are converted to pseudo videos through augmentations following Wu et al. (). We resize all the training data to $512\times 512$ for training and evaluate under the original resolution. We freeze the CLIP encoder and VAE during training. We use AdamW (Loshchilov et al., ) for optimization with a constant learning rate of 1e-6 in both stages. The training batch size is 4, and for each sample, we randomly load a 8-frame video clip. We train our model using 4 NVIDIA 80GB A100 GPUs, and it takes about 1 week to finish the whole training process.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: * How does REM handle ambiguous or overlapping concepts (e.g., scenes where multiple dynamic elements, like water and smoke, interact)?

* How critical is the quality of synthetic data for REM’s generalization across diverse concepts?
soundness: 3
strengths: * REM achieves good generalization to out-of-domain concepts and non-object entities by leveraging Internet-scale pretraining on video-language data, outperforming specialized methods.

* The Ref-VPS benchmark provides a way to evaluate models on dynamic processes, filling a gap in existing RVS datasets by focusing on temporal events and continuous appearance changes.
summary: The REM paper introduces a novel framework for Referral Video Segmentation (RVS) that uses a video-text diffusion model to generalize segmentation to a wide range of visual concepts.REM effectively segments both common and rare objects, as well as non-object dynamic concepts like smoke, water, or changing light. The paper also introduces Ref-VPS, a new segmentation benchmark on dynamic processes that require both spatial and temporal modeling.
weaknesses: * The REM method performs comparably to MUTR  and only slightly better then VLMO on Ref-YTB.

* Given use of a high-capacity diffusion model, there is limited discussion on its computational efficiency or inference speed. A comparison of REM's computational requirements with prior methods would be beneficial.

* Analysis on the impact of the quantity and quality of synthetic data would be useful - whether reducing/increasing the amount of synthetic data affect REM's performance.

* Since REM is applied to video segmentation tasks, especially focusing on continuous changes, it would be great to have some evaluation of temporal consistency.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: In order to match the pre-training input distribution of the model, the authors add noise to the input (as performed during pre-training).
However, it corrupts the input image and reduces the information amount accessed by the model.
Have the authors experimented with not adding noise to the input? That would simplify the approach further.

How many frames does the model process at once? (What is the value of T ?)

Minor typos:
L309: "enities"
L386: "UINNEXT"
L495: "StableDiffsuion"
L523: "changing a little as possible"
soundness: 3
strengths: The paper tackles an interesting problem in video segmentation.
The approach is straightforward and well-motivated, leveraging existing diffusion models.
The contributions are clearly articulated, and the paper is well-structured.
The authors commit to releasing code, models, and data, which supports reproducibility of their results by the community.
summary: This paper presents a method to adapt a video diffusion model (a text-to-video generative model) into a referral video segmenter (a text+video to masklet model).
Instead of segmenting clearly defined objects as conventionally done in image and video object segmentation, the authors focus on segmenting dynamic concepts such as a wave or a cloud of smoke. In order to evaluate their model on this task, the authors introduce the Referral Video Process Segmentation (Ref-VPS) benchmark, consisting of 111 annotated Tiktok videos.
Because of the lack of training data for segmenting such dynamic "entities", the authors propose to use a video diffusion model, which has learned the mapping from linguistic descriptions to video on massive amounts of text-video pairs. The model is very minimally adapted to the Referral Video Segmentation task in order to conserve its capabilities acquired during pretraining.
weaknesses: Evaluation of Ref-VPS:
Dynamic concepts such as a wave or a cloud of smoke don't have clearly defined edges. Providing a single ground-truth contour for them is necessarily arbitrary. How then to evaluate? 
For example, in Figure 1, the mask for “the smoke dissipating” arguably does not cover all the smoke. Similarly, for ”the wave crashing in the ocean”, the ground-mask does not include the wave foam on the left. A model prediction that includes these elements would be unjustly penalised.
Segmentations on this kind of "dynamic concepts" are notoriously hard to evaluate but this issue is mostly not covered in this paper. The authors exclude the contour accuracy F metric but the J metric also suffers from this issue. There are possible solutions to alleviate this problem and their discussion is expected for a paper introducing a benchmark on segmenting "Dynamic concepts".

Runtime:
In order to be practical, the approach presented in this paper should have a reasonable running time. The paper does not give details on this subject.
For example, how long does it take for the model to segment a video of 256 frames, using an A100 GPU?

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Questions:
1. Can you clarify whether you are proposing Referring Video Process Segmentation as a new/separate task?
2. What is the definition of "concept" and how is it different from a "class"? Is there high inter-annotator agreement on Ref-VPS? My concern is that the challenges of Referring Video Process Segmentation are due to the ambiguity of the task and the arbitrary nature of the label, rather than the lack of generalizable representations in the model.
3. In l. 157 can you clarify what you mean by "universal"? Do you mean generalizing to many domains?
4. In l. 184-188, what are the advantages of using such models? One can imagine that they are more lightweight or less computationally demanding than diffusion models.
5. In l. 275 what is t usually set to?
6. In l. 314 can you clarify what is meant by the "physical nature of the event"?
7. What architecture if REM specifically built on top of in the experiments in Section 5.1?
8. In Section 5.2 are the methods trained on the datasets as reported in Table 1?
9. In the "Fine-tuning strategy" section on page 10, out of curiosity, have you tried freezing the encoder and fine-tuning the decoder to preserve more of the representation?

Minor Typos/Writing Suggestions:
- l. 011 include what REM stands for
- l. 018 "crushing"->"crashing"
- "Referral" is used in l. 036, 039 but "Referring" is used in l. 134. It would be good to stay consistent to avoid confusion.
- l. 122 "approaches,"->"approaches"
- l. 160 "de-fact" -> "de-facto"
- l. 172 "highly-effective"-> "highly effective"
- l. 179 "RVOS,"->"RVOS"
- l. 260 "Figure 3 REM,"-> "Figure 3, REM"
- l. 263 "below"->"below."
- l. 313 "queries however" -> "queries; however"
- l. 331 "expression"->"expressions"
- l. 376 "video-segmentation"->"video segmentation"
- l. 377 "and we"->"and as we"
- l. 377 "by up to 46% out-of-domain in terms of Region Similarity" awkward wording
- l. 495 "StableDiffsuion"-> "StableDiffusion"
soundness: 3
strengths: 1. REM has a straightforward design for training and inference that is presented clearly. In general the writing is clear.
2. REM results are strong on Ref-VPS and Ref-YTB and the ablation study effectively validates the author's claim that retaining the architecture of diffusion models is important for taking advantage of their generalizable representations for referring video segmentation. The finding that improving video generation models leads to enhanced video segmentation performance is interesting.
summary: The authors present a diffusion-model based method for the task of Referring Video Segmentation, called REM, and a new dataset for the task called Ref-VPS. The motivation for this work stems from the observation that current datasets for the task of Referring Video Segmentation are typically extensions of object-centric datasets for Referring Video Object Segmentation, which has prevented development of general video region segmentation methods. To address this, the authors create their carefully curated dataset, Ref-VPS, from TikTok videos, and label it with "concepts" that are not necessarily object-centric. They create REM from diffusion models that, due to their pretraining on Internet-scale data, have generalizable representations. The authors show that on Ref-VPS and two Referral Video Object Segmentation, REM either outperforms or performs favorably compared to current Referring Video Segmentation baselines. On an ablation study, the authors validate their claim that preserving the architecture of diffusion models during fine-tuning for Referring Video Segmentation leads to better performance.
weaknesses: 1. The motivation of the paper is not very clear. Specifically, it is not clear whether the authors are advocating for distinguishing the Referring Video Object Segmentation benchmarks from the Referring Video Process Segmentation benchmark that they propose. If the authors are advocating for this (and thus for two different tasks), then the final output of the model should be evaluated differently for each task, but this doesn't seem to be the case.
2. The Ref-VPS dataset was created from heavily curated TikTok videos and thus may not be representative of real-world scenes and videos. A more in-depth comparison of Ref-VPS to a dataset comprised of random videos (such as YouTube-BoundingBoxes [1]) would strengthen the paper if it is found that Ref-VPS adequately represents a wide variety of scenes, especially since the authors claim that the generalization ability of their method is a contribution.
3. The details of the REM method's computational efficiency and a comparison to that of the baselines are missing. Specifically, the training time, inference time and amount of memory use on the GPU should be included and compared to the baselines. The ease of use of the method is important in determining its contribution.

[1] "YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video". Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan and Vincent Vanhoucke. CVPR, 2017.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: My biggest concern is listed in the Weaknesses.
soundness: 3
strengths: 1. The paper is well-written and easy to follow. 

2. The task of Referral Video Process Segmentation is interesting. 

3. The REM achieves comparable performance with the state-of-the-art method UNINEXT on the RES task and exhibits stronger performance on out-of-domain benchmarks.
summary: This paper explores using a pre-trained video diffusion model in referring segmentation. The paper proposes a new setting: Referral Video Process Segmentation, which is interesting. The model provides valuable insights on using generative prior knowledge to perform segmentation better.
weaknesses: REM uses a one-step approach to predict the segmentation mask during training. As a result, REM is more like resuming the pre-trained weights from the video diffusion model and training a U-Net segmentation model. I believe the model could achieve comparable performance without using the VAE Encoder to process the mask and instead using a few simple MLP layers. If this is the case, REM's performance gain may not be due to the authors' claim ``preserving as much of the generative model's original representation." Therefore, I think the authors need to add more detailed ablation studies to analyze the key factors leading to performance improvement.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: The definition of RVOS in the paper is **Referral** Video Object Segmentation in the introduction section, but it should be **Referring** Video Object Segmentation.
soundness: 3
strengths: 1. The focus on segmenting video processes presents a new research direction, which is valuable for exploring and evaluating model performance in this context.
2. The application of video diffusion to Referring Video Object Segmentation (RVOS) tasks is an interesting and promising approach.
summary: The paper aims to segment a wide range of concepts in video through a natural language. The paper proposes to rely on the powerful representation capabilities of the video diffusion model. Besides, to facilitate the research community, the paper introduces a new benchmark for Referral Video Process Segmentation (RVPS), which captures dynamic phenomena that exist at the intersection of video and language.
weaknesses: 1. The integration of video diffusion into referring segmentation appears incremental and lacks novelty, as it primarily adapts an existing video diffusion model to RVOS without specific tailoring for the task.
2. A detailed computational complexity analysis is needed to better understand the efficiency of the proposed method.
3. Both the figures and the overall writing quality require further refinement for better clarity and presentation.

Meta-review: This paper receives mixed reviews from five reviewers.

The positive reviewers think the new setting is interesting, and it evaluates models on dynamic processes.

The negative reviewers were concerned about the novelty of technical parts, ablation study details, and several writing problems in particular for the main highlighted contributions.

During the rebuttal, the authors have give detailed feedbacks on several aspects, including generalization to unseen domains (more experiments), inference time, temporal consistency, more ablation studies on hyper-parameters and improved writing.  Parts of these issues are solved. One reviewer raises the score from 3 to 5. One reviewer raises the score from 4 to 6.  However, after discussion with the reviewers, there are still two negative reviews in the final stage.

The AC have checked the refined version of the submission and agrees with the negative sides.

The next submission must include a more detailed comparison with existing diffusion-based segmentation methods, more clear motivation, a clearer contribution statement, more detailed ablation studies, and future works on bad cases.

Decision: Reject