ReferEverything: Towards segmenting everything we can speak of in videos

Authors: Anurag Bagchi, Zhipeng Bao, Yu-Xiong Wang, Pavel Tokmakov, Martial Hebert
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Referring Video Segmentation, Video Diffusion Models

Abstract:
We present REM, a framework for segmenting a wide variety of concepts in video that can be described through natural language. To achieve this level of generalization, our method capitalizes on visual-language representations learned by video diffusion models on Internet-scale datasets. A key insight of our approach is preserving as much of the generative model’s original representation as possible, while fine-tuning it on narrow-domain Referral Object Segmentation datasets. As a result, despite being exclusively trained on object masks from a limited set of categories, our framework is able to accurately segment and track both rare, unseen objects and non-object, dynamic concepts, such as waves crushing in the ocean. To better quantify the generalization capabilities of our model, we introduce a new benchmark for Referral Video Process Segmentation (RVPS), which captures dynamic phenomena that exist at the intersection of video and language. Our experiments show that REM performs comparably to state-of-the-art approaches on in-domain datasets while outperforming them by up to 28\% out-of-domain, leveraging the power of Internet-scale pre-training.

Decision: reject
Reviewer scores: 6, 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1729307472919

thread_type:
official_review

<thread_object>
creator: reviewer_bagm

<content>
summary:
The paper aims to segment a wide range of concepts in video through a natural language. The paper proposes to rely on the powerful representation capabilities of the video diffusion model. Besides, to facilitate the research community, the paper introduces a new benchmark for Referral Video Process Segmentation (RVPS), which captures dynamic phenomena that exist at the intersection of video and language.

strengths:
1. The focus on segmenting video processes presents a new research direction, which is valuable for exploring and evaluating model performance in this context.
2. The application of video diffusion to Referring Video Object Segmentation (RVOS) tasks is an interesting and promising approach.

weaknesses:
1. The integration of video diffusion into referring segmentation appears incremental and lacks novelty, as it primarily adapts an existing video diffusion model to RVOS without specific tailoring for the task.
2. A detailed computational complexity analysis is needed to better understand the efficiency of the proposed method.
3. Both the figures and the overall writing quality require further refinement for better clarity and presentation.

questions:
The definition of RVOS in the paper is **Referral** Video Object Segmentation in the introduction section, but it should be **Referring** Video Object Segmentation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730291673207

thread_type:
official_review

<thread_object>
creator: reviewer_hmhl

<content>
summary:
This paper explores using a pre-trained video diffusion model in referring segmentation. The paper proposes a new setting: Referral Video Process Segmentation, which is interesting. The model provides valuable insights on using generative prior knowledge to perform segmentation better.

strengths:
1. The paper is well-written and easy to follow. 

2. The task of Referral Video Process Segmentation is interesting. 

3. The REM achieves comparable performance with the state-of-the-art method UNINEXT on the RES task and exhibits stronger performance on out-of-domain benchmarks.

weaknesses:
REM uses a one-step approach to predict the segmentation mask during training. As a result, REM is more like resuming the pre-trained weights from the video diffusion model and training a U-Net segmentation model. I believe the model could achieve comparable performance without using the VAE Encoder to process the mask and instead using a few simple MLP layers. If this is the case, REM's performance gain may not be due to the authors' claim ``preserving as much of the generative model's original representation." Therefore, I think the authors need to add more detailed ablation studies to analyze the key factors leading to performance improvement.

questions:
My biggest concern is listed in the Weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730572944131

thread_type:
official_review

<thread_object>
creator: reviewer_bxba

<content>
summary:
The authors present a diffusion-model based method for the task of Referring Video Segmentation, called REM, and a new dataset for the task called Ref-VPS. The motivation for this work stems from the observation that current datasets for the task of Referring Video Segmentation are typically extensions of object-centric datasets for Referring Video Object Segmentation, which has prevented development of general video region segmentation methods. To address this, the authors create their carefully curated dataset, Ref-VPS, from TikTok videos, and label it with "concepts" that are not necessarily object-centric. They create REM from diffusion models that, due to their pretraining on Internet-scale data, have generalizable representations. The authors show that on Ref-VPS and two Referral Video Object Segmentation, REM either outperforms or performs favorably compared to current Referring Video Segmentation baselines. On an ablation study, the authors validate their claim that preserving the architecture of diffusion models during fine-tuning for Referring Video Segmentation leads to better performance.

strengths:
1. REM has a straightforward design for training and inference that is presented clearly. In general the writing is clear.
2. REM results are strong on Ref-VPS and Ref-YTB and the ablation study effectively validates the author's claim that retaining the architecture of diffusion models is important for taking advantage of their generalizable representations for referring video segmentation. The finding that improving video generation models leads to enhanced video segmentation performance is interesting.

weaknesses:
1. The motivation of the paper is not very clear. Specifically, it is not clear whether the authors are advocating for distinguishing the Referring Video Object Segmentation benchmarks from the Referring Video Process Segmentation benchmark that they propose. If the authors are advocating for this (and thus for two different tasks), then the final output of the model should be evaluated differently for each task, but this doesn't seem to be the case.
2. The Ref-VPS dataset was created from heavily curated TikTok videos and thus may not be representative of real-world scenes and videos. A more in-depth comparison of Ref-VPS to a dataset comprised of random videos (such as YouTube-BoundingBoxes [1]) would strengthen the paper if it is found that Ref-VPS adequately represents a wide variety of scenes, especially since the authors claim that the generalization ability of their method is a contribution.
3. The details of the REM method's computational efficiency and a comparison to that of the baselines are missing. Specifically, the training time, inference time and amount of memory use on the GPU should be included and compared to the baselines. The ease of use of the method is important in determining its contribution.

[1] "YouTube-BoundingBoxes: A Large High-Precision Human-Annotated Data Set for Object Detection in Video". Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan and Vincent Vanhoucke. CVPR, 2017.

questions:
Questions:
1. Can you clarify whether you are proposing Referring Video Process Segmentation as a new/separate task?
2. What is the definition of "concept" and how is it different from a "class"? Is there high inter-annotator agreement on Ref-VPS? My concern is that the challenges of Referring Video Process Segmentation are due to the ambiguity of the task and the arbitrary nature of the label, rather than the lack of generalizable representations in the model.
3. In l. 157 can you clarify what you mean by "universal"? Do you mean generalizing to many domains?
4. In l. 184-188, what are the advantages of using such models? One can imagine that they are more lightweight or less computationally demanding than diffusion models.
5. In l. 275 what is t usually set to?
6. In l. 314 can you clarify what is meant by the "physical nature of the event"?
7. What architecture if REM specifically built on top of in the experiments in Section 5.1?
8. In Section 5.2 are the methods trained on the datasets as reported in Table 1?
9. In the "Fine-tuning strategy" section on page 10, out of curiosity, have you tried freezing the encoder and fine-tuning the decoder to preserve more of the representation?

Minor Typos/Writing Suggestions:
- l. 011 include what REM stands for
- l. 018 "crushing"->"crashing"
- "Referral" is used in l. 036, 039 but "Referring" is used in l. 134. It would be good to stay consistent to avoid confusion.
- l. 122 "approaches,"->"approaches"
- l. 160 "de-fact" -> "de-facto"
- l. 172 "highly-effective"-> "highly effective"
- l. 179 "RVOS,"->"RVOS"
- l. 260 "Figure 3 REM,"-> "Figure 3, REM"
- l. 263 "below"->"below."
- l. 313 "queries however" -> "queries; however"
- l. 331 "expression"->"expressions"
- l. 376 "video-segmentation"->"video segmentation"
- l. 377 "and we"->"and as we"
- l. 377 "by up to 46% out-of-domain in terms of Region Similarity" awkward wording
- l. 495 "StableDiffsuion"-> "StableDiffusion"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730578179133

thread_type:
official_review

<thread_object>
creator: reviewer_nxrs

<content>
summary:
This paper presents a method to adapt a video diffusion model (a text-to-video generative model) into a referral video segmenter (a text+video to masklet model).
Instead of segmenting clearly defined objects as conventionally done in image and video object segmentation, the authors focus on segmenting dynamic concepts such as a wave or a cloud of smoke. In order to evaluate their model on this task, the authors introduce the Referral Video Process Segmentation (Ref-VPS) benchmark, consisting of 111 annotated Tiktok videos.
Because of the lack of training data for segmenting such dynamic "entities", the authors propose to use a video diffusion model, which has learned the mapping from linguistic descriptions to video on massive amounts of text-video pairs. The model is very minimally adapted to the Referral Video Segmentation task in order to conserve its capabilities acquired during pretraining.

strengths:
The paper tackles an interesting problem in video segmentation.
The approach is straightforward and well-motivated, leveraging existing diffusion models.
The contributions are clearly articulated, and the paper is well-structured.
The authors commit to releasing code, models, and data, which supports reproducibility of their results by the community.

weaknesses:
Evaluation of Ref-VPS:
Dynamic concepts such as a wave or a cloud of smoke don't have clearly defined edges. Providing a single ground-truth contour for them is necessarily arbitrary. How then to evaluate? 
For example, in Figure 1, the mask for “the smoke dissipating” arguably does not cover all the smoke. Similarly, for ”the wave crashing in the ocean”, the ground-mask does not include the wave foam on the left. A model prediction that includes these elements would be unjustly penalised.
Segmentations on this kind of "dynamic concepts" are notoriously hard to evaluate but this issue is mostly not covered in this paper. The authors exclude the contour accuracy F metric but the J metric also suffers from this issue. There are possible solutions to alleviate this problem and their discussion is expected for a paper introducing a benchmark on segmenting "Dynamic concepts".

Runtime:
In order to be practical, the approach presented in this paper should have a reasonable running time. The paper does not give details on this subject.
For example, how long does it take for the model to segment a video of 256 frames, using an A100 GPU?

questions:
In order to match the pre-training input distribution of the model, the authors add noise to the input (as performed during pre-training).
However, it corrupts the input image and reduces the information amount accessed by the model.
Have the authors experimented with not adding noise to the input? That would simplify the approach further.

How many frames does the model process at once? (What is the value of T ?)

Minor typos:
L309: "enities"
L386: "UINNEXT"
L495: "StableDiffsuion"
L523: "changing a little as possible"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730798859093

thread_type:
official_review

<thread_object>
creator: reviewer_qta2

<content>
summary:
The REM paper introduces a novel framework for Referral Video Segmentation (RVS) that uses a video-text diffusion model to generalize segmentation to a wide range of visual concepts.REM effectively segments both common and rare objects, as well as non-object dynamic concepts like smoke, water, or changing light. The paper also introduces Ref-VPS, a new segmentation benchmark on dynamic processes that require both spatial and temporal modeling.

strengths:
* REM achieves good generalization to out-of-domain concepts and non-object entities by leveraging Internet-scale pretraining on video-language data, outperforming specialized methods.

* The Ref-VPS benchmark provides a way to evaluate models on dynamic processes, filling a gap in existing RVS datasets by focusing on temporal events and continuous appearance changes.

weaknesses:
* The REM method performs comparably to MUTR  and only slightly better then VLMO on Ref-YTB.

* Given use of a high-capacity diffusion model, there is limited discussion on its computational efficiency or inference speed. A comparison of REM's computational requirements with prior methods would be beneficial.

* Analysis on the impact of the quantity and quality of synthetic data would be useful - whether reducing/increasing the amount of synthetic data affect REM's performance.

* Since REM is applied to video segmentation tasks, especially focusing on continuous changes, it would be great to have some evaluation of temporal consistency.

questions:
* How does REM handle ambiguous or overlapping concepts (e.g., scenes where multiple dynamic elements, like water and smoke, interact)?

* How critical is the quality of synthetic data for REM’s generalization across diverse concepts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731909485097

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General response

comment:
We thank the reviewers for their feedback and valuable suggestions, which helped us further strengthen the paper. We are glad that reviewers found our approach to be **straightforward and well-motivated** (reviewers NxrS and bXbA), its **zero-shot out-of-domain generalization** performance to be strong (reviewers QTA2, bXbA, and hMhL), the proposed extension of **referring video segmentation outside of the domain of objects** to be valuable (reviewers QTA2, NxrS, hMhL, and BAgM) and the paper to be **well written** (reviewers NxrS, bXbA, and hMhL). We have addressed all the comments individually, and highlight a couple of common concerns in this general response.


### **Ground-truth annotations**
We would like to emphasize that Figure 1 in the manuscript shows REM predictions, not the ground-truth. We have clarified this in the caption to avoid confusion. While our model’s predictions are accurate, the ground-truth annotations in the Ref-VPS benchmark offer even greater precision, and explicitly address the natural ambiguity of the problem with Ignore labels. We have added Figure B with a corresponding discussion to the appendix to illustrate this. In particular, the entire extent of the wave is labeled in the third row. Rows 1 and 4 illustrate the handling of ambiguous scenarios, where only the confident regions of the glowing water and of the light column are labeled as target, and the ambiguous regions are labeled as Ignore (shown in gray). Pixels inside the Ignore regions are not included in the metric calculation.  

### **Inference costs**
Several reviewers have raised concerns about the computational complexity of our approach. While REM is based on a video diffusion model, which is inherently computationally intensive due to its multiple inference steps, we emphasize that REM is trained and evaluated as a feed-forward model for referring video segmentation. Consequently, its memory requirements and runtime are comparable to those of other methods in the literature. 

We report the inference speed and the memory consumption of REM alongside the primary baselines from our paper using their public implementations below:

Model | Inference speed (FPS)  | Memory (GB)
:---|:---:|:---:
MUTR| 13.6 |34.1
UNINEXT| 3.3 |9.7
VDIT| 7.1 |72.8
Ours| 7.1 |41.8

These values are estimated under the following protocol: inference was performed on 32-frame clips from Ref-DAVIS on a single A100 GPU with averages computed over 80 runs. As shown, the inference costs of REM align with those of other state-of-the-art approaches. We have incorporated these results into Table C in the manuscript.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733279643176

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Final response

comment:
Dear Reviewers,  

We deeply appreciate your thoughtful feedback and constructive engagement, which helped to improve our paper. Below, we summarize the key contributions of our work and highlight the additional analyses provided in the rebuttal, which address your comments and demonstrate the strengths of our approach:    


**Novel Contributions:**    
- We introduce **REM**, a Referring Video Segmentation (RVS) framework that can segment a wide range of object-centric and non-object-centric concepts in video that can be described through natural language, while being trained **only on standard referring object segmentation datasets** (Ref-COCO and Ref-YTB).
- By capitalizing on visual-language representations learned by video diffusion models, REM achieves strong generalization to **out-of-domain datasets** (BURST, VSPW, and Ref-VPS), outperforming both traditional approaches, and methods like VDIT that share the same backbone, by up to 28% in terms of relative improvement.
- We present **Ref-VPS**, the first benchmark dedicated to **dynamic processes** involving spatiotemporal changes beyond object-centric segmentation tasks. Ref-VPS evaluates **zero-shot** model generalization to challenging dynamic concepts, filling a critical gap in the existing literature.    


**Key Insights from Rebuttal Results:**    
- **Generalization to Unseen Domains:** The superior performance of REM across BURST’s 482 diverse object categories, despite only training on 94 categories, underscores the method’s ability to generalize beyond the training domain. 
- **Inference Speed and Compute:** REM’s inference speed and compute are in line with the state-of-the-art methods, while delivering much better OOD generalization.  
- **Impact of Hyperparameters:** Ablations on the mask decoder design and time-step parameter $t$ confirmed that our choices are optimal.
- **Temporal Consistency:** We implemented and reported a new temporal consistency metric in the rebuttal, showing REM’s superior ability to model continuous changes compared to state-of-the-art baselines.
- **Clarifications on Ref-VPS:** We take great care to ensure robust evaluation of challenging dynamic concepts in Ref-VPS by labeling all ambiguous regions as Ignore.
- **Clarity of the presentation:** We have updated the manuscript to address all specific suggestions regarding clarity and writing.     


**Conclusion:**    
Our work pioneers a novel direction in RVS, with REM delivering unparalleled generalization across diverse and dynamic scenarios while introducing Ref-VPS as a new, valuable benchmark. We hope the provided results and clarifications demonstrate the rigor, impact, and potential of this research.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper receives mixed reviews from five reviewers.

The positive reviewers think the new setting is interesting, and it evaluates models on dynamic processes.

The negative reviewers were concerned about the novelty of technical parts, ablation study details, and several writing problems in particular for the main highlighted contributions.

During the rebuttal, the authors have give detailed feedbacks on several aspects, including generalization to unseen domains (more experiments), inference time, temporal consistency, more ablation studies on hyper-parameters and improved writing.  Parts of these issues are solved. One reviewer raises the score from 3 to 5. One reviewer raises the score from 4 to 6.  However, after discussion with the reviewers, there are still two negative reviews in the final stage.

The AC have checked the refined version of the submission and agrees with the negative sides.

The next submission must include a more detailed comparison with existing diffusion-based segmentation methods, more clear motivation, a clearer contribution statement, more detailed ablation studies, and future works on bad cases.