SMITE: Segment Me In TimE

Authors: Amirhossein Alimohammadi, Sauradip Nag, Saeid Asgari, Andrea Tagliasacchi, Ghassan Hamarneh, Ali Mahdavi Amiri
Primary area: generative models
Keywords: video segmentation, diffusion models, video diffusion, part segmentation

Abstract:
Segmenting an object in a video presents significant challenges. Each pixel must be accurately labeled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a
pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives. The project page is available at https://segment-me-in-time.github.io/

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 8, 6

Reviews and discussion:
<thread>
creation date:
1730014170531

thread_type:
official_review

<thread_object>
creator: reviewer_askr

<content>
summary:
This paper presents SMITE, a few-shot video segmentation method with fine granularity tracking. SMITE is based on SLiMe, a few-shot image segmentation method that finetunes text tokens for a diffusion network and probes attention masks to output a mask.  SMITE additionally introduces cross-attention finetuning of the diffusion network, a segment tracking and voting module based on point tracking, and a low-pass regularization term. SMITE achieves better segmentation performance than applying baseline SLiME per frame in a newly proposed dataset SMITE-50.

strengths:
This paper tackles the hard yet useful task of few-shot, fine-granularity video segmentation. This task has potentially wide applications. I think the main contribution of this work is taking the initial crack at this problem with reasonable performance in segmentation quality and temporal stability when compared to applying the SLiME per frame. 

Technique-wise, segmentation tracking/voting and low-pass regularization are reasonable in handling temporal flickering unique to videos, and the results also supported this.

The proposed dataset SMITE-50 is also potentially useful for future work, as it contains densely labeled segmentation which is a rare type of data.

weaknesses:
I don’t see it mentioned in the paper, but I suspect that SMITE might be too slow for most practical applications. In addition to finetuning the text tokens and the cross-attention layers during a training stage, SMITE additionally performs dense, windowed point tracking – there seem to be a lot of windows (linear to video length) and a lot of repeated computation in the point tracker (linear to windows size). SMITE also minimizes an energy function which is also costly. Besides, given that SMITE expands the spatial self-attention to the spatiotemporal volume, it would be quite memory intensive for longer videos.

I think the authors can also compare SMITE to existing few-shot video object segmentation methods like [Multi-grained Temporal Prototype Learning for Few-shot Video Object Segmentation, ICCV 2023]. This can happen in two dimensions: 1) applying few-shot VOS to the fine-grained segmentation setting, and 2) applying SMITE to the few-shot VOS setting. This can highlight whether the fine-grained segmentation setting significantly differs from the object setting and supplements the existing sparse baselines.

In Table 2, should there be a row with neither of the loss and another row with only the low-pass loss? Also, can the ablations be run on the entire SMITE-50 dataset rather than just on the CARS subset? This would make the results more significant.

questions:
What is the speed and memory requirement of the proposed method? How many frames can it process?

What if an inflated U-Net is not used during text embedding optimization (e.g., the segment prompts are just images, not videos) but only used during inference?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730191138129

thread_type:
official_review

<thread_object>
creator: reviewer_k95k

<content>
summary:
This work presents a framework for segmenting videos into parts at adjustable levels of granularity, using only a single or few annotated images as references. The framework comprises several main components:
First, it optimizes text embeddings and fine-tunes a stable diffusion model so that the WAS attention maps (a combination of self and cross-attention maps) align with the annotated reference images.
Next, temporal consistency is enforced through specific loss terms, enabling the framework to generate segmentation maps from the input video using the optimized text embeddings and fine-tuned diffusion model.

strengths:
Novel and interesting problem formulation. 
The motivations behind the design choices are presented clearly.
Qualitative results seem decent.
A new video part-level dataset and SOTA performance.

weaknesses:
1. Limited competitors - The baseline (frame-by-frame) might be a bit too simplistic. 
Maybe a better baseline would be a simple video extension of SLIME (without fine-tuning the cross-attention layers). This video extension is pretty trivial and should have some consistency.
Another baseline could be using SLIME (single frame) with tracking.
2. Limited ablation study - 
I would have liked to know what is the contribution of each step.
In addition, I think more ablation would make the paper stronger. For example, how similar should be the input object to the one in the video? Would horse segmentation work on zebras? and on dogs?

In addition: Table 2 only states what happens if only the regularization step is dropped. What about dropping the tracking? And what about dropping both?

questions:
1. The differences in Figure 3 could be better presented (the difference between (c) and (d) is hardly visible).

2. It was slightly challenging to follow all the technical details. 
Few small modifications might reduce the reading time:
For example: 
a. In section 4.2 3 loss terms were mentioned and in section 4.4 2 other loss terms were mentioned; It will be clearer if the authors stated clearly how and in what phase each loss term is used (all together? two-phase optimization?).
b. lines 267-268, sliding window -> temporal sliding window

3. Small technical note:
line 148: should the epsilon_theta after UNet be deleted?

4. Lines 303-305:
It is not clear how this scenario is different from using only cross attention maps. 
If I understand what the authors suggested correctly: average voting can be computed directly on the masks as a "tracking post-process step" without the need for an optimization for tracking.
If so, why can't you simply calculate the average vote on the S_WAS as well?
If not, what was your intention?

5. Line 105 - you mention that some parts cannot be described by text. 
Yet, you do find text embeddings that describe them, although it is reasonable to claim that these embeddings do not represent natural text descriptions. 
However, it will be interesting to acknowledge/discuss this distinction. 
Why does it make sense to find text embeddings for parts that cannot be described by texts?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730344098246

thread_type:
official_review

<thread_object>
creator: reviewer_acvu

<content>
summary:
This paper proposes a SMITE framework for segmenting object fragments across video frames. The method consists of denosing module, diffusion module and tracking module. Overall, it achieves satisfactory performance for video segmentation over various levels of granularity.

strengths:
1)This proposed method combines information from text, tracker and spatial temporal features for video segmentation at arbitrary level.
2)SMITE achieves state of the art performance on the SMITE-50 dataset.

weaknesses:
1)The proposed method is only compared with two methods in Table 1, while there are lots of few-shot and zero-shot segmentation methods. 
Besides, it seems that the proposed method can be directly applied on multi-target video segmentation datasets like DAVIS2017. So, there should be more comprehensive comparison, for both methods and datasets.
2)Speed is important for online tracking; so there should be some runtime analysis.
3)Limitations of the method is not discussed in experiments.
4)How does the proposed method work on random videos and random annotation references?

questions:
Same as weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730540695865

thread_type:
official_review

<thread_object>
creator: reviewer_rk61

<content>
summary:
This paper presents SMITE (Segment Me In Time), a novel approach for one/few-shot video object part segmentation with flexible granularity. Building upon the image-based SLiMe method, SMITE leverages a pre-trained text-to-image diffusion model (Stable Diffusion) and introduces a tracking mechanism and low-pass filtering to maintain temporal consistency in video segmentations. Given one or a few annotated reference images, SMITE aims to segment unseen videos of objects from the same category, respecting the provided segmentation granularity. The authors introduce a new dataset, SMITE-50, to benchmark performance and compare against relevant baselines.

strengths:
1. Originality : The proposed method innovatively combines pre-trained text-to-image diffusion models with a temporal tracking mechanism to achieve video segmentation with flexible granularity using only a few reference images.
1. Quality : The comprehensive experiments, including quantitative evaluations on the new SMITE-50 dataset, qualitative comparisons, ablation studies, and user studies, provide solid support for the method's effectiveness.
1. Clarity : The paper is well-structured, with clear explanations of the problem, methodology, and results. Figures and diagrams are used effectively to enhance understanding.
1. Significance: Addressing the challenge of few-shot video object part segmentation has practical implications for various applications, including VFX and video editing.

weaknesses:
1. Methodological Clarity: While the overall explanation is detailed, some of the methodology is difficult to follow, especially for readers not familiar with diffusion models and the previous work SLiMe. The process of how tracking integrates into the attention map refinement could be clearer, with more intuitive descriptions of its working mechanism.
1. Dataset & Baselines: The new SMITE-50 dataset is an important contribution, but it appears somewhat small (with 50 videos). Expanding the dataset size or providing an analysis of its generalization for more categories could strengthen the paper’s claims. A comparison to more widespread benchmarks for video segmentation, such as DAVIS or other VOS datasets, would help position SMITE relative to not only a niche dataset but also larger, well-accepted benchmarks in the community.
1. Inference Efficiency: The paper does not deeply explore the computational cost or inference time of the method in practical settings, which would be relevant given the addition of several components (e.g., temporal voting, low-pass filters). Addressing this in the discussion or providing benchmarks on efficiency could help increase understanding of real-world applicability.

questions:
1. How sensitive is the method's performance to the choice of tracking algorithm? Have you experimented with other tracking methods besides CoTracker?
1. The low-pass regularization is mentioned as a key component. Can the authors elaborate on how this regularization term is formulated and how it balances preserving segment structures with smoothing transitions?
1. How does the proposed method handle videos with rapid movements, severe occlusions, or significant lighting changes? Are there strategies to mitigate potential performance degradation in such scenarios?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730832459862

thread_type:
official_review

<thread_object>
creator: reviewer_9vaw

<content>
summary:
This paper defines a task of flexible granularity, which aims to segment user-intended granularity temporally using one or a few reference images. To do so, it proposes several techniques to achieve temporal consistent multi granularity segmentation including alternative learning of text embeddings and cross attention module, tracking modules with voting mechanism to be aware of neighbor pixels and low frequency regularizer for spatial consistency. It shows its effectiveness on the proposed dataset, SMITE-50

strengths:
Belows are the strong points that this paper has:

- It is well-written and well-organized, which help reader to understand new task and motivation of the proposed method.

- The proposed techniques including learning generalizable segments using alternative tuning of text embeddings and cross attention, temporal consistency via tracking voting mechanism, and spatial consistency with low-pass filter are in good harmony, which can outperform the other baselines with high margin.

weaknesses:
Belows are points that the reviewer feel concerned about.

- In line #247, the authors mention that learning text embeddings and cross-attention layers is conducted in two phases to provide a better initialization for the next phase. However, this phased or alternating training approach could also increase training complexity. Could the authors provide an ablation study to compare the results of joint training versus alternating training to justify the chosen training setup? Specifically, analysis on training time with final performance of two training strategies can be provided.

- In the supplementary video, specifically at 1:46 in the "Qualitative_Comparison" section, it appears that the proposed method captures a relatively smaller region of the "eye" area compared to the baseline. Is this due to the proposed voting mechanism diluting the effective area by averaging nearby major segments? Would this issue be mitigated by replacing the current hard voting with a soft voting mechanism? Is it doable that quantitative analysis on different voting system can be measured?

- Is the proposed method shape-agnostic within the same class? To demonstrate that the method can segment other unseen images, the authors should provide test samples that differ in shape from the reference object. For example, as similar to what the author showed in "Cars" of Figure 7, if the reference image includes a detailed segmentation map of a "sedan" car, the test samples can be replaced with different shapes of cars, such as "SUVs?

questions:
- Is the proposed method also applicable to "Multi-object flexible granularity", which indicates that reference images annotated with multiple objects (or multiple classes) different granularity can be used as input to the proposed model?

- Could the author provide any failure cases?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732634624137

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response for Rebuttal

comment:
We sincerely thank all the reviewers for their valuable feedback and excellent suggestions.

We appreciate their recognition of our paper as well-written ($R_{9Vaw}$), well-structured ($R_{rK61}$), innovative ($R_{rK61}$), novel ($R_{K95K}$), interesting ($R_{K95K}$), and reasonable ($R_{askr}$). Additionally, we are grateful for their acknowledgment of the state-of-the-art performance of our method ($R_{9Vaw}, R_{aCvua}, R_{K95K}$) and the merit of our proposed dataset ($R_{K95K}, R_{askr}$). 

We revised our paper to incorporate reviewer feedback (revisions are denoted in blue text). Here, we address the shared concerns and comments raised by the reviewers. Individual reviewer questions are answered under each review. We are happy to respond to any additional questions and welcome further suggestions or updated opinions from the reviewers.

**More ablations on the entire SMITE-50** ($R_{askr}, R_{K95K}$): 
We have merged previous Tables 2 and 3 and added a new table (current Table 2) that shows the impact of each component of our method on the entire SMITE-50 dataset. It is evident that our method performs the best when all of our components are being utilized. Please note that low-pass regularization loss without tracking has no impact on the results. Therefore, we did not report the ablation number for that setting.


**Comparison with more baselines including variants of SLiMe ($R_{K95K}$) :** We have updated Table 1 to include more baselines including SLiMe+CoTracker and SLiMe+Inflated UNet. As it is evident, SMITE still outperforms all these other baselines. Please refer to Section 5, Quantitative comparison. We also updated our user study section and Table 4 and we are still ranked highest among the alternatives. 

**Time and memory analysis ($R_{aCvu}, R_{9Vaw}, R_{rK61}, R_{askr}$) :** Using 10 images with a batch size of 1 on 15 GB of GPU VRAM, SMITE's training takes 20 minutes. By increasing the batch size and utilizing more GPU VRAM, training time can be reduced to 7 minutes. We train for 300 epochs, but the best accuracy is often reached before 200 epochs, therefore this training time is an upper bound for the samples we showed in the paper. For inference, each frame takes 26 seconds and it requires 60 GB of GPU VRAM to process the entire video. We have added a discussion about time and memory analysis in Appendix.

**Comparison on other datasets including DAVIS ($R_{rK61},R_{aCvu}$) :** To demonstrate SMITE's performance on other datasets compared to the state-of-the-art in video segmentation, we had already included a comparison with PumaVOS in our original submission (Section 7.2). To further compare our method with SOTA for single-object segmentation, we evaluated it on DAVIS-2016. Since DAVIS-2017 (as suggested by aCvu) involves multiple instances, our method cannot be directly applied to DAVIS-2017 without additional modifications. Therefore, our experiments are performed on DAVIS-2016. It is important to note that the strength of our method lies in its ability to perform arbitrary granularity and multi-granular segmentation, while DAVIS-2016 contains only single objects without varying levels of granularity. Therefore, DAVIS-2016 does not fully capture the advantages of our approach. Nonetheless, we discussed this in Appendix; Section 7.3, where we also presented our method's performance in both semi-supervised and zero-shot settings. Despite most alternative methods being specifically trained for video segmentation using datasets like YouTubeVOS or DAVIS, our approach, which requires only a few frames for training, achieves competitive results.

**Limitations ($R_{aCvu},R_{9Vaw}$) :** SMITE has some limitations, which were discussed in Section 6 such as lower performance on smaller objects and also possibly inheriting the limitations of the tracking mechanism. We expanded that discussion in Section 6 and we reiterate it here. Firstly, as it relies on an off-the-shelf tracking technique, its tracking mechanism and voting system are constrained by the performance of this underlying tracking system. Secondly, the generation of WAS maps involves the use of low-resolution cross-attention mechanisms, which may result in reduced accuracy when segmenting fine details or small segments, such as thin lines or minor components. Thirdly, it cannot offer an interactive segmentation since it involves diffusion and optimization processes.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper addresses an interesting problem in the video-object segmentation (VOS) space. 

Most VOS methods assume an object query in the first video frame (queries can come in the form of segmentation masks, points, or bounding boxes), and the task is to segment the queried object in the remaining video frames. 

Different from that, this paper suggests a "query" that consists of a few images (e.g., three images) that contain an object class of interest (e.g., a horse), with object parts annotated. Given these, the proposed work segment any horse, and its parts, in any video. It can be understood as a few-shot video-object segmentation with image-level supervision. 

The method builds upon prior art for a few-shot image segmentation model (SLiMe) that builds upon a text-to-image generation model (Stable Diffusion). On top of that, this paper introduces a video segmentation method based on point tracking and a low-pass regularization term that helps maintain temporal consistency. Empirical validation on a newly introduced dataset, SMITE-50, shows that the proposed network outperforms SLiME baseline (applied per-frame), and several relevant few-shot/zero-shot VOS methods. 


Reviewers appreciate that the paper is well-written and organized, find the overall approach innovative, agree that it addresses a new problem in the video object segmentation space, and find that the proposed work has practical applications in the industry (especially VFX and video editing). Overall, there appears to be a consensus that this is a strong paper. 

Reviewers also provided constructive feedback. Reviewers note that the paper may not be easy to follow for readers who are not well-versed in the space of generative models or unfamiliar with stable diffusion or prior work this work builds upon (SLiMe). 

Reviewers also note that while a new dataset is always a welcome contribution, it is a rather small dataset, and comparisons using canonical VOS benchmarks (DAVIS&related) would strengthen this paper's contributions. 

Overall, there is a consensus that this is a good paper (ratings are 8, 8, 6, 6, 6). AC agrees with reviewers; the paper addresses an interesting new VOS task and an interesting model that successfully tackles this problem. 

While there was some criticism regarding the empirical validation (reviewers would like to see evaluation beyond the proposed dataset), reviewers agree that the evaluation does support the claims made in the paper. The authors also amended the paper after the rebuttal with additional baselines and results on the DAVIS dataset. 

AC suggests revising the paper to make it more accessible to people unfamiliar with SLiMe (the presentation already improved in rebuttal revision, but AC finds more space for streamlining the presentation). By having paper that is self-contained and explanations consolidated and simpler, it will reach a broader audience and have more potential for impact.  

The authors responded well to comments regarding dataset size compared to existing VOS datasets and showed that the dataset size is comparable to early VOS datasets. The table provided in the rebuttal does not include more recent and larger datasets (e.g., YT-VOS). Please add this to a revised paper version (it can be in the appendix).