SMITE: Segment Me In TimE

## Abstract

Abstract Segmenting an object in a video presents significant challenges. Each pixel must be accurately labeled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives . The project page is available at https://segment-me-in-time.github.io/ .

## 1 Introduction

Segmenting an object in a video poses a significant challenge in computer vision and graphics, frequently employed in applications such as visual effects, surveillance, and autonomous driving. However, segmentation is inherently complex due to variations in a single object (scale, deformations, etc.), within the object class (shape, appearance), as well as imaging (lighting, viewpoint). In addition, difficulty arises due to the segmentation requirements, such as its granularity (i.e., number of segments), as demanded by the downstream tasks. For example, in face segmentation, one VFX application might need to isolate the forehead for wrinkle removal, while another, such as head tracking, might treat it as part of the whole face. Creating a comprehensive dataset for every possible segmentation scenario to develop a supervised segmentation technique is extremely time-consuming and labor-intensive. Therefore, there is a need to segment images or videos based on a reference image. We call this type of segmentation *flexible granularity*.

When flexible granularity segmentation is applied on a large scale, it significantly improves downstream tasks such as VFX production, which involves managing numerous shots and videos. By segmenting one or a few reference images only once and then using those images to segment any video that features a target object of the same class, we can eliminate the need for separate segmentation of each video, thereby making the process far more efficient. In this paper, we tackle the challenge of video segmentation using one or few reference images that are not derived from the video frames themselves. For example, as shown in Fig. 1, a few annotated images are provided as references to our model, and our method, SMITE, successfully segments videos, exhibiting an object from the same class, in the same level of granularity. Importantly, none of the frames from these videos are included in the reference images, yet SMITE is capable of segmenting the videos with objects that exhibit different colors, poses, and even occlusions.
This is an important feature when working with large-scale videos requiring consistent segmentation (such as VFX videos needing the same enhancements), since there is no need for manual intervention to segment each video’s frames.

While recent work has explored flexible granularity segmentation for objects in images by leveraging the semantic knowledge of pretrained text-to-image diffusion models Khani et al. (), the complexity increases in videos. Ensuring label consistency across frames and managing instances where image segmentation may fail to produce accurate results require additional considerations.

To tackle these challenges and achieve consistent segmentation across frames, we utilize the semantic knowledge of pretrained text-to-image diffusion models, equipped with additional *temporal attentions* to promote temporal consistency. We also propose a *temporal voting* mechanism by tracking and projecting pixels over attention maps to maintain label consistency for each pixel. This approach results in segmentations with significantly reduced flickering and noise compared to per-frame segmentation methods while segments still follow the reference images thanks to our *low-pass regularization* technique that ensure preserving the structure of segments provided by attention maps and optimized according to the reference images.

Moreover, rather than simply optimizing a token for each segment Khani et al. (), we also *fine-tune cross-attentions* to enhance segmentation accuracy and better align with the reference images. Consequently, our method not only supports videos with temporal consistency but also outperforms flexible granularity image segmentation techniques in segmenting a single image.

We validate our design choices and methodology through comprehensive experiments detailed in the paper. As existing datasets with arbitrary semantic granularity are lacking, we introduce a small dataset, SMITE-50, to demonstrate the superior performance of our method against baselines. Additionally, we conduct user studies that highlight our method’s effectiveness in terms of segmentation accuracy and temporal consistency.

## 2 Related Work

#### Part-based semantic segmentation.

In computer vision, semantic segmentation, wherein a class label is assigned to each pixel in an image, is an important task with several applications such as scene parsing, autonomous systems, medical imaging, image editing, environmental monitoring, and video analysis (Sohail et al., ; He et al., ; Chen et al., ; Zhao et al., ; He et al., ; Chen et al., ; Sandler et al., ; Chen et al., ; Ravi et al., ). A more fine-grained derivative of semantic segmentation is semantic part segmentation, which endeavors to delineate individual components of objects rather than segmenting the entirety of objects.
Despite notable advancements in this domain (Li et al., ; ), a limitation of such methodologies is their reliance on manually curated information specific to the object whose parts they aim to segment. To solve the annotation problem, some works(Pan et al., ; Wei et al., ) proposed open-set part segmentation frameworks, achieving category-agnostic part segmentation by disregarding part category labels during training. Building on this, further works such as SAM (Kirillov et al., ), Grounding-SAM (Ren et al., ) explored utilizing foundation models to assist in open-vocabulary part segmentation. However, most of these methods can only segment the parts that are semantically described by text. With the influx of Stable Diffusion (SD) based generative segmentation approaches (Khani et al., ; Namekata et al., ), such issues have been partly solved by allowing SD features to segment semantic parts at any level of detail, even if they cannot be described by text. Despite such progress, applying such fine-grained segmentations on videos is challenging and unexplored. Our proposed SMITE presents the first part-segmentations in videos wherein it segments utilizing the part features from a pre-trained SD and generalizes it to any-in-the wild videos.

Video segmentation. Video segmentation methods can be categorized as video semantic segmentation (VSS) (Zhu et al., ; Zhang et al., ; ; Li et al., ; Ke et al., ; Wang et al., ), video instance segmentation (VIS) (Yang et al., ) and video object segmentation (VOS) (Xie et al., ; Wang et al., ; Cheng et al., ; ; Bekuzarov et al., ). VSS and VIS extends image segmentation to videos, assigning pixel labels across frames while maintaining temporal consistency despite object deformations and camera motion. VOS, in contrast, focuses on tracking and isolating specific objects throughout the video. Both tasks leverage temporal correlations through techniques like temporal attention (Mao et al., ; Wang et al., ), optical flow (Xie et al., ; Zhu et al., ), and spatio-temporal memory (Wang et al., ; Cheng & Schwing, ). Recent efforts, such as UniVS (Li et al., ), proposes unified models for various segmentation tasks, utilizing prior frame features as visual prompts. However, these methods struggle with fine-grained part segmentation and generalization to unseen datasets (Zhang et al., ). Bekuzarov et al. () leverage a spatio-temporal memory module and a frame selection mechanism to achieve high-quality video part segmentation with partial annotations. However, it requires frame annotations from the same video complicating video segmentation at scale. In contrast, we only need segmentation references for a few *arbitrary selected* images and we can segment an *unseen* given video respecting the segementation references. Therefore, per video manual annotation is not needed in our method.

Video diffusion models.
Recently, diffusion models (Ho et al., ; Song et al., ; ) have gained popularity due to their training stability and have been used in various text-to-image (T2I) methods (Ramesh et al., ; ; Saharia et al., ; Balaji et al., ), achieving impressive results.
Video generation (Le Moing et al., ; Ge et al., ; Chen et al., ; Cong et al., ; Yu et al., ; Luo et al., ) can be viewed as an extension of image generation with an additional temporal dimension.
Recent video generation models (Singer et al., ; Zhou et al., ; Ge et al., ; Nag et al., ; Cong et al., ) attempt to extend successful text-to-image generation models into the spatio-temporal domain by inflating the T2I UNet.
VDM (Ho et al., ) adopted this inflated UNet for denoising while LDM (Blattmann et al., ) implement video diffusion models in the latent space.
Video diffusion models can be categorized into inversion-based and inversion-free methods. Inversion-based approaches (Cong et al., ; Jeong & Ye, ) use DDIM inversion to control attention features ensuring temporal consistency, while inversion-free methods (Zhang et al., ) focus on more flexible conditioning, wider compatibility and better generation quality. However, inversion-free methods can suffer from flickering due to a lack of DDIM inversion guidance.
Recent works (Wang et al., ; Zhu et al., ) explored inversion-free T2V diffusion models to segment objects in videos, but they fail to generate fine-grained segments and often produce flickering segmentation results. Building on this, our method solves the seminal problem of video part-segmentation which combines an inversion-free model coupled with point-tracking algorithms (Karaev et al., ) to generate consistent, generalizable, flicker-free segmentations.

## 3 Preliminaries

Latent diffusion models and WAS maps.
Latent Diffusion Models perform the denoising operation on the latent space of a pretrained image autoencoder. Each latent pixel corresponds to a patch in the generated image. Starting from pure random noise $z_{T}$, at each timestep $t$, the current noisy latent $z_{t}$ is passed through a denoising UNet $\epsilon_{\theta}$, which is trained to predict the current noise $\epsilon_{\theta}(z_{t},y,t)$ using text prompt $y$.
In each block, the UNet employs residual convolution layers to generate intermediate features, which are then fed into attention layers. These attention layers average various values based on pixel-specific weights. The cross-attention layers (denoted by $A_{ca}$) incorporate semantic contexts from the prompt encoding, whereas the self-attention layers (denoted by $A_{sa}$) leverage global information from the latent representation itself.
SLiMe (Khani et al., ) demonstrated that text embeddings can be learned from a few images to segment other unseen images by leveraging both cross attention and self-attention layers in a novel representation called Weighted Accumulated Self-Attention (WAS) map, $S_{\text{WAS}}$.
This is defined as follows:

$$ $S_{\text{WAS}}=\text{Sum}(\text{Flatten}(R_{ca})\odot A_{sa}),$ (1) $$

where $R_{ca}$ is the downsampled latent of $A_{ca}$.

Inflated UNet.
A T2I diffusion model, such as LDM (Rombach et al., ), usually utilizes a U-Net(Ronneberger et al., ) architecture, which involves a downsampling phase, followed by an upsampling with skip connections. The architecture consists of layered 2D convolutional residual blocks, spatial attention blocks, and cross-attention blocks that incorporate textual prompt embeddings. To extend the T2I model for T2V tasks, the convolutional residual blocks and spatial attention blocks are inflated. Following earlier approaches (Cong et al., ; Wu et al., ), the $3\times 3$ convolution kernels in the residual blocks are adjusted to $1\times 3\times 3$ by introducing a pseudo temporal channel. To enhance the temporal coherence, we further extend the spatial self-attention mechanism to
the spatio-temporal domain. The original spatial self-attention method focused on patches within a single frame. However, in inflated UNet, we use all patch embeddings from the entire video as the queries, keys, and values. This allows for a full understanding of the video’s context. Additionally, we reuse the parameters from the original spatial attention blocks in the new dense spatio-temporal attention blocks.

## 4 Method

Here, we first introduce and formalize our method (SMITE) designed for achieving temporally consistent video segmentation with varying levels of granularity, guided by one or more reference images (Sec. 4.1). To achieve this and capture fine-grained segments, we first propose a new training strategy applied on the inflated UNet (Sec. 4.2).
The segmentation obtained from the inflated UNet may lack temporal consistency. To address this, we employ a voting mechanism guided by a tracking method that is projected onto the attention maps. However, relying solely on tracking to adjust the segments might lead to deviations from the provided samples. To mitigate this, we incorporate a frequency-based regularization technique to maintain detailed segmentations across frames (Sec. 4.3). Since tracking and frequency-based regularization may pull the segmentation in different directions, we use a energy based guidance optimization technique to balance both approaches (Sec. 4.4).

### 4.1 Problem Setting

Problem statement. Given one or few images, $\mathcal{I}_{n}\in\mathbb{R}^{H\times W\times 3}$, of a subject, along with its segment annotations $\mathcal{Y}_{n}=\{\mathcal{Y}^{i}_{n}|i=1:K\}$ where $\mathcal{Y}^{i}$ denotes binary segment mask $i$, and $n$ and $K$ are respectively the number of images and segments. Our objective is to learn temporally consistent segments of the subject for a given video $\mathcal{V}=\{v^{M}_{j=1}\}$ where $v_{j}$ represents video frames.

Figure: Figure 2: SMITE pipeline. During inference (a), we invert a given video into a noisy latent by iteratively adding noise. We then use an inflated U-Net denoiser (b) along with the trained text embedding as input to denoise the segments. A tracking module ensures that the generated segments are spatially and temporally consistent via spatio-temporal guidance. The video latent $z_{t}$ is updated by a tracking energy $\mathcal{E}_{track}$ (c) that makes the segments temporally consistent and also a low-frequency regularizer (d) $\mathcal{E}_{reg}$ which guides the model towards better spatial consistency.
Refer to caption: extracted/5950859/img/fig2_final_v4.png

Our framework.
We use Stable Diffusion’s (SD) semantic knowledge to learn the segments defined by few images and then generalize them to the video of the subject in any pose, color or size. This implies that the model needs to share information across frames to enforce temporal consistency.
Differently from the UNet structure used in Khani et al. (),
we apply an inflation to the T2I models across the temporal dimension (Wu et al., ) to enable temporal attention across all video frames.
Also, we incorporate a tracking module combined with low-frequency regularization to enhance spatio-temporal consistency across frames. The overall inference pipeline of our model, SMITE, is illustrated in Fig. 2.

### 4.2 Learning generalizable segments

We first learn segments provided by the reference images of the subject by optimizing a text embedding for each segment of the reference images that can be used for segmenting the given videos. We also fine tune the cross attentions of the SD to better match the provided segments since the text embeddings alone may not be able to fully capture the masks’ details.

Learning text embeddings.
We begin by passing the reference images $\mathcal{I}$ and text embeddings $\mathcal{T}$ into into SMITE (denoted by $\psi(.)$) . Similar to SLiMe (Khani et al., ), we obtain image resolution WAS maps (denoted by $\mathcal{S}$) from our inflated UNet as follows:

$$ $\mathcal{S}=\psi_{\theta}(\mathcal{I},\mathcal{T})\hskip 7.22743pt\text{,} \hskip 7.22743pt\hat{Y}=argmax(\mathcal{S}),$ (2) $$

where $\theta$ represents the learnable parameters of the model and $\mathcal{S}$ is defined the same as Eq. 1. Text embeddings $\mathcal{T}$, which are initialized randomly or with the names of segments, correspond to segment masks $\mathcal{S}=[S^{1}_{\text{WAS}},S^{2}_{\text{WAS}},...,S^{K}_{\text{WAS}}]$, and $\hat{Y}$ is the segmentation output.
Since the inflated UNet is designed for videos, we pass reference images in $\mathcal{I}$ to $\psi(.)$ as videos with a single frame. Together with the ground-truth mask $\mathcal{Y}$, we find the optimized text embeddings, $\mathcal{T}^{*}$, that are correlated with the segments in $\mathcal{S}$.

[Uncaptioned image]: extracted/5950859/img/fig3_v7.png

Network fine-tuning.
Only learning text embeddings fails to capture complex granularities. To extract more customizable fine-grained segments from the video, we need to fine-tune the existing SD weights (denoted by $\theta$) using the available provided segment annotations. As shown in the inset figure, optimizing solely the text embeddings struggles with asymmetrical segmentation (e.g., segmenting only one eye).
We hypothesize that such issues arises because the model may get stuck in local minima when relying exclusively on text embeddings for optimization. To mitigate this, we update the cross-attention layers $A_{ca}$ in the UNet along with the text-embedding but do so in two phases. First, the model is frozen, while the text embeddings corresponding to the segments denoted by $\mathcal{T}$ are optimized using a high learning rate. Thus, an initial embedding ($\mathcal{T}^{*}$) is achieved quickly without detracting from the generality of the model, which then serves as a good starting point for the next phase.
Second, we unfreeze the cross-attention weights and optimize them along with the text embeddings, using a significantly lower learning rate. This gentle fine-tuning of the cross-attention and the embeddings enables faithful generation of segmentation masks with varied granularity.
We use the same combination of losses ($\mathcal{L}_{\text{CE}}$, $\mathcal{L}_{\text{MSE}}$ and $\mathcal{L}_{\text{LDM}}$) used in SLiMe (Khani et al., ) for network finetuning.
This results in an optimized SMITE model (denoted by $\Psi_{\theta}^{*}$) which can generate segments that is generalizable across different videos.

### 4.3 Temporal Consistency

To enhance temporal consistency, we use temporal attention in SMITE’s Inflated UNet denoiser. While it improves temporal consistency compared to independent frame processing (Fig. 3(a)), inconsistencies remain due to the need for segmenting at different granularities, often with imprecise boundaries. These inconsistencies can cause flickering or unnatural transitions (Fig. 3(b)).

Figure: Figure 3: Best viewed in Adobe Acrobat.

Figure: Figure 4: Segment tracking module ensures that segments are consistent across time. It uses co-tracker to track each point of the object’s segment (here it is nose) and then finds point correspondence of this segment (denoted by blue dots) across timesteps. When the tracked point is of a different class (e.g,. face) then it is recovered by using temporal voting. The misclassified pixel is then replaced by the average of the neighbouring pixels of adjacent frames. This results are temporally consistent segments without visible flickers.
Refer to caption: extracted/5950859/img/fig4_smite_v3.png

Segment tracking and voting.
The first step to ensure segment consistency is tracking the segments across time. Point tracking methods like CoTracker (Karaev et al., ) are well-suited for our approach because they use point correspondences to minimize pixel drift over time. However, since our segments come from attention maps, tracking needs to occur directly on these maps. Since CoTracker is trained on spatial domains, we first apply it on the frames of video $\mathcal{V}$ using a sliding window, project the results onto the attention maps via simple scaling, and then update the attention maps accordingly. A pixel’s label in the attention map is updated based on the most frequent label it receives across the visible frames in the window.
Formally, for a pixel with coordinates $(x_{t},y_{t})$ on the frame latent at timestep $t$, its coordinates on all subsequent frame latents in the window can be derived from the tracker. The coordinates are linked, and the trajectory sequence can be presented as:

$$ $\{(x_{t-\frac{w}{2}},y_{t-\frac{w}{2}}),...,(x_{t},y_{t}),...,(x_{t+\frac{w}{2 }},y_{t+\frac{w}{2}})\}=\phi(\mathcal{P}(\mathcal{V})),\hskip 7.22743pt$ (3) $$

where $\mathcal{P}$ is the tracker, $\phi(.)$ is the scaling operator, and $w$ denotes the length of the sliding window. We choose the point corresponding to the center frame of the window $(x_{t},y_{t})$ as the first point and track bidirectional correspondences within the window. Each of these trajectory points $(x,y)$ is assigned a segment label $\hat{Y}$. However, certain
pixels in the window may disappear and reappear over time as shown in Fig 4. To handle such scenarios, we discard labels for which the tracking pixel is invisible and then use temporal voting to update its segment labels. Therefore, we update the WAS maps (denoted by $\mathcal{S}_{\text{WAS}}$) corresponding to the tracking pixel $(x_{t},y_{t})$ and use the following formula to assign the labels:

$$ $\mathcal{S}_{\text{Tracked}}(x_{t},y_{t})=\text{Avg}(\mathcal{S}(x_{l},y_{l}) \mid\hat{Y}(x_{l},y_{l})=F)\hskip 7.22743pt\forall l\in(t-\frac{w}{2},t+\frac{ w}{2}),$ (4) $$

where $F$ denotes the most frequent label in the window.
After correcting the unstable pixel tracking over multiple-step denoising, we obtain a consistent set of segments ($\mathcal{S}_{tracked}$). Note that the window size $w$ is small (e.g, seven), hence to track the segments for the entire video, we need to slide the windows and track in an iterative fashion.
More details are provided in the supplementary material.

Intuitively, if we would have relied only on cross-attention for segmentation, we could have adjusted it directly to align with the tracking output. However, since segments $\mathcal{S}$ are derived from the combination of cross and self-attention, direct manipulation is challenging. To tackle this, we propose an energy function to measure how well the segments before tracking (denoted by $\mathcal{S}$) align with the segments after correcting (denoted by $\mathcal{S}_{tracked}$) it using temporal voting.
This energy is defined as:

$$ $\mathcal{E}_{\text{Tracking}}=\textit{CE}(\text{S},\text{S}_{\text{tracked}}),$ (5) $$

where $CE(.)$ represents cross-entropy objective. This optimization ensures that the segment drift is corrected throughout the video to make it temporally consistent with low flicker.

Low-pass regularization.
When tracking is applied, label modifications should be repeated through several denoising steps to ensure full temporal label propagation and achieve better consistency. This process may cause deviations (Fig. 3) from the original segmentation provided by the reference images that is captured in the segments ($\mathcal{S}$) in the initial denoising step.
Our goal is to preserve the overall structure of the segments $\mathcal{S}$ from the WAS maps while smoothing boundary transitions for temporal consistency via tracking. Low Pass Filters (LPF) have been effective in video generation to maintain temporal correlations and spatial frame structure (Wu et al., ; Si et al., ).
Therefore, we use an LPF on the finally denoised segments $\mathcal{S}$ ensuring the structure of the LPF on the segments $\mathcal{S}$ predicted in the initial denoising step (denoted by $\mathcal{S}_{ref}$) is respected by the following function:

$$ $\mathcal{E}_{\text{Reg}}=\left\|\omega(\mathcal{S})-\omega(\mathcal{S}_{ref}) \right\|_{1}$ (6) $$

where $\omega(S)=\mathcal{H}_{l}\odot\text{DCT}_{3D}(\text{S})$, $\text{DCT}_{3D}$ refers to Discrete Cosine Transform and $\mathcal{H}_{l}$ refers to an LPF.
This ensures that the segments structure is progressively corrected across the video frames at the end of the denoising phase as illustrated in Fig. 3.

### 4.4 Spatio-Temporal Guidance

Our overall energy function is minimized by backpropagating through the diffusion process, as described in Chen et al. (); Safaee et al. (), which updates the latent representation to achieve more consistent segmentation over time:

$$ $\mathcal{E}_{\text{Total}}=\lambda_{\text{Tracking}}\cdot\mathcal{E}_{\text{ Tracking}}+\lambda_{\text{Reg}}\cdot\mathcal{E}_{\text{Reg}},$ (7) $$

where $\lambda_{\text{Tracking}}$ and $\lambda_{\text{Reg}}$ are the coefficients for the tracking and reference loss functions, respectively.
Optimizing this function encourages better spatial and temporal consistency in the WAS attention maps (S). Specifically, we update the latent $z_{t}$ using gradient descent on $\mathcal{E}_{\text{Total}}$:

$$ $z_{t}^{\prime}\leftarrow z_{t}-\alpha_{t}\cdot\nabla_{z_{t}}\mathcal{E}_{Total},$ (8) $$

where $\alpha_{t}$ is the learning rate. Results indicate that this strategy leads to superior performance compared to when we either omit the voting mechanism or do not apply low-pass regularization.

## 5 Results and Experiments

Dataset and benchmark.
To evaluate our method, we introduce a benchmark dataset called SMITE-50, primarily sourced from [Pexels](www.pexels.com). SMITE-50 features multi-granularity annotations and includes visually challenging scenarios such as pose changes and occlusions. To our knowledge, no existing datasets focus exclusively on multi-granular and multi-segment annotations. While the PumaVOS dataset Bekuzarov et al. () contains a limited number of multi-part annotated videos across a diverse range, it does not offer multiple videos for specific granularities and categories.

Figure: Figure 5: SMITE-50 Dataset sample.
Refer to caption: extracted/5950859/img/SMITE_50_teaser.png

We focus on three main categories: (a) Horses, (b) Human Faces, and (c) Cars, encompassing 41 videos. Each subset includes ten segmented reference images for training and densely annotated videos for testing. The granularity varies from human eyes to animal heads, etc. relevant for various applications such as VFX (see Fig. 5). All segments are labeled consistently with the part names used in existing datasets.
Additionally, we provide nine challenging videos featuring faces with segments that cannot be described textually, as shown in Fig. 3 (Non-Text). Overall, our dataset comprises 50 video clips, each at least five seconds long. For dense annotations, we followed a similar approach to (Ding et al., ; Bekuzarov et al., ), creating masks for every fifth frame with an average of six parts per frame across three granularity types (more info in Appendix). While PumaVOS dataset Bekuzarov et al. () has 8% annotations, our SMITE-50 dataset has 20% dense annotations.

Evaluation protocol.
In our setting, few reference images per class are used to train SMITE and then it is evaluated on videos from the same categories but not the same objects in the training data. For all the cases, we report the standard metrics (higher is better): Mean Intersection Over Union (mIOU), and Contour Accuracy $F_{measure}$ respectively.

Figure: Figure 6: Visual comparisons with other methods demonstrate that SMITE maintains better motion consistency of segments and delivers cleaner, more accurate segmentations. Both GSAM2 and Baseline-I struggle to accurately capture the horse’s mane, and GSAM2 misses one leg (Left), whereas our method yields more precise results. Additionally, both alternative techniques create artifacts around the chin (Right), while SMITE produces a cleaner segmentation.
Refer to caption: extracted/5950859/img/quali_v2.png

Quantitative comparison.
Since there are no available few-shot video part segmentation methods that can be applied to our setting, we build the following two state-of-the-art baselines for quantitative comparison: (1) Few-shot Image segmentation based approach, in which SLiMe (Khani et al., ) is applied to the video frame-by-frame. We term this as Baseline-I. (2) Grounded SAM2 (Ren et al., ; Ravi et al., ), a recently introduced zero-shot foundation model for video segmentation approach which is applied directly on the video. For Baseline-I, we applied the same training procedure and used SMITE-50 dataset.
As apparent in Tab. 1, our method produces the most accurate results in comparison with other methods across all categories and all metrics.

Generally, XMem++ (Bekuzarov et al., ) cannot be utilized in our setting since we do not want to use any frames from the provided video. However, in the Appendix, we included a comparison with XMem++ on a subset of their proposed dataset PUMaVOS with flexible granularity to provide a contrast and show that our method is effective on datasets beyond SMITE-50. Although XMem++ is a semi-supervised technique, our method performs comparably and even outperforms it with fewer frames (e.g., a single shot). When XMem++ is given more frames per video (e.g., 10), its performance slightly surpasses ours, which is expected since SMITE does not require any substantial video pre-training. However, with a smaller number of frames (e.g., one or five), our method either outperforms or matches the performance of XMem++. Additionally, we report our performance on the image segmentation task in the Appendix, showing that we outperform SLiME (Baseline-I), highlighting the effectiveness of our design choices, including the cross attention optimization.

**Table 1: Quantitative evaluation on SMITE-50 dataset. The results are presented for each category (Face, Horse, Car, Non-Text) having 10 reference image during training.**
|  | Faces | Horses | Cars | Non-Text |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Methods | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU |
| Baseline-I | 0.81 | 72.95 | 0.64 | 65.48 | 0.57 | 61.38 | 0.67 | 66.69 |
| GSAM2 | 0.73 | 63.28 | 0.76 | 72.76 | 0.64 | 63.56 | - | - |
| Ours | 0.89 | 77.28 | 0.79 | 75.09 | 0.82 | 75.10 | 0.77 | 73.08 |

Qualitative comparisons.
Qualitative comparisons are presented in Fig. 6. GSAM2 struggles
to locate the boundaries accurately and produces coarse segments.
SLiMe can usually segment the first
frame accurately but struggles to preserve temporal consistency. Our method produces the best segmentation maps in terms of segmentation quality and temporal consistency. The
maps have sharper boundaries and clean clusters compared to other methods. Our SMITE performs
better than its SLiMe counterpart.
In Fig. 7, we present results from other categories, demonstrating the versatility of our model in handling various object categories. As shown, despite significant differences in pose, expression, gender, and other properties between the video and the annotated images, our method still delivers high-quality results. This is particularly evident in the pineapple example, where the object is cut in half, yet the method successfully tracks it and produces accurate segmentation. For more challenging cases, we include results from additional difficult videos in Fig. 8. In one instance, the ice cream cone is occluded by a paper napkin, and the ice cream itself is obscured and blended by a face, yet SMITE is still able to generate correct results. Furthermore, the turtle nearly blends with the background in terms of color and visual patterns, but our method successfully tracks the segments.

Ablation study.
To demonstrate the effectiveness of our design choices, we conducted three ablation studies. The first ablation on the Car dataset (Tab. 3), illustrates that combining $\mathcal{E}_{\text{Tracking}}$ and $\mathcal{E}_{\text{Reg}}$ improves the results. It’s important to note that tracking without low pass regularization leads to unsatisfactory outcomes, as results tend to align too closely with the voting method. Tab. 3 shows the impact of cross attention tuning also on the Car dataset. By optimizing both cross attention and text embedding, we achieve the highest accuracy, demonstrating the effectiveness of our tuning strategy. We have also ablated the number of images for training our few shot setting $k=1/5/10$. While our performance improves with more training images, our one shot setting still performs reasonably well (Tab. 5).

**Table 2: Loss ablation. When two losses are combined, the best performance is achieved.**
| $\mathcal{E}_{\text{Tracking}}$ | $\mathcal{E}_{\text{Reg}}$ | mIOU |
| --- | --- | --- |
| ✓ | ✗ | 69.85 |
| ✓ | ✓ | 75.10 |

User study.
We conduct a user study because human judgment is best for assessing perception when both temporal consistency and faithfulness to segmentations are important. We collected 16 segmented videos (4 from Non-Text category) and asked 40 participants to rank the methods (i.e., 1 best and 3 worst) based on *segmentation quality* (i.e., fidelity to the segmentation reference) and *motion consistency* (i.e., reduced flicker), encouraging them to prioritize segmentation quality in their evaluations in terms of a tie in motion consistency. We do not report the scores of GSAM2 on Non-Text segments as the segments in the videos are not describable by texts. Tab. 5 demonstrates that SMITE achieves higher preference both for textual and non-textual segmentation.

**Table 4: Few shot ablation on cars in SMITE-50. The performance increases with more training images but still performs well in one shot setting.**
| Training sample # | mIOU |
| --- | --- |
| 1-shot | 63.03 |
| 5-shot | 71.55 |
| 10-shot | 75.10 |

## 6 Conclusion

In this work, we introduce SMITE, a video segmentation technique that supports flexible granularity segmentation of objects within a video. SMITE leverages the semantic knowledge of a pre-trained text-to-image diffusion model to segment a video with minimal additional training (i.e., only a few images), while utilizing the temporal consistency of the video to maintain motion consistency within the segments. Notably, SMITE can be trained with a few images that are not necessarily from the video, yet it effectively segments objects within an unseen video.
To better show the capabilities of our method, we also collected a flexible granularity dataset called SMITE-50 that will be publicly available along with our code. Through various quantitative and qualitative experiments, as well as user studies, we demonstrate the effectiveness of our method and its components. However, our method still faces limitations that motivate future work. Specifically, SMITE does not perform well when the target objects or segments are too small, and its performance degrades when dealing with with low video resolution. Addressing these issues presents interesting avenues for future research. Additionally, while we use Co-tracker to track segments, exploring other tracking systems and evaluating their performance would be valuable.

Figure: Figure 7: Additional results. We visualize the generalization capability of SMITE model (trained on the reference images) in various challenging poses, shape, and even in cut-shapes.
Refer to caption: extracted/5950859/img/fig8_smite.png

Figure: Figure 8: Segmentation results in challenging scenarios . SMITE accurately segments out the objects under occlusion (”ice-cream") or camouflage (”turtle") highlighting the robustness of our segmentation technique.
Refer to caption: x2.png

## 7 Appendix

### 7.1 SMITE-50 Dataset

SMITE-50 is a video dataset that covers challenging segmentation with multiple parts of the object is being segmented in difficult scenarios like occlusion. It consists of 50 videos, up to 20 seconds long, from 24 frames to 400 frames with different aspect ratios (both vertical and horizontal). Frame samples of the dataset are provided in Fig.5 in the main paper. Primarily our dataset consists of 4 different classes "Horses", "Faces", "Cars" and "Non-Text". Out of these sequences, "Horses" and "Cars" have videos which have been captured outdoors hence it has challenging scenarios including occlusion, view-point changes and fast-moving objects with dynamic background scenes. "Faces" sequence on the other hand have videos with occlusion, scale changes and more fine-grained parts which are difficult to track and segment across time. The "Non-Text" category has videos which have parts that cannot be described by natural language cues.
Hence, these videos are difficult for zero-shot video segmentation (Ren et al., ) approaches as most of the models are reliant on textual vocabularies for the segmentations.
SMITE-50 is a working progress and we intend to expand it beyond what it currently includes and make it publicly available.

### 7.2 Extra Ablations

Performance on PUMaVOS and comparison with XMem++. Here, we provide a comparison with XMem++ on
their proposed dataset PUMaVOS to show that our method is effective on other datasets than our proposed SMITE-50.
For this evaluation, we considered the following seven video splits namely :Chair, Full face 1, Full face 2, Half face 1, Half face 2, Long Scene scale and Vlog repectively. We chose the following categories as the object parts in these videos are not too small and also have flexible granularity.
Note that XMem++ is a semi-supervised video segmentation technique meaning that it has been trained on largescale video segmentation datasets such as DAVIS (Pont-Tuset et al., ) or YoutubeVOS (Xu et al., ). Nevertheless, SMITE performs comparably and even outperforms it in case of working with fewer frames (e.g., a single shot; see Tab. 6). Predictably, when XMem++ is provided with more frames per video (e.g., 10), its performance slightly exceeds ours since SMITE is not trained on a large segmentation dataset. However, with a smaller number of frames (e.g., one or five), our method either outperforms or matches the performance of XMem++. See Tables 6 and 7, for the quantitative comparisons.

**Table 6: Comparison on a subset of PUMaVOS dataset (Bekuzarov et al. ()) when only one frame is used for training.**
|  | Chair | Full face 1 | Full Face 2 | Half Face 1 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU |
| GSAM2 | 0.49 | 58.82 | 0.99 | 97.47 | 0.94 | 94.78 | 0.29 | 57.66 |
| Baseline-I | 0.46 | 73.15 | 0.61 | 85.23 | 0.7 | 86.9 | 0.02 | 82.83 |
| XMem++ | 0.99 | 95.72 | 0.71 | 90.75 | 0.80 | 89.92 | 0.82 | 90.52 |
| Ours | 0.32 | 63.32 | 0.98 | 96.46 | 0.85 | 90.38 | 0.55 | 79.75 |
|  | Half Face 2 | Long Scene Scale | Vlog | Mean |  |  |  |  |
| Method | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU | $\textbf{F}_{meas.}$ | mIOU |
| GSAM2 | 0.54 | 74.78 | 0.99 | 97.39 | 0.16 | 42.99 | 0.63 | 74.84 |
| Baseline-I | 0.18 | 55.78 | 0.74 | 87.74 | 0.73 | 78.90 | 0.5 | 74.91 |
| XMem++ | 0.48 | 71.03 | 0.87 | 95.48 | 0.16 | 31.11 | 0.69 | 80.65 |
| Ours | 0.37 | 69.91 | 0.98 | 96.27 | 0.75 | 78.91 | 0.69 | 82.14 |

**Table 7: Quantitative results on a subset of PUMaVOS dataset (Bekuzarov et al. ()). At $k=1$, we achieve higher quality in terms of $\textbf{F}_{measure}$ and mIoU, outperforming XMem++. For other settings, we experience a small margin of loss, which is acceptable given that XMem++ is a fully supervised method, whereas our approach is a few-shot method.**
| Methods | 1 frame | 5 frames | 10 frames |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | $\textbf{F}_{meas.}$ | mIoU | $\textbf{F}_{meas.}$ | mIoU | $\textbf{F}_{meas.}$ | mIoU |
| Full Face 1 (XMem++) | 0.71 | 90.75 | 1.0 | 98.78 | 1.0 | 99.01 |
| Full Face 1 (Ours) | 0.98 | 96.46 | 0.99 | 96.76 | 1.0 | 96.73 |
| Full Face 2 (XMem++) | 0.80 | 89.92 | 0.96 | 96.64 | 0.97 | 97.35 |
| Full Face 2 (Ours) | 0.85 | 90.38 | 0.91 | 93.10 | 0.93 | 93.78 |
| Chair (XMem++) | 0.99 | 95.72 | 1.0 | 96.57 | 1.0 | 96.65 |
| Chair (Ours) | 0.32 | 63.32 | 0.98 | 90.62 | 0.99 | 89.82 |
| Half Face 1 (XMem++) | 0.82 | 90.52 | 0.94 | 94.54 | 0.96 | 95.49 |
| Half Face 1 (Ours) | 0.55 | 79.75 | 0.92 | 90.69 | 0.93 | 91.37 |
| Half Face 2 (XMem++) | 0.48 | 71.03 | 0.77 | 87.87 | 0.85 | 91.41 |
| Half Face 2 (Ours) | 0.37 | 69.91 | 0.66 | 81.06 | 0.83 | 87.17 |
| Long Scene Scale (XMem++) | 0.87 | 95.48 | 0.99 | 98.36 | 1.0 | 98.91 |
| Long Scene Scale (Ours) | 0.98 | 96.27 | 1.0 | 96.87 | 1.0 | 96.79 |
| Vlog (XMem++) | 0.16 | 31.11 | 0.55 | 62.84 | 0.82 | 82.52 |
| Vlog (Ours) | 0.75 | 78.91 | 0.86 | 84.01 | 0.90 | 85.29 |
| Mean (XMem++) | 0.69 | 80.65 | 0.89 | 90.80 | 0.94 | 94.48 |
| Mean (Ours) | 0.69 | 82.14 | 0.90 | 90.44 | 0.94 | 91.56 |

### 7.3 Extra Qualitative Results

We have included qualitative comparison videos in our project webpage to provide more examples in video format. The actual videos are in higher resolution, we have compressed the video to upload it on the website easily.

### 7.4 Implementation Details and Design Choices Explanation

All experiments were conducted on a single NVIDIA RTX 3090 GPU. During the learning phase, we initially optimized only the text embeddings for the first 100 iterations. For the subsequent iterations, we optimized the cross-attention to_k and to_v parameters.

For the categories of horses, cars, faces and non-text, we used 10 reference images from our SMITE-50 benchmark for both our method and the baseline comparisons.

Regarding the window size in our tracking module, we found that fast-moving objects benefit from a smaller window size to mitigate potential bias. Consequently, we set the window size to 7 for horses and 15 for other categories.
To better work on smaller objects, we employ a two-step approach. First, our model generates an initial segmentation estimate. We then crop the image around this initial estimate and reapply our methods to obtain a more fine-grained segmentation.

During inference, we added noise corresponding to 100 timesteps and performed a single denoising pass when segment tracking and voting were not employed. When using segment tracking and voting, we applied spatio-temporal guidance at each denoising step and conducted backpropagation 15 times per denoising timestep. For the regularization parameters, we set $\lambda_{\text{Reg}}$ across all experiments. The tracking parameter $\lambda_{\text{Tracking}}$ was set to 1 for horses, 0.5 for faces, and either 0.2 or 1 for cars. Additionally, we applied a Discrete Cosine Transform (DCT) low-pass filter with a threshold of 0.4.

Pseudo Code of Temporal Voting

Figure: Algorithm 1 Temporal Voting

Bidirectional Tracking and Resolution Reduction in CoTracker
Note that points queried at different frames are tracked incorrectly before the query frame. This is because CoTracker is an online algorithm and only tracks points in one direction. However, we can also run it backward from the queried point to track in both directions. So by setting $backward_tracking$ to True we are able to track points in both directions which is crucial for our voting mechanism. Also explain 512*512 and reduce size to 64*64.

Long video processing.
A key goal for us is to adapt our method to work efficiently on smaller GPUs. Unlike most video editing techniques that require high-end hardware like an A100 GPU and typically handle up to 24 frames, our approach aims for broader applicability. We identified that gradient computation during inference optimization is particularly demanding on resources. To address this, we segment the latent space into smaller windows—for example, (1 to k), (k+1 to 2k), (2k+1 to 3k), and so on—across different timesteps and optimize each window independently. This segmentation has proven not to compromise the final results.
Additionally, for tasks such as segmenting 200 frames consistently, our method allows processing in batches of 20 frames at a time. Then we save the last 7 frames WAS and replace them with the WAS of reference in the new iteration.
These strategies enable our model to process longer videos effectively on GPUs with as little as 24 GB of VRAM.

Enhanced Convergence.
The strategy to accelerate convergence and simplify parameter tuning in the code involves the use of an Adam-like optimization approach that dynamically adapts the learning rate and gradient updates for the latent variables. Specifically, the code implements the first and second moment estimates, denoted as $M1$ and $M2$, which accumulate the gradients and squared gradients, respectively.

In each iteration, the first moment estimate $M1$ captures the exponentially weighted average of the gradients, while the second moment estimate $M2$ tracks the squared gradients. These moment estimates are then bias-corrected by dividing them by $1-\beta_{1}^{t+1}$ and $1-\beta_{2}^{t+1}$, where $\beta_{1}$ and $\beta_{2}$ are the momentum parameters typically set to 0.9 and 0.999, respectively. This bias correction ensures that the estimates are unbiased, particularly in the initial steps of optimization.

The learning rate $\alpha_{t}$ is dynamically scaled based on these corrected moment estimates, where the update step for the latent variables is computed as:

$$ $z_{t}^{\prime}\leftarrow z_{t}-\alpha_{t}\cdot\frac{M1_{\text{corrected}}}{ \sqrt{M2_{\text{corrected}}}+\epsilon}$ $$

This adaptive approach allows the optimizer to adjust the learning rate on a per-parameter basis, depending on the variance of the gradients, leading to faster convergence. By using this method, the optimizer can take larger steps when the gradients are consistent and smaller steps when they are noisy, which helps in avoiding overshooting or getting stuck in local minima. The combination of momentum-based updates and dynamic learning rate scaling makes the optimization process more robust, reducing the need for extensive manual tuning of hyperparameters such as the learning rate, and enabling more efficient convergence.

Bidirectional tracking.
We use bidirectional tracking instead of unidirectional tracking for two main reasons. First, to manage longer videos, we implement a slicing approach where the last frames of the first slice are retained in the second slice to ensure continuity. Bidirectional tracking speeds up consistency between slices by allowing new frames in the second slice to directly reference frames from the first slice, while unidirectional tracking delays this process due to the need for updates to propagate. Second, tracking methods often struggle with fast-moving objects, as accuracy decreases with distance from the query pixel, risking loss of track. Bidirectional tracking enhances robustness in these situations. Additionally, in cases of occlusion, unidirectional tracking may fail without visible pixels to propagate information. Bidirectional tracking mitigates this by leveraging data from both past and future frames, maintaining accuracy even during occlusions.

### 7.5 Image Segmentation Results

We tested our method on an image dataset (e.g PASCAL-Part(Chen et al., )) to demonstrate the enhancements achieved through modifications to our architecture and optimization. As shown in Tables 8 and 9, our approach shows significant improvement over SLiMe even for image segmentation for car and horse split of PASCAL-Part dataset. This highlights the effectiveness of our design choices in SMITE, particularly the cross-attention tuning.

**Table 8: Image segmentation results for class car SMITE consistently outperforms SLiMe. The first two rows show the supervised methods, for which we use the reported numbers in ReGAN. The second two rows show the methods with 1-sample setting and the last three rows refer to the 10-sample setting methods. ^⋆ indicates the supervised methods.**
|  | Body | Light | Plate | Wheel | Window | Background | Average |
| --- | --- | --- | --- | --- | --- | --- | --- |
| $\text{CNN}^{\star}$ | 73.4 | 42.2 | 41.7 | 66.3 | 61.0 | 67.4 | 58.7 |
| $\text{CNN+CRF}^{\star}$ | 75.4 | 36.1 | 35.8 | 64.3 | 61.8 | 68.7 | 57.0 |
| $\text{SegGPT \cite[citep]{(\@@bibref{AuthorsPhrase1Year}{wang2023seggpt}{<br>\@@citephrase{, }}{})}}^{\star}$ | 62.7 | 18.5 | 25.8 | 65.8 | 69.5 | 77.7 | 53.3 |
| OIParts (Dai et al., ) | 77.7 | 59.1 | 57.2 | 66.9 | 59.2 | 71.1 | 65.2 |
| ReGAN (Tritrong et al., ) | 75.5 | 29.3 | 17.8 | 57.2 | 62.4 | 70.7 | 52.15 |
| SLiMe (Khani et al., ) | 81.5 | 56.8 | 54.8 | 68.3 | 70.3 | 78.4 | 68.3 |
| Ours | 82.3 | 57.5 | 55.9 | 70.1 | 72.6 | 80.1 | 69.8 |

**Table 9: Image segmentation results for class horse. SMITE outperforms ReGAN, OPParts, SegDDPM, SegGPT and SLiMe on average and most of the parts. The first two rows show the supervised methods, for which we use the reported numbers in ReGAN. The middle two rows show the 1-sample setting, and the last four rows are the results of the 10-sample settings. ^⋆ indicates the supervised methods.**
|  | Head | Leg | Neck+Torso | Tail | Background | Average |
| --- | --- | --- | --- | --- | --- | --- |
| $\text{Shape+Appereance}^{\star}$ | 47.2 | 38.2 | 66.7 | - | - | - |
| $\text{CNN+CRF}^{\star}$ | 55.0 | 46.8 | - | 37.2 | 76 | - |
| $\text{SegGPT \cite[citep]{(\@@bibref{AuthorsPhrase1Year}{wang2023seggpt}{<br>\@@citephrase{, }}{})}}^{\star}$ | 41.1 | 49.8 | 58.6 | 15.5 | 36.4 | 40.3 |
| OIParts (Dai et al., ) | 73.0 | 50.7 | 72.6 | 60.3 | 77.7 | 66.9 |
| ReGAN (Tritrong et al., ) | 50.1 | 49.6 | 70.5 | 19.9 | 81.6 | 54.3 |
| SegDDPM (Baranchuk et al., ) | 41.0 | 59.1 | 69.9 | 39.3 | 84.3 | 58.7 |
| SLiMe (Khani et al., ) | 63.8 | 59.5 | 68.1 | 45.4 | 79.6 | 63.3 |
| Ours | 64.5 | 61.9 | 73.2 | 48.1 | 83.5 | 66.2 |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: - Is the proposed method also applicable to "Multi-object flexible granularity", which indicates that reference images annotated with multiple objects (or multiple classes) different granularity can be used as input to the proposed model?

- Could the author provide any failure cases?
soundness: 3
strengths: Belows are the strong points that this paper has:

- It is well-written and well-organized, which help reader to understand new task and motivation of the proposed method.

- The proposed techniques including learning generalizable segments using alternative tuning of text embeddings and cross attention, temporal consistency via tracking voting mechanism, and spatial consistency with low-pass filter are in good harmony, which can outperform the other baselines with high margin.
summary: This paper defines a task of flexible granularity, which aims to segment user-intended granularity temporally using one or a few reference images. To do so, it proposes several techniques to achieve temporal consistent multi granularity segmentation including alternative learning of text embeddings and cross attention module, tracking modules with voting mechanism to be aware of neighbor pixels and low frequency regularizer for spatial consistency. It shows its effectiveness on the proposed dataset, SMITE-50
weaknesses: Belows are points that the reviewer feel concerned about.

- In line #247, the authors mention that learning text embeddings and cross-attention layers is conducted in two phases to provide a better initialization for the next phase. However, this phased or alternating training approach could also increase training complexity. Could the authors provide an ablation study to compare the results of joint training versus alternating training to justify the chosen training setup? Specifically, analysis on training time with final performance of two training strategies can be provided.

- In the supplementary video, specifically at 1:46 in the "Qualitative_Comparison" section, it appears that the proposed method captures a relatively smaller region of the "eye" area compared to the baseline. Is this due to the proposed voting mechanism diluting the effective area by averaging nearby major segments? Would this issue be mitigated by replacing the current hard voting with a soft voting mechanism? Is it doable that quantitative analysis on different voting system can be measured?

- Is the proposed method shape-agnostic within the same class? To demonstrate that the method can segment other unseen images, the authors should provide test samples that differ in shape from the reference object. For example, as similar to what the author showed in "Cars" of Figure 7, if the reference image includes a detailed segmentation map of a "sedan" car, the test samples can be replaced with different shapes of cars, such as "SUVs?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. How sensitive is the method's performance to the choice of tracking algorithm? Have you experimented with other tracking methods besides CoTracker?
1. The low-pass regularization is mentioned as a key component. Can the authors elaborate on how this regularization term is formulated and how it balances preserving segment structures with smoothing transitions?
1. How does the proposed method handle videos with rapid movements, severe occlusions, or significant lighting changes? Are there strategies to mitigate potential performance degradation in such scenarios?
soundness: 3
strengths: 1. Originality : The proposed method innovatively combines pre-trained text-to-image diffusion models with a temporal tracking mechanism to achieve video segmentation with flexible granularity using only a few reference images.
1. Quality : The comprehensive experiments, including quantitative evaluations on the new SMITE-50 dataset, qualitative comparisons, ablation studies, and user studies, provide solid support for the method's effectiveness.
1. Clarity : The paper is well-structured, with clear explanations of the problem, methodology, and results. Figures and diagrams are used effectively to enhance understanding.
1. Significance: Addressing the challenge of few-shot video object part segmentation has practical implications for various applications, including VFX and video editing.
summary: This paper presents SMITE (Segment Me In Time), a novel approach for one/few-shot video object part segmentation with flexible granularity. Building upon the image-based SLiMe method, SMITE leverages a pre-trained text-to-image diffusion model (Stable Diffusion) and introduces a tracking mechanism and low-pass filtering to maintain temporal consistency in video segmentations. Given one or a few annotated reference images, SMITE aims to segment unseen videos of objects from the same category, respecting the provided segmentation granularity. The authors introduce a new dataset, SMITE-50, to benchmark performance and compare against relevant baselines.
weaknesses: 1. Methodological Clarity: While the overall explanation is detailed, some of the methodology is difficult to follow, especially for readers not familiar with diffusion models and the previous work SLiMe. The process of how tracking integrates into the attention map refinement could be clearer, with more intuitive descriptions of its working mechanism.
1. Dataset & Baselines: The new SMITE-50 dataset is an important contribution, but it appears somewhat small (with 50 videos). Expanding the dataset size or providing an analysis of its generalization for more categories could strengthen the paper’s claims. A comparison to more widespread benchmarks for video segmentation, such as DAVIS or other VOS datasets, would help position SMITE relative to not only a niche dataset but also larger, well-accepted benchmarks in the community.
1. Inference Efficiency: The paper does not deeply explore the computational cost or inference time of the method in practical settings, which would be relevant given the addition of several components (e.g., temporal voting, low-pass filters). Addressing this in the discussion or providing benchmarks on efficiency could help increase understanding of real-world applicability.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Same as weakness.
soundness: 3
strengths: 1)This proposed method combines information from text, tracker and spatial temporal features for video segmentation at arbitrary level.
2)SMITE achieves state of the art performance on the SMITE-50 dataset.
summary: This paper proposes a SMITE framework for segmenting object fragments across video frames. The method consists of denosing module, diffusion module and tracking module. Overall, it achieves satisfactory performance for video segmentation over various levels of granularity.
weaknesses: 1)The proposed method is only compared with two methods in Table 1, while there are lots of few-shot and zero-shot segmentation methods. 
Besides, it seems that the proposed method can be directly applied on multi-target video segmentation datasets like DAVIS2017. So, there should be more comprehensive comparison, for both methods and datasets.
2)Speed is important for online tracking; so there should be some runtime analysis.
3)Limitations of the method is not discussed in experiments.
4)How does the proposed method work on random videos and random annotation references?

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. The differences in Figure 3 could be better presented (the difference between (c) and (d) is hardly visible).

2. It was slightly challenging to follow all the technical details. 
Few small modifications might reduce the reading time:
For example: 
a. In section 4.2 3 loss terms were mentioned and in section 4.4 2 other loss terms were mentioned; It will be clearer if the authors stated clearly how and in what phase each loss term is used (all together? two-phase optimization?).
b. lines 267-268, sliding window -> temporal sliding window

3. Small technical note:
line 148: should the epsilon_theta after UNet be deleted?

4. Lines 303-305:
It is not clear how this scenario is different from using only cross attention maps. 
If I understand what the authors suggested correctly: average voting can be computed directly on the masks as a "tracking post-process step" without the need for an optimization for tracking.
If so, why can't you simply calculate the average vote on the S_WAS as well?
If not, what was your intention?

5. Line 105 - you mention that some parts cannot be described by text. 
Yet, you do find text embeddings that describe them, although it is reasonable to claim that these embeddings do not represent natural text descriptions. 
However, it will be interesting to acknowledge/discuss this distinction. 
Why does it make sense to find text embeddings for parts that cannot be described by texts?
soundness: 3
strengths: Novel and interesting problem formulation. 
The motivations behind the design choices are presented clearly.
Qualitative results seem decent.
A new video part-level dataset and SOTA performance.
summary: This work presents a framework for segmenting videos into parts at adjustable levels of granularity, using only a single or few annotated images as references. The framework comprises several main components:
First, it optimizes text embeddings and fine-tunes a stable diffusion model so that the WAS attention maps (a combination of self and cross-attention maps) align with the annotated reference images.
Next, temporal consistency is enforced through specific loss terms, enabling the framework to generate segmentation maps from the input video using the optimized text embeddings and fine-tuned diffusion model.
weaknesses: 1. Limited competitors - The baseline (frame-by-frame) might be a bit too simplistic. 
Maybe a better baseline would be a simple video extension of SLIME (without fine-tuning the cross-attention layers). This video extension is pretty trivial and should have some consistency.
Another baseline could be using SLIME (single frame) with tracking.
2. Limited ablation study - 
I would have liked to know what is the contribution of each step.
In addition, I think more ablation would make the paper stronger. For example, how similar should be the input object to the one in the video? Would horse segmentation work on zebras? and on dogs?

In addition: Table 2 only states what happens if only the regularization step is dropped. What about dropping the tracking? And what about dropping both?

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: What is the speed and memory requirement of the proposed method? How many frames can it process?

What if an inflated U-Net is not used during text embedding optimization (e.g., the segment prompts are just images, not videos) but only used during inference?
soundness: 3
strengths: This paper tackles the hard yet useful task of few-shot, fine-granularity video segmentation. This task has potentially wide applications. I think the main contribution of this work is taking the initial crack at this problem with reasonable performance in segmentation quality and temporal stability when compared to applying the SLiME per frame. 

Technique-wise, segmentation tracking/voting and low-pass regularization are reasonable in handling temporal flickering unique to videos, and the results also supported this.

The proposed dataset SMITE-50 is also potentially useful for future work, as it contains densely labeled segmentation which is a rare type of data.
summary: This paper presents SMITE, a few-shot video segmentation method with fine granularity tracking. SMITE is based on SLiMe, a few-shot image segmentation method that finetunes text tokens for a diffusion network and probes attention masks to output a mask.  SMITE additionally introduces cross-attention finetuning of the diffusion network, a segment tracking and voting module based on point tracking, and a low-pass regularization term. SMITE achieves better segmentation performance than applying baseline SLiME per frame in a newly proposed dataset SMITE-50.
weaknesses: I don’t see it mentioned in the paper, but I suspect that SMITE might be too slow for most practical applications. In addition to finetuning the text tokens and the cross-attention layers during a training stage, SMITE additionally performs dense, windowed point tracking – there seem to be a lot of windows (linear to video length) and a lot of repeated computation in the point tracker (linear to windows size). SMITE also minimizes an energy function which is also costly. Besides, given that SMITE expands the spatial self-attention to the spatiotemporal volume, it would be quite memory intensive for longer videos.

I think the authors can also compare SMITE to existing few-shot video object segmentation methods like [Multi-grained Temporal Prototype Learning for Few-shot Video Object Segmentation, ICCV 2023]. This can happen in two dimensions: 1) applying few-shot VOS to the fine-grained segmentation setting, and 2) applying SMITE to the few-shot VOS setting. This can highlight whether the fine-grained segmentation setting significantly differs from the object setting and supplements the existing sparse baselines.

In Table 2, should there be a row with neither of the loss and another row with only the low-pass loss? Also, can the ablations be run on the entire SMITE-50 dataset rather than just on the CARS subset? This would make the results more significant.

Meta-review: This paper addresses an interesting problem in the video-object segmentation (VOS) space. 

Most VOS methods assume an object query in the first video frame (queries can come in the form of segmentation masks, points, or bounding boxes), and the task is to segment the queried object in the remaining video frames. 

Different from that, this paper suggests a "query" that consists of a few images (e.g., three images) that contain an object class of interest (e.g., a horse), with object parts annotated. Given these, the proposed work segment any horse, and its parts, in any video. It can be understood as a few-shot video-object segmentation with image-level supervision. 

The method builds upon prior art for a few-shot image segmentation model (SLiMe) that builds upon a text-to-image generation model (Stable Diffusion). On top of that, this paper introduces a video segmentation method based on point tracking and a low-pass regularization term that helps maintain temporal consistency. Empirical validation on a newly introduced dataset, SMITE-50, shows that the proposed network outperforms SLiME baseline (applied per-frame), and several relevant few-shot/zero-shot VOS methods. 


Reviewers appreciate that the paper is well-written and organized, find the overall approach innovative, agree that it addresses a new problem in the video object segmentation space, and find that the proposed work has practical applications in the industry (especially VFX and video editing). Overall, there appears to be a consensus that this is a strong paper. 

Reviewers also provided constructive feedback. Reviewers note that the paper may not be easy to follow for readers who are not well-versed in the space of generative models or unfamiliar with stable diffusion or prior work this work builds upon (SLiMe). 

Reviewers also note that while a new dataset is always a welcome contribution, it is a rather small dataset, and comparisons using canonical VOS benchmarks (DAVIS&related) would strengthen this paper's contributions. 

Overall, there is a consensus that this is a good paper (ratings are 8, 8, 6, 6, 6). AC agrees with reviewers; the paper addresses an interesting new VOS task and an interesting model that successfully tackles this problem. 

While there was some criticism regarding the empirical validation (reviewers would like to see evaluation beyond the proposed dataset), reviewers agree that the evaluation does support the claims made in the paper. The authors also amended the paper after the rebuttal with additional baselines and results on the DAVIS dataset. 

AC suggests revising the paper to make it more accessible to people unfamiliar with SLiMe (the presentation already improved in rebuttal revision, but AC finds more space for streamlining the presentation). By having paper that is self-contained and explanations consolidated and simpler, it will reach a broader audience and have more potential for impact.  

The authors responded well to comments regarding dataset size compared to existing VOS datasets and showed that the dataset size is comparable to early VOS datasets. The table provided in the rebuttal does not include more recent and larger datasets (e.g., YT-VOS). Please add this to a revised paper version (it can be in the appendix).

Decision: Accept (Poster)