SMITE: Segment Me In TimE
Authors: Amirhossein Alimohammadi
Sauradip Nag
Saeid Asgari
Andrea Tagliasacchi
Ghassan Hamarneh
Ali Mahdavi Amiri
Venue: ICLR 2025 Poster (2025)
Keywords: video segmentation
diffusion models
video diffusion
part segmentation
Primary area: generative models
TL;DR: Segmenting any video by annotating few reference images

Abstract: Segmenting an object in a video presents significant challenges. Each pixel must be accurately labeled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or a few sample images. In this paper, we address this issue by employing a
pre-trained text to image diffusion model supplemented with an additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of-the-art alternatives. The project page is available at https://segment-me-in-time.github.io/

Published as a conference paper at ICLR 2025
SMITE: SEGMENT ME IN TIME
Amirhossein Alimohammadi', Sauradip Nag', Saeid Asgari Taghanaki Andrea Tagliasacchi Ghassan Hamarneh', Ali Mahdavi Amiri'
Simon Fraser University *Autodesk Research University of Toronto *Google DeepMind
Generolize to ln- Te--The-Uiee hu-lUld Videos
Annotate Few Exemplors
Suymertatioo Viddo Frones
Images
Armotate Segments
Few Annotated Images
Figure 1: SMITE. Using only few segmentation references with fine granularity (left), method learns to segment different unseen videos respecting the segmentation references.
ABSTRACT
Segmenting object in a video presents significant challenges. Each pixel must be accurately labeled, and these labels must remain consistent across frames. The difficulty increases when the segmentation is with arbitrary granularity, meaning the number of segments can vary arbitrarily, and masks are defined based on only one or few sample images. In this paper, we address this issue by employing a pre-trained text image diffusion model supplemented with additional tracking mechanism. We demonstrate that our approach can effectively manage various segmentation scenarios and outperforms state-of- of-the-an art alternatives. The project page is available at https: //segment me- in- time. wwt.ub.tt......... ic/
INTRODUCTION
Segmenting an object in a video poses significant challenge in computer vision and graphics, frequently employed in applications such as visual effects, surveillance, and autonomous driving. However, segmentation is inherently complex due to variations in single object (scale, deformations, etc.), within the object class (shape, appearance), as well imaging (lighting, viewpoint). In addition, difficulty arises due to the segmentation requirements, such as its granularity (i.e., number of segments), as demanded by the downstream tasks. For example, in face segmentation, one VFX application might need to isolate the forchead for wrinkle removal, while another, such head tracking, might treat it as part of the whole face. Creating a comprchensive dataset for every possible segmentation scenario to develop supervised segmentation technique is extremely time-consuming and iabor-intensive. Therefore, there is a need to segment images or videos based on reference image. We call this type of segmentation flexible granularity.
When flexible granularity segmentation is applied on a large scale, it significantly improves down- stream tasks such as VFX production, which involves managing numerous shots and videos. By

---

Published as a conference paper at ICLR 2025
segmenting one few reference images only once and then using those images to segment any video that features a target object of the same class, we can eliminate the need for separate segmen- tation of each video, thereby making the process far more efficient. In this paper, tackle the challenge of video segmentation using one or few reference images that are not derived from the video frames themselves. For example, shown in Fig. [ a few annotated images are provided as references to our model, and our method, SMITE, successfully segments videos, exhibiting an object from the same class, in the same level of granularity. Importantly, none of the frames from these videos are included in the reference images, yet SMITE is capable of segmenting the videos with objects that exhibit different colors, poses, and even occlusions. This is an important feature when working with large-scale videos requiring consistent segmentation (such as VFX videos needing the same enhancements), since there no need for manual intervention to segment each video's 's frames.
While recent work has explored flexible granularity segmentation for objects in images by leveraging the semantic knowledge of pretrained text-to-image diffusion models Khani et al. (2024), the com- plexity increases in videos. Ensuring label consistency across frames and managing instances where image segmentation may fail to produce accurate results require additional considerations.
To tackle these challenges and achieve consistent segmentation across frames, we utilize the semantic knowledge of pretrained text-to- text-to-image diffusion models, equipped with additional temporal attentions to promote temporal consistency. We also propose temporal voting mechanism by tracking and projecting pixels over attention maps to maintain label consistency for each pixel. This approach results in segmentations with significantly reduced flickering and noise compared to per-frame segmentation methods while segments still follow the reference images thanks to our low-pass regularization technique that ensure preserving the structure of segments provided by attention maps and optimized according to the reference images.
Moreover, rather than simply optimizing a token for each segment Khani et al. (2024), we also fine-tume pross-attentions to enhance segmentation accuracy and better align with the reference images. Consequently, our method not only supports videos with temporal consistency but also outperforms flexible granularity image segmentation techniques in segmenting a single image.
We validate our design choices and methodology through comprehensive experiments detailed in the paper. As existing datasets with arbitrary semantic granularity are lacking, we introduce a small dataset, SMITE-50, to demonstrate the superior performance of our method against baselines. Addi- tionally, we conduct user studies that highlight our method's effectiveness in terms of segmentation accuracy and temporal consistency.
RELATED WORK
Part-based semantic segmentation. In computer vision, semantic segmentation, wherein class label is assigned to each pixel in an image, is an important task with several applications such as scene parsing. autonomous systems, medical imaging, image editing, environmental monitoring,
and video analysis (Sohail et al. 20222 He et al. 2016 Chen et al., 2017a; Zhao et al. 20177: Hc ct 2017 fine-grained derivative of semantic segmentation is semantic part segmentation, which endeavors to delineate individual components of objects rather than segmenting the entirety of objects. Despite notable advancements in this domain (Li et al. 2023 2022). a limitation of such methodologies is their reliance on manually curated information specific to the object whose parts they aim segment. To solve the annotation problem, some works (Pan et al. 2023 Wei et al. 2024) proposed open-set part segmentation frameworks, achieving dategory-agnostie part segmentation by disregarding part category labels during training. Building on this, further works such as SAM (Kirillov et al., 2023). Grounding (Ren et al. 2024) explored utilizing foundation models to assist in open-vocabulan part segmentation. However, most of these methods can only segment the parts that are semantically described by text. With the influx of Stable Diffusion (SD) based generative segmentation approaches (Khani et al. 2024: Namekata et al.) 2024), such issues have been partly solved by allowing SD features to segment semantic parts at any level of detail, even if they cannot be described by text. Despite such progress, applying such fine- mmn-iiaieed segmentations on videos is challenging and unexplored. Our proposed SMITE presents the first part-segmentations in videos wherein it segments utilizing the part features from pre-trained SD and generalizes it to any-in-the wild videos.
[0en::Chen Chen et al. 2017b; Sandler al. 2018 Chen et al. 2018; Ravi et al. 2024 A more
2

---

Published as a conference paper at ICLR 2025
Video segmentation. Video segmentation methods can be categorized as video semantic segmentation
(VSS) (Zhu et al., (024, Zhang et al. 2023a b Li et al. 2024; Ke et al. 2023; Wang et al., 2024). video instance segmentation (VIS)( Yang et al. (019) and video object segmentation (VOS) (Xte et al., 2021a Wang et al. 2021b Cheng et al. 2021c di Bekuzarov et al. 2023). VSS and VIS extends image segmentation videos, assigning pixel labels across frames while maintaining temporal consistency despite object deformations and camera motion. VOS, in contrast, focuses on tracking and isolating specific objects throughout the video. Both tasks leverage temporal correlations through techniques like temporal attention (Mao et al., 2021a Wang et al.) 2021a). optical flow (Xie et al. 2021a Zhu et al., 2017). and spatio-temporal memory (Wang et al. 2021b] Cheng & Schwing 2022). Recent efforts, such as Uni (Li et al., 2024), proposes unified models for various segmentation tasks, utilizing prior frame features as visual prompts. However, these methods struggle with fine- grained part segmentation and generalization to unseen datasets (Zhang et al. (023b). [Bekuzarov et al. (2023) leverage a spatio-tempore memory module and a frame selection mechanism to achieve high- quality video part segmentation with partial annotations. However, it requires frame annotations from the same video complicating video segmentation at scale. In contrast, we only need segmentation references for few arbitrary selected images and we can segment an unseen given video respecting the segementation references. Therefore, per video manual annotation is not needed in our method.
Video diffusion models. Recently, diffusion models (Ho al., 2020 Song al. 2020a b) have gained popularity due to their training stability and have been used in various text-to-image (T21)
methods (Ramesh et al. 2021 2022] Saharia al. 2022 Balaji et al. 2022), achieving impressive results. Video generation (Le Moing ct al. 2021 Ge ct al. [022 Chen et al. 2023b Cong et al.
2023 Yu et al.
2023 Luo al. 2023) can be viewed as an extension of image generation with an
additional temporal dimension. Recent video generation models (Singer et al. 2022 Zhou et al.
2022; Ge Ge at. 2023] Nag et al. 2023 Cong et al.] 2024) attempt to extend successful text-to-image
generation models into the spatio-temporal domain by inflating the T21 UNet. VDM (Ho et al., 2022) adopted this inflated UNet for denoising while LDM (Blattmann et al. 2023) implement video diffusion models in the latent space. Video diffusion models can be categorized into inversion-basee based and inversion-free methods. inversion-basee approaches (Cong et al. 2024 Jeong & Ye (e22222)) use DDIM inversion control attention features ensuring temporal consistency, while inversion-free methods (Zhang et al. 2023c) focus on more flexible conditioning, wider compatibility and better generation quality. However, inversion-free methods can suffer from flickering due lack of DDIM inversion guidance. Recent works (Wang et al.) 2024 Zhu et al. 2024) explored inversion-free T2V diffusion models segment objects in videos, but they fail to generate fine-grained segments and often produce flickering segmentation results. Building on this, our method solves the seminal problem of video part-segmentation which combines an inversion-fre free model coupled with trackkga algorithms (Karaev et al. 2023) to generate consistent, generalizable, flicker-free segmentations.
3 PRELIMINARIES
Latent diffusion models and WAS maps. Latent Diffusion Models perform the denoising operation on the latent space of a pretrained image autoencoder. Each latent pixel corresponds patch in the generated image. Starting from pure random noise 27, at each timestep t, the current noisy latent 2, passed through a denoising UNet (Ea). which is trained predict the current noise y, using text prompt y. In each block, the UNet employs residual convolution layers to generate intermediate features, which are then fed into attention layers. These attention layers average various values based on pixel-specific weights. The cross- attention layers (denoted by Amx incorporate semantic contexts from the prompt encoding, whereas the self-attention layers (denoted by Asa) leverage global information from the latent representation itself. SLiMe (Khani et al., 2024) demonstrated that text embeddings can be learned from a few images to segment other unseen images by leveraging both cross attention and self self-attention layers. In fact, each segment of an image is linked with a text token that is optimized to align its associated cross-attention and self-attention maps with the corresponding segment in the provided training image. For segmentations with multiple levels of granularity, multiple tokens are optimized, with each token representing a distinct segment. The optimization process of attention maps captured in a novel representation called Weighted Accumulated Self-Attention (WAS) map, SwAS- This is defined as follows:
Swas Sum(Flatten Rea) @ Asa),
(1)
where Reg is the downsampled latent of Apa-
3

---

Published as a conference paper at ICLR 2025
Input video
(0.3)
Tedl Entreiding
SMITE Oulpet
Noise Addition
Neki
InfattedUUnet
Inflated-Uret
Noiey Video
X;
Mack Exreder
Hask Ediecior
Argnax (URS
DDIM Denoising
Sapeaate 34AG Hagh-
Geidence
Misk Thacking
(c) Segmant Tracking
Focct
Haok Eslrsttrr
UNet Denoiser
Untimdiioal Siageennss
Tradioaa Singments
Naig Labent.
Freeueecyy Regelarizer
Segmented Video
Texl. Erbensding
Z5
Plazs
(o) SMITE Pipeline
Rydereact Segenst
Hiler
Sngannt
SpstiaConnideeea
Figure 2: SMITE pipeline. During inference (a), we invert a given video into a noisy latent by iteratively adding noise. We then an inflated U-Net denoiser (b) along with the trained text embedding as imput to denoise the segments. A tracking module ensures that the generated segments are spatially and temporally consistent via spatio-temporal guidance. The video latent 24 is updated by a tracking energy &track (c) that makes the segments temporally consistent and also a low-frequency regularizer (d) Ereg which guides the model towards better spatial consistency.
Inflated UNet. A T2I diffusion model, such as LDM (Rombach et al., 2022). usually utilizes a E.Neet(Ronnebrge. et al.] 2015) architecture, which involves a downsampling phase, followed by upsampling with skip connections. The architecture consists of layered 2D convolutional residual blocks, spatial attention blocks, and cross-attention blocks that incorporate textual prompt embeddings. To extend the T2I model for T2V tasks, the convolutional residual blocks and spatial attention blocks are inflated. Following earlier approaches (Cong et al., 2024, Wu et al., 2022). the by introducing pseudo
3 x 3 convolution kernels in the residual blocks are adjusted to I x temporal channel. To enhance the temporal coherence, we further extend the spatial self self-attention mechanism the opatio-temporal domain. The original spatial self attentttntion method focused on patches within a single frame. However, in inflated UNet, we use all patch embeddings from the entire video as the queries, keys, and values. This allows for a full understanding of the video's context. Additionally, we reuse the parameters from the original spatial attention blocks in the new dense temporal attention blocks.
4
METHOD
Here, we first introduce and formalize our method (SMITE) designed for achieving temporally consistent video segmentation with varying levels of granularity, guided by one or more reference images (Sec. [41). To achieve this and capture fine-grainer segments, we first propose a new training strategy applied on the inflated UNet (Sec. 4.2p. The segmentation obtained from the inflated UNet may lack temporal consistency. To address this, we employ a voting mechanism guided by tracking method that is projected onto the attention maps. However, relying solely tracking to adjust the segments might lead to deviations from the provided samples. To mitigate this, we incorporate a frequency- based regularization technique maintain detailed segmentations across frames (Sec. 4.3) Since tracking and frequency-bree based regularization may pull the segmentation in different directions, we use energy based guidance optimization technique to balance both approaches (Sec. 4.4
4.1 PROBLEM SETTING
Problem statement. Given one or few images, I, E Hx of subject, along with its segment annotations Dn ( 1 K} where )" denotes binary segment mask i, and and K are respectively the number of images and segments. Our objective is to learn temporally consistent segments of the subject for given video > (u } where U/ represents video frames.
Our framework. We use Stable Diffusson'ss 's (SD) semantic knowledge to learn the segments defined by few images and then generalize them to the video of the subject in any pose, color size. This
4

---

Published as a conference paper at ICLR 2025
implies that the model needs to share information across frames enforce temporal consistency. Differently from the UNet structure used in Khani et al. (2024), we apply an inflation to the T2I models across the temporal dimension (Wu et al. [022] enable temporal attention across all video frames. Also, we incorporate a tracking module combined with low-frequency regularization to enhance spatio-temporal consistency across frames. The overall inference pipeline of our model, SMITE, is illustrated in Fig 2
4.2 LEARNING GENERALIZABLE SEGMENTS
We first learn segments provided by the reference images of the subject by optimizing a text embedding for each segment of the reference images that can be used for segmenting the given videos. We also fine tune the cross attentions of the SD to better match the provided segments since the text embeddings alone may not be able to fully capture the masks' details.
Learning text embeddings. We begin by passing the reference images I and text embeddings into into SMITE (denoted by vi))). Similar to SLiMe (Khani et al. 2024), we obtain image resolution WAS maps (denoted by S) from our inflated UNet as follows:
= (I. Y argmax(())
where 0 represents the learnable parameters of the model and S is defined the same as Eq. | Text embeddings T, which are initialized randomly or with the names of segments, correspond to segment masks = [Swss SwAs: ..., WWs). and Y is the segmentation output. Since the inflated UNet is designed for videos, we pass reference images in I to (b) as videos with single frame. Together with the ground-truth mask y. we find the optimized text embeddings, T that are correlated with the segments in S.
Network fine-tuning. Only learning text embeddings fails to capture complex granularities. To extract more cus- tomizable fine-gres grained segments from the video, we need to fine-tune the existing SD weights (denoted by 0) using the available provided segment annotations. As shown in the inset figure, optimizing solely the text embeddings struggles with asymmetrical segmentation (e.g., segment- ing only one eye). We hypothesize that such issues arises because the model may get stuck in local minima when relying exclusively on text embeddings for optimization. To mitigate this. we update the cross-attention layers Aco in the UNet along with the text-embedding but do so in two phases. First, the model is frozen, while the text embeddings corresponding the segments denoted by are optimized using a high learning rate. Thus, an initial embedding (T*) is achieved quickly without detracting from the generality of the model, which then serves as a good starting point for the next phase. Second, we unfreeze the cross-attention weights and optimize them along with the text embeddings, using significantly lower learning rate. This gentle fine-tume teee-tunig of the cross-attention and the embeddings enables faithful generation of segmentation masks with varied granularity. For both training phases, of losses (LCF, LMSE and LIDM) used in SLiMe (Khani et al., 2024). This results in an optimized SMITE model (denoted by V7)) generating generalizable segment across different videos.
au Text Oytinizaton or
w/ Teat e Croasiite, Cytivization
the same combination
4.3 TEMPORAL CONSISTENCY
To enhance temporal consistency, we use temporal attention in SMITE's Inflated UNet denoiser. While improves temporal consistency compared to independent frame processing (Fig B a)). inconsistencies remain due to the need for segmenting at different granularities, often with imprecise boundaries. These inconsistencies can cause flickering or unnatural transitions (b))..)).
Segment tracking and voting. The first step to ensure segment consistency is tracking the segments across time. Point tracking methods like CoTracker (Karaev et al., 2023) are well-suited for our approach because they use point correspondences to minimize pixel drift over time. However, since our segments come from attention maps, tracking needs to occur directly on these maps. Since CoTracker is trained spatial domains, of video V bidirectionalyy as (X, Y} = P(V) where X, Y are set of tracked pixel coordinates. To project these tracking onto the the attention maps, use sclaing operator ((-) and linearly seale the
first apply CoTracker (denoted by P) on the frames
5

---

Published as a conference paper at ICLR 2025
Frome
Pronse
Franc
Pranel
Tracking Nuse Part
Co Tradker
Prob.
0.7
Pool Logit
Nase Prot.
Segment Tracking Voting
Mejortty Voting
Franel)
Franel
Pranee
Frovce
Face Prub.
b.2
0.2
Reptaoed Laght
Consened Ohies
Figure 4: Segment tracking module ensures that segments consistent across time. It uses tracker to track each point of the object's segment (here is nose) and then finds point correspondence of this segment (denoted by blue dots) across timesteps. When the tracked point is of different class (e.g.. face) then it is recovered by using temporal voting. The misclassified pixel is then replaced by the average of the neighbouring pixels of adjacent frames. This results are temporally consistent segments without visible flickers.
pixel trajectories onto the attention space. We process the entire video in a temporal sliding window fashion having window length For each of such temporal window, we then leverage the projected trajectories and update the attention pixel's label based on the most frequent label it receives across the visible frames in the window. Formally, for pixel with coordinates (Iz, yr) on the attention map latent at timestep I, its coordinates on all subsequent latents in the window can be derived from the tracker. The coordinates linked, and the trajectory sequence can be presented
{I y. 3t- )..... (Its yc), .... (I++ y. M/+++ y }) (P(()))
We choose the point corresponding to the center frame of the window (Ze,ye) as the first point and leverage bidirectional correspondences within the window. Each of these trajectory points (æ, y) is assigned a segment label Y. However, certain pixels in the window may disappear and reappear over time shown in Fig 4 To handle such scenarios, we discard labels for which the tracking pixel is invisible and then use temporal voting to update its segment labels. Therefore, we update the WAS maps (denoted by SwAS) corresponding the tracking pixel (Ie, Mr) and use the following formula to assign the labels:
STackad wt) Avg(S(x1, vi) (21, F) AI E (t
W
2
w
(4)
2
where F denotes the most frequent label in the window.
After correcting the unstable pixel tracking over
(a) frame by frome
(b) w/ tracking
multiple-step denoising, we obtain a consistent of segments (Stracked). Note that the window size is small (e.g, seven), hence to track the segments for the entire video, we need to slide the windows and track an iterative fashion. More details are provided in the supplementary material.
Intuitively, if we would have relied only on cross- attention for segmentation, we could have adjusted it directly to align with the tracking output. However, since segments S are derived from the combination of cross and self-attention, direct manipulation challenging. To tackle this, propose energy function to measure how well the segments before tracking (denoted by S) align with the segments after correcting (denoted by Strackea) it using temporal voting. This energy defined as:
(c) w/o lour
(d) SMITE
Figure 3: Video best viewed in Acrobat.
Eracking = CE(S, Suncked),
(5)
where C. represents cross-entropy objective. This optimization ensures that the segment drift is corrected throughout the video to make it temporally consistent with low flicker.
Low-pass regularization. When tracking is applied, label modifications should be repeated through several denoising steps ensure full temporal label propagation and achieve better consistency.

---

Published as a conference paper at ICLR 2025
This process may cause deviations (Fig 3) from the original segmentation provided by the reference images that is captured in the segments (S) in the initial denoising step. Our goal is to preserve the
overall structure of the segments S from the WAS maps while smoothing boundary transitions for
temporal consistency via tracking (refer to Fig 8 Low Pass Filters (LPF) have been effective in video generation maintain temporal correlations and spatial frame structure (Wu al. 2023 Sii et al. 2024). Therefore, we use an LPF on the finally denoised segments S ensuring the structure of the LPF on the segments & predicted in the initial denoising step (denoted by Syef) is respected by the following function:
RReg = [[](S) w(Sref)lls
where w(S) = Hi @ DS) DCT3p refers to Discrete Cosine Transform and Hi refers to an LPF. This ensures that the segments structure is progressively corrected across the video frames at the end of the denoising phase as illustrated in Fig
4.4 SPATIO-TEMPORAL GUIDANCE
Our overall energy function is minimized test time by backpropagating through the diffusion pro- cess, as described in Chen et al. (2023a); [Safaee et al. (023)) which updates the latent representation
to achieve more consistent segmentation over time:
Evoall
Arrcking &Tracking + AReg EReg-
where Arrcking and AReg are the coefficients for the tracking and reference loss functions, respectively, We have two different energy functions: Tracking which reduces flicker and ensures segment consis- tency over time, while Ekey which maintains correct semanties while preventing segment drift. Thus, optimizing Eooal encourages better spatial and temporal consistency in the WAS attention maps (S). Specifically, we update the latent 2, using gradient descent on Toeal
+ 24 a, V Tooall
(8)
where Oe is the learning rate. Results indicate that this strategy leads superior performance compared to when we either omit the voting mechanism or do not apply low-pass regularization.
5 RESULTS AND EXPERIMENTS
Dataset and benchmark. To evaluate our method, we introduce a benchmark dataset called SMITE- 50, primarily sourced from Pexels SMITE-50 features multi-granularity annotations and includes visually challenging scenarios such as pose changes and occlusions. To our knowledge, no existing datasets focus exclusively on nullt-granular and multi-segment annotations. While the Puma dataset Bekuzarov et al. (023) contains limited number of multi-part annotated videos across diverse range, it does not offer multiple videos for specific granularities and categories.
We focus three main categories: (a) Horses, (b) Human Faces, and (c) Cars, encompassing 41 videos. Each sub- set includes segmented reference images for training and densely annotated videos for testing. The granularity varies from human eyes to animal heads, etc. relevant for
various applications such as VFX (see Fig. 5). All seg- ments are labeled consistently with the part names used in existing datasets. Additionally, we provide nine challeng- ing videos featuring faces with segments that cannot described textually, as shown in Fig. 3 (Non-Text). Over- all, our dataset comprises 50 video clips, each at least five followed simi- lar approach to (Ding et al. 2023 Bekuzarov et al., 2023). creating masks for every fifth frame with average of six parts per frame across three granularity types (more info in Appendix). While PumaVOs dataset Bekuzarov et al. 20023 has 8% annotations, SMITE-50 dataset has 20% dense annotations.
seconds long. For dense annotations,
Non-Texx
Horse
Figure 5: SMITE-50 Dataset sample.
Evaluation protocol. In our setting, few reference images per class are used to train SMITE and then it is evaluated on videos from the same categories but not the same objects in the training data.
7

---

Published as a conference paper at ICLR 2025
RGB BGBvels
Baeeline-
GSAM2
Ours
Figure 6: Visual comparisons with other methods demonstrate that SMITE maintains better motion consistency of segments and delivers cleaner, more accurate segmentations. Both GSAM2 and Baseline-I struggle to accurately capture the horse's mane, and GSAM2 misses one leg (Left), whereas our method yields more precise results. Additionally, both alternative techniques create artifacts around the chin (Right), while SMITE produces a cleaner segmentation.
For all the cases, we report the standard metrics (higher is better): Mean Intersection Over Union (mlOU), and Contour Accuracy Fmeasure respectively.
Quantitative comparison. Since there are no available few-shot video part segmentation methods that can be applied to our setting, we build the multiple baselines for quantitative comparison. First, a few-shot image segmentation based approach, in which SLiMe (Khani et al., 2024) is applied to the video frame-by-frame, We term this as Baseline-l. Second, Grounded SAM2 (Ren al.) 2024 Ravi et al. 2024), recently introduced zero-shot foundation model for video segmentation approach which is applied directly the video. For Baseline-l,
applied the same training procedure and used SMITE-50 50 dataset. Another baseline, Baseline-Il is to use SLiMe and CoTracker simultaneously. However, directly applying tracking in the pixel space does not produce improved results, as it can lead to drift in segmentation. In contrast, SMITE leverages tracking on the attention maps, focusing on where the object and its parts are located, which minimizes impact of pixel color shifts. Additionally, we have incorporated regularization mechanism into our tracking process to further mitigate such drifts, ensuring alignment with the true semantic segmentations. We have also provided another baseline, Baseline-Ill, which is the modification of SLiMe by only changing its UNet to the Inflated UNet. As shown in Tab. I our method produces the most accurate results compared to other baselines across all categories and metries.
Generally, XMem++ (Bekuzarov et al. (003) cannot be utilized in our setting since we do not want use any frames from the provided video. However, in the Appendix, we included comparison with XMem+ on a subset of their proposed dataset PUMaVOS with flexible granularity to provide a contrast and show that our method is effective on datasets beyond SMITE-50. Although XMem+ is semi-supervised technique, our method performs comparably and even outperforms it with fewer frames (e.g., a single shot). When XMem++ is given more frames per video (e.g.. 10), its performance slightly surpasses ours, which is expected since SMITE does not require any substantial video pre- training. However, with a smaller number of frames (e.g., one or five), our method either outperforms or matches the performance of XMem++. Additionally, we report our performance on the image segmentation task the Appendix, showing that we outperform SL.ME (Baseline-I), highlighting the effectiveness of our design choices, including the cross attention optimization.
Qualitative comparisons. Qualitative comparisons are presented in Fig. 6 GSAM2 struggles to locate the boundaries accurately and produces coarse segments. SLiMe can usually segment the first frame accurately but struggles to preserve temporal consistency. Our method produces the best segmentation maps in terms of segmentation quality and temporal consistency. The maps have sharper boundaries and clean clusters compared to other methods. Our SMITE performs better than its SLiMe present results from other categories, demonstrating the versatility of counterpart. In Fig.
8

---

Published as a conference paper at ICLR 2025
Table Quantitative evaluation on SMITE-50 dataset. The results are presented for each category (Face, Horse, Car, Non-Text) having 10 reference image during training.
Faces
Horses
Cars
Non-Ten
Mean
Methods
meas.
mIOu
F meas.
mIOU
meas.
m
Fmess.
mIOU
meass
mIOU
Baseline-1
0.81
72.95
0.64
65.48
0.57
61.38
0.67
66.69
0.67
66.62
Baseline-Il II
0.78
71.08
0.62
64.01
0.65
61.95
0.65
64.96
0.67
65.50
Baseline-II III
0.82
73.25
0.64
67.48
0.70
68.49
0.66
68.22
0.70
69.36
GSAM2
0.73
63.28
0.76
72.76
0.64
63.56
0.71
66.53
Ours
0.89
77.28
0.79
75.09
0.82
75.10
0.77
73.08
0.82
75.14
Table 2: Network Ablation on SMITE-50 dataset. Ablations the entire SMITE-50 dataset. We respectively show the impact of inflated UNet (IUNet), cross attention tuning (CA Tuning), Tracking, and Low-pass (LP) regularization.
Method Baseline (SLiMe) mIOU
+1UNet (Baseline III) +CA Tuning
+Tracking 72.64
+LP Reg (SMITE)
66.22
69.36
71.70
75.14
our model in handling various object categories. As shown, despite significant differences in pose, expression, gender, and other properties between the video and the annotated images, our method still delivers high-quality results. This particularly evident in the pineapple example, where the object is cut in half, the method successfully tracks it and produces accurate segmentation. More qualitative results are provided in Appendix 7.4
A Ablation study. To demonstrate the effectiveness of our design choices, we conducted comprehen- sive ablation study showing the impact of cach component of our method one by one on the entire SMITE-50 dataset. Tab. shows that final setting provides the highest accuracy while the impact of each component is noticeable. Note that the low-frequency regularization (LP Reg) helps maintain spatial structure and prevents segment drift when tracking is applied to Sw Without tracking, LP Reg has minimal effect since no segment drift occurs in WAS maps (thus change of energy in &Reg is almost zero), so we do not ablate LP Reg separately without tracking. We have also performed an ablation on the number of training images illustrated in Tab. As expected the accuracy improves when the number of training samples increases.
User study. We conduct user study because human judgment is best for assessing perception when both temporal consistency and faithfulness segmentations are important. We collected 16 segmented videos (4 from Non-Text category) and asked 25 participants to rank the methods (i.e., 1 best and 5 worst) based on segmentation quality (i.e., fidelity the segmentation reference) and motion consistency (i.e., reduced flicker), encouraging them to prioritize segmentation quality in their evaluations in terms a tie in motion consistency. We do not report the scores of GSAM2 on Non-Text segments as the segments in the videos are not describable by texts. Tab. demonstrates that SMITE achieves higher preference both for textual and non-ten non-textual segmentation.
Table 3: Few shot ablation on SMITE-50. The performance increases with more training images but still performs well in one shot setting.
Table 4: User study. We are ranked the best for both textual and Non-Text classes.
Motion Consistency
Methods
Horse, Car, Face
Non-Text
Baseline-I
3.59
2.41
Training sample #
mIOU
Baseline-II
3.42
3.24
I-shot
58.55
Baseline-lll
2.98
3.32
5-shot
73.88
GSAM2
3.37
10-shot
75.14
Ours
1.46
1.03
CONCLUSION, LIMITATIONS, FUTURE WORK
In this work, we introduce SMITE, a video segmentation technique that supports flexible granularity segmentation of objects within video. SMITE leverages the semantic knowledge of pre-trained

---

Published as a conference paper at ICLR 2025
Video Segment Generalization
Reference Annotation
Faces
1
Penguin
Cars
Pineapple
Figure 7: Additional results. We visualize the generalization capability of SMITE model (trained on the reference images) in various challenging poses, shape, and even in cut-shapes.
text-to-image diffusion model to segment a video with minimal additional training (i.e., only a few images), while utilizing the temporal consistency of the video to maintain motion consistency within the segments. Notably, SMITE can be trained with a few images that are not necessarily from the video, yet it effectively segments objects within unseen video. To better show the capabilities of our method, we also collected a flexible granularity dataset called SMITE-50 that will be publicly available along with our code. Through various quantitative and qualitative experiments, as well as user studies, we demonstrate the effectiveness of our method and its components. However, our method still faces limitations that motivate future work. Firstly, although, our regularization ensures that tracking does not dominate the segmentation process deviate from the anticipated semantic segmentations, since SMITE utilizes an off-the-shelf tracking technique, its tracking mechanism and voting system are still limited by the performance of the underlying tracking system to some extent. Secondly, the generation WAS maps involves the use of low-resolution cross-attention mechanisms, which may result in reduced accuracy when segmenting fine details small segments, such as tiny strings or minor components. Thirdly, cannot offer an interactive segmentation since it involves the diffusion and optimization processes. In this work, did particularly focus optimizing training and inference time but this can be an interesting future work.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. This work supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant. We sincerely thank Pouria Laghaee for his invalu- able contributions and assistance in dataset annotation. We also appreciate the support of Helena Dehkharghanian, Ali Mohammadzade Shabestari, Soniya Almasi, and Fatemeh Kamani in the annotation process.
REFERENCES
Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, et al. ediffi: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.013 2022.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint arXiv:2112.03126, 2021.
Maksym Bekuzarov, Ariana Bermudez, Joon-Young Lee, and Hao Li. Xmem++: Production-level video segmentation from few annotated frames. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 635-644, 2023.
Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dockhorn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22563-22575, 2023.
Liang-Chite Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):833-848, 2017a.
Liang-Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous 2017b.
convolution for semantic image segmentation. arXiv preprint
Liang-Chich Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. 2023a. URL htRLl/ttss///arri...////////////
Shoufa Chen, Mengmeng Xu, Jiawei Ren, Yuren Cong, Sen He, Yanping Xie, Animesh Sinha, Ping Luo, Tao Xiang, and Juan-Manuel Perez-Rua. Gentron: Delving deep into diffusion transformers for image and video generation. arXiv preprini arXiv:2312.0455// 2023b.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1971-1978, 2014.
Yen Chen, Xiaojie Jin, Xiaohui Shen, and Ming- Yang. Video salient object detection via contrastive features and attention modules. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2022.
Hao- Hao-Kang Cheng, Yu-Wing Tai, and Chi-Keung Tang. Modular interactive video object segmentation: interaction-t-mmast propagation, and difference-aware fusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 202
Hao- Kang Cheng. Yu-Wing Tai, and Chi Chi-Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
Ho Kei Cheng and Alexander G Schwing. Xmem: Long-term video object segmentation with an Sainsonn-shifrin memory model. In European Conference on Computer Vision, pp. 640-658. Springer, 2022.
11

---

Published as a conference paper at ICLR 2025
Ho Kei Cheng, Yu- Yu-Wing Tai, and Chi Khu-Kenng Tang. Modular interactive video object segmentation: interactiontoomasse propagation and difference aware fusion. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5559-5568, 2021c.
Ho Kei Cheng, Yu-Win Yu-Wing Tai, and Chi-Keun Keung Tang. Rethinking space-time networks with improved memory coverage for efficient video object segmentation. Advances in Neural Information Processing Systems, 34: 1781-11794, 2021d.
Suhwan Cho, Minhyeok Lee, Seunghoon Lee, Chaewon Park, Donghyeong Kim, and Sangyoun Lee. Treating motion as option to reduce motion dependency in unsupervised video object segmentation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), 2023.
Yuren Cong, Jinhui Yi, Bodo Rosenhahn, and Michael Ying Yang. Ssgvs: Semantic scene graph-to- video synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 2554-2564, June 2023.
Yuren Cong, Mengmeng Xu, Christian Simon, Shoufa Chen, Jiawei Ren, Yanping Xie, Mann-l Perez-Rua, Bodo Rosenhahn, Tao Xiang, and Sen He. Flatten: optical flow-guided attention for consistent text-to-video editing. 2024. A
/
Zhenqi Dai, Ting Liu, Xingxing Zhang, Yunchao Wei, and Yanning Zhang. One-shot in-context part segmentation. In ACM Multimedia 2024, 2024.
id cd=0mXBMML91M
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 00224420234, 2023.
Jiaqing Fan, Tiankang Su, Kaihua Zhang, and Qingshan Liu. Bidirectionally learning dense spatio- temporal feature propagation network for unsupervised video object segmentation. Proceedings of the ACM International Conference on Multimedia (ACM MM), 2022.
Songwei Ge, Thomas Hayes, Harry Yang, Xi Yin, Guan Pang. David Jacobs, Jia-Bin Huang, and Devi Parikh. Long video generation with time-agnostic vggan and time-sensitive transformer. In Earopean Conference on Computer Vision, pp. 102-118. Springer, 2022.
Songwei Ge, Seungjun Nah, Guilin Liu, Tyler Poon, Andrew Tao, Bryan Catanzaro, David Jacobs, Jia-Bin Huang, Ming- Yu Liu, and Yogesh Balaji. Preserve your own correlation: A noise prior for video diffusion models. arXiv preprint arXiv:2305. 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho. Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J. Fleet. Video diffusion models, 2022.
Hyconho Jeong and Jong Chul Ye. Ground-a-video: Zero-shot grounded video editing using text-to- image diffusion models. arXiv preprint arXiv:2310.01107, 2023.
Ge-Peng Ji, Keren Fu, Zhe Wu, Deng-Ping Fan, Jianbing Shen, and Ling Shao. Full-duplex strategy for vidco object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Nikita Karaev, Ignacio Rocco, Benjamin Graham, Natalia Neverova, Andrea Vedaldi, and Christian Rupprecht. Cotracker: It is better to track together. arXiv preprint arXiv:2307.07635, 2023.
12

---

Published as a conference paper at ICLR 2025
Lei Ke, Martin Danelljan, Henghui Ding, Yu- Wing Tai, Chi-Keung Tang, and Fisher Yu. Mask-free video instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22857-28222
2023.
Aliasghar Khani, Saeid Asgari Taghanaki, Aditya Sanghi, Ali Mahdavi Amiri, and Ghassan Hamarnch. Slime: Segment like me. ICLR 2024, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan- Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
Guillaume Le Moing, Jean Ponce, and Cordelia Schmid. Cevs: context-aware controllable video synthesis. Advances in Neural Information Processing Systems, 34: 14:14042-14055, 2021.
Youngjo Lee, Hongje Seong. and Euntai Kim. Iteratively selecting an easy reference frame makes unsupervised video object segmentation easier. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2022.
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven semantic segmentation. In International Conference on Learning Representations, 2022. URL
https: /attps://openreveee net htw.:/eettoormmmoo-rirooooo...o
Minghan Li, Shuai Li, Xindong Zhang, and Lei Zhang. Univs: Unified and universal video segmenta- tion with prompts as queries. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 3227-3238, 2024.
Xiangtai Li, Shilin Xu, Yiho Yang, Haobo Yuan, Guangliang Cheng, Yunhai Tong, Zhouchen Lin, and Dacheng Tao. Panopticpartformeent:: A unified and decoupled view for panoptic part segmentation. arXiv preprint arXhv:2301.0095. 2023.
Yunchao Liang, Xia Li, Nima Jafari, and Jianbing Chen. Video object segmentation with adaptive feature bank and uncertain-region refinement. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Daizong Liu, Dongdong Yu, Changhu Wang, and Pan Zhou. F2net: Learning to focus on the foreground for unsupervised video object segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2021.
Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion models for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10218, 2023.
Yunyao Mao, Ning Wang, Wengang Zhou, and Houqiang Li. Joint inductive and transductive learning for video object segmentation. In Proceedings of the IEEF/CVF international conference computer vision, pp. 9670-9679, 202
Yuxin Mao, Ning Wang, Wengang Zhou, and Houqiang Li. Joint inductive and transductive learning for video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021b.
Sauradip Nag, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, and Tao Xiang. Difftad: Temporal action detection with proposal denoising diffusion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10362-10374, 2023.
Koichi Namekata, Amirmojtaba Sabour. Sanja Fidler, and Seung Wook Kim. Emerdiff: Emerging pixel-level semantic knowledge diffusion models. arXiv preprint arXiv:2401.1 2024.
Scoung Wug Oh, Joon- Young Lee, Ning Xu, and Scon Joo Kim. Video object segmentation using space-time memory networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) 2019.
13

---

Published as a conference paper at ICLR 2025
Tai- Yu Pan, Qing Liu, Wei-Lum Chao, and Brian Price. Towards open open-world segmentation of parts. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15392-15401, 2023.
Gensheng Pei, Fumin Shen, Yazhou Yao, Guo-Sen Xie, Zhenmin Tang, and Jinhui Tang. Hierarchical feature alignment network for unsupervised video object segmentation. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
Federico Perazzi, Jordi Pont-Tuset, Brian McWilliams, Luc Van Gool, Markus Gross, and Alexander Sorkine- A benchmark dataset and evaluation methodology for video object segmentation. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alexander Sorkine- and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv: Computer Vision and Pattern Recognition, 2017.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Nikhila Ravi, Valentin Gabeur, Yuan- Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudey Alwala, Nicolas Carion, Chao- Yuan Wu, Ross Girshick, Piotr Dollár, and Christoph Feichtenhofer. Sam 2: Segment anything in images and videos, 2024. URL https: /ttpsss///axv..///e org/ass/ 2408. 00714]
Sucheng Ren, Wenxi Liu, Yongtuo Liu, Haoxin Chen, Guoqiang Han, and Shengfeng He. Reciprocal transformations for unsupervised video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, et al. Grounded sam: Assembling open- open-world models for diverse visual tasks. arXiv preprint arXiv:2401.1411 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10695, 2022.
Olaf Ronneberger, Philipp
Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted Intervention-MICCA. 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part IIl 18, pp. 234-241 Springer, 2015.
Mehdi Safaee, Aryan Mikaeili, Or Patashnik, Daniel Cohen-Or, and Ali Mahdavi-Amiri, Clic: Concept learning in context. 2023. URL
http:/bbs/2311111 17083
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479-36494, 2022.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Blang-Chihh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4510-4520, 2018
Christian Schmidt, Ali Athar, Sabarinath Mahadevan, and Bastian Leibe. D2conv3d: Dynamic dilated convolutions for object segmentation videos. Proceedings of the IEEE/CVE Winter Conference Applications of Computer Vision (WACV), 2022.
14

---

Published as a conference paper at ICLR 2025
Hyeonseob Seong, Scoung Wug Oh, Joon- Young Lee, Scoung Wug Lee, Seokju Lee, and Euntai Kim. Hierarchical memory matching network for video object segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). 2021.
Chenyang Si, Ziqi Huang, Yuming Jiang, and Ziwei Liu. Freeu: Free lunch in diffusion u-net. In Proceedings of the IEEE/CVF Conference Computer Vision and Pullern Recognition, pp. 4733-4743, 2024.
Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14770000
2022.
Ali Sohail, Naeem A Nawaz, Asghar Ali Shah, Saim Rasheed, Sheeba Ilyas, and Muhammad Khurram Ehsan. A systematic literature review on machine learning and deep learning methods for semantic segmentation. IEEE Access, 2022.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint 3xXiv:2011.134560 2020b.
Tiankang Su, Huihui Song, Dong Liu, Bo Liu, and Qingshan Liu. Unsupervised video object segmentation with online adversarial self-tuning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 688-698, 2023.
Nontawat Tritrong, Pitchaporn Rewatbowornwongg and Supasorn Suwajanakorn. Repurposing gans for one-shot semantic part segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp.
2021.
Hao Wang, Weining Wang, and Jing Liu. Temporal memory attention for video semantic segmentation. In 202 IEEE International Conference on Image Processing (ICIP), pp. 2254-22222222 2258. IEEE, 2021 l.
Haochen Wang, Xiaolong Jiang, Haibing Ren, Yao Hu, and Song Bai. Swiftnet: Real-time video object segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1296-1305, 2021b.
Qian Wang, AbdeIrahman Eldesokey, Mohit Mendiratta, Fangneng Zhan, Adam Kortylewski, Chris- tian Theobalt, and Peter Wonka. Zero-shot video semantic segmentation based on pre-trained diffusion models. arXiv preprint arXiv:2405. 2024.
Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang. Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context. arXiv preprint arXiv:2304.03284 2023.
Meng Wei, Xiaoyu Yue, Wenwei Zhang, Shu Kong, Xihui Liu, and Jiangmiao Pang. Ov-parts: Towards open-vocabulary part segmentation. Advances in Neural Information Processing Systems, 36, 2024.
Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. arXiv preprint arXiv:2212.11560 2022.
Tianxing Wu, Chenyang Si, Yuming Jiang, Ziqi Huang, and Ziwei Liu. Freeinit: Bridging initializa- tion gap in video diffusion models. arXiv preprint arXiv:2312.07537, 2023.
Haozhe Xie, Hongxun Yao, Shangchen Zhou, Shengping Zhang, and Wenxiu Sun. Efficient regional memory network for video object segmentation. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 1286-1295, 202
Hongwei Xie, Hongxun Yao, Shun Zhou, Shengping Zhang, and Wenxiu Sun. Efficient regional memory network for video object segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), 2021
15

---

Published as a conference paper at ICLR 2025
Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang. Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-tor to--uueeee--- video object segmentation, 2018a. /tthttps::/ http//axxivv.......... wwwpdvv.rrgabb//aaa0 aww.aass./55 00461
Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. arXiv preprint
arXiv:1809.03327, 2018b.
Linjie Yang, Yuchen Fan, and Ning Xu. Video instance segmentation. Proceedings of the IEEE/CVF international conference computer vision, pp. 5188-5197,
Shu Yang, Lu Zhang, Jinqing Qi, Huchuan Lu, Shuo Wang, and Xiaoxing Zhang. Learning motion- appearance co-attention for zero-shot video object segmentation. In Proceedings of the IEEF/CVF International Conference Computer Vision (ICCV), 2021a.
Zehao Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by foreground- background integration. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Zehao Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In Advances in Neural Information Processing Systems (NeurIPS), 2021b.
Lijun Yu, Yong Cheng, Kihyuk Sohn, José Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming-Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10459-10469, 2023.
Kaihua Zhang, Zicheng Zhao, Dong Liu, Qingshan Liu, and Bo Liu. Deep transport network for unsupervised video object segmentation, Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2021.
Tao Zhang, Xingye Tian, Yu Wu, Shumping Ji, Xuebo Wang, Yuan Zhang, and Pengfei Wan. Dvis: Decoupled video instance segmentation framework. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 1282-1291, 2023a.
Tao Zhang, Xingye Tian, Yikang Zhou, Shumping Ji, Xuebo Wang, Xin Tao, Yuan Zhang, Pengfei Wan, Zhongyuan Wang, and Yu Wu. Dvis++: Improved decoupled framework for universal video segmentation. arXiv preprint arXiv:2312.13305. 2023b.
Yabo Zhang, Yuxiang Wei, Dongsheng Jiang, Xiaopeng Zhang, Wangmeng Zuo, and Qi Tian. Con- trolvideo: Training-fris frre controllable text-to-video generation. arXiv preprint arXiv:2305.13077 2023c.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2881-2890, 2017.
Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022.
Xizhou Zhu, Yuwen Xiong, Jifeng Dai, Lu Yuan, and Yichen Wei. Deep feature flow for video recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2349-2358, 2017.
Zixin Zhu, Xuelu Feng, Dongdong Chen, Junsong Yuan, Chunming Qiao, and Gang Hua. Exploring pre-trained text-to-video diffusion models for referring video object segmentation. arXiv preprint 1rXiv:2403.12042 2024.
16

---

Published as a conference paper at ICLR 2025
7 APPENDIX
7.1 SMITE-50 DATASET
SMITE-50 is video dataset that covers challenging segmentation with multiple parts of the object being segmented in difficult scenarios like occlusion. consists of 50 videos, up to 20 seconds long, from 24 frames to 400 frames with different aspect ratios (both vertical and horizontal). Frame samples of the dataset are provided in Fig in the main paper. Primarily our dataset consists of different classes "Horses", "Faces", "Cars" and Next"" Out of these sequences, "Horses" and "Cars" have vidcos which have been captured outdoors hence it has challenging scenarios including occlusion, view-point changes and fast-moving objects with dynamic background scenes. "Faces" sequence on the other hand have videos with occlusion, scale changes and more fine-grained parts which are difficult to track and segment across time. The "Non-Text" category has videos which have parts that cannot be described by natural language cues. Hence, these videos are difficult for zero-shot video segmentation (Ren et al. 2024) approaches most of the models are reliant on textual vocabularies for the segmentations. As compared to existing VOS datasets (refer to Table 5 like DAVIS-2016 AAVIs-2016/Prrzzzi al. (2016), PUMAVOS Bekuzarov et al. (2023) and Youtube- VOS Xu et al.) (2088)), SMITH-50 has more parts per video and many multi-par videos. Although Youtube- (YT-VOS) has more videos but all of the videos lack multi-pat part annotations. SMITE-50 is working progress and we intend to expand it beyond what it currently includes and make it publicly available.
Table 5: Comparison of various VOS datasets with SMITE-50.
Dataset
DAVIS-2016
PUMAVOS
YT-VOS
SMITE-50
training/tcsting videos
30/20
0/24
3945/508
0/50
Avg parts per video
1.08
1
6.39
Number of multi-part videos
0
0
50
7.2 EXTRA ABLATIONS
Performance on PUMaVOS and comparison with XMem++. Here, we provide comparison with XMem++ on their proposed dataset PUMaVOS to show that our method is effective on other datasets than our proposed SMITE-50. For this evaluation, we considered the following seven video splits namely :Chair, Full face 1, Full face 2. Half face 1, Half face 2, Long Scene scale and Vlog repectively. We chose the following categories as the object parts in these videos are not too small and also have flexible granularity. Note that XMem++ is a semi-supervised video segmentation technique meaning that it has been trained on largescale video segmentation datasets such as DAVIS (Pont-T iet al. (207)) or Youtube VOS (Xu et al. (0018)). Nevertheless, SMITE performs comparably and even outperforms in case of working with fewer frames (e.g., a single shot; see Tab. 66. Predictably, when XMem++ is provided with more frames per video (e.g., 10), its performance slightly exceeds ours since SMITE not trained on large segmentation dataset. However, with smaller number of frames (e.g.. one or five), our method either outperforms or matches the performance of XMem++ See Tables 6 and 7 for the quantitative comparisons.
Table 6: Comparison on a subset of PUMaVOS dataset (Bekuzarov et al. 2023
when only one
frame is used for training.
Chair
Full face 1
Full Face 2
Half Face
Method
F Fmaus.
mIOU
F Fmaos.
mIOU
F Fnecas.
mIOU
F
mIOU
GSAM2
0.49
58.82
0.99
97.47
0.94
94.78
0.29
57.66
Baseline-I
0.46
73.15
0.61
85.23
0.7
86.9
0.02
82.83
XMem+
0.99
95.72
0.71
90.75
0.80
89.92
0.82
90.52
Ours
0.32
63.32
0.98
96.46
0.85
90.38
0.55
79.75
Half Face 2
Long Scene Scale
Vlog
Mean
Method
meous.
mIOU
F Fmoox.
mIOU
moons.
mIOU
mIOU
GSAM2
0.54
74.78
0.99
97.39
0.16
42.99
0.63
74.84
Baseline-1
0.18
55.78
0.74
87.74
0.73
78.90
0.5
74.91
XMem++
0.48
71.03
0.87
95.48
0.16
31.11
0.69
80.65
Ours
0.37
69.91
0.98
96.27
0.75
78.91
0.69
82,14
17

---

Published as a conference paper at ICLR 2025
Table Quantitative results on subset of PUMaVOS dataset (Bekuzarov et al. (2023)). At k we achieve higher quality in terms of F, measare and mIoU, outperforming XMem++. For other settings, we experience small margin of loss, which acceptable given that XMem++ is fully supervised method, whereas our approach is a fho-shot method. Methods
frame
frames
10 frames
Fmeas.
mloU
F mecas.
mloU
F Fmeas.
mloU
Full Face (XMem++)
0.71
90.75
1.0
98.78
1.0
99.01
Full Face (Ours)
0,98
96.46
0.99
96.76
1.0
96.73
Full Face (XMem++)
0.80
89.92
0.96
96.64
0.97
97.35
Full Face (Ours)
0.85
90.38
0,91
93.10
0.93
93.78
Chair (XMem++)
0.99
95.72
1.0
96.57
1.0
96.65
Chair (Ours)
0.32
63.32
0.98
90.62
0.99
89.82
Half Face I (XMem++)
0.82
90.52
0,94
94.54
0.96
95.49
Half Face 1 (Ours)
0.55
79.75
0.92
90.69
0.93
91.37
Half Face 2 (XMem+++
0.48
71.03
0.77
87.87
0.85
91.41
Half Face 2 (Ours)
0.37
69.91
0.66
81.06
0.83
87.17
Long Scene Scale (XMem++)
0.87
95.48
0.99
98.36
1.0
98.91
I Long Scene Scale (Ours)
0.98
96.27
1.0
96.87
1.0
96.79
Vlog (XMem++)
0.16
31.11
0,55
62.84
0,82
82.52
Vlog (Ours)
0.75
78.91
0.86
84.01
0.90
85,29
Mean (XMem++)
0.69
80,65
0.89
90.80
0.94
94,48
Mean (Ours)
0.69
82.14
0.90
90.44
0.94
91.56
Joint training vs two-phases training, We employ two-phase training instead of joint training due its increased stability and general improved performance. We have ablated our training recipe on SMITE-50 dataset in this experiment. Based on the results in Tab. 8 it is evident that two phase training gives us better performance quantitatively. In addition this, we performed Levene's test on the training strategies to asses the training stability of both. We observed significant difference (p 0.0036) between the variance losses in joint training and two-phase training. This ensures that two-phase training is more stable which can also be reflected visually in Fig which confirms findings. In both the cases, the number of epochs and the training time is same.
Table 8: Ablation of joint vs two-phase training on SMITE-50.
Training Strategy mIOU
Training Time
Joint 62.17
20 mins
Two-Phase 75.14
20 mins
Cross-category generalization. SMITE capable of of segmenting objects that differ from the train- ing samples. For example, when trained horses, it is able to perform acceptable segmentation on camels and giraffes as shown in Fig a0 a). For intra-class categories like in car category, even we train SMITE with reference images of Sedan car, our model can still segment SUV car which is structurally varying.
Multi-Object Flexible Granularity. SMITE supports multi-object flexible granularity. For instance, when presented with a training image containing both a horse and a car, the model learns distinct representations for each object simultaneously shown in Fig 11 This enables the model to generalize effectively, handling multiple objects with varying levels detail in a cohesive manner.
7.3 EXTRA QUANTITATIVE RESULTS
Performance on PUMaVOS and comparison with XMem++. Here, we provide comparison with XMem++ on their proposed dataset PUMaVOS to show that our method is effective on other datasets than our proposed SMITE-50. For this evaluation, we considered the following seven video splits namely :Chair, Full face 1, Full face 2, Half face 1, Half face 2, Long Scene scale and Vlog respectively. We chose the following categories as the object parts in these videos are not too small and also have flexible granularity. Note that XMem++ is semi- cemi-supervised video segmentation technique meaning that it has been trained on large scale video segmentation datasets such as DAVIS or Youtube VOS (Xu et al. 2018b). Nevertheless, SMITE performs comparably and even outperforms
18

---

Published as a conference paper at ICLR 2025
Low Filer Segnerth
Tracked Segnent
Correctted Segrent
Ouile
LowFon Regaartahon
Smooth and Conderint
Figure 8: Importance of using LPF While applying tracking it leads to segments which has jagged edges and pixel drifts. Using LPF on the segments, treats it as a guide which corrects the inconsistent segments after tracking into spatially consistent segments which preserves the original structure.
Comparison of Mean Validation Loss with Standard Deviation
Mean Loss Two phase Training
Mean Loss joint Training
74-
72
Loss
68-
66
50
100
150
200
250
300
Epoch
Figure 9: Two phase training highlighted as blue shows significant less oscillation in training and provides more stability.
it in case of working with fewer frames (e.g., a single shot; see Tab, 6 Predictably, when XMem++ provided with more frames per video (e.g., 10), its performance slightly exceeds ours since SMITE is not trained on a large segmentation dataset. However, with smaller number of frames (e.g., one or five), our method either outperforms or matches the performance of XMem++. See Tables and for the quantitative comparisons.
Performance on DAVIS Dataset. Note that the strength of our method to perform arbitrary granularity and multi- multi-granular segmentation and DAVIS only contains single objects rather than different granularity levels. To further contrast our method to SOTA for single object segmentation, we evaluate our dataset DAVIS-2016 for the video object segmentation (VOS) task using two different settings: a) Zero-Sho VOS in Table b) Semi- Supervsuee VOS in Table 10 respectively. The first setting follows zero-shot approach, where train model video from the training set and report the results. However, our method has a limitation struggles with small objects. This is common limitation for most existing diffusion based segmentation
randomly selected image per
19

---

Published as a conference paper at ICLR 2025
Table 9: Quantitative evaluation on DAVIS-2016 in a zero-shot setting. Our method excels in multi-granular segmentation, but DAVIS-2016 only contains single objects. To compare with SOTA methods, we evaluate DAVIS- 2016 in zero-shot setting, training on one image per video from the training set. The results are reported SMITE on the entire dataset and as SMITE* on the dataset excluding two samples (2/20) that are small and unsuitable for our method.
Methods
Jf
Ff
F'NNet (Ji al.
2021
83.4
83.1
F2Net (Lau et al.
2021
83.1
84.4
TransportNet (Zhang
2021
84.5
85.0
AMCNet Yang al 20222
84.5
84.6
RTNet (Ren et al. 2021)
85.6
84.7
CFANet (Chen al 1022
83.5
82.0
D2Conv3D (Schmid ci
2022
85.5
86.5
IMPNet (Lec
21122)
84.5
86.7
DBSNet Fan ct al
2022
85.9
84.7
HFAN (Per el al. 2022
86.2
87.1
TMO C ho et al. 2025
85.6
86.6
OAST Su et al.
2023
86.6
87.4
SMITE
80.8
82.9
SMITE
82.39
87.6
Table 10: Quantitative evaluation on DAVIS-2016 in semi-supervised setting. Our method is designed for multi-granular segmentation, but DAVIS-20 2016 only contains single objects. To compare with SOTA methods, we evaluate DAVIS-2016 in semi-supervised setting, training one image per video from the validation set. The results are reported as SMITE on the entire dataset and as SMITE the dataset excluding two samples (2/20) that are small and unsuitable for our method. Method
Jf
Ff
STM (Oh et al. 2019)
88.7
89.9
AFB-URR (Liang al.
2020
88.3
90.5
CFBI Yang et al. 2020
88.7
RMNet X1e ct al.
202
89.6
92.0
HMMN Scong ct al
2021
89.6
92.4
MIVOS
Cheng ct
202
90.8
92.5
STCN
Cheng
2021b
90.1
92.1
JOINT
Mao et al.
2021b)
90.4
92.7
AOT
Yang et al.
2021b)
91.1
92.1
XMem
Cheng & Schwing
2022
91.5
92.7
SMITE
83.4
80.7
SMITE
92.0
88.7
20

---

Published as a conference paper at ICLR 2025
Homse Giraffe Generoiization
Tran SMIE with "Norse"
Reference Iniign
Horse Greneraizatio
(a)
Sedon - SUV Generdizztion
main SMIT with "Sedon"
Reference brages
(b)
Figure 10: Cross-Categon Generalization SMITE supports cross-category generalization where dispite of training with reference images of (a) horse category, our model can successfully generalize to ciraffe.camee which has large topological changes and training with reference images of (b) sedan car, our model can successfully generalize to SUV car which has intra-category variations.
approaches] Namekata et al. (2024); Khani al. (2024). Additionally, two out of the 20 videos contain thin lines to be segmented (Fig 13). making them unsuitable for segmentation using our method. Therefore, in addition to reporting numbers on the entire test-set, we also exclude these two videos and report the average performance (SMITE+) on the remaining DAVIS-201 videos.
Secondly, we evaluate our method a semi-supervised segmentation setting, where only a single frame is used for each test video. While many models, both in zero-shot and semi-supervised settings, are specifically trained for video segmentation using videos from datasets like YouTube VOS or DAVIS, approach requires only few frames for training. Despite this minimal requirement, achieve competitive performance on most examples, except in cases involving small objects, where method faces limitations.
21

---

Published as a conference paper at ICLR 2025
Figure Multi-Object Segmentation. Our method is capable of multi-object segmentation.
Imput Video
Opterization
Figure 12: Importance of applying Tracker. We ablated the usage tracking algorithm in two scenarios (a) During postprocessing (second row) and (b) During inference optimization (last row). From the figure, can be observed that applying tracker on the segments Swas during postprocessing (Baseline-II] leads to segment drifts thus resulting jagged edges. Such problems can be mitigated if we use tracking during inference optimization (Eq. (5) stage since the optimization process slowly propagates the corrections across the entire video during the denoising stage, thus resulting in lesser segment drift and temporally consistent predictions.
7.4 EXTRA QUALITATIVE RESULTS
For more challenging cases, include more results in Fig 14 In one instance, the ice cream cone is occluded by paper napkin, and the ice cream itself is obscured and blended by a face, yet SMITE still able to generate correct results. Furthermore, the turtle nearly blends with the background in terms of color and visual patterns, but our method successfully tracks the segments.
We have included qualitative comparison videos in the supplementary file provide examples in video format. The actual videos are in higher resolution, we have compressed the video to upload the website easily.
22

---

Published as a conference paper at ICLR 2025
Input
SMITE Prediction
DAVI GT
Figure 13: DAVIS Test Set Visualization SMITE segments poorly in scenes which has thin structures like strings in both the scenes.
"Occhalonn"
"Camouflage"
Figure 14: Segmentation results in challenging scenarios. SMITE accurately segments the objects under occlusion ("ice-cream") or camouflage ("turtle") highlighting the robustness of our segmentation technique.
7.5 IMPLEMENTATION DETAILS AND DESIGN CHOICES EXPLANATION
All experiments conducted single NVIDIA RTX 3090 GPU. During the learning phase, initially optimized only the text embeddings for the first 100 iterations. For the subsequent iterations, optimized the cross-attention
and to v parameters.
For the categories of horses, cars, faces and non-text, we used 10 reference images from our SMITE- 50 benchmark for both our method and the baseline comparisons.
Regarding the window size in our tracking module, we found that fast-moving objects benefit from smaller window size mitigate potential bias. Consequently, set the window size to 7 for horses
23

---

Published as a conference paper at ICLR 2025
and 15 for other categories. To better work on smaller objects, we employ two-step approach. First, our model generates an initial segmentation estimate. We then crop the image around this initial estimate and reapply our methods to obtain more fine-grained segmentation.
During inference, we added noise corresponding to 100 timestops and performed a single denoising pass when segment tracking and voting were not employed. When using segment tracking and voting, we applied spatio-temporal guidance at each denoising step and conducted backpropagation 15 times per denoising timestep. For the regularization parameters, we AReg across all experiments. The tracking parameter Ärrcking was set to I for horses, 0.5 for faces, and either 0.2 or for cars. Additionally, we applied Discrete Cosine Transform (DCT) low-pass filter with threshold of 0.4.
Pseudo Code of Temporal Voting
Algorithm Temporal Voting
1: Input: X: a pixel at frame f. W: window size
2: X, <- Correspondence of at frame & (obtained by CoTracker(X,
3: Vis(X,, visibility of X, (obtained by CoTracker)
4: Visible t fi E range (- W W) if Vis(X,.) == 1}
2
5: P < Most_Occurrence (S( (S(X;).arra ..x.).aaggaxxxiin = 0)) where i Visible_Set
6: total < 0, count <
7: for all e Visiblc_Set do
8: if' ((Xxx))arrgaxx((mm=))
P then
9:
total +- total + S(X;)
count +- count +
11: end if
12: end for
Streeked (X) total
count
Bidirectional Tracking and Resolution Reduction in CoTracker Note that points queried at different frames are tracked incorrectly before the query frame. This is because CoTracker is an online algorithm and only tracks points in one direction. However, we can also run it backward from the queried point track in both directions. So by setting backward tracking to True we are able to track points in both directions which is crucial for our voting mechanism.
Time and Memory Analysis. Using 10 training images with a batch size of on 15 GB of GPU VRAM, training takes 20 minutes. By increasing the batch size and utilizing more GPU VRAM, training time can be reduced to minutes. Although we use 300 epochs, the best accuracy is often reached before epoch 200, so training time can be further decreased. Training time is unaffected by whether we learn the text embedding or U-Net weights. For inference, each frame takes 26 seconds and requires 60 GB of GPU VRAM to process the entire video. However, is possible adjust settings to use only 15 GB, though this increases the inference time.
Long video processing, A key goal for us is adapt our method to work efficiently smaller GPUs. Unlike most video editing techniques that require high-end hardware like an A100 GPU and typically handle up 24 frames, our approach aims for broader applicability. We identified that gradient computation during inference optimization is particularly demanding on resources. To address this, we segment the latent space into smaller windows-for example, (1 k). (k+1 to 2k), (2k+1 3k), and so ae--cross different timesteps and optimize each window independently. This segmentation has proven not to compromise the final results. Additionally, for tasks such as segmenting 1,000 frames (PUMAVOS samples) consistently, our method processes batches of 2xT frames at a time (you can set based on your GPU's capacity). We then save the states of the last (T+1) frames and replace them with the reference state in the next iteration. These strategies enable our model to process longer videos effectively on GPUs with as little as 24 GB of VRAM.
Enhanced Convergence. The strategy to accelerate convergence and simplify parameter tuning in the code involves the use of an Adam-like optimization approach that dynamically adapts the learning rate and gradient updates for the latent variables. Specifically, the code implements the first and second moment estimates, denoted as M1 and M2, which accumulate the gradients and squared gradients, respectively.
24

---

Published as a conference paper at ICLR 2025
In each iteration, the first moment estimate M1 captures the exponentially weighted average of the gradients, while the second moment estimate M2 tracks the squared gradients. These moment estimates then corr-coree by dividing them by 1 - 0|+1 and _ where 3n and 82 are the momentum parameters typically set to 0.9 and 0.999, respectively. This bias correction ensures that the estimates unbiased, particularly in the initial steps of optimization.
The learning rate a is dynamically scaled based on these corrected moment estimates, where the update step for the latent variables computed as:
M corrected 2 2: 04 M2correcedd +
This adaptive approach allows the optimizer to adjust the learning rate on per-parameter basis, depending on the variance of the gradients, leading to faster convergence. By using this method, the optimizer can take larger steps when the gradients are consistent and smaller steps when they are noisy, which helps in avoiding overshooting or getting stuck in local minima. The combination of momentum-baas based updates and dynamic learning rate scaling makes the optimization process more robust, reducing the need for extensive manual tuning of hyperparameters such as the learning rate, and enabling more efficient convergence.
Bidirectional tracking. We use bidirectional tracking instead of unidirectional tracking for two main reasons. First, to manage longer videos, implement slicing approach where the last frames of the first slice are retained in the second slice to ensure continuity. Bidirectional tracking speeds up consistency between slices by allowing new frames in the second slice to directly reference frames from the first slice, while unidirectional tracking delays this process due to the need for updates to propagate. Second, tracking methods often struggle with fast-moving objects, as accuracy decreases with distance from the query pixel, risking loss of track. Bidirectional tracking enhances robustness in these situations. Additionally, in cases of occlusion, unidirectional tracking may fail without visible pixels to propagate information. Bidirectional tracking mitigates this by leveraging data from both past and future frames, maintaining accuracy even during occlusions.
7.6 How DOES WAS ATTENTION WORK?
SLiMe refines latent diffusion models by learning special text embeddings that guide segmentation using the Weighted Accumulated Self-Artention (WAS) map. In each denoising step. a latent diffusion U-Net uses cross attention to link latent features with text tokens, and self-attention capture global spatial information. SLiMe combines these two forms of attention through WAS, enabling it to identify clear boundaries and consistent object regions, even in unseen images.
Weighted Accumulated Self-Attention. Let S be the cross- attention map and S be the self- attention map at layer L. We first average resized coss-attention maps:
A Averago, Resize ((S%
and do the same for self-attention maps:
Aga Average((St (a)
Next, we resize A (the map for the k-th text embedding) to match the shape of Asa:
R = Resize (A
then flatten R& and multiply it clement-wise by Asa. Summing the result across channels gives the WAS map:
WAS = Σ flatten R @ Asa).
This step combines the text-based localization (from cross-attention) with the detailed spatial infor- mation (from self-attention) leading sharper and more meaningful segmentations.
25

---

Published as a conference paper at ICLR 2025
Loss Functions. The text embeddings optimized using three loss functions. The cross-entropy loss encourages the cross- attention maps to align with the ground truth mask M:
LEE
CE(Sca; M).
The MSE loss on the WAS map aligns self-attention with the ground truth:
MSSE
Σ
[Resize(SWi. - Mx
Finally, the diffusion based regularization term keeps the learned embeddings within the pretrained latent diffusion distribution:
LLOM = Ban, c, 60(2t, L, 1) w)1l
The total loss is:
L LCE + LMSE LLDM.
where a and B control how strongly we emphasize spatial accuracy versus staying consistent with the original diffusion model. Once trained, SLiMe's embeddings can segment new images similar granularity without extra fine-tuning.
7.7 IMAGE SEGMENTATION RESULTS
We tested our method on an image dataset (e.g Phen et al. 2014) to demonstrate the enhancements achieved through modifications to architecture and optimization. As shown in Tables [] and 12 our approach shows significant improvement over SLiMe even for image segmentation for and horse split of PASCAL-Part dataset. This highlights the effectiveness of design choices in SMITE, particularly the ross-attention tuning.
Table 11: Image segmentation results for class car SMITE consistently outperforms SLiMe. The first two rows show the supervised methods, for which we use the reported numbers in ReGAN. The second two rows show the methods with 1-sample setting and the last three rows refer to the 10-sample setting methods. indicates the supervised methods.
Body
Light
Plate
Wheel
Window
Background
Average
CNN
73.4
42.2
41.7
66.3
61.0
67.4
58.7
CNN+CRF
75.4
36.1
35.8
64.3
61.8
68.7
57.0
SegGPT (Wang et al. 2202)**
62.7
18.5
25.8
65.8
69.5
77.7
53.3
OlParts (Dai et al. 2024)
77.7
59.1
57.2
66.9
59.2
71.1
65.2
ReGAN Tritrong el a 2021)
75.5
17.8
57.2
62.4
70.7
52.15
SLiMe (Khan el al. 2024)
81.5
56.8
54.8
68.3
70.3
78.4
68.3
Ours
82.3
57.5
55.9
70.1
72.6
80.1
69.8
Table 12: Image segmentation results for class horse. SMITE outperforms ReGAN, OPParts, SegDDPM, SegGPT and SLiMe on average and most of the parts. The first two rows show the supervised methods, for which use the reported numbers ReGAN. The middle rows show the 1-sample setting, and the last four rows are the results of the 10-sample settings. indicates the supervised methods.
Hcad
Leg
Neck+ Torso
Tail
Background
Average
Appereance CNF+CRE"
47.2
38.2
66.7
55.0
46.8
SegGPT (Wang et al. 2023
41.1
49.8
58.6
15.5
36.4
40.3
OlParts Dai al. 2024
73.0
50.7
72.6
60.3
77.7
66.9
ReGAN (Tntrong al.
50.1
49.6
70.5
19.9
81.6
54.3
ScgDDPM
Baranchuk et all 2021
41.0
59.1
69.9
39.3
84.3
58.7
SLiMe
Khami et al. 2024
63.8
59.5
68.1
45.4
79.6
63.3
Ours
64.5
61.9
73.2
48.1
83.5
66.2
26


Reviews:
[{"review_id": "xdfIbElj3r", "reviewer": "Reviewer_9Vaw", "summary": "This paper defines a task of flexible granularity, which aims to segment user-intended granularity temporally using one or a few reference images. To do so, it proposes several techniques to achieve temporal consistent multi granularity segmentation including alternative learning of text embeddings and cross attention module, tracking modules with voting mechanism to be aware of neighbor pixels and low frequency regularizer for spatial consistency. It shows its effectiveness on the proposed dataset, SMITE-50", "questions": "- Is the proposed method also applicable to \"Multi-object flexible granularity\", which indicates that reference images annotated with multiple objects (or multiple classes) different granularity can be used as input to the proposed model?\n\n- Could the author provide any failure cases?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Belows are the strong points that this paper has:\n\n- It is well-written and well-organized, which help reader to understand new task and motivation of the proposed method.\n\n- The proposed techniques including learning generalizable segments using alternative tuning of text embeddings and cross attention, temporal consistency via tracking voting mechanism, and spatial consistency with low-pass filter are in good harmony, which can outperform the other baselines with high margin.", "weaknesses": "Belows are points that the reviewer feel concerned about.\n\n- In line #247, the authors mention that learning text embeddings and cross-attention layers is conducted in two phases to provide a better initialization for the next phase. However, this phased or alternating training approach could also increase training complexity. Could the authors provide an ablation study to compare the results of joint training versus alternating training to justify the chosen training setup? Specifically, analysis on training time with final performance of two training strategies can be provided.\n\n- In the supplementary video, specifically at 1:46 in the \"Qualitative_Comparison\" section, it appears that the proposed method captures a relatively smaller region of the \"eye\" area compared to the baseline. Is this due to the proposed voting mechanism diluting the effective area by averaging nearby major segments? Would this issue be mitigated by replacing the current hard voting with a soft voting mechanism? Is it doable that quantitative analysis on different voting system can be measured?\n\n- Is the proposed method shape-agnostic within the same class? To demonstrate that the method can segment other unseen images, the authors should provide test samples that differ in shape from the reference object. For example, as similar to what the author showed in \"Cars\" of Figure 7, if the reference image includes a detailed segmentation map of a \"sedan\" car, the test samples can be replaced with different shapes of cars, such as \"SUVs?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7nvMdWOuaA", "reviewer": "Reviewer_rK61", "summary": "This paper presents SMITE (Segment Me In Time), a novel approach for one/few-shot video object part segmentation with flexible granularity. Building upon the image-based SLiMe method, SMITE leverages a pre-trained text-to-image diffusion model (Stable Diffusion) and introduces a tracking mechanism and low-pass filtering to maintain temporal consistency in video segmentations. Given one or a few annotated reference images, SMITE aims to segment unseen videos of objects from the same category, respecting the provided segmentation granularity. The authors introduce a new dataset, SMITE-50, to benchmark performance and compare against relevant baselines.", "questions": "1. How sensitive is the method's performance to the choice of tracking algorithm? Have you experimented with other tracking methods besides CoTracker?\n1. The low-pass regularization is mentioned as a key component. Can the authors elaborate on how this regularization term is formulated and how it balances preserving segment structures with smoothing transitions?\n1. How does the proposed method handle videos with rapid movements, severe occlusions, or significant lighting changes? Are there strategies to mitigate potential performance degradation in such scenarios?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Originality : The proposed method innovatively combines pre-trained text-to-image diffusion models with a temporal tracking mechanism to achieve video segmentation with flexible granularity using only a few reference images.\n1. Quality : The comprehensive experiments, including quantitative evaluations on the new SMITE-50 dataset, qualitative comparisons, ablation studies, and user studies, provide solid support for the method's effectiveness.\n1. Clarity : The paper is well-structured, with clear explanations of the problem, methodology, and results. Figures and diagrams are used effectively to enhance understanding.\n1. Significance: Addressing the challenge of few-shot video object part segmentation has practical implications for various applications, including VFX and video editing.", "weaknesses": "1. Methodological Clarity: While the overall explanation is detailed, some of the methodology is difficult to follow, especially for readers not familiar with diffusion models and the previous work SLiMe. The process of how tracking integrates into the attention map refinement could be clearer, with more intuitive descriptions of its working mechanism.\n1. Dataset & Baselines: The new SMITE-50 dataset is an important contribution, but it appears somewhat small (with 50 videos). Expanding the dataset size or providing an analysis of its generalization for more categories could strengthen the paper’s claims. A comparison to more widespread benchmarks for video segmentation, such as DAVIS or other VOS datasets, would help position SMITE relative to not only a niche dataset but also larger, well-accepted benchmarks in the community.\n1. Inference Efficiency: The paper does not deeply explore the computational cost or inference time of the method in practical settings, which would be relevant given the addition of several components (e.g., temporal voting, low-pass filters). Addressing this in the discussion or providing benchmarks on efficiency could help increase understanding of real-world applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TSe3NocAV4", "reviewer": "Reviewer_aCvu", "summary": "This paper proposes a SMITE framework for segmenting object fragments across video frames. The method consists of denosing module, diffusion module and tracking module. Overall, it achieves satisfactory performance for video segmentation over various levels of granularity.", "questions": "Same as weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1)This proposed method combines information from text, tracker and spatial temporal features for video segmentation at arbitrary level.\n2)SMITE achieves state of the art performance on the SMITE-50 dataset.", "weaknesses": "1)The proposed method is only compared with two methods in Table 1, while there are lots of few-shot and zero-shot segmentation methods. \nBesides, it seems that the proposed method can be directly applied on multi-target video segmentation datasets like DAVIS2017. So, there should be more comprehensive comparison, for both methods and datasets.\n2)Speed is important for online tracking; so there should be some runtime analysis.\n3)Limitations of the method is not discussed in experiments.\n4)How does the proposed method work on random videos and random annotation references?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CkMo3IhVnB", "reviewer": "Reviewer_K95K", "summary": "This work presents a framework for segmenting videos into parts at adjustable levels of granularity, using only a single or few annotated images as references. The framework comprises several main components:\nFirst, it optimizes text embeddings and fine-tunes a stable diffusion model so that the WAS attention maps (a combination of self and cross-attention maps) align with the annotated reference images.\nNext, temporal consistency is enforced through specific loss terms, enabling the framework to generate segmentation maps from the input video using the optimized text embeddings and fine-tuned diffusion model.", "questions": "1. The differences in Figure 3 could be better presented (the difference between (c) and (d) is hardly visible).\n\n2. It was slightly challenging to follow all the technical details. \nFew small modifications might reduce the reading time:\nFor example: \na. In section 4.2 3 loss terms were mentioned and in section 4.4 2 other loss terms were mentioned; It will be clearer if the authors stated clearly how and in what phase each loss term is used (all together? two-phase optimization?).\nb. lines 267-268, sliding window -> temporal sliding window\n\n3. Small technical note:\nline 148: should the epsilon_theta after UNet be deleted?\n\n4. Lines 303-305:\nIt is not clear how this scenario is different from using only cross attention maps. \nIf I understand what the authors suggested correctly: average voting can be computed directly on the masks as a \"tracking post-process step\" without the need for an optimization for tracking.\nIf so, why can't you simply calculate the average vote on the S_WAS as well?\nIf not, what was your intention?\n\n5. Line 105 - you mention that some parts cannot be described by text. \nYet, you do find text embeddings that describe them, although it is reasonable to claim that these embeddings do not represent natural text descriptions. \nHowever, it will be interesting to acknowledge/discuss this distinction. \nWhy does it make sense to find text embeddings for parts that cannot be described by texts?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Novel and interesting problem formulation. \nThe motivations behind the design choices are presented clearly.\nQualitative results seem decent.\nA new video part-level dataset and SOTA performance.", "weaknesses": "1. Limited competitors - The baseline (frame-by-frame) might be a bit too simplistic. \nMaybe a better baseline would be a simple video extension of SLIME (without fine-tuning the cross-attention layers). This video extension is pretty trivial and should have some consistency.\nAnother baseline could be using SLIME (single frame) with tracking.\n2. Limited ablation study - \nI would have liked to know what is the contribution of each step.\nIn addition, I think more ablation would make the paper stronger. For example, how similar should be the input object to the one in the video? Would horse segmentation work on zebras? and on dogs?\n\nIn addition: Table 2 only states what happens if only the regularization step is dropped. What about dropping the tracking? And what about dropping both?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4tKt9covO2", "reviewer": "Reviewer_askr", "summary": "This paper presents SMITE, a few-shot video segmentation method with fine granularity tracking. SMITE is based on SLiMe, a few-shot image segmentation method that finetunes text tokens for a diffusion network and probes attention masks to output a mask.  SMITE additionally introduces cross-attention finetuning of the diffusion network, a segment tracking and voting module based on point tracking, and a low-pass regularization term. SMITE achieves better segmentation performance than applying baseline SLiME per frame in a newly proposed dataset SMITE-50.", "questions": "What is the speed and memory requirement of the proposed method? How many frames can it process?\n\nWhat if an inflated U-Net is not used during text embedding optimization (e.g., the segment prompts are just images, not videos) but only used during inference?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper tackles the hard yet useful task of few-shot, fine-granularity video segmentation. This task has potentially wide applications. I think the main contribution of this work is taking the initial crack at this problem with reasonable performance in segmentation quality and temporal stability when compared to applying the SLiME per frame. \n\nTechnique-wise, segmentation tracking/voting and low-pass regularization are reasonable in handling temporal flickering unique to videos, and the results also supported this.\n\nThe proposed dataset SMITE-50 is also potentially useful for future work, as it contains densely labeled segmentation which is a rare type of data.", "weaknesses": "I don’t see it mentioned in the paper, but I suspect that SMITE might be too slow for most practical applications. In addition to finetuning the text tokens and the cross-attention layers during a training stage, SMITE additionally performs dense, windowed point tracking – there seem to be a lot of windows (linear to video length) and a lot of repeated computation in the point tracker (linear to windows size). SMITE also minimizes an energy function which is also costly. Besides, given that SMITE expands the spatial self-attention to the spatiotemporal volume, it would be quite memory intensive for longer videos.\n\nI think the authors can also compare SMITE to existing few-shot video object segmentation methods like [Multi-grained Temporal Prototype Learning for Few-shot Video Object Segmentation, ICCV 2023]. This can happen in two dimensions: 1) applying few-shot VOS to the fine-grained segmentation setting, and 2) applying SMITE to the few-shot VOS setting. This can highlight whether the fine-grained segmentation setting significantly differs from the object setting and supplements the existing sparse baselines.\n\nIn Table 2, should there be a row with neither of the loss and another row with only the low-pass loss? Also, can the ablations be run on the entire SMITE-50 dataset rather than just on the CARS subset? This would make the results more significant.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)