From ViT Features to Training-free Video Object Segmentation via Streaming-data Mixture Models
Authors: Roy Uziel
Or Dinari
Oren Freifeld
Venue: NeurIPS 2023 poster (2023)
Keywords: Unsupervised
video segmentation
clustering
TL;DR: We propose a novel method for SVOS that combines deep learning and classical clustering techniques within a memory-efficient streaming framework.

Abstract: In the task of semi-supervised video object segmentation, the input is the binary mask of an object in the first frame, and the desired output consists of the corresponding masks of that object in the subsequent frames. Existing leading solutions have two main drawbacks: 1) an expensive and typically-supervised training on videos; 2) a large memory footprint during inference. Here we present a training-free solution, with a low-memory footprint, that yields state-of-the-art results. The proposed method combines pre-trained deep learning-based features (trained on still images) with more classical methods for streaming-data clustering. Designed to adapt to temporal concept drifts and generalize to diverse video content without relying on annotated images or videos, the method eliminates the need for additional training or fine-tuning, ensuring fast inference and immediate applicability to new videos. Concretely, we represent an object via a dynamic ensemble of temporally- and spatially-coherent mixtures over a representation built from pre-trained ViT features and positional embeddings. A convolutional conditional random field further improves spatial coherence and helps reject outliers. We demonstrate the efficacy of the method on key benchmarks: the DAVIS-2017 and YouTube-VOS 2018 validation datasets. Moreover, by the virtue of the low-memory footprint of the compact cluster-based representation, the method scales gracefully to high-resolution ViT features. Our code is available at https://github.com/BGU-CS-VIL/Training-Free-VOS

From ViT Features to Training-free Video Object Segmentation via Streaming-date -dta Mixture Models
Roy Uziel
Or Dinari
Ben-Gurion University of the Negev, Israel crielr@post.bgu.. ac.
Ben-Gurion University of the Negev, Israel hwprripptttt.uu..... ac.il
Oren Freifeld
Ben- Len-Gurion University of the Negev, Israel orenfrêcs bgu. ac.
Abstract
In the task of semi-supervised vidco object segmentation, the input is the binary mask of an object in the first frame, and the desired output consists of the corre- sponding masks of that object in the subsequent frames. Existing leading solutions have two main drawbacks: 1) an expensive and typically-supervei videos; large memory footprint during inference, Here we present training- free solution, with low-memory footprint, that yields state-of-ther fi-the--rt results. The proposed method combines pre-trained deep learning- features (trained on still images) with more classical methods for streaming-data clustering. Designed to adapt to temporal concept drifts and generalize diverse video content with- out relying annotated images videos, the method eliminates the need for additional training or fine-tuning, ensuring fast inference and immediate applica- bility to new videos. Concretely, we represent an object via dynamic ensemble temporally- and mnatially-cherree mixtures over a representation built from pre-trained VIT features and positional embeddings. A convolutional conditional random field further improves spatial coherence and helps reject outliers. We demonstrate the efficacy of the method key benchmarks: the DAVIS-201 and YouTube-VOOS 2018 validation datasets. Moreover, by the virtue of the low-memory footprint of the compact cluster-based representation, the method scales gracefully to high-resolution ViT features. Our code available at http:://ithub..om/BBCC CS W-wVVLLraannng-Free----
training on
Introduction
Video Object Segmentation (VOS), key computer-vision task, aims to distinguish objects of interest from the background across sequence of video frames. In Semi-supervised VOS (SVOS), the topic of this paper, the provides the mask of the object in the first frame (while in the case of unsupervised VOS, no mask is given), and the task is to propagate that mask to the next frames. Prior to the revolution of Deep Learning (DL), SVOS relied heavily traditional techniques like color-based segmentation, point trajectories, and optical flow. However, the emergence of DL-based methods has significantly outpaced those traditional techniques. Thus, today DL-based methods dominate the field of SVOS. Note that according to the standard terminology in the area of VOS, the term "semiisuperrviee" in the name SVOS refers to the nature of the task (that is, the fact that user provides the mask in the first frame), not the nature of the solution. In other words, solutions to the SVOS task can be either supervised, sumi-supervieed, or unsupervised. For example, a supervised
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Background
Object
Bockground
Object
Festuree extraction
clustering object
Pixel to to-jectt atsignment
CRP Refinement
Figure 1: Method overview. Starting from the initial frame, the features of the object are extracted and modeled using an ensemble of von Mises-Fisher (vMF) mixtures, where each mixture has different number of components. The higher the number of components, the finer the scale that the mixture captures. This multi-scale representation lets capture phenomena such as drastic changes in the size/resolution
the object. The background is modeled using another such mixture ensemble. Next, all of the mixtures are updated in a streaming and memory-efficient manner at each subsequent frame. A maximun-aaposterion rule applied over all of the vMF components in the ensemble to generate preliminary segmentation mask, which is then further refined by convolutional Conditional Random Field. The iterative process progresses through the frames.
DL solution to SVOS typically requires multiple labeled videos train on. In contrast, the approach proposed in this paper requires neither supervision nor learning on vidcos.
The main strategy employed by DL-based SVOS methods revolves around pixel-level matching to establish inter-frame correspondence. This is achieved by creating pixel-to-pixel connections between the current frame and the preceding ones. That strategy, however, often has two main limitations: 1) the need for an extensive training on large-scale (and typically annotated) video datasets; significant memory footprint due the storage requirements of (dense representations of) previous frames for pixel matching. Addressing those prevalent challenges, we propose an unsupervised, adaptive, and compact SVOS method. By combining DL and classical techniques and capitalizing on the widespread practice of representing features as points on hypersphere, the proposed model incorporates an ensemble of temporally- and spatially-coherent von Mises-Fisher (vMF) mixtures. The mixtures are updated dynamically in a streaming manner, together with convolutional implementation of a Conditional Random Field (CRF).
Our method uses features from a pre-trained unsupervised deep neural network trained on still images, not videos but does not require training by itself only performs inference. The method fits multiple vMF mixtures to each object of interest (and to the background) in the initial frame. The fact that employ multi-scale modeling (where each mixture has its own number of components) lets us represent the object at multiple levels of granularity. In subsequent frames, pixels are assigned the most probable component (across all of the components in all of the mixtures in all of the objects). The mixture ensembles themselves are updated dynamically while taking into account temporal and spatial coherence. Following established stream clustering techniques [13], apply weighted window with a kernel function that reduces the weight of older frames, mitigating memory consumption and improving computational efficiency without storing extensive history of dense representations of past frames. Our model adapts to concept drifts in the objects of interest or the background. We improve spatial consistency by constraining pixel-component membership based on location and refine results using Pixel-Adaptive Convolutional CRF (PAC-CRF) [37], fostering pixel consistency and awareness of outliers. Together with outlier-rejection mechanisms, the method mitigates error propagation across frames.

---

To summarize, our key contributions are: 1) We demonstrate that synergy between classical and DL techniques can provide an effective solution for SVOS that achieves state-of-the-art results on the DAVIS-2017 7 and MerTue---OS 2018 datasets. 2) By leveraging unsupervised pre-trained image-based features, our method eliminates the need for further training, and in particular no training on videos is required (unlike most competitors which require large-scale video datasets for training). 3) Our method has low-memory footprint by storing only cluster-level information, as opposed to dense past-frame representations. Thus, the method does not only scale better than competing ones but also enables handling high-resolution features for higher-quality segmentation.
2 Related work
The evolution of SVOS methods. From older SVOS methods based on classic vision tech- niques [11, 9, 21, 50, 10, 5], the focus has shifted to DL-based SVOS methods, also known as one-shot VOS. Early semi-supervised methods focused on online fine-tuning of a pre-trained network, exemplified by OS [6]. In fact, OSVOS pioneered the online fine-tuning of pre-trained networks in SVOS. However, OSVOS is prone to overfitting and fails to adapt to gradual concept drifts in the objects of interest. To address these issues, numerous extensions and refinements OSVOS were proposed. OnAVOS [41] introduced additional fine-tuning with high-confidence frames in order to handle gradual concept drifts, albeit at the cost of increased computational complexity. Maninis et al. [28] integrated semantic information into the predictions via Mask R-CNN [17], thereby reducing the dependency on temporal smoothness. In an innovative departure from common practice, BubbleNets [16] proposed selecting the most suitable frame for annotation, challenging the conventional practice of annotating the first frame. Other works addressed different aspects of the OSVOS algorithm. For example, OSNM [47] utilized network modulation for more efficient fine-tuning, while A-Game [20] employed a Gaussian Mixture Model to better capture both the foreground and the background. Other noteworthy and relevant works include, but are not limited to, FRTM [35], LWI. [4], and TAODA [51].
The emergence of matching- based methods. Recently, pixel matching methods have gained prominence due to their superior speed and results [15]. These methods, which either implicitly predict inter-frame similarities or explicitly match features between pixel pairs across frames, have been adopted by recent SVOS methods [22, 18, 3], leading to state-of-the-an performance.
Unresolved issues and evolving developments in SVOS. Constructing discriminative feature embeddings with temporal consistency is crucial for reliable correspondence. This is typically achieved by training backbone networks on large-scale video datasets such as OxUvA [39], YouTube- VOS [46], TrackingNet [46], and Kinetics-400 [8]. Most existing methods store dense feature representations of multiple previous frames and perform spatial correlation to propagate information across frames. However, this significantly increases memory consumption, while reducing the number of reference frames degrades the performance. Hence, finding a memory-efficient way to propagate the information is of paramount importance for achieving accurate and robust segmentation results.
3 Method
We start with the task definition. In the SVOS task, the user provides (only for the first frame) a binary mask for each object of interest. Let (Mo)) denote the masks where Nobj is the number of objects. Those (non-overlappingg Nobj masks imply another mask, M Nong which represents the background at time t 1 (i.e., the first frame). That mask is formed by taking the binary complement of the union of (M2 N We will henceforth treat the background as just another object unless stated otherwise. For each time / 2, T, where T is the total number of frames, the goal is to estimate the corresponding masks, denoted by (Mg) Nobj+1
Our unsupervised method for the SVOS task, depicted in Fig. can be summarized as follows. We extract features from video frames ($ 3. and model the within-object feature distribution via a multi-scale dynamic ensemble of vMF mixtures ($ 3.2). We use a streaming-data Expectation- Maximization algorithm to update the parameters of the mixtures ($ 3.3). We assign pixels to objects based on the component-we wise posterior probabilities in the entire ensemble of each object (S 3.4). For robustness, we disregard outliers during the parameter- eestimttt-sstimation steps, and further
3

---

improve the results by incorporating Pixel-Adaptive Convolutional CRF ($ 3.5) for additional outlier management and segmentation refinement. For examples of qualitative results see, e.g., Fig.
3.1 Feature extraction
The feature representation we use consists of ViT features [1] and positional embeddings [40]. For the ViT features, we use the publicly-avilable pre-trained weights of XCIT Pross-Covariance Image Transformer) [1]. that had been trained in a self-supervised manner as outlined in [7].
( denote the VIT features extracted from I (the video frame at time E), where N is the number of features. For example, if the VIT features are extracted using non-overlapping 8 x patches, then N = Npixels/ /64 The XCIT variant, like its DINO-ViT counterpart, excels in extracting rich and discriminative features from images [2], but with an added advantage of better handling high-resolution images. In our case, we employ an XCIT model that yields feature vector Øt E I8884 for each image patch i. To capture the spatial relationships between neighboring pixels, we extend each VIT feature, z with its corresponding rotary positional embedding, p(t). as per the methodology detailed in Su et al. [38]. This embedding represents the 2D spatial location of the ViT feature in 64-dimensional space. Consequently, we form the extended feature vector xf (az', Wp p(i)) (where wp is the weight of the positional embeddings), normalized such that || 1. We denote by X = (==)) the collection of all these unit-length (extended) features time L. Finally, given the initial masks, (M Nobj+1 we partition X the features at t = 1. into , Nast1 such that, for each o Nob. + 1). X represents the features that correspond to M
Let Npixels be the number of pixels in each frame. Let X =
3.2 Modeling the within-objet feature distribution via a multi-scale vMF mixture
In our pursuit of capturing intricate patterns in high- nigh-dimensional data while ensuring computational efficiency, we turn to vMF mixtures; namely, mixture model whose each component is vMF distribution (defined below). These mixtures, known for their simple structure that depends solely on mean directions and concentration parameters, are highly efficient computationally. They are also adept at modeling data on a unit hypersphere, which is a prevalent form of representation for deep features. Since often the feature distribution of an object is not only complex but also dependent on its size (e.g., when the object-to-camest distance increases, the object resolution decreases and fine details tend to disappear, which in term affects the distribution), we advocate for multi-scale approach, using a mixture ensemble. For each object, we employ S independent vMF mixture models, each corresponding a different scale granularity. This way, capture details multiple scales.
The vMF distribution is probability distribution over gd-1 {ær x E Rd ||x= 1}. the unit sphere in Rd. Its probability density function (pdf), evaluated gd-1,
MMF ,a, 7) Ca(T) exp(Tu
(1)
where f E Sd-1 is the mean and イ >> 0 is the concentration parameter. The normalizer Ca(r) depends only on r and has a closed form that appears Appendix A.
Let E (1....,S}, Mixture s for object at time t is defined as follows:
K, A. Hb.e.k To.
(2)
Here, a represents the mixing proportions while v and s. are the mean and concentration parameter, respectively, of component time t. The predetined number of components, K,, is different for every s and represents the scale of the mixture. The full set of parameters describing object o time is 0 ((at)
K, where 6e &,,.k = (H' T k).
3.3 Parameter estimation via Expectation- Maximization
We adapt the widely-used Expectation MititationnMaxiizzzttinn (EM) framework [12] to estimate the parameters of our vMF mixtures a streaming-dalt manner, tailoring specifically for VOS. The algorithm applied independently and in parallel to each mixture.

---

Our EM algorithm consists of two stages. The Expectation step (E-step) calculates the posterior probabilities of component membership (i.e., caaturr-to-comoonee assignment), integrating spatial coherence to improve segmentation accuracy. In the Maximization step (M-step), we update the mixture parameters to maximize the expected log-likelihood, introducing modifications to handle the streaming nature of video data and manage memory efficiently.
3.3.1 E-step: calculating posterior probability with spatial coherence
In the E-step we incorporate spatial coherence, crucial factor in VOS. Let assume that, time t, pixel was assigned to object (we will later explain how pixels are assigned to objects). For each mixture s of the S mixtures associated with object o, we do the following. For every k = L,.... K.. let La,, be the posterior probability that, at time t, pixel i belongs to component &o (of mixture s) given the observed feature ar and the current estimate of (, H k).
K. Let N? be the number of
nonzero pixels in Mr and, in slight abuse of notation, let (z:)), denote the object features at time We modify the calculation of the posterior probability, e by integrating a spatial constraint. indicator function that respects both the current parameter estimates
This constraint is expressed and the spatial information inherited from the previous frame, t 1. This approach ensures smooth transition between frames and improves the spatial integrity of objects in the video sequence. Taken together, the modified posterior probability is
Lo,s,k,i P(zo.s. = k|22,, 0 00,,,)) I( (o,.,k)
(3)
,,,,,) x where 2o is the assignment of æ a component mixture in object o, p(zo,a.i = k| ; of p(ar) 00 (k..,) = (a/; p To,,,)), and the binary- indicator function,
p p(P T
I( (o,s,)
(0.
(4)
otherwise
returns 1 if x has positional embedding that closely matches p( the positional part of the mean of cluster & from the previous frame. The quality of the match is measured terms of cosine similarity. Note that the assignment also depends on a certain threshold, T.. which in turn depends on the number of clusters, Kg. As K2 increases, a higher degree of similarity is required, reflecting the finer granularity of the mixture. If no such pixel exists, the function returns O, effectively preventing the current pixel from being assigned cluster k. This approach ensures spatial coherence.
3.3.2 M-step: parameter estimation
We first provide the details for the M-step while ignoring the streaming nature the data, and then later, in 3.3.3, explain how account for the streaming. In the M-step, the parameters updated to maximize the expected log-likelihood. We iteratively update set of sufficient statistics that encapsulate the critical properties of the data and the model. Concretely, time we compute
K,
H Ho,.k k = (h o. k=
1.2
K.
Σ i=1 Lo.o.b.ds Σ i= di So
(5)
k=1
The M-step uses well-known closed-form solutions for the estimated parameters the vMF distribu- tion. In particular, to estimate the concentration parameters we employ an approximation scheme based on the continued fraction method, supplemented with a correction term:
1 |16.2.11
||,.22211 1.1
To d. R.Ra (R.
R R
0,
1- -RR
Σk
h
(6)
To circumvent the computational expense of directly calculating the modified Bessel function (which appears in Ca(-)), we adopt an alternative method compute the log- likelliod log p( (0, (
using the mean vector & and the concentration parameter T. Specifically, we leverage the fact that the normalizing constant Ca(Tá,,, depends solely on T. and can be approximated [36] by a simpler expression (see Appendix A). This enables us to express the log-likelihood as follows:
log p( k. = -o (H. E + Nº log Ca(To,s.k)
(7)
5

---

100 21 Reference framnes
Max 1965
Relerence Tremnss
3 Reterence Trame
Memory Consumption (GB)
Ours
200
62
Hax
28
Moo 13
300 150
1250
1500
Feature Map Size (H-W)
Figure 3: Memory utilization. This graph illus- trates the memory consumption as a function of the feature map size, comparing method with scenarios using different numbers of reference frames (1, 5, and 21). Notably, our method re- quires less memory than even the scenario with single reference frame. The feature dimension for this evaluation is set to 384.
Figure 2:
Qualitative examples.
where T denotes transpose and æ denotes the (Fuclidean) sample mean of the data points. We now address the dynamic characteristics of video data.
3.3.3 Memory--fficient dynamic updates and temporal coherence
In the dynamic environment of video sequences, objects may experience occlusion, reemergence, or gradual appearance changes due to, e-g., varying lighting conditions. Addressing these challenges, and inspired by [13], our method integrates temporal coherence into the clustering procedure using time-weighted sufficient statistics. We devise these statistics a weighted sum of data values across multiple frames, with weights diminishing exponentially over time. By maintaining historical records of a limited length, and updating the time-weighted statisties for each frame, we efficiently manage space, memory, and time complexity. Data from the preceding frame is retained only if its (time-decaying) weight exceeds a certain threshold. Concretely, we employ a time-decaying weight (C(t, t') 2 where the user-defined À > 0 controls the correlation between sequential frames. As result, older frames are assigned lower weights. Similarly to [13], we modify the sufficient statistics accordingly, and obtain a streaming-dae variant of Eq. 5:
K.
N do
N
H = -hhi 422 Ke
A
Σ 1=1
K(L
EC(O 0==1
(8)
k=1
Our method's efficient information propagation from prior frames results in smoother transitions and improves the temporal coherence of the segmentation. Retaining (weighted) sufficient statistics from previous frames helps with the detection of reappearing, previously-occluded. objects parts. Importantly, our memory consumption is determined by the number of clusters the model, not N, the (much larger) feature map size. This drastically reduces memory usage during inference (see Fig. 3) and enables us to obtain a compact representation. Moreover, this drastic memory reduction (in comparison to methods which rely saving history of dense representations) enables to higher-resolution features than the competitors (see, e.g., Fig. 4). Importantly, as IC(t, L') decreases when It f'l increases and eventually decays zero, practice store the sufficient statistics from time t' only until time + 15 and then discard it.
3.4 Pixel-to-object assignment
Given the estimated models from time I, in the subsequent frame we extract features X and assign labels to each pixel based on the mixture (from time that has the highest probability for any of its components. By employing the maximum-a-posteee postterioii (MAP) rule, we determine the component

---

Figure 4: Benefits of high-resse riseuueesoltiin. Each row presents an original video frame (leff) and our results obtained when using features either low resolution (middle) or high resolution (right). Note the finer details achieved high resolution. While it fairly obvious that higher resolution hetter, our point here is that competing methods (e.g., [3, 24D usually cannot handle such resolution due to the increased computational and memory demands. This is unlike our method which has a low-memory footprint. Please zoom in for optimal viewing.
with the highest probability for each pixel, and then assign the pixel to the object associated with that component. In other words, for every E {1, f Nobs 1} compute
Pio A Pr(pixel i time belongs object o) x max s:sE s:sEl.....5
max I..El..... K,
p(a) fec 10....)))
(9)
and then assign the pixel to object according to arg max (I..., Ps.0 the (non-overlapping) masks for time i + 1, 1-f--1y1 (M Jos1
4+1 Next, we generate
Nobi+1
via
1, if arg max, Pioo
M (i)
(10)
0, otherwise
where M5+1(1
denotes pixel of M
3.5 Refining the segmentation using outlier rejection and a convolutional CRF
Recall that the initial mask is the sole ground truth reference that the method has access to. Thus, as the video sequence unfolds, the presence of outliers poses substantial challenge to the accuracy of the clustering process. These outliers, which stem from factors such as occlusions, illumination changes, noise, might hurt the integrity of existing clusters or lead to confusing them with unrelated clusters. Our approach for mitigating such potential sources of eгтor incorporates two strategies: outlier rejection and refinement using convolutional CRF.
In the initial phase, we exclude points from the parameter-estimation step when the pixel-to-object assignment is inconclusive. Specifically, if the score of pixel's second-ber best match closely rivals that of its best match, we consider that pixel an outlier. We then employ the Pixel-Adaptive Convolutional CRF (PAC-CRF) [37] to further refine the segmentation. PAC-CRF, with its fixed window connections around each pixel, effectively filters noise while preserving segmentation coherence. bolsters our outlier handling strategy by enabling the exclusion of points where the PAC-CRF prediction conflicts with our initial prediction. After several EM steps (where the M step excludes the outliers), we apply PAC-CRF again, leveraging the source image to improve the pixel-wise accuracy and the segmentation quality. Furthermore, we leverage the learnability of the PAC-CRF's pairwise potentials' weights and optimize them using Focal-Loss [26, 30]. This optimization, carried out over a few gradient steps with the ground-truth annotation from the first frame, ensures that our method adapts well to the given data, leading to improved performance and more accurate segmentation maps (Fig. 2).
4 Results
In this section, we present the performance the proposed method on the DAVIS-2017 and YouTube- VOS datasets. A subsequent subsection delves into adeptness at harnessing features from deeper
7

---

INO [3]]
Ours
Figure 5: Qualitative examples DAVIS-2017, comparing our results (third row) with of the key competitors (second row).
Table I: Results on DAVIS-2017 validation in terms of the mask (T) and boundary (F) accuracy (IoU). The subscript [-]r denotes the recall of the metric, while [:]m signifies the mean.
Method
Training Video Dataset (Duration)
J&F
Im
J.
F
F
Unsupervised
Colorization [42]
Kinetics (800 hours)
34.0
34.6
34.1
32.7
26.8
CorrFlow [23]
OxUvA (14 hours)
50.3
48.4
53.2
52.2
56.0
TimeCycle[44]
VLOG (344 hours)
48.7
46.4
50.0
50.0
48.0
UVC[25]
Kinetics (800 hours)
60.9
59.3
68.8
62.7
70.9
MuG[27]
OxUvA (14 hours)
54.3
52.6
57.4
56.1
58.1
MAST[22]
Youtube- VOS (5.58 hours)
65.5
63.3
73.2
67.6
77.7
CRW[18]
Kinetics (800 hours)
68.3
65.5
78.6
71.0
82.9
ContrastComr[43]
TrackingNet (140 hours)
63.0
60.5
65.5
VFS[45]
Kinetics (800 hours)
66.7
64.0
69.4
JSTG[49]
Kinetics (800 hours)
68.7
65.8
77.7
71.6
84.3
DULVS[3]
TrackingNet (140 hours)
69.4
67.1
80.9
71.7
84.8
CLTC
Youtube- VOS (5.58 hours)
70.3
67.9
78.2
72.6
83.7
DINO [7]
No training video is required Youtube- VOS (5.58 hours)
71.4
67.9
80.7
74.9
87.8
LIR [24]
72.1
69.7
81.4
74.5
85.9
INO [33]
Kinetics (833 hours) No training video required
72.5
68.7
82.0
76.3
89.0
Ours
76.3
73.8
85.6
78.7
89.2
Supervised
OSVOS [6]
DAVIS-2016 (0.5 hours)
60.3
56.6
63.8
63.9
73.8
TVOS[48]
DAVIS-2017 Youtube- VOS (6 hours)
72.6
69.9
74.7
STM[31]
DAVIS-2011 + Youtube-VOS (6 hours)
81.7
79.2
84.3
networks and superior resolutions. Video demonstrations can be found here. Details regarding implementation are available in Appendix B.
4.1 Evaluation on DAVIS-2017 and YouTube- VOS
We report the results on two widely-used SVOS benchmarks: YouTube- VOS [46] and DAVIS 2017 [34]. Commonly used metrics for assessing VOS performance include the Jaccard's index
(J) [14], which calculates the interrectionn-vve-union (IoU) of the object mask, and the F-measure (F) [29], which assesses the contour accuracy. These metrics provide comprehensive insight into the segmentation performance by evaluating both the object area and contour precision. We report the mean IoU for mask and contour, denoted as Im and Fm. as well as the recall J, and F,, which are

---

Table DAVIS-2017 (train)
Table 3: DAVIS-2017 (test-dev)
Method
I&Fm
Jm
Jr
F
Fr
Method
J&F m
Im
Jr
Fm
F,
DULVS
68.0
64.4
75.4
71.7
81.3
DULVS
58.3
54.5
64.7
62.0
70.1
LIIR
63.9
60.3
69.7
67.5
75.4
LIIR
50.3
47.6
54.7
52.9
60.2
INO
67.7
62.4
71.7
73.0
84.1
INO
61.9
56.5
63.8
67.3
77.6
Ours
73.9
80.0
77.9
77.9
88.1
Ours
66.5
63.3
74.5
69.6
80.1
calculated using IoU threshold of 0.5. The metric J&Fm m represents the average of Im and F..- For fair comparison, we use the same video resolution as the competing methods, despite the fact that our method can process higher resolutions than them.
The DAVIS-2017 dataset [34], featuring high-quality videos with diverse object appearances and challenging motion patterns, is widely-accepe SVOS benchmark. On the DAVIS-2017 validation set (Table 1), our method demonstrated top-tier scores the Jaccard index, F-measure, and J&FF,. m metries. Specifically, our results include J&F score of 76.3, Im of 73.8, J, of 85.6, Fm.. of 78.7, 7, and Fr of 89.2. These results clearly outperform the other methods (see also Fig. 5). We also evaluated our method on the AAVIS-2017 training and test-dev sets, collectively constituting 90 additional sequences. These sets allowed for a broader performance assessment, with the training set offering an extra dimension of evaluation since it
DAVIS 2017.
included in the training data of the methods we compared with. Results from the test-dev set, officially validated by the evaluation server, further demonstrate our method's efficacy (Table Table 3). Qualitative results from these extended evaluations are available in Appendix C.
You'l Tobe-VOS. The You Tube- VOS dataset [46] serves large-scale benchmark for SVOS, intro- ducing a variety of complexities such as occlusions, scale changes, and motion blur. Unique to this dataset is the division of object classes between the training and validation sets, where only subset is shared, allowing the benchmark to distinguish between 'seen' and 'unseen' categories. Our method's scores, obtained through the official evaluation server, showcase its adaptability in navigating these complex situations, surpassing the performance of several state-of-ther thett--rtttttttt approaches (Table 6).
4.2 Performance and feature analysis
Our method adeptly processes high-resolution features, distinguishing it from other models con- strained by memory, as illustrated in Table 4. While merely adopting deeper backbones does not guarantee superior outcomes [45], our model showcases its capability to harness the strengths of DINOv2's [32] features. To counteract the reduced spatial information inherent in DINOv2 due to its larger patch size, We concatenated the DINOv2 features with the XCIT features had pre- viously used. This approach ensures a rich and comprehensive representation of the visual scene. As demonstrated in Table 5, our method effectively capitalizes on the enhanced features from both deeper backbones and higher spatial resolutions.
Table 4: FPS across resolutions. Comparison Tesla V100- excluding feature extrac- tion. OOM stands for "Out Of Memory".
Table DAVIS-2017 (val) with DINOv2 fea- tures. Fusion with our features elevates perfor- mance. "x2" denotes using features with twice the spatial resolution. DINOv2 "S" and "B" stand for "small" and "base", respectively.
Resolution Feat. Size
Ours
[33]
[3]
384
8.25
0.13
0.14
120 X 210
768
6.50
0.11
0.12
Method
J&FF m
Jm
F
1 1152
5.25
0.10
0.11
+ DINO2-S
0.776
0.754
0.798
384
1.55
OOM OOM
+ DINO2-B
0.782
0.760
0.804
240 x 420
768
1.28
OOM OOM
x2 + DINO2-S
0.784
0.762
0.806
1152
1.10
OOM OOM
x2 + DINO2-B
0.803
0.780
0.825
4.3 Ablation study
We performed an ablation study (Fig. 6) to analyze the influence of different parts of the method on the performance. The combination of outlier detection and segmentation refinement showed

---

Table 6: Results on You'Tube-VOS 2018 (val)
we outleers removal Varying
Varying S
Varying 10p
Seen
Unseen
Mcan
Method
Im
F
Im
F m
T4.1
A
Wp
Unsupervised
73.0
=
2.0
Colorize[42]
43.1
38.6
36.6
37.4
38.9
75.0
10
73.9
ConFlow [23]
50.6
46.6
43.8
45.6
46.6
73.2
73.8
MAST[22]
63.9
64.9
60.3
67.7
64.2
D3LVS
70.2
71.9
66.5
74.8
70.6
CLTC[19]
66.2
67.9
63.2
71.7
67.3
LUR [24]
67.9
69.7
65.7
73.8
69.3
74.1
30
[NO[33]
70.7
73.2
65.6
75.6
71.3
S
74,7
Ours
72.0
74.2
66.2
73.7
71.5
9
Supervised
74.2
OSVOS
59.8
60.5
54.2
60.7
58.8
Figure 6: Ablation study of model parame- ters. We report the J J&Fm m score evaluated on DAVIS 2017 val. Our baseline configura- tion (the centered one) is: 10, 0.33, wpp = 15.
TVOS
67.1
69.4
56.5
63.0
71.6
STM[31]
79.7
84.2
72.8
80.9
79.4
synergistic improvement in performance, with each contributing significantly when implemented separately. The necessity of streaming mechanism underscored when observed notable performance drop while solely relying on the previous frame for initializing mixture parameters (A x.). Furthermore, examining the number of mixtures per object, notice substantial performance boost when increasing the number of mixtures from one to two, underscoring the importance of multi-scale object representations. However, the added benefit started to decay with further increase, leading us to settle on S = 10 mixtures in our reported results. Concerning spatial components, removing positional embeddings (i.e., wp = 0) resulted in a major performance drop. confirming their essential role. Additionally, the introduction of the indicator function led to an additional positive effect on performance.
5 Conclusion
We proposed novel SVOS approach that achieves state-of-the-art results on challenging bench- marks. Our memory-conscious strategy also opens possibilities for processing high-resolution videos, suggesting valuable avenue for future video understanding research.
5.1 Limitations
The proposed method faces difficulties in scenarios (see Appendix D for examples). The first is related to re-identification challenges, where occlusion between multiple similar instances of the same object type (e.g., two camels) makes challenging assign correct labels each instance. This limitation stems from the method's lack of explicit object identity representation and modeling of temporal dynamics, which could help disambiguate overlapping objects. The second scenario occurs in the presence of strong motion blur. The method is sensitive to such a blur, likely because the ViT features were trained still images, not videos. This limitation becomes more pronounced when motion blur persists across consecutive frames, as then the eттог propagation is more significant.
Acknowledgments and disclosure of funding
This work was supported by: the BGU CS Lynn and William Frankel Center; the Israeli Council for Higher Education via the BGU Data Science Research Center; Israel Science Foundation Personal Grant #360/21.
10

---

References
[1] Alaacldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Mathhij Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, al. Xcit: Cross-covariance image trans- formers. NeurIPS, 2021.
[2] Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features dense visual descriptors. In ECCV Workshop on What is Motion For?, 2022.
[3] Nikita Araslanov, Simone Schaub-Meyer, and Stefan Roth. Dense unsupervised learning for video segmentation. NeurlPS, 2021. 3, 7, 8, 9, 10
Goutam Bhat, Felix Järemo Lawin, Martin Danelljan, Andreas Robinson, Michael Felsberg, Lue Van Gool, and Radu Timofte. Learning what to learn for video object segmentation. In ECCV, 2020. 3
[5] Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In ECCV. Springer, 2010. 3
[6] Sergi Caclles, Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Lue Van Gool. One-shot video object segmentation. CVPR, 2017. 10
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, ICCV, 2021. 4, 8
[8] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Shao-Yr Yi Chien, Shyh-Y7 Yih Ma, and Liang-Gee Chen. Efficient moving object segmentation algorithm using background registration technique. IEEE Transactions on Circuits and Syxtems for Video Technology. 2002.
[10] Prakash Chockalingam, Nalin Pradeep, and Stan Birchfield. Adaptive fragments- based tracking of non-rigid objects using level sets. In ICCV. IEEE, 2009.
[11] Rita Cucchiara, Costantino Grana, Massimo Piccardi, and Andrea Prati. Detecting moving objects, ghosts, and shadows video streams. JEEE TPAMI, 2003. 3
[12] Arthur P Dempster, Nan Laird, and Donald Rubin. Maximum likelihood from incomplete data the EM algorithm. Journal of the royal statistical society: series B (methodological), 1977. 4
[13] Dinari and Oren Freifeld. Sampling dirichlet process mixture models clustering streaming data.
In AISTATS. PMLR, 2022. 2, 6
[14] Mark Everingham. Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV, 2009.
[15] Mingqi Gao, Feng Zheng, James JQ Yu. Caifeng Shan, Guiguang Ding, and Jungong Han. Deep learning for video object segmentation: a review. Artificial Intelligence Review, 2023. 3
[16] Brent Griffin and Jason J Corso. Bubblenets: Learning select the guidance frame in video object segmentation by deep sorting frames. CVPR, 2019.
[17] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. ICCV, 2017.
[18] Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence contrastive random walk. In NewrlPS, 2020. 3, 3
[19] Sangryul Jeon, Dongbo Min, Scungryong Kim, and Kwanghoon Sohn. Mining better samples for contrastive learning of temporal correspondence. CVPR, 2021. 8.
[20] Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg. generative appearance model for end-to-end video object segmentation. In CVPR, 2019. 3
[21] Changick Kim and -eng Hwang. Fast and automatic video object segmentation and tracking for content-based applications. IEEE transactions on circuits and systems for video technology. 2002.
[22] Zihang Lai, Erika Lu, and Weidi Xic. Mast: A memory augmenee self- tracker. In CVPR, 2020.
3, 8, 10
[23] Zihang Lai and Weidi Xie. supervued learning for video correspondence flow. In BMVC, 2019. 8, 10

---

[24] Liulei Li, Tianfei Zhou, Wenguan Wang, Lu Yang, Jianwu Li, and Yi Yang. Locality- awarr inter-and intra-video reconstruction for self-supervised correspondence learning. In CVPR. 2022. 7. 8, 10
[25] Xueting Li, Sifei Liu, Shalini De Mello. Xiaolong Wang. Jan Kautz, and Ming-Hsuan Yang. Joint-task self-supervised learning for temporal correspondence. NeurlPS, 2019. 8
[26] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017. 7
[27] Xiankai Lu, Wenguan Wang. Jianbing Shen, Yu- Wiig Tai, David J Crandall, and Steven CH Hoi. Learning vidco object segmentation from unlabeled videos. In CVPR, 2020. 8
[28] K-K Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. Video object segmentation without temporal information. JEEF TPAMI, 2018. 3
[29] David R Martin, Charless C Fowlkes, and Jitendra Malik. Learning to detect natural image boundaries using local brightness, color, and texture cues. JEEE TPAMI, 2004.
[30] Ty Nguyen, Tolga Ozaslan, Ian D Miller, James Keller, Giuseppe Loianno, Camillo J Taylor, Daniel D Lee, Vijay Kumar, Joseph H Harwood, and Jennifer Wozencraft. U-net for mav-based penstock inspection: an investigation of focal loss in multi-class segmentation for corrosion identification. arXiv preprint arXiv: 1809.06376.6 2018. 7
[31] Scoung Wug Oh, Joon- Young Lee. Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV, 2019. 8, 10
[32] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719. 2023. 9
[33] Xiao Pan, Peike Li, Zongxin Yang, Huiling Zhou, Chang Zhou, Hongxia Yang, Jingren Zhou, and Yi Yang. In-n-out generative learning for dense unsupervised video segmentation. In Proceedings of the 30th ACM International Conference on Multimedia, 2022. 8, 9, 10
(34] Jordi Pont-Tusct, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Dorkine-Homunng and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: (tl::/7000006555
2017. 8,9 9
[35] Andreas Robinson, Felix Jaremo Lawin, Martin Danelljan. Fahad Shahbaz Khan, and Michael Felsberg. Learning fast and robust target models for video object segmentation. In CVPR, 2020.
[36] Tyler R Scott, Andrew C Gallagher, and Michael C Mozer. von mises-fisher loss: An exploration of embedding geometries for supervised learning. In ICCV, 2021. 5
[37] Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, and Jan Kautz. Pixel-adaptive convolutional neural networks. In CVPR, 2019. 2.7
[38] Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint /04iv:2/04.09864 2021.
[39] Jack Valmadre, Luca Bertinetto, Joao F Henriques, Ran Tao, Andrea Vedaldi, Aroold WM Smeulders, Philip HS Torr, and Efstratios Gavves. Long-term tracking in the wild: A henchmark. In ECCV, 2018. 3
[40] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NewrIPS, 2017. 4
[41] Paul Voigtlaender and Bastian Leibe. Online adaptation of convolutional neural networks for video object segmentation. arXiv preprint /tti::/706.09360 2017. 3
[42] Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In ECCV, 2018. 8, 10
[43] Ning Wang. Wengang Zhou, and Houqiang Li. Contrastive transformation for self-supervised correspon- dence learning. In Proceedings of the AAAI Conference on Artificial Intelligence, 2021. 8
[44] Wenguan Wang, Hongmei Song, Shuyang Zhao, Jianbing Shen. Sanyuan Zhao, Steven CH Hoi, and Haibin Ling. Learning unsupervised video object segmentation through visual attention. In CVPR, 2019. 8
[45] Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. In ICCV, 2021. 8,9
12

---

[46] Ning Xu. Linjie Yang. Yuchen Fan. Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-lo-sequencce video object segmentation. ECCV, 2018. 8,9 9
[47] Linjie Yang, Yanran Wang. Xuehan Xiong. Jianchao Yang. and Aggelos K. Katsaggelos. Efficient video object segmentation network modulation. CVPR, 2018.
[48] Yizhuo Zhang, Zhirong Wu, Houwen Peng, and Stephen Lin. A transductive approach for video object segmentation. In CVPR, 2020. 10
[49] Zixu Zhao, Yueming Jin, and Pheng-Ann Heng. Modelling neighbor relation in joint space-time graph for video correspondence learning. ICCV, 2021.
[50] Di Zhong and Shih-Fu Chang- An integrated approach for content-based video object segmentation and retrieval. IEEE Transactions on Circuits and Systems for Video Technology. 1999.
[51] Tianfei Zhou, Jianwu Li, Xueyi Li, and Ling Shao. Target-aware object discovery and association for unsupervised video multi-object segmentation. CVPR, 2021. 3
13


Reviews:
[{"review_id": "IUrgVtPmCa", "reviewer": "Reviewer_5uUQ", "summary": "The paper essentially shows, that a combination of classic techniques such as stream-data clustering, using an EM-algorithm and dynamic updates, in combination with strong pre-trained features, allows to achieve state-of-the-art performance on two standard video segmentation datasets (Davis 2017 and YouTube-VOS-2018). ", "questions": "The paper is well written and the performance is interesting to know about. I suppose the key question to me is essentially, how surprising the results are? If I had been asked prior to reading this paper if a sensible combination of stream-based clustering with strong pretrained features can do the job on datasets such as DAVIS and YouTube (these are not particularly difficult after all)  - I would have clearly set yes. In that sense the surprise and novel insight to me for this work is rather marginal. Having said this, I would assume that not everyone has the same intuitions and thus it is imho worthwhile to report the paper in some form as a publication. However, in my personal judgement the current paper and contribution seems below the bar for such a high-profile venue such as NeurIPS. ", "limitations": "ok for me", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Clearly achieving good performance with a combination of classic techniques, chosen well for the task at hand, is interesting to report and know about. Performance is strong on DAVIS (probably the dataset where most parameters are set) and somewhat less impressive on YouTube - but still sota", "weaknesses": "There is essentially no novelty in the paper - except to propose a well-chosen combination of classic techniques. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and useful review. \nWe also thank the reviewer for recognizing the strong performance of our method on both the DAVIS and YouTube datasets, as well as appreciating the integration of classic techniques. \n\nWe would like to underscore the unique contribution that our work makes, which goes far beyond combining and adapting established techniques. The success of our method lies in the precision with which the techniques are adapted, tailored, and validated to deliver state-of-the-art results on benchmarks that continue to be challenging. The fact that after reading our paper the reviewer feels that the approach is logical and that it makes sense that it works, does not contradict the fact that no one did it before and does not dispute the novelty. Moreover, the details are crucial and many of our judicious choices are far from being obvious (e.g., opting to go with an ensemble of mixtures instead of model selection or nonparametric clustering techniques) and were always made while taking into account not only performance and principled modeling considerations but also efficiency and scalability.   In any case, we argue that if it were so obvious that setting new state-of-the-art results on a key computer-vision task can be done this way, people would have already done it before… \n\nMoreover, we are the first to demonstrate a low memory footprint in this domain, an essential quality in real-world applications. Our ability to easily scale to higher resolution features, while relying solely on static images during training, represents a breakthrough in unsupervised video object segmentation. This innovation not only builds on classic techniques but extends them in a way that could redefine best practices in the field."}, {"review_id": "2zcEiZBcbc", "reviewer": "Reviewer_RHPb", "summary": "The method tries to combine classical technique for SVOS task namely vMF and CRFs with advance Deep Learning based ViT representations. By doing so, the proposed method eliminates the training requirements on video data. The authors perform comprehensive experimentation to evaluate their approach.", "questions": "- As claimed by authors, their method has a low-memory footprint. How does it compare to the baseline in regard to the speed of performing the VOS over a video? I want to know if there is a trade-off between the memory footprint and performance speed.", "limitations": "As the authors are using pre-trained VITs to extract the features. All the limitations of VITs carry forward to this approach.", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The paper is well written and easy to follow. Authors have done a thorough job in providing comprehensive experimental evaluation for their technique. The major strength of this approach is it does not require video training data.", "weaknesses": "- The performance of the model decreases when it encounters the unseen examples as can be observed from the table 2. If the proposed model does not require training then why would the performance on unseen examples is lower?\n-  I'm not sure I understand what Fig. 3 represents? Is it a comparison of memory utilization by proposed method and baselines that require 1, 5, 21 reference frames? \n- The claim of low-memory footprint is unclear to me? is the claim made with respect to other baselines? If so, by how much delta is the proposed method better?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful, useful, and overall-positive review. \n\n-  **_\"The performance of the model decreases when it encounters unseen examples\"_**.\nThe decrease in performance on unseen examples, as noted in Table 2, does not stem from a need for training, but rather the complexity of these examples. The unseen categories likely comprise more challenging cases. This observation aligns with other unsupervised models not trained on the YouTube-VOS 2018 dataset, where a similar drop in performance occurs. This pattern emphasizes that the challenge lies in the unique difficulty of the unseen examples, rather than a specific limitation of our model.\n\n- **Clarification on Figure 3 and the memory footprint**.\nFigure 3 compares memory consumption for the retention of history information between our proposed method and recent works in a similar setting (unsupervised, correspondence based matching), which often need up to 21 reference frames.\nThe cumulative memory utilization during inference is affected by:\n    1.  **Memory Storage**: Traditional methods store multiple dense feature maps from prior frames, increasing the memory footprint based on the quantity and size of these maps. Our method significantly reduces this by tying memory usage to the number of clusters, which demands far less memory than even a single dense feature map.\n    2. **Computational Load**: Standard label propagation techniques compute cosine similarity for each feature point in the current frame with all feature points within a certain radius in all reference frames. That approach struggles with scalability, particularly when the feature map resolution increases. The higher the resolution, the larger the radius needed for cosine similarity computation, intensifying the computational challenge. In stark contrast, our method improves on this by computing similarities relative to clusters, not individual feature points, thus enhancing scalability.\n- **_\"Trade-off between the memory footprint and performance speed:\"_**\n\n  **Performance Comparison in Frames Per Second (FPS):**\n\n  | Resolution       | Feature Dimension | Ours  | INO  | DULVS |\n  |------------------|-------------------|-------|------|-------|\n  | 1/4 (120 x 210)  | 384               | 8.25  | 0.13 | 0.14  |\n  |                  | 768               | 6.50  | 0.11 | 0.12  |\n  |                  | 1152              | 5.25  | 0.10 | 0.11  |\n  | 1/2 (240 x 420)  | 384               | 1.55  | OOM  | OOM   |\n  |                  | 768               | 1.28  | OOM  | OOM   |\n  |                  | 1152              | 1.10  | OOM  | OOM   |\n\n  _OOM: Out of Memory_\n\n  We evaluated three models using feature maps at resolutions of 1/4 and 1/2 of the image size (480 x 840) across three feature dimensions (384, 768, 1152). This comparison involved computing the frames per second (FPS) after the first 21 frames, exclusive of the feature extraction step. Our model consistently outperformed the others, exhibiting particular strength in higher resolutions where competing models faced out-of-memory (OOM) issues. \n  (Tested using Tesla V100-32GB)\n\n"}, {"review_id": "MHTmAjGCLp", "reviewer": "Reviewer_sD4x", "summary": "This paper proposes a training free framework for semi-supervised VOS. Here, the features of objects from a pretrained ViT are represented as vMF, where the objects across frames are associated to perform propagation from initial masks. The results are better than previous self-supervised methods that trained on unlabeled videos.", "questions": "-More details should be illustrated in Table 1 and Table 2, e.g., backbone, resolutions. The proposed method uses ViT while the others always use res18 or res50. It is necessary to demonstrate that the superior results come from the well designed algorithm instead of the usage of more powerful backbone. \n", "limitations": "The authors discussed the limitations of their methods.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The idea of associating objects in vMF space for SVOS is interesting, which is a new insight different from the previous contrastive learning based self-supervised methods. The paper is well organized and sufficient comparison visual results are given. ", "weaknesses": "-The claimed limitations in SVOS is not true especially for the large memory footprint. Actually, some lightweight designs have been proven in VOS like mobilevos[1*] and AOT[2*]. Here, [1*] focuses a lightweight for real-time vos and [2*] tackles the memory storage issue in STM and the repeated inference for each object id at testing. It's true that the heuristic association method gets rid of the usage of complicated decoders where temporal correspondence and mask generation are performed. However, the large memory is not only from the network parameters but the computational costs. For now, the authors did't give convincing results to demonstrate this contribution like comparisons in terms of params, gflops or fps.\n-For the temporal coherent, is it possible to extend to model both inter-object contrast and intra-object consistency, so the method can directly predict multiple instances at once.\n--Although the method is training free, many hyperparameters need to be tuned during testing. Figure 5 also demonstrates that the change of hyperparameters would impact the final performance. \n\n[1*] MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation, cvpr23.\n[2*] Associating Objects with Transformers for Video Object Segmentation, neurips 21.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful, useful, and overall-positive review. \n\n- **Large memory footprint**. MobileVOS and AOT are supervised methods that indeed accomplish admirable memory management and computational efficiency. However, these achievements are grounded in the utilization of training labels, which streamline label propagation, subsequently reducing computational and memory demands. In contrast, our approach operates within the unsupervised realm, where label propagation is inherently more complex and memory-intensive.\nIt's also noteworthy that, to our knowledge, no existing unsupervised methods have effectively navigated the memory-intensive challenges associated with label propagation. Our method bridges this gap, providing an innovative solution in the unsupervised paradigm.\n\n  **Performance Comparison in Frames Per Second (FPS):**\n\n  | Resolution       | Feature Dimension | Ours  | INO  | DULVS |\n  |------------------|-------------------|-------|------|-------|\n  | 1/4 (120 x 210)  | 384               | 8.25  | 0.13 | 0.14  |\n  |                  | 768               | 6.50  | 0.11 | 0.12  |\n  |                  | 1152              | 5.25  | 0.10 | 0.11  |\n  | 1/2 (240 x 420)  | 384               | 1.55  | OOM  | OOM   |\n  |                  | 768               | 1.28  | OOM  | OOM   |\n  |                  | 1152              | 1.10  | OOM  | OOM   |\n\n  _OOM: Out of Memory_\n\n  We evaluated three models using feature maps at resolutions of 1/4 and 1/2 of the image size (480 x 840) across three feature dimensions (384, 768, 1152). This comparison involved computing the frames per second (FPS) after the first 21 frames, exclusive of the feature extraction step. Our model consistently outperformed the others, exhibiting particular strength in higher resolutions where competing models faced out-of-memory (OOM) issues. \n  (Tested using Tesla V100-32GB)\n- **Increasing the inter-object variability and increasing the intra-object similarity**. Please note that our method does not make any assumption about the object class. Thus, it is not designed for instance segmentation per se. That said, and as demonstrated in many of our videos (please also visit the url provided with the paper which shows many such videos), if the user provides multiple masks in the first frame, then our method successfully segments multiple objects, including different instances of the same class (e.g. see the video with several dogs). \n- **Hyperparameters** exist in most machine-learning models. The number of hyperparameters in our method is not higher than, e.g., in a typical pure deep-learning method. In comparison to traditional training-based models, tuning hyperparameters in our approach is in fact decidedly more straightforward. Of note, we used the same values of hyperparamters for both Youtube-VOS and Davis, even though these datasets differ from each other in multiple ways. \n- **Backbones and Resolutions:**\n\n  | Model  | Backbone   | Resolution          |\n  |--------|------------|---------------------|\n  | DINO   | ViT B/8    | ⅛ of image resolution |\n  | LIIR   | Resnet-18  | ¼ of image resolution |\n  | VFS    | Resnet-50  | ⅛ of image resolution |\n  | DULVS  | Resnet-18  | ⅛ of image resolution |\n  | INO    | ViT B/8    | ⅛ of image resolution |\n\n  In the revised version of our paper, we will ensure these details are included.\n- **Adopting Deeper Networks:**\nIt is important to note that simply adopting deeper networks doesn't necessarily guarantee superior results. As observed in previous self-supervised learning approaches like VFS [1], performance sometimes remains unchanged or even degrades when using deeper backbones like ResNet-50. It can even introduce additional complications which already exist during training, such as issues with memory and convergence. Thus, our method provides an effective and much-needed solution to a practical problem.\nTo further dispel any concerns about the effectiveness of our algorithm, we direct  attention to our comparison with INO, which employs ViT B/8. Despite using the same powerful backbone, our method outperforms INO, thereby demonstrating that our superior results are indeed a product of our well-designed algorithm, and not merely the result of utilizing a more powerful backbone.\n\n- [1] Xu, Jiarui, and Xiaolong Wang. \"Rethinking self-supervised correspondence learning: A video frame-level similarity perspective.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n \n\n\n"}, {"review_id": "pGgUwpCl0M", "reviewer": "Reviewer_He22", "summary": "This paper tackles the semi-supervised video object segmentation problem. It presents a method that relies on clustering features from a pre-trained ViT model. The presented method has a low memory footprint and does not need any additional training. It shows SOTA results on DAVIS-2017 and YouTube-VOS 2018 validation datasets.", "questions": "Most of my questions are mentioned above. \nAnother question that I will like to know the answer to is about the time consideration - how much time does it take, as opposed to other methods for SVOS, and how does it scale with the spatial size of the features? ", "limitations": "The limitations of the paper are discussed and strongly relate to the choice of the pre-trained model and its corresponding features, and might be addressed by using a model pre-trained on videos.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presented algorithm is the first to show a low memory footprint and indeed can scale easily. \n\nThe paper presents a comprehensive study of the different components of the algorithm and an interesting ablation that shows the necessity of the different components. \n\nThe results of the paper are both visually and numerically impressive. ", "weaknesses": "While the algorithm in the paper results in impressive results, it is hard to follow all the notations and the optimization objectives, mostly due to the nested indexing. While I was able to follow it eventually, I suggest rewriting it (one possibility is to start with a simple 2 classes foreground and background case -  and extend it later to N classes).\n\nOne element that is not ablated in this paper, is the quality of the pre-trained model and the features that are used. To show that this algorithm can improve and produce better results in the future, one should show how the performance of SVOS improves when \"better\" features are given to it. e.g. does a ViT model that was pre-trained on more data result in better downstream performance? Does a larger model improve the SVOS? Does the algorithm improve with higher resolution features? All of these questions are important for understanding if this algorithm will survive the \"test of time\". ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful, useful, and positive review. \n\n**Adopting Deeper Networks:**\nIt is important to note that simply adopting deeper networks doesn't necessarily guarantee superior results. As observed in previous self-supervised learning approaches like VFS [1], performance sometimes remains unchanged or even degrades when using deeper backbones like ResNet-50. It can even introduce additional complications which already exist during training, such as issues with memory and convergence. Thus, our method provides an effective and much-needed solution to a practical problem.  \n\nThe table below represents an evaluation conducted on the DAVIS-2017 validation set.\n\n| Features                | J&F-Mean | J-Mean  | J-Recall | F-Mean  | F-Recall |\n|-------------------------|----------|---------|----------|---------|----------|\n| ours+dinov2_small       | 0.776    | 0.754    | 0.854    | 0.798    | 0.890    |\n| ours+dinov2_base        | 0.782    | 0.760    | 0.880    | 0.804    | 0.900    |\n| oursx2+dinov2_small     | 0.784    | 0.762    | 0.873    | 0.806    | 0.905    |\n| oursx2+dinov2_base      | 0.803    | 0.780    | 0.885    | 0.825    | 0.916    |\n\nIn the experiment, DINOv2's [2] features were concatenated with our original features, a necessity due to DINOv2's large patch size of 14. This fusion provided more effective information utilization. The results showed a marked improvement in J&F-Mean with the \"oursx2\" configurations (features with higher spatial dimensions). This improvement is attributed to our method's ability to exploit and use higher resolutions, facilitated by XCiT's capability to handle them. A comparison between the DINOv2 versions (\"small\" vs \"base\") revealed that the \"base\" version consistently performed better, underscoring the influence of stronger pre-trained models. Overall, these findings demonstrate that the method's performance can be elevated by using higher resolutions and selecting more advanced pre-trained models, delineating a promising direction for future enhancements.\n\n- [1] Xu, Jiarui, and Xiaolong Wang. \"Rethinking self-supervised correspondence learning: A video frame-level similarity perspective.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.\n- [2] Oquab, Maxime, et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\n\n\n\n"}, {"review_id": "QS4SdfwuSC", "reviewer": "Reviewer_XSWn", "summary": "This paper proposes a novel method for semi-supervised video object segmentation. The method combines pre-trained deep features from still images with streaming-data clustering techniques to model the object and the background as dynamic ensembles of von Mises-Fisher mixtures. The method does not require any additional training or fine-tuning on videos, and has a low memory footprint by storing only cluster-level information. The method also incorporates spatial coherence, outlier rejection, and convolutional conditional random fields to improve the segmentation quality. The paper demonstrates that the method achieves state-of-the-art results on two challenging benchmarks: DAVIS-2017 and YouTube-VOS 2018.\n\n\n", "questions": "Please refer to weakness.", "limitations": "The limitation has been well discussed in Supplemental Material.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper shows that using pre-trained features from still images can eliminate the need for costly and supervised training on videos, while using streaming-data clustering can adapt to temporal changes and reduce memory consumption.\n1. The idea of using vMF distribution to model the changes of features in stream data is interesting. \n2. The memory-conscious strategy opens possibilities for processing long videos, while the current solution is limited to videos with only about one hundred frames.", "weaknesses": "1. Almost all of the methods compared in this work target correspondence learning, which can not only handle VOS but also pose/object tracking. Thus it is reasonable that this work specifically tailored for VOS will achieve SOTA performance.\n2. Though no training video is required, the backbone has to be pre-trained on million static images which is much larger than and has more diverse scenes than YouTube-VOS or Kinetics.\n3. The improvement of performance on YouTube-VOS 2018 is not as considerable as that on DAVIS17 val. Moreover, it is even inferior to [1] on YouTube-VOS 2018 (i.e., 71.5 v.s. 72.4) with a stronger backbone (i.e., XCiT-small v.s. ResNet-50).\n4. How about the inference speed of the proposed method compared to existing work? Will multiple vMF mixture models impose a tremendous burden on inference?\n5. Please check the bib carefully, e.g. [7] and [8] are the same.\n\n[1]. Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentation. CVPR23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful, useful, and positive review. \n\n- Our claim is for SOTA results only on the VOS task, and we make no claims to solve other tasks. However, the methods we cited and compared to are the current SOTA (which we beat…) in VOS, so we had to compare with them even if they can also solve non-VOS tasks. \n- The uniqueness of our approach stems from its capability to understand video dynamics without any exposure to video sequences during training. This is a non-trivial task as it necessitates the model to contend with video-specific attributes such as occlusion, camera movement, and changes in object appearances, which are absent in static images. Therefore, the proficiency to learn from static images and apply this knowledge effectively to video sequences enhances the novelty and complexity of our method.\n- The evaluation of our method on the YouTube-VOS 2018 dataset yielded a score of 71.5, slightly below [1]'s score of 72.4. It's significant to note that [1] utilized the same self-supervised pretraining scheme as [2], followed by additional specialized training on this dataset, whereas the pre-trained features used by our method were trained using only static images. However, this specialized approach by [1] didn't result in superior performance on the DAVIS2017 validation and test sets. Moreover, [1] demonstrates limitations in frame rate and an inability to utilize higher-resolution features, as discernible from their pseudo code. In contrast, our method's performance underscores its more robust adaptability across different datasets and superior handling of high-resolution features, even when solely employing static images during training.\nFurthermore, when we examine the performance of our method against INO, which employs a backbone of similar strength to ours (ViT B/8), our method still delivers superior results. This direct comparison demonstrates that our results aren't solely a product of the powerful backbone.\n- Performance Comparison in Frames Per Second (FPS):\n\n  | Resolution       | Feature Dimension | Ours  | INO  | DULVS |\n  |------------------|-------------------|-------|------|-------|\n  | 1/4 (120 x 210)  | 384               | 8.25  | 0.13 | 0.14  |\n  |                  | 768               | 6.50  | 0.11 | 0.12  |\n  |                  | 1152              | 5.25  | 0.10 | 0.11  |\n  | 1/2 (240 x 420)  | 384               | 1.55  | OOM  | OOM   |\n  |                  | 768               | 1.28  | OOM  | OOM   |\n  |                  | 1152              | 1.10  | OOM  | OOM   |\n\n  _OOM: Out of Memory_\n\n  We evaluated three models using feature maps at resolutions of 1/4 and 1/2 of the image size (480 x 840) across three feature dimensions (384, 768, 1152). This comparison involved computing the frames per second (FPS) after the first 21 frames, exclusive of the feature extraction step. Our model consistently outperformed the others, exhibiting particular strength in higher resolutions where competing models faced out-of-memory (OOM) issues. \n  (Tested using Tesla V100-32GB)\n\n\n- [1] Li, Liulei, et al. \"Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n- [2] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n\n\n"}]

Author rebuttal:
We thank the reviewers for their thoughtful reviews and constructive criticism. We are glad that, overall, the paper was positively received. 

Here, in the general response, we touch upon several points common to more than one reviewer. Below, we reply to each reviewer separately. 

### **Memory Utilization and Computational Load:**
The cumulative memory utilization during inference is affected by:
- **Memory Storage**: Traditional methods store multiple dense feature maps from prior frames, increasing the memory footprint based on the quantity and size of these maps. Our method significantly reduces this by tying memory usage to the number of clusters (as opposed to the much larger number of features), which demands far less memory than even a single dense feature map.
- **Computational Load**: Standard label propagation techniques compute cosine similarity for each feature point in the current frame with all feature points within a certain radius in all reference frames. That approach struggles with scalability, particularly when the feature map resolution increases. The higher the resolution, the larger the radius needed for cosine similarity computation, intensifying the computational challenge. In stark contrast, our method improves on this by computing similarities relative to the clusters present within a certain radius, not individual feature points, thus enhancing scalability (since the number of clusters within a region is much smaller than the number of features in that region).


### **Performance Comparison in Frames Per Second (FPS):**
| Resolution       | Feature Dimension | Ours  | INO  | DULVS |
|------------------|-------------------|-------|------|-------|
| 1/4 (120 x 210)  | 384               | 8.25  | 0.13 | 0.14  |
|                  | 768               | 6.50  | 0.11 | 0.12  |
|                  | 1152              | 5.25  | 0.10 | 0.11  |
| 1/2 (240 x 420)  | 384               | 1.55  | OOM  | OOM   |
|                  | 768               | 1.28  | OOM  | OOM   |
|                  | 1152              | 1.10  | OOM  | OOM   |
_OOM: Out of Memory_

We evaluated three models using feature maps at resolutions of 1/4 and 1/2 of the image size (480 x 840) across three dimensions (384, 768, 1152). This comparison involved computing the frames per second (FPS) after the first 21 frames, exclusive of the feature extraction step. Our model consistently outperformed the others, exhibiting particular strength in higher resolutions where competing models faced out-of-memory (OOM) issues. 
(Tested using Tesla V100-32GB)


### **Adopting Deeper Networks:**
It is important to note that simply adopting deeper networks doesn't necessarily guarantee superior results. As observed in previous self-supervised learning approaches like VFS [1], performance sometimes remains unchanged or even degrades when using deeper backbones like ResNet-50. It can even introduce additional complications which already exist during training, such as issues with memory and convergence. Thus, our method provides an effective and much-needed solution to a practical problem.  
The table below represents an evaluation conducted on the DAVIS-2017 validation set.
| Features                | J&F-Mean | J-Mean  | J-Recall | F-Mean  | F-Recall |
|-------------------------|----------|---------|----------|---------|----------|
| ours+dinov2_small       | 0.776    | 0.754    | 0.854    | 0.798    | 0.890    |
| ours+dinov2_base        | 0.782    | 0.760    | 0.880    | 0.804    | 0.900    |
| oursx2+dinov2_small     | 0.784    | 0.762    | 0.873    | 0.806    | 0.905    |
| oursx2+dinov2_base      | 0.803    | 0.780    | 0.885    | 0.825    | 0.916    |

In the experiment, DINOv2's [2] features were concatenated with our original features, a necessity due to DINOv2's large patch size of 14. This fusion provided more effective information utilization. The results showed a marked improvement in J&F-Mean with the "oursx2" configurations (features with higher spatial dimensions). This improvement is attributed to our method's ability to exploit and use higher resolutions, facilitated by XCiT's capability to handle them. A comparison between the DINOv2 versions ("small" vs "base") revealed that the "base" version consistently performed better, underscoring the influence of stronger pre-trained models. Overall, these findings demonstrate that the method's performance can be elevated by using higher resolutions and selecting more advanced pre-trained models, delineating a promising direction for future enhancements.

### **References:**
- [1] Xu, Jiarui, and Xiaolong Wang. "Rethinking self-supervised correspondence learning: A video frame-level similarity perspective." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
- [2] Oquab, Maxime, et al. "Dinov2: Learning robust visual features without supervision." arXiv preprint arXiv:2304.07193 (2023).




Meta-review: Four reviewers vote for accept, and one reviewer gave borderline reject, and not willing to argue strongly against acceptance.

I would encourage the authors to take the reviewers' comments into consideration in the final camera ready paper.
Decision: Accept (poster)