Title: From ViT Features to Training-free Video Object Segmentation via Streaming-data Mixture Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Unsupervised

video segmentation

clustering
Abstract: In the task of semi-supervised video object segmentation, the input is the binary mask of an object in the first frame, and the desired output consists of the corresponding masks of that object in the subsequent frames. Existing leading solutions have two main drawbacks: 1) an expensive and typically-supervised training on videos; 2) a large memory footprint during inference. Here we present a training-free solution, with a low-memory footprint, that yields state-of-the-art results. The proposed method combines pre-trained deep learning-based features (trained on still images) with more classical methods for streaming-data clustering. Designed to adapt to temporal concept drifts and generalize to diverse video content without relying on annotated images or videos, the method eliminates the need for additional training or fine-tuning, ensuring fast inference and immediate applicability to new videos. Concretely, we represent an object via a dynamic ensemble of temporally- and spatially-coherent mixtures over a representation built from pre-trained ViT features and positional embeddings. A convolutional conditional random field further improves spatial coherence and helps reject outliers. We demonstrate the efficacy of the method on key benchmarks: the DAVIS-2017 and YouTube-VOS 2018 validation datasets. Moreover, by the virtue of the low-memory footprint of the compact cluster-based representation, the method scales gracefully to high-resolution ViT features. Our code is available at https://github.com/BGU-CS-VIL/Training-Free-VOS

From ViT Features to Training-free Video Object Segmentation via Streaming-data Mixture Models

Abstract

In the task of semi-supervised video object segmentation, the input is the binary mask of an object in the first frame, and the desired output consists of the corresponding masks of that object in the subsequent frames. Existing leading solutions have two main drawbacks: 1) an expensive and typically-supervised training on videos; 2) a large memory footprint during inference. Here we present a trainingfree solution, with a low-memory footprint, that yields state-of-the-art results. The proposed method combines pre-trained deep learning-based features (trained on still images) with more classical methods for streaming-data clustering. Designed to adapt to temporal concept drifts and generalize to diverse video content without relying on annotated images or videos, the method eliminates the need for additional training or fine-tuning, ensuring fast inference and immediate applicability to new videos. Concretely, we represent an object via a dynamic ensemble of temporally- and spatially-coherent mixtures over a representation built from pre-trained ViT features and positional embeddings. A convolutional conditional random field further improves spatial coherence and helps reject outliers. We demonstrate the efficacy of the method on key benchmarks: the DAVIS-2017 and YouTube-VOS 2018 validation datasets. Moreover, by the virtue of the low-memory footprint of the compact cluster-based representation, the method scales gracefully to high-resolution ViT features. Our code is available at https://github.com/BGUCS-VIL/Training-Free-VOS.

1 Introduction

Video Object Segmentation (VOS), a key computer-vision task, aims to distinguish objects of interest from the background across a sequence of video frames. In Semi-supervised VOS (SVOS), the topic of this paper, the user provides the mask of the object in the first frame (while in the case of unsupervised VOS, no mask is given), and the task is to propagate that mask to the next frames. Prior to the revolution of Deep Learning (DL), SVOS relied heavily on traditional techniques like color-based segmentation, point trajectories, and optical flow. However, the emergence of DL-based methods has significantly outpaced those traditional techniques. Thus, today DL-based methods dominate the field of SVOS. Note that according to the standard terminology in the area of VOS, the term "semi-supervised" in the name SVOS refers to the nature of the task (that is, the fact that a user provides the mask in the first frame), not the nature of the solution . In other words, solutions to the SVOS task can be either supervised, semi-supervised, or unsupervised. For example, a supervised

Background

Object

Background

Object

Feature extraction and

clustering per object

assignment

CRF Refinement

DL solution to SVOS typically requires multiple labeled videos to train on. In contrast, the approach proposed in this paper requires neither supervision nor learning on videos.

The main strategy employed by DL-based SVOS methods revolves around pixel-level matching to establish inter-frame correspondence. This is achieved by creating pixel-to-pixel connections between the current frame and the preceding ones. That strategy, however, often has two main limitations: 1) the need for an extensive training on large-scale (and typically annotated) video datasets; 2) a significant memory footprint due to the storage requirements of (dense representations of) previous frames for pixel matching. Addressing those prevalent challenges, we propose an unsupervised, adaptive, and compact SVOS method. By combining DL and classical techniques and capitalizing on the widespread practice of representing features as points on a hypersphere, the proposed model incorporates an ensemble of temporally- and spatially-coherent von Mises-Fisher (vMF) mixtures. The mixtures are updated dynamically in a streaming manner, together with a convolutional implementation of a Conditional Random Field (CRF).

Our method uses features from a pre-trained unsupervised deep neural network - trained on still images, not videos - but does not require training by itself as it only performs inference. The method fits multiple vMF mixtures to each object of interest (and to the background) in the initial frame. The fact that we employ multi-scale modeling (where each mixture has its own number of components) lets us represent the object at multiple levels of granularity. In subsequent frames, pixels are assigned to the most probable component (across all of the components in all of the mixtures in all of the objects). The mixture ensembles themselves are updated dynamically while taking into account temporal and spatial coherence. Following established stream clustering techniques [13], we apply a weighted window with a kernel function that reduces the weight of older frames, mitigating memory consumption and improving computational efficiency without storing extensive history of dense representations of past frames. Our model adapts to concept drifts in the objects of interest or the background. We improve spatial consistency by constraining pixel-component membership based on location and refine results using a Pixel-Adaptive Convolutional CRF (PAC-CRF) [37], fostering pixel consistency and awareness of outliers. Together with outlier-rejection mechanisms, the method mitigates error propagation across frames.

To summarize, our key contributions are: 1) We demonstrate that a synergy between classical and DL techniques can provide an effective solution for SVOS that achieves state-of-the-art results on the DAVIS-2017 and YouTube-VOS 2018 datasets. 2) By leveraging unsupervised pre-trained image-based features, our method eliminates the need for further training, and in particular no training on videos is required (unlike most competitors which require large-scale video datasets for training). 3) Our method has a low-memory footprint by storing only cluster-level information, as opposed to dense past-frame representations. Thus, the method does not only scale better than competing ones but also enables handling high-resolution features for higher-quality segmentation.

2 Related work

The evolution of SVOS methods . From older SVOS methods based on classic vision techniques [11, 9, 21, 50, 10, 5], the focus has shifted to DL-based SVOS methods, also known as one-shot VOS. Early semi-supervised methods focused on online fine-tuning of a pre-trained network, exemplified by OSVOS [6]. In fact, OSVOS pioneered the online fine-tuning of pre-trained networks in SVOS. However, OSVOS is prone to overfitting and fails to adapt to gradual concept drifts in the objects of interest. To address these issues, numerous extensions and refinements to OSVOS were proposed. OnAVOS [41] introduced additional fine-tuning with high-confidence frames in order to handle gradual concept drifts, albeit at the cost of increased computational complexity. Maninis et al . [28] integrated semantic information into the predictions via Mask R-CNN [17], thereby reducing the dependency on temporal smoothness. In an innovative departure from common practice, BubbleNets [16] proposed selecting the most suitable frame for annotation, challenging the conventional practice of annotating the first frame. Other works addressed different aspects of the OSVOS algorithm. For example, OSNM [47] utilized network modulation for more efficient fine-tuning, while A-Game [20] employed a Gaussian Mixture Model to better capture both the foreground and the background. Other noteworthy and relevant works include, but are not limited to, FRTM [35], LWL [4], and TAODA [51].

The emergence of matching-based methods . Recently, pixel matching methods have gained prominence due to their superior speed and results [15]. These methods, which either implicitly predict inter-frame similarities or explicitly match features between pixel pairs across frames, have been adopted by recent SVOS methods [22, 18, 3], leading to state-of-the-art performance.

Unresolved issues and evolving developments in SVOS . Constructing discriminative feature embeddings with temporal consistency is crucial for reliable correspondence. This is typically achieved by training backbone networks on large-scale video datasets such as OxUvA [39], YouTubeVOS [46], TrackingNet [46], and Kinetics-400 [8]. Most existing methods store dense feature representations of multiple previous frames and perform spatial correlation to propagate information across frames. However, this significantly increases memory consumption, while reducing the number of reference frames degrades the performance. Hence, finding a memory-efficient way to propagate the information is of paramount importance for achieving accurate and robust segmentation results.

3 Method

We start with the task definition. In the SVOS task, the user provides (only for the first frame) a binary mask for each object of interest. Let ( M 1 o ) N obj o =1 denote the masks where N obj is the number of objects. Those (non-overlapping) N obj masks imply another mask, M 1 N obj +1 , which represents the background at time t = 1 ( i.e ., the first frame). That mask is formed by taking the binary complement of the union of ( M 1 o ) N obj o =1 . We will henceforth treat the background as just another object unless stated otherwise. For each time t = 2 , 3 , . . . , T , where T is the total number of frames, the goal is to estimate the corresponding masks, denoted by ( M t o ) N obj+1 o =1 .

Our unsupervised method for the SVOS task, depicted in Fig. 1, can be summarized as follows. We extract features from video frames (§ 3.1) and model the within-object feature distribution via a multi-scale dynamic ensemble of vMF mixtures (§ 3.2). We use a streaming-data ExpectationMaximization algorithm to update the parameters of the mixtures (§ 3.3). We assign pixels to objects based on the component-wise posterior probabilities in the entire ensemble of each object (§ 3.4). For robustness, we disregard outliers during the parameter-estimation steps, and further

improve the results by incorporating a Pixel-Adaptive Convolutional CRF (§ 3.5) for additional outlier management and segmentation refinement. For examples of qualitative results see, e.g ., Fig. 2.

3.1 Feature extraction

The feature representation we use consists of ViT features [1] and positional embeddings [40]. For the ViT features, we use the publicly-available pre-trained weights of XCiT (Cross-Covariance Image Transformer) [1], that had been trained in a self-supervised manner as outlined in [7].

Let N pixels be the number of pixels in each frame. Let ˜ X t = ( ˜ x t i ) N i =1 denote the ViT features extracted from I t (the video frame at time t ), where N is the number of features. For example, if the ViT features are extracted using non-overlapping 8 × 8 patches, then N = N pixels / 64 . The XCiT variant, like its DINO-ViT counterpart, excels in extracting rich and discriminative features from images [2], but with an added advantage of better handling high-resolution images. In our case, we employ an XCiT model that yields a feature vector ˜ x t i ∈ R 384 for each image patch i . To capture the spatial relationships between neighboring pixels, we extend each ViT feature, ˜ x t i , with its corresponding rotary positional embedding, ρ ( i ) , as per the methodology detailed in Su et al. [38]. This embedding represents the 2D spatial location of the ViT feature in a 64-dimensional space. Consequently, we form the extended feature vector x t i = ( ˜ x i t , w ρ · ρ ( i )) (where w ρ > 0 is the weight of the positional embeddings), normalized such that ∥ x t i ∥ ℓ 2 = 1 . We denote by X t = ( x t i ) N i =1 the collection of all these unit-length (extended) features at time t . Finally, given the initial masks, ( M t o ) N obj+1 o =1 , we partition X 1 , the features at t = 1 , into ( X 1 o ) N obj +1 o =1 such that, for each o ∈ (1 , . . . , N obj +1) , X 1 o represents the features that correspond to M 1 o .

3.2 Modeling the within-object feature distribution via a multi-scale vMF mixture

In our pursuit of capturing intricate patterns in high-dimensional data while ensuring computational efficiency, we turn to vMF mixtures; namely, a mixture model whose each component is a vMF distribution (defined below). These mixtures, known for their simple structure that depends solely on mean directions and concentration parameters, are highly efficient computationally. They are also adept at modeling data on a unit hypersphere, which is a prevalent form of representation for deep features. Since often the feature distribution of an object is not only complex but also dependent on its size ( e.g ., when the object-to-camera distance increases, the object resolution decreases and fine details tend to disappear, which in term affects the distribution), we advocate for a multi-scale approach, using a mixture ensemble. For each object, we employ S independent vMF mixture models, each corresponding to a different scale or granularity. This way, we capture details at multiple scales.

The vMF distribution is a probability distribution over S d -1 ≜ { x : x ∈ R d , ∥ x ∥ ℓ 2 = 1 } , the unit sphere in R d . Its probability density function (pdf), evaluated at x ∈ S d -1 , is

where µ ∈ S d -1 is the mean and τ > 0 is the concentration parameter. The normalizer C d ( τ ) depends only on τ and has a closed form that appears in Appendix A .

Let s ∈ { 1 , . . . , S } . Mixture s for object o at time t is defined as follows:

Here, α t o,s,k represents the mixing proportions while µ t o,s,k and τ t o,s,k are the mean and concentration parameter, respectively, of component k at time t . The predefined number of components, K s , is different for every s and represents the scale of the mixture. The full set of parameters describing object o at time t is θ t o = ( ( α t o,s,k , θ t o,s,k ) K s k =1 ) S s =1 , where θ t o,s,k = ( µ t o,s,k , τ t o,s,k ) .

3.3 Parameter estimation via Expectation-Maximization

We adapt the widely-used Expectation-Maximization (EM) framework [12] to estimate the parameters of our vMF mixtures in a streaming-data manner, tailoring it specifically for VOS. The algorithm is applied independently and in parallel to each mixture.

Our EM algorithm consists of two stages. The Expectation step (E-step) calculates the posterior probabilities of component membership ( i.e ., feature-to-component assignment), integrating spatial coherence to improve segmentation accuracy. In the Maximization step (M-step), we update the mixture parameters to maximize the expected log-likelihood, introducing modifications to handle the streaming nature of video data and manage memory efficiently.

3.3.1 E-step: calculating posterior probability with spatial coherence

In the E-step we incorporate spatial coherence, a crucial factor in VOS. Let us assume that, at time t , pixel i was assigned to object o (we will later explain how pixels are assigned to objects). For each mixture s of the S mixtures associated with object o , we do the following. For every k = 1 , . . . , K s , let q t o,s,k,i be the posterior probability that, at time t , pixel i belongs to component k (of mixture s ) given the observed feature x t i and the current estimate of ( α t o,s,k , θ t o,s,k ) K s k =1 . Let N t o be the number of nonzero pixels in M t o and, in a slight abuse of notation, let ( x t i ) N t o i =1 denote the object features at time t . We modify the calculation of the posterior probability, q t o,s,k,i , by integrating a spatial constraint. This constraint is expressed as an indicator function that respects both the current parameter estimates and the spatial information inherited from the previous frame, t -1 . This approach ensures a smooth transition between frames and improves the spatial integrity of objects in the video sequence. Taken together, the modified posterior probability is

where z o,s,i is the assignment of x t i to a component in mixture s in object o , p ( z o,s,i = k | x t i , θ t o,s,k ) ∝ p ( x t i | θ t o,s,k ) = vMF( x t i ; µ t o,s,k , τ t o,s,k ) , and the binary-valued indicator function,

returns 1 if x t i has a positional embedding that closely matches ρ ( µ t -1 o,s,k ) , the positional part of the mean of cluster k from the previous frame. The quality of the match is measured in terms of cosine similarity. Note that the assignment also depends on a certain threshold, r s , which in turn depends on the number of clusters, K s . As K s increases, a higher degree of similarity is required, reflecting the finer granularity of the mixture. If no such pixel exists, the function returns 0, effectively preventing the current pixel from being assigned to cluster k . This approach ensures spatial coherence.

3.3.2 M-step: parameter estimation

We first provide the details for the M-step while ignoring the streaming nature of the data, and then later, in § 3.3.3, explain how we account for the streaming. In the M-step, the parameters are updated to maximize the expected log-likelihood. We iteratively update a set of sufficient statistics that encapsulate the critical properties of the data and the model. Concretely, at time t we compute

The M-step uses well-known closed-form solutions for the estimated parameters of the vMF distribution. In particular, to estimate the concentration parameters we employ an approximation scheme based on the continued fraction method, supplemented with a correction term:

To circumvent the computational expense of directly calculating the modified Bessel function (which appears in C d ( · ) ), we adopt an alternative method to compute the log-likelihood log p ( x t i | θ t o,s,k ) using the mean vector µ t o,s,k and the concentration parameter τ t o,s,k . Specifically, we leverage the fact that the normalizing constant C d ( τ t o,s,k ) depends solely on τ t o,s,k and can be approximated [36] by a simpler expression (see Appendix A ). This enables us to express the log-likelihood as follows:

140

.......

21ReferenceFrames

Max: 126.0

5 ReferenceFrames

120

1Reference Frame

Memory Consumption (GB)

Ours

100

80

60

40

20

Max:1.8

0

250

500

750

100012501500

1750

2000

where ⊤ denotes transpose and x denotes the (Euclidean) sample mean of the data points. We now address the dynamic characteristics of video data.

3.3.3 Memory-efficient dynamic updates and temporal coherence

In the dynamic environment of video sequences, objects may experience occlusion, reemergence, or gradual appearance changes due to, e.g ., varying lighting conditions. Addressing these challenges, and inspired by [13], our method integrates temporal coherence into the clustering procedure using time-weighted sufficient statistics. We devise these statistics as a weighted sum of data values across multiple frames, with weights diminishing exponentially over time. By maintaining historical records of a limited length, and updating the time-weighted statistics for each frame, we efficiently manage space, memory, and time complexity. Data from the preceding frame is retained only if its (time-decaying) weight exceeds a certain threshold. Concretely, we employ a time-decaying weight K ( t, t ′ ) = 2 λ ( t -t ′ ) where the user-defined λ > 0 controls the correlation between sequential frames. As a result, older frames are assigned lower weights. Similarly to [13], we modify the sufficient statistics accordingly, and obtain a streaming-data variant of Eq. 5:

Our method's efficient information propagation from prior frames results in smoother transitions and improves the temporal coherence of the segmentation. Retaining (weighted) sufficient statistics from previous frames helps with the detection of reappearing, previously-occluded, objects or parts. Importantly, our memory consumption is determined by the number of clusters in the model, not N , the (much larger) feature map size. This drastically reduces memory usage during inference (see Fig. 3) and enables us to obtain a compact representation. Moreover, this drastic memory reduction (in comparison to methods which rely on saving history of dense representations) enables us to use higher-resolution features than the competitors (see, e.g ., Fig. 4). Importantly, as K ( t, t ′ ) decreases when | t -t ′ | increases and eventually decays to zero, in practice we store the sufficient statistics from time t ′ only until time t ′ +15 and then discard it.

3.4 Pixel-to-object assignment

Given the estimated models from time t , in the subsequent frame we extract features X t +1 and assign labels to each pixel based on the mixture (from time t ) that has the highest probability for any of its components. By employing the maximum-a-posteriori (MAP) rule, we determine the component

with the highest probability for each pixel, and then assign the pixel to the object associated with that component. In other words, for every o ∈ { 1 , . . . , N obj +1 } we compute

and then assign the pixel to object according to arg max o : o ∈{ 1 ,...,N obj +1 } p t +1 i,o . Next, we generate the (non-overlapping) masks for time t +1 , ( M t +1 ) N obj+1 o =1 via

where M t +1 o ( i ) denotes pixel i of M t +1 o ′ .

3.5 Refining the segmentation using outlier rejection and a convolutional CRF

Recall that the initial mask is the sole ground truth reference that the method has access to. Thus, as the video sequence unfolds, the presence of outliers poses a substantial challenge to the accuracy of the clustering process. These outliers, which stem from factors such as occlusions, illumination changes, or noise, might hurt the integrity of existing clusters or lead to confusing them with unrelated clusters. Our approach for mitigating such potential sources of error incorporates two strategies: outlier rejection and a refinement using a convolutional CRF.

In the initial phase, we exclude points from the parameter-estimation step when the pixel-to-object assignment is inconclusive. Specifically, if the score of a pixel's second-best match closely rivals that of its best match, we consider that pixel an outlier. We then employ the Pixel-Adaptive Convolutional CRF (PAC-CRF) [37] to further refine the segmentation. PAC-CRF, with its fixed window connections around each pixel, effectively filters noise while preserving segmentation coherence. It bolsters our outlier handling strategy by enabling the exclusion of points where the PAC-CRF prediction conflicts with our initial prediction. After several EM steps (where the M step excludes the outliers), we apply PAC-CRF again, leveraging the source image to improve the pixel-wise accuracy and the segmentation quality. Furthermore, we leverage the learnability of the PAC-CRF's pairwise potentials' weights and optimize them using Focal-Loss [26, 30]. This optimization, carried out over a few gradient steps with the ground-truth annotation from the first frame, ensures that our method adapts well to the given data, leading to improved performance and more accurate segmentation maps (Fig. 2).

4 Results

In this section, we present the performance of the proposed method on the DA VIS-2017 and YouTubeVOS datasets. A subsequent subsection delves into our adeptness at harnessing features from deeper

INO [33]

Ours

networks and superior resolutions. Video demonstrations can be found here. Details regarding our implementation are available in Appendix B .

4.1 Evaluation on DAVIS-2017 and YouTube-VOS

We report the results on two widely-used SVOS benchmarks: YouTube-VOS [46] and DAVIS 2017 [34]. Commonly used metrics for assessing VOS performance include the Jaccard's index (J) [14], which calculates the intersection-over-union (IoU) of the object mask, and the F-measure (F) [29], which assesses the contour accuracy. These metrics provide comprehensive insight into the segmentation performance by evaluating both the object area and contour precision. We report the mean IoU for mask and contour, denoted as J m and F m , as well as the recall J r and F r , which are

calculated using an IoU threshold of 0.5. The metric J & F m represents the average of J m and F m . For a fair comparison, we use the same video resolution as the competing methods, despite the fact that our method can process higher resolutions than them.

DAVIS 2017 . The DAVIS-2017 dataset [34], featuring high-quality videos with diverse object appearances and challenging motion patterns, is a widely-accepted SVOS benchmark. On the DAVIS-2017 validation set (Table 1), our method demonstrated top-tier scores in the Jaccard index, F-measure, and J & F m metrics. Specifically, our results include a J & F m score of 76.3, J m of 73.8, J r of 85.6, F m of 78.7, and F r of 89.2. These results clearly outperform the other methods (see also Fig. 5). We also evaluated our method on the DA VIS-2017 training and test-dev sets, collectively constituting 90 additional sequences. These sets allowed for a broader performance assessment, with the training set offering an extra dimension of evaluation since it is not included in the training data of the methods we compared with. Results from the test-dev set, officially validated by the evaluation server, further demonstrate our method's efficacy (Table 2 & Table 3). Qualitative results from these extended evaluations are available in Appendix C .

YouTube-VOS . The YouTube-VOS dataset [46] serves as a large-scale benchmark for SVOS, introducing a variety of complexities such as occlusions, scale changes, and motion blur. Unique to this dataset is the division of object classes between the training and validation sets, where only a subset is shared, allowing the benchmark to distinguish between 'seen' and 'unseen' categories. Our method's scores, obtained through the official evaluation server, showcase its adaptability in navigating these complex situations, surpassing the performance of several state-of-the-art approaches (Table 6).

4.2 Performance and feature analysis

Our method adeptly processes high-resolution features, distinguishing it from other models constrained by memory, as illustrated in Table 4. While merely adopting deeper backbones does not guarantee superior outcomes [45], our model showcases its capability to harness the strengths of DINOv2's [32] features. To counteract the reduced spatial information inherent in DINOv2 due to its larger patch size, We concatenated the DINOv2 features with the XCiT features we had previously used. This approach ensures a rich and comprehensive representation of the visual scene. As demonstrated in Table 5, our method effectively capitalizes on the enhanced features from both deeper backbones and higher spatial resolutions.

4.3 Ablation study

We performed an ablation study (Fig. 6) to analyze the influence of different parts of the method on the performance. The combination of outlier detection and segmentation refinement showed a

76

.

3

74

.

1

74

.

2

66

.

6

λ

=

∞

73

.

9

λ

= 2

.

0

75

.

6

λ

= 1

.

0

70

S

= 1

73

.

1

S

= 2

74

.

1

S

= 5

76

.

0

73

.

6

w

ρ

= 0

75

.

0

w

ρ

= 5

75

.

2

w

ρ

= 10

74

,

7

w

ρ

= 30

w/o CRF refinement

w/o outliers removal

Varying

λ

w/o

I

Varying

S

Varying

w

ρ

synergistic improvement in performance, with each contributing significantly when implemented separately. The necessity of our streaming mechanism was underscored when we observed a notable performance drop while solely relying on the previous frame for initializing mixture parameters ( λ = ∞ ). Furthermore, examining the number of mixtures per object, we notice a substantial performance boost when increasing the number of mixtures from one to two, underscoring the importance of multi-scale object representations. However, the added benefit started to decay with further increase, leading us to settle on S = 10 mixtures in our reported results. Concerning spatial components, removing positional embeddings ( i.e ., w ρ = 0 ) resulted in a major performance drop, confirming their essential role. Additionally, the introduction of the indicator function led to an additional positive effect on performance.

5 Conclusion

We proposed a novel SVOS approach that achieves state-of-the-art results on challenging benchmarks. Our memory-conscious strategy also opens possibilities for processing high-resolution videos, suggesting a valuable avenue for future video understanding research.

5.1 Limitations

The proposed method faces difficulties in two scenarios (see Appendix D for examples). The first is related to re-identification challenges, where occlusion between multiple similar instances of the same object type ( e.g ., two camels) makes it challenging to assign correct labels to each instance. This limitation stems from the method's lack of explicit object identity representation and modeling of temporal dynamics, which could help disambiguate overlapping objects. The second scenario occurs in the presence of a strong motion blur. The method is sensitive to such a blur, likely because the ViT features were trained on still images, not videos. This limitation becomes more pronounced when motion blur persists across consecutive frames, as then the error propagation is more significant.

Acknowledgments and disclosure of funding

This work was supported by: the BGU CS Lynn and William Frankel Center; the Israeli Council for Higher Education via the BGU Data Science Research Center; Israel Science Foundation Personal Grant #360/21.

References

Alaaeldin Ali, Hugo Touvron, Mathilde Caron, Piotr Bojanowski, Matthijs Douze, Armand Joulin, Ivan Laptev, Natalia Neverova, Gabriel Synnaeve, Jakob Verbeek, et al. Xcit: Cross-covariance image transformers. NeurIPS , 2021. 4

Shir Amir, Yossi Gandelsman, Shai Bagon, and Tali Dekel. Deep vit features as dense visual descriptors. In ECCV Workshop on What is Motion For? , 2022. 4

Nikita Araslanov, Simone Schaub-Meyer, and Stefan Roth. Dense unsupervised learning for video segmentation. NeurIPS , 2021. 3, 7, 8, 9, 10

Goutam Bhat, Felix Järemo Lawin, Martin Danelljan, Andreas Robinson, Michael Felsberg, Luc Van Gool, and Radu Timofte. Learning what to learn for video object segmentation. In ECCV , 2020. 3

Thomas Brox and Jitendra Malik. Object segmentation by long term analysis of point trajectories. In ECCV . Springer, 2010. 3

Sergi Caelles, Kevis-Kokitsi Maninis, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. In CVPR , 2017. 3, 8, 10

Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV , 2021. 4, 8

Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In CVPR , 2017. 3

Shao-Yi Chien, Shyh-Yih Ma, and Liang-Gee Chen. Efficient moving object segmentation algorithm using background registration technique. IEEE Transactions on Circuits and Systems for Video Technology , 2002. 3

Prakash Chockalingam, Nalin Pradeep, and Stan Birchfield. Adaptive fragments-based tracking of non-rigid objects using level sets. In ICCV . IEEE, 2009. 3

Rita Cucchiara, Costantino Grana, Massimo Piccardi, and Andrea Prati. Detecting moving objects, ghosts, and shadows in video streams. IEEE TPAMI , 2003. 3

Arthur P Dempster, Nan M Laird, and Donald B Rubin. Maximum likelihood from incomplete data via the EM algorithm. Journal of the royal statistical society: series B (methodological) , 1977. 4

Or Dinari and Oren Freifeld. Sampling in dirichlet process mixture models for clustering streaming data. In AISTATS . PMLR, 2022. 2, 6

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. IJCV , 2009. 8

Mingqi Gao, Feng Zheng, James JQ Yu, Caifeng Shan, Guiguang Ding, and Jungong Han. Deep learning for video object segmentation: a review. Artificial Intelligence Review , 2023. 3

Brent A Griffin and Jason J Corso. Bubblenets: Learning to select the guidance frame in video object segmentation by deep sorting frames. In CVPR , 2019. 3

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV , 2017. 3

Allan Jabri, Andrew Owens, and Alexei Efros. Space-time correspondence as a contrastive random walk. In NeurIPS , 2020. 3, 8

Sangryul Jeon, Dongbo Min, Seungryong Kim, and Kwanghoon Sohn. Mining better samples for contrastive learning of temporal correspondence. In CVPR , 2021. 8, 10

Joakim Johnander, Martin Danelljan, Emil Brissman, Fahad Shahbaz Khan, and Michael Felsberg. A generative appearance model for end-to-end video object segmentation. In CVPR , 2019. 3

Changick Kim and Jenq-Neng Hwang. Fast and automatic video object segmentation and tracking for content-based applications. IEEE transactions on circuits and systems for video technology , 2002. 3

Zihang Lai, Erika Lu, and Weidi Xie. Mast: A memory-augmented self-supervised tracker. In CVPR , 2020. 3, 8, 10

Zihang Lai and Weidi Xie. Self-supervised learning for video correspondence flow. In BMVC , 2019. 8, 10

Liulei Li, Tianfei Zhou, Wenguan Wang, Lu Yang, Jianwu Li, and Yi Yang. Locality-aware inter-and intra-video reconstruction for self-supervised correspondence learning. In CVPR , 2022. 7, 8, 10

Xueting Li, Sifei Liu, Shalini De Mello, Xiaolong Wang, Jan Kautz, and Ming-Hsuan Yang. Joint-task self-supervised learning for temporal correspondence. NeurIPS , 2019. 8

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV , 2017. 7

Xiankai Lu, Wenguan Wang, Jianbing Shen, Yu-Wing Tai, David J Crandall, and Steven CH Hoi. Learning video object segmentation from unlabeled videos. In CVPR , 2020. 8

K-K Maninis, Sergi Caelles, Yuhua Chen, Jordi Pont-Tuset, Laura Leal-Taixé, Daniel Cremers, and Luc Van Gool. Video object segmentation without temporal information. IEEE TPAMI , 2018. 3

David R Martin, Charless C Fowlkes, and Jitendra Malik. Learning to detect natural image boundaries using local brightness, color, and texture cues. IEEE TPAMI , 2004. 8

Ty Nguyen, Tolga Ozaslan, Ian D Miller, James Keller, Giuseppe Loianno, Camillo J Taylor, Daniel D Lee, Vijay Kumar, Joseph H Harwood, and Jennifer Wozencraft. U-net for mav-based penstock inspection: an investigation of focal loss in multi-class segmentation for corrosion identification. arXiv preprint arXiv:1809.06576 , 2018. 7

Seoung Wug Oh, Joon-Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV , 2019. 8, 10

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193 , 2023. 9

Xiao Pan, Peike Li, Zongxin Yang, Huiling Zhou, Chang Zhou, Hongxia Yang, Jingren Zhou, and Yi Yang. In-n-out generative learning for dense unsupervised video segmentation. In Proceedings of the 30th ACM International Conference on Multimedia , 2022. 8, 9, 10

Jordi Pont-Tuset, Federico Perazzi, Sergi Caelles, Pablo Arbeláez, Alex Sorkine-Hornung, and Luc Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv:1704.00675 , 2017. 8, 9

Andreas Robinson, Felix Jaremo Lawin, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg. Learning fast and robust target models for video object segmentation. In CVPR , 2020. 3

Tyler R Scott, Andrew C Gallagher, and Michael C Mozer. von mises-fisher loss: An exploration of embedding geometries for supervised learning. In ICCV , 2021. 5

Hang Su, Varun Jampani, Deqing Sun, Orazio Gallo, Erik Learned-Miller, and Jan Kautz. Pixel-adaptive convolutional neural networks. In CVPR , 2019. 2, 7

Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. arXiv preprint arXiv:2104.09864 , 2021. 4

Jack Valmadre, Luca Bertinetto, Joao F Henriques, Ran Tao, Andrea Vedaldi, Arnold WM Smeulders, Philip HS Torr, and Efstratios Gavves. Long-term tracking in the wild: A benchmark. In ECCV , 2018. 3

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS , 2017. 4

Paul Voigtlaender and Bastian Leibe. Online adaptation of convolutional neural networks for video object segmentation. arXiv preprint arXiv:1706.09364 , 2017. 3

Carl Vondrick, Abhinav Shrivastava, Alireza Fathi, Sergio Guadarrama, and Kevin Murphy. Tracking emerges by colorizing videos. In ECCV , 2018. 8, 10

Ning Wang, Wengang Zhou, and Houqiang Li. Contrastive transformation for self-supervised correspondence learning. In Proceedings of the AAAI Conference on Artificial Intelligence , 2021. 8

Wenguan Wang, Hongmei Song, Shuyang Zhao, Jianbing Shen, Sanyuan Zhao, Steven CH Hoi, and Haibin Ling. Learning unsupervised video object segmentation through visual attention. In CVPR , 2019. 8

Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. In ICCV , 2021. 8, 9

Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-to-sequence video object segmentation. In ECCV , 2018. 3, 8, 9

Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang, and Aggelos K Katsaggelos. Efficient video object segmentation via network modulation. In CVPR , 2018. 3

Yizhuo Zhang, Zhirong Wu, Houwen Peng, and Stephen Lin. A transductive approach for video object segmentation. In CVPR , 2020. 8, 10

Zixu Zhao, Yueming Jin, and Pheng-Ann Heng. Modelling neighbor relation in joint space-time graph for video correspondence learning. In ICCV , 2021. 8

Di Zhong and Shih-Fu Chang. An integrated approach for content-based video object segmentation and retrieval. IEEE Transactions on Circuits and Systems for Video Technology , 1999. 3

Tianfei Zhou, Jianwu Li, Xueyi Li, and Ling Shao. Target-aware object discovery and association for unsupervised video multi-object segmentation. In CVPR , 2021. 3

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel method for semi-supervised video object segmentation. The method combines pre-trained deep features from still images with streaming-data clustering techniques to model the object and the background as dynamic ensembles of von Mises-Fisher mixtures. The method does not require any additional training or fine-tuning on videos, and has a low memory footprint by storing only cluster-level information. The method also incorporates spatial coherence, outlier rejection, and convolutional conditional random fields to improve the segmentation quality. The paper demonstrates that the method achieves state-of-the-art results on two challenging benchmarks: DAVIS-2017 and YouTube-VOS 2018.\n\nSTRENGTHS:\nThe paper shows that using pre-trained features from still images can eliminate the need for costly and supervised training on videos, while using streaming-data clustering can adapt to temporal changes and reduce memory consumption.\n1. The idea of using vMF distribution to model the changes of features in stream data is interesting. \n2. The memory-conscious strategy opens possibilities for processing long videos, while the current solution is limited to videos with only about one hundred frames.\n\nWEAKNESSES:\n1. Almost all of the methods compared in this work target correspondence learning, which can not only handle VOS but also pose/object tracking. Thus it is reasonable that this work specifically tailored for VOS will achieve SOTA performance.\n2. Though no training video is required, the backbone has to be pre-trained on million static images which is much larger than and has more diverse scenes than YouTube-VOS or Kinetics.\n3. The improvement of performance on YouTube-VOS 2018 is not as considerable as that on DAVIS17 val. Moreover, it is even inferior to [1] on YouTube-VOS 2018 (i.e., 71.5 v.s. 72.4) with a stronger backbone (i.e., XCiT-small v.s. ResNet-50).\n4. How about the inference speed of the proposed method compared to existing work? Will multiple vMF mixture models impose a tremendous burden on inference?\n5. Please check the bib carefully, e.g. [7] and [8] are the same.\n\n[1]. Unified Mask Embedding and Correspondence Learning for Self-Supervised Video Segmentation. CVPR23\n\nQUESTIONS:\nPlease refer to weakness.\n\nLIMITATIONS:\nThe limitation has been well discussed in Supplemental Material.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the semi-supervised video object segmentation problem. It presents a method that relies on clustering features from a pre-trained ViT model. The presented method has a low memory footprint and does not need any additional training. It shows SOTA results on DAVIS-2017 and YouTube-VOS 2018 validation datasets.\n\nSTRENGTHS:\nThe presented algorithm is the first to show a low memory footprint and indeed can scale easily. \n\nThe paper presents a comprehensive study of the different components of the algorithm and an interesting ablation that shows the necessity of the different components. \n\nThe results of the paper are both visually and numerically impressive.\n\nWEAKNESSES:\nWhile the algorithm in the paper results in impressive results, it is hard to follow all the notations and the optimization objectives, mostly due to the nested indexing. While I was able to follow it eventually, I suggest rewriting it (one possibility is to start with a simple 2 classes foreground and background case -  and extend it later to N classes).\n\nOne element that is not ablated in this paper, is the quality of the pre-trained model and the features that are used. To show that this algorithm can improve and produce better results in the future, one should show how the performance of SVOS improves when \"better\" features are given to it. e.g. does a ViT model that was pre-trained on more data result in better downstream performance? Does a larger model improve the SVOS? Does the algorithm improve with higher resolution features? All of these questions are important for understanding if this algorithm will survive the \"test of time\".\n\nQUESTIONS:\nMost of my questions are mentioned above. \nAnother question that I will like to know the answer to is about the time consideration - how much time does it take, as opposed to other methods for SVOS, and how does it scale with the spatial size of the features?\n\nLIMITATIONS:\nThe limitations of the paper are discussed and strongly relate to the choice of the pre-trained model and its corresponding features, and might be addressed by using a model pre-trained on videos.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a training free framework for semi-supervised VOS. Here, the features of objects from a pretrained ViT are represented as vMF, where the objects across frames are associated to perform propagation from initial masks. The results are better than previous self-supervised methods that trained on unlabeled videos.\n\nSTRENGTHS:\nThe idea of associating objects in vMF space for SVOS is interesting, which is a new insight different from the previous contrastive learning based self-supervised methods. The paper is well organized and sufficient comparison visual results are given.\n\nWEAKNESSES:\n-The claimed limitations in SVOS is not true especially for the large memory footprint. Actually, some lightweight designs have been proven in VOS like mobilevos[1*] and AOT[2*]. Here, [1*] focuses a lightweight for real-time vos and [2*] tackles the memory storage issue in STM and the repeated inference for each object id at testing. It's true that the heuristic association method gets rid of the usage of complicated decoders where temporal correspondence and mask generation are performed. However, the large memory is not only from the network parameters but the computational costs. For now, the authors did't give convincing results to demonstrate this contribution like comparisons in terms of params, gflops or fps.\n-For the temporal coherent, is it possible to extend to model both inter-object contrast and intra-object consistency, so the method can directly predict multiple instances at once.\n--Although the method is training free, many hyperparameters need to be tuned during testing. Figure 5 also demonstrates that the change of hyperparameters would impact the final performance. \n\n[1*] MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation, cvpr23.\n[2*] Associating Objects with Transformers for Video Object Segmentation, neurips 21.\n\nQUESTIONS:\n-More details should be illustrated in Table 1 and Table 2, e.g., backbone, resolutions. The proposed method uses ViT while the others always use res18 or res50. It is necessary to demonstrate that the superior results come from the well designed algorithm instead of the usage of more powerful backbone.\n\nLIMITATIONS:\nThe authors discussed the limitations of their methods.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The method tries to combine classical technique for SVOS task namely vMF and CRFs with advance Deep Learning based ViT representations. By doing so, the proposed method eliminates the training requirements on video data. The authors perform comprehensive experimentation to evaluate their approach.

STRENGTHS:
The paper is well written and easy to follow. Authors have done a thorough job in providing comprehensive experimental evaluation for their technique. The major strength of this approach is it does not require video training data.

WEAKNESSES:
- The performance of the model decreases when it encounters the unseen examples as can be observed from the table 2. If the proposed model does not require training then why would the performance on unseen examples is lower?
-  I'm not sure I understand what Fig. 3 represents? Is it a comparison of memory utilization by proposed method and baselines that require 1, 5, 21 reference frames? 
- The claim of low-memory footprint is unclear to me? is the claim made with respect to other baselines? If so, by how much delta is the proposed method better?

QUESTIONS:
- As claimed by authors, their method has a low-memory footprint. How does it compare to the baseline in regard to the speed of performing the VOS over a video? I want to know if there is a trade-off between the memory footprint and performance speed.

LIMITATIONS:
As the authors are using pre-trained VITs to extract the features. All the limitations of VITs carry forward to this approach.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-09

comments: SUMMARY:
The paper essentially shows, that a combination of classic techniques such as stream-data clustering, using an EM-algorithm and dynamic updates, in combination with strong pre-trained features, allows to achieve state-of-the-art performance on two standard video segmentation datasets (Davis 2017 and YouTube-VOS-2018).

STRENGTHS:
Clearly achieving good performance with a combination of classic techniques, chosen well for the task at hand, is interesting to report and know about. Performance is strong on DAVIS (probably the dataset where most parameters are set) and somewhat less impressive on YouTube - but still sota

WEAKNESSES:
There is essentially no novelty in the paper - except to propose a well-chosen combination of classic techniques.

QUESTIONS:
The paper is well written and the performance is interesting to know about. I suppose the key question to me is essentially, how surprising the results are? If I had been asked prior to reading this paper if a sensible combination of stream-based clustering with strong pretrained features can do the job on datasets such as DAVIS and YouTube (these are not particularly difficult after all)  - I would have clearly set yes. In that sense the surprise and novel insight to me for this work is rather marginal. Having said this, I would assume that not everyone has the same intuitions and thus it is imho worthwhile to report the paper in some form as a publication. However, in my personal judgement the current paper and contribution seems below the bar for such a high-profile venue such as NeurIPS.

LIMITATIONS:
ok for me
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09