SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Roy Uziel
Ben-Gurion University of the Negev, Israel
uzielr@post.bgu.ac.il
Or Dinari
dinari@post.bgu.ac.il
Oren Freifeld
orenfr@cs.bgu.ac.il
In the task of semi-supervised video object segmentation, the input is the binary mask of an object in the first frame, and the desired output consists of the corresponding masks of that object in the subsequent frames. Existing leading solutions have two main drawbacks: 1) an expensive and typically-supervised training on videos; 2) a large memory footprint during inference. Here we present a training-free solution, with a low-memory footprint, that yields state-of-the-art results. The proposed method combines pre-trained deep learning-based features (trained on still images) with more classical methods for streaming-data clustering. Designed to adapt to temporal concept drifts and generalize to diverse video content without relying on annotated images or videos, the method eliminates the need for additional training or fine-tuning, ensuring fast inference and immediate applicability to new videos. Concretely, we represent an object via a dynamic ensemble of temporally- and spatially-coherent mixtures over a representation built from pre-trained ViT features and positional embeddings. A convolutional conditional random field further improves spatial coherence and helps reject outliers. We demonstrate the efficacy of the method on key benchmarks: the DAVIS-2017 and YouTube-VOS 2018 validation datasets. Moreover, by the virtue of the low-memory footprint of the compact cluster-based representation, the method scales gracefully to high-resolution ViT features. Our code is available at https://github.com/BGU-CS-VIL/Training-Free-VOS.
Video Object Segmentation (VOS), a key computer-vision task, aims to distinguish objects of interest from the background across a sequence of video frames. In Semi-supervised VOS (SVOS), the topic of this paper, the user provides the mask of the object in the first frame (while in the case of unsupervised VOS, no mask is given), and the task is to propagate that mask to the next frames. Prior to the revolution of Deep Learning (DL), SVOS relied heavily on traditional techniques like color-based segmentation, point trajectories, and optical flow. However, the emergence of DL-based methods has significantly outpaced those traditional techniques. Thus, today DL-based methods dominate the field of SVOS. Note that according to the standard terminology in the area of VOS, the term "semi-supervised" in the name SVOS refers to the nature of the task (that is, the fact that a user provides the mask in the first frame), not the nature of the solution. In other words, solutions to the SVOS task can be either supervised, semi-supervised, or unsupervised. For example, a supervised
Figure 1: Method overview. Starting from the initial frame, the features of the object are extracted and modeled using an ensemble of von Mises-Fisher (vMF) mixtures, where each mixture has a different number of components. The higher the number of components, the finer the scale that the mixture captures. This multi-scale representation lets us capture phenomena such as drastic changes in the size/resolution of the object. The background is modeled using another such mixture ensemble. Next, all of the mixtures are updated in a streaming and memory-efficient manner at each subsequent frame. A maximum-a-posteriori rule is applied over all of the vMF components in the ensemble to generate a preliminary segmentation mask, which is then further refined by a convolutional Conditional Random Field. The iterative process progresses through the frames.
DL solution to SVOS typically requires multiple labeled videos to train on. In contrast, the approach proposed in this paper requires neither supervision nor learning on videos.
The main strategy employed by DL-based SVOS methods revolves around pixel-level matching to establish inter-frame correspondence. This is achieved by creating pixel-to-pixel connections between the current frame and the preceding ones. That strategy, however, often has two main limitations: 1) the need for an extensive training on large-scale (and typically annotated) video datasets; 2) a significant memory footprint due to the storage requirements of (dense representations of) previous frames for pixel matching. Addressing those prevalent challenges, we propose an unsupervised, adaptive, and compact SVOS method. By combining DL and classical techniques and capitalizing on the widespread practice of representing features as points on a hypersphere, the proposed model incorporates an ensemble of temporally- and spatially-coherent von Mises-Fisher (vMF) mixtures. The mixtures are updated dynamically in a streaming manner, together with a convolutional implementation of a Conditional Random Field (CRF).
Our method uses features from a pre-trained unsupervised deep neural network – trained on still images, not videos – but does not require training by itself as it only performs inference. The method fits multiple vMF mixtures to each object of interest (and to the background) in the initial frame. The fact that we employ multi-scale modeling (where each mixture has its own number of components) lets us represent the object at multiple levels of granularity. In subsequent frames, pixels are assigned to the most probable component (across all of the components in all of the mixtures in all of the objects). The mixture ensembles themselves are updated dynamically while taking into account temporal and spatial coherence. Following established stream clustering techniques [13], we apply a weighted window with a kernel function that reduces the weight of older frames, mitigating memory consumption and improving computational efficiency without storing extensive history of dense representations of past frames. Our model adapts to concept drifts in the objects of interest or the background. We improve spatial consistency by constraining pixel-component membership based on location and refine results using a Pixel-Adaptive Convolutional CRF (PAC-CRF) [37], fostering pixel consistency and awareness of outliers. Together with outlier-rejection mechanisms, the method mitigates error propagation across frames.
To summarize, our key contributions are: 1) We demonstrate that a synergy between classical and DL techniques can provide an effective solution for SVOS that achieves state-of-the-art results on the DAVIS-2017 and YouTube-VOS 2018 datasets. 2) By leveraging unsupervised pre-trained image-based features, our method eliminates the need for further training, and in particular no training on videos is required (unlike most competitors which require large-scale video datasets for training). 3) Our method has a low-memory footprint by storing only cluster-level information, as opposed to dense past-frame representations. Thus, the method does not only scale better than competing ones but also enables handling high-resolution features for higher-quality segmentation.
The evolution of SVOS methods. From older SVOS methods based on classic vision techniques [11, 9, 21, 50, 10, 5], the focus has shifted to DL-based SVOS methods, also known as one-shot VOS. Early semi-supervised methods focused on online fine-tuning of a pre-trained network, exemplified by OSVOS [6]. In fact, OSVOS pioneered the online fine-tuning of pre-trained networks in SVOS. However, OSVOS is prone to overfitting and fails to adapt to gradual concept drifts in the objects of interest. To address these issues, numerous extensions and refinements to OSVOS were proposed. OnAVOS [41] introduced additional fine-tuning with high-confidence frames in order to handle gradual concept drifts, albeit at the cost of increased computational complexity. Maninis et al. [28] integrated semantic information into the predictions via Mask R-CNN [17], thereby reducing the dependency on temporal smoothness. In an innovative departure from common practice, BubbleNets [16] proposed selecting the most suitable frame for annotation, challenging the conventional practice of annotating the first frame. Other works addressed different aspects of the OSVOS algorithm. For example, OSNM [47] utilized network modulation for more efficient fine-tuning, while A-Game [20] employed a Gaussian Mixture Model to better capture both the foreground and the background. Other noteworthy and relevant works include, but are not limited to, FRTM [35], LWL [4], and TAODA [51].
The emergence of matching-based methods. Recently, pixel matching methods have gained prominence due to their superior speed and results [15]. These methods, which either implicitly predict inter-frame similarities or explicitly match features between pixel pairs across frames, have been adopted by recent SVOS methods [22, 18, 3], leading to state-of-the-art performance.
Unresolved issues and evolving developments in SVOS. Constructing discriminative feature embeddings with temporal consistency is crucial for reliable correspondence. This is typically achieved by training backbone networks on large-scale video datasets such as OxUvA [39], YouTube-VOS [46], TrackingNet [46], and Kinetics-400 [8]. Most existing methods store dense feature representations of multiple previous frames and perform spatial correlation to propagate information across frames. However, this significantly increases memory consumption, while reducing the number of reference frames degrades the performance. Hence, finding a memory-efficient way to propagate the information is of paramount importance for achieving accurate and robust segmentation results.
We start with the task definition. In the SVOS task, the user provides (only for the first frame) a binary mask for each object of interest. Let  $(M_o^1)_{o=1}^{N_{\mathrm{obj}}}$  denote the masks where  $N_{\mathrm{obj}}$  is the number of objects. Those (non-overlapping)  $N_{\mathrm{obj}}$  masks imply another mask,  $M_{N_{\mathrm{obj}}+1}^1$ , which represents the background at time  $t = 1$  (i.e., the first frame). That mask is formed by taking the binary complement of the union of  $(M_o^1)_{o=1}^{N_{\mathrm{obj}}}$ . We will henceforth treat the background as just another object unless stated otherwise. For each time  $t = 2,3,\ldots,T$ , where  $T$  is the total number of frames, the goal is to estimate the corresponding masks, denoted by  $(M_o^t)_{o=1}^{N_{\mathrm{obj}}+1}$ .
Our unsupervised method for the SVOS task, depicted in Fig. 1, can be summarized as follows. We extract features from video frames (§ 3.1) and model the within-object feature distribution via a multi-scale dynamic ensemble of vMF mixtures (§ 3.2). We use a streaming-data Expectation-Maximization algorithm to update the parameters of the mixtures (§ 3.3). We assign pixels to objects based on the component-wise posterior probabilities in the entire ensemble of each object (§ 3.4). For robustness, we disregard outliers during the parameter-estimation steps, and further
improve the results by incorporating a Pixel-Adaptive Convolutional CRF (§ 3.5) for additional outlier management and segmentation refinement. For examples of qualitative results see, e.g., Fig. 2.
3.1 Feature extraction
The feature representation we use consists of ViT features [1] and positional embeddings [40]. For the ViT features, we use the publicly-available pre-trained weights of XCiT (Cross-Covariance Image Transformer) [1], that had been trained in a self-supervised manner as outlined in [7].
Let  $N_{\mathrm{pixels}}$  be the number of pixels in each frame. Let  $\widetilde{X}^t = (\widetilde{\boldsymbol{x}}_i^t)_{i=1}^N$  denote the ViT features extracted from  $I_t$  (the video frame at time  $t$ ), where  $N$  is the number of features. For example, if the ViT features are extracted using non-overlapping  $8 \times 8$  patches, then  $N = N_{\mathrm{pixels}} / 64$ . The XCiT variant, like its DINO-ViT counterpart, excels in extracting rich and discriminative features from images [2], but with an added advantage of better handling high-resolution images. In our case, we employ an XCiT model that yields a feature vector  $\widetilde{\boldsymbol{x}}_i^t \in \mathbb{R}^{384}$  for each image patch  $i$ . To capture the spatial relationships between neighboring pixels, we extend each ViT feature,  $\widetilde{\boldsymbol{x}}_i^t$ , with its corresponding rotary positional embedding,  $\rho(i)$ , as per the methodology detailed in Su et al. [38]. This embedding represents the 2D spatial location of the ViT feature in a 64-dimensional space. Consequently, we form the extended feature vector  $\boldsymbol{x}_i^t = (\widetilde{\boldsymbol{x}} i^t, w_\rho \cdot \rho(i))$  (where  $w_\rho > 0$  is the weight of the positional embeddings), normalized such that  $\| \boldsymbol{x}_i^t \|_{\ell_2} = 1$ . We denote by  $X^t = (\boldsymbol{x}_i^t)_{i=1}^N$  the collection of all these unit-length (extended) features at time  $t$ . Finally, given the initial masks,  $(M_o^{t})_{o=1}^{N_{\mathrm{obj}}+1}$ , we partition  $X^1$ , the features at  $t=1$ , into  $(X_o^1)_{o=1}^{N_{\mathrm{obj}}+1}$  such that, for each  $o \in (1, \ldots, N_{\mathrm{obj}}+1)$ ,  $X_o^1$  represents the features that correspond to  $M_o^1$ .
3.2 Modeling the within-object feature distribution via a multi-scale vMF mixture
In our pursuit of capturing intricate patterns in high-dimensional data while ensuring computational efficiency, we turn to vMF mixtures; namely, a mixture model whose each component is a vMF distribution (defined below). These mixtures, known for their simple structure that depends solely on mean directions and concentration parameters, are highly efficient computationally. They are also adept at modeling data on a unit hypersphere, which is a prevalent form of representation for deep features. Since often the feature distribution of an object is not only complex but also dependent on its size (e.g., when the object-to-camera distance increases, the object resolution decreases and fine details tend to disappear, which in term affects the distribution), we advocate for a multi-scale approach, using a mixture ensemble. For each object, we employ  $S$  independent vMF mixture models, each corresponding to a different scale or granularity. This way, we capture details at multiple scales.
The vMF distribution is a probability distribution over  $\mathbb{S}^{d - 1}\triangleq \{\pmb {x}:\pmb {x}\in \mathbb{R}^d,\| \pmb {x}\|_{\ell_2} = 1\}$ , the unit sphere in  $\mathbb{R}^d$ . Its probability density function (pdf), evaluated at  $\pmb {x}\in \mathbb{S}^{d - 1}$ , is
$$
\operatorname {v M F} (\boldsymbol {x}; \boldsymbol {\mu}, \tau) \triangleq C _ {d} (\tau) \exp \left(\tau \boldsymbol {\mu} ^ {\top} \boldsymbol {x}\right) \tag {1}
$$
where  $\pmb{\mu} \in \mathbb{S}^{d-1}$  is the mean and  $\tau > 0$  is the concentration parameter. The normalizer  $C_d(\tau)$  depends only on  $\tau$  and has a closed form that appears in Appendix A.
Let  $s \in \{1, \dots, S\}$ . Mixture  $s$  for object  $o$  at time  $t$  is defined as follows:
$$
\sum_ {k = 1} ^ {K _ {s}} \alpha_ {o, s, k} ^ {t} \mathrm {v M F} (x; \boldsymbol {\mu} _ {o, s, k} ^ {t}, \tau_ {o, s, k} ^ {t}). \tag {2}
$$
Here,  $\alpha_{o,s,k}^{t}$  represents the mixing proportions while  $\pmb{\mu}_{o,s,k}^{t}$  and  $\tau_{o,s,k}^{t}$  are the mean and concentration parameter, respectively, of component  $k$  at time  $t$ . The predefined number of components,  $K_{s}$ , is different for every  $s$  and represents the scale of the mixture. The full set of parameters describing object  $o$  at time  $t$  is  $\theta_{o}^{t} = \left((\alpha_{o,s,k}^{t},\theta_{o,s,k}^{t})_{k = 1}^{K_{s}}\right)_{s = 1}^{S}$ , where  $\theta_{o,s,k}^{t} = (\pmb{\mu}_{o,s,k}^{t},\tau_{o,s,k}^{t})$ .
3.3 Parameter estimation via Expectation-Maximization
We adapt the widely-used Expectation-Maximization (EM) framework [12] to estimate the parameters of our vMF mixtures in a streaming-data manner, tailoring it specifically for VOS. The algorithm is applied independently and in parallel to each mixture.
Our EM algorithm consists of two stages. The Expectation step (E-step) calculates the posterior probabilities of component membership (i.e., feature-to-component assignment), integrating spatial coherence to improve segmentation accuracy. In the Maximization step (M-step), we update the mixture parameters to maximize the expected log-likelihood, introducing modifications to handle the streaming nature of video data and manage memory efficiently.
3.3.1 E-step: calculating posterior probability with spatial coherence
In the E-step we incorporate spatial coherence, a crucial factor in VOS. Let us assume that, at time  $t$ , pixel  $i$  was assigned to object  $o$  (we will later explain how pixels are assigned to objects). For each mixture  $s$  of the  $S$  mixtures associated with object  $o$ , we do the following. For every  $k = 1, \dots, K_s$ , let  $q_{o,s,k,i}^t$  be the posterior probability that, at time  $t$ , pixel  $i$  belongs to component  $k$  (of mixture  $s$ ) given the observed feature  $\pmb{x}_i^t$  and the current estimate of  $(\alpha_{o,s,k}^t, \theta_{o,s,k}^t)_{k=1}^{K_s}$ . Let  $N_o^t$  be the number of nonzero pixels in  $M_o^t$  and, in a slight abuse of notation, let  $(\pmb{x}_i^t)_{i=1}^{N_o^t}$  denote the object features at time  $t$ . We modify the calculation of the posterior probability,  $q_{o,s,k,i}^t$ , by integrating a spatial constraint. This constraint is expressed as an indicator function that respects both the current parameter estimates and the spatial information inherited from the previous frame,  $t - 1$ . This approach ensures a smooth transition between frames and improves the spatial integrity of objects in the video sequence. Taken together, the modified posterior probability is
$$
q _ {o, s, k, i} ^ {t} = p \left(z _ {o, s, i} = k \mid \boldsymbol {x} _ {i} ^ {t}, \theta_ {o, s, k} ^ {t}\right) \cdot \mathcal {I} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {\mu} _ {o, s, k} ^ {t - 1}\right) \tag {3}
$$
where  $z_{o,s,i}$  is the assignment of  $\pmb{x}_i^t$  to a component in mixture  $s$  in object  $o$ ,  $p(z_{o,s,i} = k|\pmb{x}_i^t, \theta_{o,s,k}^t) \propto p(\pmb{x}_i^t|\theta_{o,s,k}^t) = \mathrm{vMF}(\pmb{x}_i^t; \pmb{\mu}_{o,s,k}^t, \tau_{o,s,k}^t)$ , and the binary-valued indicator function,
$$
\mathcal {I} \left(\boldsymbol {x} _ {i} ^ {t}, \boldsymbol {\mu} _ {o, s, k} ^ {t - 1}\right) = \left\{ \begin{array}{l l} 1 & \rho (i) ^ {\top} \rho \left(\boldsymbol {\mu} _ {o, s, k} ^ {t - 1}\right) > r _ {s} \\ 0, & \text {o t h e r w i s e} \end{array} , \right. \tag {4}
$$
returns 1 if  $\pmb{x}_i^t$  has a positional embedding that closely matches  $\rho(\pmb{\mu}_{o,s,k}^{t-1})$ , the positional part of the mean of cluster  $k$  from the previous frame. The quality of the match is measured in terms of cosine similarity. Note that the assignment also depends on a certain threshold,  $r_s$ , which in turn depends on the number of clusters,  $K_s$ . As  $K_s$  increases, a higher degree of similarity is required, reflecting the finer granularity of the mixture. If no such pixel exists, the function returns 0, effectively preventing the current pixel from being assigned to cluster  $k$ . This approach ensures spatial coherence.
3.3.2 M-step: parameter estimation
We first provide the details for the M-step while ignoring the streaming nature of the data, and then later, in § 3.3.3, explain how we account for the streaming. In the M-step, the parameters are updated to maximize the expected log-likelihood. We iteratively update a set of sufficient statistics that encapsulate the critical properties of the data and the model. Concretely, at time  $t$  we compute
$$
H _ {o, s, k} ^ {t} = \left(h _ {o, s, k} ^ {t, 1}, h _ {o, s, k} ^ {t, 2}\right) _ {k = 1} ^ {K _ {s}} \triangleq \left(\sum_ {i = 1} ^ {N _ {o} ^ {t}} q _ {o, s, k, i} ^ {t}, \sum_ {i = 1} ^ {N _ {o} ^ {t}} q _ {o, s, k, i} ^ {t} \boldsymbol {x} _ {i} ^ {t}\right) _ {k = 1} ^ {K _ {s}}. \tag {5}
$$
The M-step uses well-known closed-form solutions for the estimated parameters of the vMF distribution. In particular, to estimate the concentration parameters we employ an approximation scheme based on the continued fraction method, supplemented with a correction term:
$$
\boldsymbol {\mu} _ {o, s, k} ^ {t} = \frac {h _ {o , s , k} ^ {t , 2}}{\left\| h _ {o , s , k} ^ {t , 2} \right\|}, R _ {o, s, k} ^ {t} = \frac {\left\| h _ {o , s , k} ^ {t , 2} \right\|}{h _ {o , s , k} ^ {t , 1}}, \tau_ {o, s, k} ^ {t} \approx \frac {d \cdot R _ {o , s , k} ^ {t} - \left(R _ {o , s , k} ^ {t}\right) ^ {3}}{1 - \left(R _ {o , s , k} ^ {t}\right) ^ {2}}, \alpha_ {o, s, k} ^ {t} = \frac {h _ {o , s , k} ^ {t , 1}}{\sum_ {k = 1} ^ {K _ {s}} h _ {o , s , k} ^ {t , 1}}. \tag {6}
$$
To circumvent the computational expense of directly calculating the modified Bessel function (which appears in  $C_d(\cdot)$ ), we adopt an alternative method to compute the log-likelihood  $\log p(\pmb{x}_i^t|\theta_{o,s,k}^t)$  using the mean vector  $\pmb{\mu}_{o,s,k}^t$  and the concentration parameter  $\tau_{o,s,k}^t$ . Specifically, we leverage the fact that the normalizing constant  $C_d(\tau_{o,s,k}^t)$  depends solely on  $\tau_{o,s,k}^t$  and can be approximated [36] by a simpler expression (see Appendix A). This enables us to express the log-likelihood as follows:
$$
\log p \left(\boldsymbol {x} _ {i} ^ {t} \mid \theta_ {o, s, k} ^ {t}\right) = N _ {o} ^ {t} \tau_ {o, s, k} ^ {t} \left(\boldsymbol {\mu} _ {o, s, k} ^ {t}\right) ^ {\top} \bar {\boldsymbol {x}} + N _ {o} ^ {t} \log C _ {d} \left(\tau_ {o, s, k} ^ {t}\right), \tag {7}
$$
Figure 2: Qualitative examples.
Figure 3: Memory utilization. This graph illustrates the memory consumption as a function of the feature map size, comparing our method with scenarios using different numbers of reference frames (1, 5, and 21). Notably, our method requires less memory than even the scenario with a single reference frame. The feature dimension for this evaluation is set to 384.
where  $\top$  denotes transpose and  $\overline{x}$  denotes the (Euclidean) sample mean of the data points. We now address the dynamic characteristics of video data.
3.3.3 Memory-efficient dynamic updates and temporal coherence
In the dynamic environment of video sequences, objects may experience occlusion, reemergence, or gradual appearance changes due to, e.g., varying lighting conditions. Addressing these challenges, and inspired by [13], our method integrates temporal coherence into the clustering procedure using time-weighted sufficient statistics. We devise these statistics as a weighted sum of data values across multiple frames, with weights diminishing exponentially over time. By maintaining historical records of a limited length, and updating the time-weighted statistics for each frame, we efficiently manage space, memory, and time complexity. Data from the preceding frame is retained only if its (time-decaying) weight exceeds a certain threshold. Concretely, we employ a time-decaying weight  $\mathcal{K}(t,t^{\prime}) = 2^{\lambda (t - t^{\prime})}$  where the user-defined  $\lambda >0$  controls the correlation between sequential frames. As a result, older frames are assigned lower weights. Similarly to [13], we modify the sufficient statistics accordingly, and obtain a streaming-data variant of Eq. 5:
$$
H _ {o, s, k} ^ {t} = \left(h _ {o, s, k} ^ {t, 1}, h _ {o, s, k} ^ {t, 2}\right) _ {k = 1} ^ {K _ {s}} \triangleq \left(\sum_ {t ^ {\prime} = 1} ^ {t} \mathcal {K} (t, t ^ {\prime}) \sum_ {i = 1} ^ {N _ {o} ^ {t}} q _ {o, s, k, i} ^ {t ^ {\prime}}, \sum_ {t ^ {\prime} = 1} ^ {t} \mathcal {K} (t, t ^ {\prime}) \sum_ {i = 1} ^ {N _ {o} ^ {t}} q _ {o, s, k, i} ^ {t ^ {\prime}} \boldsymbol {x} _ {i} ^ {t}\right) _ {k = 1} ^ {K _ {s}}. \tag {8}
$$