SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our method's efficient information propagation from prior frames results in smoother transitions and improves the temporal coherence of the segmentation. Retaining (weighted) sufficient statistics from previous frames helps with the detection of reappearing, previously-occluded, objects or parts. Importantly, our memory consumption is determined by the number of clusters in the model, not  $N$ , the (much larger) feature map size. This drastically reduces memory usage during inference (see Fig. 3) and enables us to obtain a compact representation. Moreover, this drastic memory reduction (in comparison to methods which rely on saving history of dense representations) enables us to use higher-resolution features than the competitors (see, e.g., Fig. 4). Importantly, as  $\mathcal{K}(t,t^{\prime})$  decreases when  $|t - t^{\prime}|$  increases and eventually decays to zero, in practice we store the sufficient statistics from time  $t^{\prime}$  only until time  $t^{\prime} + 15$  and then discard it.
3.4 Pixel-to-object assignment
Given the estimated models from time  $t$ , in the subsequent frame we extract features  $X^{t + 1}$  and assign labels to each pixel based on the mixture (from time  $t$ ) that has the highest probability for any of its components. By employing the maximum-a-posteriori (MAP) rule, we determine the component
Figure 4: Benefits of high-resolution. Each row presents an original video frame (left) and our results obtained when using features at either low resolution (middle) or high resolution (right). Note the finer details achieved at high resolution. While it is fairly obvious that higher resolution is better, our point here is that competing methods (e.g., [3, 24]) usually cannot handle such a resolution due to the increased computational and memory demands. This is unlike our method which has a low-memory footprint. Please zoom in for optimal viewing.
with the highest probability for each pixel, and then assign the pixel to the object associated with that component. In other words, for every  $o \in \{1, \dots, N_{\mathrm{obj}} + 1\}$  we compute
$$
p _ {i, o} ^ {t + 1} \triangleq \Pr (\text {p i x e l} i \text {a t t i m e} t + 1 \text {b e l o n g s t o o b j e c t} o) \propto \max  _ {s: s \in 1, \dots , S} \left(\max  _ {k: k \in 1, \dots , K _ {s}} p \left(\boldsymbol {x} _ {i} ^ {t + 1} \mid \theta_ {o, s, k} ^ {t}\right)\right) \tag {9}
$$
and then assign the pixel to object according to  $\arg \max_{o: o \in \{1, \dots, N_{\mathrm{obj}} + 1\}} p_{i,o}^{t + 1}$ . Next, we generate the (non-overlapping) masks for time  $t + 1$ ,  $(M^{t + 1})_{o = 1}^{N_{\mathrm{obj}} + 1}$  via
$$
M _ {o} ^ {t + 1} (i) = \left\{ \begin{array}{l l} 1, & \text {i f} o = \arg \max  _ {o ^ {\prime}} p _ {i, o} ^ {t + 1} \\ 0, & \text {o t h e r w i s e} \end{array} \right. \tag {10}
$$
where  $M_o^{t + 1}(i)$  denotes pixel  $i$  of  $M_{o'}^{t + 1}$ .
3.5 Refining the segmentation using outlier rejection and a convolutional CRF
Recall that the initial mask is the sole ground truth reference that the method has access to. Thus, as the video sequence unfolds, the presence of outliers poses a substantial challenge to the accuracy of the clustering process. These outliers, which stem from factors such as occlusions, illumination changes, or noise, might hurt the integrity of existing clusters or lead to confusing them with unrelated clusters. Our approach for mitigating such potential sources of error incorporates two strategies: outlier rejection and a refinement using a convolutional CRF.
In the initial phase, we exclude points from the parameter-estimation step when the pixel-to-object assignment is inconclusive. Specifically, if the score of a pixel's second-best match closely rivals that of its best match, we consider that pixel an outlier. We then employ the Pixel-Adaptive Convolutional CRF (PAC-CRF) [37] to further refine the segmentation. PAC-CRF, with its fixed window connections around each pixel, effectively filters noise while preserving segmentation coherence. It bolsters our outlier handling strategy by enabling the exclusion of points where the PAC-CRF prediction conflicts with our initial prediction. After several EM steps (where the M step excludes the outliers), we apply PAC-CRF again, leveraging the source image to improve the pixel-wise accuracy and the segmentation quality. Furthermore, we leverage the learnability of the PAC-CRF's pairwise potentials' weights and optimize them using Focal-Loss [26, 30]. This optimization, carried out over a few gradient steps with the ground-truth annotation from the first frame, ensures that our method adapts well to the given data, leading to improved performance and more accurate segmentation maps (Fig. 2).
In this section, we present the performance of the proposed method on the DAVIS-2017 and YouTube-VOS datasets. A subsequent subsection delves into our adeptness at harnessing features from deeper
Figure 5: Qualitative examples on DAVIS-2017, comparing our results (third row) with one of the key competitors (second row).
Table 1: Results on DAVIS-2017 validation in terms of the mask  $(\mathcal{J})$  and boundary  $(\mathcal{F})$  accuracy (IoU). The subscript  $[\cdot ]_r$  denotes the recall of the metric, while  $[\cdot ]_m$  signifies the mean.
Method	Training Video Dataset (Duration)	J&Fm	Jm	Jr	Fm	Fr
Unsupervised
Colorization[42]	Kinetics (800 hours)	34.0	34.6	34.1	32.7	26.8
CorrFlow[23]	OxUvA (14 hours)	50.3	48.4	53.2	52.2	56.0
TimeCycle[44]	VLOG (344 hours)	48.7	46.4	50.0	50.0	48.0
UVC[25]	Kinetics (800 hours)	60.9	59.3	68.8	62.7	70.9
MuG[27]	OxUvA (14 hours)	54.3	52.6	57.4	56.1	58.1
MAST[22]	Youtube-VOS (5.58 hours)	65.5	63.3	73.2	67.6	77.7
CRW[18]	Kinetics (800 hours)	68.3	65.5	78.6	71.0	82.9
ContrastCorr[43]	TrackingNet (140 hours)	63.0	60.5	-	65.5	-
VFS[45]	Kinetics (800 hours)	66.7	64.0	-	69.4	-
JSTG[49]	Kinetics (800 hours)	68.7	65.8	77.7	71.6	84.3
DULVS[3]	TrackingNet (140 hours)	69.4	67.1	80.9	71.7	84.8
CLTC[19]	Youtube-VOS (5.58 hours)	70.3	67.9	78.2	72.6	83.7
DINO[7]	No training video is required	71.4	67.9	80.7	74.9	87.8
LIIR[24]	Youtube-VOS (5.58 hours)	72.1	69.7	81.4	74.5	85.9
INO[33]	Kinetics (833 hours)	72.5	68.7	82.0	76.3	89.0
Ours	No training video is required	76.3	73.8	85.6	78.7	89.2
Supervised
OSVOS[6]	DAVIS-2016 (0.5 hours)	60.3	56.6	63.8	63.9	73.8
TVOS[48]	DAVIS-2017 + Youtube-VOS (6 hours)	72.6	69.9	-	74.7	-
STM[31]	DAVIS-2017 + Youtube-VOS (6 hours)	81.7	79.2	-	84.3	-
networks and superior resolutions. Video demonstrations can be found here. Details regarding our implementation are available in Appendix B.
4.1 Evaluation on DAVIS-2017 and YouTube-VOS
We report the results on two widely-used SVOS benchmarks: YouTube-VOS [46] and DAVIS 2017 [34]. Commonly used metrics for assessing VOS performance include the Jaccard's index (J) [14], which calculates the intersection-over-union (IoU) of the object mask, and the F-measure (F) [29], which assesses the contour accuracy. These metrics provide comprehensive insight into the segmentation performance by evaluating both the object area and contour precision. We report the mean IoU for mask and contour, denoted as  $\mathcal{I}_m$  and  $\mathcal{F}_m$ , as well as the recall  $\mathcal{I}_r$  and  $\mathcal{F}_r$ , which are
Table 2: DAVIS-2017 (train)
Method	J&Fm	Jm	Jr	Fm	Fr
DULVS	68.0	64.4	75.4	71.7	81.3
LIIR	63.9	60.3	69.7	67.5	75.4
INO	67.7	62.4	71.7	73.0	84.1
Ours	73.9	80.0	77.9	77.9	88.1
Table 3: DAVIS-2017 (test-dev)
Method	J&Fm	Jm	Jr	Fm	Fr
DULVS	58.3	54.5	64.7	62.0	70.1
LIIR	50.3	47.6	54.7	52.9	60.2
INO	61.9	56.5	63.8	67.3	77.6
Ours	66.5	63.3	74.5	69.6	80.1
calculated using an IoU threshold of 0.5. The metric  $\mathcal{J} \& \mathcal{F}_m$  represents the average of  $\mathcal{J}_m$  and  $\mathcal{F}_m$ . For a fair comparison, we use the same video resolution as the competing methods, despite the fact that our method can process higher resolutions than them.
DAVIS 2017. The DAVIS-2017 dataset [34], featuring high-quality videos with diverse object appearances and challenging motion patterns, is a widely-accepted SVOS benchmark. On the DAVIS-2017 validation set (Table 1), our method demonstrated top-tier scores in the Jaccard index, F-measure, and  $\mathcal{J}\& \mathcal{F}_m$  metrics. Specifically, our results include a  $\mathcal{J}\& \mathcal{F}_m$  score of 76.3,  $\mathcal{J}_m$  of 73.8,  $\mathcal{J}_r$  of 85.6,  $\mathcal{F}_m$  of 78.7, and  $\mathcal{F}_r$  of 89.2. These results clearly outperform the other methods (see also Fig. 5). We also evaluated our method on the DAVIS-2017 training and test-dev sets, collectively constituting 90 additional sequences. These sets allowed for a broader performance assessment, with the training set offering an extra dimension of evaluation since it is not included in the training data of the methods we compared with. Results from the test-dev set, officially validated by the evaluation server, further demonstrate our method's efficacy (Table 2 & Table 3). Qualitative results from these extended evaluations are available in Appendix C.
YouTube-VOS. The YouTube-VOS dataset [46] serves as a large-scale benchmark for SVOS, introducing a variety of complexities such as occlusions, scale changes, and motion blur. Unique to this dataset is the division of object classes between the training and validation sets, where only a subset is shared, allowing the benchmark to distinguish between 'seen' and 'unseen' categories. Our method's scores, obtained through the official evaluation server, showcase its adaptability in navigating these complex situations, surpassing the performance of several state-of-the-art approaches (Table 6).
4.2 Performance and feature analysis
Our method adeptly processes high-resolution features, distinguishing it from other models constrained by memory, as illustrated in Table 4. While merely adopting deeper backbones does not guarantee superior outcomes [45], our model showcases its capability to harness the strengths of DINOv2's [32] features. To counteract the reduced spatial information inherent in DINOv2 due to its larger patch size, We concatenated the DINOv2 features with the XCiT features we had previously used. This approach ensures a rich and comprehensive representation of the visual scene. As demonstrated in Table 5, our method effectively capitalizes on the enhanced features from both deeper backbones and higher spatial resolutions.
Table 4: FPS across resolutions. Comparison on Tesla V100-32GB, excluding feature extraction. OOM stands for "Out Of Memory".
Resolution	Feat. Size	Ours	[33]	[3]
120 × 210	384	8.25	0.13	0.14
768	6.50	0.11	0.12
1152	5.25	0.10	0.11
240 × 420	384	1.55	OOM	OOM
768	1.28	OOM	OOM
1152	1.10	OOM	OOM
Table 5: DAVIS-2017 (val) with DINOv2 features. Fusion with our features elevates performance. "×2" denotes using features with twice the spatial resolution. DINOv2 "S" and "B" stand for "small" and "base", respectively.
Method	J&Fm	Jm	Fm
+ DINO2-S	0.776	0.754	0.798
+ DINO2-B	0.782	0.760	0.804
×2 + DINO2-S	0.784	0.762	0.806
×2 + DINO2-B	0.803	0.780	0.825
We performed an ablation study (Fig. 6) to analyze the influence of different parts of the method on the performance. The combination of outlier detection and segmentation refinement showed a
Table 6: Results on YouTube-VOS 2018 (val)
Method	Seen	Unseen	Mean
Jm	Fm	Jm	Fm
Unsupervised
Colorize[42]	43.1	38.6	36.6	37.4	38.9
CorrFlow[23]	50.6	46.6	43.8	45.6	46.6
MAST[22]	63.9	64.9	60.3	67.7	64.2
DULVS[3]	70.2	71.9	66.5	74.8	70.6
CLTC[19]	66.2	67.9	63.2	71.7	67.3
LIIR[24]	67.9	69.7	65.7	73.8	69.3
INO[33]	70.7	73.2	65.6	75.6	71.3
Ours	72.0	74.2	66.2	73.7	71.5
Supervised
OSVOS[6]	59.8	60.5	54.2	60.7	58.8
TVOS[48]	67.1	69.4	56.5	63.0	71.6
STM[31]	79.7	84.2	72.8	80.9	79.4
Figure 6: Ablation study of model parameters. We report the  $\mathcal{J}\& \mathcal{F}_m$  score evaluated on DAVIS-2017 val. Our baseline configuration (the centered one) is:  $S = 10, \lambda = 0.33$ ,  $w_{\rho} = 15$ .
synergistic improvement in performance, with each contributing significantly when implemented separately. The necessity of our streaming mechanism was underscored when we observed a notable performance drop while solely relying on the previous frame for initializing mixture parameters  $(\lambda = \infty)$ . Furthermore, examining the number of mixtures per object, we notice a substantial performance boost when increasing the number of mixtures from one to two, underscoring the importance of multi-scale object representations. However, the added benefit started to decay with further increase, leading us to settle on  $S = 10$  mixtures in our reported results. Concerning spatial components, removing positional embeddings (i.e.,  $w_{\rho} = 0$ ) resulted in a major performance drop, confirming their essential role. Additionally, the introduction of the indicator function led to an additional positive effect on performance.
We proposed a novel SVOS approach that achieves state-of-the-art results on challenging benchmarks. Our memory-conscious strategy also opens possibilities for processing high-resolution videos, suggesting a valuable avenue for future video understanding research.
The proposed method faces difficulties in two scenarios (see Appendix D for examples). The first is related to re-identification challenges, where occlusion between multiple similar instances of the same object type (e.g., two camels) makes it challenging to assign correct labels to each instance. This limitation stems from the method's lack of explicit object identity representation and modeling of temporal dynamics, which could help disambiguate overlapping objects. The second scenario occurs in the presence of a strong motion blur. The method is sensitive to such a blur, likely because the ViT features were trained on still images, not videos. This limitation becomes more pronounced when motion blur persists across consecutive frames, as then the error propagation is more significant.
This work was supported by: the BGU CS Lynn and William Frankel Center; the Israeli Council for Higher Education via the BGU Data Science Research Center; Israel Science Foundation Personal Grant #360/21.
[46] Ning Xu, Linjie Yang, Yuchen Fan, Jianchao Yang, Dingcheng Yue, Yuchen Liang, Brian Price, Scott Cohen, and Thomas Huang. Youtube-vos: Sequence-to-sequence video object segmentation. In ECCV, 2018. 3, 8, 9
[47] Linjie Yang, Yanran Wang, Xuehan Xiong, Jianchao Yang, and Aggelos K Katsaggelos. Efficient video object segmentation via network modulation. In CVPR, 2018. 3
[48] Yizhuo Zhang, Zhirong Wu, Houwen Peng, and Stephen Lin. A transductive approach for video object segmentation. In CVPR, 2020. 8, 10
[49] Zixu Zhao, Yueming Jin, and Pheng-Ann Heng. Modelling neighbor relation in joint space-time graph for video correspondence learning. In ICCV, 2021. 8
[50] Di Zhong and Shih-Fu Chang. An integrated approach for content-based video object segmentation and retrieval. IEEE Transactions on Circuits and Systems for Video Technology, 1999. 3
[51] Tianfei Zhou, Jianwu Li, Xueyi Li, and Ling Shao. Target-aware object discovery and association for unsupervised video multi-object segmentation. In CVPR, 2021. 3