# SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale Benchmark +
## 5 Experiments
### Sketch vs. Other References  
5.2.1. Compare vs. Other Methods  
We compare our Sketch-VOS with the existing VOS works incorporated with different references that appear in the literature. Furthermore, we conduct comprehensive comparisons between several references: language, sketch, mask, and interactive annotations including cross, circle, scribble, and object contour. Competitors  
VOSwL [14] predicts masks through a two-stage process involving localization and segmentation. URVOS [15] follows a similar approach but concatenates visual and linguistic features before passing them through the cross-modal attention mechanism. YOFO [30] employs meta-learning to transfer object information. HINet [29] integrates language and visual features through a hierarchical fusion process. MLRL [31] combines linguistic features with video, frame, and object features at multiple levels. LBDT [32] uses language to transfer both spatial and temporal visual features. MTTR [33], ReferFormer [34], SgMg [35] and LoSh [36] apply transformer-based detectors for both localization and mask segmentation. STM [44] introduces a spatial-temporal memory bank, which was later enhanced by STCN [45] through a more efficient affinity module. Comparison  
Table 3 summarizes the performance of various video segmentation methods on the YouTube-VOS, DAVIS17, and DAVIS16 datasets. The proposed model demonstrates competitive results across all three benchmarks, outperforming existing methods in several key areas. Notably, our method achieves significant improvements over all language-based approaches, with a gain of approximately 10.1%, 9.5%, and 10.9% in the \(\mathcal{J}\&\mathcal{F}\), \(\mathcal{J}\), and \(\mathcal{F}\) metrics, respectively, compared to LoSh [36]. For a fair comparison with mask-based methods, we retrain STM [44] and STCN [45] without using any additional image datasets. Although our model outperforms STM, it slightly lags behind STCN. Compared to our Vanilla Baseline [25], we achieveimprovements of 1.9, 1.5, and 2.4 on three metrics, respectively, on the Youtube-VOS dataset. We also evaluate our model on the DAVIS17 validation set, which presents longer videos and more stringent annotations, making it a more challenging benchmark for video object segmentation. Despite these challenges, our model shows strong generalization and surpasses all language-based methods without the use of extra image datasets. In comparison to mask-based methods, our model outperforms STM by 2.9 points but is slightly behind STCN by 2.0 points in terms of \(\mathcal{J}\&\mathcal{F}\). On DAVIS16, our model is evaluated against VOSwL [14] and HINet [29], both mask-based methods. Similarly, our method outperforms the Vanilla Baseline [25] by 2.2 and 0.7 on the \(\mathcal{J}\&\mathcal{F}\) metric on the DAVIS17 and DAVIS16 datasets, respectively. Despite the smaller scale of DAVIS16, our sketch-based method delivers performance on par with these two approaches. Importantly, we conduct the evaluation in a zero-shot setting, testing the model trained on YouTube-VOS without using any DAVIS data for training. Even under these conditions, our model achieves satisfactory results, further demonstrating its robustness and effectiveness across different datasets. Visualizations  
We visualize the generated attention maps by querying the first frame from various videos in the Sketch-YouTube-VOS validation set using sketch references. As shown in Figure 8, the VOS model effectively focuses on the areas indicated by the sketch references, regardless of the categories or scales of the target objects. Even objects that are too small to be accurately described by language can be precisely localized using sketches. Figure 9 illustrates the segmentation results on the Sketch-YouTube-VOS dataset. Given a sketch reference, our model successfully segments the object mask in each frame.