# SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale Benchmark +
###### Abstract  
In this paper, we propose sketch-based video object segmentation (SKVOS), a novel task that segments objects consistently across video frames using human-drawn sketches as queries. Traditional reference-based methods, such as photo masks and language descriptions, are commonly used for segmentation. Photo masks provide high precision but are labor intensive, limiting scalability. While language descriptions are easy to provide, they often lack the specificity needed to distinguish visually similar objects within a frame. Despite their simplicity, sketches capture rich, fine-grained details of target objects and can be rapidly created, even by non-experts, making them an attractive alternative for segmentation tasks. We introduce a new approach that utilizes sketches as efficient and informative references for video object segmentation. To evaluate sketch-guided segmentation, we introduce a new benchmark consisting of three datasets: Sketch-DAVIS16, Sketch-DAVIS17, and Sketch-YouTube-VOS. Building on a memory-based framework for semi-supervised video object segmentation, we explore effective strategies for integrating sketch-based references. To ensure robust spatiotemporal coherence, we introduce two key innovations: the Temporal Relation Module and Sketch-Anchored Contrastive Learning. These modules enhance the model's ability to maintain consistency both across time and across different object instances. Our method is evaluated on the Sketch-VOS benchmark, demonstrating superior performance with overall improvements of 1.9%, 3.3%, and 2.0% over state-of-the-art methods on the Sketch-YouTube-VOS, Sketch-DAVIS 2016, and Sketch-DAVIS 2017 validation sets, respectively. Additionally, on the YouTube-VOS validation set, our method outperforms the leading language-based VOS approach by 10.1%. sketches; video object segmentation; sketch-based datasets +
Footnote †: journal: Applied Sciences  
1 School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China 1,*  
Footnote †: journal: Applied Sciences  
1 School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing 100876, China 1,*
However, traditional Semi-VOS methods rely heavily on pixel-level annotations for the first frame, which are time-consuming to create, especially for large-scale datasets like YouTube-VOS [13]. Recent efforts to reduce annotation burden have introduced referring VOS, which utilizes natural language expressions as references [14; 15]. Despite their potential, language-based references often struggle to convey fine-grained visual details, leading to ambiguities in object segmentation. For instance, as shown in Figure 1a, language descriptions may fail to distinguish between visually similar objects, whereas sketches provide precise shape and pose information that can resolve such ambiguities. Sketches, as a natural and efficient form of visual expression, offer a compelling alternative to language-based references. Compared to pixel-level masks, sketches are faster to create and encode both categorical and fine-grained information as demonstrated in sketch-based image retrieval [16; 17; 18; 19; 20; 21; 22] and editing tasks [23; 24]. Building on these advantages, we previously introduced sketch-based video object segmentation (SKVOS) [25], a novel task that leverages sketches as first-frame references for segmenting video objects. To support this task, we constructed three large-scale datasets by extending DAVIS16 [26], DAVIS17 [27], and YouTube-VOS [13] with freehand sketches, following established protocols from Ref-DAVIS [14] and Ref-YouTube [15]. We proposed a framework that begins by locating relevant regions with sketch query in the first frame and subsequently propagates segmentation masks across frames.