# SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale Benchmark +
###### Abstract
While the vanilla baseline for SKVOS above, established in the earlier version of this work [25], primarily focused on fusing sketch information with the first video frame, it left room for improvement in leveraging temporal information across frames. In this work, we address two key limitations of the baseline:  
Figure 1: **(a)** Motivation: a comparison example between three different annotation types for the Semi-VOS task. (1) Mask reference. (2) Language reference. (3) Sketch reference (Ours). In this case, pixel-level mask annotation is time-consuming and resource intensive, while language annotation has clear ambiguities. Sketch, on the other hand, can balance the annotation cost, providing more intuitive visual and semantic information compared to text. (**b**) The previous vanilla SKVOS recipe relied solely on the interaction between the sketch and the first frame, using memory to assist in generating the target mask (the black curved arrows). (**c**) Our approach not only utilizes the sketch to establish associations across multiple frames (the orange curved arrows) but also leverages information from the previous frame to improve segmentation performance for targets of varying scales (the blue curved arrow). * Temporal Continuity: Existing methods typically rely on sparse memory frames for segmentation propagation, often sampling video frames at fixed intervals and storing them in a memory bank. While this approach reduces the computational and storage overhead to some extent, it overlooks a fundamental characteristic of videos: objects can undergo significant changes in appearance, position, shape, size, and viewpoint within very short intervals. Consequently, relying solely on distant frames for matching can introduce interference, leading to inaccurate segmentation results. To address this limitation, we propose a Temporal Relation Module (TRM) that explicitly leverages information from adjacent frames. The TRM establishes multi-scale relational connections and multi-level association fusion between consecutive frames, enabling the model to capture fine-grained temporal dynamics and ensuring smoother, more consistent segmentation over time. * Sketch-Video Fusion: Although our baseline method fuses sketch information with the first frame [25], it does not fully exploit the potential of sketches as temporal anchors. Sketches offer a significant advantage over textual descriptions in video object segmentation (VOS): while language relies on general semantics, sketches encode precise visual cues, providing both high-level semantic information and fine-grained visual details. This dual capability is particularly advantageous for instance-level tracking, where maintaining accurate object identities over time requires detailed segmentation. As demonstrated by [PERSON] et al. [28], sketches can effectively guide detection at the object level, instance level, and even at the part level, making them a powerful modality for VOS. To fully harness this potential, we introduce Sketch-Anchored Temporal Contrastive Learning (STCL), which uses sketches as anchors to establish consistency in object representations across multiple frames. By aligning object features temporally, STCL enhances the model's ability to handle complex object dynamics, such as changes in appearance, scale, and motion. This mechanism not only improves object-level segmentation but also strengthens instance-level tracking, ensuring robust and accurate segmentation results over time. Our contributions are summarized as follows:
* Temporal Relation Module (TRM): We propose a novel module that captures hierarchical spatiotemporal features by strengthening the correlation between adjacent frames. This ensures smoother transitions and more consistent segmentation results, particularly in videos with dynamic object movements. * Sketch-Anchored Temporal Contrastive Learning (STCL): We introduce a contrastive learning framework that uses sketches as anchors to align object representations across multiple frames.