# SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale Benchmark +
## 5 Experiments
### Experimental Setup  
**Datasets:** Our experiments are conducted on the Sketch-Video dataset (Section 3), which consists of three subsets: Sketch-DAVIS16, Sketch-DAVIS17, and Sketch-YouTube-VOS. The specific splits are as follows: Sketch-DAVIS16 contains 30 training videos and 20 validation videos; Sketch-DAVIS17 includes 60 training videos and 30 validation videos; and Sketch-YouTube-VOS comprises 3412 training videos and 507 validation videos. The exact number of targets can be found in Table 2. Notably, the difference between DAVIS16/17, Ref-DAVIS16/17, and Sketch-DAVIS16/17 lies solely in the annotation format of the first frame (mask, text, and sketch, respectively); while the distinction between YouTube-VOS and Sketch-YouTube-VOS also lies in the annotation format of the first frame (mask and sketch, respectively), the number of training videos differs between the two datasets.  
**Evaluation Metrics:** We evaluate our results using the standard metrics for VOS tasks [26], specifically the following:  
* Region similarity (J): The region similarity, also known as Intersection over Union (IoU), quantifies the overlap between the predicted and ground truth segmentation masks. It is computed as the ratio of the intersection to the union of the two masks.
* Contour accuracy (F): The contour accuracy combines precision and recall into a single metric. It is the harmonic mean of these two metrics. A higher F-measure reflects a more accurate segmentation, balancing both false positives and false negatives.
* J&F (Combined Jaccard and F-measure): The J&F metric averages the Jaccard Index and F-measure to provide a more comprehensive evaluation of segmentation performance. It incorporates both the overlap accuracy (J) and the balance between precision and recall (F), offering a well-rounded assessment.  
For the DAVIS dataset, we use the official evaluation code ([[https://github.com/davisideochallenge/davis2017-evaluation](https://github.com/davisideochallenge/davis2017-evaluation)]([https://github.com/davisideochallenge/davis2017-evaluation](https://github.com/davisideochallenge/davis2017-evaluation)), (accessed on 22 April 2019)). Experiments on the YouTube datasets are evaluated on the competition server ([[https://youtube-vos.org/dataset/rvs/](https://youtube-vos.org/dataset/rvs/)]([https://youtube-vos.org/dataset/rvs/](https://youtube-vos.org/dataset/rvs/)), (accessed on 20 March 2021)), in line with the Ref-YouTube-VOS method [34]. Note that only 202 videos from the validation set are available for evaluation on the server, and all subsequent results are based on these 202 videos.  
**Implementation Details:** We use STCN [45] as our baseline model. Both the video frames and corresponding sketches are downscaled to a resolution of 384p. The model is trained using the Adam optimizer with an initial learning rate of \(1\times 10^{-5}\). The frame encoder is initialized with classification weights pre-trained on ImageNet [75], while the sketch encoder is initialized with classification weights pre-trained on QuickDraw [76]. Unlike STCN, we select the first frame as the reference frame and randomly sample two additional temporal frames to form a training clip. For contrastive learning, we empirically set \(|\mathbf{M}|=3\). During training, data augmentation includes random affine transformations, cropping, and horizontal flipping. To optimize computational efficiency, the size of the memory bank is set to 3. During inference, frames and sketches are resized to 480 \(\times\) 480, with the memory bank updated every 5 frames. Temporal mask propagation is performed without any post-processing. For fair comparison, the text encoder remains frozen throughout the training. All experiments are conducted with a batch size of 8 and a weight decay of \(1\times 10^{-7}\). The learning rate is reduced by a factor of 10 after every 2/3 of the total iterations. The model is trained end-to-end for 180,000 iterations on a single NVIDIA RTX 3090 GPU.