# SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale Benchmark +
## 5 Experiments
### Ablation Study  
#### 5.3.1 Fusion Designs  
We encode sketches and video frames using two separate ResNet50 [80] networks. To assess the impact of different design choices, we conducted an ablation study by remov  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Reference** & \(\mathcal{J}\)**\&\(\mathcal{F}\)** & \(\mathcal{J}\)** & \(\mathcal{F}\)** & **Average Annotating Time** & **FPS** \\ \hline Text & 47.5 & 41.0 & 54.5 & 5.0 s [14] & 12.3 \\  & +click & 58.6 & 56.6 & 60.6 & - & 12.3 \\  & +box & 59.3 & 57.7 & 60.8 & \textgreater{}30.6 s & 12.7 \\ Cross & 56.8 & 56.0 & 57.6 & - & 19.2 \\ Circle & 60.4 & 58.9 & 62.0 & - & 19.2 \\ Contour & 73.8 & 72.0 & 75.8 & \textgreater{}30.6 s & 18.8 \\ Scribble & 69.6 & 67.2 & 72.1 & 1.25 s [79] & 19.2 \\ Sketch (Ours) & 77.3 & 74.6 & 79.9 & 30.6 s & 18.8 \\ Mask & 79.6 & 77.1 & 82.1 & 109.0 s [56] & 13.2 \\ \hline \hline \end{tabular}
\end{table}
Table 4: Fair comparison with different references on YouTube-VOS validation dataset. The rightmost column shows the average time of annotating one object.  
ing the TRM and STCL modules, which allowed us to evaluate their individual influence on the performance. In the study, we also experiment with different feature levels and interaction variants between the sketch reference and video frames. The following configurations are tested: _GAP sketch features_: After extracting the output from the layer4 of the sketch encoder, we apply global average pooling (GAP) to aggregate the global information of the sketch. This approach serves to verify the significance of spatial information embedded within the sketch features, enabling us to assess its contribution to segmentation accuracy. _Spatial sketch features_: In this variant, we directly use the spatial feature maps generated by the sketch encoder, which preserve the detailed spatial information of the sketch. This allows for a finer representation of the sketch's shape and structure, potentially improving the localization and segmentation precision. _Multi-level visual features_: This refers to the use of features from multiple layers of the video encoder, which capture information at different levels of abstraction. By leveraging features from various layers, the model can potentially enhance its ability to segment objects across a range of spatial and temporal scales. These experiments aim to highlight the role of spatial and multi-level features in improving the performance of sketch-based video object segmentation.  
Table 5 presents the results of various sketch-based video object segmentation (VOS) models. The designs incorporating cross-attention and input concatenation consistently deliver strong results, with the best model achieving a \(\mathcal{J}\&\mathcal{F}\) score of 74.9, \(\mathcal{J}\) of 72.9, and \(\mathcal{F}\) of 77.0. In contrast, sketch-based weight generation does not perform well in this task. This could be because, in this setup, the sketch reference interacts less effectively with the video frames, making it difficult for the reference information to propagate across the frames. Among the cross-attention variations, Cross-Q outperforms Cross-KV, improving the \(\mathcal{J}\&\mathcal{F}\) by approximately 7 points. This finding challenges prior studies [34; 78], suggesting that the design of sketch-based references may require different approaches than those used for language models. Additionally, the use of GAP features yields poor performance, highlighting the importance of the spatial information provided by the sketch in the VOS task. While multi-level visual features offer a slight improvement, the added computational cost may not justify their use.  
In conclusion, utilizing sketch features as queries in cross-modal attention, along with spatial sketch features, proves effective for the video object segmentation task.