# SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale Benchmark +
## 4 SKVOS Model
### Sketch-Video Fusion Designs
#### 4.3.3 Latent Fusion: Cross Attention  
In existing Transformer-based image segmentation methods [65], the decoder typically relies on a set of static learnable parameters as queries, which are designed to capture the generic features of objects in images and are commonly used for segmentation tasks with closed-set categories. However, we aim for the model to generalize to unknown categories, and sketches, as an intuitive interaction method, can convey rich structural and semantic information in a concise form. Previous studies have demonstrated the potential of sketches in zero-shot learning [16; 17] and fine-grained matching [20; 21; 53]. Based on this, we propose a sketch-based dynamic query mechanism. This mechanism uses user-drawn sketches as prior information to generate a set of instance-aware queries closely related to the sketch content. These queries not only encode the geometric structure of the sketch but also incorporate the implicit semantic information within the sketch, enabling the more precise localization and segmentation of target objects. Unlike traditional fixed queries, this dynamic query mechanism fully leverages the flexibility and expressiveness of sketches, generating diverse local correspondences and ensuring the model focuses on user-specified target regions. Specifically, we transform the problem of fusing sketch features with video frame features into an image matching task, effectively achieving this goal by establishing deep pixel-level feature correspondences in the latent space. Given the first frame \(I_{0}\) and the sketch reference \(S^{i}\), their features are extracted using a visual encoder \(E_{\texttt{p}}(\cdot)\) and a sketch encoder \(E_{\texttt{s}}(\cdot)\), respectively. These features are denoted as \(F_{0}\in\mathbb{R}^{C_{\texttt{s}}\times H\times W}\) and \(F^{i}_{\texttt{s}}\in\mathbb{R}^{C_{\texttt{s}}\times M\times N}\), where \(H\), \(W\), \(M\), and \(N\) are the spatial dimensions, and \(C_{\texttt{p}}\) and \(C_{\texttt{s}}\) represent the channel dimensions. To fuse these features, we introduce two cross-modal attention strategies. Figure 5c illustrates the first cross-modal attention module, termed Cross-KV. In this module, sketch features are encoded as the key and value through two \(1\times 1\) convolution mappings, while visual features are encoded as the query via another \(1\times 1\) convolution filter to match the dimensions of the sketch features. This yields the following outputs: key \(K^{i}_{\texttt{s}}\in\mathbb{R}^{C\times MN}\), value \(V^{i}_{\texttt{s}}\in\mathbb{R}^{C\times MN}\), and query \(Q_{0}\in\mathbb{R}^{C\times HW}\). A dot-product attention [72] is then computed between query and key, storing the correspondences between visual features and the sketch reference. Subsequently, the value is transformed by the attention map to generate the output features \(R^{i}\in\mathbb{R}^{C\times HW}\) using the following equations:  
\[W=\Phi(\frac{Q_{0}\texttt{T}K^{i}_{\texttt{s}}}{\sqrt{C}}), \tag{4}\] \[R^{i}=V^{i}_{\texttt{s}}W^{\texttt{T}}\odot Q_{0}, \tag{5}\]  
where \(\Phi(\cdot)\) represents the softmax operation, and \(\odot\) denotes element-wise multiplication. Alternative strategies for \(\odot\) can be explored to improve the effectiveness of this module. Figure 5d presents the second module, referred to as Cross-Q. In this design, the sketch feature acts as the query, reflecting its role as a reference condition for segmenting the target mask. The visual features are encoded as the key and value. The resulting outputs are key \(K_{0}\in\mathbb{R}^{C\times HW}\), value \(V_{0}\in\mathbb{R}^{C\times HW}\), and query \(Q^{i}_{\texttt{s}}\in\mathbb{R}^{C\times MN}\). The attention map in this case is computed between the query from the sketch feature and the key from the visual feature,  
effectively transposing the attention map from the Cross-KV module.