# SKVOS: Sketch-Based Video Object Segmentation with a Large-Scale Benchmark +
## 4 SKVOS Model
### Temporal Relation Module  
Video scenes are complex and diverse, characterized by the coexistence of multi-scale targets as shown in Figure 2. Although leveraging prior information through a memory bank aids in learning target features in subsequent frames, changes in target appearance, motion, or viewpoint may introduce interference when randomly sampled memory frames are used. We observe that adjacent frames in a video typically exhibit only subtle changes. Based on this, we design the Temporal Relation Module (TRM) as illustrated in Figure 6. This module explicitly computes multi-level feature relationships between adjacent frames and extracts highly responsive components, thereby enhancing temporal continuity in segmentation and effectively suppressing noise caused by target motion.  
Figure 6: Implementation of Temporal Relation Module.  
Given the pair of consecutive frames, \(I_{t-1}\) and \(I_{t}\), their key and value can be obtained through the previous encoder, denoted as \(K_{t-1}\in\mathbb{R}^{C_{k}\times HW}\), \(K_{t}\in\mathbb{R}^{C_{k}\times HW}\), \(V_{t-1}\in\mathbb{R}^{C_{v}\times HW}\) and \(V_{t}\in\mathbb{R}^{C_{v}\times HW}\). We first construct a cost volume to model the global pixel-wise relations over time. The affinity matrix, \(H\in\mathbb{R}^{H\times W\times HW}\), is computed as the cosine similarity between each point in the keys of frame \(t\) and frame \(t-1\):  
\[H(i,j)=\frac{K_{t}(i)K_{t-1}(j)}{\big{|}|K_{t}(i)|\big{|}|\big{|}|K_{t-1}(j)| \big{|}|}, \tag{10}\]  
where \(H\) and \(W\) are the shapes of the key/value. The \(C_{k}\) and \(C_{v}\) are the feature dimensions. The \(H(i,j)\) represents the relation between the \(i\)-th feature vector in \(K_{t}\) and \(j\), the feature vector in \(K_{t-1}\). In addition, we design a multi-scale relation block to effectively fuse the hierarchical appearance feature and enhance the temporal correlation of adjacent frames. Specifically, inspired by [73], we reduce the correlation map resolution by performing pooling operations on the last two dimensions, resulting in multi-scale correlation representations \(\{H^{l1},H^{l2},H^{l3}\}\), followed by upsampling and concatenation layers to form the final feature map, where \(H^{l}\in\mathbb{R}^{H\times W\times H^{l}\times W^{l}}\). The number of pyramid levels and size of each level can be modified. Our pyramid pooling module is a three-level one. To minimize unnecessary connections, we define a searching window based on pyramid correlation as follows:  
\[\mathcal{P}(x)_{l}=\{x+r:||r||_{1}\leq R\}, \tag{11}\]  
where \(x\in\mathbb{Z}^{2}\) is the coordinate in \(K_{t-1}\) in the \(l\)-th layer in the correlation pyramid, and \(r\) is the displacement from point \(x\), and \(R\) is the size of the search window. This window-based patch search leverages the spatial continuity between adjacent frames to eliminate the need for a comprehensive global search in Figure 6. Then, we employ the bilinear operator to interpolate the multi-scale correlation maps, aligning them to a uniform size to obtain \(\{A^{l1},A^{l2},A^{l3}\}\), where \(A_{l}\in\mathbb{R}^{HW\times HW}\). Next, the attention weights are applied on the visual features of previous frame to obtain the final feature representation \(F_{trm}\):  
\[F_{trm}=\sum_{l=1}^{3}V_{t-1}A_{l}+V_{t}, \tag{12}\]  
Finally, we fuse the relation map with output of memory-based network, following  
\[F_{fused}=F_{vr}\oplus F_{trm} \tag{13}\]  
Then, the \(F_{fused}\) features are gradually processed and upsampled by the decoder \(D(\cdot)\). Specifically, the decoder reconstructs the object mask by compressing the input feature map to 256 channels, followed by multiple refinement modules that upscale the map by a factor of two. Each module integrates skip-connections from the query encoder. The final output is bilinearly upsampled to match the original resolution.