Learning Spatial-Semantic Features for Robust Video Object Segmentation

## Abstract

Abstract Tracking and segmenting multiple similar objects with complex or separate parts in long-term videos is inherently challenging due to the ambiguity of target parts and identity confusion caused by occlusion, background clutter, and long-term variations.
In this paper, we propose a robust video object segmentation framework equipped with spatial-semantic features and discriminative object queries to address the above issues.
Specifically, we construct a spatial-semantic network comprising a semantic embedding block and spatial dependencies modeling block to associate the pretrained ViT features [ 18 , 34 ] with global semantic features and local spatial features, providing a comprehensive target representation.
In addition, we develop a masked cross-attention module to generate object queries that focus on the most discriminative parts of target objects during query propagation, alleviating noise accumulation and ensuring effective long-term query propagation.
The experimental results show that the proposed method set a new state-of-the-art performance on multiple datasets, including DAVIS2017 test ( 89.1% ), YoutubeVOS 2019 ( 88.5% ), MOSE ( 75.1% ), LVOS test ( 73.0% ), and LVOS val ( 75.1% ), which demonstrate the effectiveness and generalization capacity of the proposed method.
We will make all source code and trained models publicly available.

## 1 Introduction

Video Object Segmentation (VOS) aims to track and segment target objects that are specified in the initial frame with mask annotations in a video sequence [, , , ].
This practice holds significant promise across various applications, particularly as the prevalence of video content surges in domains like autonomous driving, augmented reality, and interactive video editing [, ].
The main challenges faced by VOS include drastic target appearance change, occlusion, and identity confusion caused by similar objects and background clutter, which becomes more difficult, particularly in long-term videos.

VOS methods [, ] typically perform video object segmentation by comparing the test frame with past frames.
Specifically, they first use an association model to generate the correlated features of the test sample and target templates pixel-wise and then predict target masks based on the correlated features.
The pixel-wise correlated features [] contribute to accurate mask predictions.
To account for targets that appear differently over time, some methods [, ] utilize a memory module to store these changing appearances.
In addition, several recent methods [, ] introduce object queries to help distinguish different target objects for alleviating identity confusion.

Despite the good performance on short-term test videos with simple-structured objects, existing methods do not work well in the following challenging scenarios.
First, when dealing with target objects with multiple complex or separate parts caused by occlusion, background clutter, and shape complexity, existing methods often generate incomplete prediction masks.
This is because the pixel-wise correlation adopted by existing methods mainly focuses on detailed spatial information at the pixel level and neglects the semantic information at the instance level.
Second, although object query improves the ID association accuracy, it does not work well in sequences with dramatic target appearance changes.
Existing methods [, ] update target queries relying on the entire predicted sample, which can introduce noise and errors and further lead to tracking failures, where targets are missed or their identities are swapped.

Figure: Figure 1: Performance comparison in terms of $\mathcal{J}\&\mathcal{F}$ on various datasets. The proposed method achieves favorable performance against the state-of-the-art methods, including AOT_L [], XMem [], and Cutie [] on all the datasets. All methods are trained using the YoutubeVOS and DAVIS datasets.
Refer to caption: x1.png

To address the aforementioned issues, we propose a VOS framework to learn spatial-semantic features and generate discriminative object queries for robust VOS.
To be specific, we construct a Spatial-Semantic Block comprising a semantic embedding module and a spatial dependencies modeling module to efficiently leverage the semantic information and local details of the pre-trained ViTs for VOS without training all the parameters of the ViT backbone.
In addition, we develop a discriminative query to focus more on the representative target parts for more reliable target representation and query update, which can alleviate noise accumulation in query propagation and ensure robustness in long-term videos.
We evaluate the proposed method extensively on five benchmarks, including DAVIS 2017 [], Youtube VOS2018 [], Youtube VOS2019, LVOS [], and MOSE [].
The experimental results show that our approach set new state-of-the-art performance on all these datasets including DAVIS2017 test (89.1%), YoutubeVOS 2019 (88.5%), MOSE (75.1%), and LVOS test (73.0%).
In addition, the results achieved by only using the DAVIS and Youtube datasets for training show that our method generalizes well to different datasets, as shown in Figure 1.

In this work, we make the following contributions:

- •
We propose a spatial-semantic network block to incorporate semantic information with spatial information for Video Object Segmentation.
We use the cls token from a pre-trained ViT backbone to extract global semantic information for interaction with current features.
In addition, we design a spatial dependencies modeling module to leverage the features of the visual transformer better.
The proposed spatial-semantic block enables robust VOS performance in handling the target objects with multiple complex or separate parts.
- •
We develop a discriminative query mechanism to capture the representative region of the target for better target representation learning and updating.
This achieves significant improvement in VOS datasets, especially in long-term VOS datasets.
- •
We conduct extensive experiments on five diverse datasets to demonstrate the effectiveness of the proposed method and comprehensive ablation studies to evaluate the contribution of each proposed component.
We set a new state-of-the-art performance on all these datasets.

## 2 Related Work

Video Object Segmentation.
Video Object Segmentation (VOS) aims to identify and separate objects within a video sequence from the background and from each other based on one annotated target frame, which is also known as semi-supervised VOS.
Early VOS methods [, , , , ] use test-time learning to adapt pre-trained segmentation models for separating the specified targets online.
Test-time learning of a deep model is time-consuming and significantly reduces the run speed.
To avoid online learning, propagation-based methods [, , ] model the temporal correlations between adjacent frames with offline shift attention learning to propagate target masks from the previous frame to the current.
Despite the promising performance, these methods suffer from error accumulation caused by inaccurate predictions.
To achieve fast and efficient inference, matching-based methods [, , , ] identify targets by comparing the target template to the test image and predict target masks based on matching features.
For instance, OML [] and VideoMatch [] conduct pixel matching between the nearest frames for mask prediction.
Some methods [, , , ] further improve the matching-based VOS framework by introducing target memory to enrich target templates and developing advanced matching strategies to generate more comprehensive correlated features.
For example, SimVOS [] and JointFormer [] use the ViT blocks [] to model features and patch correspondence for object modeling jointly.
In addition, some recent methods [, ] introduce object queries to enrich target representation for better ID identification.

Despite the rich target features and sufficient interactions, existing methods lack modeling of object-level semantic information, so they do not work well on dealing with objects with complex structures and long-term deformation.
Unlike existing methods, the proposed method learns features and associates templates with test frames at both pixel and object levels achieved through a novel object-aware backbone and a pixel-and-query level correlation component, leading to robust VOS performance on complex and long-term videos.

Query Transformer Networks.
DETR [] utilizes queries to represent target regions for object detection tasks.
In the segmentation domain, some methods [, ] introduce masked cross-attention to use queries to represent targets for segmentation tasks.
Motivated by the above methods, ISVOS [] utilizes additional queries generated by Mask2Former [] to represent instance information for target modeling, enhancing target representation in matching-based VOS methods.
To achieve end-to-end training, Cutie [] learns object-level information and facilitates interaction between object and pixel information, improving segmentation accuracy and efficiency.
However, the query-based VOS methods suffer from query drift due to ineffective propagation.
These methods model the entire target sample in the query, incorporating redundant information and degrading saliencies of the target object.
Instead, the proposed method generates discriminative queries for target representation and query propagation, which is more robust to variations and preserves discriminative information.

Adapters for Dense Tasks.
Vit-Adapter [] first addresses the problem of utilizing pre-trained models to boost detection and segmentation performance and proposes a dense adapter to exchange information with ViT blocks.
To enhance spatial hierarchical features, ViT-CoMer [] introduces a multi-receptive field feature pyramid module to provide multi-scale spatial features.
However, those two methods require full fine-tuning for the best performance, which is inefficient, especially in the video domain.
Meanwhile, they focus on spatial information enhancement while lacking semantic feature interaction between ViT blocks and multi-scale features.

To better suit VOS tasks, our method features several design distinctions compared to existing adapters.
First, we fuse the feature from ViT to multi-scale features and do not need extra modules to transmit information to ViT blocks, which is more efficient.
Second, the proposed fusion block associates semantic priors and local dense features.
Third, the proposed method freezes the ViT blocks during training and does not need full fine-tuning.

## 3 Proposed Algorithm

Our method aims to learn comprehensive target features that contain semantic, spatial, and discriminative information for video object segmentation. This helps us cope with complex target appearance variations and ID confusion between target objects with similar appearances in long-term videos.
To this end, we propose a spatial-semantic feature learning network that first embeds semantic information from a trained ViT network with multi-scale features from a trainable CNN network. Then, we model spatial dependencies upon the fused feature, enabling spatial-semantic features for VOS.
In addition, to ensure that the target query retains discriminative information during long-term appearance changes, we develop a discriminative query propagation module to capture and update the local representations in the target object.

### 3.1 Overall Framework

Figure 2(a) shows the overall framework of the proposed method, consisting of two core modules: a spatial-semantic feature generation module and a pixel-and-query dual-level target association module.
Given a test frame $\mathbf{X}_{t}\in\mathbb{R}^{3\times H\times W}$, the proposed method predicts the mask of all specified objects within it based on the initial frame $\mathbf{X}_{init}\in\mathbb{R}^{3\times H\times W}$ and its corresponding annotation mask $\mathbf{M}_{init}\in\mathbb{R}^{H\times W}$.
First, the feature generation module takes the test frame $\mathbf{X}_{t}$ as input and generates spatial-semantic features.
Then, the target association module associates the spatial-semantic features with the reference samples using pixel- and query-wise correlation to generate the correlated features.
Finally, a decoder predicts the mask based on these correlated features.

To better represent video targets that vary over frames, we develop a dual-level memory module consisting of the base features generated by the feature generation module and target queries that capture more global information.
Both types of memory are updated every few frames using the online predicted masks, enabling adaptive target representations for effective target association.

Figure: Figure 2: Architecture of the proposed method. (a) The overall framework of the proposed method. (b) Implementation of the spatial-semantic block. Semantic priors and location details are fused by semantic embedding and spatial dependencies modeling, respectively. (c) Illustration of the discriminative query propagation, which learns more discriminative queries to represent high-level object information.
Refer to caption: x2.png

### 3.2 Spatial-Semantic Feature Learning

Figure 2(b) shows the spatial-semantic network, which consists of a semantic embedding module that incorporates semantic information from a trained ViT model into multi-scale features and a spatial dependency modeling module that learns spatial dependencies based on these incorporated features.

Semantic embedding.
As the VOS task is designed for generic objects and no class labels are given, it is challenging to learn semantic representations directly from the VOS dataset during training.
Fortunately, the CLS token in a trained ViT model aggregates semantic information from the entire image and provides a rich, global representation of the image content.
Thus, we incorporate the CLS token with the multi-scale features generated from CNN networks to obtain detailed semantic features at different scales.
To be specific, we use a cross-attention operation to perform the corporation of these two features by taking the semantic feature (CLS token $\mathcal{F}_{cls}^{i}$) concatenated with global token representation ( $\mathcal{F}_{g}^{i}$) as the Key and Value, and the spatial-temporal features ($\mathcal{F}_{spsem}^{i-1}$) from the $(i-1)$-th block as the Query to produce the semantic-aware features $\mathcal{F}_{sem}^{i}$ as:

$$ $\mathcal{F}_{sem}^{i}=\textit{f}_{softmax}\left(\frac{(W^{q}(\mathcal{F}_{cls} ^{i},\mathcal{F}_{g}^{i}))(W^{k}\mathcal{F}_{spsem}^{i-1})^{\top}}{\sqrt{d}} \right)(W^{v}\mathcal{F}_{spsem}^{i-1}),$ (1) $$

where $\textit{f}_{softmax}$ is the SoftMax operation, and $W^{q}$, $W^{k}$, and $W^{v}$ are the projection matrices for Query, Key, and Value.

Spatial dependency modeling.
We note that it is important to learn spatial dependencies between semantic features for the VOS model to understand the relationships between different object parts, which helps handle objects with complex structures or separate parts.
Given the features of the input image patches generated by the ViT backbone model $\mathcal{F}_{vit}^{i}$, we adopt the Multi-scale Deformable Cross-Attention operation [] to capture the spatial information ${\mathcal{F}}_{spa}^{i}$:

$$ ${\mathcal{F}}_{spa}^{i}=\sum_{j}\textit{f}{{}_{\alpha}}(\mathcal{F}_{vit}^{i}, \textit{f}_{gs}(\mathcal{F}_{sem}^{i},\Delta p))\cdot\textit{f}_{gs}(\mathcal{ F}_{sem}^{i},\Delta p),$ (2) $$

where $\Delta p$ indicates the deformable offsets generated based on the query features $\mathcal{F}_{vit}^{i}$, $\textit{f}_{gs}$ performs grid sampling on $\mathcal{F}_{sem}^{i}$ using the offsets $\Delta p$, $\textit{f}_{\alpha}$ denotes the attention weights generated based on the query features $\mathcal{F}_{vit}^{i}$ and the grid sampled features, and $\sum_{j}$ is the summation over all sampled points.

To enable an effective long-range propagation through the cascade blocks, we use the residual connection [] followed by a convolution-feed-forward to generate the final spatial-semantic feature of the current block:

$$ $\mathcal{F}_{spsem}^{i}={\mathcal{F}}_{spa}^{i}+\textit{f}_{ConFFN}\left({ \mathcal{F}}_{spa}^{i}+\mathcal{F}_{spsem}^{i-1}\right).$ (3) $$

The spatial-semantic block leverages the strengths of both a pre-trained ViT network for providing semantic information and a multi-scale CNN network for rich spatial information, enabling comprehensive spatial-semantic feature learning for robust VOS.

### 3.3 Discriminative Query Propagation

We note that updating the target query memory directly with entire object patches generated based on online predicted masks is ineffective as the predicted masks often cover background noise, reducing target distinctiveness and leading to accumulating errors over time.
To propagate target queries effectively across frames, we update the target queries with the most distinctive feature of the target object.

Discriminative feature generation.
We select the discriminative feature of a target object by comparing the target query $\mathcal{Q}\in\mathbb{R}^{N\times C}$ with every channel activation in the correlated feature map of the target and taking the most similar one.
Formally, given the feature map $\mathcal{F}\in\mathbb{R}^{H\times W\times C}$, the salient channel activation feature of a target is computed as:

$$ $\mathcal{Q}_{s}=\mathcal{F}[\underset{i\in(1,H*W)}{\arg\max}(\mathcal{F} \mathcal{Q}^{T}\odot\mathcal{\mathcal{M}})],$ (4) $$

where $\mathcal{M}$ represents a binary mask of the target, and $\mathcal{F}[i]$ denotes the $i$-th channel activation of the feature map $\mathcal{F}$.
We use the strategy in Cutie [] to generate the target mask $\mathcal{M}$, which uses the predicted segmentation results in the previous frame.

Discriminative query update.
Based on the discriminative target feature $\mathcal{Q}_{s}$ generated from a new target sample, we can update target queries by dynamically calculating the relationship between the salient query and salient pixel features in an additive manner, which can be formulated as:

$$ $\mathcal{Q}_{\text{out }}=(\alpha\odot\frac{\mathcal{A}}{\|\mathcal{A}\|_{2}}+ \mathcal{Q}_{s})\mathcal{W}_{\text{out }}+\mathcal{Q},$ (5) $$

where $\alpha$ is a learnable parameter to scale the matrix, and $\mathcal{A}$ denotes the correspondence between salient query and salient pixel features, which is computed as.

$$ $\mathcal{A}=\mathcal{Q}\odot\mathcal{Q}_{s}.$ (6) $$

The proposed discriminative query propagation scheme adaptively refines target queries with the most representative features, which helps deal with the challenges of dramatic appearance variations in long-term videos.

## 4 Experiments

### 4.1 Experimental Settings

Pretrained Backbone & Baseline.
We conduct different pre-trained weights obtained using various pre-training methods, including MAE [], DINOV2 [], and Depth Anything [].
To validate the effectiveness of our proposed discriminative query propagation, we first choose ResNet50 as the backbone for a baseline model.
The multi-scale features from the backbone are utilized in the Decoder for fine prediction.
Soft aggregation operation is used to merge the predicted masks in the multi-target scenario.

Training.
Our training settings are similar to Cutie [].
Note that, apart from ResNet pre-training, we do not pre-train other models on static images and directly trained them on video datasets (YoutubeVOS [] and DAVIS []).
To enhance the performance of our model, we utilize the MEGA dataset constructed by Cutie, which includes the YouTubeVOS [], DAVIS [], OVIS [], MOSE [], and BURST [] datasets.
We sample eight frames to train the model, and three are randomly selected to train the matching process.
For each sequence, we randomly choose at most three targets for training.
Point supervision in loss is adopted to reduce the memory requirements.

For optimization, AdamW [] optimizer is performed with a learning rate of 5e-5 and a weight decay of 0.5.
We train the model for 125K with batch size 16 on the video dataset, and 195k on the MEGA dataset.
For specific parameter settings, please refer to our supplementary materials.
All our models are trained on 8 x NVIDIA V100 GPUs and tested on an NVIDIA V100 GPU.

Inference.
Our feature and query memory are updated every 5th frame during the testing phase.
For longer sequences, we employ a long-term fusion strategy [] for updating. To enhance storage quality, we skip frames without targets and do not store them.
The test input size contains two scales: 480 for fair comparison and 600/720 for further improvement.

**Table 1: Ablation study. All the variants are trained on the YoutubeVOS and DAVIS datasets.**
| Dataset | MOSE-val | DAVIS 2017 test | LVOS test | LVOS val | YouTube-VOS 2019 val |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{G}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{G}$ | $\mathcal{J}_{s}$ | $\mathcal{F}_{s}$ | $\mathcal{J}_{u}$ | $\mathcal{F}_{u}$ |
| XMem [] (Baseline) | 53.3 | 62.0 | 57.6 | 81.0 | 77.4 | 84.5 | 50.0 | 45.5 | 54.4 | 49.5 | 45.2 | 53.7 | 85.5 | 84.3 | 88.6 | 80.3 | 88.6 |
| Cutie [] | 64.0 | 60.0 | 67.9 | 84.2 | 80.6 | 87.7 | 56.2 | 51.8 | 60.5 | 60.1 | 55.9 | 64.2 | 86.1 | 85.8 | 90.5 | 80.0 | 88.0 |
| +Discriminative Query | 64.2 | 60.3 | 68.1 | 85.2 | 81.8 | 88.5 | 57.4 | 53.3 | 61.5 | 62.1 | 57.9 | 66.2 | 86.5 | 86.2 | 90.7 | 80.6 | 88.8 |
| +ViT | 64.2 | 60.2 | 68.3 | 85.6 | 82.0 | 89.2 | 58.3 | 53.7 | 62.8 | 58.9 | 54.1 | 63.6 | 86.7 | 86.4 | 90.3 | 81.0 | 88.7 |
| +Spatial | 68.2 | 64.0 | 72.4 | 86.2 | 82.4 | 90.1 | 67.4 | 62.9 | 71.9 | 68.4 | 63.7 | 73.1 | 87.3 | 86.4 | 91.3 | 81.5 | 90.3 |
| +Semantic (Full) | 68.5 | 64.5 | 72.6 | 86.7 | 82.7 | 90.8 | 66.5 | 62.1 | 70.8 | 67.7 | 63.1 | 72.2 | 87.5 | 86.8 | 91.8 | 81.3 | 89.9 |

### 4.2 Ablation Study

Effect of different components.
Table 1 provides a comprehensive analysis of the components of our method.
The first two rows describe current methods upon which we build improvements.
Cutie enhances performance by embedding a query transformer within the model.

+ Discriminative Query, which performs the discriminative query generation by adaptively refining target queries with the most representative features in the query transformer.
The performance gains compared to Cutie demonstrate the advantages of target modeling of discriminative query for VOS, especially in long-term benchmarks.

+ ViT, which directly utilizing ViTDet [] to incorporate pre-trained model features at multiple scales.
The performance denotes that directly employing ViT to generate multi-scale features is minimally beneficial for VOS tasks.

+ Spatial, which applies spatial dependencies modeling in learning multi-scale detail features.
It is clear that the proposed spatial dependencies modeling significantly boosts model performance in terms of $\mathcal{J}\&\mathcal{F}$ on all the VOS benchmarks.
It validates the superiority of using spatial dependencies learning to conduct feature generation.

+ Semantic, which inserts semantic information to embed the multi-scale features.
The performance in three benchmarks, MOSE, DAVIS 2017, and YouTubeVOS 2019, is further improved.
These improvements validate the benefits of semantic embedding, which improves the semantic understanding of targets.

Effect of different pre-trained weights.
Table 2 also compares the effects of different training parameters on segmentation performance.
We conduct experiments using three different pre-training parameters, and the results indicate that better pre-training parameters lead to more significant improvements in the model, especially those from Depth Anything [] (8-query in the table).
The Depth Anything model implicitly contains depth information about the scene, which contributes to its stronger representation capability, consistent with the findings of the Depth Anything paper.

Effect of different query numbers.
Table 2 shows the impact of different query numbers.
Performance decreases as the number of queries increases. This is because more queries introduce additional background noise, given the fewer targets present in most VOS benchmarks, except MOSE [].

Effect of adapters.
Table 2 presents a performance comparison between different adapters and our method. We replace the fusion module with various adapters and fully fine-tune them on the MEGA dataset, as indicated in related research that full fine-tuning achieves optimal performance [, ]. The experimental results demonstrate that our method is efficient in training and outperforms the other two methods in effectiveness.

**Table 2: Effect of different settings in every component. The ablation study examines the impact of pre-trained weights and the number of queries on training results for the YoutubeVOS and DAVIS datasets. Additionally, it explores the effects of different adapters on the MEGA dataset.**
| Dataset | MOSE-val | DAVIS 2017 test | LVOS test | YouTube-VOS 2018 val | YouTube-VOS 2019 val |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{G}$ | $\mathcal{J}_{s}$ | $\mathcal{F}_{s}$ | $\mathcal{J}_{u}$ | $\mathcal{F}_{u}$ | $\mathcal{G}$ | $\mathcal{J}_{s}$ | $\mathcal{F}_{s}$ | $\mathcal{J}_{u}$ | $\mathcal{F}_{u}$ |
| Effect of different pre-trained weights. |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| ResNet [] (Baseline) | 64.2 | 60.3 | 68.1 | 85.2 | 81.8 | 88.5 | 57.4 | 53.3 | 61.5 | 86.3 | 86.2 | 91.0 | 79.9 | 88.3 | 86.5 | 86.2 | 90.7 | 80.6 | 88.8 |
| MAE [] | 67.0 | 62.6 | 71.3 | 85.3 | 81.3 | 89.3 | 64.4 | 60.2 | 68.7 | 86.4 | 85.6 | 90.2 | 80.6 | 89.0 | 86.4 | 85.4 | 89.7 | 81.2 | 89.3 |
| DINOv2 [] | 67.5 | 63.5 | 71.6 | 86.1 | 82.7 | 89.6 | 63.2 | 58.7 | 67.6 | 87.3 | 86.6 | 91.7 | 80.9 | 89.9 | 86.7 | 86.2 | 91.0 | 80.6 | 89.1 |
| ViT-small [] | 64.3 | 60.0 | 68.5 | 85.3 | 81.5 | 89.1 | 55.9 | 51.6 | 60.1 | 86.0 | 85.3 | 90.4 | 79.9 | 88.5 | 86.3 | 85.3 | 90.1 | 80.6 | 89.1 |
| Effect of different query numbers. |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| 32-query | 68.3 | 64.3 | 72.3 | 85.8 | 81.7 | 89.9 | 64.8 | 60.4 | 69.6 | 86.8 | 86.9 | 92.0 | 79.8 | 88.6 | 86.9 | 86.8 | 91.6 | 80.3 | 88.8 |
| 16-query | 67.7 | 63.6 | 71.8 | 86.6 | 82.7 | 90.2 | 66.4 | 62.0 | 70.9 | 86.9 | 86.7 | 91.8 | 80.1 | 88.9 | 87.0 | 86.6 | 91.6 | 80.6 | 89.3 |
| 8-query | 68.5 | 64.5 | 72.6 | 86.7 | 82.7 | 90.8 | 66.5 | 62.1 | 70.8 | 87.4 | 87.0 | 92.0 | 80.9 | 89.7 | 87.5 | 86.8 | 91.8 | 81.3 | 89.9 |
| Effect of different adapters. |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Ours (Adapter [] ) | 72.2 | 67.8 | 76.4 | 87.2 | 84.1 | 90.4 | 71.1 | 66.7 | 75.6 | 87.1 | 86.1 | 91.0 | 81.6 | 89.8 | 87.1 | 85.9 | 90.9 | 81.4 | 90.1 |
| Ours (CoMer []) | 73.2 | 69.1 | 77.3 | 87.0 | 83.2 | 90.8 | 73.4 | 68.8 | 77.9 | 87.3 | 87.1 | 92.2 | 80.7 | 89.4 | 87.4 | 87.0 | 91.7 | 81.3 | 89.6 |
| Ours (Spatial-Semantic) | 74.0 | 69.8 | 78.3 | 87.8 | 84.0 | 91.7 | 73.0 | 68.3 | 77.8 | 88.0 | 87.0 | 91.8 | 82.5 | 90.7 | 88.1 | 87.4 | 92.5 | 81.9 | 90.7 |

### 4.3 Evaluations with State-Of-The-Art Methods

DAVIS 2017 [] is a benchmark that offers densely annotated, high-quality, full-resolution videos with multiple objects of interest.
Table 3 shows that our method achieves favorable performance (86.7%) against state-of-the-art only trained on YoutubeVOS and DAVIS.
The gains are primarily due to the injection of semantic information, which helps handle sequences with object occlusion.
With more training data and improved inference size, our method performs best on the DAVIS test set (89.1%).

**Table 3: Quantitative comparisons on the MOSE, LVOS, DAVIS 2017, YouTube-VOS 2018 & 2019 dataset. The best two results are shown in red and blue color. In the table, * denotes the models are pre-trained using the additional datasets, including static image datasets [, , , , ] and the large BL30K dataset, and + means using a larger input image size while testing.**
| Dataset | MOSE-val | LVOS test | DAVIS 2017 test | YouTube-VOS 2018 val | YouTube-VOS 2019 val | FPS |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{G}$ | $\mathcal{J}_{s}$ | $\mathcal{F}_{s}$ | $\mathcal{J}_{u}$ | $\mathcal{F}_{u}$ | $\mathcal{G}$ | $\mathcal{J}_{s}$ | $\mathcal{F}_{s}$ | $\mathcal{J}_{u}$ | $\mathcal{F}_{u}$ |  |
| Trained on the YouTube VOS, and DAVIS datasets |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| STM [] * | - | - | - | - | - | - | - | - | - | 79.4 | 79.7 | 84.2 | 72.8 | 80.9 | - | - | - | - | - | - |
| AFB-URR [] * | - | - | - | - | - | - | - | - | - | 79.6 | 78.8 | 83.1 | 74.1 | 82.6 | - | - | - | - | - | - |
| CFBI [] * | - | - | - | 50.0 | 45.0 | 55.1 | 74.8 | 71.1 | 78.5 | 81.4 | 81.1 | 85.8 | 75.3 | 83.4 | 81.0 | 80.6 | 85.1 | 75.2 | 83.0 | 2.9 |
| MiVOS []* | - | - | - | - | - | - | 78.6 | 74.9 | 82.2 | 82.6 | 81.1 | 85.6 | 77.7 | 86.2 | 82.4 | 80.6 | 84.7 | 78.1 | 86.4 | - |
| STCN [] * | 52.5 | 48.5 | 56.6 | 45.8 | 41.1 | 50.5 | 77.8 | 74.3 | 81.3 | 84.3 | 83.2 | 87.9 | 79.0 | 87.3 | 84.2 | 82.6 | 87.0 | 79.4 | 87.7 | 13.2 |
| Swin-B-AOT-L [] * | 59.4 | 53.6 | 65.2 | 59.4 | 53.6 | 65.2 | 81.2 | 77.3 | 85.1 | 85.1 | 85.1 | 90.1 | 78.4 | 86.9 | 85.3 | 84.6 | 89.5 | 79.3 | 87.7 | 12.1 |
| DeAOT-R50 [] | 59.0 | 54.6 | 63.4 | - | - | - | 80.7 | 76.9 | 84.5 | 86.0 | 84.9 | 89.9 | 80.4 | 88.7 | 85.6 | 84.2 | 89.2 | 80.2 | 88.8 | 11.7 |
| XMem | - | - | - | - | - | - | 79.8 | 76.3 | 83.4 | 84.3 | 83.9 | 88.8 | 77.7 | 86.7 | 84.2 | 83.8 | 88.3 | 78.1 | 86.7 |  |
| XMem[] * | 53.3 | 62.0 | 57.6 | 50.0 | 45.5 | 54.4 | 81.0 | 77.4 | 84.5 | 85.7 | 84.6 | 89.3 | 80.2 | 88.7 | 85.5 | 84.3 | 88.6 | 80.3 | 88.6 | 22.6 |
| ISVOS [] * | - | - | - | - | - | - | 82.8 | 79.3 | 86.2 | 86.3 | 85.5 | 90.2 | 80.5 | 88.8 | 86.1 | 85.2 | 89.7 | 80.7 | 88.9 | 5.8 |
| SimVOS-B [] | 61.6 | 57.9 | 65.3 | - | - | - | 82.3 | 78.7 | 85.8 | - | - | - | - | - | 84.2 | 83.1 | 87.5 | 79.1 | 87.2 | 3.3 |
| Cutie []* | 64.0 | 60.0 | 67.9 | - | - | - | 84.2 | 80.6 | 87.7 | 86.1 | 85.5 | 90.0 | 80.6 | 88.3 | 86.1 | 85.8 | 90.5 | 80.0 | 88.0 | 36.4 |
| Cutie []*+ | 66.2 | 62.3 | 70.1 | 56.2 | 51.8 | 60.5 | 85.9 | 82.6 | 89.2 | - | - | - | - | - | 86.9 | 86.2 | 90.7 | 81.6 | 89.2 | 17.9 |
| JointFormer [] | - | - | - | - | - | - | 87.0 | 83.4 | 90.6 | 86.0 | 86.0 | 91.0 | 79.5 | 87.5 | 86.2 | 85.7 | 90.5 | 80.4 | 88.2 |  |
| JointFormer []* | - | - | - | - | - | - | 88.1 | 84.7 | 91.6 | 87.6 | 86.4 | 91.0 | 82.2 | 90.7 | 87.4 | 86.5 | 90.9 | 82.0 | 90.3 | 3.0 |
| Ours | 68.5 | 64.5 | 72.6 | - | - | - | 86.7 | 82.7 | 90.8 | 87.4 | 87.0 | 92.0 | 80.9 | 89.7 | 87.5 | 86.8 | 91.8 | 81.3 | 89.9 | 8.0 |
| Ours+ | 70.5 | 66.5 | 74.6 | 66.5 | 62.1 | 70.8 | 87.8 | 84.6 | 91.3 | 87.6 | 86.9 | 91.7 | 81.5 | 90.1 | 87.7 | 86.8 | 91.6 | 82.2 | 90.8 | 4.5 |
| Trained on the MEGA dataset |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| DEVA [] | 66.5 | 62.3 | 70.8 | 54.0 | 49.0 | 59.0 | 83.2 | 79.6 | 86.8 | 86.2 | 85.4 | 89.9 | 80.5 | 89.1 | 85.8 | 84.8 | 89.2 | 80.3 | 88.8 | 25.3 |
| Cutie [] * | 69.9 | 65.8 | 74.1 | - | - | - | 86.1 | 82.4 | 89.9 | 87.0 | 86.4 | 91.1 | 81.4 | 89.2 | 87.0 | 86.0 | 90.5 | 82.0 | 89.6 | 36.4 |
| Cutie []*+ | 71.7 | 67.6 | 75.8 | 66.7 | 62.4 | 71.0 | 88.1 | 84.7 | 91.4 | - | - | - | - | - | 87.5 | 86.3 | 90.6 | 82.7 | 90.5 | 17.9 |
| Ours | 74.0 | 69.8 | 78.3 | - | - | - | 87.8 | 84.0 | 91.7 | 88.0 | 87.0 | 91.8 | 82.5 | 90.7 | 88.1 | 87.4 | 92.5 | 81.9 | 90.7 | 8.0 |
| Ours+ | 75.1 | 71.0 | 79.2 | 73.0 | 68.3 | 77.8 | 89.1 | 85.8 | 92.4 | 88.5 | 87.6 | 92.6 | 82.7 | 91.3 | 88.5 | 87.3 | 92.0 | 83.1 | 91.4 | 4.5 |

YouTubeVOS 2018 []
has 3471 videos with 65 categories for training and videos for validation. In the validation, there are 26 categories that the model has not seen in its training, enabling us to evaluate its generalization ability for 350 class-agnostic targets.
Table 3 illustrates that our method achieves better performance on seen and unseen categories than previous state-of-the-art models.
The performance gain is attributed to spatial-semantic feature representation and representative target correlation.

YouTubeVOS 2019 is an extension of YouTube-VOS 2018, featuring a greater number of masked targets and including more challenging sequences in its val set.
In Table 3, our approach exhibits competitive performance relative to the leading state-of-the-art methods. Remarkably, our model achieves the highest performance (87.5%) without relying on pre-training.
This underscores the effectiveness of our method in leveraging inherent semantic-spatial information and discriminative query representation.

MOSE [] contains 2149 video clips and 5200 objects from 36 categories under more complex scenes.
Compared to existing state-of-the-art methods, our model gains great improvement in this dataset.
The benefits primarily stem from the discriminative query capturing more representative target information.
Additionally, the feature extraction module provides matching with more semantic and spatially informative features.

LVOS [] is a new long-term dataset to validate the robustness of VOS methods when facing these challenges.
It contains videos 20 times longer than those in the existing VOS datasets, typically featuring a single target per video. In this benchmark, our method achieves a performance gain (66.5% compared to 56.2%).
This improvement is attributed to the superior quality of memorized features and queries, which enhance the robustness of our approach in long-term video scenarios.
Our method demonstrates enhanced stability and effectiveness in long-duration contexts by incorporating more discriminative queries that capture crucial target information into the object memory.

Results with larger test size.
Following Cutie, we notice that the performance of our method can be further improved by changing hyperparameters, such as test input scales, memory interval, memorize or not, and the size of the memory bank during inference.
As is shown in Table 3, owing to the robust spatial-semantic feature learning and discriminative query propagation, our method achieves state-of-the-art performance on most benchmarks (e.g. 75.1% on MOSE [] val set).
Besides, our method is only trained with video data, without any pre-train, but still achieves better performance.

Visualized Results.
We visualize some challenging sequences, including scenarios involving part-to-whole changing and faint objects, as well as those where targets closely resemble the background.
In Figure 3(a), our approach shows favorable segmentation performance against the state-of-the-art methods.
For instance, our model accurately segments targets and provides detailed results in a video that includes gorillas, necks, and a riding man.
This is attributed to our proposed discriminative query propagation, which enables our model to capture more detailed target information by focusing on the most representative features, thereby facilitating precise segmentation in these challenging scenarios
Additionally, incorporating semantic features in our model ensures the model understands the semantic prior of the interested targets, such as in the case of the sequence of man riding from YoutubeVOS 2019.

Figure 3(b) shows the segmentation and tracking performance on long-term samples with fast motion and dramatic appearance variations.
In this sequence, two similar dogs continuously change their position and posture, which is challenging for the model to identify and track them accurately.
Our VOS method performs accurately in tracking and segmenting the targets, whereas Cutie fails to
identify and segment the targets in this scenario.
Good performance in scenarios like this is due to the semantic-aware feature and discriminative query representation learned by our model.
More visualization and videos can be found in the appendix.

Figure: Figure 3: Visualized results on challenging scenarios. The proposed method performs well on these challenging sequences, while other methods suffer from semantic understanding and thus cannot segment the whole objects. (b) Our method handles the challenging sequence with fast-motion objects better than Cutie [] .
Refer to caption: x3.png

## 5 Limitations

Our method enhances target representation by introducing spatial-semantic features and discriminative query propagation, achieving significant improvements in numerous challenging scenarios.
However, in the scenes with object parts as the target (some sequences in the VOT challenge, shown in the appendix), the proposed model finds it difficult to represent a part of the target and often extends the predicted mask to the whole target.
For example, when the target is the head of a person, our model may gradually shift to segmenting the entire body.
This tendency arises because our method tends to interpret the target as the entire object seen during the training phase.
A potential direction to improve this issue could be letting the model learn more part-aware semantic features, which also needs a new benchmark.

## 6 Conclusion

In this paper, we propose a spatial-semantic network to learn comprehensive representations to improve the robustness of video object segmentation in handling objects with complex structures or separate parts.
In addition, we develop a discriminative query propagation module that selects representative features in target objects for query update, enabling more reliable query propagation in long-term videos.
Our approach achieves favorable performance compared with the state-of-the-art methods on numerous datasets.
Both quantitative and qualitative results demonstrate the effectiveness of the proposed method in tracking and segmenting video objects in long-term and complex scenes.

## Appendix A Appendix

The appendix is organized as follows:

1. We provide a detailed implementation of training and inference.

2. We provide more experiments with different training settings and test hyperparameters.

3. More visualization and failure cases are shown.

### A.1 Implement Details.

#### A.1.1 Multi-Object Processing

In this section, we provide additional implementation details for completeness. Our training and inference code will be released to ensure reproducibility.

Following the previous work [, ], we extend our methods to the multi-object setting.
The predicted mask of each frame undergoes channel separation for different targets.
The separated masks for each target are then concatenated with the RGB image and the aggregated mask of other targets for feature extraction.
The extracted features are fused with RGB features to obtain pixel-level value features, which are then stored in memory.
It’s important to note that the pixel-level features stored in memory include both the key (RGB feature) and the value.

#### A.1.2 Training Details

As mentioned in the main paper, we train our network only on video-level datasets, which is different from other methods [, , ].
The backbone weights are initialized from MAE pretraining, similar to the approaches used in SimVOS [] and JointFormer [].
We conduct spatial-semantic fusion every three ViT layers and five discriminative query transformer blocks for query propagation.
We implement our network using PyTorch and train the model on 8 NVIDIA V100s, employing automatic mixed precision (AMP) during training.

Two different training datasets are used for better improvements.
The "DAVIS$\&$YTVOS" settings combine the training sets of YouTubeVOS-2019 and DAVIS-2017, with the DAVIS-2017 dataset expanded five times to increase the data ratio.
The "MEGA" set contains four different training sets, including DAVIS-2017, YouYubeVOS-2019, MOSE, and BURST datasets, with the DAVIS-2017 dataset expanded five times to increase the data ratio.
To sample a training sequence, we first randomly select a "seed" frame from all the frames and then randomly select 7 other frames from the same video.
We re-sample if any two consecutive frames have a temporal distance greater than the max-skip.
Following the previous work [, ], the max-skip is set to [5, 10, 15, 5] after [0%, 10%, 30%, 80%] of the training iterations, respectively.
To avoid sampling frames without targets, we use the data statistics provided by Cutie [].

**Table A1: Training parameters.**
| Config | DAVIS YTVOS | MEGA |
| --- | --- | --- |
| optimizer | AdamW | AdamW |
| base learning rate | 5e-5 | 5e-5 |
| weight decay | 0.05 | 0.05 |
| droppath rate | 0.15 | 0.15 |
| batch size | 16 | 16 |
| num ref frames | 3 | 3 |
| num frames | 8 | 8 |
| max-skip | [5, 10, 15, 5] | [5, 10, 15, 5] |
| max-skip-itr | [0.1,0.3,0.8,1] | [0.1, 0.3, 0.8, 1] |
| Iterations | 150,000 | 190,000 |
| learning rate schedule | steplr | steplr |

We conduct data augmentation with methods of random horizontal mirroring, random affine transformations, and cut-and-paste.
The training samples are also resized by crop (480 x 480).
Then, random color jittering and random grayscaling are performed in the training samples.
The same augmentation is performed in the same video for stability.
To ensure the images can be input into the patch embedding of ViT, we apply padding to both the images and masks so that their dimensions are multiples of 32.

In the training phase, we initialize the model with the first frame and ground truth.
When the number of past frames is three or fewer, we use all of them as memory frames.
If there are more than three past frames, we randomly sample three of them to be the memory frames.
We compute the loss at all frames except the first one and perform back-propagation through time.

**Table A2: Additional ablation study.**
| Dataset | MOSE-val | DAVIS 2017 test | LVOS test | YouTube-VOS 2019 val |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{G}$ | $\mathcal{J}$ | $\mathcal{F}$ | $\mathcal{G}$ | $\mathcal{J}_{s}$ | $\mathcal{F}_{s}$ | $\mathcal{J}_{u}$ | $\mathcal{F}_{u}$ |
| Trained on the YouTubeVOS, and DAVIS datasets |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| XMem [] (Baseline) | 53.3 | 62.0 | 57.6 | 81.0 | 77.4 | 84.5 | 50.0 | 45.5 | 54.4 | 85.5 | 84.3 | 88.6 | 80.3 | 88.6 |
| Cutie [] | 64.0 | 60.0 | 67.9 | 84.2 | 80.6 | 87.7 | 56.2 | 51.8 | 60.5 | 86.1 | 85.8 | 90.5 | 80.0 | 88.0 |
| +Discriminative Query | 64.2 | 60.3 | 68.1 | 85.2 | 81.8 | 88.5 | 57.4 | 53.3 | 61.5 | 86.5 | 86.2 | 90.7 | 80.6 | 88.8 |
| +ViT | 64.2 | 60.2 | 68.3 | 85.6 | 82.0 | 89.2 | 58.3 | 53.7 | 62.8 | 86.7 | 86.4 | 90.3 | 81.0 | 88.7 |
| +Spatial | 68.2 | 64.0 | 72.4 | 86.2 | 82.4 | 90.1 | 67.4 | 62.9 | 71.9 | 87.3 | 86.4 | 91.3 | 81.5 | 90.3 |
| +Semantic (Full) | 68.5 | 64.5 | 72.6 | 86.7 | 82.7 | 90.8 | 66.5 | 62.1 | 70.8 | 87.5 | 86.8 | 91.8 | 81.3 | 89.9 |
| Improved test size (600/720) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Cutie [] | 66.2 | 62.3 | 70.1 | 85.9 | 82.6 | 89.2 | 56.2 | 51.8 | 60.5 | 86.9 | 86.2 | 90.7 | 81.6 | 89.2 |
| +Discriminative Query | 66.4 | 62.4 | 70.1 | 87.9 | 84.6 | 91.2 | 57.4 | 53.3 | 61.5 | 87.1 | 86.3 | 90.6 | 82.0 | 89.5 |
| +Spatial | 69.9 | 67.5 | 74.1 | 87.0 | 83.7 | 90.2 | 67.4 | 62.9 | 71.9 | 87.5 | 86.8 | 91.6 | 81.6 | 90.2 |
| +Semantic ( full) | 70.5 | 66.5 | 74.6 | 87.8 | 84.6 | 91.3 | 66.5 | 62.1 | 70.8 | 87.7 | 86.8 | 91.6 | 82.2 | 90.8 |
| Trained on the MEGA datasets |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Cutie [] | 69.9 | 65.8 | 74.1 | 86.1 | 82.4 | 89.9 | 66.7 | 62.4 | 71.0 | 87.0 | 86.0 | 90.5 | 82.0 | 90.7 |
| +Discriminative query | 70.6 | 66.5 | 74.6 | 86.6 | 82.7 | 90.5 | 66.5 | 62.1 | 70.8 | 87.5 | 86.0 | 90.6 | 82.8 | 90.6 |
| +Spatial | 73.5 | 69.1 | 77.7 | 87.6 | 83.8 | 91.5 | 68.8 | 64.4 | 73.1 | 87.9 | 86.9 | 91.8 | 82.3 | 90.4 |
| +Semantic (Full) | 74.0 | 69.8 | 78.3 | 87.8 | 84.0 | 91.7 | 73.0 | 68.3 | 77.8 | 88.1 | 87.4 | 92.5 | 81.9 | 90.7 |
| Improved test size (600/720) |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| Cutie [] | 71.7 | 67.6 | 75.8 | 88.1 | 84.7 | 91.4 | 66.7 | 62.4 | 71.0 | 87.5 | 86.3 | 90.6 | 82.7 | 90.5 |
| +Discriminative query | 71.6 | 67.7 | 75.5 | 88.1 | 84.6 | 91.4 | 66.5 | 62.1 | 70.8 | 88.0 | 86.2 | 90.6 | 83.6 | 91.5 |
| +Spatial | 75.3 | 71.3 | 79.2 | 89.0 | 85.8 | 92.3 | 68.8 | 64.4 | 73.1 | 88.3 | 87.0 | 91.9 | 83.0 | 91.1 |
| +Semantic(Full) | 75.1 | 71.0 | 79.2 | 89.1 | 85.8 | 92.4 | 73.0 | 68.3 | 77.8 | 88.5 | 87.3 | 92.0 | 83.1 | 91.4 |

#### A.1.3 Inference Details

During the inference phase, the model memorizes the feature every 5th frame like  [, ].
The pixel-level memory always stores the first frame and its mask.
The top-K filter is used to augment the memory-reading in pixel memory.
For object-level memory, we use the streaming average to accumulate the discriminative query representation like the operation in  [].

### A.2 Additional Quantitative Results

In this section, we provide more additional ablation studies and sota comparisons with different settings.
In the main paper, we provide ablation studies for different components.
To provide a broader comparison, we conduct tests under various combinations of training data and testing settings.
The experimental results in Table A2 detail the impact of different components of our model on the performance, facilitating a better understanding of the effectiveness of our model.

To compare with more methods on MOSE, we follow the training strategy from MOSE [], and replace the YouTubeVOS with MOSE in the training process.
Table A3 shows the results of the variants of our method.
All the variants achieve great improvement in terms of $\mathcal{J}\&\mathcal{F}$, which further validates the effectiveness of the proposed spatial-semantic feature learning and discriminative query representation.

**Table A3: Results only trained on MOSE datasets.**
| Dataset | MOSE-val |  |  |
| --- | --- | --- | --- |
| Method | $\mathcal{J}\&\mathcal{F}$ | $\mathcal{J}$ | $\mathcal{F}$ |
| RDE [] | 48.8 | 44.6 | 52.9 |
| STCN [] | 50.8 | 46.6 | 55.0 |
| AOT [] | 57.2 | 53.1 | 61.3 |
| XMem [] | 57.6 | 53.3 | 62.0 |
| DeAOT [] | 59.4 | 55.1 | 63.8 |
| ResNet+Discriminative query | 69.9 | 65.8 | 73.9 |
| +Spatial | 72.7 | 68.3 | 77.0 |
| +Semantic | 72.9 | 68.4 | 77.3 |
| Improved test size (720) |  |  |  |
| ResNet+Discriminative query | 71.6 | 67.7 | 75.5 |
| +Spatial | 74.0 | 69.9 | 78.1 |
| +Semantic | 74.5 | 70.2 | 78.5 |

Figure: Figure A.1: Additional qualitative comparison against the state-of-the-art methods including XMem, Swin-DeAOT, and Cutie. All the sequences are picked from YouTubeVOS 2019 and DAVIS 2017 datasets.
Refer to caption: x4.png

Figure: Figure A.2: Additional qualitative comparison on sequence with giant appearance variance. All the sequences are picked from MOSE [] dataset.
Refer to caption: x5.png

### A.3 Additional Qualitative Comparison

We present a more qualitative comparison of most benchmarks in Figure A.1 and Figure A.2.

In Figure A.1, we categorize the scenarios into three main types: overall-to-part segmentation (man on bike and deer), small object segmentation (small monkey), blurry object segmentation (cat toy), and segmentation under varying lighting conditions (man-bike and golf).
In these three cases, our method demonstrates precise segmentation compared to other methods. The semantic embedding allows our model to consider semantic information when parsing objects, spatial dependence modeling enhances the understanding of spatial positions and details of the object, and the discriminative query accurately represents the target.
The integration of these three components enables the model to achieve excellent segmentation and tracking performance in complex and challenging scenarios.

In Figure A.2, we select more sequence results on MOSE, primarily divided into two categories: positional changes of similar targets and rapid movement of targets.
In both cases, our method outperforms Cutie.
This is attributed to our proposed spatial-semantic feature representation and discriminative query expression, which enable the model to distinguish appearance-similar targets and consistently track the target.

Figure A.3 compares sequences that require a semantic understanding of the target.
Without the semantic embedding module, the method suffers from segmenting parts of the target.
The addition of the semantic embedding module enables the model to fully understand the semantic information of the target, allowing for precise segmentation and tracking in these scenarios.

Failure cases are shown in Figure A.4, in which our method tends to segment the whole instance rather than the part of them.
The challenging sequences need the model to segment and track the head of the man, flamingo, and kangaroo.
Our model interprets semantics as the entire instance during tracking and segmentation, such as a person, bird, or kangaroo, resulting in segmentation outcomes gradually favoring the whole instance.

Figure: Figure A.3: Qualitative comparison between with and without semantic embedding block. All the sequences are picked from YouTubeVOS 2019 dataset.
Refer to caption: x6.png

Figure: Figure A.4: Failure cases. All the sequences are picked from VOT Challenge [].
Refer to caption: x7.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weakness
soundness: 2
strengths: 1. The motivation is clear and the architecture makes sense. Integrating high-level semantics and low-level spatial cues is promising in video object segmentation.
2. The experiments are thorough and the ablation studies can well reflect the effectiveness of each module.
summary: This paper focuses on video object segmentation. The authors analyze the existing challenges like structural complexity, occlusion, and dramatic appearance changes, and correspondingly propose spatial-semantic feature augmentation as well as discriminative query association. The ablation studies and visualizations verify the effectiveness of each module.
weaknesses: 1. The method is complicated. What is the advantage of using spatial offsets with deformable convolution compared to simple position encodings?
2. The second row of Figure 3(a) seems strange. With semantic feature augmentation, the feature maps can well highlight the desired object instance. Adding spatial cues on the contrary suppresses the emphasis on the target instance but enhances object instances with the same semantics.
3. Compared to SAM2, which designs a memory to prompt the segmentation of new frames, what is the advantage of this architecture?

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: See weakinesses
soundness: 3
strengths: This paper’s S3 algorithm for Video Object Segmentation (VOS) demonstrates notable strengths:

1.Spatial-Semantic Integration: By combining semantic embedding with spatial dependency modeling, it effectively captures complex object structures without requiring extensive ViT retraining.

2.Discriminative Query Mechanism: The adaptive query approach improves target focus and reduces noise in long-term tracking, enhancing robustness.

3.Extensive Validation: State-of-the-art results on multiple benchmarks highlight its strong generalization across datasets.
summary: This paper addresses the complex task of tracking and segmenting multiple similar objects in long-term videos, where identifying target objects becomes challenging due to factors like occlusion, cluttered backgrounds, and appearance changes. To tackle these issues, the authors propose a new framework for robust video object segmentation, focusing on learning spatial-semantic features and generating discriminative object queries.

The framework introduces a spatial-semantic block that combines global semantic embedding with local spatial dependency modeling, which enhances the representation of target objects by capturing both broad context and fine details. Additionally, a masked cross-attention module refines object queries, concentrating on the most distinctive parts of target objects and reducing noise accumulation over time. This approach aids in effective long-term query propagation, a critical factor for high-performance tracking over extended sequences.

The experimental results are strong, showing state-of-the-art performance across several benchmarks.
weaknesses: 1.This paper claims to address the challenges of long-term tracking and segmentation. However, as far as I know, memory mechanisms are crucial for tackling these challenges in long-term tracking and segmentation, yet the authors do not seem to have conducted ablation experiments on the number of frames in the memory bank.

2.I believe that the ablation study on the number of queries is insufficient with only 8, 16, and 32 as tested values. A wider range of query counts should be explored to more thoroughly validate the effectiveness of the proposed method.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1.In Figure 2, there are N spatial-semantic blocks, but N is not specified later in the paper.

2.In Figure 2, the argmax operation in the distinctive query propagation is non-differentiable. Will this prevent the gradient from being propagated through the model?

3.If the introduced ViT backbone is not fine-tuned, will its performance degrade on the new dataset? A comparison experiment between freezing and not freezing the parameters is needed here.

4.The number of different queries should be related to the number of targets. However, using 8 queries yields better results. When faced with more than 8 targets, can 8 queries adequately represent the different targets?

5.In Table 3, there are two XMem entries, one of which is not referenced. It is unclear what the unreferenced entry represents, and why it lacks FPS results needs to be clarified.

6.Table 3 lacks a comparison of Joint Former results trained on the MEGA dataset. Please provide the results for Joint Former trained on the MEGA dataset in detail. If the original Joint Former was not trained on this dataset, can it be trained and then compared for performance?

7.The spatial-semantic block consists of two parts: first, the global feature cls token is fused with the semantic features, and then further enhanced through Deformable Cross Attention. It is necessary to separately validate the effects of directly fusing the features versus applying Deformable Cross Attention for further enhancement.
soundness: 3
strengths: This paper presents a spatial-semantic modeling method and a discriminative query mechanism that significantly enhance the model's performance. Extensive experiments have been conducted to demonstrate the effectiveness of the model, and several visual examples are provided to clearly illustrate the results at different processing stages. Additionally, the final results showcase the model's considerable potential.
summary: This paper presents a novel spatial-semantic block that effectively integrates semantic information with spatial features, resulting in a more comprehensive representation of target objects, especially those with complex or distinct parts. By utilizing a pre-trained Vision Transformer (ViT) backbone without the need to retrain all parameters, the proposed method significantly enhances the efficiency of video object segmentation (VOS).

Additionally, the development of a discriminative query mechanism marks a substantial advancement in the field. This mechanism prioritizes the most representative regions of target objects, thereby improving the reliability of target representation and query updates. This is particularly advantageous in long-term video scenarios, where appearance changes and occlusions can lead to noise accumulation during query propagation.

The authors also highlight the importance of learning comprehensive target features that encompass semantic, spatial, and discriminative information. This holistic approach effectively addresses challenges related to appearance variations and identity confusion among similar-looking objects in long-term videos, making it a valuable contribution to the VOS community.

Finally, extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance across multiple benchmark datasets, including DAVIS 2017, YouTube VOS 2019, MOSE, and LVOS.
weaknesses: Writing Style:
1. The writing language is not concise enough, with many long sentences that significantly reduce readability. This is particularly evident in the introduction, such as on the second page: "We construct a Spatial-Semantic Block comprising a semantic embedding module and a spatial dependencies modeling module to efficiently leverage the semantic information and local details of the pre-trained ViTs for VOS without training all the parameters of the ViT backbone."

Image Details:
1. In Figure 2, there are N spatial-semantic blocks, but N is not specified later in the paper.

Method:
1. In Figure 2, the argmax operation in the distinctive query propagation is non-differentiable. Will this prevent the gradient from being propagated through the model?

2. If the introduced ViT backbone is not fine-tuned, will its performance degrade on the new dataset? A comparison experiment between freezing and not freezing the parameters is needed here.

3. The number of different queries should be related to the number of targets. However, using 8 queries yields better results. When faced with more than 8 targets, can 8 queries adequately represent the different targets?

4. In Table 3, there are two XMem entries, one of which is not referenced. It is unclear what the unreferenced entry represents, and why it lacks FPS results needs to be clarified.

5. Table 3 lacks a comparison of Joint Former results trained on the MEGA dataset. Please provide the results for Joint Former trained on the MEGA dataset in detail. If the original Joint Former was not trained on this dataset, can it be trained and then compared for performance?

6. The spatial-semantic block consists of two parts: first, the global feature cls token is fused with the semantic features, and then further enhanced through Deformable Cross Attention. It is necessary to separately validate the effects of directly fusing the features versus applying Deformable Cross Attention for further enhancement.

---

confidence: 5
contribution: 4
initial_rating: 6
presentation: 4
questions: 1、	How many trainable parameters and total parameters does the model have?
2、	Why does "DepthAnything" achieve the best results?
3、	If the backbone of the Cutie model is replaced with ViT, would it achieve good results?
soundness: 4
strengths: This paper addresses key issues in current applications of VOS methods: long-term tracking, occlusion, and object representation changes. 
1.	The proposed approach utilizes semantic and spatial information from upstream pre-trained models, enriching the target's semantic and detailed information. It is novel in the field of VOS. 
2.	The paper proposes a discriminative query generation mechanism to provide the model with more distinctive target information, which is validated on LVOS datasets.
3.	The proposed method is validated on various VOS datasets and achieves state-of-the-art results.
summary: The authors focus on the problem of video object segmentation in long-term tracking and complex environments. To improve the model's robustness, a spatial-semantic network block is proposed to integrate semantic information with spatial information for video object segmentation. Additionally, a discriminative query mechanism is developed to capture the most representative region of the target for better target representation learning and updating. The proposed method achieves state-of-the-art results on most VOS dataset.
weaknesses: The paper does not have obvious weaknesses, but there are still some issues. 
1.	In Table 1, why is there no separate ablation study for the spatial block and semantic block? Please provide this part of the experiment.
2.	Some detail issues: In Figure 3, the blue results are not clearly marked and the position of * is not aligned.

Meta-review: This paper proposes a robust video object segmentation framework by learning spatial-semantic features and discriminative object queries to address challenges encountered in the video object segmentation task. The overall idea seems like a combination of existing techniques, yet experimental results are solid. Four reviewers reach a consensus to accept this paper by discussing it with the authors through rebuttals. After reviewing the comments and rebuttals, AC agrees with the merits proposed in this paper. Therefore, AC recommends accepting this paper.

Decision: Accept (Poster)