Learning Spatial-Semantic Features for Robust Video Object Segmentation
Authors: Xin Li
Deshui Miao
Zhenyu He
Yaowei Wang
Huchuan Lu
Ming-Hsuan Yang
Venue: ICLR 2025 Poster (2025)
Keywords: Video Object Segmentation
Spatial-Semantic Feature
Long-Term
Discriminative Object Queries
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Tracking and segmenting multiple similar objects with distinct or complex parts in long-term videos is particularly challenging due to the ambiguity in identifying target components and the confusion caused by occlusion, background clutter, and changes in appearance or environment over time. In this paper, we propose a robust video object segmentation framework that learns spatial-semantic features and discriminative object queries to address the above issues. Specifically, we construct a spatial-semantic block comprising a semantic embedding component and a spatial dependency modeling part for associating global semantic features and local spatial features, providing a comprehensive target representation. In addition, we develop a masked cross-attention module to generate object queries that focus on the most discriminative parts of target objects during query propagation, alleviating noise accumulation to ensure effective long-term query propagation. The experimental results show that the proposed method sets new state-of-the-art performance on multiple data sets, including the DAVIS2017 test (\textbf{87.8\%}), YoutubeVOS 2019 (\textbf{88.1\%}), MOSE val (\textbf{74.0\%}), and LVOS test (\textbf{73.0\%}), which demonstrate the effectiveness and generalization capacity of the proposed method. We will make all the source code and trained models publicly available.

Published as a conference paper at ICLR 2025
LEARNING
FEATURES FOR ROBUST VIDEO OBJECT SEGMENTATION
Xin Li Deshui Mi Zhenyu Heale, Yaowei Wang". Huchuan Lu', and Ming-Hsuan Yang !engghhnng Laboratory Harbin Institute of Technology, Shenzhen Pazhou Lab (Huangpu) *Dalian University of Technology UUiiversity of California at Merced Yonsei University
ABSTRACT
Tracking and segmenting multiple similar objects with distinct or complex parts in long-term videos is particularly challenging due to the ambiguity in identifying target components and the confusion caused by occlusion, background clutter, and changes in appearance environment over time. In this paper, propose robust video object segmentation framework that learns spatial-semantic features and discriminative object queries to address the above issues. Specifically, construct a appatil--semanin block comprising a semantic embedding component and a spatial dependency modeling part for associating global semantic features and local spatial features, providing a comprehensive target representation. In addition, we develop masked cross- cooss-attention module to generate object queries that focus the most discriminative parts of target objects during query propagation, alleviating noise accumulation to ensure effective long-term query propagation. Extensive experi- mental results show that the proposed method achieves state-of-ther of-theeart performance on benchmark data sets, including the DAVIS2017 test (87.8% 8%).. ), Youtube 2019 (88.1 )) MOSE val (74.0%), and LVOS test (73.0 %) ), and demonstrate the effectiveness and generalization capacity of our model. The source code and trained models are released at https//gitub..cm/yaaoooo.
INTRODUCTION
Video Object Segmentation (VOS) aims to track and segment target objects specified in the initial frame with mask annotations a video sequence (Xu et al., 2018; Pont-Tuset et al., 2017: Voigtlaender et al., 2019; Yang et al., 2021). This practice holds significant promise in various applications, particularly as the prevalence video content increases in domains such as autonomous driving, augmented reality, and interactive video editing (Park al., 2022; Bao et al., 2018). The main challenges faced by VOS include drastic target appearance change, occlusion, and identity confusion caused by similar objects and background clutter, which becomes more difficult in long-term videos.
Existing VOS methods (Oh et al., 2019; Cheng et al., 2021 typically perform video object segmen- tation by comparing the test frame with past frames. Specifically, they first use an association model to generate the correlated features of the test sample and target templates. Then, the target masks are predicted based on the correlated features. The pixel-wise correlated features (Wu et al., 2023) contribute to accurate mask predictions. To account for targets that appear differently over time, some methods (Yang al., 2021; Cheng & Schwing, 2022) utilize a memory module to store these changing appearances. In addition, several recent approaches ( (Wang et al., 2023; Cheng et al., 2023a) introduce object queries to help distinguish different target objects to alleviate identity confusion.
Although significant advances have been made in video object segmentation, existing methods do not perform well in challenging scenarios. First, when dealing with target objects with multiple complex separate parts caused by occlusion, background clutter, and shape complexity, existing methods often generate incomplete prediction masks. This is because the pixel-wise correlation adopted by existing methods mainly focuses detailed spatial information at the pixel level and neglects the semantic information at the instance level. Second, although object query improves the ID association
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
LVOS-val
LVOS-test
67.7
66.9
60.1
56.2
54.1
D17-test
49.
50.0
45.8 45.8
MOSE
86.784.2
1.2
76.0
0.88
57.267. 67.6
83.0 82.7
85.1
85.7
86 1
86.1
87.4 4
87.5
YT18
YT19
STCN
AOT
XMem
Cutie
Ours
Figure I: Performance comparison in terms of the J&F score various datasets. The proposed method achieves favorable performance against state-of-the-ar methods, including AOT L (Yang al., 2021), XMem (Cheng & Schwing, 2022), and Cutie (Cheng al., 2023a) on all the datasets. All methods are trained using the Youtube VOS and DAVIS datasets.
accuracy, it does not work well in sequences with dramatic target appearance changes. Existing methods (Wang al., 2023; Cheng al., 2023a) update target queries based on the entire predicted sample, which can introduce noise and errors and further lead to tracking failures when targets are missed their identities are swapped.
To address the aforementioned issues, we propose VOS algorithm, called S3, to learn spatial- semantic features and generate discriminative object queries robust VOS. Specifically, we design spatial semantic block with semantic embedding module and spatial dependencies modeling part. This block efficiently utilizes the semantic information and local details from pre-trained ViTs for VOS, avoiding the need to train all VIT backbone parameters. In addition, query that emphasizes representative target parts, enabling more reliable target representation and query updates. This approach mitigates noise accumulation during query propagation and enhances robustness in long-term videos. We evaluate the proposed method extensively on five benchmarks, including DAVIS 2017 (Pont-Tuset et al., 2017). Youtube VOS2018 (Xu al., 2018), Youtube VOS2019, LVOS (Hong et al., 2023), and MOSE (Ding et al., 2023). Extensive experimental results show that our approach achieves ffate-of-the-an the-art performance on all these datasets including the DAVIS2017 test (89.1%) %). Youtube VOS 2019 (88.5%), MOSE (75.1% %), and LVOS test (73.0% datasets. addition, the results achieved by only using the DAVIS and Youtube datasets for training show that our method generalizes well to different scenarios, as shown in Figure 1.
propose a discriminative
In this work, we make the following contributions:
We propose a spatial-semantic network block to incorporate semantic information with spatial information for Video Object Segmentation. The spatial-semantic block integrates global semantic information from the CLS token of a pre-trained ViT backbone into the base features of the input sample and then models spatial dependencies using a spatial dependency modeling module, enhancing robust VOS performance when handling target objects with complex separate parts.
2

---

Published as a conference paper at ICLR 2025
We develop discriminative query mechanism to capture the most representative region of the target for better target representation learning and updating. This significantly improves VOS datasets, especially in long-term VOS datasets.
We demonstrate that the proposed method achieves state-of ofte-ooftthe-rt performance on five di- verse datasets and evaluate the contribution of each proposed component with comprehensive ablation studies.
RELATED WORK
Video Object Segmentation. Video Object Segmentation (VOS) aims separate objects from the background and identify each target object across frames based on the target mask annotations given in the initial frame. Early VOS methods (Caelles et al., 2017; Xiao et al., 2018; Voigtlaender & Leibe, 2017; Hu et al., 2017: Luiten et al., 2018) use test-time learning to adapt pre-trained segmentation models to separate the specified targets online. The test-time learning of deep model is time-consuming and significantly reduces the run speed. To avoid online learning, propagation-basee methods (Duke et al., 2021; Cheng et al., 2018; Tsai al., 2016) offline shift attention model temporal correlations between adjacent frames, propagating target masks from the previous frame to the current one. Despite promising performance, these methods suffer from error accumulation caused by inaccurate predictions. Matching- based methods (Chen et al., 2018 8: Hu et al., 2018; Bhat al., 2020; Yang et al., 2020) enable fast and efficient inference by comparing the target template with the test image and predicting target masks based on feature matching. For example, OML (Chen et al., 2018) and VideoMatch (Hu et al., 2018) perform pixel matching between adjacent frames for mask prediction. Several methods (Oh et al., 2019; Yang et al., 2021; Yang & Yang, 2022; Cheng & Schwing, 2022) improve the matching- bbased VOS framework by introducing target memory to enrich target templates and developing advanced matching strategies to generate more comprehensive correlated features. For example, SimVOS (Wu et al., 2023) and JointFormer (Mao et al., 2021) use VIT blocks (Dosovitskiy et al., 2021) to model features and patch correspondence for object modeling jointly. For better target association across frames, some recent methods (Wang et al., 2023; Cheng al., 2023a) introduce object queries to enrich target representation. Despite good performance on simple target objects, existing methods lack modeling of object-level semantic information, so they do not work well on dealing with objects with complex structures and long-ter -eeg deformation. Unlike existing methods, the proposed approach performs pixel- and object-level feature learning through a novel object-aware backbone and a query-level correlation mechanism, enabling robust. vOS performance, particularly in complex and long-term video scenarios.
Query-Based VOS Frameworks. DETR (Carion et al., 2020) introduces the transformer net- work (Dosovitskiy et al., 2021) to object detection tasks by using queries to represent target regions. In the segmentation domain, some methods (Cheng et al., 2022; Cavagnero et al., 2024) introduce masked cross-attention modules use queries to represent targets for segmentation tasks. Mo- tivated by the above methods, ISVOS (Wang et al.. 2023) uses additional queries generated by Mask2Former (Cheng al., 2022) represent instance information for target modeling, enhanc- ing target representation in the matching-base based VOS framework. To enable end-to-end training, Cutie (Cheng et al., 2023a) learns object-level features and enhances the interaction between object- level and pixel-level information, leading to improved segmentation accuracy and efficiency. However, query-based VOS methods suffer from query drift due to ineffective propagation. These methods model the entire target sample in the query. incorporating redundant information that diminishes the saliency of the target object. Instead, the proposed method generates discriminative queries for target representation and propagation, which preserves key distinguishing information of target objects and offers greater robustness to variations.
Adapters for Dense Tasks. The ViT-Adapter (Chen et al., 2022) efficiently leverages pre-trained models by introducing a dense adapter for enhanced information exchange within ViT blocks, improving detection and segmentation performance. To enhance spatial hierarchical features, ViT- CoMer (Xia et al., 2024) introduces a nelti-receptive field feature pyramid module to provide multi-scale spatial features. However, both methods require complete fine-tuning step, which is inefficient, particularly in the video domain. Meanwhile, they focus on spatial information enhancement while lacking semantic feature interaction between ViT blocks and multi-scale features.
3

---

Published as a conference paper at ICLR 2025
To better adapt to VOS tasks, the proposed method improves adapter-based VOS frameworks in the following aspects. First, we introduce a spatial-semantic module to integrate the global semantic features from ViT multi-scale features. This module does not need extra modules to transmit information to ViT blocks, which is more efficient. Second, the proposed spatial-semantic integration block associates semantic priors with dense local features. Third, the proposed method freezes the ViT blocks during training, eliminating the need for full fine-tuning
3 PROPOSED ALGORITHM
Our method aims to learn comprehensive target features that contain semantic, spatial, and discrimi- native information for video object segmentation. This helps us cope with complex target appearance variations and ID confusion between target objects with similar appearances long-term videos. To this end, propose a spatial-semantic feature learning network that first embeds semantic information from a trained VIT network with multi-scale features from a trainable CNN network. Then, model spatial dependencies upon the fused feature, enabling spatial-semantic features for VOS. addition, to ensure that the target query retains discriminative information during long-term appearance changes, we develop diseriminative query propagation module to capture and update local representations of the target object.
3.1 OVERALL FRAMEWORK
Figure 2(a) shows the overall framework of the proposed method. consisting of two core modules: a spatial-semanti feature generation module and pixel-and-ques dual-level target association module. Given a test frame X, E 3x3 the proposed method predicts the mask of all specified objects within it based on the initial frame Xinit E 3R3* H XW and its corresponding annotation mask Minit HxW, First, the feature generation module takes the test frame X, as input and generates spatial-semantic features. Then, the target association module associates the spatial-semantic features with the reference samples pixel-wise and query-wise to better represent the correlated features. Finally, a decoder predicts the mask based on these correlated features. In addition, we develop dual-level memory module consisting of target features and target queries to better represent video targets that vary over frames. Both kinds of memory are updated every few frames using the online predicted masks, enabling adaptive target representations for effective target association.
3.2 SPATIAL-SEMANTES FEATURE LEARNING
Figure 2(b) shows the spatial-semantic network, which consists of a semantic embedding module that incorporates semantic information from pre-trained ViT model into multi-scale features and a spatial dependency modeling module that learns spatial dependencies based on semantic features.
Semantic embedding, As the VOS task is designed for generic objects and no class labels are given, is challenging to learn semantic representations directly from the VOS dataset during training. Fortunately, the CLS token in trained ViT model aggregates semantic information from the entire image and provides a rich, global representation of the image content. Thus, we incorporate the semantic features generated from the CLS token with the multi-scale features generated from CNN networks to obtain detailed semantic features at different scales. To be specific, we first concatenate the CLS token F with global token representation Fg.. which is generated by averagely pooling the features from a VIT block, to generate global semantic features denoted as (Fi... .F)). Then, we use cross attsntattnntion operation to generate the semantic-as aware features Fi by taking (Fii cla+ Fg) as the Key and Value, and the spatial-temporal features (Fimtm) from the (i - 1)-th spatial-semant block as the Query, which is formulated as:
(W9 clai (g) (W spsem)
Fi sem =Sofftman
(W"Fi Fipsem);
Vd
where f fmma is the SoftMax operation, and W W and W are the projection matrices for Query, Key, and Value
Spatial dependency modeling, We note that learning spatial dependencies between semantic features important for the VOS model to understand the relationships between different object parts, which
4

---

Published as a conference paper at ICLR 2025
VIT Block
Patch tokens
Putcs embolcings
Nvgiooo
Feattree generation
Softree
JOC Mar's TE
VIT
Modelng
Target ansodtation
Red love
Project
CofFFN
Hidl Canckac
Multionale
Semanti: emdealding
Sputial dependencics modeliag
Query
Spatial-Semantic
Dovodar
Decoder
Modeling Spalisl-Senamicc
lnk qarles
Addiicn
+
FPN
Object
memory
Sinllity
(5) Overall Framework
(c) Distinctive query propagation
Figure 2: Architecture of the proposed method. (a) shows the overall framework of the proposed method comprising a feature generation part, a target association component, and a decoder- based prediction part. The feature generation part contains multiple spatial- semantic blocks, as illustrated in (b), which learn spatiil-semantic features by integrating semantic priors and spatial details. (c) illustrates the discriminative query propagation process, which learns to generate discriminative queries that represent high-level object information.
helps handle objects with complex structures or separate parts. Given the features of the input image patches generated by ViT backbone model Fine. we adopt the Multi-scale Deformable Cross-Attention operation (Carion et al., 2020) to capture the spatial information F as:
F&pa - (Feem: Ap)) 'go (F Ap),
where Ap indicates the deformable offsets generated based on the query features Fo performs grid sampling on Fi using the offsets Ap. denotes the attention weights generated based on the query features Fost and the grid sampled features, and Σj the summation over all sampled points. To enable an effective long-range propagation through the cascade blocks, we use the residual connection (He et al., 2016) followed by spatial-semante feature of the current block:
wealuutioonfed-ffrww-ee to generate the final
F spseem Fi + +SCoFFFN (F) spa Fspsem)
F
(3)
The spatial-semantic block leverages the strengths of both pre-trained VIT network for provid- ing semantic information and a multi-scale CNN network for rich spatial information, enabling comprehensive apatial-semantic feature learning for robust VOS.
Figure 3(a) shows how the feature changes during the processing of the proposed model by visualizing the feature maps in different processing steps. It shows that the semantic embedding module enhances the semantic information (boundaries) of the objects in the feature maps. In contrast, after spatial dependence modeling, the feature maps pay more attention the details of the objects. Figure 3(b) compares the features generated by different networks. Compared to ViT and ResNet models, the proposed Spatial-Semantic model exhibits better target representation capabilities with significantly higher activations in target object areas.
3.3 DISCRIMINATIVE QUERY PROPAGATION
To propagate target queries effectively across frames, we update the target queries with the most distinctive feature of the target object.
5

---

Published as a conference paper at ICLR 2025
Frames
Original
+Semantic
"Spatial-Semamn
(a)
Visualized features at different process stages
Frames
ViT
ResNet
Ours SSpatial-Smantii
(b) Visualization of features generated by different networks
Figure 3: Visualized feature maps from different backbones and stages. It shows that the proposed spatial-semantic model generates effective features for target representation.
Discriminative feature generation. We select the discriminative feature of target object by comparing the target query Q E NxC with every channel activation in the correlated feature map of the target and taking the most similar one. Formally, given the feature map F R x xC the salient channel activation feature of a target is computed as:
Fl arg max HE(L.H-W)
(FQ" @.M)..
4
where denotes the dot product, M represents a binary mask of the target, and Fli] denotes the i-th channel activation of the feature map F. We use the strategy Cutie (Cheng et al., 2023a) to generate the target mask M, which uses the predicted segmentation results in the previous frame.
Query propagation. Based on the discriminative target feature Q. generated from the target sample and the input target query Qin at the current stage, the output query Qou is propagated as:
A
Q.n. = (a
+ 2.) + Qi..
(5)
|
where a is a learnable parameter to scale the matrix, Wour is learnable projection matrix, and A denotes the correspondence between salient query and salient pixel features, which computed
Q Q..
(6)
The proposed discriminative query propagation scheme adaptively refines target queries with the most representative features, which helps deal with the challenges of dramatic appearance variations in long-term videos. We also present the similarity map between the discriminative query and the feature in the last two rows, which indicates that our proposed discriminative query can represent and distinguish different objects.
EXPERIMENTS
4.1 EXPERIMENTAL SETTINGS
Network. The full version the proposed method uses a pre-trained ViT-base (Dosovitskiy et al., 2021) model the VIT branch and constructs simple CNN module for multi-scale feature generation.

---

Published as a conference paper at ICLR 2025
Table Ablation study, All the variants are well aligned using the same settings and trained on the Youtube VOS and DAVIS datasets. S3 denotes the proposed algorithm.
Dataset
MOSE val
DAVIS 2017 test
LYOS tstt
LYOS val
YouTube- VOS 2019 val
Jku
JULF
J
J
J:
XMen (Cheng Schving, 2022) (Baxelin))
53.3
02.0
57.5
$1.0
77.4
84.5
50.0
45.5
54.4
40.5
45.2
53.7
$5.5
$4.3
59.6
90.3
83.6
Cose (Cheng 20130)
56.2
51.8
55.9
99.5
-Deecimintttee Query
64.3
06.3
68.1
85.2
81.8
89.5
57.4
53.3
61.5
62.1
57.9
66.3
86.5
16.2
9437
90.6
88.8
89.2
58.3
537
-Spraal-emrriic (53)
68.
64.5
72.6
86.7
87.7
90.8
06.5
62.1
79.8
67.7
63.1
72.3
87.5
10.8
91.8
81.3
89.9
The Spatial-Semann block performs feature modeling after every block of the VIT branch. Upon the target association part, the Decoder predicts target masks based multi-scale features. We use soft aggregation operation for the scenario containing multiple target objects to merge the predicted masks.
In all experiments, we set the number of Spatial-Semantie Blocks to N=4.
Training. We adopt similar training settings as those of Cutie (Cheng al., 2023a). Note that, apart from ResNet pre-training, we do not pre-train other models on static images and directly trained them on video datasets (Youtube VOS (Xu et al., 2018 and DAVIS (Pont-Tuset al., 2017)). To enhance the performance of our model, we use the MEGA dataset constructed by Cutie, which includes the YouTubeVOS (Xu al., 2018), DAVIS (Pont-Tuset et al., 2017). OVIS (Qi et al., 2022). MOSE (Ding et al., 2023) and BURST (Athar et al., 2023) datasets. We sample 8 frames to train the model and randomly select of them to train the matching process. For each sequence, we randomly choose at most 3 targets for training. Point supervision in loss is adopted to reduce the memory requirements.
For optimization, the AdamW (Kingma & Ba, 2014) optimizer is used with a learning rate of 5e-5 and weight decay of 0.5. We train the model for 125K with a batch size of 16 on the video dataset, and 195k on the MEGA dataset. For specific parameter settings, please refer to our supplementary material. All our models are trained on a machine with 8 NVIDIA V100 GPUs and tested using one NVIDIA V100 GPU.
Our feature and query memories are updated every 5th frame during the testing phase. For longer sequences, we employ a long-term fusion strategy (Cheng & Schwing, 2022) for updating. We skip frames without targets. We two kinds of input sizes, 480 and 600/720, for more detailed comparisons, In this section, we compare all methods with input size 480p.
Inference.
4.2 ABLATION STUDY
Effect of every component. Table provides a comprehensive analysis of the components of our method. The first two rows describe the current methods upon which we build improvements. Cutie improves performance by using a query transformer within the model.
Baseline. We adopt XMem as the baseline model, representative VOS framework with well-designed memory mechanism.
+ Discriminative Query, which performs discriminative query generation by adaptively refin- ing target queries with the most representative features of the query transformer. To validate the effectiveness of our proposed discriminative query propagation, we first choose ResNet50 as the backbone for feature generation (simply comparing query propagation with Cutie (Cheng et al., 2023a)).. Performance gains compared to Cutie demonstrate the advantages of target modeling of discriminative queries for VOS, especially in long-term benchmarks.
VIT. which directly utilizes ViTDet (Li et al., 2022b) to incorporate pre-trained model features at multiple scales. The performance indicates that directly employing ViT to generate multi-scale features is minimally beneficial for VOS tasks.
S-patial-semattic, which applies the opatial-seaanee blocks to integrate semantic information and multi-scale features. It is clear that the proposed spatial dependency modeling significantly boosts model performance in terms of J&F on all VOS benchmarks. These improvements validate the benefits of the spatial-semantic network, which improves the semantic understanding of targets.
7

---

Published as a conference paper at ICLR 2025
Table Effect of different settings in every component. The ablation study examines the impact of pre-trained weights and the number of queries on training results for the Youtube VOS and DAVIS datasets. Additionally, it explores the effects of different adapters on the MEGA dataset.
MOSE-ssi
LYOS
Yauflab--VSS 2018
YouTlbb-
Methood
JkeF
J.
F.
J.
J.
Edfct of difieen pre snained vnggltt.
Realeer tHe et (Bocctie)
64.2
00.3
65.1
K52
$1.8
$5.5
57.
53.3
61.5
86.3
16.2
91.0
713
15.5
86.5
96.2
90.7
50.5
58.8
MAR 2003)
67.0
62.6
66.2
66.7
90.2
900
39.0
86.4
85.4
(Ogpah
WT-mmll (ang at, 2024)
64.3
65.5
81.5
$1.6
60.1
79.9
$8.5
96.3
Depth Anything
68.5
66.5
77.6
16.7
K2.7
90.8
66.5
67.1
7048
87.4
87111
92.0
805
19.7
87.5
86.3
91.8
81.3
69.9
Affict 4ificrmii murdlrr
gunr
08.3
60.A
75.1
19.6
16,9
18.8
16 16-quary -quccy
67.7
62.0
50.1
58.9
$9.3
68.5
66.5
72.6
66.7
K2.7
90.5
66.5
62.1
70.8
S7.4
57.00
92.0
90.9
$9.7
47.5
56.3
91.8
81.3
$0.9
Ouzs (Adiaper (Clten et al, 2022))
12.2
67.8
76.4
57.2
$4.1
90.4
71.1
66.7
75.6
S7.1
96.1
91.0
816
89.8
87.1
65.9
90.9
$1.4
90.1
(CoMer
(Sputial Semastic)
74.0
68.3
77.8
90.7
38.1
Effect of different pre-trained weights. Table 2 also compares the performance of using different training parameters. We conduct experiments using three different pre-training parameters, and the results indicate that better pre-training parameters lead to more significant improvements in the model, especially those from Depth Anything (Yang et al., 2024) (8-query in Table 2). The Depth Anything model implicitly contains depth information about the scene, which contributes to its stronger representation capability, consistent with the findings of the Depth Anything paper.
Effect of different query numbers. Table shows that performance drops the number of queries increases. This is because more queries introduce additional background noise, given the fewer targets in most VOS benchmarks, except MOSE (Ding et al., 2023).
Effect of adapters. Table presents a performance comparison between different adapters and our method. We replace the fusion module with various adapters and fully fine-tune them on the MEGA dataset, as indicated in related research that full fine-tuning achieves optimal performance (Chen al., 2022; Xia al., 2024). The experimental results demonstrate that our method efficient in training and outperforms the other two methods in accuracy.
4.3 COMPARISONS WITH STATE-OF-THE-ARR METHODS
DAVIS 2017 (Pont-Tusct et al., 2017) is a benchmark that offers densely annotated, high-quality, full-resolution videos with multiple objects of interest. Table 3 shows that our method achieves favorable performance (86.7% 7)) against state-of-the DAVIS, without any pre-training. The gains are primarily due the injection of semantic information, which helps handle sequences with object occlusion. When using the MEGA dataset as the training data, our method performs best on the DAVIS test (87.8%)
methods when trained on Youtube VOS and
YouTube 2018 (Xu et al., 2018) contains 347 videos of 65 categories for training and videos for validation. For validation, there are 26 categories that the model has not seen in its training, allowing us to evaluate its generalization ability for 350 class-agnostic targets. Table 3 illustrates that our method performs better on seen and unseen categories than existing state-of-the-an models. The performance gain can be attributed to spatial-semantic feature representation and representative target correlation.
YouTube VOS 2019 is an extension of YouTube- 2018, featuring a larger number of masked targets and more challenging sequences in its validation set. In Table 3, our approach exhibits competitive performance compared to the leading state-of-the-art methods. Remarkably, achieves the highest performance (87.5% without relying on pre-training, This underscores the effectiveness of our method in leveraging inherent semantic-spatial information and discriminative query representation.
model
MOSE (Ding et al., 2023) contains 2149 video clips and 5200 objects from 36 categories under more complex scenes. Compared to existing state-of-the-as methods, our model gains great improvement in this dataset. The benefits stem primarily from the discriminative query that captures more
8

---

Published as a conference paper at ICLR 2025
Table 3: Quantitative comparisons on the MOSE, LVOS, DAVIS 2017, YouTube-VOOS 2018 & 2019 dataset. The best two results are shown in red and blue color. In the table, denotes that the models are pre-trained using the static image datasets (Shi et al., 2016; Li et al., 2020; Zeng et al., 2019; Cheng et al., 2020; Wang et al., 2017). In the Appendix, we compare methods with external training data (e.g., large BL30K dataset for pre-training) and improved test size (600p/720p).
Mo 3
DAVIS
YouTube-VVon F.
Methool
JkF
JKF
3
Jo.F
J
3.
F.
J.
FL
G
J.
J.
Tinited only the Your Tisbe VCS
and I3AV/S Alstcurts
MIVOS (Chesg dl., 20216>*
74.9
82.6
77.7
86.2
$2.4
$6.4
STCN (Cheng et cl., 2021al
52.5
48.5
56.5
15.8
41.1
50.5
77.8
14.3
$1.3
84.3
85.2
57.9
79.0
57.3
84.2
82.6
87.9
79.6
57.7
13.2
Swin- (om-----T--J-C Ching 2001)
59.6
536
54.4
49.3
59.4
81.2
77.3
85.1
90.1
78.4
86.9
35.3
54.6
89.5
79.5
87.7
12.1
DeAOT (Ying Yong, 2022)
59.0
54.5
63.4
80.7
76.9
84.5
86.0
84.9
59.9
50.4
89.7
85.6
84.2
89.2
80.2
98.8
11.7
XMem (C (Ching& Schtiing 20121
76.3
83.4
14.3
83.9
77.7
86.7
34.2
85.3
76.1
86.7
XMar (Cheng Schoinng 2022)
53.3
60.0
90.0
$4.4
81.0
77.4
84.5
59.3
50.3
88.7
55.5
80.3
88.6
22.6
15VO5 (Warg sl., 2023)
12.8
79.3
96.2
86.3
85.5
90.2
50.5
88.8
86.1
55.2
89.7
90.7
88.9
5.8
SinVOS (Wa al. 2023)
61.6
57.9
75.7
85.8
94.2
79.1
87.2
3.3
Cree (Cheng at, 3025ay*
64.0
00.0
67.
M62
51.8
00.5
64.2
80.0
87.7
96.1
85.5
90.0
50.6
55.55
56.1
65.8
9835
W00
85.0
36.4
Inintormmzz (Zhing al.,
3.0
JuiaFoemc (Zlang et sl., 20233*
87.6
84.2
91.1
87.0
86.2
91.0
81.4
89.3
$7.0
56.1
90.6
82.0
89.5
3.0
(Ours)
64.5
66.5
70.5
92.7
89.7
59.9
13.1
DEVA (Chong et al. 2k2
66.5
67.3
70.8
54.0
49.0
59.0
K3.
79.6
86.8
W6.2
85-4
59.9
50.5
89.1
85.8
54.8
89:3
80.3
88.8
25.3
Curic (Cheng clall 2023a)
69.9
65.8
74.1
66.7
62.4
71.0
86.1
82.4
89.0
87.0
86
91.1
81.4
89.2
57.0
56
90.5
82.0
89.6
36.4
(Ours)
74.0
73.0
77.8
87.8
8400
91.7
98.7
90.7
13.1
Table 4: Comparasion with SAM 2 (Ravi et al., 2024). It shows that our method performs favorably against SAM
Dattset
LVOSVZ.ol Jh.
LVOS-val
DAVIS17-tod
MOSE-va
YeuThibe-Vo VOS 2019
Method
F
J&F
5
JKA
J
5
9
SAM (Herra
75,8
72.0
79.6
74.9
79.6
850
75.8
718
799
58.4
91.6
SAM
78.1
92.2
53 (ViThbb))
(37.9)8377
(-1.1)
80.8
(+1.5)
8644
(40.1)
718
(6..3)88.7
91.9
representative target information. Additionally, the feature extraction module provides matching with more semantic and spatially informative features.
LVOS (Hong et al., 2023) is new long-term dataset to validate the robustmess of VOS methods when faced with these challenges. It contains videos 20 times longer than existing VOS datasets, typically featuring a single target per video. In this benchmark, our method achieves performance gain (66.5% compared to 56.2%). )... This improvement attributed to the quality of memorized features and queries, which enhances the robustness of our approach in long-term video scenarios. Our method demonstrates enhanced stability and effectiveness in long-duration contexts by incorporating more discriminative queries that capture crucial target information into the object memory.
Comparison with Segment Anything (Ravi et al., 2024). Table 4 shows the detailed compari- son results between our method and SAM2. Specifically, SAM2 used large-scale mixed training dataset (comprising 1. 3% DAVIS, approximately 9.4% MOSE, approximately 9.2% YouTube VOS, approximately 15.5% SA-1B, approximately 49.5% SA-V, and approximately 15. 1 1% Internal, approximately 120.6k videos). In contrast, our model is trained solely on VOS datasets (DAVIS, MOSE, YouTube VOS, OVIS, and BURST, about 11.4K videos). To ensure a relatively fair compari- son, we use the same model size (ViTb) and the same input image size (1024p). In all benchmarks evaluated, our method outperforms SAM2 in all metrics, especially on the challenging LVOSV2 val dataset (+7.9%). Even when SAM2 (Ravi et al., 2024) uses Hiera-L as feature extractor, our model demonstrates comparable results, especially on LVOS V2 (Hong et al., 2023) val dataset (83.7% % vs 78.1% %))
Visualized Results. We visualize some challenging sequences, including scenarios involving part- to-whole-changing and faint objects and those where targets closely resemble the background. In Figure 4(a), our approach shows favorable segmentation performance against the sfate-of-the-art methods. For example, our model accurately segments targets and provides detailed results in a video that includes gorillas, necks, and a riding man. This can be attributed to our proposed discriminative query propagation, which enables model to capture more detailed target information by focusing on the most representative features, thereby facilitating precise segmentation in these challenging scenarios Additionally, incorporating semantic features in our model ensures the model understands

---

Published as a conference paper at ICLR 2025
Search frame
XMem
SwinB-DeAQTI
Cutie
Ours
5348066 3
bufu
(a) Qualitative comparisons from various sequences.
Frame
#29
Cutie
#0 GT
#20
#29
#59
#91
Ours
(b) Qualitative comparisons with target appcarance variance.
Figure Visualized results on challenging scenarios. The proposed method performs well on these challenging sequences, while other methods suffer from semantic understanding and thus cannot segment the whole objects. (b) Our method handles the challenging sequence with fast-motion objects better than Cutie (Cheng et al., 2023a)
the semantic prior of the interested targets, such as in the case of the sequence of man riding from YoutubeVOS 2019.
Figure 4(b) shows the segmentation and tracking performance on long-term sequences with fast motion and dramatic appearance variations. In this sequence, two similar dogs continuously change their position and posture, making it challenging for the model identify and track them accurately, Our VOS method performs accurately in tracking and segmenting targets, whereas Cutie fails to identify and segment the targets in this scenario. Good performance scenarios like this due to the semantic aware feature and discriminative query representation learned by model. More visualization and videos can be found in the Supplementary Material.
4.4 LIMITATIONS
Our method enhances target representation by introducing spatial- semantic features and discriminative query propagation, significantly improving numerous challenging scenarios. However, in scenes with object parts as the target (some sequences the VOT challenge, shown the appendix), the proposed model finds it difficult to represent part of the target and often extends the predicted mask to the whole target. For example, when the target is the head of a person, our model may gradually shift segmenting the entire body. This tendency arises because our method tends interpret the target as the entire object seen during the training phase. A potential direction to improve this issue could be to let the model learn part-aware semantic features, which also needs new benchmark. Moreover, the improvement of the inference speed will also be addressed in our future work.
CONCLUSION
In this paper, we propose spatial-semantic network to learn comprehensive representations to improve the robustness of video object segmentation in handling objects with complex structures or separate parts. In addition, we develop a discriminative query propagation module that selects representative features from target objects for query update, enabling more reliable query propagation in long-term videos. Our approach achieves favorable performance compared with of-the art methods on numerous datasets. Both quantitative and qualitative results demonstrate the effectiveness of the proposed method in tracking and segmenting video objects in complex and long-term scenes.
10

---

Published as a conference paper at ICLR 2025
Acknowledgments This work is supported by the Guangdong Basic and Applied Basic Research Foundation (Grant No. 2024A1515011222) the National Natural Science Foundation of China (Grant No.62476148, 62172126), and the Shenzhen Research Council (No. NM0.JCCY2200033410000000
REFERENCES
Ali Athar, Jonathon Luiten, Paul Voigtlaender, Tarasha Khurana, Achal Dave, Bastian Leibe, and Deva Ramanan. Burst: A benchmark for unifying object recognition, segmentation and tracking in video. In WACV, pp. 1674-1683, 2023.
L. Bao, B. Wu, and W. Liu. Can in mrf: Video object segmentation via inference in enn-based higher-order spatiotemporal mrf. In CVPR, 2018.
Goutam Bhat, Felix Jaremo Lawin, Martin Danelljan, Andreas Robinson, Michael Felsberg, Luc Van Gool, and Radu Timofte. Learning what to learn for video object segmentation. In ECCV, pp. 777-794, 2020.
Sergi Caelles, Kevis-Kokits, Maninis, Jordi Pont-Tuset, Laura Lexl-Taixé, Daniel Cremers, and Luc Van Gool. One-shot video object segmentation. CVPR, pp. 221-230, 2017. 3
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229. Springer, 2020. 5,5
Niccolò Cavagnero, Gabriele Rosi. Claudia Ruttano, Francesca Pistilli, Marco Ciccone, Giuseppe Averta, and Fabio Cermelli. Pem: Prototype-bes efficient maskformer for image segmentation. arXiv preprint arXiv:2402.114222 2024. 3
Y. Chen, Pont- JonteT Montes, and L.. Van Gool. Blazingly fast video object segmentation with pixel-wise metric learning. In CVPR, pp. 1189-11999 2018. 3
Zhe Chen, Yuchen Duan, Wenhai Wang, Junjun Hc, Tong Lu, Jifeng Dai, and Yu Qiao. Vision transformer adapter for dense predictions. In ICLR, 2022. 3,8
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, pp. 1290-1299, 2022.
H. K. Cheng and A. G. Schwing. Xmem: Long-term video object segmentation with an atkinson- shiffrin memory model. In ECCV, 2022. 1, 2. 3, 7, 14, 15, 17
H. K. Cheng, Y. W. Tai, and C. K. Tang. Rethinking spacetime networks with improved memory coverage for efficient video object segmentation. In NeurlPS, pp. 11781-11794, 2021a. 1, 14, 17
H.K. Cheng, Y.W. Tau, and C.K. Tang. Modular interactive video object segmentation: Interaction- to-mask, propagation and difference-awart fusion. In CVPR, 2021b. 9, 15
Ho Kei Cheng. Jihoon Chung, Yu-Wing Tai, and Chi-Keung Tang. Cascadepsp: Toward class-agnostic and very high-resolution segmentation via global and local refinement. In CVPR, pp. 8890-8899, 2020.
Ho Kei Cheng, Seoung Wug Oh, Brian Price, Joon- object back into video object segmentation. arXiv preprint arXiv:2310. 12982, 2023a. 2, 3, 6, 7,
Joon-Young Young Lee, and Alexander Schwing. Putting the
9. 10, 14, 15, 17
Kei Cheng, Seoung Wug Oh, Brian Price, Alexander Schwing, and Joon- Joon-Young Lee. Tracking anything with decoupled video segmentation. In ICCV, pp. 1316-1326, 2023b.
Cheng. Y.H. Tsai, W.C. Hung, S. Wang, and M.H. Yang. Fast and accurate online video object segmentation via tracking parts. In CVPR, 2018. 3

---

Published as a conference paper at ICLR 2025
Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, Philip HS Torr, and Song Bai. Mose: A new dataset for video object segmentation in complex scenes. ICCV. pp. 22224-2023,, 2023. 2, 7, 8, 15, 16
A. Dosovitskiy, L. Beyer, and A. Kolesnikov. An image worth 16x16 words: image recognition at scale. In ICLR, 2021. 3,6
Transformers for
B. Duke, A. Ahmed, C. Wolf, and G. W. Taylor. Sstvos: Sparse spatiotemporal transformers for video object segmentation. In CVPR, pp. 5912-5921, 2021. 3
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pp. 770-778, 2016. 5, 8
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, pp. 16000-16009, 2022. 8
Lingyi Hong, Wenchao Chen, Zhongying Liu, Wei Zhang, Pinxue Guo, Zhaoyu Chen, and Wenqiang Zhang. Lvos: A benchmark for long-term video object segmentation. In ICCV, 2023. 2,9
Yuan-Ting Hu, Jia-Bin Huang, and Alexander Schwing. Maskrnn: Instance level video object segmentation. In NewrlPS, 2017. 3
Yuan-Ting Hu, Jia-Bin Huang, and Alexander G Schwing. Videomatch: Matching based video object segmentation. In ECCV, 2018. 3
Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.7
Matej Kristan, Jifí Matas, Martin Danelljan, Michael Felsberg, Hyung Jin Chang, Luka Cehovin Zaje, Alan Lukežič, Ondrej Drbohlav, Zhongqun Zhang, Khanh-Tung Tran, al. The first visual object tracking segmentation vots2023 challenge results. In ICCV, pp. 1796-1818, 2023. 15, 20
Mingxing Li, Li Hu, Zhiwei Xiong, Bang Zhang, Pan Pan, and Dong Liu. Recurrent dynamic embedding for video object segmentation. In CVPR, 2022a. 17
Xiang Li, Tianhan Wei, Yau Pun Chen, Yu- Wing Tai, and Chi- Chi-Keung Tang. Fss-1000: A 1000-class dataset for few-shot segmentation. In CVPR, June 2020. 5. 9
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022b. 7
J. Luiten, P. Voigtlaender, and B. Leibe. Premvos: Proposal-generatce refinement and merging for video object segmentation. In CVPR, pp. 565-580, 2018.
Y. Mao, N. Wang, W. Zhao, and H. Li. Joint inductive and transductive learning for video object segmentation. In ICCV, 2021. 3, 14
Scoung Wug Oh, Joon- Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space-time memory networks. In ICCV, pp. 9226-9235, 2019. 1,
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023. 8
Kwanyong Park, Sanghyun Woo, Scoung Wug Oh, In So Kweon, and Joon-Young Lee. Per-clip video object segmentation. In CVPR, 2022.
Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. Van Gool. The 2017 davis challenge on video object segmentation. arXiv preprint arXiv: 1704.00677. 2017. 1, 2, 7, 8
Jiyang Qi, Yan Gao, Yao Hu, Xinggang Wang. Xiaoyu Liu, Xiang Bai, Serge Belongie, Alan Yuille, Philip HS Torr, and Song Bai. Occluded video instance segmentation: A benchmark. IJCV, 130 (8):2022-200 2039, 2022.
12

---

Published as a conference paper at ICLR 2025
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408. 00714, 2024. 9
Jianping Shi, Qiong Yan, Li Xu, and Jiaya Jia. Hierarchical image saliency detection on extended cssd. IEEE TPAMI, 38(4):717-729, 2016. 5. 9
Yi-Hsuan Tsai, Ming- Hsuan Yang, and Michael J Black. Video segmentation via object flow. In CVPR, pp. 3899-3908, 2016.
Voigtlaender and B. Leibe. Online adaptation of convolutional neural networks for video object
segmentation. In BMVC, 2017. 3
Voigtlaender, Y. Chai, F. Schroff, H. Adam, and L.C. Chen. Feelvos: Fast end-to-end embedding learning for video object segmentation. In CVPR, pp. 9481-9490, 2019.
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Chuanxin Tang, Xiyang Dai, Yucheng Zhao, Yujia Xie, Lu Yuan, and Yu-Gang Jiang. Look before you match: Instance understanding matters in video object segmentation. In CVPR, 2023. 1, 2, 3, 15
Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. CVPR, 2017. 5. 9
Qianggiang Wu, Tianyu Yang. Wei Wu, and Antoni Chan. Scalable video object segmentation with 9, 14
simplified framework. In ICCV, 2023.
Chunlong Xia, Xinliang Wang, Feng Lv, Xin Hao, and Yifeng Shi. Vit-comer: Vision trans- former with convolutional multi-scale feature interaction for dense predictions. arXiv preprint arXiv:2403.07392, 2024. 8,8 8
Huaxin Xiao, Jiashi Feng. Guosheng Lin, Yu Liu, and Maojun Zhang. Monet: Deep motion
exploitation for video object segmentation. In CVPR, pp. 1140-1148 2018. 3
Ning Xu, Linjie Yang, Yuchen Fan, Dingcheng Yue, Yuchen Liang, Jianchao Yang, and Thomas Huang. Youtube-vos: A large-scale video object segmentation benchmark. In ECCV, 2018. 1, 7,
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint arXiv:2401.10891, 2024. 8
Zongxin Yang and Yi Yang. Decoupling features in hierarchical propagation for video object
segmentation. In NeurlPS, 2022. 3, 9, 14, 17
Zongxin Yang, Yunchao Wei, and Yi Yang. Collaborative video object segmentation by foreground- background integration. In ECCV, 2020. 3
Zongxin Yang, Yunchao Wei, and Yi Yang. Associating objects with transformers for video object segmentation. In NeurlPS, 2021. 2, 3, 9, 17
Yi Zeng, Pingping Zhang, Jianming Zhang, Zhe Lin, and Huchuan Lu. Towards high-resolution salient object detection. In ICCV, 2019. 5. 9
Jiaming Zhang, Yutao Cui, Gangshan Wu, and Limin Wang. Joint modeling of feature, correspon- dence, and compressed memory for video object segmentation. arXiv preprint arXiv:2308..3500 2023. 9, 15
13

---

Published as a conference paper at ICLR 2025
A APPENDIX
The appendix is organized as follows:
We provide a detailed implementation of training and inference.
2. We provide more experiments with different training settings and test hyperparameters.
3. More visualization and failure cases are shown.
A.1 IMPLEMENT DETAILS.
A.1.1 MULTI-OBIECT PROCESSING
In this section, we provide additional implementation details for completeness. Our training and inference code will be released to ensure reproducibility.
Following the previous work (Cheng & Schwing, 2022; Cheng et al., 2021a), extend methods to the multi-object setting. The predicted mask of each frame undergoes channel separation for different targets. The separated masks for each target then concatenated with the RGB image and the aggregated mask of other targets for feature extraction. The extracted features are fused with RGB features to obtain pixel-level value features, which are then stored in memory. It is important to note that the pixel-level features stored in memory include the key (RGB feature) and the value.
A.1.2 TRAINING DETAILS
As mentioned in the main paper, we train our network only video-level datasets, which differs from other methods (Cheng et al., 2023a; Cheng & Schwing, 2022; Yang Yang, 2022). The backbone weights are initialized from MAE pre-training, similar to the approaches used in VOS (Wu et al., 2023) and JointFormer (Mao et al., 2021). We perform spatial-semantic fusion every three ViT layers and five discriminative query transformer blocks for query propagation. We implement our network using PyTorch and train the model on 8 NVIDIA V100s, employing automatic mixed precision (AMP) during training.
Two different training datasets are used for better improvements. The "DAVIS&YTVOS settings combine the training sets of YouTube VOS-2219 and DAVIS-2017, with the DAVIS-2011 2017 dataset expanded five times to increase the data ratio. The "MEGA" set contains four different training sets, including DAVIS-2017, You dataset expanded five times to increase the data ratio. To sample training sequence, we randomly select "seed" frame from all the frames and then randomly choose 7 other frames from the same video. We ra-sallle if any two consecutive frames have a temporal distance greater than the max-skip. Following the previous work (Cheng & Schwing, 2022; Cheng al., 2023a), the max-skip set to [5, 10, 15, 5] after [0%, 10%, 30,, 80%] of the training iterations, respectively. To avoid sampling frames without targets, we use the data statisties provided by Cutie (Cheng al., 2023a).
MOSE, and BURST datasets, with the DAVIS-2017
Table 5: Training parameters.
Config
DAVIS YTVOS
MEGA
optimizer
Adam
Adam W
base learning rate
5e-5
5e-5
weight decay
0.05
0.05
droppath rate
0.15
0.15
batch size
16
16
num ref frames
3
3 8
num frames
8
max-skip max-skip-itr
[5, 10, 1] 10.
[5. 10, 15, S]
10.1, 0.3, 0.8,
Iterations
150,000 steplr
190,000
learning rate schedule
steplr
14

---

Published as a conference paper at ICLR 2025
Table 6: Quantitative comparisons with high-resolution input on the MOSE, LVOS, DAVIS 2017,
2018 & 2019 dataset. means improve the input size.
Taiset
MOSE-val
DAVIS 2017 lest
YorTabb-VDS 2015 val
You Tuls--Voo VOI 2010 val
Method
Cusle buse (Chenx ctd. 2023g
64.0
60.0
67.9
$4.2
90.6
62.7
96.1
55.3
90.5
$0.0
$5.0
$6.1
65.5
90.0
90.6
88.3
(Wang etal. 21037--81 +BL3K 20210)
89.0
55.2
AsidForm (Zaang 3023))+BL33K (Chesg et 2021b)
88.1
64.7
91.6
87.6
96.4
91.0
92.2
90.7
$7.4
96.5
90.9
82.0
90.3
64.5
72.6
53.1
92.0
50.9
$7.5
55.8
91.8
(Cheng 200. 20230)
62.3
70.1
89.2
96.2
903
81.6
89.2
Unrs
703
06.5
74.5
K7M
94.0
91.3
57.6 36.
91.7
$1.5
90.1
$7.8
56.8
91.6
K2.2
103
Diie-bas" (Chrgg 700110 MIXGA
65.8
74.1
87.4
89.9
87.0
36.4
91.1
$1.4
$7.0
$5.0
90.5
81.0
19.0
Ours WMEGA
73.2
68.8
77.5
58.2
94.3
92.1
88.1
97.4
92.5
$1.9
90.7
58.0
$5.0
91.8
82.5
10.8
(Changettl 202347
75.6
Unne WIMEGA
253
71.0
79.2
85.8
92.4
19.5
87.6
92.6
52.7
91.3
88.5
87.3
91.0
81.1
41.4
We conduct data augmentation with methods of random horizontal mirroring, random affine trans- formations, and cut-and random color jittering and random grayscaling augmentation performed in the same video for stability. To ensure the images can be input into the patch embedding of VIT, apply padding to both the images and masks so that their dimensions multiples of 32.
The training samples are also resized by crop (480 x 480). Then, performed in the training samples. The same
We initialize the model with the first frame and ground truth in the training phase. When the number of past frames is three or fewer, we use all of them as memory frames. If there are more than three past frames, randomly sample three of them be the memory frames. We compute the loss all frames except the first one and perform back poockpropagation through time.
A.1.3 INFERENCE DETAILS
During the inference phase, the model memorizes the feature every 5th frame like (Cheng & Schwing, 2022; Cheng et al., 2023a). The pixel-Ievel memory always stores the first frame and its mask. The top-K K filter is used augment memory reading in pixel memory. We use the streaming average for object-level memory to accumulate the discriminative query representation like the operation in (Cheng et al., 2023a).
A.2 ADDITIONAL QUANTITATIVE RESULTS
In this section, we provide additional ablation studies and SOTA comparisons with different settings.
Table 6 shows detailed results about methods with further pre-training (e.g. JointFormer (Zhang et al., 2023) and ISVOS (Wang et al., 2023)) and improved test size (Cutie (Cheng et al., 2023a) We note that our method achieves comparable results against SOTA methods (Cutie (Cheng al., 2023a) and JoimtFormer (Zhang et al., 2023)) without any pre-training When trained with MEGA datasets or improved test image size, our method achieves state-of-the-art results in all the benchmarks.
In the main paper, we provide ablation studies for different components. To provide broader comparison, we conduct tests using various combinations of training data and testing settings. The experimental results in Table detail the impact of different components of our model on the performance, facilitating a better understanding of the effectiveness our model.
To compare with more methods MOSE, we follow the training strategy from MOSE (Ding et al., 2023), and replace the YouTube VOS with MOSE the training process. Table shows the results of the variants of our method. All the variants achieve great improvement in terms of J &.F, which further validates the effectiveness the proposed spatial-semant feature learning and discriminative query representation.
Table 9 shows the results of our method on the more challenging VOTS2023 (Kristan et al., 2023) benchmark, which indicates that method outperforms the winner (DMAOT) of last year most metrics. Our method achieves the highest robustness thanks to our representation of target saliency information and the modeling of target semantic-spatial information.
15

---

Published as a conference paper at ICLR 2025
Search frame
XMem
SwinB-DeAOTT
Cutie
Quri
8dea7458de
b2258e258c br4cc89b18 16ed698281
Man-bike
Qualitative comparisons from various sequences.
Figure 5: Additional qualitative comparison against state-of-the-art methods including XMem,
Swin-DeAOT, and Cutie. All the sequences are selected from YouTube 2019 and DAVIS 2017 datasets.
Cutie
#1 7
#35
#47
#83
Ours
#2
#6
#8
Cutie
88
Ours
#17
#20
#22
#2.
Cutle
#17
#20
225
Ours
Qualitative comparisons from various sequences with appearance variance.
Figure 6: Additional qualitative comparison sequence with giant appearance variance. All the sequences are picked from MOSE (Ding et al., 2023) dataset.
16

---

Published as a conference paper at ICLR 2025
Table 7: Model detailed ablation study.
Dutaset
MOSE-val
DAVIS 2017 test
LVOS
YohT VOS 2019
Method
J&J
J
J
J,
Ju
J
Trained the YouTabe and DAVIS dutusetr
XMem (Cheng Schwing. 2023) (Baseline)
53.3
62.0
57.6
81.0
77.4
84.5
50.0
45.5
54.1
85.5
$4.3
88.5
80.3
88.6
Cutie (Cheng et al., 2023a)
64.0
60.0
67.9
84.2
80.5
87.7
56.2
51.8
60.5
86.1
85.8
90.5
80.0
88.0
+Disciminative Query
64.2
60.3
68.1
85.2
81.8
88.5
57.4
53.3
61.5
86.5
86.2
90.7
80.6
88.8
.WT
64.2
60.2
68.3
85.6
82.0
89.2
58.3
53.7
62.8
96.7
86.4
90.3
81.0
88.7
Sppntil (Fall)
64.0
72.4
90.1
87.3
86.4
91.3
Ieproved hest stor (600/220)
(Cheng et 2023a)
Discriminative Query
Spputal
69,9
67.5
74.1
87.0
83.7
90.2
67.4
71.9
87.5
86.8
91,5
81.6
90.2
Semmniic ( (ull)
70.5
66.5
74.6
87.8
84.6
91.3
66.5
62.1
70.8
87.7
86.8
91.6
82.2
90.8
Trained on the MEGA datarets
Cutie (Cheng et al., 2023a)
69.9
65.8
74.1
86.1
82.4
89.9
66.7
62.4
71.0
87.0
86.0
90.5
82.0
90.7
Diseriminative quary
70.6
66.5
74.6
86.6
82.7
90.5
62.1
70.8
87.5
86.0
90.6
82.8
90.6
+Sputial
Semantic
69.8
78.3
87.8
84.0
87.4
92.5
Improved vizr (6007720)
Cutie (Cheng 2023g)
71.7
67.6
75.8
88.1
84.7
91.1
66.7
62.4
71.0
87.5
86.3
90.5
82.7
90.5
Discriminnrive query
71.6
67.7
75.5
88.1
84.6
91.4
66.5
62.1
70.8
88.0
86.2
90.6
83.6
91.5
Sputial
75.3
71,3
79.2
89.0
85.8
92.3
68.8
64.4
73.1
88.3
87.0
91.9
83.0
91.1
Semanii(Pall)
75.1
71.0
79.2
89.1
85.8
92.4
73.0
68.3
77.8
88.5
87.3
92.0
83.1
91.4
Table Results only trained on MOSE datasets.
Dataset Method
MOSE-val
J &FF
F
RDE (Li al., 2022a)
48.8
44,6
52.9
STCN (Cheng et al., 2021a)
50.8
46.6
55.0
AOT (Yang ct al., 2021)
57.2
53.1
61.3
XMem (Cheng & Schwing, 2022)
57.6
53.3
62.0
DeAOT (Yang & Yang, 2022)
59.4
55.1
63.8
DesNeitDDiermmnnativ query
69.9
65.8
73.9
+Spatial
72.7
68.3
77.0
+Semantic
72.9
68.4
77.3
Improved test size (720)
ResNettDiscrimiaatiee query
71.6
67.7
75.5
+Spatial
74.0
69.9
78.1
+Semantic
74.5
70.2
78.5
Table 9: Results VOTS2023 Challenge.
Dataset
VOTS2023
Method
Q1
Af
H
NER |
DRE
ADQ
Dynamic_ DEAOT
0.59
0.69
0.84
0.07
0.1
0.57
M-VOSTrack
0.61
0.75
0.76
0.16
0.08
0.71
HQTrack
0.62
0.75
0.77
0.16
0.08
0.70
DMAOT
0.64
0.75
0.80
0.14
0.07
0.73
Ours
0.67 (+0.3)
0.74
0.86 (+0.6)
0.09 (+0.5)
0.05 (+0.02)
0.75 (+0.02)
A.3 ADDITIONAL QUALITATIVE COMPARISON
We present more qualitative comparison of most benchmarks in Figure 5 and Figure 6.
In Figure 5, we categorize the scenarios into three main types: overall-to-part segmentation (man on bike and deer), small object segmentation (small monkey), blurry object segmentation (cat toy), and segmentation under varying lighting conditions (man-bike and golf). In these three cases, our method demonstrates precise segmentation compared to other methods. The semantic embedding allows our model to consider semantic information when parsing objects, spatial dependence modeling enhances the understanding of spatial positions and details of the object, and the discriminative query
17

---

Published as a conference paper at ICLR 2025
(Spatia) Semantic)
Frames
Block innut Semantic cobbekding Sparial modeline
(a) Visualization of the feature map output from different backbones
(b) Visualization of the feature map after proposed semantic and spatial layers
Figure 7: Visualizations of feature maps from different architectural components.
accurately represents the target. The integration of these three components enables the model to achieve excellent segmentation and tracking performance in complex and challenging scenarios.
In Figure 6, we select more sequence results on MOSE, primarily divided into two categories: positional changes of similar targets and rapid movement of targets. In both cases, our method outperforms Cutie. This is attributed to our proposed spatial-semantic feature representation and discriminative query expression, which enable the model to distinguish appearance similar targets and consistently track the target.
Figure 9 compares sequences that require semantic understanding of the target. Without the semantic embedding module, the method suffers from segmenting parts of the target. The addition of the semantic embedding module enables the model fully understand the semantic information of the target, allowing for precise segmentation and tracking in these scenarios.
Visualization of feature maps. Figure 7(a) shows that the proposed Spatial-Semanté block assigns higher activation weights to complex target objects compared to the CNN or ViT model, which indicates that the proposed model makes target representations more distinguishable than ViT and ResNet. Moreover, we show how the feature changes during the processing of the proposed model by visualizing the feature maps at different processing steps in 7(b). After passing through the semantic embedding module, the feature maps (semantic embedding) demonstrate that the module makes the contours of objects clearer, indicating richer semantic information. After passing through the spatial dependence module, the feature maps (spatial modeling) show that the details of the targets are more pronounced. The changes in the feature maps confirm that the proposed two modules significantly enhance the semantic and spatial information of the targets.
We also visualize the similarity between the learned discriminative queries and the feature map using heat maps in Figure 8. The visualization shows that the learned diseriminative queries distinctly express the target areas. We also mark the positions of the discriminative queries, which represent the locations of the pixels with the highest similarity.
Failure cases are shown in Figure 10, in which our method tends to segment the whole instance rather than the part of it. The challenging sequences require the model to segment and track the head of the man, the flamingo, and the kangaroo. Our model interprets semantics as the entire instance during tracking and segmentation, such as a person, bird, or kangaroo, resulting in segmentation outcomes gradually favoring the whole instance.
18

---

Published as a conference paper at ICLR 2025
Frames
Prediction
Targel
Target
Target
Figure 8: Visualization of the similarity map between query and features. The map indicates the discrimination of the target representation. The discriminative queries of different targets are plotted the similarity map, which can be found in the black box.
b05faf5417
7a43728d3 8dea7458de
Test frame
W/O Semantic
W Semantic
Figure 9; Qualitative comparison between with and without semantic embedding block. All the sequences are picked from the YouTube VOS 2019 dataset.
19

---

Published as a conference paper at ICLR 2025
Time line
Failure cases
Figure 10: Failure cases. All the sequences are picked from VOT Challenge (Kristan et al., 2023).
20


Reviews:
[{"review_id": "UbUWdnnEZt", "reviewer": "Reviewer_TGSj", "summary": "This paper focuses on video object segmentation. The authors analyze the existing challenges like structural complexity, occlusion, and dramatic appearance changes, and correspondingly propose spatial-semantic feature augmentation as well as discriminative query association. The ablation studies and visualizations verify the effectiveness of each module.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The motivation is clear and the architecture makes sense. Integrating high-level semantics and low-level spatial cues is promising in video object segmentation.\n2. The experiments are thorough and the ablation studies can well reflect the effectiveness of each module.", "weaknesses": "1. The method is complicated. What is the advantage of using spatial offsets with deformable convolution compared to simple position encodings?\n2. The second row of Figure 3(a) seems strange. With semantic feature augmentation, the feature maps can well highlight the desired object instance. Adding spatial cues on the contrary suppresses the emphasis on the target instance but enhances object instances with the same semantics.\n3. Compared to SAM2, which designs a memory to prompt the segmentation of new frames, what is the advantage of this architecture?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dm43fcrBCE", "reviewer": "Reviewer_dBZh", "summary": "This paper addresses the complex task of tracking and segmenting multiple similar objects in long-term videos, where identifying target objects becomes challenging due to factors like occlusion, cluttered backgrounds, and appearance changes. To tackle these issues, the authors propose a new framework for robust video object segmentation, focusing on learning spatial-semantic features and generating discriminative object queries.\n\nThe framework introduces a spatial-semantic block that combines global semantic embedding with local spatial dependency modeling, which enhances the representation of target objects by capturing both broad context and fine details. Additionally, a masked cross-attention module refines object queries, concentrating on the most distinctive parts of target objects and reducing noise accumulation over time. This approach aids in effective long-term query propagation, a critical factor for high-performance tracking over extended sequences.\n\nThe experimental results are strong, showing state-of-the-art performance across several benchmarks.", "questions": "See weakinesses", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper’s S3 algorithm for Video Object Segmentation (VOS) demonstrates notable strengths:\n\n1.Spatial-Semantic Integration: By combining semantic embedding with spatial dependency modeling, it effectively captures complex object structures without requiring extensive ViT retraining.\n\n2.Discriminative Query Mechanism: The adaptive query approach improves target focus and reduces noise in long-term tracking, enhancing robustness.\n\n3.Extensive Validation: State-of-the-art results on multiple benchmarks highlight its strong generalization across datasets.", "weaknesses": "1.This paper claims to address the challenges of long-term tracking and segmentation. However, as far as I know, memory mechanisms are crucial for tackling these challenges in long-term tracking and segmentation, yet the authors do not seem to have conducted ablation experiments on the number of frames in the memory bank.\n\n2.I believe that the ablation study on the number of queries is insufficient with only 8, 16, and 32 as tested values. A wider range of query counts should be explored to more thoroughly validate the effectiveness of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QMKAMFTGuX", "reviewer": "Reviewer_ZJce", "summary": "This paper presents a novel spatial-semantic block that effectively integrates semantic information with spatial features, resulting in a more comprehensive representation of target objects, especially those with complex or distinct parts. By utilizing a pre-trained Vision Transformer (ViT) backbone without the need to retrain all parameters, the proposed method significantly enhances the efficiency of video object segmentation (VOS).\n\nAdditionally, the development of a discriminative query mechanism marks a substantial advancement in the field. This mechanism prioritizes the most representative regions of target objects, thereby improving the reliability of target representation and query updates. This is particularly advantageous in long-term video scenarios, where appearance changes and occlusions can lead to noise accumulation during query propagation.\n\nThe authors also highlight the importance of learning comprehensive target features that encompass semantic, spatial, and discriminative information. This holistic approach effectively addresses challenges related to appearance variations and identity confusion among similar-looking objects in long-term videos, making it a valuable contribution to the VOS community.\n\nFinally, extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance across multiple benchmark datasets, including DAVIS 2017, YouTube VOS 2019, MOSE, and LVOS.", "questions": "1.In Figure 2, there are N spatial-semantic blocks, but N is not specified later in the paper.\n\n2.In Figure 2, the argmax operation in the distinctive query propagation is non-differentiable. Will this prevent the gradient from being propagated through the model?\n\n3.If the introduced ViT backbone is not fine-tuned, will its performance degrade on the new dataset? A comparison experiment between freezing and not freezing the parameters is needed here.\n\n4.The number of different queries should be related to the number of targets. However, using 8 queries yields better results. When faced with more than 8 targets, can 8 queries adequately represent the different targets?\n\n5.In Table 3, there are two XMem entries, one of which is not referenced. It is unclear what the unreferenced entry represents, and why it lacks FPS results needs to be clarified.\n\n6.Table 3 lacks a comparison of Joint Former results trained on the MEGA dataset. Please provide the results for Joint Former trained on the MEGA dataset in detail. If the original Joint Former was not trained on this dataset, can it be trained and then compared for performance?\n\n7.The spatial-semantic block consists of two parts: first, the global feature cls token is fused with the semantic features, and then further enhanced through Deformable Cross Attention. It is necessary to separately validate the effects of directly fusing the features versus applying Deformable Cross Attention for further enhancement.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper presents a spatial-semantic modeling method and a discriminative query mechanism that significantly enhance the model's performance. Extensive experiments have been conducted to demonstrate the effectiveness of the model, and several visual examples are provided to clearly illustrate the results at different processing stages. Additionally, the final results showcase the model's considerable potential.", "weaknesses": "Writing Style:\n1. The writing language is not concise enough, with many long sentences that significantly reduce readability. This is particularly evident in the introduction, such as on the second page: \"We construct a Spatial-Semantic Block comprising a semantic embedding module and a spatial dependencies modeling module to efficiently leverage the semantic information and local details of the pre-trained ViTs for VOS without training all the parameters of the ViT backbone.\"\n\nImage Details:\n1. In Figure 2, there are N spatial-semantic blocks, but N is not specified later in the paper.\n\nMethod:\n1. In Figure 2, the argmax operation in the distinctive query propagation is non-differentiable. Will this prevent the gradient from being propagated through the model?\n\n2. If the introduced ViT backbone is not fine-tuned, will its performance degrade on the new dataset? A comparison experiment between freezing and not freezing the parameters is needed here.\n\n3. The number of different queries should be related to the number of targets. However, using 8 queries yields better results. When faced with more than 8 targets, can 8 queries adequately represent the different targets?\n\n4. In Table 3, there are two XMem entries, one of which is not referenced. It is unclear what the unreferenced entry represents, and why it lacks FPS results needs to be clarified.\n\n5. Table 3 lacks a comparison of Joint Former results trained on the MEGA dataset. Please provide the results for Joint Former trained on the MEGA dataset in detail. If the original Joint Former was not trained on this dataset, can it be trained and then compared for performance?\n\n6. The spatial-semantic block consists of two parts: first, the global feature cls token is fused with the semantic features, and then further enhanced through Deformable Cross Attention. It is necessary to separately validate the effects of directly fusing the features versus applying Deformable Cross Attention for further enhancement.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4ZQa1x3HWE", "reviewer": "Reviewer_eHC5", "summary": "The authors focus on the problem of video object segmentation in long-term tracking and complex environments. To improve the model's robustness, a spatial-semantic network block is proposed to integrate semantic information with spatial information for video object segmentation. Additionally, a discriminative query mechanism is developed to capture the most representative region of the target for better target representation learning and updating. The proposed method achieves state-of-the-art results on most VOS dataset.", "questions": "1、\tHow many trainable parameters and total parameters does the model have?\n2、\tWhy does \"DepthAnything\" achieve the best results?\n3、\tIf the backbone of the Cutie model is replaced with ViT, would it achieve good results?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "This paper addresses key issues in current applications of VOS methods: long-term tracking, occlusion, and object representation changes. \n1.\tThe proposed approach utilizes semantic and spatial information from upstream pre-trained models, enriching the target's semantic and detailed information. It is novel in the field of VOS. \n2.\tThe paper proposes a discriminative query generation mechanism to provide the model with more distinctive target information, which is validated on LVOS datasets.\n3.\tThe proposed method is validated on various VOS datasets and achieves state-of-the-art results.", "weaknesses": "The paper does not have obvious weaknesses, but there are still some issues. \n1.\tIn Table 1, why is there no separate ablation study for the spatial block and semantic block? Please provide this part of the experiment.\n2.\tSome detail issues: In Figure 3, the blue results are not clearly marked and the position of * is not aligned.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)