Learning Spatial-Semantic Features for Robust Video Object Segmentation

Authors: Xin Li, Deshui Miao, Zhenyu He, Yaowei Wang, Huchuan Lu, Ming-Hsuan Yang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video Object Segmentation, Spatial-Semantic Feature, Long-Term, Discriminative Object Queries

Abstract:
Tracking and segmenting multiple similar objects with distinct or complex parts in long-term videos is particularly challenging due to the ambiguity in identifying target components and the confusion caused by occlusion, background clutter, and changes in appearance or environment over time. In this paper, we propose a robust video object segmentation framework that learns spatial-semantic features and discriminative object queries to address the above issues. Specifically, we construct a spatial-semantic block comprising a semantic embedding component and a spatial dependency modeling part for associating global semantic features and local spatial features, providing a comprehensive target representation. In addition, we develop a masked cross-attention module to generate object queries that focus on the most discriminative parts of target objects during query propagation, alleviating noise accumulation to ensure effective long-term query propagation. The experimental results show that the proposed method sets new state-of-the-art performance on multiple data sets, including the DAVIS2017 test (\textbf{87.8\%}), YoutubeVOS 2019 (\textbf{88.1\%}), MOSE val (\textbf{74.0\%}), and LVOS test (\textbf{73.0\%}), which demonstrate the effectiveness and generalization capacity of the proposed method. We will make all the source code and trained models publicly available.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 8

Reviews and discussion:
<thread>
creation date:
1730114570975

thread_type:
official_review

<thread_object>
creator: reviewer_ehc5

<content>
summary:
The authors focus on the problem of video object segmentation in long-term tracking and complex environments. To improve the model's robustness, a spatial-semantic network block is proposed to integrate semantic information with spatial information for video object segmentation. Additionally, a discriminative query mechanism is developed to capture the most representative region of the target for better target representation learning and updating. The proposed method achieves state-of-the-art results on most VOS dataset.

strengths:
This paper addresses key issues in current applications of VOS methods: long-term tracking, occlusion, and object representation changes. 
1.	The proposed approach utilizes semantic and spatial information from upstream pre-trained models, enriching the target's semantic and detailed information. It is novel in the field of VOS. 
2.	The paper proposes a discriminative query generation mechanism to provide the model with more distinctive target information, which is validated on LVOS datasets.
3.	The proposed method is validated on various VOS datasets and achieves state-of-the-art results.

weaknesses:
The paper does not have obvious weaknesses, but there are still some issues. 
1.	In Table 1, why is there no separate ablation study for the spatial block and semantic block? Please provide this part of the experiment.
2.	Some detail issues: In Figure 3, the blue results are not clearly marked and the position of * is not aligned.

questions:
1、	How many trainable parameters and total parameters does the model have?
2、	Why does "DepthAnything" achieve the best results?
3、	If the backbone of the Cutie model is replaced with ViT, would it achieve good results?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730524123483

thread_type:
official_review

<thread_object>
creator: reviewer_zjce

<content>
summary:
This paper presents a novel spatial-semantic block that effectively integrates semantic information with spatial features, resulting in a more comprehensive representation of target objects, especially those with complex or distinct parts. By utilizing a pre-trained Vision Transformer (ViT) backbone without the need to retrain all parameters, the proposed method significantly enhances the efficiency of video object segmentation (VOS).

Additionally, the development of a discriminative query mechanism marks a substantial advancement in the field. This mechanism prioritizes the most representative regions of target objects, thereby improving the reliability of target representation and query updates. This is particularly advantageous in long-term video scenarios, where appearance changes and occlusions can lead to noise accumulation during query propagation.

The authors also highlight the importance of learning comprehensive target features that encompass semantic, spatial, and discriminative information. This holistic approach effectively addresses challenges related to appearance variations and identity confusion among similar-looking objects in long-term videos, making it a valuable contribution to the VOS community.

Finally, extensive experimental results demonstrate that the proposed method achieves state-of-the-art performance across multiple benchmark datasets, including DAVIS 2017, YouTube VOS 2019, MOSE, and LVOS.

strengths:
This paper presents a spatial-semantic modeling method and a discriminative query mechanism that significantly enhance the model's performance. Extensive experiments have been conducted to demonstrate the effectiveness of the model, and several visual examples are provided to clearly illustrate the results at different processing stages. Additionally, the final results showcase the model's considerable potential.

weaknesses:
Writing Style:
1. The writing language is not concise enough, with many long sentences that significantly reduce readability. This is particularly evident in the introduction, such as on the second page: "We construct a Spatial-Semantic Block comprising a semantic embedding module and a spatial dependencies modeling module to efficiently leverage the semantic information and local details of the pre-trained ViTs for VOS without training all the parameters of the ViT backbone."

Image Details:
1. In Figure 2, there are N spatial-semantic blocks, but N is not specified later in the paper.

Method:
1. In Figure 2, the argmax operation in the distinctive query propagation is non-differentiable. Will this prevent the gradient from being propagated through the model?

2. If the introduced ViT backbone is not fine-tuned, will its performance degrade on the new dataset? A comparison experiment between freezing and not freezing the parameters is needed here.

3. The number of different queries should be related to the number of targets. However, using 8 queries yields better results. When faced with more than 8 targets, can 8 queries adequately represent the different targets?

4. In Table 3, there are two XMem entries, one of which is not referenced. It is unclear what the unreferenced entry represents, and why it lacks FPS results needs to be clarified.

5. Table 3 lacks a comparison of Joint Former results trained on the MEGA dataset. Please provide the results for Joint Former trained on the MEGA dataset in detail. If the original Joint Former was not trained on this dataset, can it be trained and then compared for performance?

6. The spatial-semantic block consists of two parts: first, the global feature cls token is fused with the semantic features, and then further enhanced through Deformable Cross Attention. It is necessary to separately validate the effects of directly fusing the features versus applying Deformable Cross Attention for further enhancement.

questions:
1.In Figure 2, there are N spatial-semantic blocks, but N is not specified later in the paper.

2.In Figure 2, the argmax operation in the distinctive query propagation is non-differentiable. Will this prevent the gradient from being propagated through the model?

3.If the introduced ViT backbone is not fine-tuned, will its performance degrade on the new dataset? A comparison experiment between freezing and not freezing the parameters is needed here.

4.The number of different queries should be related to the number of targets. However, using 8 queries yields better results. When faced with more than 8 targets, can 8 queries adequately represent the different targets?

5.In Table 3, there are two XMem entries, one of which is not referenced. It is unclear what the unreferenced entry represents, and why it lacks FPS results needs to be clarified.

6.Table 3 lacks a comparison of Joint Former results trained on the MEGA dataset. Please provide the results for Joint Former trained on the MEGA dataset in detail. If the original Joint Former was not trained on this dataset, can it be trained and then compared for performance?

7.The spatial-semantic block consists of two parts: first, the global feature cls token is fused with the semantic features, and then further enhanced through Deformable Cross Attention. It is necessary to separately validate the effects of directly fusing the features versus applying Deformable Cross Attention for further enhancement.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730555928322

thread_type:
official_review

<thread_object>
creator: reviewer_dbzh

<content>
summary:
This paper addresses the complex task of tracking and segmenting multiple similar objects in long-term videos, where identifying target objects becomes challenging due to factors like occlusion, cluttered backgrounds, and appearance changes. To tackle these issues, the authors propose a new framework for robust video object segmentation, focusing on learning spatial-semantic features and generating discriminative object queries.

The framework introduces a spatial-semantic block that combines global semantic embedding with local spatial dependency modeling, which enhances the representation of target objects by capturing both broad context and fine details. Additionally, a masked cross-attention module refines object queries, concentrating on the most distinctive parts of target objects and reducing noise accumulation over time. This approach aids in effective long-term query propagation, a critical factor for high-performance tracking over extended sequences.

The experimental results are strong, showing state-of-the-art performance across several benchmarks.

strengths:
This paper’s S3 algorithm for Video Object Segmentation (VOS) demonstrates notable strengths:

1.Spatial-Semantic Integration: By combining semantic embedding with spatial dependency modeling, it effectively captures complex object structures without requiring extensive ViT retraining.

2.Discriminative Query Mechanism: The adaptive query approach improves target focus and reduces noise in long-term tracking, enhancing robustness.

3.Extensive Validation: State-of-the-art results on multiple benchmarks highlight its strong generalization across datasets.

weaknesses:
1.This paper claims to address the challenges of long-term tracking and segmentation. However, as far as I know, memory mechanisms are crucial for tackling these challenges in long-term tracking and segmentation, yet the authors do not seem to have conducted ablation experiments on the number of frames in the memory bank.

2.I believe that the ablation study on the number of queries is insufficient with only 8, 16, and 32 as tested values. A wider range of query counts should be explored to more thoroughly validate the effectiveness of the proposed method.

questions:
See weakinesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730699958919

thread_type:
official_review

<thread_object>
creator: reviewer_tgsj

<content>
summary:
This paper focuses on video object segmentation. The authors analyze the existing challenges like structural complexity, occlusion, and dramatic appearance changes, and correspondingly propose spatial-semantic feature augmentation as well as discriminative query association. The ablation studies and visualizations verify the effectiveness of each module.

strengths:
1. The motivation is clear and the architecture makes sense. Integrating high-level semantics and low-level spatial cues is promising in video object segmentation.
2. The experiments are thorough and the ablation studies can well reflect the effectiveness of each module.

weaknesses:
1. The method is complicated. What is the advantage of using spatial offsets with deformable convolution compared to simple position encodings?
2. The second row of Figure 3(a) seems strange. With semantic feature augmentation, the feature maps can well highlight the desired object instance. Adding spatial cues on the contrary suppresses the emphasis on the target instance but enhances object instances with the same semantics.
3. Compared to SAM2, which designs a memory to prompt the segmentation of new frames, what is the advantage of this architecture?

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1732029479586

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
&nbsp;&nbsp; We thank the reviewers for their feedback and valuable suggestions, which helped us further strengthen the paper. 

&nbsp;&nbsp; We are glad that all the reviewers found our approach to be **well-motivated** and **effective**, the integration of high-level semantics and low-level spatial cues to be **promising for VOS**, the **thoroughness of our experiments and ablation studies**, and the **state-of-the-art results** achieved on multiple VOS benchmarks. According to the comments, we have presented more details about the algorithm design and provided more experiments. We highlight a couple of common concerns in this general response and please find the detailed feedback under each comment.

### **(1) Experiments with a wider range of query counts to more thoroughly validate the effectiveness of the proposed method.**  
&nbsp;&nbsp; In the table below, we provide results for query numbers 4, 10, and 64. The results show that increasing query numbers does not consistently improve performance, as more queries may introduce noise and cause slow convergence. The best performance is achieved with 8 queries, which are related to the target object number (mostly 1–5) across datasets, effectively covering most samples.
| Queries | D17 (J&F) | MOSE-val (J&F) | YT19 (J&F) | LVOS (J&F) | YT18 (J&F) |
|---------|-------------|----------------|------------|------------|------------|
| 64      | 85.0        | 65.5           | 86.7       | 65.2       | 86.6       |
| 32      | 85.8        | 68.3           | 86.9       | 64.8       | 86.8       |
| 16      | 86.6        | 67.7           | 87.0       | 66.4       | 86.9       |
| 10      | 86.4        | 68.2           | 87.0       | 66.4       | 87.1       |
| **8**   | **86.7**    | **68.5**       | **87.5**   | **66.5**   | **87.4**   |
| 4       | 83.1        | 60.2           | 85.2       | 53.5       | 85.2       |

| Target numbers | YTVOS19 | MOSE | LVOSv2 |
|-------------------|---------|------|--------|
| 1                 | 168     | 188  | 91     |
| 2                 | 171     | 58   | 31     |
| 3                 | 132     | 15   | 8      |
| 4                 | 26      | 22   | 4      |
| 5                 | 7       | 11   | 0      |
| 6                 | 3       | 8    | 2      |
| 7                 | 0       | 4    | 1      |
| >=8               | 0       | 5    | 2      |

### **(2) More ablation experiments about the Spatial-Semantic block.**   
&nbsp;&nbsp; To save space, we placed the detailed ablation study in Table 7 in the Appendix section. Table 7 shows that spatial dependence modeling significantly improves performance across all VOS datasets, with a 3.0%+ J&F gain on MOSE val. This improvement comes from better detail modeling during feature extraction. Adding semantic information further boosts performance, reaching 73% J&F on the LVOS test, demonstrating its value in enhancing target semantic understanding. We provide some experiments in the table below. For more details, please refer to Table 7 in the  Appendix.

| **Dataset**                        | **MOSE** | **D17 test** | **LVOS test** | **YT19** |
|------------------------------------|----------|--------------|---------------|----------|
| **Method**                         | **J&F**  | **J&F**      | **J&F**       | **J&F**  |
| **Trained on YouTubeVOS and DAVIS datasets** |          |              |               |          |
| XMem (Baseline)                    | 53.3     | 81.0         | 50.0          | 85.5     |
| Cutie                              | 64.0     | 84.2         | 56.2          | 86.1     |
| +Discriminative Query               | 64.2     | 85.2         | 57.4          | 86.5     |
| +Spatial                            | 68.2     | 86.2         | 67.4          | 87.3     |
| +Semantic (Full)                    | 68.5     | 86.7         | 66.5          | 87.5     |
| **Trained on the MEGA datasets**   |          |              |               |          |
| Cutie                              | 69.9     | 86.1         | 66.7          | 87.0     |
| +Discriminative Query               | 70.6     | 86.6         | 66.5          | 87.5     |
| +Spatial                            | 73.5     | 87.6         | 68.8          | 87.9     |
| +Semantic (Full)                    | 74.0     | 87.8         | 73.0          | 88.1     |
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a robust video object segmentation framework by learning spatial-semantic features and discriminative object queries to address challenges encountered in the video object segmentation task. The overall idea seems like a combination of existing techniques, yet experimental results are solid. Four reviewers reach a consensus to accept this paper by discussing it with the authors through rebuttals. After reviewing the comments and rebuttals, AC agrees with the merits proposed in this paper. Therefore, AC recommends accepting this paper.