Context-Aware Video Instance Segmentation

Authors: Seunghun Lee, Jiwan Seo, Kiljoon Han, Minwoo Choi, Sunghoon Im
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Video Instance Segmentation, Contrastive Learning, Instance Prototype, Context-aware Learning

Abstract:
In this paper, we introduce the Context-Aware Video Instance Segmentation (CAVIS), a novel framework designed to enhance instance association by integrating contextual information adjacent to each object. To efficiently extract and leverage this information, we propose the Context-Aware Instance Tracker (CAIT), which merges contextual data surrounding the instances with the core instance features to improve tracking accuracy. Additionally, we introduce the Prototypical Cross-frame Contrastive (PCC) loss, which ensures consistency in object-level features across frames, thereby significantly enhancing instance matching accuracy. CAVIS demonstrates superior performance over state-of-the-art methods on all benchmark datasets in video instance segmentation (VIS) and video panoptic segmentation (VPS). Notably, our method excels on the OVIS dataset, which is known for its particularly challenging videos.

Decision: reject
Reviewer scores: 3, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730043014139

thread_type:
official_review

<thread_object>
creator: reviewer_fndi

<content>
summary:
This paper addresses the video segmentation task and presents CAVIS, a method that enhances instance association by leveraging contextual information surrounding the instances. Specifically, the paper introduces the Context-Aware Instance Tracker (CAIT), which incorporates both surrounding features and instance features. The context feature encodes the “contextual information” related to each object. To further utilize this contextual information, the authors propose context-aware instance matching through a modified cross-attention module. Additionally, to improve the learning of instance matching, the paper introduces the Prototypical Cross-frame Contrastive (PCC) loss. With these components, the proposed method achieves state-of-the-art performance across all benchmarks.

strengths:
This paper introduces a new perspective on the video segmentation task. The authors argue that better identification requires not only focusing on the object itself but also considering the contextual cues around it. This approach is well-motivated, and the experiments demonstrate the effectiveness of the proposed method.

weaknesses:
**Major Weaknesses:**

**1. Insufficient Support for the Motivation**

The proposed components are insufficient to fully support the paper’s motivation. The surrounding feature is generated by extracting the edge area of the context feature, which aggregates nearby information. However, it still contains noisy data from small regions, making it unclear whether the surrounding feature provides meaningful “contextual information” that can effectively distinguish target instances. For example, in Fig. 1, the surrounding feature of the bicycle should ideally capture information about the person riding it, but the proposed approach seems to struggle with incorporating such essential contextual details. Moreover, the ablation study on context-aware feature learning is insufficient to support the motivation. Further analysis is required to demonstrate how the extracted surrounding feature can consistently capture relevant context and contribute to better instance distinction.

**2. Similarity to previous work**

The design of the PCC loss closely resembles the appearance-guided enhancement method introduced in VISAGE [1]. In VISAGE, the appearance query is generated using the same procedure described in Eq. 11. Moreover, the generated appearance query is employed for contrastive learning to ensure high appearance similarity, which aligns closely with the motivation behind PCC. Therefore, the proposed PCC method appears to be a re-implementation of VISAGE’s approach.


**Minor Weaknesses:**

**1. Missing Citations**

Some relevant works are omitted in the citations. As mentioned above, VISAGE [1] presents a similar learning objective and also incorporates additional information to enhance tracking quality by resolving ambiguities.

**2. Unconvincing Ablation Studies**

The ablation studies are not well-designed. Since the settings for the ablation experiments differ from those in the main experiments, it is difficult to assess the impact of each component on the final performance.

**3. Redundant Notations and Reduced Readability**

The paper contains many equations and redundant notations, which detract from its readability.


[1] Kim, Hanjung, et al. "VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement." European Conference on Computer Vision. Springer, Cham, 2025.

questions:
**Q1: What Information is Encoded in the Surrounding Feature?**

To address the weakness mentioned in Major.1, it is important to analyze what type of information the surrounding feature encodes. A useful experiment would be to plot the attention map of the surrounding feature to identify which areas are being attended to. This would help determine whether the surrounding feature captures meaningful contextual information.

**Q2: Do the Ablation Study Trends Hold Across Experiment Settings?**

If the settings of the ablation studies were aligned with those of the main experiments, would the overall performance trends remain consistent? This is crucial to confirm that the ablation studies reflect the real contribution of each component under consistent conditions.

**Q3: Why Does a Wider Context Filter Reduce Performance?**

The ablation studies show that increasing the context filter size degrades performance. Why does a wider filter (e.g., size 11) lead to worse results compared to a smaller one (e.g., size 9)? Plotting the difference in attention maps between filter sizes 9 and 11 could provide insights into how the filter size influences attention distribution and performance.

**Q4: Why Use the Instance Feature as the Value in Context-Aware Matching?**

In context-aware instance matching, the value is set to the instance feature rather than the context feature. Since the context feature already encompasses the instance feature, it seems more intuitive to use the context feature as the value. Has any analysis been conducted to explain this design choice? Exploring this further could clarify whether using the context feature would offer better performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730226598646

thread_type:
official_review

<thread_object>
creator: reviewer_5yhw

<content>
summary:
This paper proposes a method called Context-Aware Video Instance Segmentation (CAVIS), which leverages contextual information around segmented instances to enhance instance differentiation across video frames. Specifically, CAVIS uses an average filter on feature maps and a Laplacian filter to extract surrounding features of each instance. By aggregating both instance features and surrounding context features during instance matching, the approach can aligns instances across frames more accurately. Additionally, a Prototypical Cross-frame Contrastive (PCC) loss is employed to further enforce inter-frame instance consistency

strengths:
1. The paper is well-organized and easy to follow.

2. The method is simple yet effective. Introducing contextual information to improve instance segmentation is a clear and logical approach.

3. Comprehensive experiments demonstrate the effectiveness of the model, which achieves state-of-the-art performance across various metrics when tested with different backbones

weaknesses:
1. Provide a clear notation table or explicitly define each variant of Q when introduced. For example, there are multiple definitions for "Q" which make the equations confusing. For Eq. (6), specify the dimension over which the average is calculated.
2. Clarify why surrounding pixels are chosen as context and how this compares to using full context information. Explain the specific benefits of using surrounding pixels for improving segmentation. Ablation study to show individual effects of CAIT and PCC components.
3. More visualization in the ablation study could strengthen the paper. In addition to t-SNE visualizations, examples of segmentation that showcase the impact of Instance Surrounding Features and Context-Aware Instance Features would help illustrate the added benefits. For example, visualizations highlighting whether in-context features enhance instance segmentation precision, particularly at object boundaries, could clarify the practical effects of the proposed method like having side-by-side comparisons of segmentation results with and without the context-aware features, focusing on challenging cases like object boundaries or occluded instances.
4. How does the addition of context features affect the computational efficiency of CAVIS? How does the performance of CAVIS change as the number of frames increases and the context potentially changes more significantly over time?

questions:
Please see the `Weaknesses’ above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730408222211

thread_type:
official_review

<thread_object>
creator: reviewer_ln3a

<content>
summary:
This paper proposes a context-aware model for video instance segmentation. Context-aware features are extracted and used for instance matching. Prototypical cross-frame contrastive loss is introduced to reduce computational cost and maintain pixel consistency. Experiments on multiple public benchmarks demonstrate the effectiveness of the proposed method.

strengths:
1.The motivation of utilizing context information for VIS to improve training accuracy makes sense.

2.The implementation of context-aware feature extraction and instance matching is reasonable.

3.The prototypical cross-frame contrastive loss improves cross-frame pixel level consistency with affordable computational cost.

4.Experiments on multiple benchmarks demonstrate the effectiveness of the proposed method. It outperforms existing works across these benchmarks.

weaknesses:
1.Table 3(d) shows that learnable filter performs worse than average filter. Although the authors explain that it's because the average filter is well-defined, but the learnable filter is not. The reviewer is still curious about the result, and more analysis should be provided. For example, will other filters, like maximum or median filter generate similar performance as the average filter?

2.The results on the more recent dataset YouTube-VIS 2022 is missing. It contains more videos than previous YouYube-VIS 2019 and 2021 benchmarks. It would be good if the experimental results on this dataset are provided.

questions:
Please check the Weaknesses part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731087399495

thread_type:
official_review

<thread_object>
creator: reviewer_rjta

<content>
summary:
The paper introduces a video instance segmentation framework focused on improving instance tracking in complex video scenarios by integrating contextual information for each instance. The authors claim two key contributions: (i) a Context-Aware Instance Tracker (CAIT) that explicitly incorporates contextual information for each instance, and (ii) a Prototypical Cross-frame Contrastive (PCC) loss to improve the consistency of enriched features across frames, further enhancing tracking robustness. Experiments are performed on the YouTube-VIS 2019, 2021, OVIS, and VIPSeg datasets.

strengths:
(i) Comprehensive experiments across several video segmentation datasets.
(ii) The paper is easy to follow.
(iii) The authors have shared the code.

weaknesses:
Weaknesses:


(i) The authors should more clearly explain the novel contributions of the proposed approach compared to closely related work, such as (Choudhuri et al., 2023). Currently, the authors mention that “Choudhuri et al., 2023 introduces a context-aware relative query which reflects global and local context by aggregating multi-level image features from consecutive frames” without explicitly explaining how the contextual information captured by the proposed method is superior. Additionally, the authors are expected to include a comparison with Choudhuri et al., 2023, and recent approaches in the state-of-the-art comparison table. Similarly, the novel contributions of PCC compared to closely related loss formulations also need to be clarified.  
Reference: Choudhuri et al., 2023: Context-Aware Relative Object Queries to Unify Video Instance and Panoptic Segmentation, CVPR 2023.

(ii) It is unclear whether the performance gain is due to the additional number of duplicate queries or from capturing contextual information, as claimed. For example, some conventional object detection methods, such as GroupDETR (ICCV 2023) [https://arxiv.org/abs/2207.13085], demonstrate the benefits of using additional object queries.

(iii) The method seems somewhat ad-hoc to me . For instance, it involves obtaining instance-level masks for each frame using Mask2Former, then using edge-filtered instance masks to capture surrounding features of instances, which are subsequently leveraged to improve mask prediction and tracking. Would performance suffer if the initial masks obtained from Mask2Former were inaccurate? In cases of heavy occlusion, I assume that temporal information from previous frames would be beneficial for accurate mask prediction as well. However, since the initial masks are predicted independently by Mask2Former, could this affect overall performance?

(iv) It appears that the proposed method mainly targets improving tracking or association performance, rather than mask quality. If so, why isn’t the proposed approach evaluated on multi-object tracking datasets like MOTS2020 and KITTI-MOTS in addition to the VIS datasets, similar to Choudhuri et al.?

questions:
See the weaknesses mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732109910014

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank all reviewers for their constructive comments. We appreciate the positive feedback on the effectiveness of the proposed method (Ln3A, 5Yhw), strong motivation behind the approach (Fndi), comprehensive experiments (rJtA, Ln3A, 5Yhw), and good readability (rJtA, 5Yhw). To effectively address the reviewers' questions, we have uploaded additional visualization materials to the [[annonymous url]](https://anonymous.4open.science/r/CAVIS_ICLR2025-73E7/rebuttal/).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a video object segmentation framework by leveraging contextual information for each instance. Context-aware features are extracted and used for matching. Prototypical cross-frame contrastive loss is introduced to reduce computational cost and maintain pixel consistency. The experimental results on YTVIS and OVIS datasets are given.

The main strengths are 1) good performance and 2) well presentation.

The main weaknesses are 1) lack of novelty (the proposed method is similar to previous methods, like VISAGE), and 2) the contribution of PCC loss is not substantial.  

After the author-reviewer discussion, the main issue of lack of novelty still remains ( recognized by Reviewers Fndi and rJtA). As this issue significantly degrades the contribution of this paper, the AC does not recommend it be accepted at this conference.
The authors are encouraged to consider the comments for their future submission.