Context-Aware Video Instance Segmentation

## Abstract

Abstract In this paper, we introduce the Context-Aware Video Instance Segmentation (CAVIS), a novel framework designed to enhance instance association by integrating contextual information adjacent to each object. To efficiently extract and leverage this information, we propose the Context-Aware Instance Tracker (CAIT), which merges contextual data surrounding the instances with the core instance features to improve tracking accuracy. Additionally, we introduce the Prototypical Cross-frame Contrastive (PCC) loss, which ensures consistency in object-level features across frames, thereby significantly enhancing instance matching accuracy. CAVIS demonstrates superior performance over state-of-the-art methods on all benchmark datasets in video instance segmentation (VIS) and video panoptic segmentation (VPS). Notably, our method excels on the OVIS dataset, which is known for its particularly challenging videos. Project page: this https URL

## 1 Introduction

Video Instance Segmentation (VIS) is a crucial task that involves segmenting and identifying individual objects within video sequences, applicable in a variety of fields including video understanding, autonomous driving, and video editing (Yang et al., ).
VIS has seen considerable advancements, with developments in both online methods (Yang et al., ; Cao et al., ; Yang et al., ; Huang et al., ; Wu et al., ; Ying et al., ; Kim et al., ), which process videos frame-by-frame to adapt in real-time, and offline methods (Wang et al., ; Hwang et al., ; Wu et al., ; Cheng et al., ; Heo et al., ), which analyze entire videos to understand inter-frame dependencies thoroughly.

Recent advances have brought robust query-based segmentation architectures (Cheng et al., ; ), designed to detect instance centers and cluster pixels into instance-specific groups within images. These modern VIS approaches increasingly employ instance center associations across frames to improve tracking accuracy.
Techniques such as contrastive learning (Wu et al., ; Ying et al., ) and transformer-based trackers (Heo et al., ; Zhang et al., ) leverage the similarities between instance centers for consistent identification of objects across frames.
However, challenges persist in scenarios with significant occlusions or when multiple similar objects are present, leading to potential tracking inaccuracies as shown in the top row of Fig. 1.
While some strategies (Heo et al., ; ) attempt to use instance features for tracking across segments or entire videos, difficulties remain.

To address this issue, we propose Context-Aware Video Instance Segmentation (CAVIS), a novel framework designed to improve object identification by incorporating contextual information surrounding each instance into the tracking process. This approach draws from insights in neuroscience and cognitive science (Bar, ; Oliva & Torralba, ), emphasizing the importance of contextual cues in human perception for deciphering complex scenes and resolving visual ambiguities.
An example of the practical application of this principle is shown in Fig. 1, where recognizing a person is riding a bicycle, rather than just identifying the bicycle, greatly enhances the accuracy of object identification.

To achieve this, we design the Context-Aware Instance Tracker (CAIT), featuring advanced modules for extracting and matching context-aware instance features. The context-aware instance feature extractor combines the contextual information at the object’s boundary with the core features of each instance.
Then, we incorporate these context-aware instance features into a transformer-based tracking architecture (Wu et al., ; Heo et al., ; Zhang et al., ), enhanced by our novel context-aware cross-attention mechanism.
This adjustment allows for the precise utilization of detailed contextual nuances within each scene.

Figure: Figure 1: Importance of contextual information. Comparative results showing the state-of-the-art model (Zhang et al., ) (Top) and CAVIS (Bottom). The frame on the left precedes the right by four frames, during which an occlusion takes place. The standard model, lacking contextual data, fails to consistently track the same bicycle post-occlusion, while CAVIS effectively maintains accurate instance tracking.
Refer to caption: x1.png

Furthermore, we introduce the Prototypical Cross-frame Contrastive (PCC) loss to ensure feature consistency across frames. Utilizing a query-based segmenter, PCC loss correlates each pixel with an object by comparing core instance features against pixel embeddings, enhancing regional consistency within frames and establishing an intra-frame constraint reflective of the feature map’s similarity. This method crucially maps inter-frame pixel relationships, bolstering object association—a vital aspect for contrastive learning that differentiates similar from dissimilar objects across frames. However, maintaining individual pixel consistency is memory-intensive. To address this, PCC loss strategically constructs instance-wise prototypes to maintain frame-to-frame consistency, thereby boosting training efficiency and ensuring robust performance in dynamic environments.

Our extensive testing shows that CAVIS significantly outperforms existing state-of-the-art methods across major video segmentation benchmarks, including YTVIS19 (Yang et al., ), YTVIS21 (Yang et al., ), OVIS (Qi et al., ), and VIPSeg (Miao et al., ), particularly excelling on OVIS dataset that include complex video sequences.
Our contributions to the field are manifold and can be summarized as follows:

- 1.
We present Context-Aware Instance Tracker (CAIT), a novel framework designed to extract context-aware instance features and utilize them for enhanced instance matching.
- 2.
We propose a Prototypical Cross-frame Contrative (PCC) loss that enhances the learning of instance matching by ensuring consistency in object-level features across frames.
- 3.
Our model demonstrates robustness in challenging videos environments, establishing state-of-the-art performance in Video Instance Segmentation and Video Panoptic Segmentation.

## 2 Related Works

Video Instance Segmentation.
VIS methods learn to associate features frame-to-frame based on instance segmentation architectures. The pioneering MaskTrack R-CNN (Yang et al., ) integrates a tracking head into Mask R-CNN (He et al., ), utilizing heuristic cues for instance association
Following advancements include SipMask (Cao et al., ) and CrossVIS (Yang et al., ), which enhance temporal links through cross-frame learning.
IDOL(Wu et al., ) a contrastive learning approach with query-based architectures (Zhu et al., ), boosting online method performance.
Conversely, offline approaches like VisTR (Wang et al., ) and Seqformer (Wu et al., ) use the entire video for mask trajectory predictions, with VisTR applying DETR (Carion et al., ) at the clip level and Seqformer aggregating temporal information via inter-frame queries.
Innovations like IFC (Hwang et al., ) and TeViT (Yang et al., ) improve efficiency by adjusting attention mechanisms within transformer architectures.

Advancements in Query-based Networks.
Strong query-based segmentation networks have become prevalent in current VIS methods, with many relying on Mask2Former (Cheng et al., ) as their foundation.
MinVIS (Huang et al., ) achieves tracking through simple post-processing based on cosine similarity between object features, without video learning.
VITA (Heo et al., ) temporally associates frame-level queries to find instance prototypes within a video.
GenVIS (Heo et al., ) adopts object association approach of VITA and designs a tracking network at the sub-clip level.
Inspired by SimCLR (Chen et al., ), CTVIS (Ying et al., ) utilizes contrastive learning with a larger number of frames for comprehensive frame association.
DVIS (Zhang et al., ) introduces a decoupled framework for VIS, dividing it into segmentation, tracking, and refinement tasks, thereby enabling efficient and effective learning.

Object Tracking with Additional Cues.
Tracking methodologies have been developed across various domains, including video object segmentation (VOS) (Xu et al., ; Oh et al., ; Cheng et al., ), multiple object tracking (MOT) (Milan et al., ; Bergmann et al., ; Zhou et al., ; Zhang et al., ), and VIS.
Despite advancements, many challenging cases persist, prompting research into object association with supplementary data.
Early approaches leverage spatial-temporal information such as geometric relation between adjacent frames (Tang et al., ) and aggregated object features of previous frames (Xu et al., ).
BeyondPixel (Sharma et al., ) improves inter-frame object matching by proposing a new cost that captures 3D pose and shape based on monocular geometry.
BATMAN (Yu et al., ) combines optical flow and object query features to encode motion and appearance information into bilateral space.
(Choudhuri et al., ) introduces a context-aware relative query which reflects global and local context by aggregating multi-level image features from consecutive frames.

## 3 Preliminary

This section offers a concise introduction to the fundamentals of a query-based instance segmentation pipeline and outlines a VIS approach that incorporates contrastive learning (Wu et al., ; Li et al., ; Ying et al., ; Li et al., ).

### 3.1 Query-based instance segmentation

Modern VIS methods adopt a query-based instance segmentation pipeline (Cheng et al., ; ), including three main components: a backbone encoder, a pixel decoder, and a transformer decoder.
The backbone encoder and pixel decoder are responsible for extracting multi-scale feature maps from the input image. The transformer decoder employs object queries—sequences of latent vectors—as initial guesses for object centers and utilizes these features to generate object-level features.
These queries undergo refinement through multiple transformer blocks via a cross-attention mechanism between the object queries and the feature maps.
The refined instance features are then used for classification and segmentation tasks through respective prediction heads.
Typically, the number of object queries, $N$, exceeds the actual number of objects, $N_{GT}$, present in the image.
Traditionally, the process involves finding a permutation of $N$ elements, $\sigma\in\mathfrak{S}_{N}$, that optimally assigns the prediction set $\{\hat{y}_{i}\}_{i=1}^{N}$ to maximize total similarity to the ground truth (GT) set $\{y_{i}\}_{i=1}^{N_{GT}}$. This is achieved by minimizing a pair-wise matching cost $\mathcal{L}_{\text{Match}}$, as defined in (Cheng et al., ):

$$ $\begin{gathered}\hat{\sigma}=\underset{\sigma\in\mathfrak{S}_{N}}{\arg\min} \sum_{k=1}^{N_{GT}}\mathcal{L}_{\text{Match}}\left(y_{k},\hat{y}_{\sigma(k)} \right).\end{gathered}$ (1) $$

The network is trained with an objective function, $\mathcal{L}_{\text{Inst}}$, which consists of a categorical loss ($\mathcal{L}_{\text{Cls}}$), a binary cross-entropy loss for masks ($\mathcal{L}_{\text{Bce}}$), and a dice loss ($\mathcal{L}_{\text{Dice}}$) with the weights $\lambda_{\text{Cls}}$, $\lambda_{\text{Bce}}$, and $\lambda_{\text{Dice}}$ balancing the contributions of each loss component as follows:

$$ $\begin{gathered}\mathcal{L}_{\text{Inst}}=\lambda_{\text{Cls}}\mathcal{L}_{ \text{Cls}}+\lambda_{\text{Bce}}\mathcal{L}_{\text{Bce}}+\lambda_{\text{Dice}} \mathcal{L}_{\text{Dice}}.\end{gathered}$ (2) $$

The loss is used to train VIS framework with the frame-wise matching relation $\hat{\sigma}^{t}$ as follows:

$$ $\begin{gathered}\mathcal{L}_{\text{VIS}}=\sum_{t=1}^{T}\sum_{n=1}^{N_{GT}} \mathcal{L}_{\text{Inst}}\left(y^{t}_{n},\hat{y}^{t}_{\hat{\sigma}^{t}(n)} \right).\end{gathered}$ (3) $$

### 3.2 Contrastive learning for VIS

In query-based architectures, the order of instance features effectively serves as the identity of each object. By aligning the sequence of instance features across frames, we can facilitate object tracking. Since instance features represent specific objects, inter-frame feature association is used for this alignment.
To enhance the robustness of instance features for matching objects between frames, the following contrastive loss is integrated within the VIS framework (Wu et al., ):

$$ $\displaystyle\mathcal{L}_{\text{Emb}}(v_{t})$ $\displaystyle=-\log\frac{\sum_{k^{+}\in\mathbf{K}^{+}_{{v_{t}}}}\exp\left(v_{t }\cdot k^{+}\right)}{\sum_{k^{+}\in\mathbf{K}^{+}_{{v_{t}}}}\exp\left(v_{t} \cdot k^{+}\right)+\sum_{k^{-}\in\mathbf{K}^{-}_{{v_{t}}}}\exp\left(v_{t}\cdot k ^{-}\right)},$ (4) $\displaystyle=\log\left[1+\sum_{k^{+}\in\mathbf{K}^{+}_{{v_{t}}}}\sum_{k^{-} \in\mathbf{K}^{-}_{{v_{t}}}}\exp\left(v_{t}\cdot k^{-}-v_{t}\cdot k^{+}\right) \right],~{}~{}\forall t\in\{1,...,T\},$ $$

where $\mathbf{K}^{+}_{v_{t}}$ represents the sets of positive embeddings corresponding to the same object as $v_{t}$ from frames other than the $t$-th frame, while $\mathbf{K}^{-}_{v_{t}}$ includes negative embeddings featuring characteristics of objects different from that of $v_{t}$.

## 4 Method

This section describes our Context-Aware Video Instance Segmentation (CAVIS) whose overall pipeline is illustrated in Fig. 2.
Our CAVIS consists of two key components: Context-Aware Instance Tracker (CAIT) and Prototypical Cross-frame Contrastive (PCC) loss, which are detailed in Sec. 4.1 and Sec. 4.2, respectively.
We describe the training losses for each network in Sec. 4.3.

Figure: Figure 2: Overview of CAVIS. (a) The extraction of context-aware instance features from the output of an instance segmentation network. (b) CAVIS pipeline through context-aware instance matching. This includes the organization of the surrounding instance features $\tilde{Q}_{t}$, facilitated by Hungarian matching between the ordered instance features $\hat{Q}^{*}_{t}$ and unordered instance features $\hat{Q}_{t}$.
Refer to caption: x2.png

### 4.1 Context-Aware Instance Tracker

#### 4.1.1 Context-aware feature extraction

Following the method outlined in previous VIS studies (Huang et al., ; Heo et al., ; ; Ying et al., ; Zhang et al., ), we employ Mask2Former (Cheng et al., ) as our segmentation network $\mathcal{S}$.
This framework ingests a series of input frames $\{I_{t}\}_{t=1}^{T}$, with $T$ denoting the total number of frames. It extracts feature maps $F$, identifies instance features $\hat{Q}$, and computes both classification scores $O$ and generates segmentation masks $M$ as follows:

$$ $\displaystyle\left\{F_{t},\hat{Q}_{t},O_{t},M_{t}\right\}_{t=1}^{T}$ $\displaystyle=\mathcal{S}\left(\left\{I_{t}\right\}_{t=1}^{T}\right),$ (5) $\displaystyle F_{t}\in\mathbb{R}^{H\times W\times C},~{}\hat{Q}_{t}\in\mathbb{ R}^{N\times C},$ $\displaystyle~{}O_{t}\in\mathbb{R}^{N\times K},~{}M_{t}\in\mathbb{R}^{N\times H \times W},$ $$

where $H$, $W$, and $C$ denote the height, width, and channel dimensions of the feature maps, respectively. $N$ indicates the maximum number of detactable objects in a single frame, and $K$ signifies the number of object classes.
We then extract the instance surrounding features $\tilde{Q}_{t}\in\mathbb{R}^{N\times C}$ capturing data around the object’s boundaries essential for detailed context analysis as follows:

$$ $\displaystyle\begin{split}\tilde{Q}^{n}_{t}=&\frac{\sum_{h=1}^{H}\sum_{w=1}^{W }\bar{F}^{\{h,w\}}_{t}*\mathbbm{1}\left(\acute{M}_{t}^{\{n,h,w\}}>0\right)}{ \sum_{h=1}^{H}\sum_{w=1}^{W}\mathbbm{1}\left(\acute{M}_{t}^{\{n,h,w\}}>0\right )},~{}~{}\forall n=\{1,...,N\},\\ &\text{where}~{}\bar{F}=\text{Avg}(F),~{}\acute{M}=\text{Lap}(M),\end{split}$ (6) $$

where $\text{Avg}(\cdot)$ denotes an average filtering process, $\text{Lap}(\cdot)$ signifies the application of a Laplacian filter, and $\mathbbm{1}(\cdot)$ is the indicator function that tests for the presence of the object within the filtered mask. We employ the average filter specifically configured with a $9\times 9$ kernel size.

Finally, we combine the core and surrounding features to further enhance instance representations. The context-aware instance feature $Q^{n}_{t}\in\mathbb{R}^{C}$ is generated by concatenating the core instance feature $\hat{Q}^{n}_{t}$ and the instance surrounding feature $\tilde{Q}^{n}_{t}$, and subsequently processing this combined feature through a multi-layer perceptron (MLP) as follows:

$$ $Q^{n}_{t}=\text{MLP}\left(\text{Concat}\left(\hat{Q}^{n}_{t},\tilde{Q}^{n}_{t} \right)\right),~{}~{}\forall n=\{1,...,N\}.$ (7) $$

The MLP is structured with three linear layers, each followed by a ReLU activation function.
To promote the learning of discriminative context-aware instance features, we implement a contrastive loss specifically for these features. The context-aware contrastive loss, denoted as $\mathcal{L}_{\text{CTX}}$, leverages the established contrastive loss framework $\mathcal{L}_{\text{Emb}}$ detailed in Eq. (4) as follows:

$$ $\begin{gathered}\mathcal{L}_{\text{CTX}}=\sum_{t=1}^{T}\sum_{n=1}^{N_{GT}} \mathcal{L}_{\text{Emb}}\left(Q_{t}^{\hat{\sigma}^{t}(n)}\right),\end{gathered}$ (8) $$

where $\hat{\sigma}^{t}$ is a frame-wise matching relation as described in Eq. (1).
This design enhances our ability to identify and track the same instances across the entire video. By not solely relying on instance centers, and instead utilizing context-aware instance embeddings, we can more accurately recognize instances throughout the video sequence.

#### 4.1.2 Context-aware instance matching

We introduce our context-aware tracking network $\mathcal{T}$, which employs a transformer-based tracking network (Heo et al., ; Zhang et al., ) to learn associations across adjacent frames. The network comprises six transformer blocks, each featuring cross-attention, self-attention, and feed-forward layers.
The conventional cross-attention mechanism, denoted as $\text{Attn}(Q,K,V)$, traditionally aligns the current unordered instance features, $\hat{Q}_{t}$, (serving as both the key and value), with the ordered instance features from the previous frame, $\hat{Q}^{*}_{t-1}$, (used as the query).
To enhance accuracy, our model employs context-aware instance features, ${Q}^{*}_{t-1}$ and ${Q}_{t}$, as the query and key, respectively while we still use the instance features $\hat{Q}_{t}$ as value.
This modification leads to a context-aware cross-attention mechanism, formulated as:

$$ $\text{Attn}({Q}_{t-1}^{*},{Q}_{t},\hat{Q}_{t})=\text{softmax}\left(\frac{{Q}_{ t-1}^{*}\cdot\left({Q}_{t}\right)^{T}}{\sqrt{C}}\right)\hat{Q}_{t},$ (9) $$

where $C$ is the channel dimensions of the feature maps.
The aligned context-aware instance features $Q_{t}^{*}$ is built by concatenating the aligned instance features $\hat{Q}_{t}^{*}$ and the aligned instance surrounding features $\tilde{Q}_{t}^{*}$ same as in Eq. (7).
We obtain the aligned instance surrounding features $\tilde{Q}_{t}^{*}$ by using Hungarian matching algorithm (Kuhn, ) on cosine similarity between $\hat{Q}_{t}$ and $\hat{Q}_{t}^{*}$ as follows:

$$ $\displaystyle\begin{split}\tilde{Q}_{t}^{*^{\sigma_{H}(n)}}=\tilde{Q}_{t}^{n}, ~{}\forall n=\{1,...,N\},~{}\text{where}~{}\sigma_{H}=~{}\text{Hungarian}(\hat {Q}_{t}^{*},\hat{Q}_{t}),~{}\sigma_{H}\in\mathbb{R}^{N}.\end{split}$ (10) $$

This process is similarly applied to the aligned context-aware features $Q_{t-1}^{*}$ for the previous frame.

### 4.2 Prototypical Cross-frame Contrastive Loss

Our method employs a query-based segmenter to assess each pixel’s relationship to an object by comparing core instance features, $\hat{Q}_{t}^{n}\in\mathbb{R}^{C}$, with corresponding pixel embeddings, $F_{t}^{\{h,w\}}\in\mathbb{R}^{C}$.
This process ensures consistent feature representation within object-containing regions and introduces an intra-frame constraint that reflects similarity within the feature map. By examining the inter-frame relationships of pixel embeddings, our method improves object association, essential for contrastive learning methods that strive to differentiate between similar and dissimilar objects across frames.

Given the memory-intensive nature of maintaining individual pixel consistency, we introduce Prototypical Cross-frame Contrastive (PCC) loss. This loss $\mathcal{L}_{\text{PCC}}$ maintains frame-to-frame consistency of pixel embeddings for each instance feature by constructing instance-wise prototypes from predicted masks, defined as follows:

$$ $\mathcal{L}_{\text{PCC}}=\sum_{t=1}^{T}\sum_{n=1}^{N_{GT}}\mathcal{L}_{\text{ Emb}}\left(\eta_{t}^{\hat{\sigma}^{t}(n)}\right),~{}~{}\eta_{t}^{n}=\frac{\sum _{h=1}^{H}\sum_{w=1}^{W}F_{t}^{\{h,w\}}*\mathbbm{1}\left(M_{t}^{\{h,w\}}==1 \right)}{\sum_{h=1}^{H}\sum_{w=1}^{W}\mathbbm{1}\left(M_{t}^{\{h,w\}}==1\right )}.$ (11) $$

### 4.3 Training Loss

To train the segmentation network $\mathcal{S}$, we implement an objective function that incorporates the standard video instance segmentation loss $\mathcal{L}_{\text{VIS}}$ in Eq. (3), context-aware constrative loss $\mathcal{L}_{\text{CTX}}$ in Eq. (8) and Prototypical Cross-frame Contrastive (PCC) loss in Eq. (11) as follows:

$$ $\begin{gathered}\mathcal{L}_{\mathcal{S}}=\mathcal{L}_{\text{VIS}}+\lambda_{ \text{CTX}}\mathcal{L}_{\text{CTX}}+\lambda_{\text{PCC}}\mathcal{L}_{\text{PCC }},\end{gathered}$ (12) $$

where $\lambda_{\text{CTX}}$ and $\lambda_{\text{PCC}}$ are the weights assigned to balance these losses.

To train the tracking network $\mathcal{T}$, we calculate the matching cost only for objects that appear for the first time to ensure consistent video-level matching pairs, as implemented in prior work (Zhang et al., ).
The objective function $\mathcal{L}_{\mathcal{T}}$ incorporating the matching relation $\hat{\sigma}_{\text{con}}$ is defined as follows:

$$ $\begin{gathered}\mathcal{L}_{\mathcal{T}}=\sum_{t=1}^{T}\sum_{n=1}^{N_{GT}} \mathcal{L}_{\text{Inst}}\left(y^{t}_{n},\hat{y}^{t}_{\hat{\sigma}_{\text{con} }(n)}\right),~{}~{}\hat{\sigma}_{\text{con}}=\underset{\sigma\in\mathfrak{S}_{ N}}{\arg\min}\sum_{k=1}^{N_{GT}}\mathcal{L}_{\text{Match }}\left(y_{k}^{f(k)}, \hat{y}_{\sigma(k)}^{f(k)}\right),\end{gathered}$ (13) $$

where $f(k)$ denotes the frame in which the $k$-th instance first appears.

## 5 Experiments

We evaluate CAVIS on two major tasks: video instance segmentation (VIS) and video panoptic segmentation (VPS) on four benchmark datasets recognized for their challenges and prevalence in the research community: YouTubeVIS-2019 (Yang et al., ), YouTubeVIS-21 (Yang et al., ), OVIS (Qi et al., ), and VIPSeg (Miao et al., ).
For VIS, performance metrics include average precision (AP) and average recall (AR) as established in previous studies (Yang et al., ).
In the realm of VPS (Kim et al., ), we further examine our model’s capabilities using the Segmentation and Tracking Quality (STQ) metric, and the Video Panoptic Quality (VPQ) metric.

Table: Table 1: Comparisons on the validation sets of YouTube-VIS 2019, 2021, and OVIS datasets. The best and second-best scores are highlighted in red and blue, respectively. $\dagger$ denotes the model trained with the temporal refiner (Zhang et al., ). Rows in cyan indicate comparisons with a top-performing model.

### 5.1 Implementation Details

We employ Mask2Former (Cheng et al., ) as our segmentation network, utilizing three distinct backbone encoders: ResNet-50 (He et al., ), Swin-L (Liu et al., ), and ViT-L (Dosovitskiy et al., ). The ResNet-50 and Swin-L backbones are initialized with parameters pre-trained on the COCO dataset (Lin et al., ), while the ViT-L backbone uses initialization parameters from DINOv2 (Oquab et al., ). Additionally, for the ViT-L backbone, we employ a memory-efficient version of VIT-Adapter (Chen et al., ), aligning with recent advancements in network efficiency (Zhang et al., ).
Further details are described in Sec. A.2.2.

### 5.2 Comparison to State-of-the-Art Methods

Video Instance Segmentation (VIS).
We benchmark CAVIS against leading methods on three established VIS datasets, as detailed in Tab. 1. CAVIS sets a new state-of-the-art, outperforming the previous top model, DVIS++, by margins of 1.1, 1.4, and 3.7 average precision (AP) points on YouTube-VIS2019, YouTube-VIS2021, and OVIS, respectively. Particularly noteworthy is CAVIS’s performance on the OVIS dataset, where it significantly outstrips all competitors. This dataset is renowned for its diversity and the complexity of its video sequences.
Fig. 3 illustrates how our model proficiently tracks objects even in scenarios marked by severe occlusion.
This capability underscores the strength of our context-aware video learning approach, which effectively leverages information from surrounding objects for accurate instance matching, even under severe occlusion.

Video Panoptic Segmentation (VPS).
In the realm of VPS, CAVIS also achieves the best performance on the VIPSeg dataset as shown in Tab. 2. For the ResNet-50 backbone, it achieves 45.3 in both Video Panoptic Quality (VPQ) and Segmentation and Tracking Quality (STQ). For the ViT-L backbone, the figures reach 58.5 VPQ and 56.1 STQ, demonstrating substantial advancements. Specifically, our model shows significant gains in $\text{VPQ}^{\text{Th}}$—which assesses performance on ‘thing’ classes—with increases of 3.0 and 1.9 for ResNet-50 and ViT-L backbones, respectively, over the previous best models. These improvements highlight the versatility of our context-aware object matching strategy across various video segmentation tasks.

Figure: Figure 3: Qualitative comparisons of CAVIS (ours) against state-of-the-art methods: CTVIS (Ying et al., ) and DVIS++ (Zhang et al., ) on the OVIS dataset.
Refer to caption: x3.png

### 5.3 Ablation Study

We conduct ablation studies on the OVIS dataset (Qi et al., ) with the ResNet-50 (He et al., ) backbone over 40k training iterations, detailed in Tab. 3(f).
For these tests, we sample two frames from each input video for experiments listed in Tab. 3(f)-(a), (b), and (d).
To evaluate the segmentation network across these setups in Tab. 3(f)-(a-d), we employ the minimal post-processing method proposed by MinVIS (Huang et al., ).

Ablation study on technical contributions of CAVIS.
We conduct a series of experiments to demonstrate the effectiveness of our key components: the context-aware instance tracker (CAIT) and prototypical cross-frame contrastive (PCC) loss.
The results, shown in Tab. 3(f)-(a) for the segmentation network and Tab. 3(f)-(e) for the tracking network, demonstrate significant performance enhancements.

Tab. 3(f)-(a) includes six experiments (i- vi), where experiment (i) present our baseline performance of retraining MinVIS (Huang et al., ) to match our settings.
Experiments (i- iii) show that implementing contrastive learning, whether with standard or context-aware instance features, leads to significant performance gains. Particularly, context-aware instance features result in a notable +2.7 AP improvement over the baseline, a considerable increase compared to the +1.3 AP improvement observed with standard instance features.
Further performance boosts are noted when PCC is introduced in experiments (iv- vi). The most substantial improvement is seen in experiment (vi), which records the highest performance of 29.5 AP by utilizing both $\mathcal{L}_{\text{CTX}}$ and $\mathcal{L}_{\text{PCC}}$. This highlights the synergistic effect of integrating context-aware tracking with cross-frame contrastive loss, significantly enhancing the system’s accuracy and effectiveness.

Table: Table 2: Comparison on VIPSeg validation sets. ‘Th’ and ‘St’ denote ‘things’ and ‘stuff’ classes.

In Tab. 3(f)-(e), we evaluate the tracking network by initially using a fixed pre-trained segmentation network. Employing standard cross-attention with $\hat{Q}$ achieved a high performance of 34.4 AP. Substituting this with our newly designed context-aware cross-attention further increases the performance by +1.7 AP, reaching 36.1 AP. These findings validate the efficacy of the context-aware feature, confirming its significant advantages for instance matching in complex video scenarios.

**Table 3: Ablation studies on each component of CAVIS. (a-d) present the results from the segmentation network, while the others present those from the tracking network. “CL” denotes contrastive learning.**
|  | CL with $\hat{Q}$ | $\mathcal{L}_{\text{CTX}}$ | $\mathcal{L}_{\text{PCC}}$ | AP |
| --- | --- | --- | --- | --- |
| (i) |  |  |  | 26.4 |
| (ii) | ✓ |  |  | 27.9 |
| (iii) |  | ✓ |  | 29.1 |
| (iv) |  |  | ✓ | 27.6 |
| (v) | ✓ |  | ✓ | 28.3 |
| (vi) |  | ✓ | ✓ | 29.5 |

Context filter.
Given that images feature objects at various scales, identifying the optimal receptive field size that functions effectively across different scenarios is essential. Our experiments, detailed in Tab. 3(f)-(b), explore the effects of varying context filter sizes from 3 to 11. The optimal performance is achieved with a filter size of 9; larger sizes led to decreased performance, suggesting that excessively large receptive fields may detract from effective object matching by homogenizing the context features across all objects.
Further investigation into different types of context filters shown in Tab. 3(f)-(d) reveals that the average filter, which evenly reflects surrounding information, offers a well-defined benefit. In contrast, the learnable filter, which lacks specific directives on characterizing surrounding information, performs similarly to scenarios without enhanced context features, as demonstrated in Tab. 3(f)-(a)-(v). This indicates the importance of a clearly defined context filter in improving the segmentation and tracking accuracy.

The number of adjacent frames used during training.
Our fundamental assumption is that the surrounding information between adjacent frames remains relatively stable, thereby aiding object matching. Tab. 3(f)-(c) details the performance comparison based on the number of adjacent frames used during training. Utilizing three frames results in the highest performance, achieving 30.0 AP. Increasing the number of frames decreases performance, likely due to larger gaps between sampled frames which lead to more significant changes in the surrounding information, thus complicating object matching. Consequently, we have found that using three frames optimizes training effectiveness.

Context feature alignment.
The tracking network aligns instance features and produces outputs in varying orders, necessitating the precise alignment of context features for accurate matching in subsequent frames. Misalignment of these features can lead to incorrect matching of context information for each object, significantly impacting performance. As demonstrated in Tab. 3(f)-(f), misalignment results in a notable performance drop of 3.3 AP. This underscores the critical need for accurate alignment of context information to ensure robust object tracking performance.

## 6 Conclusion

In this paper, we introduce Context-Aware Video Instance Segmentation (CAVIS), a pioneering framework designed to enhance the accuracy and reliability of object tracking in complex video scenarios by integrating contextual information surrounding each instance. The introduction of the Context-Aware Instance Tracker (CAIT) and the innovative Prototypical Cross-frame Contrastive (PCC) loss are central to CAVIS’s effectiveness. CAIT leverages the surrounding context to enrich the core features of each instance, providing a more holistic view that significantly improves instance detection and segmentation under challenging conditions. Simultaneously, PCC loss ensures consistency of these enriched features across frames, reinforcing the temporal linkage between instances and enhancing the overall tracking robustness.
Our experiments across multiple challenging benchmarks demonstrate that CAVIS significantly outperforms existing state-of-the-art methods, particularly excelling in scenarios that demand robust tracking capabilities. The integration of CAIT and PCC not only addresses the primary challenges of occlusions and motion but also effectively manages the presence of visually similar objects that can often mislead traditional VIS approaches.

#### Acknowledgments

This work was supported by Korea Research Institute for defense Technology planning and advancement through Defense Innovation Vanguard Enterprise Project, funded by Defense Acquisition Program Administration(R230206), Institute of Information & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea government(MSIT) (No.2014-3-00123, Development of High Performance Visual BigData Discovery Platform for Large-Scale Realtime Data Analysis) and the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00210908).

## Appendix A Appendix

### A.1 Limitation

Video Instance Segmentation (VIS) is an advanced technology designed to perform segmentation and tracking concurrently, capturing the trajectories of individual instances within a video. While this technology has significant benefits, it also poses potential risks if misused, particularly in surveillance applications. Such misuse could lead to severe privacy infringements.
It is important to note, however, that the dataset used in this study is a standard one within the VIS community and does not include any sensitive or personal information. This precaution helps mitigate the risk of our trained model being used for harmful purposes. Nonetheless, the potential for negative impacts should not be underestimated, and ethical considerations must guide the deployment of VIS technologies.

### A.2 Experimental Details

#### A.2.1 Datasets

Youtube-VIS 2019 and 2021 YouTube-VIS was introduced by Yang et al. in their pioneering study on the VIS task (Yang et al., ). This dataset comprises high-resolution YouTube videos, categorized into 40 distinct classes. The 2019 version of the dataset includes 2,238 videos for training, 302 for validation, and 343 for testing (Yang et al., ). The 2021 update expands these numbers to 2,985, 421, and 453 videos for training, validation, and testing, respectively (Yang et al., ). YouTube-VIS is utilized across various pixel-level video understanding tasks, including VIS, video semantic segmentation, and video object detection.

OVIS The OVIS dataset (Qi et al., ) presents a significant challenge with its frequent occlusions and a realistic representation of common everyday objects. This makes it highly relevant for real-world applications. OVIS videos are longer and contain more objects compared to those in YouTube-VIS, which increases the complexity of segmentation and tracking tasks. The dataset is organized into training, validation, and test sets, with 607, 140, and 154 videos, respectively.

VIPSeg VIPSeg (Miao et al., ) is a comprehensive Video Panoptic Segmentation dataset that includes 3,536 videos and 84,750 frames, annotated with pixel-level panoptic labels. Unlike earlier VPS datasets that primarily focus on street views, VIPSeg offers a broader range of challenges and practical scenarios. It features 232 diverse settings and is annotated with 58 ‘thing’ classes and 66 ‘stuff’ classes, making it one of the most diverse and challenging datasets available in the field.

#### A.2.2 Implementation

Our segmentation approach employs the Mask2Former architecture (Cheng et al., ), utilizing the officially recommended hyperparameters. For all experimental settings, we follow established practices by incorporating COCO joint training, as adopted in previous methodologies (Wu et al., ; Heo et al., ; ; Ying et al., ; Zhang et al., ).
The tracking network consists of six transformer blocks.
Within the tracking network’s transformer blocks, we innovate by replacing the standard cross-attention layer with the referring cross-attention layer, as introduced in (Zhang et al., ).
Additionally, we conduct experiments with the temporal refiner (Zhang et al., ) over 160k iterations, specifically analyzing sequences of 15 consecutive frames to enhance tracking accuracy.

For efficient training, we adopt a staged approach where the segmentation network is trained first, followed by the tracking network with all other parameters frozen, promoting stability and efficiency in learning, as suggested by previous studies (Zhang et al., ; Li et al., ). Optimization is carried out using the AdamW optimizer (Loshchilov & Hutter, ), with a starting learning rate of 1e-4 and a weight decay of 5e-2.
The training process spans 40k iterations for the segmentation network and 160k iterations for the tracking network, with learning rate reductions scheduled at 28k and 112k iterations, respectively.
During training, we sample three frames for the segmentation network and five frames for the tracking network from each of eight batched videos. These frames undergo resizing to ensure the shorter side is between 320 and 640 pixels, while the longer side does not exceed 768 pixels. The loss function weights are set to $\lambda_{\text{cls}}=2.0$, $\lambda_{\text{bce}}=5.0$, $\lambda_{\text{dice}}=5.0$, $\lambda_{\text{ctx}}=2.0$, and $\lambda_{\text{pro}}=2.0$ to balance the contributions of each component during training.
For inference, the shorter side of input frames is scaled down to 448 pixels to maintain a consistent aspect ratio across inputs. All experiments are conducted using 8 RTX2080Ti GPUs for the ResNet-50 backbone and 8 RTX3090 GPUs for the Swin-L and ViT-L backbones, ensuring adequate computational resources are available for the demands of each model configuration.

### A.3 Additional Experiments

Analysis on object embeddings.
To demonstrate the effectiveness of our context-aware instance learning, we compare the distribution of object embeddings from three different models, as shown in Fig. 4.
MinVIS does not engage in video learning, resulting in less effective distinction between objects.
Compared to MinVIS, CTVIS shows a clearer object distinction by employing contrastive learning among object embeddings, but it still exhibits some overlaps in object clusters.
In contrast, CAVIS forms much more distinct object clusters, highlighting the advantage of leveraging contextual information for object identification.

Figure: Figure 4: Visualization of object embeddings. Each point on the t-SNE (Van der Maaten & Hinton, ) plot represents the learned object embeddings. The three different colors of points indicate the embeddings of three different elephants throughout the entire video.
Refer to caption: x4.png

Qualitative results.
We provide additional qualitative results of CAVIS across various datasets, as depicted in Fig. 5-8. These results underscore the robust capability of CAVIS to track objects in diverse scenarios for both VIS and VPS tasks. Notably, CAVIS excels in environments featuring numerous similar objects, fast-moving objects, and significant occlusions, demonstrating its effectiveness across complex dynamic scenes.

Figure: Figure 5: Additional qualitative results on OVIS dataset.
Refer to caption: x5.png

Figure: Figure 6: Additional qualitative results on Youtube-VIS 2019 dataset.
Refer to caption: x6.png

Figure: Figure 7: Additional qualitative results on Youtube-VIS 2021 dataset.
Refer to caption: x7.png

Figure: Figure 8: Additional qualitative results on VIPSeg dataset.
Refer to caption: x8.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: See the weaknesses mentioned above.
soundness: 2
strengths: (i) Comprehensive experiments across several video segmentation datasets.
(ii) The paper is easy to follow.
(iii) The authors have shared the code.
summary: The paper introduces a video instance segmentation framework focused on improving instance tracking in complex video scenarios by integrating contextual information for each instance. The authors claim two key contributions: (i) a Context-Aware Instance Tracker (CAIT) that explicitly incorporates contextual information for each instance, and (ii) a Prototypical Cross-frame Contrastive (PCC) loss to improve the consistency of enriched features across frames, further enhancing tracking robustness. Experiments are performed on the YouTube-VIS 2019, 2021, OVIS, and VIPSeg datasets.
weaknesses: Weaknesses:


(i) The authors should more clearly explain the novel contributions of the proposed approach compared to closely related work, such as (Choudhuri et al., 2023). Currently, the authors mention that “Choudhuri et al., 2023 introduces a context-aware relative query which reflects global and local context by aggregating multi-level image features from consecutive frames” without explicitly explaining how the contextual information captured by the proposed method is superior. Additionally, the authors are expected to include a comparison with Choudhuri et al., 2023, and recent approaches in the state-of-the-art comparison table. Similarly, the novel contributions of PCC compared to closely related loss formulations also need to be clarified.  
Reference: Choudhuri et al., 2023: Context-Aware Relative Object Queries to Unify Video Instance and Panoptic Segmentation, CVPR 2023.

(ii) It is unclear whether the performance gain is due to the additional number of duplicate queries or from capturing contextual information, as claimed. For example, some conventional object detection methods, such as GroupDETR (ICCV 2023) [https://arxiv.org/abs/2207.13085], demonstrate the benefits of using additional object queries.

(iii) The method seems somewhat ad-hoc to me . For instance, it involves obtaining instance-level masks for each frame using Mask2Former, then using edge-filtered instance masks to capture surrounding features of instances, which are subsequently leveraged to improve mask prediction and tracking. Would performance suffer if the initial masks obtained from Mask2Former were inaccurate? In cases of heavy occlusion, I assume that temporal information from previous frames would be beneficial for accurate mask prediction as well. However, since the initial masks are predicted independently by Mask2Former, could this affect overall performance?

(iv) It appears that the proposed method mainly targets improving tracking or association performance, rather than mask quality. If so, why isn’t the proposed approach evaluated on multi-object tracking datasets like MOTS2020 and KITTI-MOTS in addition to the VIS datasets, similar to Choudhuri et al.?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please check the Weaknesses part.
soundness: 3
strengths: 1.The motivation of utilizing context information for VIS to improve training accuracy makes sense.

2.The implementation of context-aware feature extraction and instance matching is reasonable.

3.The prototypical cross-frame contrastive loss improves cross-frame pixel level consistency with affordable computational cost.

4.Experiments on multiple benchmarks demonstrate the effectiveness of the proposed method. It outperforms existing works across these benchmarks.
summary: This paper proposes a context-aware model for video instance segmentation. Context-aware features are extracted and used for instance matching. Prototypical cross-frame contrastive loss is introduced to reduce computational cost and maintain pixel consistency. Experiments on multiple public benchmarks demonstrate the effectiveness of the proposed method.
weaknesses: 1.Table 3(d) shows that learnable filter performs worse than average filter. Although the authors explain that it's because the average filter is well-defined, but the learnable filter is not. The reviewer is still curious about the result, and more analysis should be provided. For example, will other filters, like maximum or median filter generate similar performance as the average filter?

2.The results on the more recent dataset YouTube-VIS 2022 is missing. It contains more videos than previous YouYube-VIS 2019 and 2021 benchmarks. It would be good if the experimental results on this dataset are provided.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: Please see the `Weaknesses’ above.
soundness: 2
strengths: 1. The paper is well-organized and easy to follow.

2. The method is simple yet effective. Introducing contextual information to improve instance segmentation is a clear and logical approach.

3. Comprehensive experiments demonstrate the effectiveness of the model, which achieves state-of-the-art performance across various metrics when tested with different backbones
summary: This paper proposes a method called Context-Aware Video Instance Segmentation (CAVIS), which leverages contextual information around segmented instances to enhance instance differentiation across video frames. Specifically, CAVIS uses an average filter on feature maps and a Laplacian filter to extract surrounding features of each instance. By aggregating both instance features and surrounding context features during instance matching, the approach can aligns instances across frames more accurately. Additionally, a Prototypical Cross-frame Contrastive (PCC) loss is employed to further enforce inter-frame instance consistency
weaknesses: 1. Provide a clear notation table or explicitly define each variant of Q when introduced. For example, there are multiple definitions for "Q" which make the equations confusing. For Eq. (6), specify the dimension over which the average is calculated.
2. Clarify why surrounding pixels are chosen as context and how this compares to using full context information. Explain the specific benefits of using surrounding pixels for improving segmentation. Ablation study to show individual effects of CAIT and PCC components.
3. More visualization in the ablation study could strengthen the paper. In addition to t-SNE visualizations, examples of segmentation that showcase the impact of Instance Surrounding Features and Context-Aware Instance Features would help illustrate the added benefits. For example, visualizations highlighting whether in-context features enhance instance segmentation precision, particularly at object boundaries, could clarify the practical effects of the proposed method like having side-by-side comparisons of segmentation results with and without the context-aware features, focusing on challenging cases like object boundaries or occluded instances.
4. How does the addition of context features affect the computational efficiency of CAVIS? How does the performance of CAVIS change as the number of frames increases and the context potentially changes more significantly over time?

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: **Q1: What Information is Encoded in the Surrounding Feature?**

To address the weakness mentioned in Major.1, it is important to analyze what type of information the surrounding feature encodes. A useful experiment would be to plot the attention map of the surrounding feature to identify which areas are being attended to. This would help determine whether the surrounding feature captures meaningful contextual information.

**Q2: Do the Ablation Study Trends Hold Across Experiment Settings?**

If the settings of the ablation studies were aligned with those of the main experiments, would the overall performance trends remain consistent? This is crucial to confirm that the ablation studies reflect the real contribution of each component under consistent conditions.

**Q3: Why Does a Wider Context Filter Reduce Performance?**

The ablation studies show that increasing the context filter size degrades performance. Why does a wider filter (e.g., size 11) lead to worse results compared to a smaller one (e.g., size 9)? Plotting the difference in attention maps between filter sizes 9 and 11 could provide insights into how the filter size influences attention distribution and performance.

**Q4: Why Use the Instance Feature as the Value in Context-Aware Matching?**

In context-aware instance matching, the value is set to the instance feature rather than the context feature. Since the context feature already encompasses the instance feature, it seems more intuitive to use the context feature as the value. Has any analysis been conducted to explain this design choice? Exploring this further could clarify whether using the context feature would offer better performance.
soundness: 2
strengths: This paper introduces a new perspective on the video segmentation task. The authors argue that better identification requires not only focusing on the object itself but also considering the contextual cues around it. This approach is well-motivated, and the experiments demonstrate the effectiveness of the proposed method.
summary: This paper addresses the video segmentation task and presents CAVIS, a method that enhances instance association by leveraging contextual information surrounding the instances. Specifically, the paper introduces the Context-Aware Instance Tracker (CAIT), which incorporates both surrounding features and instance features. The context feature encodes the “contextual information” related to each object. To further utilize this contextual information, the authors propose context-aware instance matching through a modified cross-attention module. Additionally, to improve the learning of instance matching, the paper introduces the Prototypical Cross-frame Contrastive (PCC) loss. With these components, the proposed method achieves state-of-the-art performance across all benchmarks.
weaknesses: **Major Weaknesses:**

**1. Insufficient Support for the Motivation**

The proposed components are insufficient to fully support the paper’s motivation. The surrounding feature is generated by extracting the edge area of the context feature, which aggregates nearby information. However, it still contains noisy data from small regions, making it unclear whether the surrounding feature provides meaningful “contextual information” that can effectively distinguish target instances. For example, in Fig. 1, the surrounding feature of the bicycle should ideally capture information about the person riding it, but the proposed approach seems to struggle with incorporating such essential contextual details. Moreover, the ablation study on context-aware feature learning is insufficient to support the motivation. Further analysis is required to demonstrate how the extracted surrounding feature can consistently capture relevant context and contribute to better instance distinction.

**2. Similarity to previous work**

The design of the PCC loss closely resembles the appearance-guided enhancement method introduced in VISAGE [1]. In VISAGE, the appearance query is generated using the same procedure described in Eq. 11. Moreover, the generated appearance query is employed for contrastive learning to ensure high appearance similarity, which aligns closely with the motivation behind PCC. Therefore, the proposed PCC method appears to be a re-implementation of VISAGE’s approach.


**Minor Weaknesses:**

**1. Missing Citations**

Some relevant works are omitted in the citations. As mentioned above, VISAGE [1] presents a similar learning objective and also incorporates additional information to enhance tracking quality by resolving ambiguities.

**2. Unconvincing Ablation Studies**

The ablation studies are not well-designed. Since the settings for the ablation experiments differ from those in the main experiments, it is difficult to assess the impact of each component on the final performance.

**3. Redundant Notations and Reduced Readability**

The paper contains many equations and redundant notations, which detract from its readability.


[1] Kim, Hanjung, et al. "VISAGE: Video Instance Segmentation with Appearance-Guided Enhancement." European Conference on Computer Vision. Springer, Cham, 2025.

Meta-review: This paper proposes a video object segmentation framework by leveraging contextual information for each instance. Context-aware features are extracted and used for matching. Prototypical cross-frame contrastive loss is introduced to reduce computational cost and maintain pixel consistency. The experimental results on YTVIS and OVIS datasets are given.

The main strengths are 1) good performance and 2) well presentation.

The main weaknesses are 1) lack of novelty (the proposed method is similar to previous methods, like VISAGE), and 2) the contribution of PCC loss is not substantial.  

After the author-reviewer discussion, the main issue of lack of novelty still remains ( recognized by Reviewers Fndi and rJtA). As this issue significantly degrades the contribution of this paper, the AC does not recommend it be accepted at this conference.
The authors are encouraged to consider the comments for their future submission.

Decision: Reject