Title: SOC: Semantic-Assisted  Object Cluster for Referring Video Object Segmentation
Venue: NeurIPS 2023
Accepted: yes
Keywords: Referring Video Object Segmentation

Video-Level Multi-Modal Understanding

Object Cluster

Visual-Linguistic Contrastive Learning
Abstract: This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. Recent approaches model the RVOS task as a sequence prediction problem and perform multi-modal interaction as well as segmentation for each frame separately. However, the lack of a global view of video content leads to difficulties in effectively utilizing inter-frame relationships and understanding textual descriptions of object temporal variations. To address this issue, we propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. By associating a group of frame-level object embeddings with language tokens, SOC facilitates joint space learning across modalities and time steps. Moreover, we present multi-modal contrastive supervision to help construct well-aligned joint space at the video level. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin. Besides, the emphasis on temporal coherence enhances the segmentation stability and adaptability of our method in processing text expressions with temporal variations. Code is available at https://github.com/RobertLuo1/NeurIPS2023_SOC.

SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation

Abstract

This paper studies referring video object segmentation (RVOS) by boosting videolevel visual-linguistic alignment. Recent approaches model the RVOS task as a sequence prediction problem and perform multi-modal interaction as well as segmentation for each frame separately. However, the lack of a global view of video content leads to difficulties in effectively utilizing inter-frame relationships and understanding textual descriptions of object temporal variations. To address this issue, we propose S emantic-assisted O bject C luster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. By associating a group of frame-level object embeddings with language tokens, SOC facilitates joint space learning across modalities and time steps. Moreover, we present multi-modal contrastive supervision to help construct wellaligned joint space at the video level. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin. Besides, the emphasis on temporal coherence enhances the segmentation stability and adaptability of our method in processing text expressions with temporal variations. Code is available at https://github.com/RobertLuo1/NeurIPS2023_SOC.

1 Introduction

Referring Video Object Segmentation (RVOS) [1, 42] aims to segment the target object referred by the given text description in a video. Unlike conventional single-modal segmentation tasks according to pre-defined categories [31, 32] or visual guidance [24-26, 10], referring segmentation requires comprehensive understanding of the content across different modalities to identify and segment the target object accurately. Compared to referring image segmentation, RVOS is even more challenging since the algorithms must also model the temporal relationships of different objects and locations. This emerging topic has attracted great attention and has many potential applications, such as video editing and human-robot interaction.

Due to the varieties of video content as well as the unrestricted language expression, the critical problem of RVOS lies in how to perform pixel-level alignment between different modalities and time steps. To accomplish this challenging task, previous methods have tried various alignment workflows.

Aggregation

'a dog is waiting to catch

the ball shown to him'

Transformer

…

(a) Existing Frame-Based Paradigm

Video-Level

…

Video-Level Object Embedding

Transformer

Transformer

…

…

'a dog is waiting to catch

the ball shown to him'

…

Input Video

(b) Our Video-Centric Paradigm

Visual-Linguistic

Contrastive Learning

…

Frame Content

Aggregation

Frame Content

Aggregation

…

Early approaches [5, 9, 14, 20, 28, 44, 48] take the bottom-up or top-down paradigms to segment each frame separately, while recent works [1, 42] propose to unify cross-modal interaction with pixel-level understanding into transformer structure. Although the above-mentioned approaches have facilitated the alignment between different modalities and achieved excellent performance, they model the RVOS task as a sequence prediction problem and pay little attention to the temporal relationships between different frames. Specifically, they perform cross-modal interaction and segmentation for each frame individually, as illustrated in Fig. 1 (a). The exploitation of temporal guidance relies on the spatial-temporal backbone and manually designed hard assignment strategies [1, 42]. Such paradigms convert the referring video segmentation into stacks of referring image segmentation. While it may be acceptable for such conversion to handle the descriptions of static properties such as the appearance and color of the objects, this approach may lose the perception of target objects for language descriptions expressing temporal variations of objects due to the lack of video-level multi-modal understanding.

To alleviate the above problems and align video with text effectively, we propose Semantic-assisted Object Cluster (SOC) to perform object aggregation and promote visual-linguistic alignment at the video level, as depicted in Fig. 1 (b). Specifically, we design a Semantic Integration Module (SIM) to efficiently aggregate intra-frame and inter-frame information. With a global view of the video content, SIM can facilitate the understanding of temporal variations as well as alignment across different modalities and granularity. Furthermore, we introduce visual-linguistic contrastive learning to provide semantic supervision and guide the establishment of video-level multi-modal joint space. In addition to the remarkable improvements in generic scenarios, these efforts also allow our method to effectively handle text descriptions expressing temporal variations.

We conduct experiments on popular RVOS benchmarks, i.e. , Ref-YouTube-VOS [36], RefDAVIS [16], A2D-Sentences and JHMDB-Sentences [8], to validate the effectiveness of our method. Results show that SOC notably outperforms existing methods for all benchmarks with faster inference speed. In addition, we provide detailed ablations and analysis on components of our method.

Overall, our contributions are summarized as follows:

We present a framework called SOC for RVOS to unify temporal modeling and cross-modal alignment. In SOC, a Semantic Integration Module (SIM) is designed to efficiently aggregate inter and intra-frame information, which achieves video-level multi-modal understanding.

We introduce a visual-linguistic contrastive loss to apply semantic supervision on video-level object representations, resulting in well-aligned multi-modal joint space.

Without bells and whistles, our method outperforms existing state-of-the-art method ReferFormer [42] by a remarkable margin, e.g. , +3.0% J & F on Ref-YouTube-VOS and +3.8% J & F on Ref-DAVIS under fair comparison. Besides, our method runs at 32.3 FPS on single 3090 GPU, which is significantly faster than the 21.4 FPS of ReferFormer.

2 Related Work

Referring Image Segmentation Referring Image Segmentation (RIS) aims to localize the corresponding object referred by a text description within a static image. It is first introduced by Hu et

T Frames

A woman is

watching an

elephant

Stack

FPN

Box

Head

Mask

Head

Class

Head

c

*

Matching

＋

Element-wise Sum

c

Concatenation

*

Dynamic Convolution

Linguistic

MMF

Visual

Transformer

Encoder

Transformer

Encoder

Transformer

Encoder

Frame 1

Transformer

Decoder

Transformer

Decoder

Transformer

Decoder

Video-level Object Cluster

Object

Object

Object

Video level Objects

＋

Frame 2

Frame T

Semantic Integration Module

Backbone

Instance Matching

Visual-Linguistic

Contrastive Loss

×

1

1

1

×

Multiplication

…

…

…

…

Video

Encoder

Text

Encoder

Output

Input

Frame Content Aggregation

al. [12], who developes a simple framework that utilizes Fully Convolution Network (FCN) [37] to generate segmentation masks from concatenated visual and linguistic features. To deeply explore the intrinsic correlations among different modal features, several studies [3, 13, 41, 46] design various attention modules for modality interaction. Additionally, VLT [4] proposes a transformer-based architecture for the RIS task, which has gained more popularity than FCN-based approaches. LAVT [45] incorporates early alignment of visual and linguistic features at the intermediate layers of encoders. PolyFormer [21] further uses transformer to generate polygon vertices as the prior information to refine segmentation masks, leading to better results.

Referring Video Object Segmentation Compared to RIS, RVOS is more challenging since both the action and appearance of the referred object are required to be segmented in a dynamic video. Gavrilyuk et al. [8] first proposes the Referring Video Object Segmentation (RVOS) task. URVOS [36] introduces a large-scale RVOS benchmark and a unified framework that leverages attention mechanisms and mask propagation to increase the task's complexity and scope. ACAN [40] designs an asymmetric cross-guided attention network to establish complex visual-linguistic relationships. To improve positional relation representations in the text, PRPE [35] explores a positional encoding mechanism based on the polar coordinate system. In addition, most previous approaches [19, 22, 39, 47, 48, 30] rely on complicated pipelines. To simplify the workflow, MTTR [1] and ReferFormer [42] adopt query-based end-to-end frameworks for decoding objects from multi-modal features, achieving excellent performance. However, during the decoding phase, previous methods only concentrate on intra-frame object information, disregarding the valuable temporal context of objects across frames. To address this issue, we propose to associate object temporal context with language tokens and achieve video-level multi-modal understanding.

3 Method

Given T frames of video clip I = { I t } T t =1 , where I t ∈ R 3 × H 0 × W 0 and a referring text expression E = { e i } L i =1 , where e i denotes the i-th word in the text. Our goal is to generate a series of binary

segmentation masks S = { s t } T t =1 , s t ∈ R 1 × H 0 × W 0 of the referred object. To this end, we propose a video-centric framework called Semantic-assisted Object Cluster (SOC). We will elaborate on it in the following sections.

3.1 Visual and Linguistic Encoding

Visual Encoder Taking a video clip I as input, we utilize a spatial-temporal backbone such as Video Swin Transformer [27] to extract hierarchical vision features. Consequently, the video clip is encoded into a set of feature maps F v i ∈ R C i × H i × W i , i ∈ { 1 , 2 , 3 , 4 } . Here H i and W i denote the height and width of each scale feature map, respectively. C denotes the channel dimension.

Language Encoder Simultaneously, a transformer-based [38] language encoder encodes the given textual expression E to a word-level embedding F w ∈ R L × C t and a sentence-level embedding F s ∈ R 1 × C t . The word embedding F w contains fine-grained description information. On the other hand, the sentence embedding F s expresses the general characteristics of the referred target object.

3.2 Two Stream Multi-Modal Fusion

Having the separate visual and linguistic embedding encoded from the video clip and text expression, we design a Multi-Modal Fusion module called MMF to perform preliminary cross-modal alignment. As shown in Fig. 3, MMF is a two-stream structure. The language-to-vision (L2V) stream aims to highlight the corresponding regions of the referred object in each frame and mitigate the effect of background noise. It leverages linguistic information as guidance and addresses the potential similar visual areas. Meanwhile, a vision-to-language (V2L) stream is designed to update the textual embedding with image content, which helps to relieve the potential ambiguity of unconstrained descriptions. Specifically, we measure the relevance of all visual areas to the text query and assign weights to the useful information extracted from the visual features so as to reorganize the text embedding. The above L2V and V2L fusion process are based on the multi-head cross-attention mechanism, which can be formulated as:

where MHA( · ) stands for multi-head attention. W,W Q j , W j K , W V j are learnable weights used to map the input to the attention space.

Since visual features of different scales contain diverse content information, MMF is designed to produce a series of coarse-to-fine visual and textual feature maps {F vf i } and {F ef i } , i ∈ { 2 , 3 , 4 } . Specifically, we leverage shared parameters to perform multi-head cross-attention operations on {F v i } , i ∈ { 2 , 3 , 4 } and F w . Take F v 2 as an example. Firstly, we utilize 1 × 1 convolution and fully connected layers to transform the visual and linguistic embeddings into joint space, respectively. Then the bidirectional multi-modal fusion is applied to align information from different modalities as well as enhance the feature representation. The fusion process is:

where F v 2 ∈ R TH 2 W 2 × D and F w ∈ R L × D are embeddings projected by convolution and fully connected layers. Similar to Eq. (2) and Eq. (3), the coarse-to-fine aligned visual features {F vf i } and textual embeddings {F ef i } , i ∈ { 2 , 3 , 4 } are produced.

3.3 Semantic Integration Module

After aligning cross-modal information and activating the potential target region in MMF, we design a Semantic Integration Module (SIM) to incorporate visual content and generate compact target representations. Specifically, we first leverage frame-level content aggregation to locate objects separately for each frame. Then we associate inter-frame dependency and model the temporal relationship of objects via video-level object cluster.

Multi-Modal Fusion Module

Frame 1

Frame 2

Frame T

…

×

Video Embedding

Word

Embedding

Attention

weights

×

×

×

×

×

Linear

Linear

Linear

𝑸

𝑲

V

Attention

weights

Video Embedding

Word

Embedding

Language to Vision

Vision to Language

…

Linear

Linear

＋

Linguistic

Cross Attention

Self Attention

FFN

Add & Norm

Frame 1

Frame 2

Frame T

Linear

Video-Level Object Cluster Module

×

Word

Embedding

Linear

Linear

Linear

…

…

…

𝐻𝑊×𝐶

𝑇𝐻𝑊×𝐶

𝐿×𝐶

𝐿×𝐶

𝑸

𝑲

𝑽

𝑸

𝑲

V

𝑇𝐻𝑊×𝐶

𝐿×𝐶

…

…

…

Frame-Level Content Aggregation Having the activated visual features {F vf i } from MMF, we utilize a transformer-based structure to locate objects in each frame. Firstly, K stacks of deformable transformer encoder layer [49] are leveraged to capture intra-frame relationships and further excavate multi-modal interactions inside {F vf i } . This process can be formulated as:

where f vf t denotes the activated visual features of the t-th frame. Then, a set of learnable object queries [2, 49, 34] is introduced to aggregate image content and highlight potential target objects. Following [42], these object queries fully interact with F ′ vf i in deformable transformer decoder through cross-attention mechanism. After extracting different object representations, these object queries are turned into instance embeddings O f ∈ R T × N q × D . Note that we set up N q object queries to represent instances of each frame in a video clip so there are N q T output object queries in total.

Video-Level Object Cluster Instances typically vary in pose and location between frames and even being obscured. Previous methods [1, 5, 6, 42] only model instances separately for each frame, disregarding the temporal relationship and continuous motion of instances. Although such an approach can cover simple scenarios such as descriptions of target appearance, the lack of inter-frame interaction makes existing methods inefficient for the description of temporal relationships. To address the aforementioned weakness, inspired by [7, 11, 18], we design a video-level object cluster to capture the temporal information of instances between frames.

As shown in Fig. 3, after instances embedding O f are formulated by the frame-level content aggregation, we flatten it into O f ′ ∈ R TN q × D and employ self-attention mechanism along the temporal axis to introduce inter-frame interaction. Furthermore, we find that only introducing temporal selfattention is inferior. Simply sharing temporal object context may create redundancy due to the similarity of the representation referring to the same object in different frames, which potentially affects the model's precise understanding of objects in the video clip. To this end, we employ an object grouping decoder to perform object clustering and group the same object into video-level compact representations across frames. Specifically, we introduce N v video-level object queries O v ∈ R N v × D and they are initialized by linguistic sentence-level feature F s , which helps to promote the understanding of object descriptions and build the visual-linguistic joint space. By performing interaction between linguistic-aware video queries and condensed instance information of each frame, the object grouping decoder can effectively capture temporal object contexts and group the referred object queries across frames to output the clustered video-level object queries O v . While temporal connections can provide a lot of valuable information, the segmentation of each frame is dependent on the specific image content. Thus, we incorporate the high-level video instance information into each frame to integrate the advantages of both. In detail, as illustrated in Fig. 2, we repeat the video-level object queries T times O ′ v ∈ R T × N v × D and enhance the representation of frame-level object queries

by element-wise sum: O f = O f + O ′ v . In this way, the semantic of frame-level object queries is greatly enriched by the supplement of video-level object information.

3.4 Visual-Linguistic Contrastive Learning

Although the aggregated video-level representation can better describe object states, the simple exploitation of textual priors may lead to non-target potential responses for video-level embeddings [29]. Meanwhile, taking unrestricted expressions as aggregation guidance may introduce undesirable inductive bias. To address the issues, we present a visual-linguistic contrastive loss that explicitly focuses on the alignment between video-level features and textual information. The loss accompanies two purposes: (1) Bridge the semantic gap between textual expressions and the corresponding object queries in the multi-modal joint embedding space. (2) Mitigate the bias caused by unrestricted textual features and emphasize the referred object representations. As shown in Fig. 2, we transform the last stage of fused textual features F ef 4 from the vision-to-language stream (see Eq. (3)) to generate the textual guidance embedding F gud :

Then, we measure the similarity between the textual guidance embedding and video-level object queries by scaled dot product similarity: ˆ y sim = O v F ⊤ gud √ D . To suppress undesired region response and highlight target object, we take softmax operation along the object query axis. Finally, we compute the contrastive loss L con by matrix multiplication of ˆ y sim and y τ , where y τ is annotated to 1 for the best predicted trajectory and 0 for others [43]:

3.5 Instance Segmentation and Loss

Prediction Heads As depicted in Fig. 2, there are three lightweight heads built on top of the semantic integration module. The classification head is a concatenation of three fully connected layers. It directly predicts class probability ˆ p ∈ R T × N q × ( K +1) for frame-level object queries, where K is the number of classes. Note that if K = 0 , the role of the classification head is to judge whether the object is referred by the text description. The box head comprises three sequential linear layers, which is designed to transform the object queries to normalized bounding box information ˆ b ∈ R T × N q × 4 , i.e. , center coordinates, width, and height. Similar to [42], we adopt dynamic convolution to output segmentation masks for each frame. Specifically, the mask head produces weights of N q dynamic kernels Ω = { ω n } N q n =1 . Meanwhile, the encoded multi-scale cross-modal features {F ′ vf i } are stacked with the 4 × features from the visual backbone to form hierarchical decoding features. An FPN structure takes the hierarchical features as input and outputs highresolution semantic-aware features F seg ∈ R T × H 0 4 × W 0 4 × D . Finally, the dynamic kernels process the features F seg based on the position information output from the box head and obtain the segmentation mask ˆ m ∈ R T × N q × H 0 4 × W 0 4 , which can be formulated as:

Instance Matching As described above, the prediction heads output the prediction trajectories of N q objects, denoted as ˆ y = { ˆ y n } and the n-th object trajectory prediction is:

There is only one referred object in a video clip corresponding to the text descriptions. Therefore, we denote the ground truth object sequence as y = { p t , b t , m t } T t and search the best prediction trajectory ˆ y σ via Hungarian algorithm [17].

Total Loss We supervise the trajectory prediction ˆ y σ by four types of losses: (1) L mask ( y, ˆ y σ ) , the mask loss is a combination of Dice loss and binary focal loss, which is computed across frames. (2) L box ( y, ˆ y σ ) : the box loss aggregates the L1 loss and GIoU loss per-frame. (3) L cls ( y, ˆ y σ ) : the class

loss is focal loss and supervises the predicted object category. (4) L con ( y τ , ˆ y sim ) : visual-linguistic contrastive loss. The total loss can be formulated as:

where λ is the scale factor to balance each loss.

4 Experiment

4.1 Datasets and Metrics

Datasets. We evaluate our model on four prevalent RVOS benchmarks: Ref-YouTube-VOS [36], Ref-DAVIS17 [16], A2D-Sentences, and JHMDB-Sentences [8]. For detailed descriptions of the datasets please see the supplementary material.

Metrics. Following [1, 42], we measure the effectiveness of our model by criteria of Precision@K, Overall IoU, MeanIoU and MAP over 0.50:0.05:0.95 for A2D-Sentences and JHMDB-Sentences. Meanwhile, we adopt standard evaluation metrics: region similarity( J ), contour accuracy ( F ) and their average value ( J & F ) on Ref-YouTube-VOS and Ref-DAVIS17.

4.2 Implementation Details

We take the pretrained Video Swin Transformer [27] and RoBERTa [23] as our encoder in default. Both the frame aggregation and object cluster parts of SIM consist of three encoder and decoder layers. The number of frame-level queries O f and video-level queries O v are set as 20 in default. We feed the model windows of w = 8 frames during training. The models are trained with eight 32GB V100 GPUs in default. The coefficients for losses are set as λ cls = 2 , λ L 1 = 2 , λ giou = 2 , λ dice = 2 , λ focal = 5 , λ con = 1 . Due to space limitations, please see the supplementary materials for more training details.

4.3 Main Results

Ref-YouTube-VOS & Ref-DAVIS17. We compare our method to previous models on Ref-YouTubeVOS and Ref-DAVIS17 in Table 1. With video-level multi-modal understanding, our SOC achieves new state-of-the-art performance among different training settings: train from scratch, with image pretrain, and joint train. Without bells and whistles, our approach outperforms existing SOTA by about 3% J & F under fair comparison. On Ref-DAVIS17, we directly report the results using the model trained on Ref-YouTube-VOS without finetune.

VOC VL

✓

✓

J

&

F

55.6

58.0

57.1

J

54.3

56.5

55.8

F

56.9

59.6

58.4

✓

✓

60.5

59.2

57.8

&

J

F

J

10

15

20

25

F

59.5

59.6

60.5

58.0

58.1

59.2

56.6

56.7

57.8

58.5

59.8

57.2

&

J

F

J

3

5

8

10

F

59.3

59.7

60.5

57.9

58.4

59.2

56.5

57.1

57.8

59.1

60.3

57.8

VOC Method

K-Means

1D-Conv

Average

Ours

F

34.7

56.4

58.9

J

&

F

33.8

55.1

57.5

J

32.9

53.8

56.1

59.2

60.5

57.8

VOC Structure

&

J

F

J

None only encoder

only decoder

57.1

58.3

58.1

55.8

56.9

56.7

F

58.4

59.8

59.5

Both

59.2

57.8

60.5

Table 8: The structure design of the VOC module.

A2D-Sentences & JHMDB-Sentences. As shown in Table 2, compared with the existing SOTA method ReferFormer [42], our SOC achieves 50.4% mAP and 66.9% mIoU with the model trained from scratch, which gains a clear improvement of 1.8% mAP and 2.8% mIoU respectively. Furthermore, with image data pretraining, our method achieves new state-of-the-art on all metrics, e.g. , +4.0% P@0.9, +2.2% mIoU, and +2.3% mAP. Due to the similarity of the datasets and space limitations, please see the supplementary material for the comparison on JHMDB-Sentences [8]. Our SOC also surpasses all existing methods on it.

4.4 Ablation Studies

In this section, we conduct analysis experiments on the Ref-YouTube-VOS [36] benchmark using Video-Swin-Tiny as the visual backbone and train the model from scratch.

Component Analysis We experiment with the effectiveness of the proposed Video-level Object Cluster (VOC) and Visual-linguistic contrastive Learning (VL). As shown in Table 3, both of them bring performance improvement and their combination works better (+ 3.6% J & F ), which shows the significance of video-level multi-modal understanding.

Fusion Strategy We investigate the effects of various fusion strategies. As illustrated in Table 4, the absence of Language to Vision fusion (L2V) yields significant deterioration. This is because the frame-level content aggregation without textual guidance would introduce background noise. If only utilizing the L2V fusion, the performance drops 0.5% J & F , demonstrating that visual content helps filter out irrelevant words from unconstrained descriptions.

VOC Method The superiority of the current VOC module derives from two aspects: i) Powerful representation learning ability of the transformer structure. Compared to other clustering methods

the giraffe is on the right side of the railing and

has its head hanging over and eating grass

a person skateboarding

a skateboard is being rode by a person

wearing a red hat and jumping in the air

a black knife holder

a person is jumping over the river

a tennis player is running from right to left

a tennis player is running from right to left

a person is jumping over the river

(a)

(b)

shown in Table 5, the transformer architecture has a better perception as well as selection of global information, which helps to associate object features across frames. ii) Sufficient inter-frame interactions. Although other methods can also yield video-level representations, they lack the perception of temporal variations and the alignment of inter-frame information. As a contrast, our VOC performs adequate inter-frame interaction while clustering.

Object Query Number Although only one referred object is involved in a video, focusing on more potential regions is helpful due to the various video content. Table 6 shows that an increased number of object queries allows the model to effectively group the relevant objects from the pool of candidate regions. The performance saturates when the query number is increased to a certain extent. We believe that excessive queries may cause misunderstandings when grouping objects across frames.

Segmentation Stability Benefiting from video-level multimodal understanding, our method yields more robust segmentation results compared to existing approaches. To quantify the temporal segmentation stability, we calculate the IoU and J & F variance between each frame in a video and count the average value on Ref-YouTube-VOS [36] dataset. Results are shown in Fig. 4 ( the lower is better ). It can be seen that with video object cluster module and visual-linguistic contrastive loss building the video-level multi-modal aligned space, our SOCcan understand the overall video content better and output

0.025

0.03

0.035

0.04

SOC(Ours)

Referformer

MTTR

J Variance

J&F Variance

◼

𝑰𝒐𝑼 𝑽𝒂𝒓𝒊𝒂𝒏𝒄𝒆

◼

𝓙&𝓕𝑽𝒂𝒓𝒊𝒂𝒏𝒄𝒆

temporally stable segmentation results. More qualitative analysis of temporal consistency is shown in the supplementary material.

Frame Number Table 7 shows results with different training clip frame numbers. As expected, widening temporal context leads to performance gains yet causes expensive computation cost. For a fair comparison with existing methods, we take the frame number of 8 by default.

Variants of VOC Structure Results in Table 8 illustrate the rationality of the structure of the VOC module. Without sufficient inter-frame interaction by the encoder or aggregation of target object information by the decoder, the segmentation performance will be degraded.

Inference Speed In addition to superior segmentation performance, our method also achieves real-time inference. Specifically, our SOC runs at 32.3 FPS on single 3090 GPU, which significantly exceeds the existing SOTA method ReferFormer [42] (21.4 FPS).

a horse running to the left of another horse towards the water

(a)

(b)

(a)

(b)

the person on the bike, with a yellow hoodie and black pants, to the right

4.5 Qualitative Results

Fig. 5 demonstrates the effectiveness of SOC for complex scenarios segmentation, i.e. , similar appearance, occlusion, and large variations. To further prove the adaptability of our SOC for understanding text expressions about temporal action and variations, we design corresponding text descriptions that depict changed states of the object in temporal. As shown in Fig. 6, the global view enables SOC to understand such text and segment the target object accurately. In contrast, ReferFormer [42] only recognizes the object in specific frames mentioned in the text descriptions and fails to comprehend the content. More comparisons can be seen in the supplementary material. Fig. 7 indicates that due to the temporal modeling and inter-frame interaction in VOC, the full model comprehends text descriptions of temporal variations and tracks the target in coherence. In contrast, without VOC, the model struggles to understand the temporal relationship and fails to segment the object.

5 Conclusion

In this paper, we propose a framework called SOC for RVOS to achieve video-level multi-modal understanding. By associating frame-level object embeddings with language tokens, we unify temporal modeling and cross-modal alignment into a simple architecture. Moreover, visual-linguistic contrastive learning is introduced to build the video-level multi-modal space. Extensive experiments show that our SOC remarkably outperforms existing state-of-the-art methods. Besides, video-level understanding also allows our SOC to handle text descriptions expressing temporal variations better.

Limitations. The proposed framework, SOC, has achieved video-level multi-modal understanding and excellent performance. However, there are some potential limitations, e.g. , the current architecture cannot handle infinitely long videos. We think that devising explicit designs for long videos with complex scenarios is an interesting future direction.

References

Botach, A., Zheltonozhskii, E., Baskin, C.: End-to-end referring video object segmentation with multimodal transformers. In: CVPR. pp. 4975-4985 (2022) 1, 2, 3, 5, 7, 8

Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV. pp. 213-229 (2020) 5

Ding, H., Jiang, X., Shuai, B., Liu, A.Q., Wang, G.: Semantic correlation promoted shapevariant context for segmentation. In: CVPR. pp. 8885-8894 (2019) 3

Ding, H., Liu, C., Wang, S., Jiang, X.: Vision-language transformer and query generation for referring segmentation. In: ICCV. pp. 16301-16310 (2021) 3

Ding, H., Liu, C., Wang, S., Jiang, X.: Vlt: Vision-language transformer and query generation for referring segmentation. TPAMI (2022) 2, 5, 7

Ding, Z., Hui, T., Huang, J., Wei, X., Han, J., Liu, S.: Language-bridged spatial-temporal interaction for referring video object segmentation. In: CVPR. pp. 4964-4973 (2022) 5, 7

Duke, B., Ahmed, A., Wolf, C., Aarabi, P., Taylor, G.W.: Sstvos: Sparse spatiotemporal transformers for video object segmentation. In: CVPR. pp. 5912-5921 (2021) 5

Gavrilyuk, K., Ghodrati, A., Li, Z., Snoek, C.G.M.: Actor and action video segmentation from a sentence. In: CVPR. pp. 5958-5966 (2018) 2, 3, 7, 8

Han, K., Liu, Y., Liew, J.H., Ding, H., Wei, Y., Liu, J., Wang, Y., Tang, Y., Yang, Y., Feng, J., et al.: Global knowledge calibration for fast open-vocabulary segmentation. arXiv preprint arXiv:2303.09181 (2023) 2

He, C., Li, K., Zhang, Y., Xu, G., Tang, L., Zhang, Y., Guo, Z., Li, X.: Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. arXiv preprint arXiv:2305.11003 (2023) 1

Heo, M., Hwang, S., Oh, S.W., Lee, J., Kim, S.J.: VITA: video instance segmentation via object token association. arXiv preprint arXiv:2206.04403 (2022) 5

Hu, R., Rohrbach, M., Darrell, T.: Segmentation from natural language expressions. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV. pp. 108-124 (2016) 3, 8

Hu, Z., Feng, G., Sun, J., Zhang, L., Lu, H.: Bi-directional relationship inferring network for referring image segmentation. In: CVPR. pp. 4423-4432 (2020) 3

Huang, S., Hui, T., Liu, S., Li, G., Wei, Y ., Han, J., Liu, L., Li, B.: Referring image segmentation via cross-modal progressive comprehension. In: CVPR. pp. 10488-10497 (2020) 2

Kazemzadeh, S., Ordonez, V., Matten, M., Berg, T.: Referitgame: Referring to objects in photographs of natural scenes. In: EMNLP. pp. 787-798 (2014) 7

Khoreva, A., Rohrbach, A., Schiele, B.: Video object segmentation with language referring expressions. In: ACCV. pp. 123-141 (2018) 2, 7

Kuhn, H.W.: The hungarian method for the assignment problem. In: Naval research logistics quarterly (1955) 6

Li, X., Zhang, W., Pang, J., Chen, K., Cheng, G., Tong, Y., Loy, C.C.: Video k-net: A simple, strong, and unified baseline for video segmentation. In: CVPR. pp. 18847-18857 (2022) 5

Liang, C., Wu, Y., Luo, Y., Yang, Y.: Clawcranenet: Leveraging object-level relation for text-based video segmentation. arXiv preprint arXiv:2103.10702 (2021) 3, 8

Liang, C., Wu, Y., Zhou, T., Wang, W., Yang, Z., Wei, Y., Yang, Y.: Rethinking cross-modal interaction from a top-down perspective for referring video object segmentation. arXiv preprint arXiv:2106.01061 (2021) 2

Liu, J., Ding, H., Cai, Z., Zhang, Y., Satzoda, R.K., Mahadevan, V., Manmatha, R.: Polyformer: Referring image segmentation as sequential polygon generation. arXiv preprint arXiv:2302.07387 (2023) 3

Liu, S., Hui, T., Huang, S., Wei, Y ., Li, B., Li, G.: Cross-modal progressive comprehension for referring segmentation. TPAMI pp. 4761-4775 (2022) 3, 8

Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., Stoyanov, V.: Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692 (2019) 7

Liu, Y., Yu, R., Wang, J., Zhao, X., Wang, Y., Tang, Y., Yang, Y.: Global spectral filter memory network for video object segmentation. In: ECCV. pp. 648-665 (2022) 1

Liu, Y., Yu, R., Yin, F., Zhao, X., Zhao, W., Xia, W., Yang, Y .: Learning quality-aware dynamic memory for video object segmentation. In: ECCV. pp. 468-486 (2022)

Liu, Y., Yu, R., Zhao, X., Yang, Y.: Quality-aware and selective prior enhancement memory network for video object segmentation. In: CVPR Workshop. vol. 2 (2021) 1

Liu, Z., Ning, J., Cao, Y ., Wei, Y ., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. In: CVPR. pp. 3192-3201 (2022) 4, 7

Luo, G., Zhou, Y., Sun, X., Cao, L., Wu, C., Deng, C., Ji, R.: Multi-task collaborative network for joint referring expression comprehension and segmentation. In: CVPR. pp. 10034-10043 (2020) 2

Ma, Y., Wang, Y., Wu, Y., Lyu, Z., Chen, S., Li, X., Qiao, Y.: Visual knowledge graph for human action reasoning in videos. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 4132-4141 (2022) 6

McIntosh, B., Duarte, K., Rawat, Y.S., Shah, M.: Visual-textual capsule routing for text-based video segmentation. In: CVPR. pp. 9942-9951 (2020) 3

Miao, J., Wang, X., Wu, Y., Li, W., Zhang, X., Wei, Y., Yang, Y.: Large-scale video panoptic segmentation in the wild: A benchmark. In: CVPR. pp. 21033-21043 (2022) 1

Miao, J., Wei, Y., Wu, Y., Liang, C., Li, G., Yang, Y.: Vspw: A large-scale dataset for video scene parsing in the wild. In: CVPR. pp. 4133-4143 (2021) 1

Nagaraja, V.K., Morariu, V.I., Davis, L.S.: Modeling context between objects for referring expression understanding. In: ECCV. pp. 792-807 (2016) 7

Ni, X., Wen, H., Liu, Y., Ji, Y ., Yang, Y .: Multimodal prototype-enhanced network for few-shot action recognition. arXiv preprint arXiv:2212.04873 (2022) 5

Ning, K., Xie, L., Wu, F., Tian, Q.: Polar relative positional encoding for video-language segmentation. In: IJCAI. p. 10 (2020) 3

Seo, S., Lee, J., Han, B.: URVOS: unified referring video object segmentation network with a large-scale benchmark. In: ECCV. pp. 208-223 (2020) 2, 3, 7, 8, 9

Shelhamer, E., Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. TPAMI pp. 640-651 (2017) 3

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. NeurIPS 30 (2017) 4

Wang, H., Deng, C., Ma, F., Yang, Y.: Context modulated dynamic networks for actor and action video segmentation with language queries. In: AAAI. pp. 12152-12159 (2020) 3

Wang, H., Deng, C., Yan, J., Tao, D.: Asymmetric cross-guided attention network for actor and action video segmentation from natural language query. In: ICCV. pp. 3938-3947 (2019) 3, 8

Wang, Z., Lu, Y., Li, Q., Tao, X., Guo, Y., Gong, M., Liu, T.: Cris: Clip-driven referring image segmentation. In: CVPR. pp. 11686-11695 (2022) 3

Wu, J., Jiang, Y., Sun, P., Yuan, Z., Luo, P.: Language as queries for referring video object segmentation. In: CVPR. pp. 4964-4974 (2022) 1, 2, 3, 5, 6, 7, 8, 9, 10

Xiao, Y., Ma, Y., Li, S., Zhou, H., Liao, R., Li, X.: Semanticac: Semantics-assisted framework for audio classification. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5 (2023) 6

Xu, M., Zhang, Z., Wei, F., Lin, Y., Cao, Y., Hu, H., Bai, X.: A simple baseline for openvocabulary semantic segmentation with pre-trained vision-language model. In: ECCV. pp. 736-753 (2022) 2

Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., Torr, P.H.S.: Lavt: Language-aware vision transformer for referring image segmentation. In: CVPR. pp. 18134-18144 (2022) 3

Ye, L., Rochan, M., Liu, Z., Wang, Y.: Cross-modal self-attention network for referring image segmentation. In: CVPR. pp. 10502-10511 (2019) 3

Ye, L., Rochan, M., Liu, Z., Zhang, X., Wang, Y.: Referring segmentation in images and videos with cross-modal self-attention network. TPAMI pp. 3719-3732 (2022) 3, 8

Zhang, Y., Yuan, L., Guo, Y., He, Z., Huang, I., Lee, H.: Discriminative bimodal networks for visual localization and detection with natural language queries. In: CVPR. pp. 1090-1099 (2017) 2, 3

Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: deformable transformers for end-to-end object detection. In: ICLR (2021) 5

Peer reviews:
comments: SUMMARY:
This paper proposes a novel framework that can effectively exploit video-level visual-linguistic alignment to improve the performance of RVOS, filling a gap in existing methods. The extensive experiments on three datasets demonstrate  the effectiveness of SOC.

STRENGTHS:
The proposed method outperforms existing state-of-the-art methods on all datasets.
The paper provides a clear and concise description of the SOC framework and its experimental results.

WEAKNESSES:
None

QUESTIONS:
How to handle objects that have been occluded for a short time (> training frames number)

LIMITATIONS:
The authors have addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

comments: SUMMARY:
This work studies the referring video object segmentation problem and proposes the semantic-assisted object cluster approach. It aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment.

STRENGTHS:
1) It performs object aggregation and promotes visual-linguistic alignment at the video level, which considers the temporal variations and alignment across different modalities of frames. 

2) It introduces the visual-linguistic contrastive learning to provide semantic supervision.

3) Empirical studies on several benchmarks demonstrate the advantage of the proposed method.

WEAKNESSES:
1) The novelty seems a bit limited. The multi-modal fusion model just uses the common multi-head cross-attention to model the relations of language and visual information. 

2) Is there other clustering methods for object grouping? Some discussions are requried. 

3) Ablations studies on the hyper-parameters $\lambda$ are missing.

QUESTIONS:
What about considering the object-level temporal relations across frames compared to the video-level ones?

LIMITATIONS:
See the above.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis research paper focuses on referring video object segmentation. It combines temporal frame features and language features using a cross attention design. Additionally, it extracts object features from each frame and applies contrastive learning with language representation. The paper presents various designs to enhance the referring performance. Results from experiments on multiple benchmarks demonstrate that this method achieves state-of-the-art performance on this task.\n\nSTRENGTHS:\n1. The RVOS framework demonstrates remarkable results on various benchmarks.\n2. Through experimentation, the proposed fusion of features significantly enhances the baseline.\n3. The concept of utilizing temporal and language features is effectively communicated.\n\nWEAKNESSES:\n1. The current method is quite complex, and it is not clear what the main contributions of the RVOS in figure are. It appears that most of the modules shown in Figure 1 are not mentioned in the manuscript. This makes it difficult to understand which proposed components are essential for the given task.\n\n2. The Multi-Modal Fusion (MMF) and Semantic Integration Module (SIM) are presented in Section 3.2 and 3.3. However, there are no experimental tests conducted to validate these modules. The authors also claim that SIM contributes to the task in the introduction.\n\n3. The results of visualizing the impact of the proposed modules are missing. It is important to provide results that demonstrate how each module enhances the referring performance.\n\nQUESTIONS:\n1. Figure 6 shows a video with only one person. Why does the compared method fail in these scenes? Are there any complex scenes that demonstrate the advantages of the present method?\n2. The figures lack clarity and are difficult to understand. It is important to highlight the main contributions of the present modules.\n3. The Semantic Integration Module appears to be resource-intensive, and it would be helpful to present the model parameters compared to previous methods.\n4. The use of off-the-shelf designs in all the modules limits the novelty of the approach.\n\nLIMITATIONS:\nThe limitations are only briefly mentioned in the current manuscript.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces the Semantic-assisted Object Cluster model, which is used to solve the problem of visual language alignment between videos and texts. The model uses a semantic integration module and visual language contrastive learning to improve the understanding of video content and has achieved significant improvements in different scenes and text descriptions.\n\nSTRENGTHS:\n1．The Semantic-assisted Object Cluster (SOC) framework has been proposed, which effectively aligns video and text, realizes object aggregation and visual-language alignment, and improves the performance of the RefVOS model. \n2．The Semantic Integration Module (SIM) has been designed to efficiently aggregate intra-frame and inter-frame information, understand video content from a global perspective, and help understand temporal changes and alignment of different modalities and granularities. \n3．Visual-linguistic contrastive learning has been introduced to provide semantic supervision, guide the establishment of video-level multimodal joint space, and improve the performance of the RefVOS model. 4、It can effectively handle text descriptions that express temporal changes and significantly improve general scene processing. \n4.On popular RVOS benchmarks such as Ref-YouTube-VOS, Ref-DAVIS, A2D-Sentences, and JHMDB-Sentences, the performance is significantly better than existing methods, and inference speed is faster.\n\nWEAKNESSES:\n1．The effectiveness of Visual-Linguistic Contrastive Learning module heavily depends on the quality of the text descriptions used for supervision. If the text descriptions are inaccurate or incomplete, the performance of this module could be limited.\n2．The efficiency of SIM is not clearly explained, and it is unclear how well it would scale to larger and more complex datasets.\n3. The caption of many figures (such as Fig.3) are too simple, and more details should be provided.\n4.Ablation experiments were conducted only within each module, without verifying the effectiveness of each module on the whole network framework.\n\nQUESTIONS:\n1．For the Video-Level Object Cluster module, the design of the object clustering module does not fully consider the similarities and differences between instances, which may affect the clustering effect. The paper should provide some considerations on this issue.\n\n2．For the Visual-Linguistic Contrastive Learning module, the issue of potentially non-target responses in video-level embeddings due to the simple use of text priors should not be ignored.\n\n3．The model uses Video-Swin-B and Video-Swin-T as backbones, so it is unfair to compare it with other models that use resnet as backbone.\n\nLIMITATIONS:\n1．The model may rely on the accuracy of semantic information, as it uses the Semantic Integration Module (SIM) to aggregate information in the video.\n\n2．The model may not be able to handle some complex video scenes because it only performs inference based on known semantic information.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper mainly studies referring video object segmentation (RVOS) which is a fundermental problem in vision-and-language applications. Compared with previous works which pay little attention to temporally modeling relationships between different frames in the entire video clip, the authors proposed a framework called SOC for RVOS which conducts temporal modeling and cross-modal alignment in a unified framework. Specifically, SOC contains several modules including textual and vision encoding and multi-modal fusion, semantic integration module and vision-language contrastive learning. The experimental results show SOC outperforms existing SOTA method, e.g. ReferFormer published in CVPR 2022.\n\nSTRENGTHS:\n(1) Overall, the motivation for poposing the temporal modeling intra-frame and inter-frame information for RVOS is quite clear and straightforward. Besides, the paper is well-writen and the idear as well as each module is easy-to-follow.\n\n(2) The most valuable contribution is the video-level object cluster module in the semantic integration module (SIM).  Instead of using traditional clustering method such as k-means, the authors proposed an object grouping decoder architecure to perform the clustering process. Specifically, N_v video-level object queries are introduced to learn the overall object centroids in the entire video. After repeating such operation T times, the semantic representation enhancement is achieved by enriching the video-level object information. Figure 1 in the supplementary material demonstrates that most of these learnable queries focus on the region-of-interests (ROIs)  of he referred objects. \n\n(3) The experimental results are promising. The proposed SOC framework outperforms other existing SOTA works under different settings. The provided video demo shows the referred objects have been segmented perfectly.\n\nWEAKNESSES:\n(1) According to my understanding of this work, the main contributions is the video-level object cluster module. Hence, the albation studies or comparison between different methods should focus on such module. Specifically, is there any other clustering-based method that could obtain video-level object embeddings? Why is the proposed object grouping decoder-style model is the best solution among these optional methods? Besides such high level comparisons, some details (or parameters) of abaltion studies should be further discussed. For example, the number of object queries and number of temporal frames are tuned independently in Table 5 and 6. However, jointly tuning these two parameters should be investigated.  Another example is that the final representation is the element-wis sum of frame-level and video-level semantic embeddings. Are there any other better learning-based fusion implementations?\n\n(2) Generally, the proposed framework follows a two-stream fashion which includes a textual encoder and a vision encoder. Note that the vision encoder could be a temporal convolutional network (3D CNN) or video transformer such as in this work. No matter which implementation, it undergoes the temporal processing. The question is: could the video-level object clustering module be embedded inside such temporal encoding procedure? I think the whole inferece speed could be further decreased if it is possible.    \n\n(3) I think the main contribution and the motivation is clear.  Some trivial parts such as the visual-linguistic contrastive learning could be removed in order to make the key points stand out, which will not reduce the whole work's contribution or value.\n\nQUESTIONS:\n(1) Why the SIM module is not designed inside the video encoder? Please refer to the second part in the above weaknesses. \n\n(2) It is reported that the SOC runs at 32.3 FPS on a single 3090 GPU. I think the overhead of each module should be further decomposed, e.g. the encoding backbone, multi-modal fusion module, and the SIM.\n\n(3) As illustrated in Table 4, L2V outperforms V2L significantly. Deeper insights are expected to be uncovered. Although using both of them achieves the best performance, some missing details should be claimed. For example, what is each branch's weight? \n\n(4) Could the proposed SOC be extended to solve multiple referred objects given a textual query?\n\nLIMITATIONS:\nThe authors have mentioned that malicious use of such model could lead to negative social impacts such as unauthorized video surveillance. However, I believe the positive impacts of these vision-and-language applications would improve the embodied AI research and generally outweigh the risks.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

comments: SUMMARY:
In this paper, the author presents a framework called SOC, and a visual-linguistic contrastive loss for RVOS to unify temporal modeling and cross-modal alignment.
The performance is good according to the leaderboard (https://paperswithcode.com/sota/referring-video-object-segmentation-on-refer)
The extra module seems to have many parameters, and somehow incremental.

STRENGTHS:
In this paper, the author presents a framework called SOC, and a visual-linguistic contrastive loss for RVOS to unify temporal modeling and cross-modal alignment.
The performance is good.

WEAKNESSES:
1. Parameters
The extra module seems to have many parameters, and somehow incremental.  
- Unfair Comparison. It is unfair to compare with other methods. 
- Design of MMF is not new. Many multi-modality fusion has used similar structure such as [a]. The different  part is that authors adopt transformer structure. 
[a] Comprehensive Linguistic-Visual Composition Network for Image Retrieval 

2. Motivation in Figure 1 is not clear to me. 
I think it can be easily solved by  adopting the video-based backbone such as TSM or I3DNet, 

3. The visual-linguistic contrastive loss
I do not see significant difference from the basic contrastive loss in Clip or other works. The input is changing from image to video features. Could you further explain it?

QUESTIONS:
Please see Weakness. I am not working in this field. If the author could address my concerns, I will upgrade my rate.

LIMITATIONS:
Please see Weakness. I am not working in this field. If the author could address my concerns, I will upgrade my rate.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-24