SOC: Semantic-Assisted  Object Cluster for Referring Video Object Segmentation
Authors: Zhuoyan Luo
Yicheng Xiao
Yong Liu
Shuyan Li
Yitong Wang
Yansong Tang
Xiu Li
Yujiu Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: Referring Video Object Segmentation
Video-Level Multi-Modal Understanding
Object Cluster
Visual-Linguistic Contrastive Learning

Abstract: This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. Recent approaches model the RVOS task as a sequence prediction problem and perform multi-modal interaction as well as segmentation for each frame separately. However, the lack of a global view of video content leads to difficulties in effectively utilizing inter-frame relationships and understanding textual descriptions of object temporal variations. To address this issue, we propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. By associating a group of frame-level object embeddings with language tokens, SOC facilitates joint space learning across modalities and time steps. Moreover, we present multi-modal contrastive supervision to help construct well-aligned joint space at the video level. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin. Besides, the emphasis on temporal coherence enhances the segmentation stability and adaptability of our method in processing text expressions with temporal variations. Code is available at https://github.com/RobertLuo1/NeurIPS2023_SOC.

SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation
Zhuoyan Luo't, Yicheng Xiao
Yong Liu",,
Shuyan Li,, Yitong Wang?, Yansong Tang', Xiu Li Yujiu Yang't
Tsinghua Shenzhen International Graduate School, Tsinghua University
*ByteDance Inc.
'Engineering Department, University of Cambridge
(luozy23, xiaoyc23, liu-yong20) tssngein edu. s121410cam ac uk waugjjigghhenbbyteddnce. com (tang.
Stang-yansong, yansong, li.xiu, yang
yang.yujiu) yujiu) tsinghua...... edu.
Abstract
This paper studies referring video object segmentation (RVOS) by boosting video- level visual-linguistie alignment. Recent approaches model the RVOS task as sequence prediction problem and perform multi-modal interaction as well as segmentation for each frame separately. However, the lack of a global view of video content leads to difficulties in effectively utilizing inter-frame relationships and understanding textual descriptions of object temporal variations. To address this issue, we propose Semantic-an assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross cross-modal alignment. By associating a group of frame-level object embeddings with language tokens, SOC facilitates joint space learning across modalities and time steps. Moreover, we present multi-modal contrastive supervision to help construct well- aligned joint space the video level. We conduct extensive experiments popular RVOS benchmarks, and our method outperforms state-of-the-ar competitors on all benchmarks by a remarkable margin. Besides, the emphasis temporal coherence enhances the segmentation stability and adaptability of our method in processing text expressions with temporal variations. Code is available SOC.
Introduction
Referring Video Object Segmentation (RVOS) [1, 42] aims to segment the target object referred by the given text description in video. Unlike conventional single-modal segmentation tasks according to pre-defined categories [31, 32] or visual guidance [24-26, 10], referring segmentation requires comprehensive understanding of the content across different modalities identify and segment the target object accurately. Compared to referring image segmentation, RVOS is even more challenging since the algorithms must also model the temporal relationships of different objects and locations. This emerging topic has attracted great attention and has many potential applications, such as video editing and human-robot interaction.
Due to the varicties of video content as well as the unrestricted language expression, the critical problem of RVOS lies in how to perform pixel-level alignment between different modalities and time steps. To accomplish this challenging task, previous methods have tried various alignment workflows.
"Equal contribution. Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

"a dog is wailing catch the hall shown to him"
"a dog waiting to catch the ball shown to him"
Viual Lingusiii Contrssive leening
Transfomer
Transfomer
Videa-Level Object Claster
Cuntent Aggregation
Transformer
Ingut Video
Leput Video
Vidco-Level Object limbedddng
(a) Existing Frame-Based Paradigm
(b) Our Video-Centrie Paradigm
Figure 1: Illustration of different paradigms. Frame-based methods perform cross-modal interaction and segmentation for each frame individually. In contrast, our method unifies temporal modeling and cross-modal alignment to achieve video-level understanding.
Early approaches [5. 9, 14, 20, 28, 44, 48] take the bottom-up or top-down paradigms to segment each frame separately, while recent works [1, 42] propose to unify cross-modal interaction with pixel-level understanding into transformer structure. Although the above-mentioned approaches have facilitated the alignment between different modalities and achieved excellent performance, they model the RVOS task as sequence prediction problem and pay little attention to the temporal relationships between different frames. Specifically, they perform cross-modal interaction and segmentation for each frame individually, as illustrated in Fig. 1 (a). The exploitation of temporal guidance relies on the temporal backbone and manually designed hard assignment strategies [1, 42]. Such paradigms convert the referring video segmentation into stacks of referring image segmentation. While it may be acceptable for such conversion to handle the descriptions of static properties such the appearance and color of the objects, this approach may lose the perception of target objects for language descriptions expressing temporal variations of objects due to the lack video-level multi-modal understanding.
To alleviate the above problems and align video with text effectively, we propose Semantic-assisted Object Cluster (SOC) perform object aggregation and promote visual-linguiste alignment the video level, as depicted in Fig. (b). Specifically, we design a Semantic Integration Module (SIM) to efficiently aggregate intra-frame and inter-frame information. With global view of the video content, SIM can facilitate the understanding of temporal variations as well as alignment across different modalities and granularity. Furthermore, introduce visual-linguistic contrastive learning to provide semantic supervision and guide the establishment of video-level multi-modal joint space. In addition the remarkable improvements in generic scenarios, these efforts also allow our method to effectively handle text descriptions expressing temporal variations.
We conduct experiments popular RVOS benchmarks, i.e., Ref- Yef-YouTubbeVo vOS [36], Ref- DAVIS [16], A2D-Sentences and WHMDD-SSntent [8], to validate the effectiveness our method. Results show that SOC notably outperforms existing methods for all benchmarks with faster inference speed. In addition, we provide detailed ablations and analysis on components of our method.
Overall, our contributions are summarized as follows:
We present framework called SOC for RVOS to unify temporal modeling and cross-modee alignment. In SOC, a Semantic Integration Module (SIM) is designed to efficiently aggregate inter and intra-frame information, which achieves video-level multi-m modil understanding.
We introduce a visual- lingul-ii contrastive loss to apply semantic supervision on video-level object representations, resulting in well-aligned multi-modal joint space.
Without bells and whistles, our method outperforms existing state-of theffthh-att method Refer- Former [42] by remarkable margin, e.g. +3.0% J&F on Ref- YouTube--OOT and +3.8% J&F on Ref-DAVIS under fair comparison. Besides, our method runs at 32.3 FPS on single 3090 GPU, which is significantly faster than the 21.4 FPS of ReferFormer.
2 Related Work
Referring Image Segmentation Referring Image Segmentation (RIS) aims to localize the corre- sponding object referred by text description within a static image. It is first introduced by Hu et

---

Instance Mutching
Output
Visual- Linga--ingg
Contrastive Loss
Concatenation
FPN
Mutching
Dy Pynamic Coavolution
Multiplication
Bux Head
Mask Licad
Class
Input
Vidco-level Object Cluster
Stack
Obbject
Object
Object
Backbone
Trunsfommer
Transfocmer
Thussfmmer Decoder
Video
Tranofurtper Uncoder
Transfonner Encocer
Transformer Enctaler
1 000
T DOO Frume
Visual
Frame
Fimme Frame Content Aggregation
woman watching an elephant
Ensader
Text
Linguistic
Semantic Integration Module
Figure 2: Overview of SOC. The model takes a video clip with corresponding language descriptions as input. After the encoding process, the multi-modal fusion (MMF) module performs bidirectional fusion to build the intrinsic feature relations. Then we design Semantic Integration Module to efficiently aggregate intra-frame and inter-frame information. Meanwhile, we introduce visual- linguistic contrastive loss to benefit the establishment of video-level multi-modal space. Finally, the prediction heads decode the condensed embeddings and output segmentation masks. The transparent arrows illustrates the pipeline of our model and label the input and output.
al. [12], who developes simple framework that utilizes Fully Convolution Network (FCN) [37] to generate segmentation masks from concatenated visual and linguistic features. To deeply explore the intrinsic correlations among different modal features, several studies [3, 13, 41, 46] design various attention modules for modality interaction. Additionally, VLT [4] proposes transformer-basee archi- tecture for the RIS task, which has gained more popularity than incorporates early alignment of visual and linguistic features at the intermediate layers of encoders. PolyFormer [21] further uses transformer to generate polygon vertices as the prior information to refine segmentation masks, leading to better results.
approaches. LAVT [45]
Referring Video Object Segmentation Compared to RIS, RVOS is more challenging since both the action and appearance of the referred object are required be segmented in dynamic video. Gavri- lyuk et al. [8] first proposes the Referring Video Object Segmentation (RVOS) task. URVOS [36] in- troduces large-scale RVOS benchmark and unified framework that leverages attention mechanisms and mask propagation to increase the task's complexity and scope. ACAN [40] designs an asymmet- ric cross-guided attention network to establish complex visual-linguistic relationships. To improve positional relation representations in the text, PRPE [35] explores a positional encoding mechanism based on the polar coordinate system. In addition, most previous approaches [19, 22, 39, 47, 48, 30] rely on complicated pipelines. To simplify the workflow, MTTR [i] and ReferFormer [42] adopt query-base based end-to-end frameworks for decoding objects from multi-mode features, achieving excellent performance. However, during the decoding phase, previous methods only concentrate on intra- intra-frame object information, disregarding the valuable temporal context of objects across frames. To address this issue, we propose to associate object temporal context with language tokens and achieve video-level multi-modal understanding.
3 Method
Given T frames video clip I= (T))_ where J, E R3xHox and a referring text expression E {ex) where e; denotes the i-th word in the text. Our goal is to generate series of binary

---

segmentation masks S = { E NxxWo of the referred object. To this end, we propose video-centrie framework called cemantic-assistie Object Cluster (SOC). We will elaborate on it in the following sections.
3.1 Visual and Linguistic Encoding
Visual Encoder Taking a video clip I as input, utilize spatial-temporal backbone such as Video Swin Transformer [27] to extract hierarchical vision features. Consequently, the video clip is encoded into of feature maps F RR (X4XHH,, H, M,x,, E 21, 3, 4). Here H; and W; denote the height and width of each scale feature map, respectively. C' denotes the channel dimension.
Language Encoder Simultaneously, a transformer-based [38] language encoder encodes the given textual expression & to a word-level embedding F E RLXC, and sentence-level embedding F I The word embedding F contains ine-grained description information. On the other hand, the sentence embedding F expresses the general characteristics of the referred target object.
3.2 Two Stream Multi-Modal Fusion
Having the separate visual and linguistic emhedding encoded from the video clip and text expression, we design a Multi-Modal Fusion module called MMF to perform preliminary cross-modal alignment. As shown in Fig. MMF is two-stream structure. The language-to-vision (I.2V) stream aims to highlight the corresponding regions of the referred object in each frame and mitigate the effect of background noise. It leverages linguistic information as guidance and addresses the potential similar visual areas. Meanwhile, a vision-to-language (V2L) stream is designed to update the textual embedding with image content, which helps to relieve the potential ambiguity of unconstrained descriptions. Specifically, we measure the relevance of all visual areas to the text query and assign weights to the useful information extracted from the visual features so to reorganize the text embedding. The above I.2V and V2I. fusion process are based on the multi-head cross-attention mechanism, which can be formulated as:
MHA (X.)) Concat (head1 (X,)) headn (X,))) W, (xW9) T DW
head, (X,3) softmax
vwy.
VC
where MHA (-) stands for multi-head attention. W. W WiK, wy are learnable weights used to map the input to the attention space.
Since visual features of different scales contain diverse content information, MMF is designed to produce series of coarse-to-fing visual and textual feature maps (F.T² and (F. i E (2, 4). Specifically, we leverage shared parameters to perform multi-head cross-attention operations on (F.). i 32, 4} and F=. Take Fÿ an example. Firstly, utilize 1 x convolution and fully connected layers to transform the visual and linguistic embeddings into joint space, respectively. Then the bidirectional multi-modal fusion is applied to align information from different modalities as well as enhance the feature representation. The fusion process is:
Je MHA F") F3
(2)
Fel MHA (F)
(3)
where Fy E RTH, and F E RLXD are embeddings projected by convolution and fully connected layers. Similar Eq. (2) and Eq. (3), the coarse-to-fine aligned visual features (F!²) } and textual embeddings {F((), ) (2,3, 3, 4] are produced.
3.3 Semantic Integration Module
After aligning cross-modal information and activating the potential target region in MMF, we design Semantic Integration Module (SIM) to incorporate visual content and generate compact target representations. Specifically, we first leverage frame-level content aggregation to locate objects separately for each frame. Then we associate inter-frame dependency and model the temporal relationship of objects via video-level object cluster.

---

Language to Vision
Vision to Language Inc
Word Fimbaadinggg
Embeedding
Add & Norm
Atrention weights
Atlention weights
Self Attention
Lineer
Linear
Linear
Linear
Linear
Linear
0
A A0O
A
Video Embedding
Linear
Lincar
Lineur
Embeedding
Finheddiing
1000
A
A
Linguistic
Forre
Frumee
Frunne
Rame!
Frame:
Tone
Multi-Modal Fusion Module
Video-L.evel Object Cluster Module
Figure 3: The structure of proposed Multi-Modal Fusion module (MMF) and Video-level Object Cluster module (VOC).
Frame-Level Content Aggregation Having the activated visual features {F from MMF, we utilize a transformer-basee structure to locate objects in each frame. Firstly, K stacks of deformable transformer encoder layer [49] are leveraged to capture intra-frame relationships and further excavate multi-modal interactions inside {J.') }. This process can be formulated as:
Fmd = Decformennc (Fe (r'))
where f. denotes the activated visual features of the t-th frame. Then, set of learnable object queries [2, 49, 34] is introduced to aggregate image content and highlight potential target objects. Following [42]. these object queries fully interact with Fo in deformable transformer decoder through cross-attention mechanism. After extracting different object representations, these object queries are turned into instance embeddings Of E IR. N, x Note that we set up Ng object queries represent instances of each frame video clip so there are N₂T output object queries in total.
Video-Level Object Cluster Instances typically vary in pose and location between frames and being obseured. Previous methods [1, 5, 6, 42] only model instances separately for each frame, disregarding the temporal relationship and continuous motion of instances. Although such an approach can cover simple scenarios such as descriptions of target appearance, the lack of inter-frame, interaction makes existing methods inefficient for the description of temporal relationships. To address the aforementioned weakness, inspired by 11, 18], design video-level object cluster to capture the temporal information of instances between frames.
As shown in Fig. 3, after instances embedding O are formulated by the frame-level content aggre- gation, flatten it into O R D and employ self-attention mechanism along the temporal axis to introduce inter-frame interaction. Furthermore, we find that only introducing temporal self- attention is inferior. Simply sharing temporal object context may create redundancy due to the similarity of the representation referring to the same object in different frames, which potentially affects the model's precise understanding of objects in the video clip. To this end, employ object grouping decoder to perform object clustering and group the same object into video-level compact representations across frames. Specifically, we introduce No video-level object queries O" E IR N., D and they are initialized by linguistic sentence-lewe feature F=, which helps to promote the understanding of object descriptions and build the visual- linguis-i joint space. By performing interaction between linguistic-awar video queries and condensed instance information of each frame, the object grouping decoder can effectively capture temporal object contexts and group the referred object queries across frames to output the clustered video-level object queries O While temporal connections can provide
of valuable information, the segmentation of each frame is dependent on the specific image content. Thus, we incorporate the high-level video instance information into each frame to integrate the advantages of both. In detail, as illustrated in Fig. 2, we repeat the video-level object queries T times O" IR
7 N2 and enhance the representation of frame-level object queries
5

---

by element sum: O = Of + O"" In this way, the semantic of frame-level object queries is greatly enriched by the supplement of video-level object information.
3.4 Visual-Linguistic Contrastive Learning
Although the aggregated video-level representation can better describe object states, the simple ex- ploitation textual priors may lead to non-target potential responses for video-level embeddings [29]. Mcanwhile, taking unrestricted expressions as aggregation guidance may introduce undesirable inductive bias. To address the issues, we present a -ingu--ii contrastive loss that explicitly focuses the alignment between video-level features and textual information. The loss accompanies two purposes: (1) Bridge the semantic gap between textual expressions and the corresponding object queries the multi-modal joint embedding space. (2) Mitigate the bias caused by unrestricted textual features and emphasize the referred object representations. As shown in Fig. 2, we transform the last stage of fused textual features F from the vision-to- lo-nu-uge stream (see Eq. (3)) to generate the textual guidance embedding Fgud:
F Faud= = A AveragePooling
(FA') Food E R.D.
(5)
Then, we measure the similarity between the textual guidance embedding and video-level object To suppress undesired region response and
O"F
queries by scaled dot product similarity: Dsim VD highlight target object, take softmax operation along the object query axis. Finally, compute the contrastive loss Loon by matrix multiplication of Deim and yr. where y, is annotated to for the best predicted trajectory and 0 for others [43]:
Leon LogSoftmax (fstm) Vr-
(6)
3.5 Instance Segmentation and Loss
Prediction Heads As depicted in Fig. 2, there are three lightweight heads built on top of the semantic integration module. The classification head is concatenation of three fully connected layers. It directly predicts class probability e R x(K+1) for frame-level object queries, where K is the number classes. Note that if K = O, the role of the classification head is to judge whether the object is referred by the text description. The box head comprises three sequential linear layers, which is designed to transform the object queries to normalized bounding box information b E x i.e., center coordinates, width, and height. Similar to [42], we adopt dynamic convolution output segmentation masks for each frame. Specifically, the mask head produces weights of N₂ dynamic kernels Ω = [ Meanwhile, the encoded multi-scale cross-modal features (F are stacked with the 4x features from the visual backbone to form hierarchical decoding features. An FPN structure takes the hicrarchical features as input and outputs high- resolution semantic-aware features F #eg R'x #a xD Finally, the dynamic kernels process the features Fseg based on the position information output from the box head and obtain the segmentation mask m E TR Ha which can be formulated
{ face @
No
mr
Instance Matching As described above, the prediction heads output the prediction trajectories of Ng objects, denoted و {ūn} and the n-th object trajectory prediction is:
in = (p'.. By. m H:r
(8)
There is only one referred object in video clip corresponding to the text descriptions. Therefore, we denote the ground truth object sequence as y (p', b', mt)7 and search the best prediction trajectory ja via Hungarian algorithm [17].
Total Loss We supervise the trajectory prediction å. by four types of losses: (1) Lmask (u, _r), the mask loss is combination of Dice loss and binary focal loss, which is computed across frames. (2) Looz (y. yo ): the box loss aggregates the LI loss and GloU loss per- framfee (3) Lote (y. Do the class

---

Ref YouTube VOS
Ref-DAVISI7
Method
Backbone
J&F
J
F
JKF
J
URVOS [36]
ResNet-50
47.2
45.3
49.2
51.5
47.3
56.0
LBDT-4 [6]
ResNct-50
49.4
48.2
50.6
MTTR [1]
Video-Swin
53.3
54.0
56.6
ReferFormer [42]
Video-Swin-T
56.0
54.8
57.3
SOC (Ours)
Video Video-Swin-T
59.2
57.8
60.5
59.0
55.4
62.6
With Image Pretrain
ReferFormer [42]
Aideo-Swin-
59.4
58.0
60.9
59.7
56.6
62.8
ReferFormer [42]
Video-Swin-I
62.9
61.3
64.6
61.1
58.1
64.1
VLT [5]
Video-Swin-
63.8
65.6
61,6
64.3
SOC (Ours)
Video-Swiid
62.4
61.1
63.7
63.5
60.2
66.7
SOC (Ours)
Video-S Swin-B
66.0
64.1
67.9
64.2
61.0
67.4
Joint Train
ReferFormer ReferFormer SOC (Ours)
Video-Swin Sooo-S
62.6
59.9
63.3
Video-Swin- 64.9
62.8
67.0
Video- Swin- 65.0
63.3
66.7
64.2
60.9
67.5
SOC (Ours)
Video-Swin-2 67.3
65.3
69.3
65.8
62.5
69.1
Table 1: Comparison with the state-of-the-ee methods Ref-YYuTuube- VOS and Ref-DAVIS17 datasets. With Image Pretrain denotes the models are first pretrained on RefCOCO [15], Ref- COCO+ [15], and RefCOCOg [33] datasets. Joint Train indicates the models are trained with the combination of image datasets and video datasets.
loss is focal loss and supervises the predicted object category. (4) Leon (Ur, Dsim): visual-linguistic contrastive loss. The total loss be formulated as:
Liotal CraoskCmadk + Coow + NotoCols AconCoom
where À is the scale factor to balance each loss.
4 Experiment
4.1 Datasets and Metrics
Datasets. We evaluate our model on four prevalent RVOS benchmarks: Ref-YouTube- VOS [36], Ref-DAVISI7 [16], A2D-Sentences, and JHMDB-Sentent [8]. For detailed descriptions of the datasets please see the supplementary material.
Metrics. Following [1, 42), measure the effectiveness of our model by criteria of Precision @ K, Overall IoU, MeanloU and MAP over 0.50:0.05:0.95 for A2D-Sentences and Mcanwhile, adopt standard evaluation metries: region similarity([)), contour accuracy (F) and their average value (J&F) on Ref- YouTube- VOS and Ref-DAVIS17.
4.2 Implementation Details
We take the pretrained Video Swin Transformer [27] and RoBERTa [23] as our encoder in default. Both the frame aggregation and object cluster parts of SIM consist of three encoder and decoder layers. The number of frame-level queries Of and video-level queries O are set as 20 in default. We feed the model windows of w frames during training. The models are trained with eight 32GB V100 GPUs in default. The coefficients for losses are set Acto 2, ÅLI 2, Xgiou 2, Adice 2, Xyocat = 5, Acon. 1. Due to space limitations, please see the supplementary materials for more training details.
4.3 Main Results
Ref-YouTubb- & Ref-DAVIS17. We compare our method to previous models on Ref- YouTube- VOS and Ref-DAVIS17 in Table 1. With video-level modli--oddl understanding, our SOC achieves new state-of-the-art performance among different training settings: train from scratch, with image pretrain, and joint train. Without bells and whistles, our approach outperforms existing SOTA by about 3% J&F under fair comparison. On Ref-DAVIS17, we directly report the results using the model trained on Ref- YouTube-VooO without finetune.

---

Precision
IoU
Method
Backbone
mAP
P@0.5
P@0.6
P@0.77
P@0.8
P@0.9
Overall
Mean
Hu er al. [12]
VGG-16
34.8
23.6
13.3
3.3
0.1
47.4
35.0
13.2
Gavrilyuk et al.
[8]
I3D
47.5
34.7
21.1
8.0
0.2
53.6
42.1
19.8
CMSA + CPSA
[47]
ResNet-101
48.7
43.1
35.8
23.1
5.2
61.8
43.2
ACAN [40]
131)
55.7
45.9
31.9
16.0
2.0
60.1
49.0
27.4
CMPC-V [22]
13D
65.5
59.2
50.6
34.2
9.8
65.3
57.3
40.4
ClawCraneNet [19]
ResNet-50/101
70.4
67.7
61.7
48.9
17.1
63.1
59.9
MTTR [1]
Video-Swin-T
75.4
71.2
63.8
48.5
16.9
72.0
64.0
RefferFormer [42]
Video-Swin-T
76.0
72.2
65.4
49.8
17.9
72.3
64.1
48.6
SOC (Ours)
Video-Swin-T
79.0
75.6
68.7
53,5
19.5
74.7
66.9
50.4
With Image Pretrain
ReferFormer [42]
Video-Swin- Swin-T T
82.8
79.2
72.3
55.3
19.3
77.6
69.6
52.8
ReferFormer [42]
Video-Swin-B
83.1
80.4
74.1
57.9
21.2
78.6
70.3
55.0
SOC (Ours)
Video-Swin-T 83.1
80.6
73.9
57.7
21.8
78.3
70.6
54.8
SOC (Ours)
Video-Swin-B B
85.1
82.7
76.5
60.7
25.2
80.7
72.5
57.3
Table Comparison with the state-of-the-art methods on A2D-Sentences.
VOC VL J&F
J
F
Fusion Strategy J&F
J
F
VOC Method J&F
J
F
55.6
54.3
56.9
No Fusion
38.3
36.1
40.6
K-Means
33.8
32.9
34.7
58.0
56.5
59.6
V2L
39.8
37.6
42.0
ID-Conv
55.1
53.8
56.4
57.1
55.8
58.4
L.2V
58.7
57.3
60.0
Average
57.5
56.1
58.9
59.2
57.8
60.5
Both
59.2
57.8
Ours
59.2
57.8
60.5
Table 3: Effectiveness of our proposed modules.
Table 4: Impact of different fu- sion strategies.
Table 5: Impact of different clus- ter methods.
Query Num. J&.F
F
Frame Num. J&F
F
VOC Structure
J&F
J
58.0
56.6
59.5
3
57.9
56.5
59.3
None
57.1
55.8
58.4
58.1
56.7
59.6
5
58.4
57.1
59.7
only encoder
58.3
56.9
59.8
20
59.2
57.8
60.5
8
59.2
57.8
60.5
only decoder
58.1
56.7
59.5
25
58.5
57.2
59.8
10
59.1
57.8
60.3
Both
59.2
57.8
60.5
Table 6: Results of different ob- ject query numbers.
Table 7: Results of different
Table 8: The structure design of the VOC module.
training frame numbers.
A2D-Sentences & JHMDB-Sentencee, As shown in Table 2, compared with the existing SOTA method ReferFormer [42], SOC achieves 50.4% mAP and 66.9% mloU with the model trained from scratch, which gains clear improvement 1.8% mAP and 2.8% mloU respectively. Fur- thermore, with image data pretraining, method achieves e.g., +4.0% P.0.9, +2.2% mloU, and +2.3% mAP. Due to the similarity of the datasets and space limitations, please see the supplementary material for the comparison JHMDB-Sentences [8]. Our SOC also surpasses all existing methods on it.
state-of-the-ard on all metrics,
4.4 Ablation Studies
In this section, we conduct analysis experiments on the Ref- YouYYoTuuubeaa VOS [36] benchmark using Video-Swin-Tiin woin-Tiny&& as the visual backbone and train the model from scratch.
Component Analysis We experiment with the effectiveness of the proposed Video-level Object Cluster (VOC) and Visual-linguistic contrastive Learning (VL). As shown in Table 3, both of them bring performance improvement and their combination works better (+ 3.6% J&F), which shows the significance of video-level multi-modal understanding.
Fusion Strategy We investigate the effects of various fusion strategies. As illustrated in Table the absence of Language to Vision fusion (L2V) yields significant deterioration. This is because the frame-level content aggregation without textual guidance would introduce background noise. If only utilizing the L2V fusion, the performance drops 0.5% J J&F, demonstrating that visual content helps filter irrelevant words from unconstrained descriptions.
VOC Method representation learning ability of the transformer structure. Compared to other clustering methods
The superiority the current VOC module derives from two aspects: i) Powerful

---

the gjrnafie the right side of the railing and has bead hanging over and eating grass
black knife holder
a skateboard being rode by person weariag red hat and jumging in the air
persoz ska tabboarding
Figure 5: Segmentation results of our SOC, Best viewed in color.
person is jumping over the river
persion is jumping over the river
tecinis plaver running from right left
tetnis player ranning from right to left
(b)
Figure 6: Visualization comparison using text expressions about temporal actions. (a) and (b) are segmentation results of our SOC and ReferFormer [42], respectively.
shown in Table 5, the transformer architecture has better perception well selection of global information, which helps to associate object features across frames. ii) Sufficient inter- interac- tions. Although other methods also yield video-level representations, they lack the perception of temporal variations and the alignment inter-frame information. As a contrast, our VOC performs adequate inter-frame interaction while clustering.
Object Query Number Although only one referred object is involved in a video, focusing more potential regions helpful due to the various video content. Table shows that an increased number of object queries allows the model to effectively group the relevant objects from the pool of candidate regions. The performance saturates when the query number is increased to a certain extent. We believe that excessive queries may cause misunderstandings when grouping objects across frames.
Segmentation Stability Benefiting from video-level multi-
0.04
modal understanding, our method yields more robust segmen- tation results compared to existing approaches. To quantify the temporal segmentation stability, we calculate the loU and J&FF variance between each frame in a video and count the average value Ref- YouTube- VOS [36] dataset. Results shown in Fig. 4 (the lower is better). It can be seen that with video object cluster module and
0.03
0.025
(0ur)) Referformer MITR loll Variance
contrastive loss building the video-level multi-modal aligned space, our SOC understand the overall video content better and output temporally stable segmentation results. More qualitative analysis temporal consistency is shown in the supplementary material.
J Varianee
Figure 4: loU and J&J variance.
Frame Number Table 7 shows results with different training clip frame numbers. As expected, widening temporal context leads to performance gains yet expensive computation cost. For fair comparison with existing methods, we take the frame number of 8 by default.
Variants of VOC Structure Results in Table 8 illustrate the rationality of the structure of the VOC module. Without sufficient inter-frame interaction by the encoder or aggregation of target object information by the decoder, the segmentation performance will be degraded.
Inference Speed In addition to superior segmentation performance, our method also achieves real-time inference. Specifically, our SOC runs at 32.3 FPS on single 3090 GPU, which significantly exceeds the existing SOTA method ReferFormer [42] (21.4 FPS).

---

horse running the left of another horse towerdss the weirs
(a)
(b)
the person on the hile, with aallww and pack to the right
(a)
(b)
Figure 7: Visualization results of effectiveness on VOC. The object in the red box is the target. (a) displays the segmentation result of the completed model. While (b) shows the result of the model without VOC (Video Object Cluster SIM).
4.5 Qualitative Results
Fig. demonstrates the effectiveness of SOC for complex scenarios segmentation, i.e., similar appearance, occlusion, and large variations. To further prove the adaptability of our SOC for understanding text expressions about temporal action and variations, descriptions that depict changed states of the object in temporal. As shown in Fig. 6, the global view enables SOC understand such and segment the target object accurately. In contrast, ReferFormer [42] only recognizes the object in specific frames mentioned in the text descriptions and fails comprehend the content. More comparisons can be seen in the supplementary material. Fig. 7 indicates that due to the temporal modeling and inter-frame interaction in VOC, the full model comprehends text descriptions of temporal variations and tracks the target coherence. contrast, without VOC, the model struggles to understand the temporal relationship and fails to segment the object.
design corresponding text
Conclusion
In this paper, propose framework called SOC for RVOS to achieve video-level multi-modal understanding. By associating frame-level object embeddings with language tokens, we unify temporal modeling and cross-modal alignment into simple architecture. Moreover, visual-linguistic contrastive learning is introduced to build the video-level multi-modal space. Extensive experiments show that our SOC remarkably outperforms existing ofte--of-the-ar methods. Besides, video-level understanding also allows our SOC to handle text descriptions expressing temporal variations better.
Limitations. The proposed framework, SOC, has achieved video-level multi-modal understanding and excellent performance. However, there are some potential limitations, e.g., the current architecture cannot handle infinitely long videos. We think that devising explicit designs for long videos with complex scenarios is an interesting future direction.
Acknowledgements
This work supported by the National Natural Science Foundation of China (Grant No. U1903213) and the Shenzhen Science and Technology Program (JSGG20220831 110203007). This work was also supported by the National Key Research and Development Program of China under Grant 2020AAA0108302 and Shenzhen Stable Supporting Program (WDZ7C3000062200005000)
References
[1] Botach, A., Zheltonozhskii, E., Baskin, C.: End-to-end referring video object segmentation with multimodal transformers. In: CVPR. pp. 4975-4985 (2022) 1, 3, 5,
10

---

[2] Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV. pp. 213-229 (2020) 5
[3] Ding, H., Jiang, X., Shuai, B., Liu, A.Q., Wang, G.: Semantic correlation promoted shape- variant context for segmentation. In: CVPR. pp. 8885-8894 (2019)
[4] Ding, H., Liu, C., Wang, S., Jiang, X.: Vision-language transformer and query generation for referring segmentation. In: ICCV. pp. 16301-16310 (2021)3
[5] Ding, H., Liu, C., Wang. S., Jiang, X.: Vlt: Vision- languuge transformer and query generation for referring segmentation. TPAMI (2022) 2, 5,7
[6] Ding, Z., Hui, T., Huang, J., Wei, X., Han, J., Liu, S.: Language-bridged spatial-temporal
interaction for referring video object segmentation. In: CVPR. pp. 4964 4973 (2022) 5,
[7] Duke, B., Ahmed, A., Wolf, C., Aarabi, P., Taylor, G.W.: Sstvos: Sparse spatiotemporal transformers for video object segmentation. In: CVPR. pp. 5912-5921 (2021) 5
[8] Gavrilyuk, K., Ghodrati, A., Li, Z., Snoek, C.G.M.: Actor and action video segmentation from (2018) 2, 7,8
a sentence. In: CVPR. pp.
[9] Han, K., Liu, Y., Liew, J.H., Ding, H., Wei, Y., Liu, J., Wang, Y., Tang, Y., Yang, Y., Feng, J., et al.: Global knowledge calibration for fast open- apee-vocabblary segmentation. arXiv preprint arXiv:2303.091 109111111111111. (2023)* 2
[10] He, C., Li, K. Zhang, Y., Xu, G., Tang, L., Zhang, Y., Guo, Z., Li, X.: Weakly-supervised concealed object segmentation with sam-based pseudo labeling and multi-scale feature grouping. arXiv preprint arXiv:2305. 1003 (2023)
[11] Heo, M., Hwang. S., Oh, S.W., Lee, J., Kim, SJJ.: VITA: video instance segmentation via object token association. arXiv preprint 2XXi:2206.044000 (2022) 5
[12] Hu, R., Rohrbach, M., Darrell, T.: Segmentation from natural language expressions. In: Leibe,
B., Matas, J., Sebe, N., Welling, M. (eds.) ECCV. pp. 108-124 (2016) 3, 8
[13] Hu, Z., Feng, G., Sun, J., Zhang, L., Lu, H.: Bi-directional relationship inferring network for referring image segmentation. In: CVPR. pp. 4423-4432 (2020) 3
[14) Huang, S., Hui, T., Liu, S., Li, G., Wei, Y., Han, J., Liu, I.., Li, B.: Referring image segmentation via cross-modal progressive comprehension. In: CVPR. pp. 10488-10497 (2020)
(15) Kazemzadeh, S., Ordonez, V., Matten, M., Berg. T.: Referitgame: Referring to objects in
photographs of natural scenes. In: EMNLP. pp. 787-798 (2014)
[16] Khoreva, A., Rohrbach, A., Schiele, B.: Video object segmentation with language referring expressions. In: ACCV. pp. 123-141 (2018) 2,
[17] Kuhn, H.W.: The hungarian method for the assignment problem. In: Naval research logistics quarterly (1955)
[18) Li, X., Zhang, W., Pang, J., Chen, K., Cheng, G., Tong, Y., Loy, C.C.: Video k-net: A simple, strong, and unified baseline for video segmentation. In: CVPR. pp. 18847-18857 (2022) 5
[19] Liang, C., Wu, Y., Luo, Y., Yang, Y.: Clawcranenet: Leveraging object-level relation for text-based video segmentation. arXiv preprint arXiv:2103. (2021) 3,
[20] Liang, C., Wu, Y., Zhou, T., Wang, W., Yang, Z., Wei, Y., Yang, Y.: Rethinking cross-modal interaction from top-down perspective for referring video object segmentation. arXiv preprint arXiv:2106.0106 (2021)² 2
[21] Liu, J., Ding, H., Cai, Z., Zhang, Y., Satzoda, R.K., Mahadevan, V., Manmatha, R.: Poly- former: Referring image segmentation as sequential polygon generation. arXiv preprint arXiv:2302.07387 (2023) 3

---

[22] Liu, S., Hui, T., Huang, S., Wei, Y., Li. B., Li. G.: Cross-modal progressive comprehension for referring segmentation. TPAMI pp. 4761-4777 (2022) 3,
[23] Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer,
L., Stoyanov, V.: Roberta: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11 1692 (2019) 7
[24] Liu, Y., Yu, R., Wang, J., Zhao, X., Wang, Y., Tang, Y., Yang. Y:: Global spectral filter memory network for video object segmentation. In: ECCV. pp. 648 665 (2022)
[25] Liu, Y., Yu, R., Yin, F., Zhao, X., Zhao, W., Xia, W., Yang, Y.: Learning quality- dynamic memory for video object segmentation. In: ECCV. pp. 468-486 (2022)
[26] Liu, Y., Yu, R.. Zhao, X., Yang, Y.: Quality-aware and selective prior enhancement memory network for video object segmentation. In: CVPR Workshop. vol. (2021) 1
[27] Liu, Z., Ning, J., Cao. Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. In: CVPR. pp. 3192-3201 (2022) 4,7
[28] Luo, G., Zhou, Y., Sun, X., Cao, L., Wu, C., Deng, C., Ji, R.: Multi-task collaborative network for joint referring expression comprehension and segmentation. In: CVPR. pp. 10034-10043 (2020) 2
[29] Ma, Y., Wang, Y., Wu, Y., Lyu, Z., Chen, S., Li, X., Qiao, Y.: Visual knowledge graph for human action reasoning in videos. In: Proceedings of the 30th ACM International Conference on Multimedia. pp. 4132 4141 (2022) 6
[30] McIntosh, B., Duarte, K., Rawat, Y.S., Shah, M.: Visual-textual capsule routing for text-based video segmentation. In: CVPR. pp. 9942-9951 (2020) 3
[31] Miao, J., Wang, X., Wu, Y., Li, W., Zhang, X., Wei, Y., Yang. Y.: Large-scale video panoptic segmentation the wild: A benchmark. In: CVPR. pp. 21033-21043 (2022)
[32] Miao, J., Wei, Y., Wu, Y., Liang, C., Li, G., Yang, Y.: Vspw: A large-scale dataset for video scene parsing in the wild. In: CVPR. pp. 4133-4143 (2021) 1
[33] Nagaraja, V.K., Morariu, V.I., Davis, L.S.: Modeling context between objects for referring expression understanding. In: ECCV. pp. 792-807 (2016)
[34] Ni, X., Wen, H., Liu, Y., Ji, Y., Yang. Y.: Multimodal protorype-enhanced network for few-shot action recognition. arXiv preprint 1xXiv22212.0487: (2022) 5
[35] Ning, K., Xie, L., Wu, F., Tian, Q.: Polar relative positional encoding for video-language segmentation. In: ICAA p. 10 (2020) 3
[36] Seo, S., Lee, J., Han, B.: URVOS: unified referring video object segmentation network with large-scale benchmark. In: ECCV. pp. 208 -23 (2020) 2, 3, 7, 8,
[37] Shelhamer, E.,, Long, J., Darrell, T.: Fully convolutional networks for semantic segmentation. TPAMI pp. 640-651 (2017) 3
[38] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, L: Attention is all you need. NeurlPS 30 (2017) 4
[39] Wang, H., Deng, C., Ma, F., Yang, Y.: Context modulated dynamic networks for actor and action video segmentation with language queries. In: AAAI pp. 12152-12159 (2020) 3
[40] Wang, H., Deng, C., Yan, J., Tao, D.: Asymmetric cross-guided attention network for actor and action video segmentation from natural language query. In: ICCV, pp. 3938-3947 (2019) 3, 8
[41] Wang, Z., Lu, Y., Li, Q. Tao, X., Guo, Y., Gong, M., Liu, T.: Cris: Clip-driven referring image segmentation. In: CVPR. pp. 11686-1 1695 (2022)>
[42] Wu, J., Jiang, Y., Sun, P., Yuan, 7., Luo, P.: Language as queries for referring video object segmentation. In: CVPR. pp.
(2022) 1, 2, 3, 5, 6, 7, 8, 9, 10
12

---

[43] Xiao, Y., Ma, Y., Li, S.. Zhou, H., Liao, R., Li, X.: Semanticae: Semantics-assen assisted framework for audio classification. In: ICASSP 2023-2023 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP). pp- 1-5 (2023)
[44] Xu, M., Zhang, Z., Wei, F., Lin, Y., Cao, Y., Hu, H., Bai, X.: A simple baseline for open- vocabulary semantic segmentation with pre-trained vision-lane langugae model. In: ECCV. pp. 736-753 (2022)* 2
[45] Yang. Z... Wang, J., Tang, Y., Chen, K., Zhao, H., Torr, P.H.S.:: Lavt: Language-awai aware vision transformer for referring image segmentation. In: CVPR. pp. 18134-1814 (2022)
[46] Ye, L., Rochan, M., Liu, Z., Wang, Y.: Cross-modal self-attention network for referring image segmentation. In: CVPR. pp. 10502-10511 (2019)
[47] Ye, L.., Rochan, M., Liu, Z., Zhang, X., Wang, Referring segmentation in images and videos with cross-modal self-attention network, TPAMI pp. 3719-3732 (2022) 3, 8
[48] Zhang, Y., Yuan, L., Guo, Y., He, Z., Huang, I., Lee, H.: Discriminative bimodal networks for visual localization and detection with natural language queries. In: CVPR. pp. 1090-1099 (2017) 2,
[49] Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, Deformable DETR: deformable transformers for end-to-end object detection, In: ICLR (2021) 5
13


Reviews:
[{"review_id": "IkGofoUDeR", "reviewer": "Reviewer_iFku", "summary": "In this paper, the author presents a framework called SOC, and a visual-linguistic contrastive loss for RVOS to unify temporal modeling and cross-modal alignment.\nThe performance is good according to the leaderboard (https://paperswithcode.com/sota/referring-video-object-segmentation-on-refer)\nThe extra module seems to have many parameters, and somehow incremental. ", "questions": "Please see Weakness. I am not working in this field. If the author could address my concerns, I will upgrade my rate.", "limitations": "Please see Weakness. I am not working in this field. If the author could address my concerns, I will upgrade my rate.", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "In this paper, the author presents a framework called SOC, and a visual-linguistic contrastive loss for RVOS to unify temporal modeling and cross-modal alignment.\nThe performance is good. ", "weaknesses": "1. Parameters\nThe extra module seems to have many parameters, and somehow incremental.  \n- Unfair Comparison. It is unfair to compare with other methods. \n- Design of MMF is not new. Many multi-modality fusion has used similar structure such as [a]. The different  part is that authors adopt transformer structure. \n[a] Comprehensive Linguistic-Visual Composition Network for Image Retrieval \n\n2. Motivation in Figure 1 is not clear to me. \nI think it can be easily solved by  adopting the video-based backbone such as TSM or I3DNet, \n\n3. The visual-linguistic contrastive loss\nI do not see significant difference from the basic contrastive loss in Clip or other works. The input is changing from image to video features. Could you further explain it? \n\n ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the Reviewer iFku for his/her comments and the appreciation of our work. To the concerns expressed by Reviewer iFku in Weaknesses and Questions , our response is as follows:\n\n*Weakness-1:* Thanks for reminding us that we should make a clear demonstration of model parameters. In our supplementary materials, we compared our method with previous state-of-the-art RVOS approaches (MTTR, ReferFormer) in terms of performance, FPS, and FLOPs. Here we add parameter comparisons and list comprehensive data in the table below. Our method achieves superior performance with faster inference speed and fewer parameters compared to the previous SOTA method Referformer. While MTTR has the lowest FLOPs and the fewest parameters, the lack of elaborate multi-modal fusion and temporal interaction significantly degrades segmentation accuracy.\n\n| Model | Backbone | Parameters | $\\mathcal{J} $&$ \\mathcal{F}$ | FPS | FLOPs (a video with 43 frames)|\n|-----|-----|-----|-----|-----|-----|\n| MTTR | Video-Swin-T | 104.88M | 55.3 | 31.8 | 1018G |\n| ReferFormer | Video-Swin-T | 131.67M | 56.0 | 21.2 | 2994G |\n| SOC (Ours) | Video-Swin-T | 131.46M | 59.2 | 32.3 | 1879G |\n\n- Unfair Comparison: Thanks for the question. We would like to confirm whether the question about potential unfair comparisons stems from your concern about the parameters? If so, the above table demonstrates that our method achieves higher segmentation accuracy with fewer parameters compared to the previous SOTA method ReferFormer. Besides, all comparisons in the manuscript are based on fair settings.\n\n- Design of MMF is not new: Thanks for the question. MMF is just a component of our framework. It is not our contribution, and it is not mentioned in the contribution note in the introduction section of the paper. Our main contribution lies in the proposed framework, which unifies temporal modeling and cross-modal alignment to achieve video-level visual-linguistic understanding. The core module of our framework is the Semantic Integration Module. By fusing intra-frame and inter-frame information, our approach achieves better performance and can handle descriptions of temporal variations. Regarding the MMF module specifically, it serves to activate textual referred objects in visual features and simultaneously update the textual embeddings by incorporating visual content.\n\nWe wonder if the above response addresses your questions about the model parameters and the incremental modules. If you still have any questions about our response, we would be pleased to discuss them with you.\n\n*Weakness-2:* Good question. TSM or I3D can indeed provide temporal information. However, segmentation tasks need spatially fine-grained features and emphasize pixel-level understanding. I3D or TSM can only provide coarse-grained features (frame-level or video-level). Using I3D directly as the backbone would significantly degrade segmentation accuracy. That's why I3D is not used in video segmentation. In addition,  a viable option that can take advantage of the ability of I3D to provide video-level information may be to employ two backbones at the same time, *i.e.*, a spatial backbone and a temporal backbone (I3D). This approach, undoubtedly, introduces an excessive number of parameters and computational costs. In contrast, our method realizes the fusion of intra-frame fine-grained information and inter-frame temporal relations efficiently.\n\n*Weakness-3:* Thanks for the question. We believe that the form of the visual-linguistic loss is not significant, but rather its motivation and the supervision object. Specifically, this loss is applied to further build the multi-modal joint space at the video level. Unlike CLIP that performs  contrast learning  between different images and captions inside the training batch, what we want to establish is the contrast between different object queries and the same text descriptions. Furthermore, we would like to emphasize that the main contribution of this paper is to unify temporal modeling and cross-modal alignment to achieve video-level visual-linguistic understanding and excellent segmentation performance. In other words, the design of the pipeline, especially the Semantic Integration Module, is the core of this paper.\n\n"}, {"review_id": "74hvfhKcRp", "reviewer": "Reviewer_8Kjg", "summary": "This paper mainly studies referring video object segmentation (RVOS) which is a fundermental problem in vision-and-language applications. Compared with previous works which pay little attention to temporally modeling relationships between different frames in the entire video clip, the authors proposed a framework called SOC for RVOS which conducts temporal modeling and cross-modal alignment in a unified framework. Specifically, SOC contains several modules including textual and vision encoding and multi-modal fusion, semantic integration module and vision-language contrastive learning. The experimental results show SOC outperforms existing SOTA method, e.g. ReferFormer published in CVPR 2022. ", "questions": "\n(1) Why the SIM module is not designed inside the video encoder? Please refer to the second part in the above weaknesses. \n\n(2) It is reported that the SOC runs at 32.3 FPS on a single 3090 GPU. I think the overhead of each module should be further decomposed, e.g. the encoding backbone, multi-modal fusion module, and the SIM.\n\n(3) As illustrated in Table 4, L2V outperforms V2L significantly. Deeper insights are expected to be uncovered. Although using both of them achieves the best performance, some missing details should be claimed. For example, what is each branch's weight? \n\n(4) Could the proposed SOC be extended to solve multiple referred objects given a textual query?", "limitations": "The authors have mentioned that malicious use of such model could lead to negative social impacts such as unauthorized video surveillance. However, I believe the positive impacts of these vision-and-language applications would improve the embodied AI research and generally outweigh the risks.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "(1) Overall, the motivation for poposing the temporal modeling intra-frame and inter-frame information for RVOS is quite clear and straightforward. Besides, the paper is well-writen and the idear as well as each module is easy-to-follow.\n\n(2) The most valuable contribution is the video-level object cluster module in the semantic integration module (SIM).  Instead of using traditional clustering method such as k-means, the authors proposed an object grouping decoder architecure to perform the clustering process. Specifically, N_v video-level object queries are introduced to learn the overall object centroids in the entire video. After repeating such operation T times, the semantic representation enhancement is achieved by enriching the video-level object information. Figure 1 in the supplementary material demonstrates that most of these learnable queries focus on the region-of-interests (ROIs)  of he referred objects. \n\n(3) The experimental results are promising. The proposed SOC framework outperforms other existing SOTA works under different settings. The provided video demo shows the referred objects have been segmented perfectly. ", "weaknesses": "(1) According to my understanding of this work, the main contributions is the video-level object cluster module. Hence, the albation studies or comparison between different methods should focus on such module. Specifically, is there any other clustering-based method that could obtain video-level object embeddings? Why is the proposed object grouping decoder-style model is the best solution among these optional methods? Besides such high level comparisons, some details (or parameters) of abaltion studies should be further discussed. For example, the number of object queries and number of temporal frames are tuned independently in Table 5 and 6. However, jointly tuning these two parameters should be investigated.  Another example is that the final representation is the element-wis sum of frame-level and video-level semantic embeddings. Are there any other better learning-based fusion implementations?\n\n(2) Generally, the proposed framework follows a two-stream fashion which includes a textual encoder and a vision encoder. Note that the vision encoder could be a temporal convolutional network (3D CNN) or video transformer such as in this work. No matter which implementation, it undergoes the temporal processing. The question is: could the video-level object clustering module be embedded inside such temporal encoding procedure? I think the whole inferece speed could be further decreased if it is possible.    \n\n(3) I think the main contribution and the motivation is clear.  Some trivial parts such as the visual-linguistic contrastive learning could be removed in order to make the key points stand out, which will not reduce the whole work's contribution or value. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\nWe sincerely thank the Reviewer 8Kjg for his/her comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:\n\n*Weakness-1:* (1) Thanks for your valuable question. Table 7 in the manuscript shows the ablation of the structure of the current video-level object cluster (VOC) module.  Here we experiment with several intuitive methods to obtain video-level object embeddings. Specifically, \"Average\" means global pooling of all frames-level queries at the corresponding positions. \"1D-Conv\" denotes that we use 1D convolution (kernel size is 5) to process the frame-level query and then pooling them. It can be seen that K-Means may lead to information confusion.\n\n| VOC Method | Average | K-Means | 1D-Conv | Ours |\n| --- | :---: | :---: | :---: | :---: |\n| $\\mathcal{J} $&$ \\mathcal{F}$| 57.5 | 33.8 | 55.1 | 59.2 |\n\n(2) We believe that the superiority of the current VOC module derives from two aspects: i) Powerful representation learning ability of the transformer structure. Compared to other clustering methods, the transformer architecture has a better perception as well as selection of global information, which helps to associate object features across frames. ii) Sufficient inter-frame interactions. Although other methods can also yield video-level representations, they lack the perception of temporal variations and the alignment of inter-frame information. As a contrast, our VOC performs adequate inter-frame interaction while clustering.\n\n(3) Thanks!  Here we involve more experiments on jointly tuning the object queries and temporal frames.\n\n| Query Num. | Frame Num. | $\\mathcal{J} $&$ \\mathcal{F}$ |\n| --- | --- | --- |\n| 10 | 3 | 58.1 |\n| 15 | 5 | 57.9 |\n| 15 | 8 | 58.1 |\n| 20 | 8 | 59.2 |\n| 25 | 3 | 57.8 |\n| 25 | 10 | 58.4 |\n\n(4) We experiment with two learning-based fusion methods for comparison. Self-Attn: we concatenate the frame-level and video-level queries along the sequence length dimension to obtain a new sequence, then perform self-attention operation to this sequence, and extract the corresponding frame-level query according to the index. Cross-Attn: we take frame-level object embeddings as the query, and the video-level embeddings as the key and value. Results are shown as below. It can be seen that the performance of the learnable fusion methods is about the same as element-wise sum.\n\n| Fusion Method | Add | Self-Attn | Cross-Attn |\n| --- | :---: | :---: | :---: |\n| $\\mathcal{J} $&$ \\mathcal{F}$| 59.2 | 59.1 | 58.9 |\n\n*Weakness-2:* Interesting question! This is a valuable research direction. Limited by rebuttal time and computation resources, we are unable to provide some experiment results at this time. Nevertheless, we have thoroughly considered the matter. We believe that there exists three potential challenges to be addressed for this idea. i) The lack of multi-modal interaction within the visual backbone may lead to noise when aggregating different frame embeddings. Specifically, without highlighting the target objects (*i.e.*, the MMF module and frame-level content aggregation), video-level information fusion  may lead to feature confusion due to the diversity of video scenes. ii) Computation cost. Since inter-frame interactions are performed by compact representations, *i.e.*, object query, our VOC is computation-friendly and efficient. Video-level clustering inside the encoder is performed on feature maps, which may lead to huge computations. iii) Modifying the structure of backbone may disrupt the pretrained parameter relationship and potentially impair the backbone's ability. We will further explore this in our future work.\n\n*Weakness-3:* Thanks for your advice and appreciation of our work! We will integrate the visual-linguistic contrastive loss into the introduction of total loss in Section 3.5. Additionally, we will strive to provide more comprehensive explanations and emphasize our primary contribution.\n\n*Question-1:* Please see the response to *Weakness-2*.\n\n*Question-2:* Thanks for your great suggestion. We decompose each module of our method and report the overhead in the table below. It can be seen that our VOC module is efficient. This also shows the rationality and advantages of the design of our VOC module.\n\n| Modules | Backbone | MMF | Frame-Level Aggregation | Video-Level Clustering | Head | Total |\n| --- | :---: | :---: | :---: | :---: | :---: | :---: |\n| Time per frame (ms) | 10.4 | 0.41 | 2.48 | 0.098 | 0.46 | 30.9 |\n\n*Question-3:* (1) Deeper explanation: L2V stream highlights the corresponding regions of the referred object in each frame and mitigates the impact of background noise. It serves as an important pre-fusion procedure to enable the model to focus more on the referred object in each frame so that the following SIM module can extract the referred object embeddings and aggregate intra-frame and inter-frame information. Without the L2V stream, the model loses the perception of the referred object and can hardly output desired segmentation results. In terms of the V2L stream, it updates the textual embedding with image content, which helps to relieve the potential ambiguity of unconstrained descriptions. Without the V2L stream, the model is still able to perceive the referred object. Therefore, L2V outperforms V2L significantly. (2) Details: This question may leave us a little confused. Figure 3 in the manuscript shows that the V2L and L2V have their own function and operate separately without any balancing weights. The attention weights in the figure refers to the attention map computed by query and key. If you still have any questions, we would be glad to further discuss with you.\n\n*Question-4:* We think it's OK. Because multiple objects can be perceived by multiple object queries. The head and cross-modal interaction module may need to be modified accordingly. The main limitation may be that the current dataset and settings of RVOS task only involve single target object.\n\n"}, {"review_id": "oTMCRpFHcR", "reviewer": "Reviewer_NrNc", "summary": "This article introduces the Semantic-assisted Object Cluster model, which is used to solve the problem of visual language alignment between videos and texts. The model uses a semantic integration module and visual language contrastive learning to improve the understanding of video content and has achieved significant improvements in different scenes and text descriptions.", "questions": "1．For the Video-Level Object Cluster module, the design of the object clustering module does not fully consider the similarities and differences between instances, which may affect the clustering effect. The paper should provide some considerations on this issue.\n\n2．For the Visual-Linguistic Contrastive Learning module, the issue of potentially non-target responses in video-level embeddings due to the simple use of text priors should not be ignored.\n\n3．The model uses Video-Swin-B and Video-Swin-T as backbones, so it is unfair to compare it with other models that use resnet as backbone.\n", "limitations": "1．The model may rely on the accuracy of semantic information, as it uses the Semantic Integration Module (SIM) to aggregate information in the video.\n\n2．The model may not be able to handle some complex video scenes because it only performs inference based on known semantic information.\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1．The Semantic-assisted Object Cluster (SOC) framework has been proposed, which effectively aligns video and text, realizes object aggregation and visual-language alignment, and improves the performance of the RefVOS model. \n2．The Semantic Integration Module (SIM) has been designed to efficiently aggregate intra-frame and inter-frame information, understand video content from a global perspective, and help understand temporal changes and alignment of different modalities and granularities. \n3．Visual-linguistic contrastive learning has been introduced to provide semantic supervision, guide the establishment of video-level multimodal joint space, and improve the performance of the RefVOS model. 4、It can effectively handle text descriptions that express temporal changes and significantly improve general scene processing. \n4.On popular RVOS benchmarks such as Ref-YouTube-VOS, Ref-DAVIS, A2D-Sentences, and JHMDB-Sentences, the performance is significantly better than existing methods, and inference speed is faster.\n", "weaknesses": "1．The effectiveness of Visual-Linguistic Contrastive Learning module heavily depends on the quality of the text descriptions used for supervision. If the text descriptions are inaccurate or incomplete, the performance of this module could be limited.\n2．The efficiency of SIM is not clearly explained, and it is unclear how well it would scale to larger and more complex datasets.\n3. The caption of many figures (such as Fig.3) are too simple, and more details should be provided.\n4.Ablation experiments were conducted only within each module, without verifying the effectiveness of each module on the whole network framework.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the Reviewer NrNc for his/her comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:\n\n*Weakness-1:* Good question! Inaccurate or incomplete descriptions do affect the effectiveness of Visual-Linguistic Contrastive Learning. To alleviate this problem, we adopt the V2L stream in MMF that updates the textual embedding with video content. Besides, such ambiguous descriptions would degrade the performance of all RVOS methods. In fact, the textual description is artificially controllable. Users can easily provide clear descriptions in most cases. Finally, our approach surpasses previous methods even without Visual-Linguistic Contrastive Learning module, thanks to our video-centric paradigm design.\n\n*Weakness-2:* Thanks for your suggestion. We further investigate the overhead of each module and present the results in the table below. Results show that SIM accounts for only 8.3% of the total inference time. \n\n| Modules | Backbone | MMF | Frame-Level Aggregation | Video-Level Clustering | Head | Total |\n| --- | :---: | :---: | :---: | :---: | :---: | :---: |\n| Time per frame (ms) | 10.4 | 0.41 | 2.48 | 0.098 | 0.46 | 30.9 |\n\n\nBesides, because the VOC module in SIM is central to video-level multimodal understanding, the ablation we perform in the manuscript is for the VOC module. Here we perform detailed experiments on the effectiveness of SIM and the results are shown below.\n\n| Model | No SIM | Only Frame-Level Aggregation | Only Video-Level Clustering | Entire SIM |\n| --- | :---: | :---: | :---: | :---: |\n| $\\mathcal{J} $&$ \\mathcal{F}$ | 51.6 | 57.1 | 54.7 | 59.2 |\n\nTo the best of our knowledge, there are no larger scale annotated datasets available in this field. We have evaluated our model on all popular RVOS benchmarks and the performance is significantly better than existing methods. In addition, we also agree that large-scale RVOS datasets for complex scenarios are meaningful.\n\n*Weakness-3:* Thanks for your valuable comments! Due to page limitations in the review version (one page less than the final version), some captions may be too concise. We will revise the paper based on your suggestions to provide more details and make the caption clearer.\n\n*Weakness-4:* Thanks for your advice. We experiment with the effectiveness of each module on the whole network and the results are shown in the table below. \"W/o MMF\" indicates replacing MMF with simple cross attention operation, *i.e.*, the approach used in previous models. We will update these results in the manuscript.\n\n\n| Model | $\\mathcal{J} $&$ \\mathcal{F}$ | $\\mathcal{J} $ | $\\mathcal{F} $ |\n| :---: | :---: | :---: | :---: |\n| Full Model | 59.2 | 57.8 | 60.5 |\n| W/o SIM | 51.6 | 50.4 | 52.8 |\n| W/o MMF | 58.7 | 57.3 | 60.0 |\n| W/o VL | 58.0 | 56.5 | 59.5 |\n\n\n*Questions-1:* On the one hand, frame-level content aggregation highlights the information about the target object in each frame, enabling most queries to perceive the referred object (please see query visualization in the supplementary materials). This can effectively suppress noise caused by other instances during the fusion process. On the other hand, the transformer architecture has powerful feature recognition and extraction capabilities, which are also helpful in generating desirable video-level object embeddings. We agree that explicitly considering similarities and differences between instances may yield better video-level object representations, which is an interesting research direction. It will be our future work.\n\n*Questions-2:* Thanks for the question. The design of V2L stream in MMF is to relieve the potential ambiguity of unconstrained descriptions by incorporating visual content into linguistic embeddings. Besides, the Visual-Linguistic Contrastive Loss is designed to relieve the potentially non-target responses in video-level embeddings by discerning the optimal query for emphasis and suppressing the response of other queries. As for the potential effect of imperfect textual descriptions on Visual-Linguistic Contrastive Loss, please refer to the response to *Weakness-1*.\n\n*Questions-3:* Here we provide the comparisons of our method with others using ResNet-50  as backbone. Results show that our SOC also outperforms previous methods.\n\n\n| Method | $\\mathcal{J} $&$ \\mathcal{F}$ | $\\mathcal{J} $ | $\\mathcal{F} $ |\n| :---: | :---: | :---: | :---: |\n| URVOS | 47.2 | 45.3 | 49.2 |\n| LBDT-4 | 49.4 | 48.2 | 50.6 |\n| ReferFormer | 52.1 | 51.4 | 52.8 |\n| SOC (Ours) | **54.8** | **53.7** | **55.9** |\n\n*Limitations:* Thanks for your acknowledgement of our work. Sorry that we are not quite sure whether the semantic information here refers to the text description or the video content. If text descriptions, please see the response to *Weakness-1*. If video content or object queries, we believe the limitations mentioned here may not be significant. On the one hand, we propose frame-level content aggregation to efficiently extract information about the target object and suppress the background region (visualization of query can be seen in the supplementary materials). On the other hand, the superiority of the learnable query representation paradigm has been proven in many tasks. Related methods such as Deformable DETR and Mask2Former have also shown the generalization ability to different scenarios. We wonder if the above response addresses your questions or concerns. If you still have any questions about our response, we would be pleased to discuss them with you.\n"}, {"review_id": "tLcSUymReS", "reviewer": "Reviewer_ihRG", "summary": "This research paper focuses on referring video object segmentation. It combines temporal frame features and language features using a cross attention design. Additionally, it extracts object features from each frame and applies contrastive learning with language representation. The paper presents various designs to enhance the referring performance. Results from experiments on multiple benchmarks demonstrate that this method achieves state-of-the-art performance on this task.\n\n", "questions": "1. Figure 6 shows a video with only one person. Why does the compared method fail in these scenes? Are there any complex scenes that demonstrate the advantages of the present method?\n2. The figures lack clarity and are difficult to understand. It is important to highlight the main contributions of the present modules.\n3. The Semantic Integration Module appears to be resource-intensive, and it would be helpful to present the model parameters compared to previous methods.\n4. The use of off-the-shelf designs in all the modules limits the novelty of the approach.", "limitations": "The limitations are only briefly mentioned in the current manuscript.\n\n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The RVOS framework demonstrates remarkable results on various benchmarks.\n2. Through experimentation, the proposed fusion of features significantly enhances the baseline.\n3. The concept of utilizing temporal and language features is effectively communicated.", "weaknesses": "1. The current method is quite complex, and it is not clear what the main contributions of the RVOS in figure are. It appears that most of the modules shown in Figure 1 are not mentioned in the manuscript. This makes it difficult to understand which proposed components are essential for the given task.\n\n2. The Multi-Modal Fusion (MMF) and Semantic Integration Module (SIM) are presented in Section 3.2 and 3.3. However, there are no experimental tests conducted to validate these modules. The authors also claim that SIM contributes to the task in the introduction.\n\n3. The results of visualizing the impact of the proposed modules are missing. It is important to provide results that demonstrate how each module enhances the referring performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the Reviewer ihRG for his/her review time and valuable comments. To the concerns expressed in Weaknesses and Questions , our response is as follows:\n\n*Weakness-1*: Thanks for your suggestion. The main contribution and motivation of this paper is to understand video content from a global perspective and achieve video-level cross-modal alignment, which helps to handle descriptions about temporal variations as well as improve performance. Therefore, the Semantic Integration Module (SIM), especially the video-level object cluster, is the core component. It's worth noting that the Video Object Cluster (VOC) is a part of SIM, which may be the cause of your confusion. We have updated the framework figure to more clearly illustrate our contribution (shown in the rebuttal pdf). Besides, our SOC is a simple but effective framework. Specifically, the model are composed of feature encoding (encoder), multi-modal interaction (MMF), intra-frame content aggregation (SIM), inter-frame temporal modeling (SIM), and several output heads. Extensive comparisons of parameter, FLOPs, FPS, and performance show that our method outperforms previous state-of-the-art methods. Finally, the modules in Figure 1 have been described in the Section 3.3, *i.e.*, Frame-Level Content Aggregation and Video-Level Object Cluster. We also emphasize that in the updated framework figure.\n\n*Weakness-2*: Thanks for your great advice. Table 3 and Table 4 in the manuscript show the ablation about the VOC (temporal modeling part of SIM) and experiments on the MMF. Here we experiment with the effectiveness of each module on the whole network. \"W/o MMF\" indicates replacing MMF with simple cross attention operation, *i.e.*, the approach used in previous models. We will update these results in the manuscript.\n\n| Model | $\\mathcal{J} $&$ \\mathcal{F}$ | $\\mathcal{J} $ | $\\mathcal{F} $ |\n| :---: | :---: | :---: | :---: |\n| Full Model | 59.2 | 57.8 | 60.5 |\n| W/o SIM | 51.6 | 50.4 | 52.8 |\n| W/o MMF | 58.7 | 57.3 | 60.0 |\n| W/o VL | 58.0 | 56.5 | 59.5 |\n\n\n*Weakness-3*: Great suggestion! We provide the visualization of the impact of each module on the segmentation results and corresponding descriptions in the rebuttal pdf. We will incorporate it into the manuscript.\n\n*Question-1*: Previous methods perform multi-modal interaction in each frame separately, which leads to difficulty in comprehending the text description of temporal variations and actions. With regard to the two examples illustrated in Figure 6, in the first case, since the girl is **sitting** at the beginning, the compared method can not find an object that matches the description of **\"a person is jumping over the river\"** in this frame. For the second case, there are two athletes in the video (possibly requiring zooming in to discern the distant athlete). Due to the limitations of multimodal understanding at the frame level, previous method is unable to understand the temporal notion of \"from right to left\". Consequently, it may only understand other concepts such as \"tennis player is running\" to the best of their ability, and thus resulting in segmentation errors. Furthermore, in the supplementary material and demo, we showcase various of segmentation results in complex video scenarios *e.g.*, similar appearance, occlusion, and large variations, which demonstrates the generalization of our method  in comprehending the textual description about temporal variations.\n\n*Question-2*: Thanks for the suggestion. We will revise the manuscript to make the figures and corresponding descriptions clearer. The main contribution and motivation of this paper is to understand video content from a global perspective and achieve video-level cross-modal alignment, which allows the model can handle text descriptions about temporal variations.\n\n*Question-3*: Thanks for the advice. We perform detailed comparisons ((including model parameters)) in the table below. It can be seen that our SOC has a smaller number of parameters than the previous SOTA method ReferFormer. While MTTR has the lowest FLOPs and the fewest parameters, the lack of elaborate multi-modal fusion and temporal interaction significantly degrades segmentation accuracy.\nIn addition, our Semantic Integration Module is computation-friendly and efficient: the number of parameters and the inference time consumed by SIM are **9.8%** and **8.3%** of the whole model, respectively.\n\n| Model | Backbone | Parameters | $\\mathcal{J} $&$ \\mathcal{F}$ | FPS | FLOPs (a video with 43 frames)|\n|-----|-----|-----|-----|-----|-----|\n| MTTR | Video-Swin-T | 104.88M | 55.3 | 31.8 | 1018G |\n| ReferFormer | Video-Swin-T | 131.67M | 56.0 | 21.2 | 2994G |\n| SOC (Ours) | Video-Swin-T | 131.46M | 59.2 | 32.3 | 1879G |\n\n*Question-4*: The motivation of this work is to unify temporal modeling and cross-modal alignment to achieve video-level understanding. Some network layers such as transformer layer are indeed off-the-shelf. However, we believe that using these network layers to realize our motivation and design an efficient framework to fuse intra- and inter-frame information for video-level multi-modal understanding and improving RVOS task is technically contributory."}, {"review_id": "nBktKuViAp", "reviewer": "Reviewer_d9RC", "summary": "This work studies the referring video object segmentation problem and proposes the semantic-assisted object cluster approach. It aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. ", "questions": "What about considering the object-level temporal relations across frames compared to the video-level ones?", "limitations": "See the above.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) It performs object aggregation and promotes visual-linguistic alignment at the video level, which considers the temporal variations and alignment across different modalities of frames. \n\n2) It introduces the visual-linguistic contrastive learning to provide semantic supervision.\n\n3) Empirical studies on several benchmarks demonstrate the advantage of the proposed method. ", "weaknesses": "1) The novelty seems a bit limited. The multi-modal fusion model just uses the common multi-head cross-attention to model the relations of language and visual information. \n\n2) Is there other clustering methods for object grouping? Some discussions are requried. \n\n3) Ablations studies on the hyper-parameters $\\lambda$ are missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the Reviewer d9RC for his/her comments and the appreciation of our work. To the concerns expressed in Weaknesses and Questions, our response is as follows:\n\n*Weakness-1:* Thanks for the question. MMF is just a component of our framework. It is not our contribution, and it is not mentioned in the contribution note in the introduction section of the paper. Our main contribution lies in the proposed framework, which unifies temporal modeling and cross-modal alignment to achieve video-level visual-linguistic understanding. By fusing intra-frame and inter-frame information, our approach achieves better performance and can handle descriptions of temporal variations. Regarding the MMF module specifically, it serves to activate textual referred objects in visual features and simultaneously update the textual embeddings by incorporating visual content.\n\n*Weakness-2:* Thanks for your valuable question. Table 7 in the manuscript shows the ablation of the structure of the current video-level object cluster (VOC) module.  Here we experiment with several intuitive methods for object grouping. Specifically, \"Average\" means global pooling of all frames-level queries at the corresponding positions. \"1D-Conv\" denotes that we use 1D convolution (kernel size is 5) to process the frame-level query and then pooling them. It can be seen that such methods are underperforming. We attribute this to the absence of sufficient inter-frame interaction. Specifically, although these methods can also yield video-level representations, they lack the perception of temporal variations and the alignment of inter-frame information. As a contrast, our VOC can take advantage of the transformer structure to accomplish inter-frame interactions while clustering. We will incorporate this part into the manuscript.\n\n| VOC Method | Average | K-Means | 1D-Conv | Ours |\n| --- | :---: | :---: | :---: | :---: |\n| $\\mathcal{J} $&$ \\mathcal{F}$| 57.5 | 33.8 | 55.1 | 59.2 |\n\n\n\n*Weakness-3:* Thanks for your question. Here we show the detailed ablation experiments for the loss weight $\\lambda$.\n\n| $\\lambda_{mask}$ | $\\lambda_{box}$ | $\\lambda_{cls}$ | $\\lambda_{con}$ | $\\mathcal{J} $&$ \\mathcal{F}$ |\n| :---: | :---: | :---: | :---: | :---: |\n| 2| 5 | 2 | 1 | 59.2 |\n| 2| 5 | 1 | 1 | 58.3 |\n| 2| 5 | 1 | 2 | 58.5 |\n| 2| 2 | 1 | 1 | 58.6 |\n| 2| 2 | 2 | 2 | 57.9 |\n| 5| 2 | 2 | 1 | 59.1 |\n\n*Question:* Good question! We experiment with performing inter-frame interactions in different time scales (different temporal window size). Results show that larger temporal scale interaction is helpful and the video-level ones performs best. We believe the reason may be that the range of inter-frame interactions required for different textual descriptions is varied. In video-level interactions, the model can recognize and selectively extract information related to the target object, which leads to better performance as well as flexibility.\n\n| Temporal Window Size | $\\mathcal{J} $&$ \\mathcal{F}$ |\n| :---: | :---: |\n| 2 | 57.8 |\n| 4 | 58.2 |\n| 6 | 58.9 |\n| global | 59.2 |\n"}, {"review_id": "7MTKfjtnDc", "reviewer": "Reviewer_9DMq", "summary": "This paper proposes a novel framework that can effectively exploit video-level visual-linguistic alignment to improve the performance of RVOS, filling a gap in existing methods. The extensive experiments on three datasets demonstrate  the effectiveness of SOC.", "questions": "How to handle objects that have been occluded for a short time (> training frames number)", "limitations": "The authors have addressed the limitations and potential negative societal impact of their work.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The proposed method outperforms existing state-of-the-art methods on all datasets.\nThe paper provides a clear and concise description of the SOC framework and its experimental results.", "weaknesses": "None", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the Reviewer 9DMq for his/her comments and the appreciation of our work. Our response to the Reviewer 9DMq is as follows:\n\n*Questions:* Thanks for your question. Sorry we are not sure whether you are referring to the training stage or the inference. Therefore, we explain both of these two stage. During the training stage, if objects are occluded, the associated ground truth masks will also have no labeled area and would not impede the training process. For the inference stage, since we input the entire video and perform multi-modal understanding at the video level, the model is not affected by the number of training frames. During inference, occlusion scenes tend to severely affect the feature representations. But information gleaned from other frames where the occluded object is not obscured aids in localizing and segmenting target in specific frames. Thus, our method also performs well for occlusion scenarios by fusing intra-frame and inter-frame information. Segmentation visualization of the occluded objects by our method is also provided in the supplementary material. We wonder if the above response addresses your questions. If you still have any questions about our response, we would be pleased to discuss them with you."}]

Author rebuttal:
Dear Reviewers and ACs:

We sincerely appreciate all the reviewers. They give positive and high-quality comments on our paper with a lot of constructive feedback. We are glad to be recognized that our motivation is clear and straightforward (R2, R3, R6), the paper is well-written and easy-to-follow (R2, R6), the proposed modules helps understand temporal changes and alignment of different modalities and granularities (R3, R5), the performance is significantly better than existing methods with faster inference speed (R1, R3, R4, R5), and the experimental results are promising (R2).

We respond to each reviewer in the separate rebuttal part. If our response does not fully address the reviewer's questions, we are pleased to discuss further with the reviewers.

Besides, we will revise our manuscript according to all the reviewers' comments. Thank you all for the valuable suggestions.

Thanks,

Paper 4195 Authors.

Meta-review: The submission focuses on the referring video object segmentation (RVOS) task, whose goal is to segment an target object per video frame given a text description. The main motivation of the submission is to construct video-level object-embedding which captures temporal information, and the main contribution claimed by the authors is a unified framework with temporal modeling, spatial understanding, and cross-modal alignment for the RVOS task.

Overall, the reviewers found the submission to be well written, with clearly presented and motivated solutions, and offering "remarkable" performance on various benchmarks. However, they had concerns on the experiment design (e.g. if the number of parameters comparable with baselines), the lack of ablation on the video object clustering modules, and that further clarification is needed when presenting the primary contributions. The authors provided detailed responses to the reviewers, and four out of the five reviewers found their concerns to be effectively addressed, and recommended acceptance. Reviewer iFku had remaining concerns on the details of SIM and comparisons with TSM video encoder, and the AC believes that the authors provided sufficient explanations and supporting experiments to address the remaining concerns.

The AC thus recommends acceptance of the submission, and encourages the authors to incorporate the additional experiments during the rebuttal, and the reviewers' feedback into the final version.
Decision: Accept (poster)