SlowGuess/NeurIPS_2023_OCR huggingface dataset:
loss is focal loss and supervises the predicted object category. (4)  $\mathcal{L}_{con}(y_{\tau},\hat{y}_{sim})$ : visual-linguistic contrastive loss. The total loss can be formulated as:
$$
\mathcal {L} _ {\text {t o t a l}} = \lambda_ {\text {m a s k}} \mathcal {L} _ {\text {m a s k}} + \lambda_ {\text {b o x}} \mathcal {L} _ {\text {b o x}} + \lambda_ {\text {c l s}} \mathcal {L} _ {\text {c l s}} + \lambda_ {\text {c o n}} \mathcal {L} _ {\text {c o n}}, \tag {9}
$$
where  $\lambda$  is the scale factor to balance each loss.
Datasets. We evaluate our model on four prevalent RVOS benchmarks: Ref-YouTube-VOS [36], Ref-DAVIS17 [16], A2D-Sentences, and JHMDB-Sentences [8]. For detailed descriptions of the datasets please see the supplementary material.
Metrics. Following [1, 42], we measure the effectiveness of our model by criteria of Precision@K, Overall IoU, MeanIoU and MAP over 0.50:0.05:0.95 for A2D-Sentences and JHMDB-Sentences. Meanwhile, we adopt standard evaluation metrics: region similarity  $(\mathcal{I})$ , contour accuracy  $(\mathcal{F})$  and their average value  $(\mathcal{J}\& \mathcal{F})$  on Ref-YouTube-VOS and Ref-DAVIS17.
We take the pretrained Video Swin Transformer [27] and RoBERTa [23] as our encoder in default. Both the frame aggregation and object cluster parts of SIM consist of three encoder and decoder layers. The number of frame-level queries  $\mathcal{O}^f$  and video-level queries  $\mathcal{O}^v$  are set as 20 in default. We feed the model windows of  $w = 8$  frames during training. The models are trained with eight 32GB V100 GPUs in default. The coefficients for losses are set as  $\lambda_{cls} = 2$ ,  $\lambda_{L1} = 2$ ,  $\lambda_{giou} = 2$ ,  $\lambda_{dice} = 2$ ,  $\lambda_{focal} = 5$ ,  $\lambda_{con} = 1$ . Due to space limitations, please see the supplementary materials for more training details.
Ref-YouTube-VOS & Ref-DAVIS17. We compare our method to previous models on Ref-YouTube-VOS and Ref-DAVIS17 in Table 1. With video-level multi-modal understanding, our SOC achieves new state-of-the-art performance among different training settings: train from scratch, with image pretrain, and joint train. Without bells and whistles, our approach outperforms existing SOTA by about  $3\%$ $\mathcal{J}\& \mathcal{F}$  under fair comparison. On Ref-DAVIS17, we directly report the results using the model trained on Ref-YouTube-VOS without finetune.
Method	Backbone	Precision	IoU	mAP
P@0.5	P@0.6	P@0.7	P@0.8	P@0.9	Overall	Mean
Hu et al. [12]	VGG-16	34.8	23.6	13.3	3.3	0.1	47.4	35.0	13.2
Gavrilyuk et al. [8]	I3D	47.5	34.7	21.1	8.0	0.2	53.6	42.1	19.8
CMSA + CFSA [47]	ResNet-101	48.7	43.1	35.8	23.1	5.2	61.8	43.2	-
ACAN [40]	I3D	55.7	45.9	31.9	16.0	2.0	60.1	49.0	27.4
CMPC-V [22]	I3D	65.5	59.2	50.6	34.2	9.8	65.3	57.3	40.4
ClawCraneNet [19]	ResNet-50/101	70.4	67.7	61.7	48.9	17.1	63.1	59.9	-
MTTR [1]	Video-Swin-T	75.4	71.2	63.8	48.5	16.9	72.0	64.0	46.1
ReferFormer [42]	Video-Swin-T	76.0	72.2	65.4	49.8	17.9	72.3	64.1	48.6
SOC (Ours)	Video-Swin-T	79.0	75.6	68.7	53.5	19.5	74.7	66.9	50.4
With Image Pretrain
ReferFormer [42]	Video-Swin-T	82.8	79.2	72.3	55.3	19.3	77.6	69.6	52.8
ReferFormer [42]	Video-Swin-B	83.1	80.4	74.1	57.9	21.2	78.6	70.3	55.0
SOC (Ours)	Video-Swin-T	83.1	80.6	73.9	57.7	21.8	78.3	70.6	54.8
SOC (Ours)	Video-Swin-B	85.1	82.7	76.5	60.7	25.2	80.7	72.5	57.3
Table 2: Comparison with the state-of-the-art methods on A2D-Sentences.
VOC	VL	J&F	J	F
55.6	54.3	56.9
✓	58.0	56.5	59.6
✓	57.1	55.8	58.4
✓	✓	59.2	57.8	60.5
Table 3: Effectiveness of our proposed modules.
Fusion Strategy	J&F	J	F
No Fusion	38.3	36.1	40.6
V2L	39.8	37.6	42.0
L2V	58.7	57.3	60.0
Both	59.2	57.8	60.5
Table 4: Impact of different fusion strategies.
VOC Method	J&F	J	F
K-Means	33.8	32.9	34.7
1D-Conv	55.1	53.8	56.4
Average	57.5	56.1	58.9
Ours	59.2	57.8	60.5
Table 5: Impact of different cluster methods.
Query Num.	J&F	J	F
10	58.0	56.6	59.5
15	58.1	56.7	59.6
20	59.2	57.8	60.5
25	58.5	57.2	59.8
Table 6: Results of different object query numbers.
Frame Num.	J&F	J	F
3	57.9	56.5	59.3
5	58.4	57.1	59.7
8	59.2	57.8	60.5
10	59.1	57.8	60.3
Table 7: Results of different training frame numbers.
VOC Structure	J&F	J	F
None	57.1	55.8	58.4
only encoder	58.3	56.9	59.8
only decoder	58.1	56.7	59.5
Both	59.2	57.8	60.5
Table 8: The structure design of the VOC module.
A2D-Sentences & JHMDB-Sentences. As shown in Table 2, compared with the existing SOTA method ReferFormer [42], our SOC achieves  $50.4\%$  mAP and  $66.9\%$  mIoU with the model trained from scratch, which gains a clear improvement of  $1.8\%$  mAP and  $2.8\%$  mIoU respectively. Furthermore, with image data pretraining, our method achieves new state-of-the-art on all metrics, e.g.,  $+4.0\%$  P@0.9,  $+2.2\%$  mIoU, and  $+2.3\%$  mAP. Due to the similarity of the datasets and space limitations, please see the supplementary material for the comparison on JHMDB-Sentences [8]. Our SOC also surpasses all existing methods on it.
In this section, we conduct analysis experiments on the Ref-YouTube-VOS [36] benchmark using Video-Swin-Tiny as the visual backbone and train the model from scratch.
Component Analysis We experiment with the effectiveness of the proposed Video-level Object Cluster (VOC) and Visual-linguistic contrastive Learning (VL). As shown in Table 3, both of them bring performance improvement and their combination works better  $(+3.6\%)$ $\mathcal{J}\& \mathcal{F})$  , which shows the significance of video-level multi-modal understanding.
Fusion Strategy We investigate the effects of various fusion strategies. As illustrated in Table 4, the absence of Language to Vision fusion (L2V) yields significant deterioration. This is because the frame-level content aggregation without textual guidance would introduce background noise. If only utilizing the L2V fusion, the performance drops  $0.5\%$ $\mathcal{J}\& \mathcal{F}$ , demonstrating that visual content helps filter out irrelevant words from unconstrained descriptions.
VOC Method The superiority of the current VOC module derives from two aspects: i) Powerful representation learning ability of the transformer structure. Compared to other clustering methods
the giraffe is on the right side of the railing and has its head hanging over and eating grass
a skateboardboard is being rode by a personwearing a red hat and jumping in the air
Figure 5: Segmentation results of our SOC. Best viewed in color.
a black knife holder
a person skateboarding
a person is jumping over the river
a tennis player is running from right to left
a tennis player is running from right to left
Figure 6: Visualization comparison using text expressions about temporal actions. (a) and (b) are segmentation results of our SOC and ReferFormer [42], respectively.
shown in Table 5, the transformer architecture has a better perception as well as selection of global information, which helps to associate object features across frames. ii) Sufficient inter-frame interactions. Although other methods can also yield video-level representations, they lack the perception of temporal variations and the alignment of inter-frame information. As a contrast, our VOC performs adequate inter-frame interaction while clustering.
Object Query Number Although only one referred object is involved in a video, focusing on more potential regions is helpful due to the various video content. Table 6 shows that an increased number of object queries allows the model to effectively group the relevant objects from the pool of candidate regions. The performance saturates when the query number is increased to a certain extent. We believe that excessive queries may cause misunderstandings when grouping objects across frames.
Segmentation Stability Benefiting from video-level multimodal understanding, our method yields more robust segmentation results compared to existing approaches. To quantify the temporal segmentation stability, we calculate the IoU and  $\mathcal{J}\&\mathcal{F}$  variance between each frame in a video and count the average value on Ref-YouTube-VOS [36] dataset. Results are shown in Fig. 4 (the lower is better). It can be seen that with video object cluster module and visual-linguistic contrastive loss building the video-level multi-modal aligned space, our SOC can understand the overall video content better and output
Figure 4: IoU and  $\mathcal{J}\&\mathcal{F}$  variance.
temporally stable segmentation results. More qualitative analysis of temporal consistency is shown in the supplementary material.
Frame Number Table 7 shows results with different training clip frame numbers. As expected, widening temporal context leads to performance gains yet causes expensive computation cost. For a fair comparison with existing methods, we take the frame number of 8 by default.
Variants of VOC Structure Results in Table 8 illustrate the rationality of the structure of the VOC module. Without sufficient inter-frame interaction by the encoder or aggregation of target object information by the decoder, the segmentation performance will be degraded.
Inference Speed In addition to superior segmentation performance, our method also achieves real-time inference. Specifically, our SOC runs at 32.3 FPS on single 3090 GPU, which significantly exceeds the existing SOTA method ReferFormer [42] (21.4 FPS).
the person on the bike, with a yellow hoodie and black pants, to the right
(a)
(b)
Figure 7: Visualization results of effectiveness on VOC. The object in the red box is the target. (a) displays the segmentation result of the completed model. While (b) shows the result of the model without VOC (Video Object Cluster in SIM).
Fig. 5 demonstrates the effectiveness of SOC for complex scenarios segmentation, i.e., similar appearance, occlusion, and large variations. To further prove the adaptability of our SOC for understanding text expressions about temporal action and variations, we design corresponding text descriptions that depict changed states of the object in temporal. As shown in Fig. 6, the global view enables SOC to understand such text and segment the target object accurately. In contrast, ReferFormer [42] only recognizes the object in specific frames mentioned in the text descriptions and fails to comprehend the content. More comparisons can be seen in the supplementary material. Fig. 7 indicates that due to the temporal modeling and inter-frame interaction in VOC, the full model comprehends text descriptions of temporal variations and tracks the target in coherence. In contrast, without VOC, the model struggles to understand the temporal relationship and fails to segment the object.
In this paper, we propose a framework called SOC for RVOS to achieve video-level multi-modal understanding. By associating frame-level object embeddings with language tokens, we unify temporal modeling and cross-modal alignment into a simple architecture. Moreover, visual-linguistic contrastive learning is introduced to build the video-level multi-modal space. Extensive experiments show that our SOC remarkably outperforms existing state-of-the-art methods. Besides, video-level understanding also allows our SOC to handle text descriptions expressing temporal variations better.
Limitations. The proposed framework, SOC, has achieved video-level multi-modal understanding and excellent performance. However, there are some potential limitations, e.g., the current architecture cannot handle infinitely long videos. We think that devising explicit designs for long videos with complex scenarios is an interesting future direction.
This work was supported by the National Natural Science Foundation of China (Grant No. U1903213) and the Shenzhen Science and Technology Program (JSGG20220831110203007). This work was also supported by the National Key Research and Development Program of China under Grant 2020AAA0108302 and Shenzhen Stable Supporting Program (WDZC20200820200655001).
[1] Botach, A., Zheltonozhskii, E., Baskin, C.: End-to-end referring video object segmentation with multimodal transformers. In: CVPR. pp. 4975-4985 (2022) 1, 2, 3, 5, 7, 8
a horse running to the left of another horse towards the water
[43] Xiao, Y., Ma, Y., Li, S., Zhou, H., Liao, R., Li, X.: Semanticac: Semantics-assisted framework for audio classification. In: ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pp. 1-5 (2023) 6
[44] Xu, M., Zhang, Z., Wei, F., Lin, Y., Cao, Y., Hu, H., Bai, X.: A simple baseline for open-vocabulary semantic segmentation with pre-trained vision-language model. In: ECCV. pp. 736-753 (2022) 2
[45] Yang, Z., Wang, J., Tang, Y., Chen, K., Zhao, H., Torr, P.H.S.: Lavt: Language-aware vision transformer for referring image segmentation. In: CVPR. pp. 18134-18144 (2022) 3
[46] Ye, L., Rochan, M., Liu, Z., Wang, Y.: Cross-modal self-attention network for referring image segmentation. In: CVPR. pp. 10502–10511 (2019) 3
[47] Ye, L., Rochan, M., Liu, Z., Zhang, X., Wang, Y.: Referring segmentation in images and videos with cross-modal self-attention network. TPAMI pp. 3719–3732 (2022) 3, 8
[48] Zhang, Y., Yuan, L., Guo, Y., He, Z., Huang, I., Lee, H.: Discriminative bimodal networks for visual localization and detection with natural language queries. In: CVPR. pp. 1090–1099 (2017) 2, 3
[49] Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable DETR: deformable transformers for end-to-end object detection. In: ICLR (2021) 5