SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhuoyan Luo $^{1*}$ , Yicheng Xiao $^{1*}$ , Yong Liu $^{1*}$ , Shuyan Li $^{3}$ , Yitong Wang $^{2}$ , Yansong Tang $^{1}$ , Xiu Li $^{1\dagger}$ , Yujiu Yang $^{1\dagger}$
$^{1}$ Tsinghua Shenzhen International Graduate School, Tsinghua University  $^{2}$ ByteDance Inc.
$^{3}$ Engineering Department, University of Cambridge
{luozy23, xiaocc23, liu-yong20}@mails.tsinghua.edu.cn
sl2141@cam.ac.uk wangjingshen@bytedance.com
{tang.yansong, li.xiu, yang.yujiu}@sz.tsinghua.edu.cn
This paper studies referring video object segmentation (RVOS) by boosting video-level visual-linguistic alignment. Recent approaches model the RVOS task as a sequence prediction problem and perform multi-modal interaction as well as segmentation for each frame separately. However, the lack of a global view of video content leads to difficulties in effectively utilizing inter-frame relationships and understanding textual descriptions of object temporal variations. To address this issue, we propose Semantic-assisted Object Cluster (SOC), which aggregates video content and textual guidance for unified temporal modeling and cross-modal alignment. By associating a group of frame-level object embeddings with language tokens, SOC facilitates joint space learning across modalities and time steps. Moreover, we present multi-modal contrastive supervision to help construct well-aligned joint space at the video level. We conduct extensive experiments on popular RVOS benchmarks, and our method outperforms state-of-the-art competitors on all benchmarks by a remarkable margin. Besides, the emphasis on temporal coherence enhances the segmentation stability and adaptability of our method in processing text expressions with temporal variations. Code is available at https://github.com/RobertLuo1/NeurIPS2023_SOC.
Referring Video Object Segmentation (RVOS) [1, 42] aims to segment the target object referred by the given text description in a video. Unlike conventional single-modal segmentation tasks according to pre-defined categories [31, 32] or visual guidance [24-26, 10], referring segmentation requires comprehensive understanding of the content across different modalities to identify and segment the target object accurately. Compared to referring image segmentation, RVOS is even more challenging since the algorithms must also model the temporal relationships of different objects and locations. This emerging topic has attracted great attention and has many potential applications, such as video editing and human-robot interaction.
Due to the varieties of video content as well as the unrestricted language expression, the critical problem of RVOS lies in how to perform pixel-level alignment between different modalities and time steps. To accomplish this challenging task, previous methods have tried various alignment workflows.
(a) Existing Frame-Based Paradigm
(b) Our Video-Centric Paradigm
Figure 1: Illustration of different paradigms. Frame-based methods perform cross-modal interaction and segmentation for each frame individually. In contrast, our method unifies temporal modeling and cross-modal alignment to achieve video-level understanding.
Early approaches [5, 9, 14, 20, 28, 44, 48] take the bottom-up or top-down paradigms to segment each frame separately, while recent works [1, 42] propose to unify cross-modal interaction with pixel-level understanding into transformer structure. Although the above-mentioned approaches have facilitated the alignment between different modalities and achieved excellent performance, they model the RVOS task as a sequence prediction problem and pay little attention to the temporal relationships between different frames. Specifically, they perform cross-modal interaction and segmentation for each frame individually, as illustrated in Fig. 1 (a). The exploitation of temporal guidance relies on the spatial-temporal backbone and manually designed hard assignment strategies [1, 42]. Such paradigms convert the referring video segmentation into stacks of referring image segmentation. While it may be acceptable for such conversion to handle the descriptions of static properties such as the appearance and color of the objects, this approach may lose the perception of target objects for language descriptions expressing temporal variations of objects due to the lack of video-level multi-modal understanding.
To alleviate the above problems and align video with text effectively, we propose Semantic-assisted Object Cluster (SOC) to perform object aggregation and promote visual-linguistic alignment at the video level, as depicted in Fig. 1 (b). Specifically, we design a Semantic Integration Module (SIM) to efficiently aggregate intra-frame and inter-frame information. With a global view of the video content, SIM can facilitate the understanding of temporal variations as well as alignment across different modalities and granularity. Furthermore, we introduce visual-linguistic contrastive learning to provide semantic supervision and guide the establishment of video-level multi-modal joint space. In addition to the remarkable improvements in generic scenarios, these efforts also allow our method to effectively handle text descriptions expressing temporal variations.
We conduct experiments on popular RVOS benchmarks, i.e., Ref-YouTube-VOS [36], Ref-DAVIS [16], A2D-Sentences and JHMDB-Sentences [8], to validate the effectiveness of our method. Results show that SOC notably outperforms existing methods for all benchmarks with faster inference speed. In addition, we provide detailed ablations and analysis on components of our method.
Overall, our contributions are summarized as follows:
- We present a framework called SOC for RVOS to unify temporal modeling and cross-modal alignment. In SOC, a Semantic Integration Module (SIM) is designed to efficiently aggregate inter and intra-frame information, which achieves video-level multi-modal understanding.
- We introduce a visual-linguistic contrastive loss to apply semantic supervision on video-level object representations, resulting in well-aligned multi-modal joint space.
- Without bells and whistles, our method outperforms existing state-of-the-art method ReferFormer [42] by a remarkable margin, e.g.,  $+3.0\%$ $\mathcal{J}\& \mathcal{F}$  on Ref-YouTube-VOS and  $+3.8\%$ $\mathcal{J}\& \mathcal{F}$  on Ref-DAVIS under fair comparison. Besides, our method runs at 32.3 FPS on single 3090 GPU, which is significantly faster than the 21.4 FPS of ReferFormer.
Referring Image Segmentation Referring Image Segmentation (RIS) aims to localize the corresponding object referred by a text description within a static image. It is first introduced by Hu et
Figure 2: Overview of SOC. The model takes a video clip with corresponding language descriptions as input. After the encoding process, the multi-modal fusion (MMF) module performs bidirectional fusion to build the intrinsic feature relations. Then we design a Semantic Integration Module to efficiently aggregate intra-frame and inter-frame information. Meanwhile, we introduce a visual-linguistic contrastive loss to benefit the establishment of video-level multi-modal space. Finally, the prediction heads decode the condensed embeddings and output segmentation masks. The transparent arrows illustrate the pipeline of our model and label the input and output.
al. [12], who develops a simple framework that utilizes Fully Convolution Network (FCN) [37] to generate segmentation masks from concatenated visual and linguistic features. To deeply explore the intrinsic correlations among different modal features, several studies [3, 13, 41, 46] design various attention modules for modality interaction. Additionally, VLT [4] proposes a transformer-based architecture for the RIS task, which has gained more popularity than FCN-based approaches. LAVT [45] incorporates early alignment of visual and linguistic features at the intermediate layers of encoders. PolyFormer [21] further uses transformer to generate polygon vertices as the prior information to refine segmentation masks, leading to better results.
Referring Video Object Segmentation Compared to RIS, RVOS is more challenging since both the action and appearance of the referred object are required to be segmented in a dynamic video. Gavri-lyuk et al. [8] first proposes the Referring Video Object Segmentation (RVOS) task. URVOS [36] introduces a large-scale RVOS benchmark and a unified framework that leverages attention mechanisms and mask propagation to increase the task's complexity and scope. ACAN [40] designs an asymmetric cross-guided attention network to establish complex visual-linguistic relationships. To improve positional relation representations in the text, PRPE [35] explores a positional encoding mechanism based on the polar coordinate system. In addition, most previous approaches [19, 22, 39, 47, 48, 30] rely on complicated pipelines. To simplify the workflow, MTTR [1] and ReferFormer [42] adopt query-based end-to-end frameworks for decoding objects from multi-modal features, achieving excellent performance. However, during the decoding phase, previous methods only concentrate on intra-frame object information, disregarding the valuable temporal context of objects across frames. To address this issue, we propose to associate object temporal context with language tokens and achieve video-level multi-modal understanding.
Given  $T$  frames of video clip  $\mathcal{I} = \{I_t\}_{t=1}^T$ , where  $I_t \in \mathbb{R}^{3 \times H_0 \times W_0}$  and a referring text expression  $\mathcal{E} = \{e_i\}_{i=1}^L$ , where  $e_i$  denotes the i-th word in the text. Our goal is to generate a series of binary
segmentation masks  $\mathcal{S} = \{s_t\}_{t=1}^T$ ,  $s_t \in \mathbb{R}^{1 \times H_0 \times W_0}$  of the referred object. To this end, we propose a video-centric framework called Semantic-assisted Object Cluster (SOC). We will elaborate on it in the following sections.
3.1 Visual and Linguistic Encoding
Visual Encoder Taking a video clip  $\mathcal{I}$  as input, we utilize a spatial-temporal backbone such as Video Swin Transformer [27] to extract hierarchical vision features. Consequently, the video clip is encoded into a set of feature maps  $\mathcal{F}_i^v\in \mathrm{R}^{C_i\times H_i\times W_i},i\in \{1,2,3,4\}$ . Here  $H_{i}$  and  $W_{i}$  denote the height and width of each scale feature map, respectively.  $C$  denotes the channel dimension.
Language Encoder Simultaneously, a transformer-based [38] language encoder encodes the given textual expression  $\mathcal{E}$  to a word-level embedding  $\mathcal{F}^w\in \mathbb{R}^{L\times C_t}$  and a sentence-level embedding  $\mathcal{F}^s\in \mathbb{R}^{1\times C_t}$ . The word embedding  $\mathcal{F}^w$  contains fine-grained description information. On the other hand, the sentence embedding  $\mathcal{F}^s$  expresses the general characteristics of the referred target object.
3.2 Two Stream Multi-Modal Fusion
Having the separate visual and linguistic embedding encoded from the video clip and text expression, we design a Multi-Modal Fusion module called MMF to perform preliminary cross-modal alignment. As shown in Fig. 3, MMF is a two-stream structure. The language-to-vision (L2V) stream aims to highlight the corresponding regions of the referred object in each frame and mitigate the effect of background noise. It leverages linguistic information as guidance and addresses the potential similar visual areas. Meanwhile, a vision-to-language (V2L) stream is designed to update the textual embedding with image content, which helps to relieve the potential ambiguity of unconstrained descriptions. Specifically, we measure the relevance of all visual areas to the text query and assign weights to the useful information extracted from the visual features so as to reorganize the text embedding. The above L2V and V2L fusion process are based on the multi-head cross-attention mechanism, which can be formulated as:
$$
\operatorname {M H A} \left(\mathcal {X}, \mathcal {Y}\right) = \operatorname {C o n c a t} \left(\operatorname {h e a d} _ {1} \left(\mathcal {X}, \mathcal {Y}\right), \dots , \operatorname {h e a d} _ {h} \left(\mathcal {X}, \mathcal {Y}\right)\right) W,
$$
$$
\operatorname {h e a d} _ {j} (\mathcal {X}, \mathcal {Y}) = \operatorname {s o f t m a x} \left(\frac {\left(\mathcal {X} W _ {j} ^ {Q}\right) ^ {T} \mathcal {Y} W _ {j} ^ {K}}{\sqrt {C}}\right) \mathcal {Y} W _ {j} ^ {V}, \tag {1}
$$
where MHA  $(\cdot)$  stands for multi-head attention.  $W, W_{j}^{Q}, W_{j}^{K}, W_{j}^{V}$  are learnable weights used to map the input to the attention space.
Since visual features of different scales contain diverse content information, MMF is designed to produce a series of coarse-to-fine visual and textual feature maps  $\{\mathcal{F}_i^{vf}\}$  and  $\{\mathcal{F}_i^{ef}\}$ ,  $i\in \{2,3,4\}$ . Specifically, we leverage shared parameters to perform multi-head cross-attention operations on  $\{\mathcal{F}_i^v\}$ ,  $i\in \{2,3,4\}$  and  $\mathcal{F}^w$ . Take  $\mathcal{F}_2^v$  as an example. Firstly, we utilize  $1\times 1$  convolution and fully connected layers to transform the visual and linguistic embeddings into joint space, respectively. Then the bidirectional multi-modal fusion is applied to align information from different modalities as well as enhance the feature representation. The fusion process is:
$$
\mathcal {F} _ {2} ^ {v f} = \operatorname {M H A} \left(\mathcal {F} _ {2} ^ {v}, \mathcal {F} ^ {w}\right) \cdot \mathcal {F} _ {2} ^ {v}, \tag {2}
$$
$$
\mathcal {F} _ {2} ^ {e f} = \operatorname {M H A} \left(\mathcal {F} ^ {w}, \mathcal {F} _ {2} ^ {v}\right) \cdot \mathcal {F} ^ {w}, \tag {3}
$$
where  $\mathcal{F}_2^v\in \mathbb{R}^{TH_2W_2\times D}$  and  $\mathcal{F}^w\in \mathbb{R}^{L\times D}$  are embeddings projected by convolution and fully connected layers. Similar to Eq. (2) and Eq. (3), the coarse-to-fine aligned visual features  $\{\mathcal{F}_i^{vf}\}$  and textual embeddings  $\{\mathcal{F}_i^{ef}\}$ ,  $i\in \{2,3,4\}$  are produced.
3.3 Semantic Integration Module
After aligning cross-modal information and activating the potential target region in MMF, we design a Semantic Integration Module (SIM) to incorporate visual content and generate compact target representations. Specifically, we first leverage frame-level content aggregation to locate objects separately for each frame. Then we associate inter-frame dependency and model the temporal relationship of objects via video-level object cluster.
Figure 3: The structure of proposed Multi-Modal Fusion module (MMF) and Video-level Object Cluster module (VOC).
Frame-Level Content Aggregation Having the activated visual features  $\{\mathcal{F}_i^{vf}\}$  from MMF, we utilize a transformer-based structure to locate objects in each frame. Firstly,  $K$  stacks of deformable transformer encoder layer [49] are leveraged to capture intra-frame relationships and further excavate multi-modal interactions inside  $\{\mathcal{F}_i^{vf}\}$ . This process can be formulated as:
$$
\mathcal {F} _ {i} ^ {\prime v f} = \left\{\operatorname {D e c f o r m E n c} _ {k} \left(f _ {t} ^ {v f}\right) \right\} _ {t = 1} ^ {T}, \tag {4}
$$
where  $f_{t}^{vf}$  denotes the activated visual features of the t-th frame. Then, a set of learnable object queries [2, 49, 34] is introduced to aggregate image content and highlight potential target objects. Following [42], these object queries fully interact with  $\mathcal{F}_{i}^{\prime v f}$  in deformable transformer decoder through cross-attention mechanism. After extracting different object representations, these object queries are turned into instance embeddings  $\mathcal{O}^{f}\in \mathbb{R}^{T\times N_{q}\times D}$ . Note that we set up  $N_{q}$  object queries to represent instances of each frame in a video clip so there are  $N_{q}T$  output object queries in total.
Video-Level Object Cluster Instances typically vary in pose and location between frames and even being obscured. Previous methods [1, 5, 6, 42] only model instances separately for each frame, disregarding the temporal relationship and continuous motion of instances. Although such an approach can cover simple scenarios such as descriptions of target appearance, the lack of inter-frame interaction makes existing methods inefficient for the description of temporal relationships. To address the aforementioned weakness, inspired by [7, 11, 18], we design a video-level object cluster to capture the temporal information of instances between frames.
As shown in Fig. 3, after instances embedding  $\mathcal{O}^f$  are formulated by the frame-level content aggregation, we flatten it into  $\mathcal{O}^{f'}\in \mathbb{R}^{TN_q\times D}$  and employ self-attention mechanism along the temporal axis to introduce inter-frame interaction. Furthermore, we find that only introducing temporal self-attention is inferior. Simply sharing temporal object context may create redundancy due to the similarity of the representation referring to the same object in different frames, which potentially affects the model's precise understanding of objects in the video clip. To this end, we employ an object grouping decoder to perform object clustering and group the same object into video-level compact representations across frames. Specifically, we introduce  $N_{v}$  video-level object queries  $\mathcal{O}^v\in \mathbb{R}^{N_v\times D}$  and they are initialized by linguistic sentence-level feature  $\mathcal{F}^s$ , which helps to promote the understanding of object descriptions and build the visual-linguistic joint space. By performing interaction between linguistic-aware video queries and condensed instance information of each frame, the object grouping decoder can effectively capture temporal object contexts and group the referred object queries across frames to output the clustered video-level object queries  $\mathcal{O}^v$ . While temporal connections can provide a lot of valuable information, the segmentation of each frame is dependent on the specific image content. Thus, we incorporate the high-level video instance information into each frame to integrate the advantages of both. In detail, as illustrated in Fig. 2, we repeat the video-level object queries  $T$  times  $\mathcal{O}^{\prime v}\in \mathbb{R}^{T\times N_v\times D}$  and enhance the representation of frame-level object queries
by element-wise sum:  $\mathcal{O}^f = \mathcal{O}^f +\mathcal{O}'^v$  . In this way, the semantic of frame-level object queries is greatly enriched by the supplement of video-level object information.
3.4 Visual-Linguistic Contrastive Learning
Although the aggregated video-level representation can better describe object states, the simple exploitation of textual priors may lead to non-target potential responses for video-level embeddings [29]. Meanwhile, taking unrestricted expressions as aggregation guidance may introduce undesirable inductive bias. To address the issues, we present a visual-linguistic contrastive loss that explicitly focuses on the alignment between video-level features and textual information. The loss accompanies two purposes: (1) Bridge the semantic gap between textual expressions and the corresponding object queries in the multi-modal joint embedding space. (2) Mitigate the bias caused by unrestricted textual features and emphasize the referred object representations. As shown in Fig. 2, we transform the last stage of fused textual features  $\mathcal{F}_4^{ef}$  from the vision-to-language stream (see Eq. (3)) to generate the textual guidance embedding  $\mathcal{F}_{gud}$ :
$$
\mathcal {F} _ {g u d} = \text {A v e r a g e P o o l i n g} \left(\mathcal {F} _ {4} ^ {e f}\right), \mathcal {F} _ {g u d} \in \mathbb {R} ^ {D}. \tag {5}
$$
Then, we measure the similarity between the textual guidance embedding and video-level object queries by scaled dot product similarity:  $\hat{y}_{sim} = \frac{\mathcal{O}^v\mathcal{F}_{gud}^\top}{\sqrt{D}}$ . To suppress undesired region response and highlight target object, we take softmax operation along the object query axis. Finally, we compute the contrastive loss  $\mathcal{L}_{con}$  by matrix multiplication of  $\hat{y}_{sim}$  and  $y_{\tau}$ , where  $y_{\tau}$  is annotated to 1 for the best predicted trajectory and 0 for others [43]:
$$
\mathcal {L} _ {\text {c o n}} = - \operatorname {L o g S o f t m a x} \left(\hat {y} _ {\text {s i m}}\right) \cdot y _ {\tau}. \tag {6}
$$
3.5 Instance Segmentation and Loss
Prediction Heads As depicted in Fig. 2, there are three lightweight heads built on top of the semantic integration module. The classification head is a concatenation of three fully connected layers. It directly predicts class probability  $\hat{p} \in \mathbb{R}^{T \times N_q \times (K + 1)}$  for frame-level object queries, where  $K$  is the number of classes. Note that if  $K = 0$ , the role of the classification head is to judge whether the object is referred by the text description. The box head comprises three sequential linear layers, which is designed to transform the object queries to normalized bounding box information  $\hat{b} \in \mathbb{R}^{T \times N_q \times 4}$ , i.e., center coordinates, width, and height. Similar to [42], we adopt dynamic convolution to output segmentation masks for each frame. Specifically, the mask head produces weights of  $N_q$  dynamic kernels  $\Omega = \{\omega_n\}_{n=1}^{N_q}$ . Meanwhile, the encoded multi-scale cross-modal features  $\{\mathcal{F}_i^{vf}\}$  are stacked with the  $4 \times$  features from the visual backbone to form hierarchical decoding features. An FPN structure takes the hierarchical features as input and outputs high-resolution semantic-aware features  $\mathcal{F}_{seg} \in \mathbb{R}^{T \times \frac{H_0}{4} \times \frac{W_0}{4} \times D}$ . Finally, the dynamic kernels process the features  $\mathcal{F}_{seg}$  based on the position information output from the box head and obtain the segmentation mask  $\hat{m} \in \mathbb{R}^{T \times N_q \times \frac{H_0}{4} \times \frac{W_0}{4}}$ , which can be formulated as:
$$
\hat {m} _ {n} = \left\{\hat {f} _ {n} ^ {\text {s e g}} * \omega_ {i} \right\} _ {n = 1} ^ {N _ {q}}. \tag {7}
$$
Instance Matching As described above, the prediction heads output the prediction trajectories of  $N_{q}$  objects, denoted as  $\hat{y} = \{\hat{y}_n\}$  and the n-th object trajectory prediction is:
$$
\hat {y} _ {n} = \{\hat {p} _ {n} ^ {t}, \hat {b} _ {n} ^ {t}, \hat {m} _ {n} ^ {t} \} _ {t} ^ {T}. \tag {8}
$$
There is only one referred object in a video clip corresponding to the text descriptions. Therefore, we denote the ground truth object sequence as  $y = \{p^t, b^t, m^t\}_t^T$  and search the best prediction trajectory  $\hat{y}_{\sigma}$  via Hungarian algorithm [17].
Total Loss We supervise the trajectory prediction  $\hat{y}_{\sigma}$  by four types of losses: (1)  $\mathcal{L}_{mask}(y, \hat{y}_{\sigma})$ , the mask loss is a combination of Dice loss and binary focal loss, which is computed across frames. (2)  $\mathcal{L}_{box}(y, \hat{y}_{\sigma})$ : the box loss aggregates the L1 loss and GIoU loss per-frame. (3)  $\mathcal{L}_{cls}(y, \hat{y}_{\sigma})$ : the class
Method	Backbone	Ref-YouTube-VOS	Ref-DAVIS17
J&F	J	F	J&F	J	F
URVOS [36]	ResNet-50	47.2	45.3	49.2	51.5	47.3	56.0
LBDT-4 [6]	ResNet-50	49.4	48.2	50.6	-	-	-
MTTR [1]	Video-Swin-T	55.3	54.0	56.6	-	-	-
ReferFormer [42]	Video-Swin-T	56.0	54.8	57.3	-	-	-
SOC (Ours)	Video-Swin-T	59.2	57.8	60.5	59.0	55.4	62.6
With Image Pretrain
ReferFormer [42]	Video-Swin-T	59.4	58.0	60.9	59.7	56.6	62.8
ReferFormer [42]	Video-Swin-B	62.9	61.3	64.6	61.1	58.1	64.1
VLT [5]	Video-Swin-B	63.8	61.9	65.6	61.6	58.9	64.3
SOC (Ours)	Video-Swin-T	62.4	61.1	63.7	63.5	60.2	66.7
SOC (Ours)	Video-Swin-B	66.0	64.1	67.9	64.2	61.0	67.4
Joint Train
ReferFormer	Video-Swin-T	62.6	59.9	63.3	-	-	-
ReferFormer	Video-Swin-B	64.9	62.8	67.0	-	-	-
SOC (Ours)	Video-Swin-T	65.0	63.3	66.7	64.2	60.9	67.5
SOC (Ours)	Video-Swin-B	67.3	65.3	69.3	65.8	62.5	69.1
Table 1: Comparison with the state-of-the-art methods on Ref-YouTube-VOS and Ref-DAVIS17 datasets. With Image Pretrain denotes the models are first pretrained on RefCOCO [15], RefCOCO+ [15], and RefCOCOg [33] datasets. Joint Train indicates the models are trained with the combination of image datasets and video datasets.