SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Xiaoshuai Hao $^{1}$ , Wanqian Zhang $^{2*}$
$^{1}$ Samsung Research China-Beijing (SRC-B)
$^{2}$ Institute of Information Engineering, Chinese Academy of Sciences
xshuai.hao@samsung.com, zhangwanqian@ie.ac.cn
Video-text retrieval is an important but challenging research task in the multimedia community. In this paper, we address the challenge task of Unsupervised Domain Adaptation Video-text Retrieval (UDAVR), assuming that training (source) data and testing (target) data are from different domains. Previous approaches are mostly derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task. In addition, as to the pairwise misalignment issue in target domain, i.e., no pairwise annotations between target videos and texts, the existing method assumes that a video corresponds to a text. Yet we empirically find that in the real scene, one text usually corresponds to multiple videos and vice versa. To tackle this one-to-many issue, we propose a novel method named Uncertainty-aware Alignment Network (UAN). Specifically, we first introduce the multimodal mutual information module to balance the minimization of domain shift in a smooth manner. To tackle the multimodal uncertainties pairwise misalignment in target domain, we propose the Uncertainty-aware Alignment Mechanism (UAM) to fully exploit the semantic information of both modalities in target domain. Extensive experiments in the context of domain-adaptive video-text retrieval demonstrate that our proposed method consistently outperforms multiple baselines, showing a superior generalization ability for target data.
With the exponential growth of user-generated videos on the Internet, cross-modal retrieval between video data and natural language descriptions, known as video-text retrieval, has attracted much attention. The goal of this task is to retrieve videos which are semantically related to a given text query and vice versa. Generally, the de facto paradigm of video-text retrieval task is to learn high-level video and text embeddings respectively by off-the-shelf feature extractors. To further measure cross-modal semantic similarities between videos and texts, corresponding embeddings should be aligned and projected into a joint embedding space [3, 10, 27, 41, 44, 55, 56, 61], where the semantically-similar texts and videos are much closer to each other and vice versa.
With the advent of deep learning, video-text retrieval has shown promising performance in recent years. Despite the great success, most existing works of this task are supervised, which typically train models on a large number of aligned video-text pairs, making an assumption that training and testing data are drawn from the same distribution. If this assumption does not hold, traditional cross-modal retrieval methods may suffer a performance drop at the evaluation. To alleviate this, recently proposed methods of Unsupervised Domain Adaptation Video-text Retrieval (UDAVR) have attracted much attention, wherein no identical label set for source and target domains exists. By identical we denote that the label set of source data is exactly the same as that of target data. For instance, given a
Figure 1: Illustration of the UDA Video-text Retrieval task, existing works and the proposed method. The proposed method utilizes Distribution-based Vision-Language Domain Adaptation(D-VLDA) in domain gap and uncertainty-aware multimodal alignment mechanism in the target domain.
video-text pair labeled as play basketball in source domain, identical label set requires the existence of target pair which also labeled as it. This can barely be guaranteed due to the diversities between different datasets, such as scenarios, video scenes and lengths. Thus, the only supervision of UDAVR is the semantic relationship in source dataset, i.e., whether one source video and one source text are matched, which is also the general setting in other UDA cross-modal tasks [4, 8, 60, 62].
To alleviate the domain shift problem, recently proposed methods of Unsupervised Domain Adaptation (UDA) has attracted much attention, such as image classification [32, 52], autonomous driving [50, 51] and video-based action recognition [45, 48]. However, these methods are originally designed for classification tasks, which are not suitable for the video-text retrieval task. To that end, several pioneering UDAVR methods have been recently proposed [6, 13, 22, 37]. They try to tackle the lack of target domain supervision by directly minimizing the distribution discrepancy [13], distilling knowledge from the source domain [6], introducing pre-defined prototype assignments [37], or adaptively aligning the video-text pairs which are more likely to be relevant in target domain [22]. Despite the success on improving the retrieval performance of target data, they overlook the challenging one-to-many relationship issue in target domain, i.e., existing methods assume that one text in the target domain corresponds to only one video and vice versa. However, in real scenes, one query text usually corresponds to multiple videos or one query video corresponds to multiple texts.
As illustrated in Figure 1, previous methods are derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task. In addition, for the pairwise misalignment issue in target domain, the existing methods assume that a video corresponds to a text. This will inevitably mix target videos and texts together. Consequently, indiscriminative target video and text features will be introduced, leading to less generalized model and poor performance on target data. This is rather challenging and thus becomes the motivation of our work. In this paper, we propose a novel method named Uncertainty-aware Alignment Network (UAN). We first introduce the cross-modal semantic embedding to generate discriminative source features in a joint embedding space, where semantically relevant pairs should lie close together and vice versa. To alleviate the domain shift, we further utilize the Distribution-based Vision-Language Domain Adaptation(D-VLDA) to minimize of distribution shifts between source and target domains. To tackle the one-to-many in target domain, we propose a simple yet effective Uncertainty-aware Alignment Mechanism (UAM), which fully exploits the semantic information of both modalities in target domain. Extensive experiments on several benchmarks demonstrate the superiority of our method.
The contributions of this paper are mainly threefold:
- For the challenging Unsupervised Domain Adaptation Video-text Retrieval (UDAVR) task, we propose a simple yet effective Uncertainty-aware Alignment Network (UAN), which fully exploits the semantic information of both modalities in target domain.
- To tackle the one-to-many in target domain, the proposed Uncertainty-aware Alignment Mechanism (UAM) tries to utilize the multi-granularity relationships between each target video and text to ensure the discriminability of target features.
- Compared with the state-of-the-art methods, UAN achieves  $15.47\%$  and  $11.57\%$  relative improvements on  $\mathbb{R}@\mathbb{1}$  under the setting of TGIF  $\rightarrow$  MSRVTT and MSRVTT  $\rightarrow$  TGIF respectively, demonstrating the superiority of our method.
Video-Text Retrieval. There are two main types of video-text retrieval approaches: single-modal based and multi-modal based. Initially, single-modal based approaches extend image-text retrieval methods. HGR [7] suggests a Hierarchical Graph Reasoning model that separates text into a hierarchical semantic graph with different levels. Additionally, GPO [5] offers an innovative Generalized Pooling Operator that adapts itself to the best pooling strategy for various baselines. More recently, the widely adopted Contrastive Language-Image Pretraining(CLIP) [2] model is now commonly used in video-text retrieval [1, 18, 20, 30, 34, 36, 43]. CLIP4Clip [42] transfers the knowledge of large-scale image-text pre-training to the video-text retrieval task through fine-tuning and explores three similarity calculation mechanisms based on the pre-trained CLIP model. Similarly, CLIP2video [14] concentrates on the spatial semantics derived from the CLIP model. Another research direction has emphasized capturing multi-modal representations using various experts, which are then combined to derive the final video features. For instance, CE [35] uses video features from all modalities to encode a video, while T2VLAD [54] concentrates on automatically learning text-and-video semantic topics and emphasizing the significance of local semantic alignment between them. On the other hand, MMT [15] utilizes multi-modal information extracted by seven pretrained experts but only fuses them in a straightforward manner without specific guidance. In contrast to these methods, we explore the video-text retrieval task through the lens of unsupervised domain adaptation.
Unsupervised Domain Adaptation for Video-Text Retrieval. UDA transfers knowledge from a labeled source domain to an unlabeled target domain [12, 32, 38, 40, 49], which is realized by training the model with both labeled source data and unlabeled target data. Recently, some cross-modal tasks resort to UDA and try to utilize the unpaired data in target domain, such as image captioning [8, 60, 62] and VQA [4]. DCKT [26] focuses on UDA image-text retrieval and transfers knowledge from a large dataset to promote the model performance on small dataset. However, these UDA methods are originally designed for classification tasks, which might not be suitable for the video-text retrieval. To our knowledge, there are only a few explorations of the Unsupervised Domain Adaptation for Video-Text Retrieval task [6, 13, 22, 37]. MAN [13] proposes a Multi-level Alignment Network to alleviate different gaps in UDAVR task. CAPQ [6] introduces the inaugural benchmark for unsupervised domain adaptation in video-text retrieval. ACP [37] proposes a new adaptive cross-modal prototypes framework to minimizing uni-modal and cross-modal shifting between the source and target domains. DADA [22] introduces the Dual Alignment Consistency (DAC) mechanism adaptively aligns the most similar videos and texts in target domain. Our approach diverges from previous works in two principal ways. (1) Previous approaches are mostly derived from classification based domain adaptation methods to solve domain discrepancy problem, which are neither multi-modal nor suitable for retrieval task. (2) Prior works have not fully leveraged the semantic connections between videos and text in target domains, assuming that a video corresponds to a text. However, in the real scene, one text usually corresponds to multiple videos and vice versa (as illustrated in Figure 3). This problem is the primary motivation for our paper.
Problem Definition. For notational clarity, we first introduce the symbols and definitions used throughout this paper. Formally, assume that we have a set of samples in source domain  $\left\{(\mathcal{V}^s,\mathcal{T}^s) = (v_i^s,t_i^s)_{i = 1}^{n_s}\right\}$ , where  $n_{s}$  indicates the number of video-text pairs. Similarly, there also exists a set of samples in target domain  $\left\{\mathcal{V}^t = \{v_i^t\}_{i = 1}^{n_t},\mathcal{T}^t = \{t_j^t\}_{j = 1}^{n_t}\right\}$  with two collections of  $n_t$  videos  $\mathcal{V}^t$  and texts  $\mathcal{T}^t$ , respectively. Note that the target videos and texts are unpaired, which means the supervised information is not available in target domain. In other words, we have no idea whether one target video-text pair is semantically relevant or not during the training procedure. The objective of Unsupervised Domain Adaptation Video-Text Retrieval (UDAVR) is to enhance the model's ability to generalize to the target domain by leveraging information from the source domain. The overall framework of our method is illustrated in Figure 2.
Semantic Embedding Learning. Using the latest baseline in video-text retrieval, as described in [5], we employ a video encoder  $\varphi(\cdot)$  and a text encoder  $\psi(\cdot)$  to generate a joint embedding space for a given video-text pair, consisting of a video sample  $v$  and a text description  $t$ . For both source and
Figure 2: The overall framework of UAN. Semantic Embedding Learning generates discriminative source features in a joint embedding space  $(\mathcal{L}_S)$ . Distribution-based Vision-Language Domain Adaptation (D-VLDA) is proposed to alleviate the domain discrepancy problem in both modalities  $(\mathcal{L}_D)$ . Uncertainty-Aware Alignment Mechanism (UAM) is proposed to dig in uncertainty-aware multimodal relationships in the target domain  $(\mathcal{L}_U)$ .
target data, we share the same feature extracting modules to map them into the common space. Such a design can be interpreted as to some extent transfer the knowledge learned in the source domain to the target domain. If the text effectively describes the video, the visual embedding  $\varphi(v) \in \mathbb{R}^{M}$  and the text embedding  $\psi(t) \in \mathbb{R}^{M}$  exhibit semantic relevance, with  $M$  representing the dimension in the shared space. In the source domain, we use the symmetric cross-entropy loss to guide the semantic alignment learning. Given a batch of B text-video pairs, the model updates its parameters through maximizing the sum of the main diagonal of a  $\mathrm{B} \times \mathrm{B}$  similarity matrix. Normally, we represent the cross-entropy loss in text-to-video directions in Eq. 1 and represent the cross-entropy loss in video-to-text directions in Eq. 2. The similarity of the text-video is calculated as the inner production of  $t_{i}$  and  $v_{i}$  in Eq.3. The above formula can be formulated as:
$$
\mathcal {L} _ {t 2 v} = - \frac {1}{B} \sum_ {i} ^ {B} \log \frac {\exp (s (t _ {i} , v _ {i}))}{\sum_ {j} ^ {B} \exp (s (t _ {i} , v _ {j}))}, \tag {1}
$$
$$
\mathcal {L} _ {v 2 t} = - \frac {1}{B} \sum_ {i} ^ {B} \log \frac {\exp (s (v _ {i} , t _ {i}))}{\sum_ {j} ^ {B} \exp (s (v _ {i} , t _ {j}))}, \tag {2}
$$
$$
s \left(t _ {i}, v _ {i}\right) = \left\langle t _ {i}, v _ {i} \right\rangle = \left\langle \psi \left(t _ {i}\right) \cdot \varphi \left(v _ {i}\right) \right\rangle , \tag {3}
$$
where  $\psi(t_i)$  and  $\varphi(v_i)$  are the corresponding mapped features. Formally, the symmetric cross-entropy loss  $\mathcal{L}_S$  is the sum of text-to-video loss  $\mathcal{L}_{t2v}$  and video-to-text loss  $\mathcal{L}_{v2t}$ , formulated as:
$$
\mathcal {L} _ {S} = \mathcal {L} _ {t 2 v} + \mathcal {L} _ {v 2 t}. \tag {4}
$$
3.2 Distribution-based Vision-Language Domain Adaptation
In light of the video-text symmetric cross-entropy loss, the representations of unimodal encoders can be effectively aligned, which enables the generation of more informative samples. Thus, we can obtain discriminative and modality-invariant features of source videos and texts in the joint embedding space. However, this discriminability can not generalize well to the target domain, due to the inevitable domain discrepancy problem.
To alleviate this, existing works typically utilize classification domain adaptation (DA) methods to shift the feature distributions in different domains as close as possible for each modality, such as IDM [11] and MMD [38]. However, these methods are originally designed for classification tasks, which is not suitable for the video-text retrieval task. Thus, we propose Distribution-based Vision-Language Domain Adaptation, called D-VLDA, to relieve the divergence of domain statistics, thus the distribution shifts can be significantly diminished, improving the generalization of learned model on out-of-distribution target domain. The basic idea is that the moment calculated on different
domains are the same if the distributions of source and target domains are identical. Specifically, for two Gaussian distributions  $\mathcal{N}(\mu_1,\Sigma_1)$  and  $\mathcal{N}(\mu_2,\Sigma_2)$ , their 2-Wasserstein distance is defined as:
$$
D _ {2 W} = \left\| \mu_ {1} - \mu_ {2} \right\| _ {2} ^ {2} + \operatorname {T r} \left(\Sigma_ {1} + \Sigma_ {2} - 2 \left(\Sigma_ {1} ^ {1 / 2} \Sigma_ {2} \Sigma_ {1} ^ {1 / 2}\right) ^ {1 / 2}\right), \tag {5}
$$
In our modeled distributions,  $\Sigma_{1}$  and  $\Sigma_{2}$  are both diagonal matrices, indicating  $\Sigma_{1}^{1 / 2}\Sigma_{2}\Sigma_{1}^{1 / 2} = \Sigma_{1}\Sigma_{2}$ . The above formula can be rewritten as:
$$
\begin{array}{l} D _ {2 W} = \left\| \mu_ {1} - \mu_ {2} \right\| _ {2} ^ {2} + \operatorname {T r} \left(\left(\Sigma_ {1} ^ {1 / 2} - \Sigma_ {2} ^ {1 / 2}\right) ^ {2}\right), \tag {6} \\ = \left\| \mu_ {1} - \mu_ {2} \right\| _ {2} ^ {2} + \left\| \sigma_ {1} - \sigma_ {2} \right\| _ {2} ^ {2}, \\ \end{array}
$$
where  $\sigma$  refers to a standard deviation vector. In our task, the feature distribution consists of two folds: video domain adaptation module and text domain adaptation module. Therefore, we propose two domain alignment loss functions  $L_{DT}$  and  $L_{DV}$  as follows:
$$
L _ {D T} = a \cdot D _ {2 W} (\mathcal {V} ^ {s}, \mathcal {V} ^ {t}) + b, \tag {7}
$$
$$
L _ {D V} = a \cdot D _ {2 W} \left(\mathcal {T} ^ {s}, \mathcal {T} ^ {t}\right) + b, \tag {8}
$$
$$
L _ {D} = L _ {D T} + L _ {D V}. \tag {9}
$$
where  $a$  is a negative scale factor since similarity is inversely proportional to the distance, and  $b$  is a shift value. Thus, by reducing the distance between two distributions based on Eq. 9, we align the feature distributions of the same modal in different domains.
3.3 Uncertainty-aware Alignment Mechanism
Besides alleviating the domain discrepancy, we also need to bridge the semantic gap between videos and texts in the target domain, which is the key to the standard UDAVR task. Specifically, given the target set  $\left\{\mathcal{V}^t = \{v_i^t\}_{i=1}^{n_t}, \mathcal{T}^t = \{t_j^t\}_{j=1}^{n_t}\right\}$ , pairwise information is not available in the target domain. This means that the target videos and texts are unpaired, and as a result, the issue of one-to-many arises in the target domain. However, existing method [22] assumes that a video corresponds to a text, whereas in the real scene, one text usually corresponds to multiple videos and vice versa (as illustrated in Figure 3). To alleviate this, we propose a simple yet effective Uncertainty-Aware Alignment Mechanism (UAM). The UAM tries to utilize the multi-granularity relationships between each target video and text to guide the unsupervised learning in target dataset. We calculate the similarities  $(S_{v_i^t}^{\mathcal{T} \to \mathcal{V}})$  between a target text  $t_i^t$  and all target videos, and the similarities  $(S_{t_i^t}^{\mathcal{V} \to \mathcal{T}})$  between a target video  $v_i^t$  and all target texts, which can be defined as follows:
$$
\mathbf {S} _ {v _ {i} ^ {t}} ^ {\mathcal {T} \rightarrow \mathcal {V}} = \left[ S _ {t _ {i} ^ {t} v _ {1} ^ {t}}, S _ {t _ {i} ^ {t} v _ {2} ^ {t}}, \dots , S _ {t _ {i} ^ {t} v _ {j} ^ {t}}, \dots , S _ {t _ {i} ^ {t} v _ {n} ^ {t}} \right]. \tag {10}
$$
$$
\mathbf {S} _ {t _ {i} ^ {t}} ^ {\mathcal {V} \rightarrow \mathcal {T}} = \left[ S _ {v _ {i} ^ {t} t _ {1} ^ {t}}, S _ {v _ {i} ^ {t} t _ {2} ^ {t}}, \dots , S _ {v _ {i} ^ {t} t _ {j} ^ {t}}, \dots , S _ {v _ {i} ^ {t} t _ {n t} ^ {t}} \right]. \tag {11}
$$
$$
\mathcal {I} \left(v _ {i ^ {*}} ^ {t}, t _ {j ^ {*}} ^ {t}\right) = \left\{ \begin{array}{l l} 1, & \text {i f} v _ {i ^ {*}} ^ {t} \text {a n d} t _ {j ^ {*}} ^ {t} \text {a r e r e c i p r o c a l T O P - K s i m i l a r ,} \\ 0, & \text {o t h e r w i s e .} \end{array} \right. \tag {12}
$$
Note that this requires  $v_{i*}^{t}$  and  $t_{j*}^{t}$  to be the reciprocal TOP-K similar of each other, indicating a truly aligned (or positive) pair. To that end, we can obtain  $n_p$  positive pairs in one batch from target dataset, denoted as  $\left\{(\mathcal{V}^p,\mathcal{T}^p) = (v_i^p,t_i^p)^{n_p}_{i = 1}\right\}$ , where  $(v_i^p,t_i^p) = (v_{i*}^t,t_{j*}^t)$ . To improve the accuracy of aligned pairs, we introduce  $T$  as a threshold for sorting the similarities of all pairs in a batch in descending order and selecting the  $T$ -th value, which corresponds to the top  $T$  most similar pairs. Our rationale is that a truly positive video-text pair should not only be the most similar to each other but also have a relatively high similarity score compared to all the misaligned pairs. Formally, the uncertainty-aware alignment matching operation of Eq. 12 can be revised as:
$$
\mathcal {I} \left(v _ {i ^ {*}} ^ {t}, t _ {j ^ {*}} ^ {t}\right) = \left\{ \begin{array}{l l} 1, \text {i f} S _ {v _ {i} ^ {p} t _ {i} ^ {p}} \geq S _ {T}, \\ 0, \text {o t h e r w i s e .} \end{array} \right. \tag {13}
$$
where  $S_{T}$  denotes the top  $T$ -th value in one batch and  $S_{T} \in [\mathbf{S}_{v_{i}^{t}}^{\mathcal{V} \rightarrow \mathcal{T}}; \mathbf{S}_{t_{j}^{t}}^{\mathcal{T} \rightarrow \mathcal{V}}]_{des} = [S_{1}, S_{2}, ..., S_{T}, ..., S_{B^{2}}]_{des}$ .  $[\mathbf{S}_{v_{i}^{t}}^{\mathcal{V} \rightarrow \mathcal{T}}; \mathbf{S}_{t_{j}^{t}}^{\mathcal{T} \rightarrow \mathcal{V}}]_{des}$  denotes re-ranking in a descending order (from similar to dissimilar). The combination of uncertainty-aware alignment matching and the similarity
Figure 3: Illustration of Uncertainty-Aware Alignment Mechanism (UAM). If  $v_{i*}^{t}$  and  $t_{j*}^{t}$  to be the reciprocal TOP-K similar of each other, indicating a truly aligned (or positive) pair.
threshold  $T$  contributes to the alignment of target videos and texts in a mutually-promoted manner. By using these self-discovered matching pairs, we can approach the pairwise misalignment issue as a fully supervised problem. We can define the uncertainty-aware alignment loss as:
$$
\begin{array}{l} \mathcal {L} _ {U} = - \frac {1}{B} \left(\sum_ {i} ^ {B} \log \frac {\exp \left(s \left(t _ {i} ^ {p} , v _ {i} ^ {p}\right)\right)}{\sum_ {j} ^ {B} \exp \left(s \left(t _ {i} ^ {p} , v _ {j} ^ {p}\right)\right)} \right. \tag {14} \\ + \sum_ {i} ^ {B} \log \frac {\exp \left(s \left(v _ {i} ^ {p} , t _ {i} ^ {p}\right)\right)}{\sum_ {j} ^ {B} \exp \left(s \left(v _ {i} ^ {p} , t _ {j} ^ {p}\right)\right)}. \\ \end{array}
$$
Therefore, the full training objective is given by:
$$
\mathcal {L} = \mathcal {L} _ {S} + \lambda_ {1} \mathcal {L} _ {D} + \lambda_ {2} \mathcal {L} _ {U}, \tag {15}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are hyper parameters for balancing these terms.
Datasets. We use existing datasets across three domains to explore the UDAVR task, i.e., a comprehensive evaluation benchmark by combining three popular datasets: MSR-VTT (Mt)[59], MSVD (Md)[19], and TGIF (Tf)[33]. We assume that data from different datasets are from different domains, as they often have dissimilar data distributions and representations.
Evaluation Metrics. To evaluate the performance of our video-text retrieval, we utilize standard retrieval metrics, as outlined in previous studies ([7, 21, 23, 24, 25]). Our analysis is based on rank-based metrics, including R@K (the higher, the better) and Median Rank, i.e., MR, (the lower, the better).
Implementation Details. To ensure a fair comparison, we employ the identical architecture for the video and text encoders as used in GPO [5], which is currently the leading baseline for video-text retrieval. The length of shared embedding  $M$  is set to 1024. Moreover, we adopt the Adam optimizer for all our experiments. During the training procedure, we set the batch size to 64, and utilize a step-decayed learning rate with initialization value 0.0001. All experiments are conducted five times for the average performance on a 2080Ti GPU server. For all our experiments, we set  $a$  to -0.005 and  $b$  to 6. The hyper parameters  $\lambda_1$ ,  $\lambda_2$ ,  $K$  and  $T$  of the overall objective function loss is discussed extensively in section 4.3. We set the max epochs as 100, and early stop occurs if the validation performance does not improve in ten consecutive epochs.
4.2 Performance Comparison and Analysis