SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We compare our method with several state-of-the-art baselines across three categories, i.e., Source Only, DA methods and UDAVR methods. As a lower bound, we include the non-adapted Source Only results, which directly applies the model trained on the source domain to the target domain. We also implement five classification-based (i.e. typical) DA methods and modify them for the UDAVR task, including MMD [38], CORAL [49], DANN [16], IDM [11] and SCDA [32]. Moreover, we compare to four recently proposed UDAVR methods, i.e., MAN [13], CAPQ [6], ACP [37] and DADA [22]. For fairness, all methods adopt the same features and the backbone network as [5].
Table 1: Comparison with different baselines.
Method	Tf→Mt	Mt→Tf	Tf→Md	Md→Tf	Mt→Md	Md→Mt
R1↑	R10↑	MR↓	R1↑	R10↑	MR↓	R1↑	R10↑	MR↓	R1↑	R10↑	MR↓	R1↑	R10↑	MR↓	R1↑	R10↑	MR↓
Source Only	2.69	13.63	144	6.30	25.43	60	9.39	37.77	20	3.80	16.99	102	15.02	46.96	12	2.50	13.27	136
(a)	MMD [38]	2.68	13.59	135	6.77	27.11	54	9.11	36.11	23	3.50	16.28	119	15.31	47.65	12	2.62	13.18	136
CORAL [49]	2.74	14.07	128	6.56	26.49	52	9.44	37.87	21	3.65	17.34	108	15.65	49.43	11	2.65	13.34	138
DANN [16]	2.76	13.94	127	6.86	27.17	48	9.27	38.00	20	3.74	16.72	103	15.67	48.67	11	2.62	13.17	134
IDM [11]	2.59	13.11	149	7.12	25.35	60	8.05	35.51	23	3.24	15.78	120	13.96	47.77	12	2.54	12.39	165
SCDA [32]	2.79	14.22	130	6.92	26.70	53	9.84	37.11	22	3.30	17.02	108	15.64	48.65	11	2.55	12.98	138
(b)	MAN [13]	2.53	12.98	144	6.42	25.96	63	8.84	37.06	21	3.06	16.31	119	15.05	48.51	11	2.40	12.00	174
CAPQ [6]	3.46	17.02	110	7.33	25.64	62	9.30	37.97	21	3.97	17.75	113	15.66	49.08	11	3.35	15.47	158
ACP [37]	4.41	21.72	64	7.83	26.72	50	12.09	41.38	18	5.12	21.46	82	17.87	54.34	8	5.90	25.68	54
DADA [22]	5.30	24.54	50	8.21	28.97	45	14.34	48.77	11	6.03	22.52	78	18.97	57.93	7	6.40	27.61	42
Ours	UAN	6.12	27.23	40	9.16	31.06	37	15.15	49.34	10	6.51	23.93	69	20.25	60.23	5	6.52	28.15	40
Table 2: Effect of  ${\mathcal{L}}_{D}$  and  ${\mathcal{L}}_{U}$  .
Method	Tf→Mt	Mt→Tf
R1↑	R10↑	MR↓	R1↑	R10↑	MR↓
Source Only	2.69	13.63	144	6.30	25.43	60
UAN(w/o LD)	3.63	18.12	79	6.72	26.76	50
UAN(w/o LU)	3.87	18.45	76	6.91	27.01	48
UAN(full)	6.12	27.23	40	9.16	31.06	37
Table 3: Analysis on different alignment mechanisms.
Method	Tf→Mt	Mt→Tf
R1↑	R10↑	MR↓	R1↑	R10↑	MR↓
UAN(w/ DAC)	5.43	25.63	48	8.44	29.12	42
UAN(w/ UAM)	5.76	26.16	43	8.73	29.84	40
UAN(full)	6.12	27.23	40	9.16	31.06	37
Table 1 shows that: (1) As the lower bound, Source Only achieves the worst performance, which identifies the existed domain shift problem. Compared with Source Only, our method achieves  $3.43\%$  and  $2.86\%$  absolute performance improvements on  $\mathbf{R}@\mathbf{1}$  under the setting of  $\mathrm{Tf}\rightarrow \mathrm{Mt}$  and  $\mathrm{Mt}\rightarrow \mathrm{Tf}$  respectively, indicating the huge domain gap and the efficacy of UAN. (2) Traditional DA methods in setting (a) are ineffective for the UDAVR task, which can only slightly outperform the Source Only baseline. The reason is that these methods are originally designed for classification tasks, which might not be suitable for the video-text retrieval. (3) Our method consistently outperforms other UDAVR methods in setting (b) on all adaptation directions across the three datasets, which demonstrates the effectiveness of distribution-based vision-language domain adaptation and uncertainty-aware alignment mechanism. Compared with the SOTA method DADA, UAN achieves  $15.47\%$  and  $11.57\%$  relative improvements on  $\mathbf{R}@\mathbf{1}$  under the setting of  $\mathrm{Tf}\rightarrow \mathrm{Mt}$  and  $\mathrm{Mt}\rightarrow \mathrm{Tf}$  respectively.
Training process of different ablation models. To systematically evaluate the effectiveness of each module of our proposed UAN, we train the model by removing each component solely and present the results in Table 2. In the main ablation study, we design the following ablation models: (1)UAN(w/o  $\mathcal{L}_D$ ): we remove the distribution-based vision-language domain adaptation module (Eq. 9) from UAN; (2)UAN(w/o  $\mathcal{L}_U$ ): we remove the uncertainty-aware alignment mechanism module (Eq. 14) from UAN; (3)UAN(full): our full UAN model. The results of UAN(w/o  $\mathcal{L}_D$ ) and UAN(w/o  $\mathcal{L}_U$ ) are inferior to the full UAN method, verifying the effectiveness of distribution-based vision-language domain adaptation and uncertainty-aware alignment mechanism.
Analysis on uncertainty-aware alignment mechanisms. We conduct a comprehensive analysis of several alignment mechanisms to explore the efficacy of uncertainty-aware alignment mechanisms(UAM), and present the findings in Table 3. Specifically, (1)UAN(w/ DAC): we use dual alignment consistency mechanism adaptively aligns the most similar videos and texts in target domain. (2)UAN(w/ UAM): we use UAM to dig in uncertainty-aware multimodal relationships in the target domain. (3)UAN(full): our full UAN model with the threshold  $T$ . The results of UAN(w/ DAC) are worse than the UAN(w/ UAM), demonstrating that the uncertainty-aware alignment mechanism is superior to dual alignment consistency mechanism. Meanwhile, the result of UAN(w/ UAM) is also inferior, which proves that the constraint on high similarities of truly aligned pairs is effective. Furthermore, Figure 4 shows the effect of threshold  $K$  in UAM within one batch during the training procedure. Our method consistently performs well and achieves the best when  $K = 2$ .
Analysis on different DA methods. We conduct extensive ablation studies with several state-of-the-art domain adaptation methods to verify the effectiveness of Distribution-based Vision-Language Domain Adaptation(D-VLDA). We implement three classification-based (i.e., typical) DA methods
Table 5: Generalization to different video-text retrieval methods.
Method	Tf→Mt	Mt→Tf
R1↑	R10↑	MR↓	R1↑	R10↑	MR↓
(a)	HGR [7]	2.20	11.98	154	5.87	22.10	72
HGR + UAN	4.52	21.32	86	8.43	29.23	45
GPO [5]	2.69	13.63	144	6.30	25.43	60
GPO + UAN	6.12	27.23	40	9.16	31.06	37
(b)	CE [35]	2.93	14.7	122	6.50	26.23	56
CE + UAN	6.23	27.25	41	9.32	32.41	34
MMT [15]	4.20	22.30	78	7.32	31.46	30
MMT + UAN	6.53	28.32	34	9.63	38.89	19
(c)	CLIP4CLIP [42]	7.20	28.50	35	10.43	38.16	26
CLIP4CLIP + UAN	9.32	37.85	22	13.55	47.33	15
CLIP2Video [14]	7.80	31.50	31	11.21	39.48	23
CLIP2Video + UAN	9.72	38.43	17	14.23	47.87	12
Table 6: The results of image-text retrieval.
Method	Open Narr→COCO Narr	COCO→COCO Narr
R1↑	R10↑	MR↓	R1↑	R10↑	MR↓
(a)	SCAN [29]	17.4	52.6	9	22.3	72.98	5
VSRN [31]	19.6	54.7	7	25.1	75.4	4
CE [35]	19.6	56.4	7	24.5	75.8	4
(b)	CDAN [39]	20.6	59.2	6	22.2	73.3	5
CORAL [49]	19.4	58.3	7	25.4	74.6	4
DANN [16]	19.0	58.4	7	24.8	76.8	4
MMD [38]	17.3	50.8	9	22.6	72.0	5
OT [57]	20.3	57.1	8	25.0	75.6	4
(c)	MAN [13]	20.4	57.3	8	25.6	75.8	4
CAPQ [6]	21.8	57.4	7	26.5	76.4	4
ACP [37]	22.3	57.9	6	27.3	77.9	4
DADA [22]	22.9	58.3	5	28.1	78.3	4
Ours	UAN	23.6	59.2	4	29.5	79.5	3
Figure 4: Analysis of hyper parameters, i.e.,  $K$ ,  $T$ ,  $\lambda_1$ , and  $\lambda_2$ , with the R@10 retrieval performance across different domain adaptation directions.
and modify them for the UDAVR task, including MMD [38], CORAL [49] and GRL [17]. We also evaluate with TPN [46], CDD [28] and MSTN [58], which are all recently proposed DA methods based on conditional alignment. Table 4 shows how the proposed D-VLDA method performs when combined with different DA components. The proposed D-VLDA achieves significant improvements compared with classification-based and conditional alignment DA methods, which indicates that the D-VLDA module plays an essential role in the UDA video-text retrieval task.
Generalization to different cross-modal methods. As shown in Table 5, we implement several state-of-the-art video-text retrieval methods and the corresponding combinations with our UAM. Clearly, our method consistently improve the performances on target domain when combined with original baselines. Moreover, we thereby generalize our UAN to the image-text retrieval scenario, to further show its effectiveness on tackling cross-modal retrieval task. For image-text retrieval, we use three datasets: COCO [9], COCO Narratives [47] and Open Narratives [47] to evaluate the generalization of the proposed method. As reported in Table 6, our
Table 4: Analysis on different DA methods.
Method	Tf→Mt	Mt→Tf
R1↑	R10↑	MR↓	R1↑	R10↑	MR↓
Source Only	2.69	13.63	144	6.30	25.43	60
UAN(w/ MMD [38])	5.62	25.56	45	8.36	29.61	41
UAN(w/ CORAL [49])	5.64	25.58	44	8.38	29.64	41
UAN(w/ GRL [17])	5.66	25.67	44	8.43	29.71	40
UAN(w/ TPN [46])	5.72	25.72	43	8.46	29.75	40
UAN(w/ CDD [28])	5.75	25.73	43	8.56	29.82	39
UAN(w/ MSTN [58])	5.76	25.82	42	8.58	29.91	39
UAN(w/ D-VLDA)	6.12	27.23	40	9.16	31.06	37
method outperforms latest image-text retrieval approaches, i.e.,SCAN [29], VSRN [31] and CE [35]. Obviously, our method can be directly utilized in unsupervised domain adaptation video/image-text retrieval task, demonstrating the generalization ability of our method.
Analysis on hyperparameters. We conduct experiments under all the domain adaptation settings and present the sensitivity of hyper parameters in Figure 4. By altering the values within a feasible range while leaving the other hyper parameters as the corresponding default values, we can have a thorough understanding on how exactly each component contributes to the overall performance. As can be seen, within a wide range of  $\lambda$  in [0.1, 1.0], the performance only varies in a small range, indicating the robustness to different choices of  $\lambda$ . Similarly, when progressively increasing threshold  $T$  in UAM from 16 to 1,024, our method consistently performs well and achieves the best when  $T = 64$  under  $K = 2$  settings. Thus, we set  $\lambda_1 = 0.4$ ,  $\lambda_2 = 0.5$  and  $T$  to 64 under all settings.
(a) Source Only
(b) MMD [38]
(c) DADA [22]
(d) Our method
Figure 5: The t-SNE visualizations of (a) Source Only, (b) MMD, (c) DADA and (d) Our method.
Figure 6: Visualizations of query texts and corresponding videos along with the ranking changes in  $\mathrm{A}\leftarrow \mathrm{B}\leftarrow \mathrm{C}$ , where A denotes the rank of UAN, B the ACP method and C the Source Only.
Feature visualizations. We randomly choose 100 pairs in source and target domain respectively, and show the t-SNE [53] visualizations of Source Only, MMD, DADA and our method under Tf  $\rightarrow$  Mt setting in Figure 5. Blue/red denotes source/target domain, while circles/triangles denote videos/texts. Clearly, Source Only method performs the worst as the blue and red features fail to mix up due to the large domain gaps, whereas (b) and (c) can both to some extent alleviate the domain shift. However, they still show a distinct distributional gap between different features, which can be owed to the multimodal misalignment in target domain. Obviously, our method progressively generates truly aligned video-text pairs in target domain.
Qualitative Results. In Figure 6, we further visualize the returned results across different settings and baselines. Specifically, we show how the rank of relevant videos changes with different methods when given a query text. It is clear that our method leads to higher ranks of relevant videos compared to Source Only and ACP. One limitation is that our method performs worse given the query 'There is lady is riding through the sea waves', i.e., our method returns the relevant video with rank 10 while ACP with rank 8. One possible reason might be the high similarity of RGB color between foreground lady and background sea waves.
In this paper, we focus on the notoriously challenging task, i.e., UDA Video-text Retrieval (UDAVR), which can be classified as a cross-modal cross-domain retrieval task. To tackle the one-to-many issue, we propose the Uncertainty-aware Alignment Network (UAN), which progressively generates truly aligned target pairs and ensures the discriminability of target features. Extensive experiments and ablation studies across several benchmarks justify the superiority of our method.
Limitations and Societal Impacts. One limitation is the difficulty in handling the one-to-many issue at the early training stage. As the massive data (both videos and texts) in target domain have no semantic relations with each other, especially in the first few epochs, it is probably inaccurate when selecting the truly positive pairs. Besides, a simple and effective UDA Video-Text Retrieval
(UDAVR) method is crucial for reduce the huge performance gap between source and target data. We hope that our method can pave the way for effective and robust video-text retrieval system.
This work was supported by the National Key R&D Program of China under Grant 2022YFB3103500, the National Natural Science Foundation of China under Grant 62202459, and the China Postdoctoral Science Foundation under Grants 2022M713348 and 2022TQ0363.
[1] S. Albanie, Y. Liu, A. Nagrani, A. Miech, E. Coto, I. Laptev, R. Sukthankar, B. Ghanem, A. Zisserman, V. Gabeur, C. Sun, K. Allahari, C. Schmid, S. Chen, Y. Zhao, Q. Jin, K. Cui, H. Liu, C. Wang, Y. Jiang, and X. Hao. The end-of-end-to-end: A video understanding pentathlon challenge (2020). arXiv preprint arXiv:2008.00744, 2020.
[2] F. Bianchi, G. Attanasio, R. Pisoni, S. Terragni, G. Sarti, and S. Lakshmi. Contrastive language-image pre-training for the Italian language. arXiv preprint arXiv:2108.08688.
[3] S.-V. Bogolin, I. Croitoru, H. Jin, Y. Liu, and S. Albania. Cross modal retrieval with querybank normalisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5194-5205, 2022.
[4] W.-L. Chao, H. Hu, and F. Sha. Cross-dataset adaptation for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5716-5725, 2018.
[5] J. Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15789-15798, 2021.
[6] Q. Chen, Y. Liu, and S. Albanie. Mind-the-gap! unsupervised domain adaptation for text-video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1072-1080, 2021.
[7] S. Chen, Y. Zhao, Q. Jin, and Q. Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10635-10644, 2020.
[8] T.-H. Chen, Y.-H. Liao, C.-Y. Chuang, W.-T. Hsu, J. Fu, and M. Sun. Show, adapt and tell: Adversarial training of cross-domain image captioner. In Proceedings of the IEEE/CVF international conference on computer vision, pages 521-530, 2017.
[9] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollar, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325, 2015.
[10] I. Croitoru, S.-V. Bogolin, M. Leordeanu, H. Jin, A. Zisserman, S. Albanie, and Y. Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11583-11593, 2021.
[11] Y. Dai, J. Liu, Y. Sun, Z. Tong, C. Zhang, and L.-Y. Duan. Idm: An intermediate domain module for domain adaptive person re-id. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021.
[12] B. B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In European Conference on Computer Vision, pages 467-483, 2018.
[13] J. Dong, Z. Long, X. Mao, C. Lin, Y. He, and S. Ji. Multi-level alignment network for domain adaptive cross-modal retrieval. Neurocomputing, pages 207-219, 2021.
[14] H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image CLIP. arXiv preprint arXiv:2106.11097, 2021.
[49] B. Sun and K. Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In European Conference on Computer Vision, volume 9915, pages 443-450, 2016.
[50] Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7472-7481, 2018.
[51] Y.-H. Tsai, K. Sohn, S. Schulter, and M. Chandraker. Domain adaptation for structured output via discriminative patch representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1456-1465, 2019.
[52] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474, 2014.
[53] L. van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008.
[54] X. Wang, L. Zhu, and Y. Yang. T2VLAD: global-local sequence alignment for text-video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5079-5088, 2021.
[55] M. Wray, H. Doughty, and D. Damen. On semantic similarity in video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3650-3660, 2021.
[56] W. Wu, H. Luo, B. Fang, J. Wang, and W. Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10704-10713, 2023.
[57] S. Xiao, M. Farajtabar, X. Ye, J. Yan, X. Yang, L. Song, and H. Zha. Wasserstein learning of deep generative point process models. In Advances in Neural Information Processing Systems, pages 3247-3257, 2017.
[58] S. Xie, Z. Zheng, L. Chen, and C. Chen. Learning semantic representations for unsupervised domain adaptation. In International conference on machine learning, pages 5423-5432, 2018.
[59] J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5288-5296, 2016.
[60] M. Yang, W. Zhao, W. Xu, Y. Feng, Z. Zhao, X. Chen, and K. Lei. Multitask learning for cross-domain image captioning. IEEE Transactions on Multimedia, 21(4):1047-1061, 2018.
[61] J. Yuan, Z. Zha, Y. Zheng, M. Wang, X. Zhou, and T. Chua. Utilizing related samples to enhance interactive concept-based video search. IEEE Transactions on Multimedia, pages 1343-1355, 2011.
[62] W. Zhao, X. Wu, and J. Luo. Cross-domain image captioning via cross-modal retrieval and model adaptation. IEEE Transactions on Image Processing, pages 1180-1192, 2020.