Uncertainty-Aware Alignment  Network  for Cross-Domain Video-Text Retrieval
Authors: Xiaoshuai Hao
Wanqian Zhang
Venue: NeurIPS 2023 poster (2023)
Keywords: video-text retrieval; cross-domain;Unsupervised Domain Adaptation Video-text Retrieval;
TL;DR: In this paper, we address the challenge task of Unsupervised Domain Adaptation Video-text Retrieval (UDAVR), assuming that training (source) data and testing (target) data are from different domains.

Abstract: Video-text retrieval is an important but challenging research task in the multimedia community.  In this paper, we address the challenge task of Unsupervised Domain Adaptation Video-text Retrieval (UDAVR), assuming that training (source) data and testing (target) data are from different domains. Previous approaches are mostly derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task.  In addition, as to the pairwise misalignment issue in target domain, i.e., no pairwise annotations between target videos and texts, the existing method assumes that a video corresponds to a text. Yet we empirically find that in the real scene, one text usually corresponds to multiple videos and vice versa. To tackle this one-to-many issue, we propose a novel method named Uncertainty-aware Alignment Network (UAN). Specifically, we first introduce the multimodal mutual information module to balance the minimization of domain shift in a smooth manner. To tackle the multimodal uncertainties pairwise misalignment in target domain, we propose the Uncertainty-aware Alignment Mechanism (UAM) to fully exploit the semantic information of both modalities in target domain. Extensive experiments in the context of domain-adaptive video-text retrieval demonstrate that our proposed method consistently outperforms multiple baselines, showing a superior generalization ability for target data.

Uncertainty-Aware Alignment Network for Cross-Domain Texe Retrieval
Xiaoshuai Hao', Wanqian Zhang2*
*Samsung Research China-Beijing (SRC-B)
2 Institute of Information Engineering, Chinese Academy Sciences
xshuai.. haw@samsung com, zhangvanqian0lie
Abstract
Video-text retrieval is important but challenging research task in the multimedia community. this paper, we address the challenge task of Unsupervised Domain Adaptation Video-text Retrieval (UDAVR), assuming that training (source) data and testing (target) data are from different domains. Previous approaches are mostly derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task. addition, as to the pairwise misalignment issue in target domain, i.e., pairwise annotations between target videos and texts, the existing method assumes that a video corresponds to a text. Yet we empirically find that in the real scene, one usually corresponds to multiple videos and vice versa. To tackle this one-to-many issue, we propose a novel method named Uncertainty-aware Alignment Network (UAN). Specifically, introduce the multimodal mutual information module to balance the minimization of domain shift in smooth manner. To tackle the multimodal uncertainties pairwise misalignment in target domain, we propose the Uncertainty-awee awwre Alignment Mechanism (UAM) to fully exploit the semantic information of both modalities target domain. Extensive experiments in the context of domain-and andain-adeptive video-text retrieval demonstrate that our proposed method consistently outperforms multiple baselines, showing a superior generalization ability for target data.
first
Introduction
With the exponential growth of user-generated videos the Internet, cross-modal retrieval between video data and natural language descriptions, known as video-text retrieval, has attracted much attention. The goal of this task is to retrieve videos which are semantically related to given text query and vice versa. Generally, the facto paradigm of video-text retrieval task is to learn high-level video and text embeddings respectively by fff-the-sshef feature extractors. To further measure cross-modal semantic similarities between videos and texts, corresponding embeddings should be aligned and projected into joint embedding space [3] 10 27 41 55 [56 61]. where the semanticallyssmiile texts and videos are much closer to each other and vice versa.
With the advent of deep learning, video-text retrieval has shown promising performance in recent years. Despite the great success, most existing works of this task are supervised, which typically train models on a large number of aligned video-text pairs, making assumption that training and testing data are drawn from the same distribution. If this assumption does not hold, traditional cross-modal retrieval methods may suffer performance drop at the evaluation. To alleviate this, recently proposed methods of Unsupervised Domain Adaptation Vidco-text Retrieval (UDAVR) have attracted much attention, wherein identical label set for source and target domains exists. By identical we denote that the label set of source data is exactly the same as that target data. For instance, given a
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Source will. celi0tzzon
Source with anrosstun
spperbeed
supervbed "aannng
innutaiion
alles
Tiagst willou
zaaiigma
Flapst withou ansoation
Taaee vidiou anrecttonn
Video
Text
Video
Text
Video
Text
UDA Viden-test Retrieval Task
Existing works
Ours
Illustration of the UDA Video-text Retrieval task, existing works and the proposed method. The proposed method utilizes Distribution-baat -basd Vision-Language Domain Adaptation(D (D- (aatattiooo-DLAAA in domain gap and uncertainy--wwae multimodal alignment mechanism in the target domain.
Figure 1:
video-text pair labeled as play basketball in source domain, identical label set requires the existence of target pair which also labeled as it. This can barely be guaranteed due to the diversities between different datasets, such scenarios, video scenes and lengths. Thus, the only supervision of UDAVR is the semantic relationship in source dataset, i.e., whether one source video and one source text are matched, which is also the general setting in other UDA cross-modal tasks 4
60 62
To alleviate the domain shift problem, recently proposed methods of Unsupervised Domain Adap- tation (UDA) has attracted much attention, such as image classification B2 52] autonomous driv- ing [50 51 and video-based action recognition 45
48). However, these methods are originally designed for classification tasks, which are not suitable for the video-text retrieval task. To that end, several pioneering UDAVR methods have been recently proposed 66 13 2,,37] They try to tackle the lack of target domain supervision by directly minimizing the distribution discrepancy [13 distilling knowledge from the source domain [6), introducing pre-defined prototype assignments 37
adaptively aligning the video-text pairs which are more likely be relevant in target domain 22 Despite the success on improving the retrieval performance of target data, they overlook the challeng- ing one-to-many relationship issue in target domain, ie.. existing methods assume that one text the target domain corresponds to only one video and vice versa. However, in real scenes, one query text usually corresponds to multiple videos or one query video corresponds multiple texts.
As illustrated in Figure I. previous methods are derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task. In addition, for the pairwise misalignment issue in target domain, the existing methods assume that a video corresponds to a text. This will inevitably mix target videos and texts together. Consequently, indiscriminative target video and text features will be introduced, leading to less generalized model and poor performance on target data. This rather challenging and thus becomes the motivation of our work. this paper, we propose a novel method named Uncertainty-ave -wware Alignment Network (UAN). We first introduce the cross- modal semantic embedding generate discriminative source features in a joint embedding space, where semantically relevant pairs should lie close together and vice versa. To alleviate the domain shift, we further utilize the Distribution- based Vision-Lann LaaiuuLL Domain Adaptation(D-VIL VD-VLDA) to minimize of distribution shifts between source and target domains. To tackle the one-to-many in target domain, we propose a simple yet effective Uncertainty-awart aware Aligmment Mechanism (UAM), which fully exploits the semantic information of both modalities in target domain. Extensive experiments on several benchmarks demonstrate the superiority of our method.
The contributions of this paper are mainly threefold:
For the challenging Unsupervised Domain Adaptation Video-text Retrieval (UDAVR) task, we propose a simple yet effective Uncertainty-avee awaae Alignment Network (UAN), which fully exploits the semantic information of both modalities in target domain.
To tackle the one-to-many in target domain, the proposed Uncertainty-aware Alignment Mechanism (UAM) tries to utilize the multi-granularity relationships between each target video and text to ensure the discriminability of target features.
Compared with the state-of-the-art methods, UAN achieves 15.47% and 11.57% relative improvements on R @ under the setting of TGIF-MSRVTT and MSRVTT-:TGH respec- tively, demonstrating the superiority of our method.

---

2 Related Work
There are two main types of video-text retrieval approaches: single-modal based and multi-modal based. Initially, single-modal based approaches extend image-text retrieval methods. HGR 7 suggests a Hierarchical Graph Reasoning model that separates text into a hierar- chical semantic graph with different levels. Additionally, GPO [5] offers an innovative Generalized Pooling Operator that adapts itself to the best pooling strategy for various baselines. More recently, the widely adopted Contrastive Language- Imaauagggegaggeeeeeee Pretraining(LLIP 2 model is now commonly used in video-text retrieval [ 18 30
Video-Text Retrieval.
(3 36 43). CLIP4Clip 42 transfers the knowledge of large-
scale image-text pre-training to the video-text retrieval task through fine-tuning and explores three similarity calculation mechanisms based on the pre-trained CLIP model. Similarly, CI.IP2vdee [14 concentrates on the spatial semantics derived from the CLIP model. Another research direction has emphasized capturing multi-modal representations using various experts, which are then combined to derive the final video features. For instance, CE [35] uses video features from all modalities to encode a video, while T2VLAD [54] concentrates on automatically learning text-and-video semantic topics and emphasizing the significance of local semantic alignment between them. On the other hand, MMT [i5] utilizes multi-modal information extracted by seven pretrained experts but only fuses them in straightforward manner without specific guidance. In contrast to these methods, we explore the video-texi retrieval task through the lens of unsupervised domain adaptation.
Unsupervised Domain Adaptation for Video-Text Retrieval. UDA transfers knowledge from
a labeled source domain to an unlabeled target domain 12 32 38 40 49). which is realized by training the model with both labeled source data and unlabeled target data. Recently, some cross- modal tasks resort to UDA and try to utilize the unpaired data in target domain, such as image captioning 8 60 62 and VQA 4. DCKT (26) focuses on UDA image-text retrieval and transfers knowledge from a large dataset to promote the model performance on small dataset. However, these UDA methods are originally designed for classification tasks, which might not be suitable for the video-text retrieval. To our knowledge, there are only a few explorations of the Unsupervised Domain Adaptation for Video-Text Retrieval task 16 13 22 377. MAN [13 proposes Multi-level Alignment Network to alleviate different gaps in UDAVR task. CAPQ 6 introduces the inaugural benchmark for unsupervised domain adaptation in video-text retrieval. ACP (37] proposes new adaptive cross-modal prototypes framework to minimizing uni-modal and cross-modal shifting between the source and target domains. DADA (22) introduces the Dual Alignment Consistency (DAC) mechanism adaptively aligns the most similar videos and texts in target domain. Our approach diverges from previous works in two principal ways. (1) Previous approaches are mostly derived from classification based domain adaptation methods to solve domain discrepancy problem, which are neither multi-modal nor suitable for retrieval task. (2) Prior works have not fully leveraged the semantic connections between videos and text in target domains, assuming that video corresponds to text. However, the real scene, one text usually corresponds to multiple videos and vice versa as illustrated in Figure 3) This problem is the primary motivation for our paper.
3 Methodology
3.1 Preliminaries
Problem Definition. For notational clarity, we first introduce the symbols and definitions used throughout this paper. Formally, assume that we have a set of samples in source domain {(V, = (of, ( (0)=)) where n, indicates the number of video-text pairs. Similarly, there
nz { with two collections of n4 videos y"r and texts T respectively. Note that the target vidcos and texts are unpaired, which means the supervised information is not available in target domain. In other words, we have no idea whether one target video-text pair is semantically relevant or not during the training procedure. The objective of Unsupervised Domain Adaptation Video-Text Retrieval (UDAVR) is to enhance the model's 's ability generalize to the target domain by leveraging information from the source domain. The overall framework of our method is illustrated in Figure
also exists set of samples in target domain {-²= = [uf) T'
Semantic Embedding Learning. Using the latest baseline in video-text retrieval, as described in [5] we employ a video encoder y (:) and a text encoder & (-) to generate a joint embedding space for given video-text pair, consisting of video sample and text description t. For both source and
3

---

Video Features
Source
Video
Video Encoder
Target
o(v))
Mase
Vileo
Video
Fintums
ant rodto
Cp
Ayp +yp
Hany
Souree
Source Text
Large crond gathered
Adgestion
Faatirs
sitef)
Encoder
Target Text
There Iody iding Bonsgg the waves
Figure 2: The overall framework of UAN. Semantic Embedding Learning generates discriminative source features in joint embedding space (Lss). Distribution-basee Vision-Languag Domain Adap- tation (D-VLDA) is proposed to alleviate the domain discrepancy problem in both modalities(Lp)) Uncertainty-Aware Alignment Mechanism (UAM) is proposed to dig in uncertainty-awar multi- modal relationships in the target domain(Ly.)
target data, we share the same feature extracting modules map them into the common space. Such a design can be interpreted as to some extent transfer the knowledge learned in the source domain to the target domain. If the text effectively describes the video, the visual embedding yp (v) E TR and the text embedding v (t) R exhibit semantic relevance, with M representing the dimension in the shared space. In the source domain, we use the symmetric cross-entropy loss to guide the semantic alignment learning. Given batch of B text-video pairs, the model updates its parameters through similarity matrix. Normally, we represent
maximizing the sum of the main diagonal of a the cross-entropy loss in text-to-video directions in Eq. [ and represent the cross-entropy loss in text-video is calculated as the inner production of t; and i in Eq The above formula can be formulated as:
video-to-text directions in Eq. 2 The similarity
exp (s (41,0))
Lizu
Σ log
Σ exp (s1,
B
(u;,
Lu2s = B
Σ log Σ (v),t)))))
(2)
(1i, v.) = (14, uz) = (v) (4: (mi)
where a (t:) and y (v:) are the corresponding mapped features. Formally, the symmetrie cross- entropy loss Ls is the sum of text-to-video loss L120 and video-to-text loss Cuzt, formulated as:
Ls L1zv + Luzt
3.2 Distribution-based Vision-Language Domain Adaptation
In light the video-text symmetric cross-entropy loss, the representations of unimodal encoders can be effectively aligned, which enables the generation of more informative samples. Thus, we can obtain discriminative and modality-ivvarian invariant features of source videos and texts in the joint embedding space. However, this discriminability not generalize well to the target domain, due to the inevitable domain discrepancy problem.
To alleviate this, existing works typically utilize classification domain adaptation (DA) methods shift the feature distributions in different domains as close as possible for each modality, such as IDM I1I and MMD (38). However. these methods are originally designed for classification tasks, which not suitable for the video-text retrieval kask. Thus, we propose Distribution-ban based Vision- Language Domain Adaptation, called D-VLDA, to relieve the divergence of domain statistics, thus the distribution shifts can be significantly diminished, improving the generalization of learned model
target domain. The basic idea is that the moment calculated on different

---

domains are the same if the distributions of source and target domains are identical. Specifically, for two Gaussian distributions (M ) and N ((uz, Σz), their 2- 2-Wasserttenn distance defined as:
|ز +2 2-:1//222)/1) 1/2
Daw = |l 22112 'Ir
(5)
In our modeled distributions, Σn and Σg are both diagonal matrices, indicating Σ// Eg ,/ The above formula can be rewritten as:
E,
Daw Il Hellz + 'Ir ((21/-->2)
2) )
(6)
I| -22l2 + - 1101-0211..
where refers to a standard deviation vector. In our task, the feature distribution consists of two folds: video domain adaptation module and text domain adaptation module. Therefore, we propose two domain alignment loss functions LDr and LDV as follows:
LDr D2w (Ve,V*) V¹) +
Lnv = =a D2w (T',T) b,
(8)
LD =D=LDT LDV.
(9)
where a is a negative scale factor since similarity is inversely proportional to the distance, and b is shift value. Thus, by reducing the distance between two distributions based on Eq. 9 we align the feature distributions of the same modal in different domains.
3.3 Uncertainty-awarc Alignment Mechanism
Besides alleviating the domain discrepancy, we also need bridge the semantic gap between videos and texts in the target domain, which is the key to the standard UDAVR task. Specifically, given the target set Dd (v T {
n } pairwise information is not available the target domain. j=1
This means that the target videos and texts are unpaired, and as result, the issue of one-to- to-many arises the target domain. However, existing method [22] assumes that video corresponds to a text, whereas in the real scene, one text usually corresponds to multiple videos and vice versa (as illustrated Alignment in Figure 3p. To alleviate this, we propose a simple yet effective Uncertainty-Awer Mechanism (UAM). The UAM tries to utilize the multi-granulard relationships between each target video and text to guide the unsupervised learning targer dataset. We calculate the similarities (ST-V) between target text & and all target vidcos, and the similarities (SV-+) between target video of and all target texts, which can be defined as follows:
ST-V =[eefot Sejog. Siog Sejer
(10)
$ SK [S Soft. .. Softs Sofe.
(11)
1. if v/- and .r. are reciprocal TOP-K similar,
I (1b.,,t.)
(12)
0, otherwise.
Note that this requires v/- and 15. to be the reciprocal TOP-K similar of each other, indicating a truly aligned (or positive) pair. To that end,
obtain ny positive pairs in one batch from target dataset, denoted as {(>0) Th) = (v. {')); where (of (2) (o tf.) To improve the accuracy of aligned pairs, we introduce T as a threshold for sorting the similarities of all pairs in a batch in descending order and selecting the T-th value, which corresponds
the top T most similar pairs. Our rationale is that truly positive video-text pair should not only be the most similar to each other but also have a relatively high similarity score compared all the misaligned pairs. Formally, the uncertainty-awe aware alignment matching operation of Eq. [12 can be revised as:
I (v/., f/--) {
if Spge 2 ST, O, otherwise.
(13)
where ST denotes the top T-th value in one batch and ST [S1, S2, Sr.... Sp= (5 ST TT
des denotes re-ranking descending order (from sim- ilar to dissimilar). The combination of uncertainty-awart -warr alignment matching and the similarity
5

---

Text to- Video Uncemiinty
Vide--t-Test Unnettary
93
X1
V2
93
V4
V1
V2
13
Y4
15
Torplc the undaary pijcooo
1.77
8:44
8:37
6:36
0.05
72
3.25
8:47
0.06
4:25
6:44
0.91
15
0447
0.54
propue pogle
1424
3.54
4:51
064
6:38
0.25
0.53
0.18
0.23
Encertainty-Aware Multinodal
One-to relationships
to-toe--twweeeeo relationships
One-to thrre relationshipps
Figure 3: Illustration of Uncertainty-Awar Aligmment Mechanism (UAM). If of. and tz.- to be the reciprocal TOP-K similar of each other, indicating a truly aligned (or positive) pair.
threshold T contributes to the alignment of target videos and texts in a mutually-promoteed manner. By using these self-discovered matching pairs, we can approach the pairwise misalignment issue as fully supervised problem. We can define the uncertainty- alignment loss as:
exp (s (C, )
LU
log
Ej eap (s 29)
(14)
B exp (a (v (v))
+ Σ log B exp (8 (v,,
ΣS
Therefore, the full training objective is given by:
L Ls + XD XzLu,
(15)
where d and 2 are hyper parameters for balancing these terms.
4 Experiments
4.1 Experimental Setting
Datasets. We use existing datasets across three domains to explore the UDAVR task, i.e., a compre- hensive evaluation benchmark by combining three popular datasets: MSR-VTT (Mt)[59), MSVD [9), and TGIF (3). We assume that data from different datasets are from different domains, as they often have dissimilar data distributions and representations.
Evaluation Metries. To evaluate the performance of our video-text retrieval, utilize standard retrieval metrics, as outlined in previous studies ([7) 21 23 24 25)). Our analysis is based on based metrics, including R@ K (the higher, the better) and Median Rank, i.e., MR, (the lower, the better).
Implementation Details. To ensure fair comparison, we employ the identical architecture for the video and text encoders as used in GPO [5), which is currently the leading baseline for video-text retrieval. The length of shared embedding M is set to 1024. Moreover, we adopt the Adam optimizer for all our experiments. During the training procedure, we set the batch size to 64, and utilize step-decayed learning rate with initialization value 0.0001. All experiments are conducted five times for the average performance on a 2080Ti GPU server. For all our experiments, we set to -0.005 and The hyper parameters Å A2 K and T of the overall objective function loss is discussed We set the max epochs as 100, and early stop occurs if the validation performance does not improve in ten consecutive epochs.
extensively in section 4.3
4.2 Performance Comparison and Analysis
We compare our method with several state-of-the-an baselines across three categories, i.e., Source Only, DA methods and UDAVR methods. As lower bound, we include the non-adapted Source Only results, which directly applies the model trained on the source domain to the target domain. We also implement five classification-ban
(i.e. typical) DA methods and modify them for the UDAVR task, including MMD 38 CORAL 49). DANN [16,, IDM [1 and SCDA (32 Moreover, we compare to four recently proposed UDAVR methods, i.e., MAN [3] CAPQ (6, ACP 37 and DADA 22
For fairesss, all methods adopt the same features and the backbone network as (5]

---

Table 1: Comparison with different baselines.
Method
TI-Mi
MI-TI
TY-Mi
MI-=TI
MI-MI Md
Mi Mi
RIT
R10T
MRL
RTT
RIDT
MRL
RIT
RTOT
MR.
RIT
RTOT
MRL
RIT
RIOT
MRL
RIT
MR.
Source Only
25.43
37.77
15.02
46.96
2.50
13.27
1336
MMD
88
2,68
13.59
135
6.77
27.11
54
9.11
36.31
23
3.50
16.28
119
15.31
47.00
12
2.62
13.08
136
CORAL
2.74
14.07
128
6.56
26.49
52
9.44
57.87
21
3.65
17:34
108
15.65
49.45
11
2.65
13.34
1388
DANN
16
2776
13.94
6,86
27.17
9.27
35.00
20
3.74
16.72
15.67
48.67
2.62
13:17
134
IDM 11
2.59
13.01
149
7.12
25.35
8.05
35.51
23
3.24
15.78
120
13.96
47.77
12
2.54
12.39
165
SCDA
12
2.79
14.22
130
6.92
36.70
53
9.84
57.31
22
3.30
17.00
108
15.54
48.65
11
2:55
12.08
138
25.96
37.06
15.05
48.51
2.40
12.00
174
CAPO
3.46
17.02
110
7.33
25.64
57.99
21
113
15.66
49.08
11
3.35
15.47
138
ACP 1x]
4.41
21.72
61
7.83
26.
50
12.09
41.38
18
5.12
21.46
82
17.87
$4.34
8
5.90
25.68
54
DADA
2+
5.30
28.97
48.77
11
22.53
18.97
57.93
6.40
42
Ours
UAN
6.12
27.23
40
9.16
31.06
15.15
49.34
10
6.51
23,93
69
20.25
60.23
5
6.52
28.15
40
Table 2: Effect of LD and Cy.
Table 3: Analysis on different alignment mecha- nisms.
MT
MI-TTI
Method
Rif
R100
MKJ
R17
R10t
MRL
Source Only
2.69
13.63
144
6,30
25.43
60
Method
RIF
Tr-
R101
MRL
RI7
Mt- -Y
RIDE
MRL
UANtwio LD)
3,63
18.12
6.72
26.36
UANEW DAC)
5,43
25.63
48
8.44
29.12
42
UANiwdo Co.
3.87
18.45
76
6,91
27.01
UANNW LAM)
5.76
26.16
43
8.73
29.84
40
UAN(fall)
6.12
27.23
40
9.16
31.06
37
UANifalt
6.12
27.23
40
9.16
31.06
37
Table D shows that: (1) As the lower bound, Source Only achieves the worst performance, which identifies the existed domain shift problem. Compared with Source Only, our method achieves 3.43% and 2.86% absolute performance improvements on R@ I under the setting of Mf-Mt and Mt-Tff respectively, indicating the huge domain gap and the efficacy of UAN. (2) Traditional DA methods in setting (a) are ineffective for the UDAVR task, which can only slightly outperform the Source Only baseline. The reason is that these methods are originally designed for classification tasks, which might not be suitable for the video-text retrieval. (3) Our method consistently outperforms other UDAVR methods in setting (b) all adaptation directions across the three datasets, which demonstrates the effectiveness of distribution- alignment mechanism. Compared with the SOTA method DADA, UAN achieves 15.47% and 11.57% relative improvements on R@] I under the setting of Tf-Mt and Mt-Tf respectively.
vision-language domain adaptation and uncertainty
4.3 Ablation Studies
Training process of different ablation models. To systematically evaluate the effectiveness of each module of our proposed UAN, we train the model by removing each component solely and present the results in Table 2 In the main ablation study, we design the following ablation models: (1)AAN(w/)
CD): we remove the distribution- based vision- language domain adaptation module (Eq. 9 from
UAN: (2)UAN(w//o Lu we remove the ancertainty-awan alignment mechanism module (Eq. 14
from UAN; (3)UAN(full)::om inferior to the full UAN method, verifying the effectiveness of distribution-basee vision-language domain adaptation and uncertainty-awant alignment mechanism.
full UAN model. The results of UAN(w/o Lp) and UAN(w/o Lv) are
Analysis on uncertainty-aware alignment mechanisms. We conduct a comprehensive analysis of several alignment mechanisms to explore the efficacy of uncertainty- anisms(UAM), and present the findings in Table 3 Specifically, (I)UAN(W/ DAC): we use dual alignment consistency mechanism adaptively aligns the most similar videos and texts in target domain. multimodal relationships in the target
alignment mech-
(2)UAN(w/ UAM): we use UAM to dig in uncertainty-awart domain. (3)lAN(fll): our full UAN model with the threshold T', The results of UAN(w/ DAC) are worse than the UAN(w/ UAM), demonstrating that the uncertainty- is superior to dual alignment consistency mechanism. Meanwhile, the result of UAN(w/ UAM) is also inferior, which proves that the constraint on high similarities of truly aligned pairs is effective. Furthermore, Figure4shows shows the effect of threshold K in UAM within one batch during the training procedure. Our method consistently performs well and achieves the best whenK = 2.
alignment mechanism
Analysis on different DA methods. We conduct extensive ablation studies with several state-of-the- art domain adaptation methods to verify the effectiveness of Distribution-base based Vision-L Languaee Domain Adaptation(D-VLDD VLDA). We implement three classification- based (i.c., typical) DA methods
7

---

Table 6: The results of image-text retrieval.
Table 5: Generalization to different video-text re- trieval methods.
Open Narr->
COCO
Method
COCO Narr
COCO Narr
Method
TT-
Mit+II
RIT
RIOT MR.
RIT
RIOT
MR.
RIT
SCAN
17.4
52.6
22.3
72.98
HGR 7
2.20 11.95
154
5.87
22,10
72
VSRN
11
19.6
54.7
25.1
75.4
4
(8)
HGR CAN
4.52
21.32
8.43
29.23
45
CE
35
19.6
56.4
24.5
75.8
4
20.6
59.2
GPO CAN
6.12
27.23
9.16
31,06
CORAL
49
19.4
58.3
25.4
74.6
4
CE 35
2.95
14.7
122
6.50
26.25
56
(h)
DANN
16
19.0
58.4
24.8
76.8
MMT 11
4.20
22.30
78
7.30
31,46
30
OI
57
20.3
57.1
25.0
75.6
MMT GAN
6.53
28.32
34
9.63
38,89
19
MAN
13
20.4
57.3
25.6
75.8
4
CLIPACLIP 42
CAPQ
57.4
ACP
37
22.3
57.9
27.3
77.9
4
CLINCLIP' CAN
9.32
37.85
13.55
47.33
CLIP2Vide> 1
7,80 31.50
31
11.21
39.48
23
DADA
22
22,9
58.3
28.1
78.3
CL.IPEVideo
UAN
23,6
59.2
12--0
10- 11-114
21-440
10
11-49
Mi--00
Figure 4: Analysis of hyper parameters, i.e., K, T, Å1, and A2 with the R@ 10 retrieval performance across different domain adaptation directions.
and modify them for the UDAVR task, including MMD 33), CORAL 49 and GRL [17] We also evaluate with TPN [46). CDD 28 and MSTN (58]. which are all recently proposed DA methods based on conditional alignment. Table 4 shows how the proposed D- D-VLDA method performs when combined with different DA components. The proposed D-VLDA achieves significant improvements compared with classification- and conditional alignment DA methods, which indicates that the D-VLDA module plays an essential role the UDA video-text retrieval task.
Generalization to different cross-modal methods. As shown in 5 we implement several state-of-the- art video-text retrieval methods and the corresponding combinations with our UAM. Clearly, our method con- sistently improve the performances on target domain when combined with original baselines. Moreover, we thereby generalize our UAN to the image-text retrieval scenario, further show its effectiveness on tackling cross-modal retrieval task. For image-text retrieval, we use three datasets: COCO
Table 4: Analysis on different DA methods.
Method
RIT R1OT
MRJ
RIT
R10T
MR.
Source Only
2.69
13,63
164
6,30
25.43
60
LANNW MMD (x)
25.56
UANw/ CORAL 19
5.64
25.58
44
8:38
29.64
41
LAN/W/ GKL
17
5,66
25.67
8.43
29.71
40
UAN(W/ CDD
25
5775
25.73
43
8:56
29.82
39
COCO Narratives [47 and Open Narratives 47 evaluate the generalization of the proposed method. As reported in Table 6 our method outperforms latest image-text retrieval approaches, i.e.,SCAN VSRN 31 and CE Obviously, our method can directly utilized in unsupervised domain adaptation video/image-texx retrieval task, demonstrating the generalization ability of our method.
9
UAN/w/ MSTN
58
5,76
25.82
8.58
29.91
39
UANOw/D-VI.DA)
135
Analysis on ayperparameters. We conduct experiments under all the domain adaptation settings and present the sensitivity of hyper parameters in Figure 4 By altering the values within feasible range while leaving the other hyper parameters as the corresponding default values, we can have a thorough understanding on how exactly each component contributes to the overall performance. As can seen, within a wide range of À in [0. 1.0], the performance only varies small range, indicating the robustness to different choices of X. Similarly, when progressively increasing threshold in UAM from 16 1, 1,024, our method consistently performs well and achieves the best when T = 64 under K 2 setting. Thus, set λ₁ 0.4, X2 = 0.5 and T to 64 under all settings.
8

---

(a) Source Only
(b) MMD 38
(c)
DADA P2
(d) Our method
Figure 5: The t-SNE visualizations of (a) Source Only, (b) MMD, (c) DADA and (d) Our method.
Weras und zif svinesine Add water to the pot
boyand glartng and
grospofaidh gida petfienn
Rebeaar video that quars:
Rask that vidwe
1e-12-21
2 11-25 wouEnn conting bozal
1-133-23
3- 1--26 arcep ary prictcing hooiing
2-- 16-15
I-m 19--8
aninaloooooseee anntherii yangle
iniidiag
workslepp will
tablet
for
Rank ofth vilees
1w-17-19
3-17h-44
10--5-
MI-II
M--ME
364-Ir
Mi-
Figure 6: Visualizations of query texts and corresponding videos along with the ranking changes in At A+B+C, where A denotes the rank of UAN, the ACP method and the Source Only.
Feature visualizations. We randomly choose 100 pairs in source and target domain respectively, and show the (-SNE (53] visualizations Source Only. MMD, DADA and our method under Tf-Mt setting in Figure 5 Blue/red denotes source/target domain, while circles/triangles denote videos/texts. Clearly, Source Only method performs the worst as the blue and red features fail to mix up due large domain gaps, whereas (b) and (c) can both to extent alleviate the domain shift. However, they still show distinct distributional gap between different features, which can be owed to the multimodal misalignment in target domain. Obviously, method progressively generates truly aligned video-text pairs in target domain.
Qualitative Results. In Figure 6 we further visualize the returned results across different settings and baselines. Specifically, we show how the rank of relevant videos changes with different methods when given query text. It is clear that method leads to higher ranks of relevant videos compared to Source Only and ACP. One limitation that our method performs worse given the query 'There is lady is riding through the sea waves', i.e., our method returns the relevant video with rank 10 while ACP with rank 8. One possible reason might be the high similarity RGB color between foreground lady and background waves.
Conclusion
In this paper, focus the notoriously challenging task, i.c., UDA Video-text Retrieval (UDAVR), which can classified as cross-mode modal cross-domain retrieval task. To tackle the one-to-many issue, propose the Uncertainty-awan Alignment Network (UAN), which progressively generates truly aligned target pairs and ensures the discriminability target features. Extensive experiments and ablation studies across several benchmarks justify the superiority of our method.
Limitations and Societal Impacts. One limitation is the difficulty in handling the one-to-many issue at the early training stage. As the massive data (both videos and texts) in target domain have no semantic relations with each other, especially in the first few epochs, probably inaccurate when selecting the truly positive pairs. Besides, simple and effective UDA Video-Te Text Retrieval

---

(UDAVR) method is crucial for reduce the huge performance gap between source and target data. We hope that our method can pave the way for effective and robust video-text retrieval system.
6 Acknowledgement
This work was supported by the National Key R&D Program of China under Grant 3FB 1035000 the National Natural Science Foundation of China under Grant 62202459, and the China Postdoctoral Science Foundation under Grants 2022M7 13348 and 2022TQ0363.
References
[1] S. Albanie, Y. Liu, A. Nagrani, A. Miech, E. Coto. I. Laptev, R. Sukthankar, B. Ghanem, A. Zisserman, V. Gabeur, C. Sun, K. Alahari, C. Schmid, S. Chen, Y. Zhao. Q. Jin, K. Cui, H. Liu, C. Wang, Y. Jiang, and X. Hao. The soddoofend-to--eee A video understanding pentathlon challenge (2020). arXiv preprint arXiv:2008.0077 2020.
[2] F. Bianchi, G. Attanasio, R. Pisoni, S. Terragni, G. Sarti, and S. Lakshmi. Contrastive language- image pre-training for the italian language. arXiv preprint arXiv:2108.000 08688.
[3] S.-V. Bogolin, I. Croitoru, H. Jin, Y. Liu, and Albanie, Cross modal retrieval with querybank normalisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5194-5205, 2022.
[4] W.-L. Chao, H. Hu, and Sha. Cross-dataset adaptation for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5716-5725, 2018
[5] Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15789-157998. 2021.
[6] Q. Chen, Y. Liu, and S. Albanie. Mind-the-gap! unsupervised domain adaptation for text-video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 1072-1080, 2021.
[7] S. Chen, Y. Zhao, Q. Jin, and Q. Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10635-10644, 2020.
[8] T.-H. Chen, Y.-H. Liao, C.-Y. Chuang, W.-T. Hsu, J. Fu, and M. Sun. Show, adapt and tell: Adversarial training of cross- image captioner. In Proceedings of the IEEE/CVF international conference computer vision, pages 521-530. 2017.
[9] X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint orXiv::5044 00325, 2015.
[10] I. Croitoru, S.-V. Bogolin, M. Leordeanu, H. Jin, A. Zisserman, S. Albanie, and Y. Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1183--1599, 2021.
[11] Y. Dai, J. Liu, Y. Sun, Z. Tong, C. Zhang, and L.-Y. Duan. Idm: An intermediate domain module for domain adaptive person re-id. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
[12] B. B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In European Conference on Computer Vision, pages 467-483, 2018.
[13] J. Dong. Z. Long, X. Mao, C. Lin, Y. He, and S. Ji. Multi-level alignment network for domain adaptive mross-modal retrieval. Neurocomputing, pages 207-219, 2021.
[14] H. Fang, Xiong, I.. Xu, and Chen. Clip2video: Mastering video-text retrieval via image CLIP. arXiv preprint arX.v:2106.110%0 2021.
10

---

[15] V. Gabeur, C. Sun, K. Alabari, and C. Schmid. Multi-modal transformer for video retrieval. In European Conference on Computer Vision, pages 214-229, 2020.
[16] Y. Ganin and V. Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of the International Conference on Machine Learning, pages 1 1180-1189, 2015.
[17] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research, pages 2096 2030, 2016.
[18] S. K. Gorti, N. Vouitsis, J. Ma, K. Golestan, M. Volkovs, A. Garg, and G. Yu. X-pool: Cross- modal language-video attention for text-video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5006-5015, 2022.
[19] S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. Mooney, T. Darrell, and K. Saenko. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 2712-2719, 2013.
[20] X. Han, L. Yu, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang, Fashionvil: Fashion-focused vision-and- saioo-and-languuee and-language representation learning. In European Conference on Computer Vision, pages (34-651, 2022.
[21] X. Hao, W. Zhang, D. Wu, F. Zhu, and B. Li. Listen and look: Multi-modal aggregation and co-attention network for video-audio retrieval. In International Conference on Multimedia and Expo, pages 1-6, 2022.
[22] X. Hao, W. Zhang, D. Wu, F. Zhu, and B. Li. Dual alignment unsupervised domain adaptation for video-text retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattem Recognition, pages 18962 18972, 2023.
[23] X. Hao, Y. Zhou, D. Wu, W. Zhang, B. Li, and W. Wang. Multi-feature graph attention network for cross-modal video-text retrieval. In International Conference on Multimedia Retrieval, pages 135-143, 2021.
[24] X. Hao, Y. Zhou, D. Wu, W. Zhang, B. Li, W. Wang, and D. Meng. What matters: Attentive and relational feature aggregation network for video-text retrieval. In International Conference on Multimedia and Expo, pages 1-6, 2021.
[25] X. Hao, Y. Zhu, S. Appalaraju, A. Zhang, W. Zhang, B. Li, and M. Li. Mixgen: new modti-modal data augmentation. In Winter Conference on Applications of Computer Vision Workshops, pages 379-389, 2023.
[26] X. Huang and Y. Peng- Deep cross-media knowledge transfer. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 8837-8846, 2018.
[27] P. Jin, J. Huang, P. Xiong, S. Tian, C. Liu, X. Ji. L. Yuan, and J. Chen. Video-text game players: Hierarchical banzhaf interaction for cross-modall representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 2472-12222 -282, 2023.
[28] G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4893-4902, 2019.
[29] K. Lee, X. Chen, G. Hua, H. Hu, and X. He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision, pages 201-216, 2018.
[30] D. Li, J. Li, H. Li, J. C. Niebles, and S. C. H. Hoi. Align and prompt: Video-and- Video-and-language language pre-training with entity prompts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4943 4953, 2022.
[31] K. Li, Y. Zhang, K. Li, Y. and Y. Fu. Visual semantic reasoning for image-text matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4654 4662, 2019.
11

---

[32] S. Li, M. Xie, F. Lv, C. H. Liu, J. Liang, C. Qin, and W. Li. Semantic concentration for domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9102-9111 11 2021.
[33] Y. Li, Y. Song, L. Cao, J. Tetreault, L. Goldberg, A. Jaimes, and J. Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4641-4650, 2016.
[34] K. Q. Lin, Wang, M. Soldan, M. Wray, R. Yan, E. Z. XU, D. Gao, R.-C. Tu, W. Zhao, W. Kong, al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems, pages 7575-7586, 2022.
[35] Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv: (ttb..997..1488 2019.
[36] Y. Liu, H. Chen, L. Huang, D. Chen, B. Wang, P. Pan, and L. Wang. Animating images to transfer CLIP for video-text retrieval. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1906-1911, 2022.
[37] Y. Liu, Q Chen, and S. Albanie. Adaptive cross-modal prototypes for cross-domain visual- language retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14954-14964, 2021.
[38] M. Long, Y. Cao, Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning, pages 97-105, 2015.
[39] M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In Annual Conference on Neural Information Processing Systems, pages 1647-1657, 2018.
2018
[40] M. Long, H. Zhu, J. Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. Proceedings of the International Conference on Machine Learning, pages 2208- 2217, 2017.
[41] X. Long, C. Gan, G. D. Melo, J. Wu, and S. Wen. Attention clusters: Purely attention based local feature integration for video classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7834-7843, 2018
[42] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing, pages 293-304, 2022.
[43] Y. Ma, G. Xu, X. Sun, M. Yan. Zhang, and R. Ji. X-CLIP: end-to-end multi- multi-grained contrastive learning for video-text retrieval. In ACM International Conference on Multimedia, pages 638-647, 2022.
[44] F. Markatopoulou, D. Galanopoulos, V. Mezaris, and I. Patras. Query and keyframe represen- tations for ad-hoc video search. In Proceedings of the ACM International Conference on Multimedia Retrieval, pages 407-411, 2017.
[45] B. Pan, 7. Cao, F. Adeli, and J. C. Niebles. Adversarial cross-domain action recognition with co-attention. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 11815-1 11822, 2020.
[46] Y. Pan, T. Yao, Y. Li, Wang, C.WW. Ngo, and T. Mei. Transferrable prototypical networks for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2239-2247, 2019.
[47] J. Pont-Tuset J. R. R. Uijlings, S. Changpinyo, R. Soricut, and V. Ferrari. Connecting vision and language with localized narratives. In European Conference Computer Vision, pages 647-664, 2020.
[48] X. Song, S. Zhao, J. Yang, H. Yue, P. Xu, R. Hu, and H. Chai. Spatio-temporal contrastive domain adaptation for action recognition. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9787-9795, 2021.
12

---

[49] B. Sun and K. Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In European Conference on Computer Vision, volume 9915, pages 443 4400, 2016.
[50] Y.-H. Tsai, W.-C Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1572-7481, 2018.
[51] Y.-H. Tsai, K. Sohn, S. Schulter, and M. Chandraker. Domain adaptation for structured output via discriminative patch representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1456-1465, 2019.
[52] E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximiz- ing for domain invariance. arXiv preprint arXiv:1412.3477 2014.
[53] I.. van der Maaten and Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research, 2008.
[54] X. Wang, L. Zhu, and Y. Yang. T2VLAD: global-local sequence alignment for text-video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5079-5088, 2021.
[55] M. Wray, H. Doughty, and D. Damen. On semantic similarity in video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3650-3660, 2021.
[56] W. Wu, H. Luo, B. Fang, J. Wang, and W. Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
10713, 2023.
[57] S. Xiao, M. Farajtabar, X. Ye, J. Yan, X. Yang, L. Song, and H. Zha. Wasserstein learning of deep generative point process models. In Advances in Neural Information Processing Systems, pages 3247-3257, 2017.
[58] S. Xie, Z. Zheng, L. Chen, and C. Chen. Learning semantic representations for unsupervised domain adaptation. In International conference on machine learning, pages 5423-5432, 2018.
[59] Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5288-5296, 2016.
[60] M. Yang. W. Zhao, W. Xu, Y. Feng, Z. Zhao. X. Chen, and K. Lei. Multitask learning for cross-domain image captioning. IEFE Transactions Multimedia, 2144): 2018.
[61] J. Yuan, Z. Zha, Y. Zheng, M. Wang, X. Zhou, and T. Chua. Utilizing related samples to enhance interactive concept- based video search. IEEE Transactions on Multimedia, pages 1355, 2011.
[62] W. Zhao, X. Wu, and Luo. Cross-domain image captioning via cross-modal retrieval and model adaptation. IEEE Transactions on Image Processing, pages 1180-1192, 2020.
13


Reviews:
[{"review_id": "RzCl7BJZMh", "reviewer": "Reviewer_NzVt", "summary": "This paper addresses the unsupervised domain adaptation video-text retrieval problem by two proposed components, the D-VLDA (Distribution-based Vision-Language Domain Adaptation) and the UAM (Uncertainty-Aware Alignment). The proposed D-VLDA aims at alleviating the domain discrepancy via moment-based method, while the UAM method aims for better pseudo labeling.", "questions": "(1) The term \"(moment-based) D-VLDA module\" appears to be referred to as \"multi-modal mutual information module\" in the abstract. Could the author please clarify the relationship between these two concepts?\n\n(2) In the design of Uncertainty-Aware Alignment(UAM), the authors expand the top-1 dual alignment method from [A] to a top-k dual alignment. It appears that Table 3 aims to validate this expansion, where \"UAN(w/DAC)\" denotes the application of the top-1 dual alignment consistency mechanism to pick those most similar videos and texts in the target domain, as described in [A]. However,  I note that the results for \"UAN(w/DAC)\" match those reported in [A], while [A] utilizing a different discrepancy loss from D-VLDA. This may lead to an unfair comparison. I would suggest the authors control the variables to ensure a more equitable comparison.\n\n(3) The authors propose the uncertainty-aware alignment network based on the \"one-to-many\" assumption: one text can describe multiple videos and vice versa. This forms the fundamental premise of the paper. However, the frequency of the \"one-to-many\" phenomenon remains unclarified. I'm concerned that the efficacy of the method may not be primarily due to the \"one-to-many\" assumption, but potentially other factors like the top-K methods introduce more true positive pairs in the early training stage, providing the model with more supervision about the target domain. I think the author should verify the frequent existence of the \"one-to-many\". For example, collecting the top-K dual alignment results by the final model, removing the pairs that belong to the ground truth pairs, and then see the proportion of the remaining pairs to all the ground truth pairs, and also estimate how many of the left pairs match correctly (belongs to the \"one-to-many\").\n\n[A] X. Hao, W. Zhang, D. Wu, F. Zhu, and B. Li. Dual alignment unsupervised domain adaptation for video-text retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.\n", "limitations": "No", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "(+) This article is well written, and the description of the method pipeline is clear and easy to understand. \n\n(+) Experiments are sufficient and ablation studies show the effectiveness of the proposed components.\n\n(+) Stat-of-the-art results are demonstrated on several datasets.", "weaknesses": "(-) Unfair comparison in table 3, see in Question(2). \n\n(-) The \"one-to-many\" assumption that underpins the article lacks empirical evidence supporting its widespread occurrence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the positive feedback. \n\n**Question1**: unfair comparison in Tab. 3.\n\n**Response1**: Thank you for pointing out this, and the corrected experimental data is as follows:\n\n|**Method** | | Tf->Mt |  |  | Mt->Tf |  |\n|  :----:   |    :----:   |  :---: | :---: | :---: | :---: | :---: | \n|   | R@1 | R@10| MR | R@1 | R@10| MR |\n| UAN(w/ DAC)  | 5.43    | 25.63   | 48| 8.44|29.12|42\n| UAN(w/ UAM)  | 5.76  | 26.12   | 43| 8.73|29.84|40\n| **UAN(full)**   | **6.12**    | **27.23**   |**40**| **9.16**|**31.06**|**37**\n\n\n\n**Question2**: \"(moment-based) D-VLDA module\" vs. \"multi-modal mutual information module\".\n\n**Response2**: Thanks for the advice. We claim that by 'multi-modal' we refer to the contribution of our proposed domain alignment module, which is the first in UDAVR to tackle the domain gap with the consideration of different modalities. While `moment-based D-VLDA' denotes the specific technique we adopted (minimizing the domain gap in a distributional manner). We will align these terms for more clarity in the final version.\n\n\n**Question3**: the frequent existence of the \"one-to-many\".\n\n**Response3**: Thanks for the constructive advice. We conduct ablations to verify this as follows. We randomly select 1,000 video-text pairs and calculate the semantic similarities between each two texts (with details in [a]). We then define one video and one text as a `one-to-many' pair if the corresponding similarity is above the threshold. Results can be found below. The one-to-many relationship contributes to the selected target pairs for training and the overall performance.\n\n|**Dataset**|S=1.0 | S>0.9 |S>0.8  | S>0.7 |\n|  :----:   |    :----:   |  :---: | :---: | :---: |\n|MSR-VTT|1,000|1,232|1,398|1,478|\n|MSVD|1,000|1,268|1,426|1,520|\n|TGIF|1,000|1,198|1,306|1,406|\n\n[a] Wray, Michael, Hazel Doughty, and Dima Damen. \"On semantic similarity in video retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n"}, {"review_id": "NyipG8Qilk", "reviewer": "Reviewer_muWC", "summary": "This paper proposed a unsupervised domain adaptation method tailored for video-text retrieval by exploring the one-to-many correspondences between video and text on the target domains. The proposed method is shown effective and superior to existing approaches which considered only the one-to-one video-text matching relationships on the unlabelled target domains.", "questions": "+ It will be interesting to see how many discovered one-to-many matching relationships are true and consistent with the manual labels on a validation set.\n+ In Fig.4, the model obtained its best performance when K=2, does this mean that each video is optimally to be matched with two text sentences? This seems to be applied to all the pairwise combination of datasets but it is not intuitive why a 1-to-2 mapping is always the best. adding more discussion and analysis might provide more insights.", "limitations": "Yes, the author discussed one of the limitations of their proposed method is the false positive inter-sample relationships discovered for training, especially at the early stage.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "+ The idea to learn from the one-to-many matching relationships among video and text in real-world scenarios is intuitive, the proposed model is straighforward\n+ The proposed method not only yielded impressive performance on video-text retrieval, the authors further demonstrate its beneficial to image-text retrieval. Moreover, the authors also proved that the proposed idea is generally beneficial to the task by combining it with a wide range of existing methods.", "weaknesses": "+ The motivations are not sufficiently discussed. The two main motivations claimed in the paper is (1) existing UDAVR approaches are mostly derived from classification based DA methods which is not optimal for retrieval tasks and (2) existing methods assume a one-to-one matching relationships between video and text. For (1), it may be more straightforward to spell out the major drawbacks of adapting classification DA methods to UDAVR rather than just stating it is suboptimal. The motivation (2) should be justified in terms of whether the one-to-many matching relationships are common on the adopted training datasets. It will be more intuitive if this can be quantified.\n+ I'm not sure I fully understand what is the meaning of \"to balance the minimization of domain shift\" at L12\n+ As stated at L36, the definition of UDAVR is that there is no identical textual labels on the two domains. However, two sentences even from the same domains are not likely to be exactly identical but they can describe the same events using a similar vocabulary. Therefore, does it make more sense to measure how different are two video-text retrieval datasets by the overlaps of their vocabularies?\n+ In Fig.1, I failed to find the differences between existing methods which are claimed to be classification-based and the proposed method designs for retrieval tasks (L55)\n+ In Eq.(7)-(9), as a \"negative scale factor\", $a$ seems to be a constant, then how about $b$? is it also a constant? If yes, then why adding a constant in a loss function to be optimised?\n+ Are the symbols $S^{T\\rightarrow V}$ and $S^{V\\rightarrow T}$ used in Eq.(10) and Eq.(11) defined?\n+ minor problems (typos and etc):\n  - duplicated \"to\" at L12, L155\n  - \"similarity Top-K\" in Eq.(12)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the very constructive feedback. We will revise the typos in the final version.\n\n**Question1**: drawbacks of directly adapting classification-based DA methods in UDAVR task.\n\n**Response1**: Previous approaches are mostly derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task. \nIntuitively, (1) classification-based DA methods are designed for single-modality (like image classification) task, which can't be directly applied in multi-modal setting (UDAVR task); (2) their main assumption is the common label set for source and target domains, which still can't be satisfied in UDAVR task. Empirically, in Tab. 4, we conduct extensive ablation studies with several state-of the-art domain adaptation methods. The proposed D-VLDA achieves significant improvements compared with classification-based and conditional alignment DA methods, which indicates that the D-VLDA module plays an essential role in the UDA video-text retrieval task.\n\n**Question2**: the commonness of one-to-many.\n\n**Response2**: Thanks for the constructive advice. We conduct ablations to verify this as follows. We randomly select 1,000 video-text pairs and calculate the semantic similarities between each two texts (with details in [a]). We then define one video and one text as a `one-to-many' pair if the corresponding similarity is above the threshold. Results can be found below. The one-to-many relationship contributes to the selected target pairs for training and the overall performance.\n\n\n|**Dataset**|S=1.0 | S>0.9 |S>0.8  | S>0.7 |\n|  :----:   |    :----:   |  :---: | :---: | :---: |\n|MSR-VTT|1,000|1,232|1,398|1,478|\n|MSVD|1,000|1,268|1,426|1,520|\n|TGIF|1,000|1,198|1,306|1,406|\n\n\n\n\n[a] Wray, Michael, Hazel Doughty, and Dima Damen. \"On semantic similarity in video retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\n**Question3**: the meaning of \"to balance the minimization of domain shift\".\n\n**Response3**: Different datasets usually have inconsistent data distributions and representations, thus leading to the domain shift problem. \nTo alleviate this, we propose D-VLDA module to balance the minimization of domain shift on the source and target domains.\nBy `balance' we refer to the dynamically minimized domain gap designed for multi-modal setting.\n\n**Question4**: measure the difference of two video-text retrieval datasets by the overlaps of their vocabularies.\n\n**Response4**: Thanks for the advice. We admit that in some video-text retrieval datasets, one video may have high similarity with more than one text. For instance, in MSR-VTT, each video corresponds to 20 texts. Besides, the video may also have relatively high similarity with other texts (which belong to other videos). However, we argue that in UDAVR task, the domain gap mainly denotes that there exist large differences like video sources (cartoons, movies, sports etc.), text sources (short sentences, movie dialogues, cooking instructions etc.), video/text lengths and so on. THIS defines the uniqueness of UDAVR task.\n\n**Question5**: illustration of Fig. 1.\n\n**Response5**: Thanks for your suggestion, we will further optimize our Fig. 1 in the camera version.\nIn Fig. 1, existing works are mostly derived from classification based domain adaptation methods to align different domains.\nHowever, we propose Distribution-based Vision-Language Domain Adaptation, called D-VLDA, to relieve the divergence of domain statistics, thus the distribution shifts can be significantly diminished, improving the generalization of learned model on out-of-distribution target domain.\n\n**Question6**: details of Eq. 7-8.\n\n**Response6**: In Eq. 7-8 of D-VLDA, a is a negative scale factor since similarity is inversely proportional to the distance, and b is a shift value. For all our experiments, we set a to -0.005 and b to 6. We will add a discussion of parameters a and b in the Experimental Details section.\n\n**Question7**: why a 1-to-2 mapping is always the best.\n\n**Response7**: By '1-to-2' relationship we denote that during training time, one video can be considered to be `paired' with more than one text (the ground-truth one) and vice versa. We claim that training with one-to-many benefits the target retrieval performance even when evaluating with one-to-one setting. We will further clarity this in the final version."}, {"review_id": "YQX3bqZdQL", "reviewer": "Reviewer_dcEM", "summary": "This paper proposed a method named Uncertainty-aware Alignment Network (UAN) to address the one-to-many issue which means in the real scene, one text usually corresponds to multiple videos and vice versa. Specifically, the proposed method achieves a new state-of-the-art in Cross-Domain Video-Text Retrieval through the incorporation of design elements such multi-modal mutual information module and Uncertainty-aware Alignment Mechanism (UAM).", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe motivation for addressing “one-to-many in target domain” is clearly presented and validated.\n2.\tThe ablation study and generalization experiment in this paper is comprehensive and clearly analyzed, providing strong evidence of the validity of the proposed model.\n", "weaknesses": "1.\tIt is recommended to add baseline performance in Fig 4 to reflect the robustness of the proposed module.\n2.\tThe authors used each batch in the Uncertainty-Aware Alignment Mechanism to obtain self-discovered matching pairs instead of all data, so will the different number of batch have a significant impact on the effectiveness?\n3.\tThe phrase \"the confusion of the lady and sea waves\" mentioned in the Qualitative Results section is ambiguous. Further analysis is recommended in this section to clarify this point.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the positive feedback.\n\n**Question1**: add baseline in Fig. 4.\n\n**Response1**: Thanks for your suggestion, we will add baseline performance in Fig. 4 to reflect the robustness of the proposed module in the camera version of the paper.\n\n**Question2**: ablation on the number of batch size.\n\n**Response2**: Thanks for the advice. This is a very interesting ablation study, and we report the results as follows. As can be seen, when the batch size is from 16 to 64, the performance tends to increase. However, the performance of the batch size from 64 to 256 tends to be stable. This interesting ablation is unexplored before, and we argue that this may be partially due to that small batch size will exclude some ground-truth pairs, while large batch size will lead to the computational overhead instead of more included positive pairs. We will add discussion and analysis in the camera version.\n\n|**Batch** | | Tf->Mt |  |  | Mt->Tf |  |\n|  :----:   |    :----:   |  :---: | :---: | :---: | :---: | :---: | \n|   | R@1 | R@10| MR | R@1 | R@10| MR |\n| 16     | 4.22    | 22.36   | 79| 7.41|32.03|33\n| 32   | 5.62   | 25.56   | 45| 8.36|29.61|41\n| **64**   | **6.12**    | **27.23**   |**40**| **9.16**|**31.06**|**37**\n| 128   | 6.05 | 27.13  | 40| 9.04|30.87|37\n| 256    | 6.03 | 27.08   | 40| 9.01|30.72|37\n\n\n\n\n**Question3**: the phrase \"the confusion of the lady and sea waves\".\n\n**Response3**: Fig. 6 shows that ours achieves consistently best results compared with other SOTAs. We also report a failure case where 'There is lady is riding through the sea waves'. This is acceptable and can be owed to (1) the low visual quality and short length of this video; (2) the confusion of the front object 'lady' and the background 'sea waves'. One possible reason for the confusion is that the 'lady' target is too small and may be 'swallowed' by the `waves'. As to the future work, we will further add explanations to make it easier for readers to understand our work."}, {"review_id": "KhYcEBsRVy", "reviewer": "Reviewer_qpQ6", "summary": "The paper tackles the task of unsupervised domain adaptation for text-video retrieval. The authors propose an Uncertainty-aware Alignment Network (UAN), which exploits the semantic information of both modalities in target domain. Specifically, in order to tackle the one-to-many relationships in the target domain, the proposed Uncertainty-aware Alignment Mechanism (UAM) tries to utilize the multi-granularity relationships between each target video and text to ensures the discriminability of target features. Finally, the authors test their proposed method on several benchmarks.", "questions": "1. How are the results for CE reported for text-image retrieval since as far as I know the method only reports results on text-video retrieval? Do you re-train the method on images? What input features do you use for images?\n\n2. Same question for methods in Tab 5? Do you evaluate using the published weights? Re-train? Can an already trained model be adapted to a different domain without re-training? More details are needed\n\n3. Can you give more details on what is needed to combine the proposed method with other existing methods? What's the computational overhead? Are there any additional costs during inference? etc.", "limitations": "Limitations are discussed thorough the paper and societal impact is also discussed briefly.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper tackles an important problem and obtains good results. The results can be of high interest for the research community in order since the method seems to be relatively easy to combine with other existing methods, however I think some more clarifications are needed on this front.", "weaknesses": "I think some parts of the paper need more explanations (see questions below). Also, for completeness, I think it would be useful to also briefly describe what happens at inference.\n\nMissing papers in text-video retrieval related work\n\n[1] Gorti, Satya Krishna, et al. \"X-pool: Cross-modal language-video attention for text-video retrieval.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Bogolin, Simion-Vlad, et al. \"Cross modal retrieval with querybank normalisation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[minor] typo line 155 \"to to\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the positive feedback. We will add the missing papers in the final version.\n\n**Question1**: details of training CE method for image-text retrieval task.\n\n**Response1**: For a fair comparison, we use the same settings for UDA image-text retrieval in the follow paper ACP [a]. Specifically, we use the object, scene and face features in the image to re-train the model in the CE [b] method.\n\n[a] Y. Liu, Q. Chen, and S. Albanie. Adaptive cross-modal prototypes for cross-domain visual-language retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14954–14964, 2021.\n\n[b] Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv:1907.13487, 2019.\n\n**Question2**: details of the combination of our method with existing methods.\n\n**Response2**: In Tab. 5, we combine the proposed method with other methods by re-training the original methods according to their source codes. For example, the method of `HGR + UAN' is obtained by re-training HGR with the addition of our D-VLDA and UAM modules. This leads to a fair comparison.\n\n**Question3**: details of computation overhead and inference procedure.\n\n**Response3**: During the training phase, the calculation of pair similarities takes about 0.006s for each batch, which is reasonable and acceptable due to the dataset scale and batch size. In the inference stage, we directly use the model trained on the source domain and test it on the target domain directly, which does not involve any computational overhead.\n"}, {"review_id": "fuyqHdKfjr", "reviewer": "Reviewer_FYKf", "summary": "This paper focuses on text-video retrieval for the challenging unsupervised domain adaptation setting. For this problem, the model is trained on a source only set of supervised video/text pairs and is adapted to a target domain which consists of videos and text labels with no pairing ground truth. The proposed method, named Uncertainty-aware Alignment Network (UAN), uses the assumption that many videos can correspond to a textual item and vice-versa, which allows for the multi-granularity modelling. They evaluate their method on three different pairs of datasets between MSR-VTT, MSVD, and TGIF. Their method is found to outperform all other unsupervised domain adaptation methods for video retrieval because they adapt classifier domain adaptation approaches.", "questions": "1. Can the failure cases of the method be expanded and more information given regarding the failure case in Figure 6\n2. What are the values of a and b given in equation 7/8 when training the model and how are these chosen?\n3. With the domain adaptation requiring the one-to-one relationship between video and text during evaluation to be relaxed into a many-to-many relationship during training, do you think this has a method on overall method performance?", "limitations": "Limitations are discussed within the paper in the final paragraph, however, not much is said for the method beyond selecting pairs in the first few epochs likely causes issues. I think this could be expanded further to round out the paper.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The experiments that have been run are convincing and thorough and even include evaluation of the method for image-text domain adaptation which is nice to see and proves that the proposed approach was created from the ground up for embedding spaces instead of borrowed from unsupervised domain adaptation approaches for classification. As a final note, the experiments were all run 5 times and averaged for robustness.\n\nThe proposed model works well for unsupervised domain adaptation video retrieval and makes sense to break from the one-to-one assumption during the training to better find potential positives.", "weaknesses": "What are the values of a and b given in equation 7/8 and how are these chosen?\n\nEquations 10 and 11 could be better introduced/explained, additionally, it's easy to miss that T and K refer to the batch and for an individual example respectively.\n\nThe discussion on failure cases of the model within the paper is rather short. There is one given failure case within figure 6 which is very briefly explained (so much so that I'm not sure how the model failed on this case). I think these could be better highlighted as another future avenue for related work and better understanding of the method as a reader.\n\nThe method calls for many-to-one and one-to-many relationships between the two modalities during training, but for evaluation it is assumed that there is only a one-to-one relationship between modalities as with previous work. There has been some work that mentions the possibility of many-to-many relationships during training for images [a] and videos [b] and it might be worth discussing this point within the limitations which also mentions the one-to-many relationships at the early training stage.\n\n[a] Chun, Sanghyuk, et al. \"Probabilistic embeddings for cross-modal retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[b] Wray, Michael, Hazel Doughty, and Dima Damen. \"On semantic similarity in video retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nOther Comments\nLine 6: missing space between comma and 'as'\n[24] and [25] are duplicated citations\nLine 142: inner production -> inner product", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the positive feedback, pointing out that our method was created `FROM THE GROUND UP FOR EMBEDDING SPACES'! We will revise the typos in the final version.\n\n**Question1**: values of a and b in Eq. 7 and symbols in Eq. 10-12.\n\n**Response1**: In Eq. 7-8 of D-VLDA, a is a negative scale factor since similarity is inversely proportional to the distance, and b is a shift value. \nIn Eq. 10-12, ${S}^{\\mathcal{V} \\rightarrow \\mathcal{T}}$ denotes the calculated similarities of target $v_{i}^t$ with all the target texts and similar definition for ${S}^{\\mathcal{T} \\rightarrow \\mathcal{V}}$.\nFor all our experiments, we set a to -0.005 and b to 6. We will add the discussion and ablation of parameters a and b in the Experimental section.\n\n**Question2**: discussions of failure cases.\n\n**Response2**: Fig. 6 shows that ours achieves consistently best results compared with other SOTAs. We also report a failure case where 'There is lady is riding through the sea waves'. This is acceptable and can be owed to (1) the low visual quality and short length of this video; (2) the confusion of the front object 'lady' and the background 'sea waves'. One possible reason for the confusion is that the 'lady' target is too small and may be 'swallowed' by the `waves'. As to the future work, we will further add explanations to make it easier for readers to understand our work.\n\n**Question3**: one-to-many in training vs. one-to-one in testing.\n\n**Response3**: We consider the training with a more relaxed one-to-many relationship, which is rather straightforward since one video may have high similarity with more than one text. As for 'one video to many texts', this is widely existed in video-text retrieval datasets. While for 'one text to many videos', this is also intuitive that a textual sentence ($t_1$) may sometimes describe the contents of two similar videos ($v_1$ and $v_2$), even if the ground-truth pair is ($v_1$, $t_1$). This has been empirically identified in previous works like [a] and [b]. \nThe insight of our method is to fully exploit this one-to-many relationship during training for a better video/text embedding, and this contributes to the overall retrieval performance in target domain. \n\n[a] Bogolin, Simion-Vlad, et al. \"Cross modal retrieval with querybank normalisation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[b] Wray, Michael, Hazel Doughty, and Dima Damen. \"On semantic similarity in video retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n"}]

Author rebuttal:
We would like to thank the reviewers for their efforts, detailed reviews and interest for our submission. We will integrate all their remarks in the revised version of the paper.

Meta-review: This paper proposes  D-VLDA (Distribution-based Vision-Language Domain Adaptation) and the UAM (Uncertainty-Aware Alignment) to address the unsupervised domain adaptation video-text retrieval. Concerns were originally raised regarding unfair comparisons, unclear presentations and baseline results. 

 The authors provided a rebuttal and after the discussion period, all reviewers vote for acceptance. The AC recommends acceptance, and encourages the authors to revise paper accordingly.
Decision: Accept (poster)