Title: Uncertainty-Aware Alignment  Network  for Cross-Domain Video-Text Retrieval
Venue: NeurIPS 2023
Accepted: yes
Keywords: video-text retrieval; cross-domain;Unsupervised Domain Adaptation Video-text Retrieval;
Abstract: Video-text retrieval is an important but challenging research task in the multimedia community.  In this paper, we address the challenge task of Unsupervised Domain Adaptation Video-text Retrieval (UDAVR), assuming that training (source) data and testing (target) data are from different domains. Previous approaches are mostly derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task.  In addition, as to the pairwise misalignment issue in target domain, i.e., no pairwise annotations between target videos and texts, the existing method assumes that a video corresponds to a text. Yet we empirically find that in the real scene, one text usually corresponds to multiple videos and vice versa. To tackle this one-to-many issue, we propose a novel method named Uncertainty-aware Alignment Network (UAN). Specifically, we first introduce the multimodal mutual information module to balance the minimization of domain shift in a smooth manner. To tackle the multimodal uncertainties pairwise misalignment in target domain, we propose the Uncertainty-aware Alignment Mechanism (UAM) to fully exploit the semantic information of both modalities in target domain. Extensive experiments in the context of domain-adaptive video-text retrieval demonstrate that our proposed method consistently outperforms multiple baselines, showing a superior generalization ability for target data.

Uncertainty-Aware Alignment Network for Cross-Domain Video-Text Retrieval

Abstract

Video-text retrieval is an important but challenging research task in the multimedia community. In this paper, we address the challenge task of Unsupervised Domain Adaptation Video-text Retrieval (UDAVR), assuming that training (source) data and testing (target) data are from different domains. Previous approaches are mostly derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task. In addition, as to the pairwise misalignment issue in target domain, i.e., no pairwise annotations between target videos and texts, the existing method assumes that a video corresponds to a text. Yet we empirically find that in the real scene, one text usually corresponds to multiple videos and vice versa. To tackle this one-to-many issue, we propose a novel method named Uncertainty-aware Alignment Network (UAN). Specifically, we first introduce the multimodal mutual information module to balance the minimization of domain shift in a smooth manner. To tackle the multimodal uncertainties pairwise misalignment in target domain, we propose the Uncertainty-aware Alignment Mechanism (UAM) to fully exploit the semantic information of both modalities in target domain. Extensive experiments in the context of domain-adaptive video-text retrieval demonstrate that our proposed method consistently outperforms multiple baselines, showing a superior generalization ability for target data.

1 Introduction

With the exponential growth of user-generated videos on the Internet, cross-modal retrieval between video data and natural language descriptions, known as video-text retrieval, has attracted much attention. The goal of this task is to retrieve videos which are semantically related to a given text query and vice versa. Generally, the de facto paradigm of video-text retrieval task is to learn high-level video and text embeddings respectively by off-the-shelf feature extractors. To further measure cross-modal semantic similarities between videos and texts, corresponding embeddings should be aligned and projected into a joint embedding space [3, 10, 27, 41, 44, 55, 56, 61], where the semantically-similar texts and videos are much closer to each other and vice versa.

With the advent of deep learning, video-text retrieval has shown promising performance in recent years. Despite the great success, most existing works of this task are supervised, which typically train models on a large number of aligned video-text pairs, making an assumption that training and testing data are drawn from the same distribution. If this assumption does not hold, traditional cross-modal retrieval methods may suffer a performance drop at the evaluation. To alleviate this, recently proposed methods of Unsupervised Domain Adaptation Video-text Retrieval (UDAVR) have attracted much attention, wherein no identical label set for source and target domains exists. By identical we denote that the label set of source data is exactly the same as that of target data. For instance, given a

UDA Video-text Retrieval Task

Source with

annotation

Target  without

annotation

Text

Video

pairwise

misalignment

domain

gap

domain

gap

Existing works

Ours

Source with

annotation

Target  without

annotation

Text

Video

align

domain

align

domain

supervised

learning

Source with

annotation

Target  without

annotation

Text

Video

D-VLDA

align

D-VLDA

align

supervised

learning

Uncertainty-Aware

Multimodal

Certainty-Aware

Multimodal

video-text pair labeled as play basketball in source domain, identical label set requires the existence of target pair which also labeled as it. This can barely be guaranteed due to the diversities between different datasets, such as scenarios, video scenes and lengths. Thus, the only supervision of UDA VR is the semantic relationship in source dataset, i.e., whether one source video and one source text are matched, which is also the general setting in other UDA cross-modal tasks [4, 8, 60, 62].

To alleviate the domain shift problem, recently proposed methods of Unsupervised Domain Adaptation (UDA) has attracted much attention, such as image classification [32, 52], autonomous driving [50, 51] and video-based action recognition [45, 48]. However, these methods are originally designed for classification tasks, which are not suitable for the video-text retrieval task. To that end, several pioneering UDAVR methods have been recently proposed [6, 13, 22, 37]. They try to tackle the lack of target domain supervision by directly minimizing the distribution discrepancy [13], distilling knowledge from the source domain [6], introducing pre-defined prototype assignments [37], or adaptively aligning the video-text pairs which are more likely to be relevant in target domain [22]. Despite the success on improving the retrieval performance of target data, they overlook the challenging one-to-many relationship issue in target domain, i.e., existing methods assume that one text in the target domain corresponds to only one video and vice versa. However, in real scenes, one query text usually corresponds to multiple videos or one query video corresponds to multiple texts.

As illustrated in Figure 1, previous methods are derived from classification based domain adaptation methods, which are neither multi-modal nor suitable for retrieval task. In addition, for the pairwise misalignment issue in target domain, the existing methods assume that a video corresponds to a text. This will inevitably mix target videos and texts together. Consequently, indiscriminative target video and text features will be introduced, leading to less generalized model and poor performance on target data. This is rather challenging and thus becomes the motivation of our work. In this paper, we propose a novel method named Uncertainty-aware Alignment Network (UAN). We first introduce the crossmodal semantic embedding to generate discriminative source features in a joint embedding space, where semantically relevant pairs should lie close together and vice versa. To alleviate the domain shift, we further utilize the Distribution-based Vision-Language Domain Adaptation(D-VLDA) to minimize of distribution shifts between source and target domains. To tackle the one-to-many in target domain, we propose a simple yet effective Uncertainty-aware Alignment Mechanism (UAM), which fully exploits the semantic information of both modalities in target domain. Extensive experiments on several benchmarks demonstrate the superiority of our method.

The contributions of this paper are mainly threefold:

For the challenging Unsupervised Domain Adaptation Video-text Retrieval (UDAVR) task, we propose a simple yet effective Uncertainty-aware Alignment Network (UAN), which fully exploits the semantic information of both modalities in target domain.

To tackle the one-to-many in target domain, the proposed Uncertainty-aware Alignment Mechanism (UAM) tries to utilize the multi-granularity relationships between each target video and text to ensure the discriminability of target features.

Compared with the state-of-the-art methods, UAN achieves 15.47% and 11.57% relative improvements on R@1 under the setting of TGIF → MSRVTT and MSRVTT → TGIF respectively, demonstrating the superiority of our method.

2 Related Work

Video-Text Retrieval. There are two main types of video-text retrieval approaches: single-modal based and multi-modal based. Initially, single-modal based approaches extend image-text retrieval methods. HGR [7] suggests a Hierarchical Graph Reasoning model that separates text into a hierarchical semantic graph with different levels. Additionally, GPO [5] offers an innovative Generalized Pooling Operator that adapts itself to the best pooling strategy for various baselines. More recently, the widely adopted Contrastive Language-Image Pretraining(CLIP) [2]model is now commonly used in video-text retrieval [1, 18, 20, 30, 34, 36, 43]. CLIP4Clip [42] transfers the knowledge of largescale image-text pre-training to the video-text retrieval task through fine-tuning and explores three similarity calculation mechanisms based on the pre-trained CLIP model. Similarly, CLIP2video [14] concentrates on the spatial semantics derived from the CLIP model. Another research direction has emphasized capturing multi-modal representations using various experts, which are then combined to derive the final video features. For instance, CE [35] uses video features from all modalities to encode a video, while T2VLAD [54] concentrates on automatically learning text-and-video semantic topics and emphasizing the significance of local semantic alignment between them. On the other hand, MMT [15] utilizes multi-modal information extracted by seven pretrained experts but only fuses them in a straightforward manner without specific guidance. In contrast to these methods, we explore the video-text retrieval task through the lens of unsupervised domain adaptation.

Unsupervised Domain Adaptation for Video-Text Retrieval. UDA transfers knowledge from a labeled source domain to an unlabeled target domain [12, 32, 38, 40, 49], which is realized by training the model with both labeled source data and unlabeled target data. Recently, some crossmodal tasks resort to UDA and try to utilize the unpaired data in target domain, such as image captioning [8, 60, 62] and VQA [4]. DCKT [26] focuses on UDA image-text retrieval and transfers knowledge from a large dataset to promote the model performance on small dataset. However, these UDA methods are originally designed for classification tasks, which might not be suitable for the video-text retrieval. To our knowledge, there are only a few explorations of the Unsupervised Domain Adaptation for Video-Text Retrieval task [6, 13, 22, 37]. MAN [13] proposes a Multi-level Alignment Network to alleviate different gaps in UDAVR task. CAPQ [6] introduces the inaugural benchmark for unsupervised domain adaptation in video-text retrieval. ACP [37] proposes a new adaptive cross-modal prototypes framework to minimizing uni-modal and cross-modal shifting between the source and target domains. DADA [22] introduces the Dual Alignment Consistency (DAC) mechanism adaptively aligns the most similar videos and texts in target domain. Our approach diverges from previous works in two principal ways. (1) Previous approaches are mostly derived from classification based domain adaptation methods to solve domain discrepancy problem, which are neither multi-modal nor suitable for retrieval task. (2) Prior works have not fully leveraged the semantic connections between videos and text in target domains, assuming that a video corresponds to a text. However, in the real scene, one text usually corresponds to multiple videos and vice versa ( as illustrated in Figure 3). This problem is the primary motivation for our paper.

3 Methodology

3.1 Preliminaries

Problem Definition. For notational clarity, we first introduce the symbols and definitions used throughout this paper. Formally, assume that we have a set of samples in source domain { ( V s , T s ) = ( v s i , t s i ) n s i =1 } , where n s indicates the number of video-text pairs. Similarly, there also exists a set of samples in target domain { V t = { v t i } n t i =1 , T t = { t t j } n t j =1 } with two collections of n t videos V t and texts T t , respectively. Note that the target videos and texts are unpaired , which means the supervised information is not available in target domain. In other words, we have no idea whether one target video-text pair is semantically relevant or not during the training procedure. The objective of Unsupervised Domain Adaptation Video-Text Retrieval (UDAVR) is to enhance the model's ability to generalize to the target domain by leveraging information from the source domain. The overall framework of our method is illustrated in Figure 2.

Semantic Embedding Learning. Using the latest baseline in video-text retrieval, as described in [5], we employ a video encoder φ ( · ) and a text encoder ψ ( · ) to generate a joint embedding space for a given video-text pair, consisting of a video sample v and a text description t . For both source and

Source

Video

Large crowd gathered

in car workshop with

any cars around.

There is lady is

riding through the

sea waves.

Source

Text

Target

Text

Target

Video

𝑣

!

"

Video

Encoder

Text

Encoder

𝜑 𝑣

!

"

𝜓 𝑡

!

"

Video Features

Text Features

Video Domain

Adaptation

Text Domain

Adaptation

ℒ

#$

+

ℒ

$#

Source Video

Features

Source Text

Features

Supervised Loss

in Source

ℒ

%

ℒ

&

There is lady is

riding through the

sea waves.

Uncertainty-Aware  Alignment Mechanism

People in the park

and a boy feeding

pigeons.

A boy is feeding

food to a pigeon.

Target  Pseudo-label

Target Text

Target Video

ℒ

#

𝑣

!

'

𝑡

!

"

𝑡

(

'

𝜑 𝑣

!

)

𝜓 𝑡

(

)

target data, we share the same feature extracting modules to map them into the common space. Such a design can be interpreted as to some extent transfer the knowledge learned in the source domain to the target domain . If the text effectively describes the video, the visual embedding φ ( v ) ∈ R M and the text embedding ψ ( t ) ∈ R M exhibit semantic relevance, with M representing the dimension in the shared space. In the source domain, we use the symmetric cross-entropy loss to guide the semantic alignment learning. Given a batch of B text-video pairs, the model updates its parameters through maximizing the sum of the main diagonal of a B × B similarity matrix. Normally, we represent the cross-entropy loss in text-to-video directions in Eq. 1 and represent the cross-entropy loss in video-to-text directions in Eq. 2. The similarity of the text-video is calculated as the inner production of t i and v i in Eq.3. The above formula can be formulated as:

where ψ ( t i ) and φ ( v i ) are the corresponding mapped features. Formally, the symmetric crossentropy loss L S is the sum of text-to-video loss L t 2 v and video-to-text loss L v 2 t , formulated as:

3.2 Distribution-based Vision-Language Domain Adaptation

In light of the video-text symmetric cross-entropy loss, the representations of unimodal encoders can be effectively aligned, which enables the generation of more informative samples. Thus, we can obtain discriminative and modality-invariant features of source videos and texts in the joint embedding space. However, this discriminability can not generalize well to the target domain, due to the inevitable domain discrepancy problem.

To alleviate this, existing works typically utilize classification domain adaptation (DA) methods to shift the feature distributions in different domains as close as possible for each modality, such as IDM [11] and MMD [38]. However, these methods are originally designed for classification tasks, which is not suitable for the video-text retrieval task. Thus, we propose Distribution-based Vision-Language Domain Adaptation, called D-VLDA , to relieve the divergence of domain statistics, thus the distribution shifts can be significantly diminished, improving the generalization of learned model on out-of-distribution target domain. The basic idea is that the moment calculated on different

domains are the same if the distributions of source and target domains are identical. Specifically, for two Gaussian distributions N ( µ 1 , Σ 1 ) and N ( µ 2 , Σ 2 ) , their 2-Wasserstein distance is defined as:

In our modeled distributions, Σ 1 and Σ 2 are both diagonal matrices, indicating Σ 1 / 2 1 Σ 2 Σ 1 / 2 1 = Σ 1 Σ 2 . The above formula can be rewritten as:

where σ refers to a standard deviation vector. In our task, the feature distribution consists of two folds: video domain adaptation module and text domain adaptation module. Therefore, we propose two domain alignment loss functions L DT and L DV as follows:

where a is a negative scale factor since similarity is inversely proportional to the distance, and b is a shift value. Thus, by reducing the distance between two distributions based on Eq. 9, we align the feature distributions of the same modal in different domains.

3.3 Uncertainty-aware Alignment Mechanism

Besides alleviating the domain discrepancy, we also need to bridge the semantic gap between videos and texts in the target domain, which is the key to the standard UDA VR task. Specifically, given the target set { V t = { v t i } n t i =1 , T t = { t t j } n t j =1 } , pairwise information is not available in the target domain. This means that the target videos and texts are unpaired , and as a result, the issue of one-to-many arises in the target domain. However, existing method [22] assumes that a video corresponds to a text, whereas in the real scene, one text usually corresponds to multiple videos and vice versa (as illustrated in Figure 3). To alleviate this, we propose a simple yet effective Uncertainty-Aware Alignment Mechanism (UAM). The UAM tries to utilize the multi-granularity relationships between each target video and text to guide the unsupervised learning in target dataset. We calculate the similarities ( S T →V v t i ) between a target text t t i and all target videos, and the similarities ( S V→T t t i ) between a target video v t i and all target texts, which can be defined as follows:

Note that this requires v t i ∗ and t t j ∗ to be the reciprocal TOP-K similar of each other, indicating a truly aligned (or positive) pair. To that end, we can obtain n p positive pairs in one batch from target dataset, denoted as { ( V p , T p ) = ( v p i , t p i ) n p i =1 } , where ( v p i , t p i ) = ( v t i ∗ , t t j ∗ ) . To improve the accuracy of aligned pairs, we introduce T as a threshold for sorting the similarities of all pairs in a batch in descending order and selecting the T -th value, which corresponds to the top T most similar pairs. Our rationale is that a truly positive video-text pair should not only be the most similar to each other but also have a relatively high similarity score compared to all the misaligned pairs. Formally, the uncertainty-aware alignment matching operation of Eq. 12 can be revised as:

where S T denotes the top T -th value in one batch and S T ∈ [ S V→T v t i ; S T →V t t j ] des = [ S 1 , S 2 , ..., S T , ..., S B 2 ] des . [ S V→T v t i ; S T →V t t j ] des denotes re-ranking in a descending order (from similar to dissimilar). The combination of uncertainty-aware alignment matching and the similarity

Text-to-VideoUncertainty

Video-to-TextUncertainty

V1

V2

V3

V4

V5

V1

V2

V3

V4

V5

V1

V2

V3

V4

V5

People in the park and a boy feeding pigeons

T1

0.61

0.97

0.37

0.16

0.65

T1

0.61

0.97

0.37

0.16

0.65

T1

0.61

0.97

0.37

0.16

0.65

A feeder is feeding the pigeons in the park

T2

0.37

0.19

0.20

0.17

0.64

T2

0.37

0.19

0.20

0.17

0.64

T2

0.37

0.19

0.20

0.17

0.64

A boy is feeding food to a pigeon

T3

0.47

0.86

0.25

0.54

0.51

T3

0.47

0.86

0.25

0.54

0.51

T3

0.47

0.86

0.25

0.54

0.51

A group of people

T4

0.34

0.54

0.11

0.62

0.48

T4

0.34

0.54

0.11

0.62

0.48

T4

0.34

0.54

0.11

0.62

0.48

holding umbrellas.

T5

0.51

0.64

0.54

0.18

0.23

T5

0.51

0.64

0.54

0.18

0.23

T5

0.51

0.64

0.54

0.18

0.23

Uncertainty-AwareMultimodal

One-to-one relationships

One-to-tworelationships

One-to-threerelationships

threshold T contributes to the alignment of target videos and texts in a mutually-promoted manner. By using these self-discovered matching pairs, we can approach the pairwise misalignment issue as a fully supervised problem. We can define the uncertainty-aware alignment loss as:

Therefore, the full training objective is given by:

where λ 1 and λ 2 are hyper parameters for balancing these terms.

4 Experiments

4.1 Experimental Setting

Datasets. We use existing datasets across three domains to explore the UDAVR task, i.e., a comprehensive evaluation benchmark by combining three popular datasets: MSR-VTT (Mt)[59], MSVD (Md)[19], and TGIF (Tf)[33]. We assume that data from different datasets are from different domains, as they often have dissimilar data distributions and representations.

Evaluation Metrics. To evaluate the performance of our video-text retrieval, we utilize standard retrieval metrics, as outlined in previous studies ([7, 21, 23, 24, 25]). Our analysis is based on rank-based metrics, including R@K (the higher, the better) and Median Rank, i.e., MR, (the lower, the better).

Implementation Details. To ensure a fair comparison, we employ the identical architecture for the video and text encoders as used in GPO [5], which is currently the leading baseline for video-text retrieval. The length of shared embedding M is set to 1024. Moreover, we adopt the Adam optimizer for all our experiments. During the training procedure, we set the batch size to 64, and utilize a step-decayed learning rate with initialization value 0.0001. All experiments are conducted five times for the average performance on a 2080Ti GPU server. For all our experiments, we set a to -0.005 and b to 6. The hyper parameters λ 1 , λ 2 , K and T of the overall objective function loss is discussed extensively in section 4.3. We set the max epochs as 100, and early stop occurs if the validation performance does not improve in ten consecutive epochs.

4.2 Performance Comparison and Analysis

We compare our method with several state-of-the-art baselines across three categories, i.e., Source Only, DA methods and UDAVR methods. As a lower bound, we include the non-adapted Source Only results, which directly applies the model trained on the source domain to the target domain. We also implement five classification-based (i.e. typical) DA methods and modify them for the UDAVR task, including MMD [38], CORAL [49], DANN [16], IDM [11] and SCDA [32]. Moreover, we compare to four recently proposed UDAVR methods, i.e., MAN [13], CAPQ [6], ACP [37] and DADA [22]. For fairness, all methods adopt the same features and the backbone network as [5].

Table 1 shows that: (1) As the lower bound, Source Only achieves the worst performance, which identifies the existed domain shift problem. Compared with Source Only, our method achieves 3.43% and 2.86% absolute performance improvements on R@1 under the setting of Tf → Mt and Mt → Tf respectively, indicating the huge domain gap and the efficacy of UAN. (2) Traditional DA methods in setting (a) are ineffective for the UDA VR task, which can only slightly outperform the Source Only baseline. The reason is that these methods are originally designed for classification tasks, which might not be suitable for the video-text retrieval. (3) Our method consistently outperforms other UDA VR methods in setting (b) on all adaptation directions across the three datasets, which demonstrates the effectiveness of distribution-based vision-language domain adaptation and uncertainty-aware alignment mechanism. Compared with the SOTA method DADA, UAN achieves 15.47% and 11.57% relative improvements on R@1 under the setting of Tf → Mt and Mt → Tf respectively.

4.3 Ablation Studies

Training process of different ablation models. To systematically evaluate the effectiveness of each module of our proposed UAN, we train the model by removing each component solely and present the results in Table 2. In the main ablation study, we design the following ablation models: (1) UAN(w/o L D ) : we remove the distribution-based vision-language domain adaptation module (Eq. 9) from UAN; (2) UAN(w/o L U ) :we remove the uncertainty-aware alignment mechanism module (Eq. 14) from UAN; (3) UAN(full) :our full UAN model. The results of UAN(w/o L D ) and UAN(w/o L U ) are inferior to the full UAN method, verifying the effectiveness of distribution-based vision-language domain adaptation and uncertainty-aware alignment mechanism.

Analysis on uncertainty-aware alignment mechanisms. We conduct a comprehensive analysis of several alignment mechanisms to explore the efficacy of uncertainty-aware alignment mechanisms(UAM), and present the findings in Table 3. Specifically, (1) UAN(w/ DAC) : we use dual alignment consistency mechanism adaptively aligns the most similar videos and texts in target domain. (2) UAN(w/ UAM) : we use UAM to dig in uncertainty-aware multimodal relationships in the target domain. (3) UAN(full) : our full UAN model with the threshold T . The results of UAN(w/ DAC) are worse than the UAN(w/ UAM), demonstrating that the uncertainty-aware alignment mechanism is superior to dual alignment consistency mechanism. Meanwhile, the result of UAN(w/ UAM) is also inferior, which proves that the constraint on high similarities of truly aligned pairs is effective. Furthermore, Figure 4 shows the effect of threshold K in UAM within one batch during the training procedure. Our method consistently performs well and achieves the best when K = 2 .

Analysis on different DA methods. We conduct extensive ablation studies with several state-of-theart domain adaptation methods to verify the effectiveness of Distribution-based Vision-Language Domain Adaptation(D-VLDA). We implement three classification-based (i.e., typical) DA methods

1

2

3

4

5

K

25

30

35

40

45

50

55

60

R@10

Tf

Mt

Mt

Tf

Tf

Tf

Mt

Mt

16

32

64

128

256

512

1024

T

20

25

30

35

40

45

50

55

60

R@10

Tf

Mt

Mt

Tf

Tf

Tf

Mt

Mt

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

1

20

25

30

35

40

45

50

55

60

R@10

Tf

Mt

Mt

Tf

Tf

Tf

Mt

Mt

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

2

25

30

35

40

45

50

55

60

R@10

Tf

Mt

Mt

Tf

Tf

Tf

Mt

Mt

and modify them for the UDAVR task, including MMD [38], CORAL [49] and GRL [17]. We also evaluate with TPN [46], CDD [28] and MSTN [58], which are all recently proposed DA methods based on conditional alignment. Table 4 shows how the proposed D-VLDA method performs when combined with different DA components. The proposed D-VLDA achieves significant improvements compared with classification-based and conditional alignment DA methods, which indicates that the D-VLDA module plays an essential role in the UDA video-text retrieval task.

Generalization to different cross-modal methods.

As shown in Table 5, we implement several state-of-theart video-text retrieval methods and the corresponding combinations with our UAM. Clearly, our method consistently improve the performances on target domain when combined with original baselines. Moreover, we thereby generalize our UAN to the image-text retrieval scenario, to further show its effectiveness on tackling cross-modal retrieval task. For image-text retrieval, we use three datasets: COCO [9], COCO Narratives [47] and Open Narratives [47] to evaluate the generalization of the proposed method. As reported in Table 6, our

method outperforms latest image-text retrieval approaches, i.e.,SCAN [29],VSRN [31] and CE [35]. Obviously, our method can be directly utilized in unsupervised domain adaptation video/image-text retrieval task, demonstrating the generalization ability of our method.

Analysis on hyperparameters. We conduct experiments under all the domain adaptation settings and present the sensitivity of hyper parameters in Figure 4. By altering the values within a feasible range while leaving the other hyper parameters as the corresponding default values, we can have a thorough understanding on how exactly each component contributes to the overall performance. As can be seen, within a wide range of λ in [0.1, 1.0], the performance only varies in a small range, indicating the robustness to different choices of λ . Similarly, when progressively increasing threshold T in UAM from 16 to 1,024, our method consistently performs well and achieves the best when T = 64 under K = 2 setting. Thus, we set λ 1 = 0 . 4 , λ 2 = 0 . 5 and T to 64 under all settings.

20

10

0

10

20

20

10

0

10

20

(a) Source Only

20

10

0

10

20

20

10

0

10

20

(b) MMD [38]

20

10

0

10

20

30

20

10

0

10

20

(c) DADA [22]

30

20

10

0

10

20

30

20

10

0

10

20

(d) Our method

Relevant video

for that query:

Query:

1

12

21

Woman and girl are swimming

in swimming pool together

Add water to the pot

Rank of that video:

2

11

25

Relevant video

for that query:

Query:

An animal comes close

to another in the  jungle

A man dressed in red is

dancing

Rank of that video:

2

17

24

A woman is cutting

bread

Mt

→

Tf

Large crowd gathered  in car

workshop with any cars around

1

12

23

1

13

19

2

19

28

There is lady is riding

through the sea waves

→𝐌𝐭

A boy and a girl are

driving a car

2

16

26

A group of young people

are practicing  boxing

3

12

24

Mt

→

A man is glaring, and

someone with

sunglasses appears.

2

16

25

A cat tries to catch a

mouse on a tablet

1

13

26

→

Tf

A group of girls perform a

swimming show

4

13

10

8

← 𝟐𝟐

/

1

Tf

→

Mt

Tf

→𝐌𝐝

Feature visualizations. We randomly choose 100 pairs in source and target domain respectively, and show the t-SNE [53] visualizations of Source Only, MMD, DADA and our method under Tf → Mt setting in Figure 5. Blue/red denotes source/target domain, while circles/triangles denote videos/texts. Clearly, Source Only method performs the worst as the blue and red features fail to mix up due to the large domain gaps, whereas (b) and (c) can both to some extent alleviate the domain shift. However, they still show a distinct distributional gap between different features, which can be owed to the multimodal misalignment in target domain. Obviously, our method progressively generates truly aligned video-text pairs in target domain.

Qualitative Results. In Figure 6, we further visualize the returned results across different settings and baselines. Specifically, we show how the rank of relevant videos changes with different methods when given a query text. It is clear that our method leads to higher ranks of relevant videos compared to Source Only and ACP. One limitation is that our method performs worse given the query ' There is lady is riding through the sea waves ', i.e., our method returns the relevant video with rank 10 while ACP with rank 8. One possible reason might be the high similarity of RGB color between foreground lady and background sea waves.

5 Conclusion

In this paper, we focus on the notoriously challenging task, i.e., UDA Video-text Retrieval (UDA VR), which can be classified as a cross-modal cross-domain retrieval task. To tackle the one-to-many issue, we propose the Uncertainty-aware Alignment Network (UAN), which progressively generates truly aligned target pairs and ensures the discriminability of target features. Extensive experiments and ablation studies across several benchmarks justify the superiority of our method.

Limitations and Societal Impacts. One limitation is the difficulty in handling the one-to-many issue at the early training stage. As the massive data (both videos and texts) in target domain have no semantic relations with each other, especially in the first few epochs, it is probably inaccurate when selecting the truly positive pairs. Besides, a simple and effective UDA Video-Text Retrieval

(UDAVR) method is crucial for reduce the huge performance gap between source and target data. We hope that our method can pave the way for effective and robust video-text retrieval system.

References

S. Albanie, Y. Liu, A. Nagrani, A. Miech, E. Coto, I. Laptev, R. Sukthankar, B. Ghanem, A. Zisserman, V. Gabeur, C. Sun, K. Alahari, C. Schmid, S. Chen, Y. Zhao, Q. Jin, K. Cui, H. Liu, C. Wang, Y. Jiang, and X. Hao. The end-of-end-to-end: A video understanding pentathlon challenge (2020). arXiv preprint arXiv:2008.00744 , 2020.

F. Bianchi, G. Attanasio, R. Pisoni, S. Terragni, G. Sarti, and S. Lakshmi. Contrastive languageimage pre-training for the italian language. arXiv preprint arXiv:2108.08688 .

S.-V. Bogolin, I. Croitoru, H. Jin, Y . Liu, and S. Albanie. Cross modal retrieval with querybank normalisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5194-5205, 2022.

W.-L. Chao, H. Hu, and F. Sha. Cross-dataset adaptation for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5716-5725, 2018.

J. Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang. Learning the best pooling strategy for visual semantic embedding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15789-15798, 2021.

Q. Chen, Y. Liu, and S. Albanie. Mind-the-gap! unsupervised domain adaptation for text-video retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 1072-1080, 2021.

S. Chen, Y. Zhao, Q. Jin, and Q. Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10635-10644, 2020.

T.-H. Chen, Y.-H. Liao, C.-Y. Chuang, W.-T. Hsu, J. Fu, and M. Sun. Show, adapt and tell: Adversarial training of cross-domain image captioner. In Proceedings of the IEEE/CVF international conference on computer vision , pages 521-530, 2017.

X. Chen, H. Fang, T. Lin, R. Vedantam, S. Gupta, P. Dollár, and C. L. Zitnick. Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.00325 , 2015.

I. Croitoru, S.-V . Bogolin, M. Leordeanu, H. Jin, A. Zisserman, S. Albanie, and Y . Liu. Teachtext: Crossmodal generalized distillation for text-video retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 11583-11593, 2021.

Y. Dai, J. Liu, Y . Sun, Z. Tong, C. Zhang, and L.-Y . Duan. Idm: An intermediate domain module for domain adaptive person re-id. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

B. B. Damodaran, B. Kellenberger, R. Flamary, D. Tuia, and N. Courty. Deepjdot: Deep joint distribution optimal transport for unsupervised domain adaptation. In European Conference on Computer Vision , pages 467-483, 2018.

J. Dong, Z. Long, X. Mao, C. Lin, Y. He, and S. Ji. Multi-level alignment network for domain adaptive cross-modal retrieval. Neurocomputing , pages 207-219, 2021.

H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image CLIP. arXiv preprint arXiv:2106.11097 , 2021.

V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In European Conference on Computer Vision , pages 214-229, 2020.

Y. Ganin and V. S. Lempitsky. Unsupervised domain adaptation by backpropagation. In Proceedings of the International Conference on Machine Learning , pages 1180-1189, 2015.

Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and V. Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning research , pages 2096-2030, 2016.

S. K. Gorti, N. Vouitsis, J. Ma, K. Golestan, M. Volkovs, A. Garg, and G. Yu. X-pool: Crossmodal language-video attention for text-video retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5006-5015, 2022.

S. Guadarrama, N. Krishnamoorthy, G. Malkarnenkar, S. Venugopalan, R. Mooney, T. Darrell, and K. Saenko. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2712-2719, 2013.

X. Han, L. Yu, X. Zhu, L. Zhang, Y.-Z. Song, and T. Xiang. Fashionvil: Fashion-focused vision-and-language representation learning. In European Conference on Computer Vision , pages 634-651, 2022.

X. Hao, W. Zhang, D. Wu, F. Zhu, and B. Li. Listen and look: Multi-modal aggregation and co-attention network for video-audio retrieval. In International Conference on Multimedia and Expo , pages 1-6, 2022.

X. Hao, W. Zhang, D. Wu, F. Zhu, and B. Li. Dual alignment unsupervised domain adaptation for video-text retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18962-18972, 2023.

X. Hao, Y. Zhou, D. Wu, W. Zhang, B. Li, and W. Wang. Multi-feature graph attention network for cross-modal video-text retrieval. In International Conference on Multimedia Retrieval , pages 135-143, 2021.

X. Hao, Y. Zhou, D. Wu, W. Zhang, B. Li, W. Wang, and D. Meng. What matters: Attentive and relational feature aggregation network for video-text retrieval. In International Conference on Multimedia and Expo , pages 1-6, 2021.

X. Hao, Y. Zhu, S. Appalaraju, A. Zhang, W. Zhang, B. Li, and M. Li. Mixgen: A new multi-modal data augmentation. In Winter Conference on Applications of Computer Vision Workshops , pages 379-389, 2023.

X. Huang and Y. Peng. Deep cross-media knowledge transfer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8837-8846, 2018.

P. Jin, J. Huang, P. Xiong, S. Tian, C. Liu, X. Ji, L. Yuan, and J. Chen. Video-text as game players: Hierarchical banzhaf interaction for cross-modal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 2472-2482, 2023.

G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann. Contrastive adaptation network for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4893-4902, 2019.

K. Lee, X. Chen, G. Hua, H. Hu, and X. He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision , pages 201-216, 2018.

D. Li, J. Li, H. Li, J. C. Niebles, and S. C. H. Hoi. Align and prompt: Video-and-language pre-training with entity prompts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4943-4953, 2022.

K. Li, Y. Zhang, K. Li, Y. Li, and Y. Fu. Visual semantic reasoning for image-text matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 4654-4662, 2019.

S. Li, M. Xie, F. Lv, C. H. Liu, J. Liang, C. Qin, and W. Li. Semantic concentration for domain adaptation. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9102-9111, 2021.

Y. Li, Y. Song, L. Cao, J. Tetreault, L. Goldberg, A. Jaimes, and J. Luo. Tgif: A new dataset and benchmark on animated gif description. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 4641-4650, 2016.

K. Q. Lin, J. Wang, M. Soldan, M. Wray, R. Yan, E. Z. XU, D. Gao, R.-C. Tu, W. Zhao, W. Kong, et al. Egocentric video-language pretraining. Advances in Neural Information Processing Systems , pages 7575-7586, 2022.

Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman. Use what you have: Video retrieval using representations from collaborative experts. arXiv preprint arXiv:1907.13487 , 2019.

Y. Liu, H. Chen, L. Huang, D. Chen, B. Wang, P. Pan, and L. Wang. Animating images to transfer CLIP for video-text retrieval. In International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 1906-1911, 2022.

Y. Liu, Q. Chen, and S. Albanie. Adaptive cross-modal prototypes for cross-domain visuallanguage retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14954-14964, 2021.

M. Long, Y. Cao, J. Wang, and M. Jordan. Learning transferable features with deep adaptation networks. In International conference on machine learning , pages 97-105, 2015.

M. Long, Z. Cao, J. Wang, and M. I. Jordan. Conditional adversarial domain adaptation. In Annual Conference on Neural Information Processing Systems , pages 1647-1657, 2018.

M. Long, H. Zhu, J. Wang, and M. I. Jordan. Deep transfer learning with joint adaptation networks. In Proceedings of the International Conference on Machine Learning , pages 22082217, 2017.

X. Long, C. Gan, G. D. Melo, J. Wu, and S. Wen. Attention clusters: Purely attention based local feature integration for video classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7834-7843, 2018.

H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing , pages 293-304, 2022.

Y. Ma, G. Xu, X. Sun, M. Yan, J. Zhang, and R. Ji. X-CLIP: end-to-end multi-grained contrastive learning for video-text retrieval. In ACM International Conference on Multimedia , pages 638-647, 2022.

F. Markatopoulou, D. Galanopoulos, V. Mezaris, and I. Patras. Query and keyframe representations for ad-hoc video search. In Proceedings of the ACM on International Conference on Multimedia Retrieval , pages 407-411, 2017.

B. Pan, Z. Cao, E. Adeli, and J. C. Niebles. Adversarial cross-domain action recognition with co-attention. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 11815-11822, 2020.

Y. Pan, T. Yao, Y. Li, Y. Wang, C.-W. Ngo, and T. Mei. Transferrable prototypical networks for unsupervised domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2239-2247, 2019.

J. Pont-Tuset, J. R. R. Uijlings, S. Changpinyo, R. Soricut, and V. Ferrari. Connecting vision and language with localized narratives. In European Conference on Computer Vision , pages 647-664, 2020.

X. Song, S. Zhao, J. Yang, H. Yue, P. Xu, R. Hu, and H. Chai. Spatio-temporal contrastive domain adaptation for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9787-9795, 2021.

B. Sun and K. Saenko. Deep CORAL: correlation alignment for deep domain adaptation. In European Conference on Computer Vision , volume 9915, pages 443-450, 2016.

Y.-H. Tsai, W.-C. Hung, S. Schulter, K. Sohn, M.-H. Yang, and M. Chandraker. Learning to adapt structured output space for semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7472-7481, 2018.

Y.-H. Tsai, K. Sohn, S. Schulter, and M. Chandraker. Domain adaptation for structured output via discriminative patch representations. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 1456-1465, 2019.

E. Tzeng, J. Hoffman, N. Zhang, K. Saenko, and T. Darrell. Deep domain confusion: Maximizing for domain invariance. arXiv preprint arXiv:1412.3474 , 2014.

L. van der Maaten and G. Hinton. Visualizing data using t-SNE. Journal of Machine Learning Research , 2008.

X. Wang, L. Zhu, and Y. Yang. T2VLAD: global-local sequence alignment for text-video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5079-5088, 2021.

M. Wray, H. Doughty, and D. Damen. On semantic similarity in video retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3650-3660, 2021.

W. Wu, H. Luo, B. Fang, J. Wang, and W. Ouyang. Cap4video: What can auxiliary captions do for text-video retrieval? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10704-10713, 2023.

S. Xiao, M. Farajtabar, X. Ye, J. Yan, X. Yang, L. Song, and H. Zha. Wasserstein learning of deep generative point process models. In Advances in Neural Information Processing Systems , pages 3247-3257, 2017.

S. Xie, Z. Zheng, L. Chen, and C. Chen. Learning semantic representations for unsupervised domain adaptation. In International conference on machine learning , pages 5423-5432, 2018.

J. Xu, T. Mei, T. Yao, and Y. Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5288-5296, 2016.

M. Yang, W. Zhao, W. Xu, Y. Feng, Z. Zhao, X. Chen, and K. Lei. Multitask learning for cross-domain image captioning. IEEE Transactions on Multimedia , 21(4):1047-1061, 2018.

J. Yuan, Z. Zha, Y. Zheng, M. Wang, X. Zhou, and T. Chua. Utilizing related samples to enhance interactive concept-based video search. IEEE Transactions on Multimedia , pages 1343-1355, 2011.

W. Zhao, X. Wu, and J. Luo. Cross-domain image captioning via cross-modal retrieval and model adaptation. IEEE Transactions on Image Processing , pages 1180-1192, 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on text-video retrieval for the challenging unsupervised domain adaptation setting. For this problem, the model is trained on a source only set of supervised video/text pairs and is adapted to a target domain which consists of videos and text labels with no pairing ground truth. The proposed method, named Uncertainty-aware Alignment Network (UAN), uses the assumption that many videos can correspond to a textual item and vice-versa, which allows for the multi-granularity modelling. They evaluate their method on three different pairs of datasets between MSR-VTT, MSVD, and TGIF. Their method is found to outperform all other unsupervised domain adaptation methods for video retrieval because they adapt classifier domain adaptation approaches.\n\nSTRENGTHS:\nThe experiments that have been run are convincing and thorough and even include evaluation of the method for image-text domain adaptation which is nice to see and proves that the proposed approach was created from the ground up for embedding spaces instead of borrowed from unsupervised domain adaptation approaches for classification. As a final note, the experiments were all run 5 times and averaged for robustness.\n\nThe proposed model works well for unsupervised domain adaptation video retrieval and makes sense to break from the one-to-one assumption during the training to better find potential positives.\n\nWEAKNESSES:\nWhat are the values of a and b given in equation 7/8 and how are these chosen?\n\nEquations 10 and 11 could be better introduced/explained, additionally, it's easy to miss that T and K refer to the batch and for an individual example respectively.\n\nThe discussion on failure cases of the model within the paper is rather short. There is one given failure case within figure 6 which is very briefly explained (so much so that I'm not sure how the model failed on this case). I think these could be better highlighted as another future avenue for related work and better understanding of the method as a reader.\n\nThe method calls for many-to-one and one-to-many relationships between the two modalities during training, but for evaluation it is assumed that there is only a one-to-one relationship between modalities as with previous work. There has been some work that mentions the possibility of many-to-many relationships during training for images [a] and videos [b] and it might be worth discussing this point within the limitations which also mentions the one-to-many relationships at the early training stage.\n\n[a] Chun, Sanghyuk, et al. \"Probabilistic embeddings for cross-modal retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n[b] Wray, Michael, Hazel Doughty, and Dima Damen. \"On semantic similarity in video retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nOther Comments\nLine 6: missing space between comma and 'as'\n[24] and [25] are duplicated citations\nLine 142: inner production -> inner product\n\nQUESTIONS:\n1. Can the failure cases of the method be expanded and more information given regarding the failure case in Figure 6\n2. What are the values of a and b given in equation 7/8 when training the model and how are these chosen?\n3. With the domain adaptation requiring the one-to-one relationship between video and text during evaluation to be relaxed into a many-to-many relationship during training, do you think this has a method on overall method performance?\n\nLIMITATIONS:\nLimitations are discussed within the paper in the final paragraph, however, not much is said for the method beyond selecting pairs in the first few epochs likely causes issues. I think this could be expanded further to round out the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the task of unsupervised domain adaptation for text-video retrieval. The authors propose an Uncertainty-aware Alignment Network (UAN), which exploits the semantic information of both modalities in target domain. Specifically, in order to tackle the one-to-many relationships in the target domain, the proposed Uncertainty-aware Alignment Mechanism (UAM) tries to utilize the multi-granularity relationships between each target video and text to ensures the discriminability of target features. Finally, the authors test their proposed method on several benchmarks.\n\nSTRENGTHS:\nThe paper tackles an important problem and obtains good results. The results can be of high interest for the research community in order since the method seems to be relatively easy to combine with other existing methods, however I think some more clarifications are needed on this front.\n\nWEAKNESSES:\nI think some parts of the paper need more explanations (see questions below). Also, for completeness, I think it would be useful to also briefly describe what happens at inference.\n\nMissing papers in text-video retrieval related work\n\n[1] Gorti, Satya Krishna, et al. \"X-pool: Cross-modal language-video attention for text-video retrieval.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022.\n\n[2] Bogolin, Simion-Vlad, et al. \"Cross modal retrieval with querybank normalisation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[minor] typo line 155 \"to to\"\n\nQUESTIONS:\n1. How are the results for CE reported for text-image retrieval since as far as I know the method only reports results on text-video retrieval? Do you re-train the method on images? What input features do you use for images?\n\n2. Same question for methods in Tab 5? Do you evaluate using the published weights? Re-train? Can an already trained model be adapted to a different domain without re-training? More details are needed\n\n3. Can you give more details on what is needed to combine the proposed method with other existing methods? What's the computational overhead? Are there any additional costs during inference? etc.\n\nLIMITATIONS:\nLimitations are discussed thorough the paper and societal impact is also discussed briefly.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper proposed a method named Uncertainty-aware Alignment Network (UAN) to address the one-to-many issue which means in the real scene, one text usually corresponds to multiple videos and vice versa. Specifically, the proposed method achieves a new state-of-the-art in Cross-Domain Video-Text Retrieval through the incorporation of design elements such multi-modal mutual information module and Uncertainty-aware Alignment Mechanism (UAM).

STRENGTHS:
1.	The motivation for addressing “one-to-many in target domain” is clearly presented and validated.
2.	The ablation study and generalization experiment in this paper is comprehensive and clearly analyzed, providing strong evidence of the validity of the proposed model.

WEAKNESSES:
1.	It is recommended to add baseline performance in Fig 4 to reflect the robustness of the proposed module.
2.	The authors used each batch in the Uncertainty-Aware Alignment Mechanism to obtain self-discovered matching pairs instead of all data, so will the different number of batch have a significant impact on the effectiveness?
3.	The phrase "the confusion of the lady and sea waves" mentioned in the Qualitative Results section is ambiguous. Further analysis is recommended in this section to clarify this point.

QUESTIONS:
N/A

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a unsupervised domain adaptation method tailored for video-text retrieval by exploring the one-to-many correspondences between video and text on the target domains. The proposed method is shown effective and superior to existing approaches which considered only the one-to-one video-text matching relationships on the unlabelled target domains.\n\nSTRENGTHS:\n+ The idea to learn from the one-to-many matching relationships among video and text in real-world scenarios is intuitive, the proposed model is straighforward\n+ The proposed method not only yielded impressive performance on video-text retrieval, the authors further demonstrate its beneficial to image-text retrieval. Moreover, the authors also proved that the proposed idea is generally beneficial to the task by combining it with a wide range of existing methods.\n\nWEAKNESSES:\n+ The motivations are not sufficiently discussed. The two main motivations claimed in the paper is (1) existing UDAVR approaches are mostly derived from classification based DA methods which is not optimal for retrieval tasks and (2) existing methods assume a one-to-one matching relationships between video and text. For (1), it may be more straightforward to spell out the major drawbacks of adapting classification DA methods to UDAVR rather than just stating it is suboptimal. The motivation (2) should be justified in terms of whether the one-to-many matching relationships are common on the adopted training datasets. It will be more intuitive if this can be quantified.\n+ I'm not sure I fully understand what is the meaning of \"to balance the minimization of domain shift\" at L12\n+ As stated at L36, the definition of UDAVR is that there is no identical textual labels on the two domains. However, two sentences even from the same domains are not likely to be exactly identical but they can describe the same events using a similar vocabulary. Therefore, does it make more sense to measure how different are two video-text retrieval datasets by the overlaps of their vocabularies?\n+ In Fig.1, I failed to find the differences between existing methods which are claimed to be classification-based and the proposed method designs for retrieval tasks (L55)\n+ In Eq.(7)-(9), as a \"negative scale factor\", $a$ seems to be a constant, then how about $b$? is it also a constant? If yes, then why adding a constant in a loss function to be optimised?\n+ Are the symbols $S^{T\\rightarrow V}$ and $S^{V\\rightarrow T}$ used in Eq.(10) and Eq.(11) defined?\n+ minor problems (typos and etc):\n  - duplicated \"to\" at L12, L155\n  - \"similarity Top-K\" in Eq.(12)\n\nQUESTIONS:\n+ It will be interesting to see how many discovered one-to-many matching relationships are true and consistent with the manual labels on a validation set.\n+ In Fig.4, the model obtained its best performance when K=2, does this mean that each video is optimally to be matched with two text sentences? This seems to be applied to all the pairwise combination of datasets but it is not intuitive why a 1-to-2 mapping is always the best. adding more discussion and analysis might provide more insights.\n\nLIMITATIONS:\nYes, the author discussed one of the limitations of their proposed method is the false positive inter-sample relationships discovered for training, especially at the early stage.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the unsupervised domain adaptation video-text retrieval problem by two proposed components, the D-VLDA (Distribution-based Vision-Language Domain Adaptation) and the UAM (Uncertainty-Aware Alignment). The proposed D-VLDA aims at alleviating the domain discrepancy via moment-based method, while the UAM method aims for better pseudo labeling.\n\nSTRENGTHS:\n(+) This article is well written, and the description of the method pipeline is clear and easy to understand. \n\n(+) Experiments are sufficient and ablation studies show the effectiveness of the proposed components.\n\n(+) Stat-of-the-art results are demonstrated on several datasets.\n\nWEAKNESSES:\n(-) Unfair comparison in table 3, see in Question(2). \n\n(-) The \"one-to-many\" assumption that underpins the article lacks empirical evidence supporting its widespread occurrence.\n\nQUESTIONS:\n(1) The term \"(moment-based) D-VLDA module\" appears to be referred to as \"multi-modal mutual information module\" in the abstract. Could the author please clarify the relationship between these two concepts?\n\n(2) In the design of Uncertainty-Aware Alignment(UAM), the authors expand the top-1 dual alignment method from [A] to a top-k dual alignment. It appears that Table 3 aims to validate this expansion, where \"UAN(w/DAC)\" denotes the application of the top-1 dual alignment consistency mechanism to pick those most similar videos and texts in the target domain, as described in [A]. However,  I note that the results for \"UAN(w/DAC)\" match those reported in [A], while [A] utilizing a different discrepancy loss from D-VLDA. This may lead to an unfair comparison. I would suggest the authors control the variables to ensure a more equitable comparison.\n\n(3) The authors propose the uncertainty-aware alignment network based on the \"one-to-many\" assumption: one text can describe multiple videos and vice versa. This forms the fundamental premise of the paper. However, the frequency of the \"one-to-many\" phenomenon remains unclarified. I'm concerned that the efficacy of the method may not be primarily due to the \"one-to-many\" assumption, but potentially other factors like the top-K methods introduce more true positive pairs in the early training stage, providing the model with more supervision about the target domain. I think the author should verify the frequent existence of the \"one-to-many\". For example, collecting the top-K dual alignment results by the final model, removing the pairs that belong to the ground truth pairs, and then see the proportion of the remaining pairs to all the ground truth pairs, and also estimate how many of the left pairs match correctly (belongs to the \"one-to-many\").\n\n[A] X. Hao, W. Zhang, D. Wu, F. Zhu, and B. Li. Dual alignment unsupervised domain adaptation for video-text retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023.\n\nLIMITATIONS:\nNo", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}