Title: Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval
Venue: NeurIPS 2023
Accepted: yes
Keywords: multimodal learning

cross-modal retrieval

robust learning

uncertainty
Abstract: Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity. Concretely, we first construct a set of various learnable prototypes for each modality to represent the entire semantics subspace. Then Dempster-Shafer Theory and Subjective Logic Theory are utilized to build an evidential theoretical framework by associating evidence with Dirichlet Distribution parameters. The PAU model induces accurate uncertainty and reliable predictions for cross-modal retrieval. Extensive experiments are performed on four major benchmark datasets of MSR-VTT, MSVD, DiDeMo, and MS-COCO, demonstrating the effectiveness of our method. The code is accessible at https://github.com/leolee99/PAU.

Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval

Abstract

Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e . g ., corrupt images, fast-paced videos, and non-detailed texts. In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity. Concretely, we first construct a set of various learnable prototypes for each modality to represent the entire semantics subspace. Then Dempster-Shafer Theory and Subjective Logic Theory are utilized to build an evidential theoretical framework by associating evidence with Dirichlet Distribution parameters. The PAU model induces accurate uncertainty and reliable predictions for cross-modal retrieval. Extensive experiments are performed on four major benchmark datasets of MSR-VTT, MSVD, DiDeMo, and MS-COCO, demonstrating the effectiveness of our method. The code is accessible at https://github.com/leolee99/PAU .

1 Introduction

Cross-modal Retrieval, devoted to searching a gallery of related samples in vision/language modality given a query in another, has attracted increasing interest as the wellspring development of multimedia platforms. However, the heterogeneity between modalities casts daunting challenges for this task. To tackle these challenges, most existing methods [9, 60, 19, 15, 4, 53, 49] map the query and retrieval items from distinct modalities into a joint latent space to make similarity calculation feasible.

Some early works [9, 60] corroborate the powerful capabilities of deep neural networks to extract high-level features. Chen et al . [9] construct a hierarchical graph structure for videos and texts to build the relationships between the local and the global features. PVSE [60] utilizes the multi-head selfattention module to learn multiple and diverse representations of videos and texts for the polysemous problem. Recently, a large body of transformer-based pre-training models [19, 15, 4, 53, 49] have shown remarkable superiority with excellent generalization and performance.

However, prior approaches routinely assume the data qualities are absolutely stable and view them equally. They estimate the relevance of vision-language pairs relying only on the similarity scores

Video B

Video A

Video A

Video B

K

0.5

(a) Video modality

Text A

Text B

A man is singing on stage to a

huge audience he is holding a

microphone.

Video Gallery

A man is singing.

Positive

Negative

Semantic Similarity Distribution

K

Semantics Catergory

Cosine Similarity

Text A

Text B

0.5

(b) Text modality

generated by complex neural networks, but ignoring the confidence of these predictions, which should be taken seriously. Low-quality data like corrupt images [31], fast-paced videos [26], non-detailed sentences [34], etc ., inevitably lead to unreliable results. The model therefore not only needs to predict the relevance of vision-language pairs, but also capably answer the question ' How confident is the prediction? '. Namely, it is necessary to quantify the uncertainty of each data to guide more reliable vision-language pair similarity.

In the literature, uncertainty can be classified into Aleatoric Uncertainty and Epistemic Uncertainty based on causation [32]. The former is induced by the inherent data defects, while the latter is inherent to the model. This work mainly focuses on the aleatoric uncertainty ( i . e ., data uncertainty), defined as 'the uncertainty caused by ambiguous multi-modal data, such as fast-paced videos and non-detailed texts'. Next, we will answer the question ' Why do these ambiguous data lead to uncertainty? '. We first assume the possible semantics of each modal subspace are finite with K categories. Afterward, the fast-paced videos (multi-scene videos) and the non-detailed texts are generally highly related to multiple semantics, misleading models into confused matching, shown in the Figure 1. These alternative matches will bring uncertainty, which can be explained through Information Entropy [59]. Concretely, Information Entropy is the average uncertainty level in the random variable's possible outcomes, which can be denoted as Eq. 1 with the possible semantics as the random variable X .

where ∑ K i =1 p ( x i ) = 1 , p ( x i ) is the i th semantics probability derived from semantic cosine similarity through a softmax layer. The ambiguous data highly associated with multiple semantics have a low variance of the probability distribution, resulting in high information entropy and high uncertainty due to ' The Maximum Entropy Principle [33]' (proof in Appendix. E). Information entropy is a seemingly reasonable criterion to estimate the data uncertainty, but unfortunately, it is powerless to distinguish the instances with the similarity distribution of [0 . 8 , 0 . 8 , 0 . 8 , 0 . 8] and the distribution of [0 . 2 , 0 . 2 , 0 . 2 , 0 . 2] . After a softmax function, both distributions turn into [0 . 25 , 0 . 25 , 0 . 25 , 0 . 25] and obtain the same information entropy. In practice, we hope the first one gets a higher uncertainty score than the other because the front of the retrieval sequence with highly similar instances is more notable for the retrieval task. Thus, the Dempster-Shafer Theory of Evidence (DST) [58], a generalization of the Bayesian theory to subjective probabilities, which has developed into a general framework for uncertainty modeling, is leveraged to quantify multi-modal aleatoric uncertainty.

To precisely quantify the aleatoric uncertainty, in this work, we propose an effective Prototypebased Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from inherent data ambiguity. To be specific, we first construct a series of diverse prototypes for each modality to capture the overall K semantics of the subspace. Each prototype represents an individual semantics. Afterward, the variations of cosine similarities

between the instance and the prototypes from another modality are deployed as the belief masses used in DST [59] to model uncertainty. Finally, a re-ranking practice is utilized to further strengthen prediction reliability by regarding each data uncertainty as a similarity weight.

The main contributions of our paper are three folds. 1) We give a clear definition of the inherent aleatoric uncertainty in multi-modal data. 2) A powerful aleatoric uncertainty quantification framework PAU is built to accurately estimate the data uncertainty and effectively mitigate the negative impact on retrieval. 3) We also verify the superiority of our proposed PAU on multiple challenging benchmarks. Massive insightful ablation studies further reveal its effectiveness and generalization.

2 Related work

2.1 Cross-modal Retrieval

Cross-modal retrieval [9, 25, 67, 39, 17, 60, 54, 49, 22, 18, 66, 26, 64, 37], as one of the most popular tasks in multi-modal learning, has attracted increasing attention with the aim of searching similar semantic samples from different modalities. Early works [25, 39] build a joint embedding space to make query-positive pairs closer by canonical correlation analysis (CCA) [28]. Faghri et al . [17] first use a triplet loss to improve the model performance by focusing only on the hardest negative. PVSE [60] proposes a multi-candidate representation function to capture the diversity and build one-to-many relationships in different modalities. Recently, CLIP [54] has shown vital power and generalization in various multi-modal tasks. A rich body of CLIP-based approaches [49, 22, 18, 66, 26] have got significant achievements in cross-modal retrieval. CLIP4Clip [49] transfers the rich knowledge from image-text pre-trained model CLIP to video-text retrieval. CenterCLIP [66] proposes a multi-segment token clustering algorithm to keep the most representative tokens in consecutive frames. In this paper, we follow the setting of the above work, which inherits the knowledge from CLIP.

2.2 Uncertainty Quantification

Generally, there are two paradigms among existing uncertainty estimation approaches. The first one is the Bayesian-based paradigm [5, 51, 40], which estimates uncertainty by approximating the moments of the posterior predictive distribution. Subsequently, some algorithms focus on the variety of Bayesian such as Laplace approximation [50], Markov Chain Monte Carlo [52], and variational techniques [55, 6]. Recent work [21] leverages dropout [61] in the test phase, reducing the computational cost of uncertainty estimation. Nonetheless, limited by the increment of model parameters and poor convergence, these approaches always have expensive training costs and slow inference time. The second paradigm focuses on the non-Bayesian approach [41, 57, 27]. Lakshminarayanan et al . [41] propose an alternative to Bayesian Neural Networks (BNN) which is simple to train and produce reliable uncertainty estimates. Wang et al . [57] replace the parameter set of a categorical distribution with the parameters of a Dirichlet density and represent the uncertainties with a single abnormal class. Recently, uncertainty-based algorithms have been used in multi-view classification for trusted decision-making [27]. Although uncertainty-based methods have achieved impressive progress, they are limited to working with particular classification and segmentation tasks.

3 Method

In this section, we present our Prototype-based Aleatoric Uncertainty Quantification (PAU) framework (Figure 2). We first define the cross-modal retrieval task and introduce a commonly used pipeline in Section 3.1. Next, we detail the uncertainty theory, as well as the approach to quantify the uncertainty in Section 3.2.

3.1 Task Definition

Let D = ( V , T ) denote a vision and language dataset, where V is a set of images or videos, and T is a set of texts. The goal of cross-modal retrieval is to rank the relevance between a visual query v ∈ V (respectively a textual query t ∈ T ) and textual set T (respectively visual set V ). Recent works [54, 49, 22, 18, 66, 26] have shown CLIP's strong performance and generalization in various downstream tasks, inspiring us to employ CLIP as our backbone.

mean,

dim=0

mean,

dim=1

Diversity

Loss

Diversity

Loss

Uncertainty Loss

Uncertainty Loss

T

V

Visual Prototypes

(K, D)

(K, D)

Visual Instances

Textual Instances

Uncertainty

Modeling

Uncertainty

Modeling

(N)

(N)

(N, D)

Visiual

Encoder

Textual

Encoder

The man is

making a sauce

in the kitchen.

(N, D)

v

U

v

H

t

U

t

H

t

Z

v

Z

M

dot

dot

To be specific, given a visual instance v and a text instance t , the CLIP encodes them into a latent embedding space ❘ D through a visual encoder ϕ v ( · ) and a textual encoder ϕ t ( · ) , respectively. Then the relevance between them can be computed using a cosine similarity function as:

3.2 Uncertainty Quantification

Prototype construction. Although the cosine similarity function has shown effectiveness in most existing works [25, 60, 54, 49], some unreliable results will inevitably be produced due to the low inherent data quality. These ambiguous data are generally highly related to multiple semantics, leading to confusing matching (see Figure 1).

To assess these data uncertainty and provide trustworthy predictions, we first assume the semantic space is finite with K semantic categories. Then, K learnable prototypes Z ∈ ❘ K × D are constructed to represent the overall latent semantic subspace for each modality, making data semantic ambiguity evaluable. Each prototype represents an individual semantics. Finally, DST [12] is employed to build the uncertainty quantification framework by assigning the variations of similarities between an instance and the prototypes from another modality as the belief masses.

The theory of uncertainty. The Dempster-Shafer Theory of Evidence (DST) [12], a generalization of the Bayesian theory to subjective probabilities [12], builds a general uncertainty modeling framework by assigning belief masses to the set of mutually exclusive possible states with one state expressing ' I am not sure '. Following EDL [57], we adopt Subjective Logic (SL) [35] to associate the belief distribution in DST with the parameters of the Dirichlet distribution. More precisely, SL provides an evidential theoretical framework to quantify the belief masses b k of different categories and overall uncertainty mass ψ for the K classification task. These K +1 mass values are all non-negative and sum up to one:

where b k ≥ 0 and ψ ≥ 0 denote the belief mass of the k th semantic category ( k = 1 , · · · , K ) and overall uncertainty mass, respectively. In our setting, ψ here means the uncertainty mass of the

event 'the data is ambiguous', which is negatively related to the degree of data ambiguity. Thus, the u = 1 -ψ = ∑ K k =1 b k is designed to express the aleatoric uncertainty defined in Section 1. In brief, the data with lower ψ is more ambiguous, which should be a high aleatoric uncertainty score as higher u . A belief mass b k for the k th semantic category is computed using the evidence for the category. We term the evidence as the amount of data supports for being classified into a certain category. Let e = [ e 1 , · · · , e k ] be an evidence vector derived from similarity vector p = [ p 1 , · · · , p k ] between an instance and K prototypes (evidence generation shown in Appendix. B), then the belief b k and the uncertainty u are signified as:

where α k = e k +1 denote the Dirichlet parameters associated with a belief mass assignment in SL, and S = ∑ K i =1 α k is referred to as the Dirichlet strength. e k is the evidence of a visual (or textual) instance matching the k th semantics category. Intuitively, massive evidence indicates considerably ambiguous data, resulting in high uncertainty, which brings out fallible predictions.

Uncertainty learning. After uncertainty modeling, how to force the model to learn uncertainty from data is worth considering (see Figure 2). The key is enriching prototypes' meaningful semantics to represent overall subspace rigorously. We first give a normalized initialization proposed by Xavier [24] to all prototypes. Subsequently, the cosine similarity matrix M ∈ ❘ N × N between N visual and textual instances is calculated by Eq. 2 within a batch. Furthermore, N instances and K prototypes from different modalities build a set of similarity vectors P ∈ ❘ N × K to derive the evidence vector set and compute the uncertainty using Eq. 4. The data uncertainty sets of N visual and textual instances can be symbolized as U v = [ u v 1 , · · · , u v N ] and U t = [ u t 1 , · · · , u t N ] in parallel.

Next, an objective function is required to force prototypes to learn rich semantic knowledge. As the prototypes are expected to represent the overall semantics of each modality, the instance with high overall similarity to these prototypes should also have high overall similarity with other instances from another modality. According to Eq. 4, the aleatoric uncertainty u , denoted as the sum of belief masses, should be positively related to the mean similarities of each modality within a batch. Thereby, we compute the mean of similarity matrix M along the row and the column, symbolized as H v = [ h v 1 , · · · , h v N ] and H t = [ h t 1 , · · · , h t N ] , respectively. After that, prototypes can be optimized by employing Mean-Squared (MSE) Loss to keep the U v ( U t ) and H v ( H t ) consistent, where λ is a scaling coefficient to keep the same value range of u and h (Eq. 5).

Diversity learning. To employ the DST uncertainty framework, the mutual exclusion among possible states needs to be satisfied. Precisely, prototypes should avoid attending to the same or similar semantics. Therefore, a diversity loss is proposed to help the prototypes focus on diversified independent semantics. The formula is as:

where z v i and z t i denote i th prototypes of visual and textual modality. Intuitively, the semantic similarities between different prototypes are expected to be 0.

Re-ranking. After training, the uncertainty of each instance could be precisely assessed using Eq. 4. To further alleviate the impact of the uncertainty on prediction, we re-rank the retrieval sequences by weighting the predicted matrix based on the uncertainty vector U v and U t by Eq. 7.

where β 1 and β 2 are learnable parameters used to control the impact degree of re-ranking on prediction. M ′ and M ′′ separately signify the similarity matrices between all visual and textual instances in test processing before and after re-ranking.

4 Experiments

In this section, we first introduce the datasets and metrics used for our experiments, as well as the implementation details. Later, we compare PAU with the previous state of the arts on several tasks and public benchmarks. In the end, rich ablation studies are carried out to verify our method's effectiveness and generalization.

4.1 Datasets and Metrics

We report our experimental results on three public video-text retrieval datasets containing MSRVTT [62], MSVD [7], DiDeMo [30], and a public image-text retrieval dataset of MS-COCO [45]. Following [49], Recall@1 (R@1), Recall@5 (R@5), Recall@10 (R@10), Median Rank (MdR), and Mean Rank (MnR) are used to evaluate the performance.

MSR-VTT [62] is a commonly used benchmark in video-text retrieval, containing 10,000 Youtube videos with 20 captions for each. Following [20], we employ 9,000 videos for training and 1,000 selected video-text pairs for testing.

MSVD [7] contains 1,970 videos ranging in length from 1 to 62 seconds, with approximately 40 sentences per video. There are 1,200, 100, and 670 videos used for training, validation, and testing in the standard split.

DiDeMo [30] consists of 10,611 Flickr videos with approximately 40,000 annotations. We follow [46, 42, 3] and concatenate all captions of the same video into a paragraph, which will be regarded as a single item in a paragraph-to-video retrieval task.

MS-COCO [45] is a widely used dataset for image-text retrieval, containing 123,287 images with five captions per image. We follow the split in [36] with 113,287 images for training, 5,000 for validation, and 5,000 for testing. The results are reported on 1K and 5K test sets.

4.2 Implementation Details

We use CLIP's [54] visual-textual encoder (ViT-B/32) as the backbone and initialize all parameters from pre-trained CLIP weights. The max token and frame length are both set consistently with [49]. Specifically, the dimension of common embedding space is set to 512. In video-text retrieval, the Adam optimizer [38] is employed with the initial learning rate of 1e-7 for the visual encoder and the textual encoder, as well as the initial learning rate of 1e-4 for other layers. As for image-text retrieval, the AdamW optimizer [48] updates the parameters of all modules with the initial learning rate of 1e-5. The learning rate decays complying with the cosine schedule strategy [47] in 5, 3, 20, and 5 epochs trained on MSR-VTT, MSVD, DiDeMo, and MS-COCO. Besides, the number of prototypes K is set as 8 according to Table 8. All experiments are conducted on 1 to 4 RTX3090.

4.3 Comparison with State of the Arts

Video-text retrieval. We compare our model with other state-of-the-art methods [49, 66, 18, 22, 26, 23, 54] and our baseline (model structure see Appendix. A) on multiple datasets. For fair comparisons, all methods are CLIP-based, symbolizing the most advanced techniques of video-text retrieval. The performance of baseline trained without PAU is also given. Table 1, 2, 3 show the results of video-text retrieval on MSR-VTT, MSVD, and DiDeMo, respectively.

We can observe that PAU brings sizeable improvements on all benchmarks. For text-to-video retrieval (t2v), PAU boosts the R@1 performance of 2.9%, 1.0%, and 1.6% on MSR-VTT, MSVD, and DiDeMo. Moreover, 3.1%, 8.8%, and 1.2% improvements of R@1 are obtained on video-to-text retrieval (v2t). Notably, we find that the performance improvement of v2t on MSVD is more significant than other benchmarks. The reason is that MSVD is a small dataset with few videos (only 1,200 for training) but massive annotations (about 40 annotations per video). Meanwhile, CLIP simply utilizes cross-entropy loss, which only regards the diagonal pairs as the positives. Nonetheless, for a batch with a typical size of 256 [49], the probability of any two texts coming from separate videos is approximately 3 . 49 × 10 -13 (see Appendix. D), bringing innumerable noises and performance damage for the training on video-to-text retrieval. Thus, the significant improvement indicates the ability of PAU to capture and rectify misleading data to support robust learning.

Image-text retrieval. We also test our method on image-text retrieval task. To be specific, CLIP [54] is utilized as the baseline. We compare our model with several previous state-of-the-art approaches, including PVSE [60], SGRAF [13], NAAF [65], DAA [43], VSE ∞ [8], PCME [11], and PCME++ [10]. Table 4 shows the results of image-text retrieval on MS-COCO 1K and MS-COCO 5K, respectively.

We can observe that the overall performance improvement of image-text retrieval is smaller than that of video-text retrieval. The phenomenon reflects that video information is more complex than image information, inducing immense uncertainty in video-text retrieval. Furthermore, we also validate the robustness of PAU training in noisy datasets. The detailed results can be found in Appendix. C.4.

4.4 Comparison with Probabilistic Model

PCME [11] is a probabilistic-based approach to represent instance diversity, which can also represent uncertainty. The more diverse the instance, the more uncertain the instance. Thus, an instance's uncertainty and diversity should be positively correlated. To verify the superiority of our SL-based method over than probabilistic-based method, we conduct two experiments, pearson correlation analysis and uncertain data analysis .

Pearson correlation analysis. The pearson correlation coefficient ( PCC ) r xy ( r xy ∈ [ -1 , 1] ) is a metric to estimate the strength of the correlation between variable x and variable y . Generally, r xy > 0 . 5 means that two variables are strongly positively correlated. As mentioned before, the uncertainty and the diversity of an instance should be positively correlated for PCME. If an instance is diverse, it will have high mean

similarity h with the instances from the other modality. This means that the uncertainty score u and mean similarity h should also be positively correlated for an instance. Thus, we compute the pearson correlation coefficient between u and h to explore whether PAU correctly estimates uncertainty. The results are shown in Table 5.

In this Table, we compute the PCCs of PCME and PAU on MSCOCO, as well as PCCs of PAU on MSR-VTT. PCC V and PCC T separately indicate the instance's PCC in vision or textual modality. The low PCCs of PCME mean current probabilistic models have limited abilities to reflect uncertainty and diversity. We argue the reason arises from the strong prior distribution assumption, i . e ., gaussian or other simple distributions are powerless to fit complex and diverse relationships in high-dimension space. By contrast, PAU obtains high PCCs in all settings, proving our approach indeed leads to proper uncertainty estimations.

Uncertain data analysis . To further prove the superiority of SL-based aleatoric uncertainty estimation method than probabilistic-based method, we compare the changes of R@1 after removing top-r instances with highest uncertainty scores predicted by PCME and PAU, respectively. To fairly compare, we employ removal on both predictions arised from PAU and PCME. The results with removed ratio from 0% to 20% on MS-COCO 5K is shown in Figure 3. We can observe that PAU outperforms PCME on all directions and predictions. This means that the uncertain data found by PAU is more precise than PCME. All experiments indicate the SL-based approach (PAU) works better than probabilistic-based method (PCME) on representing aleatoric uncertainty.

1000

2000

3000

4000

5000

r

0.37

0.38

0.38

0.39

0.39

0.40

R@1

PAU

PCME

(a) i2t on CLIP

200

400

600

800

1000

r

0.40

0.41

0.42

0.43

0.44

0.45

0.46

R@1

PAU

PCME

(b) t2i on CLIP

1000

2000

3000

4000

5000

r

0.42

0.42

0.43

0.43

0.44

0.44

R@1

PAU

PCME

(c) i2t on PCME

200

400

600

800

1000

r

0.28

0.29

0.29

0.30

0.30

0.31

0.31

0.32

R@1

PAU

PCME

(d) t2i on PCME

4.5 Ablation Studies

The effect of different components. To understand the effect of each component, we exhaustively ablate three main components of PAU, including Uncertainty Loss, Diversity Loss, and Re-ranking. As shown in Table 6, all three components effectively improve the model performance. The comparison between No. 1 and No. 3 shows that uncertainty loss gives significant R@1 improvements of 1.4% on t2v and 2.2% on v2t. The experiment No. 5 outperforms the experiment No. 3 by a clear margin, demonstrating the necessity of diversity loss. Furthermore, the experiment No. 2 re-ranks the baseline prediction using uncertainty vectors from PAU. It's worth noting that R@1 performance still boosts on both t2v and v2t, reflecting the strong generalization of PAU.

The impact of uncertain data. To demonstrate the uncertain data found by PAU indeed affect the retrieval, a trial of the retrieval subsets with varying reliability levels is carried out on MSR-VTT. Specifically, the samples are first ranked according to the uncertainty calculated by PAU in descending order. Then, r samples with the highest uncertainty will be removed from the test set to produce an uncertainty-based removal subset. Meanwhile, r random samples will be removed from the test set to produce a random-based removal subset. The baseline model is tested on both subsets above and reports the results in Figure 4.

It's worth noting that the model tested on the subsets removing uncertain data always outperforms the ones removing randomly. Furthermore, the performance gap reaches a peak of 14.2 when r = 600

0

200

400

600

800

1000

r

0.5

0.6

0.7

0.8

0.9

1.0

R@1

random-based

uncertainty-based

(a) Text-to-video

0

200

400

600

800

1000

r

0.5

0.6

0.7

0.8

0.9

1.0

R@1

random-based

uncertainty-based

(b) Video-to-text

0

200

400

600

800

1000

r

0.30

0.35

0.40

0.45

R@1

baseline

PAU

(a) Text-to-video

0

200

400

600

800

1000

r

0.30

0.35

0.40

0.45

R@1

baseline

PAU

(b) Video-to-text

A person is explaining

something.

A man is driving a black car.

Multi colored horses in a barn

and outside in the snow.

Text Uncertainty

Uncertainty: 0.984

Uncertainty: 0.421

Uncertainty: 0.039

Video Uncertainty

Uncertainty: 0.026

Uncertainty: 0.446

Uncertainty: 0.881

...

...

...

for text-to-video retrieval and a peak of 12.3 when r = 700 for video-to-text retrieval, respectively. The above phenomena prove PAU can indeed find uncertain data, which easily misleading the model into incorrect predictions, resulting in extremely unreliable retrieval results.

Is the prediction more trustworthy? To validate the effectiveness of PAU in providing trustworthy predictions, the performances of PAU and baseline are compared in the MSR-VTT's subsets under a series of different uncertain degrees. Precisely, r samples with the highest uncertainty are selected to form the uncertain test subsets. As shown in Figure 5, PAU consistently outperforms the baseline with the largest gap of 8.6 (5.8) while r = 70 ( r = 120 ) for t2v (v2t). The smaller the r is, the higher the total uncertainty of the subset is, so the results confirm the ability of PAU to decrease the performance loss caused by uncertain data. Furthermore, PAU can provide trustworthy predictions even in high uncertainty sets.

Visualization of data uncertainty. Three captions and three videos with their uncertainty scores are shown in Figure 6. Obviously, the caption with high uncertainty is more semantically ambiguous, such as 'A person is explaining something'. By contrast, the caption with low uncertainty is more detailed, such as 'Multi colored horses in a barn and outside in the snow'. Moreover, the first video (complete video shown in Appendix. F) is more difficult to summarize due to the varied scenes, inducing high uncertainty. By contrast, the third video with low uncertainty is relatively simple.

5 Conclusion

In this work, we clearly define the inherent aleatoric uncertainty in multi-modal data. A Prototypebased Aleatoric Uncertainty Quantification (PAU) framework is designed to excavate the uncertain data and further provide trustworthy predictions. Uncertainty loss and diversity loss are proposed to support the uncertainty training by encouraging the learnable prototypes to represent the semantic subspace of varied modalities. Plentiful empirical experiments on a range of tasks and public benchmarks demonstrate the effectiveness and generalization of PAU.

Broader impacts statement. Cross-modal matching has become a common paradigm in large-scale multi-modal pretraining, such as CLIP [54]. However, the enormous data demand brings a huge challenge to pre-trained models. If we can accurately select high-quality data, which typically make greater contributions to the performance of pre-trained models, the pretraining process would become much more efficient. This work could be one of the first work to be aware of the aleatoric uncertainty in cross-modal retrieval. By quantifying uncertainty, higher quality data and more reliable results are provided, which could perhaps be extended to other multi-modal tasks in the future [2, 44, 63].

References

H. Akbari, L. Yuan, R. Qian, W. Chuang, S. Chang, Y. Cui, and B. Gong. VATT: transformers for multimodal self-supervised learning from raw video, audio and text. In NeurIPS , pages 24206-24221, 2021. 16, 17

P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR , pages 6077-6086, 2018. 10

M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV , pages 1708-1718, 2021. 6

M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV , pages 1708-1718, 2021. 1, 17

A. Blum, N. Haghtalab, and A. D. Procaccia. Variational dropout and the local reparameterization trick. In NeurIPS , pages 2575-2583, 2015. 3

C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In ICML , volume 37, pages 1613-1622, 2015. 3

D. L. Chen and W. B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL , pages 190-200, 2011. 6

J. Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang. Learning the best pooling strategy for visual semantic embedding. In CVPR , pages 15789-15798, 2021. 7, 17

S. Chen, Y. Zhao, Q. Jin, and Q. Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR , pages 10635-10644, 2020. 1, 3

S. Chun. Improved probabilistic image-text representations. arXiv preprint arXiv:2305.18171 , 2023. 7, 17

S. Chun, S. J. Oh, R. S. de Rezende, Y. Kalantidis, and D. Larlus. Probabilistic embeddings for cross-modal retrieval. In CVPR , pages 8415-8424, 2021. 7, 8, 17

A. P. Dempster. A generalization of bayesian inference. Journal of the Royal Statistical Society: Series B (Methodological) , 30(2):205-232, 1968. 4

H. Diao, Y. Zhang, L. Ma, and H. Lu. Similarity reasoning and filtration for image-text matching. In AAAI , pages 1218-1226, 2021. 7

R. Durrett. Probability: theory and examples , volume 49. Cambridge university press, 2019. 18, 19

M. Dzabraev, M. Kalashnikov, S. Komkov, and A. Petiushko. MDMMT: multidomain multimodal transformer for video retrieval. In CVPR Workshops , pages 3354-3363, 2021. 1

D. M. Endres and J. E. Schindelin. A new metric for probability distributions. IEEE Trans. Inf. Theory , 49(7):1858-1860, 2003. 18, 19

F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In BMVC , page 12, 2018. 3

H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image CLIP. CoRR , abs/2106.11097, 2021. 3, 6, 7

V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In ECCV , volume 12349, pages 214-229, 2020. 1

V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In ECCV , volume 12349, pages 214-229, 2020. 6

Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML , volume 48, pages 1050-1059, 2016. 3

Z. Gao, J. Liu, S. Chen, D. Chang, H. Zhang, and J. Yuan. CLIP2TV: an empirical study on transformer-based methods for video-text retrieval. CoRR , abs/2111.05610, 2021. 3, 6, 7

Y. Ge, Y. Ge, X. Liu, J. Wang, J. Wu, Y. Shan, X. Qie, and P. Luo. MILES: visual BERT pre-training with injected language semantics for video-text retrieval. In ECCV , volume 13695, pages 691-708, 2022. 6, 7, 17

X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS , pages 249-256, 2010. 5, 16

Y. Gong, L. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In ECCV , volume 8692, pages 529-545, 2014. 3, 4

S. K. Gorti, N. Vouitsis, J. Ma, K. Golestan, M. Volkovs, A. Garg, and G. Yu. X-pool: Crossmodal language-video attention for text-video retrieval. In CVPR , pages 4996-5005, 2022. 2, 3, 6, 7

Z. Han, C. Zhang, H. Fu, and J. T. Zhou. Trusted multi-view classification. In ICLR , 2021. 3

D. R. Hardoon, S. Szedmák, and J. Shawe-Taylor. Canonical correlation analysis: An overview with application to learning methods. Neural Comput. , 16(12):2639-2664, 2004. 3

K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV , pages 1026-1034, 2015. 16

L. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. C. Russell. Localizing moments in video with natural language. In ICCV , pages 5804-5813, 2017. 6

D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR , 2019. 2

S. C. Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety , 54(2-3):217-223, 1996. 2

E. T. Jaynes. Information theory and statistical mechanics. Physical review , 106(4):620, 1957. 2

J. Johnson, A. Karpathy, and L. Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR , pages 4565-4574, 2016. 2

A. Jøsang. Subjective logic , volume 4. Springer, 2016. 4

A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. IEEE Trans. Pattern Anal. Mach. Intell. , 39(4):664-676, 2017. 6

D. Kim, N. Kim, and S. Kwak. Improving cross-modal retrieval with set of diverse embeddings. In CVPR , pages 23422-23431, 2023. 3

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, ICLR , 2015. 7

B. E. Klein, G. Lev, G. Sadeh, and L. Wolf. Fisher vectors derived from hybrid gaussianlaplacian mixture models for image annotation. CoRR , abs/1411.7399, 2014. 3

S. Kohl, B. Romera-Paredes, C. Meyer, J. D. Fauw, J. R. Ledsam, K. H. Maier-Hein, S. M. A. Eslami, D. J. Rezende, and O. Ronneberger. A probabilistic u-net for segmentation of ambiguous images. In NeurIPS , pages 6965-6975, 2018. 3

B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS , pages 6402-6413, 2017. 3

J. Lei, L. Li, L. Zhou, Z. Gan, T. L. Berg, M. Bansal, and J. Liu. Less is more: Clipbert for video-and-language learning via sparse sampling. In CVPR , pages 7331-7341, 2021. 6

H. Li, J. Song, L. Gao, P. Zeng, H. Zhang, and G. Li. A differentiable semantic metric approximation in probabilistic embedding for cross-modal retrieval. In NeurIPS , volume 35, pages 11934-11946, 2022. 7

L. Li, W. Ji, Y . Wu, M. Li, Y . Qin, L. Wei, and R. Zimmermann. Panoptic scene graph generation with semantics-prototype learning. CoRR , abs/2307.15567, 2023. 10

T. Lin, M. Maire, S. J. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. In ECCV , volume 8693, pages 740-755, 2014. 6

Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman. Use what you have: Video retrieval using representations from collaborative experts. In BMVC , page 279, 2019. 6

I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR , 2017. 7

I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR , 2019. 7

H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocomputing , 508:293-304, 2022. 1, 3, 4, 6, 7, 15

D. J. C. Mackay. Bayesian methods for adaptive models . California Institute of Technology, 1992. 3

D. Molchanov, A. Ashukha, and D. P. Vetrov. Variational dropout sparsifies deep neural networks. In ICML , volume 70, pages 2498-2507, 2017. 3

R. M. Neal. Bayesian learning for neural networks , volume 118. Springer Science & Business Media, 2012. 3

J. A. Portillo-Quintero, J. C. Ortiz-Bayliss, and H. Terashima-Marín. A straightforward framework for video retrieval using CLIP. In MCPR , volume 12725 of Lecture Notes in Computer Science , pages 3-12, 2021. 1

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML , volume 139, pages 8748-8763, 2021. 3, 4, 7, 10, 15, 17

R. Ranganath, S. Gerrish, and D. M. Blei. Black box variational inference. In AISTATS , volume 33, pages 814-822, 2014. 3

A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR , 2014. 16

M. Sensoy, L. M. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. In NeurIPS , pages 3183-3193, 2018. 3, 4, 15

G. Shafer. A mathematical theory of evidence , volume 42. Princeton university press, 1976. 2

C. E. Shannon. A mathematical theory of communication. ACM SIGMOBILE Mob. Comput. Commun. Rev. , 5(1):3-55, 2001. 2, 3

Y. Song and M. Soleymani. Polysemous visual-semantic embedding for cross-modal retrieval. In CVPR , pages 1979-1988, 2019. 1, 3, 4, 7

N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. , 15(1):1929-1958, 2014. 3

J. Xu, T. Mei, T. Yao, and Y. Rui. MSR-VTT: A large video description dataset for bridging video and language. In CVPR , pages 5288-5296, 2016. 6

T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. In CVPR , pages 1316-1324, 2018. 10

P. Zeng, L. Gao, X. Lyu, S. Jing, and J. Song. Conceptual and syntactical cross-modal alignment with cross-level consistency for image-text matching. In ACM MM , pages 2205-2213, 2021. 3

K. Zhang, Z. Mao, Q. Wang, and Y. Zhang. Negative-aware attention framework for image-text matching. In CVPR , pages 15640-15649, 2022. 7

S. Zhao, L. Zhu, X. Wang, and Y. Yang. Centerclip: Token clustering for efficient text-video retrieval. In SIGIR , pages 970-981, 2022. 3, 6, 7

J. Zhu, P. Zeng, L. Gao, G. Li, D. Liao, and J. Song. Complementarity-aware space learning for video-text retrieval. IEEE Trans. Circuits Syst. Video Technol. , 33(8):4362-4374, 2023. 3

Appendix

A The Model Structure of the Baselines

Text Encoder (Transformer)

a

Video Encoder (ViT)

group

of

girls

mixing

paint

...

...

MeanPool

0

1

2

3

4

0

1

2

3

4

5

Text

representation

Frame representation

sglobal

slocal

aggregated s

...

J

For image-text retrieval, we utilize CLIP [54] as the baseline straightly. As for video-text retrieval, the model structure is shown in Figure 7. To be specific, we follow the model structure of the CLIP4CLIP [49] to generate J frame features Z = { z f i | i = 1 , 2 , .., J } for each video and a text feature z t for each caption. Afterwards, two branches are designed to catch global and local information, respectively. In global branch, We first adopt a mean pooling to aggregate J frame features to get an 'average global feature' ¯ z v . Next, the cosine similarity s global between ¯ z v and z t is calculated. In local branch, we first calculate a set of cosine similarities S ∈ R J between z t and each frame feature z f i . Then we take the maximum value in S as s local . Finally, we take the average of s global and s local as the final similarity score. This similarity score will be constrained by a cross-entropy loss.

B Evidence Generation

How to generate evidence e from the similarity s between the instance and the prototype is crucial for uncertainty modeling. Following [57], three optional functions, including ReLU function, softplus function, and exponential function, are used to generate evidence as follows:

ReLU function :

Softplus function . Let θ denote a threshold of reverting to a linear function. γ and θ are separately set to 1, 20 in the experiments. Softplus function can formulate as:

Exponential function . Let τ be a scale parameter of the exponential function (Eq. 10), indicated as 5 in our experiments.

The performance comparison of three evidence-generation functions is shown in Table 7. It's worth noting that the exponential function outperforms other methods on multiple metrics. We therefore utilize it as the evidence-generation function in all experiments.

C Additional Experimental Results

C.1 The number of prototypes

Multiple prototypes are built to represent the overall semantics of the subspace. To figure out the best number of prototypes for representing subspace, the performance variety against prototype number K is shown in Table 8. It can be seen that PAU always performs better than the baseline. Furthermore, R@1 of both t2v and v2t first increases to the peak against growing K ( K = 12 for t2v, K = 8 for v2t) and then decreases. To model the uncertainty consistently, K is set to 8 for all modalities due to the maximum mean R@1 when K = 8 .

C.2 Different Initialization Methods for Prototypes

To explore the impact of different initial methods to prototypes, we employ several initialization methods for prototypes on MSR-VTT. The methods contain Xavier [24], kaiming_normal [29], kaiming_uniform [29], orthogonal [56] initialization. The results are shown below. We can observe that Xavier init methods can get the best R@1 performance on both t2v and v2t.

C.3 Comparing Model Size and Complexity

The FLOPs and parameter number of PAU on MSR-VTT are computed to explore the computational costs and model complexity. We compare PAU with some previous methods, such as VATT [1],

Frozen [4], MILES [23]. The results are shown as below. It can be seen that the FLOPs and the parameter number of PAU are both significantly smaller than other methods, indicating that PAU is a light-weight approach and easy to deploy.

C.4 Training with Noisy Correspondences

We additionally compare PAU with some CLIP ViT-B/32 backbone-based methods [8, 11, 10, 54] fine-tuned on noisy MS-COCO dataset. The results of training on 0%, 20%, and 50% noise ratios are reported in the Table 11. We can observe that our method exhibits significant advantages while training with several noisy correspondences, especially on 50% noise ratio. Although the uncertaintyaware method, PAU, is not designed for tackling the noisy correspondences, it still successfully handles the scenario, indicating that PAU possesses powerful robust learning ability even while training with noisy correspondences.

D The Explanation of High Boost on MSVD

In Table 2, we find the performance improvement of MSVD on v2t is significantly larger than other benchmarks. Investigating its reason, the MSVD is a tiny dataset with only 1,200 videos for the training set and approximately 40 captions per video (48,000 captions in total). In this case, utilizing cross-entropy loss, which views the diagonal pairs as positives, will lead into tremendous noise, especially with a large batchsize. In a batch with the size of 256, the probability of any texts coming from different videos is about 3 . 49 × 10 -13 . The solution is as follows.

The probability of any texts coming from different videos can denote as Eq. 11, where B is the batchsize of 256, and N is the total caption number of 48,000. m i indicates the candidate number

of i th text in a batch with m i = 48000 -40 × ( i -1) in MSVD. Then the probability P can be calculated by:

The resultant noises mislead the model training, resulting in a terrible performance reduction. Our uncertainty-aware approach can mitigate the detriment caused by these data and improve the model robustness effectively.

E Proof of Ambiguous Data with High Information Entropy

In Section. 1, we propose a theorem that 'the ambiguous multi-modal data highly associated with multiple semantics, such as fast-paced videos and non-detailed texts, lead to high information entropy, standing for high uncertainty.' The mathematical Theorem and Proof are given below.

Theorem. Suppose z ∈ ❘ D is the vector of multi-modal data. Assuming the modality semantic subspace can be divided into K finite categories as X = { x i ∈ ❘ D : i = 1 , 2 , · · · , K } . The information entropy is denoted as Eq. 13:

where p ( x i ) is the i th semantics probability derived from cosine similarity cos ( z, x i ) through a softmax layer. Let Q ∼ q ( x ) be a Discrete Uniform Distribution [14], then we have:

where J ( P ∥ Q ) denotes the Jensen-Shannon Divergence (JSD) [16], a metric to measure the distance between two probability distributions. Namely, Eq. 14 means that the closer the distribution P ∼ p ( x ) is to the Uniform Distribution Q ∼ q ( x ) , the higher the information entropy H ( X ) is.

Proof. Let's first figure out what distribution of information entropy is the largest. Given an objective function (Eq. 15) and its constraint (Eq. 13):

Let λ denote Lagrange multiplier. The Lagrangian function can be constructed as:

Separately computing the partial derivatives for p ( x 1 ) , p ( x 2 ) , ..., p ( x K ) :

Considering the constraints of Eq. 13, the p ( x ) essentially satisfy:

From this:

which is the density of discrete uniform distribution. Therefore, the H ( X ) gets the great value log K when the probability distribution conforms to a Discrete Uniform Distribution [14].

Then Jensen-Shannon Divergence (JSD) [16] is used to measure the distance between the probability distribution P ∼ p ( x ) and the Discrete Uniform Distribution Q ∼ q ( x ) .

Since q ( x ) conforms to the Discrete Uniform Distribution, q ( x i ) = 1 K and ∑ K i =1 q ( x i ) log q ( x i ) = -log K . The JSD function (Eq. 20) can be converted to:

Note that there are two components affecting the J ( P ∥ Q ) , ∑ K i =1 p ( x i ) log p ( x i ) and ∑ K i =1 ( 1 K + p ( x i )) log( 1 K + p ( x i )) . For convenience, let:

It can be seen that J ( P ∥ Q ) ∝ ( A ( P ) -B ( P )) . When P ∼ p ( x ) is closer to the Discrete Uniform Distribution Q ∼ q ( x ) , i . e ., J ( P ∥ Q ) ↓ , three cases are possible for A ( P ) and B ( P ) :

A ( P ) ↓ and B ( P ) ↓ , but the former has a larger reduction, i.e., ∆ A ( P ) -∆ B ( P ) < 0 .

b. A ( P ) ↓ and B ( P ) ↑ .

A ( P ) ↑ and B ( P ) ↑ , but the latter has a larger growth, i.e., ∆ A ( P ) -∆ B ( P ) < 0 .

∆ → 0 is the amount of function change. A ( P ) are both declining in cases of a and b , reflecting the increase of H Q ∼ q ( x ) ( X ) according to Eq. 22. Thus, only case c possibly induces the decrease of H Q ∼ q ( x ) ( X ) with the decrease of J ( P ∥ Q ) . If there is no case suit c , the theorem is proved.

Let f ( x ) = x log x . Its first-order derivative and second-order derivative are f ( x ) ′ = log x +1 and f ( x ) ′′ = 1 x > 0 , respectively. It is a convex function obtaining the minimal value at x = 1 e .

Due to ∑ K i =1 p ( x i ) = 1 , when p ( x a ) → p ( x a ) + ∆ , there must be a p ( x b ) → p ( x b ) -∆ . Subject to A ( P ) ↑ , Eq. 23 should be satisfied:

When 0 < p ( x b ) -∆ < p ( x b ) < p ( x a ) < p ( x a ) + ∆ < 1 , we have:

It indicates that case c is an impossible case when J ( P ∥ Q ) ↓ . Thus, only a and b are possible when J ( P ∥ Q ) ↓ . In these two cases,

Therefore, demonstrating that the closer to a uniform distribution, the larger the information entropy is. It concludes the proof.

F The Detailed Video Visualization

In Figure 6 of the main paper, we present several video fragments with their uncertainty. In Figure 8, more detailed videos are displayed to illustrate the uncertainty degree of the videos.

Figure 8(a) is a video with a 0.881 uncertainty score. Actually, it is difficult to understand the meaning expressed in the video. Obviously, as the uncertainty diminishes, the number of scenes in the videos decreases, and the videos become easier to understand.

Uncertainty: 0.881

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(a)

Uncertainty: 0.738

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(b)

Uncertainty: 0.681

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(c)

Uncertainty: 0.546

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(d)

Ki

Uncertainty: 0.446

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(e)

Uncertainty: 0.335

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(f)

Uncertainty: 0.226

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(g)

Uncertainty: 0.119

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(h)

Uncertainty: 0.026

1

2

3

4

5

6

7

8

9

10

11

12

13

14

(i)

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper focuses on the unreliable prediction of cross-modal retrieval methods caused by low-quality data. The authors introduce evidential theoretical framework to quantify the uncertainty arising from inherent data ambiguity.  To provide trustworthy uncertainty predictions, they propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework. Extensive experiments demonstrate the effectiveness of their methods.\n\nSTRENGTHS:\n- To the best of my knowledge, this paper is the first to combine the evidential framework with cross-modal retrieval methods, so this work is quite novel.\n- The idea of introducing uncertainty quantification to mitigate the negative impact of low-quality data on retrieval is interesting and sound. The experimental evaluations in the paper also provide strong evidence to support the claims made in the paper. These experiments are wide-ranging and cover several different datasets.\n- The paper is well written and easy to follow. However, there is room for improvements as I mention below.\n\nWEAKNESSES:\n1.  While the experiments are quite wide-ranging, the details provided in the paper are quite scarce, such as how many epochs are trained and how to choose hyperparameters. Code is also not provided.\n\n2.  Although the authors provide related work on cross-modal retrieval and uncertainty quantification, it is not clear how these works differ from their proposed method.\n\n3. There are some unclear points in the paper.\n- What is the meaning of the histogram in Figure 1(b)?\n- Does PAU only contain the component of $L_{uct}$, $L_{div}$ and Re-rank? How does it constrain whether $M''$ is predicted accurately? Providing pseudocode would help readers better understand the proposed method.\n\nQUESTIONS:\n1. What is the effect of using different initialization methods for all prototypes?\n2. What is the time complexity of the algorithm? How does it compare to other algorithms?\n3. The authors argue that the uncertainty loss forces the prototypes into learning the rich semantics of subspace to realize accurate uncertainty quantification. But I'm confused that the uncertainty loss only limits the consistency of U and H, why it can achieve accurate uncertainty quantification?\n\nLIMITATIONS:\nThe authors point out the limitations of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the aleatoric uncertainty quantification in cross-modal retrieval. The authors point out that the model predictions are often unreliable due to the aleatoric uncertainty, which is induced by low-quality data. They first define the inherent aleatoric uncertainty in multi-modal data. Then K prototypes are constructed as the belief masses to employ DST and SL, two classical theories in uncertainty quantification. On top of this, the uncertainty loss and diversity loss are proposed to help prototypes represent the overall semantic space. Extensive experiments are conducted on multiple tasks and benchmarks.\n\nSTRENGTHS:\n1.\tThis work presents a novel insight for the uncertainty of cross-modal retrieval. They classify uncertainty into aleatoric uncertainty and epistemic uncertainty, and give a clear definition of aleatoric uncertainty for multi-modal data. I think it will inspire further uncertainty-aware work in cross-modal retrieval or multi-modal learning.\n2.\tIn Appendix, the proof of “the ambiguous multi-modal data highly associated with multiple semantics” is intuitive and reasonable.\n3.\tThe experiments greatly prove the claims, especially Fig. 3 and Fig. 4. These two experiments reflect that the uncertain data indeed have a significant negative impact on results, and PAU could provide more trustworthy predictions. This phenomenon may be able to give an inspiring insight for robust training.\n\nWEAKNESSES:\n1.\tIn the reranking process, beta_1 and beta_2 are two learnable parameters. But it is unclear how these parameters are trained in this paper. \n2.\tPrototypes are expected to represent the overall semantics space. Do they really represent the overall semantics space after training? \n3.\tThe paper only mentions uct loss and div loss, which seem to be proxy-based loss functions, and can not optimize the prediction directly like cross-entropy loss or triplet loss. I wonder if other basic losses are used in the training process.\n4.\tThe Table 7 in appendix shows that the exponential function is the best method to generate evidence. Why there is a large performance gap between ReLu function and exponential function? Or what is the advantages of exponential function.\n5.\tThe legend in Figure 1 (b) is empty.\n\nQUESTIONS:\nRefer to weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on improving video-text retrieval by considering the uncertainty (or confidence) when computing image-text similarity. The high-level idea is to estimate the data uncertainty and then perform re-ranking to improve the performance of retrieval.\n\nThe authors designed a PAU framework, Prototype-based Aleatoric Uncertainty Quantification with CLIP encoders, to estimate the uncertainty score of the input instances. The authors conducted task-specific fine-tuning on the target datasets, and reported marginal performance improvements.\n\nSTRENGTHS:\n- The authors addressed an interesting problem of estimating the uncertainty of data, which is used to improve retrieval performance.\n- The proposed PAU framework is presented clearly.\n- Both the uncertainty loss and diversity loss are technically valid and straightforward.   \n- The authors conducted several experiments on multiple benchmarks, and demonstrate marginal performance improvements.\n\nWEAKNESSES:\n- The goal and task definition of this paper seem not clear. Both the introduction section and Figure 1 are somewhat confusing, which makes me difficult to get the main point. An image (or a single video frame) should contain rich semantics. Typically, an image is paired with a caption (or sentence), forming a image-text pair, to describe the semantics. However, the authors seem to argue that a single-scene video should match only a single word (i.e., talking), which is counter-intuitive. The multi-scene video may have richer semantics than single-scene video, but the authors argue that matching multiple semantics is not ideal. Overall, their key idea is counter-intuitive.   \n- The authors conducted experiments solely on downstream fine-tuning. The fine-tuning datasets are usually cleaner and with higher quality. A majarity of the noisy image-text paired data should be found in VL pre-training dataset. While downstream fine-tuning data (like youcook2 or MSRVTT) are human annotated/collected, the pre-training data is web-crawled noisy imgae-text pairs. To study the capability of uncertainty estimation for low-quality data, it would be much more interesting to experiment their proposed method in the pre-training stage. The proposed method may have potential to greatly improve the pre-training, but it remains unknown in current stage.\n- The paper is not written well, making it difficult to understand the motivation. This is probably due to unprecise or uncommon wording in the writing.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nNo potential negative societal impact", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes prototype-based aleatoric uncertainty quantification (PAU) to represent aleatoric uncertainty (i.e., data uncertainty) for cross-modal retrieval tasks. The proposed method uses the CLIP backbone, where the initial similarity matrix M is computed by the output of the CLIP encoders. Then, \"evidence\" $e$ is computed by the similarity between each modality embedding output and the prototype vectors. Using $e$, the belief $b$ and the uncertainty $u$ is computed. As far as the reviewer understood, the computed $u$ represents how the current embedding is *different from* the prototype vectors, i.e., if the current embedding is specifically different (in terms of the cosine similarity) from the prototype vectors, then its uncertainty becomes large. Using the learned uncertainty $u$, the proposed method re-rank the retrieved items by cosine similarity of the output of the CLIP encoders. Two losses are introduces to make the prototype vectors better, the uct loss (minimize the distance between average embedding vectors and prototypes) and the diversity loss (orthogonal regularization for prototype vectors). The effect of the proposed method is evaluated on four datasets.\n\nSTRENGTHS:\n- Measuring aleatoric uncertainty for multi-modal learning is an interesting and important problem.\n- The proposed method looks efficient and easily adaptable to various methods.\n- The proposed method is a somewhat straightforward extension of Subjective Logic (SL) [27] from the fixed class labels to the learnable prototypes. Hence, it may have the good theoretical properties shown by the SL paper [27], although it is not explicitly described in the paper.\n\nWEAKNESSES:\nMy concerns can be summarized as:\n\n- It is not clear that replacing fixed class information of SL with learnable prototype embeddings will ensure a proper uncertainty estimation.\n- This paper misses the important previous work on the aleatoric uncertainty estimation method for cross-modal retrieval [A].\n- The comparisons of MS-COCO 5K would not be fair (only the proposed method uses the pre-trained CLIP model), and many recent state-of-the-art methods are missing.\n\n### Can the proposed method provide a proper uncertainty estimation?\n\nThe proposed method highly depends on SL [27], where SL uses fixed class information; hence, there is no learnable prototype. As this paper introduces new learnable prototypes, the proposed method requires two additional objective functions, Eq (5) and Eq (6). I am not sure how Eq (5) and Eq (6) will guarantee the quality of the proper uncertainty measure.\n\nEq (6) looks okay, but it enforces the orthogonality of the prototype embeddings, not the mutual exclusion among the embeddings. The latter one would be derived from independence, not orthogonality. Note that independence and orthogonality are highly related and somewhat exchangeable in many cases, I am not sure that smaller orthogonality regularization (i.e., Eq (6)) can ensure the weaker dependency between the prototype vectors. Moreover, I am not sure that Eq (6) is specifically better than widely-used orthogonality regularization, such as $\\| I - Z^\\top Z \\|_F^2$, where $I$ denotes the identity matrix, and $Z$ denotes the full prototype vectors. Another possible alternative is using Maximum Mean Discrepancy (MMD) or Hilbert-Schmidt Independence Criterion (HSIC), whose values directly induce the strength of the dependency between two probabilistic densities. In my opinion, Eq (6) could be better, but Eq (6) itself is okay and can be somewhat sound regularization.\n\nTo me, the meaning of Eq (5) is ambiguous. It seems that Eq (5) behaves as minimizing the distances between prototypes and the embeddings; thereby, it enforces to make the value of $e_K$ larger (i.e., make the uncertainty value $u$ smaller). I am not sure how Eq (5) helps the proper uncertainty estimation.\n\nI think both Eq (5) and (6) can be explained with some in-depth discussions with minimal changes in the manuscript, but as of now, I am not fully confident whether the proposed method can represent proper uncertainty estimations despite of using learnable prototypes rather than fixed class information.\n\n### Missing related works on uncertainty estimation\n\nThis paper misses many directly related works on uncertainty estimation and cross-modal retrieval. Especially this paper did not consider PCME [A], a probabilistic embedding approach to represent aleatoric uncertainty (i.e., the quality of the data) for cross-modal retrieval.\n\n- [A] Chun, Sanghyuk, et al. \"Probabilistic embeddings for cross-modal retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nI think [A] should be discussed in the related work section and in the experiment section if possible.\n\nThere are also more aleatoric uncertainty estimation methods based on probabilistic embeddings, such as:\n\n- Shi, Yichun, and Anil K. Jain. \"Probabilistic face embeddings.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n- Park, Jungin, et al. \"Probabilistic representations for video contrastive learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- Neculai, Andrei, Yanbei Chen, and Zeynep Akata. \"Probabilistic compositional embeddings for multimodal image retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- Sun, Jennifer J., et al. \"View-invariant probabilistic embedding for human pose.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020.\n\nI would like to suggest adding discussions of the comparisons of the SL-based aleatoric uncertainty estimation and the probabilistic embedding-based aleatoric uncertainty estimation.\n\n### Missing comparison methods for MS-COCO 5K\n\nIt could be a minor issue, but I think Table 4 misses many relevant and state-of-the-art methods for cross-modal retrieval. For example, as my previous comment, PCME [A] is a highly related work to this paper. In terms of using multiple diverse embeddings, [B] could be another relevant work to the proposed method, which is more relevant to PVSE [49]. I also would like to suggest adding more recent state-of-the-art cross-modal retrieval works, such as VSE$\\infty$ [C].\n\n- [B] Kim, Dongwon, Namyup Kim, and Suha Kwak. \"Improving Cross-Modal Retrieval with Set of Diverse Embeddings.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n- [C] Chen, Jiacheng, et al. \"Learning the best pooling strategy for visual semantic embedding.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\nEspecially, Table 4 can be unfair because all other comparison methods do not use the pre-trained CLIP backbone; hence their performances are much worse than a simple \"baseline\" of this paper. Therefore, in my opinion, Table 4 should contain the comparisons with other methods using the pre-trained CLIP backbone.\n\nHowever, I fully understand that it is not easy to compare with other cross-modal retrieval methods, especially when they apply different backbones, pre-extracted features, and optimization techniques. After a quick Google search, I found a very recent work, that uses the CLIP ViT-B/32 backbone for cross-modal retrieval:\n\n- [D] Chun, Sanghyuk. \"Improved Probabilistic Image-Text Representations.\" arXiv preprint arXiv:2305.18171 (2023).\n\nI think the experimental results of [D] can be directly comparable with Table 4 of this paper, including VSE $\\infty$, PCME, and InfoNCE. I would like to suggest adding the results of [D] to the main paper if the authors are willing to update their table to contain more recent papers. *Note that [D] is not a direct comparison method of the proposed method because it arxived after the NeurIPS submission deadline*; therefore, I don't argue that [D] is a significant related work that should be compared with the proposed method, but I bring [D] because [D] uses the same CLIP ViT-B/32 backbone with the experiments in this paper, and it reported [A] and [C] with the ViT-B/32 backbone, listed in my review.\n\n### Minor comments\n\n- The implementation details of the overall framework are somewhat hard to catch at first glance. I suggest adding the detailed process for computing $e$ and $u$ in Figure 2, and adding the details of $e$ (Table 7 in Appendix) to the main paper.\n\nQUESTIONS:\nPlease check my comments in the weakness section. My major concerns are (1) I am not sure whether applying SL to learnable prototypes can lead to proper uncertainty estimations (2) missing significant related works, such as [A] and [B] ([B] could be considered as a contemporary work because [B] is a CVPR 2023 paper, that is published after the NeurIPS deadline).\n\nLIMITATIONS:\nI don't think this paper has a potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents the Prototype-based Aleatoric Uncertainty Quantifications (PAU) framework, which enhances the reliability of a pre-trained vision-language model (CLIP) by leveraging evidential knowledge. The paper introduces learnable prototypes as semantic categories and maps images and texts to these categories for uncertainty quantification. The proposed method improves model training and ranking in image-text retrieval tasks.\n\nSTRENGTHS:\n- The paper addresses the issue of model reliability and provides a reasonable approach to quantify data uncertainty in image-text retrieval tasks.\n- The introduction of trainable prototypes complements contrastive learning-based vision-language alignment by incorporating evidence knowledge. This leads to improved reliability of the pre-trained model.\n- The paper provides extensive experimental results and a detailed ablation study, demonstrating the effectiveness of the proposed method compared to existing image-text retrieval approaches.\n\nWEAKNESSES:\n- The loss function defined in Equation 5 raises confusion. The uncertainty is computed in the K category domain, while the mean similarity is computed in the N category domain. Could the author explain why these two domains are mapped together?\n- The paper includes an ablation study on the number of trainable prototypes (K), showing that the best R@1 performance on MSR-VTT is achieved when K=8. Concerning the scalability of the method, would K vary on different datasets to achieve the best performance?\n- Figure 5 visualizes the uncertainty of videos and captions. However, the connection between the data pairs is not clear. Could the authors explain the rationale behind selecting these specific examples?\n- The paper lacks a clear introduction to the baseline models, including their training process and model structure. It would be helpful to provide more details in this regard. Also, the reporting of experimental results seems inconsistent or incomplete.\n\nQUESTIONS:\nPlease refer to my question in the Weaknesses section.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}