Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval
Authors: Hao Li
Jingkuan Song
Lianli Gao
Xiaosu Zhu
Heng Tao Shen
Venue: NeurIPS 2023 poster (2023)
Keywords: multimodal learning
cross-modal retrieval
robust learning
uncertainty
TL;DR: This paper propose a prototype-based aleatoric uncertainty quantification framework for cross-modal retrieval.

Abstract: Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity. Concretely, we first construct a set of various learnable prototypes for each modality to represent the entire semantics subspace. Then Dempster-Shafer Theory and Subjective Logic Theory are utilized to build an evidential theoretical framework by associating evidence with Dirichlet Distribution parameters. The PAU model induces accurate uncertainty and reliable predictions for cross-modal retrieval. Extensive experiments are performed on four major benchmark datasets of MSR-VTT, MSVD, DiDeMo, and MS-COCO, demonstrating the effectiveness of our method. The code is accessible at https://github.com/leolee99/PAU.

Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval
Hao Li 1echeeeel@0aa com com
Jingkuan Song*
Lianli Gao
jingkuan. song@gmail com
lianli.. Mlanli.go@ueeste gao@uestc.....ooo edu. cn
Xiaosu Zhu
Heng Tao Shen
riaosu. com
shenhengtao@hhommai.. com
Center for Future Media, University of Electronic Science and Technology of China
Abstract
Cross-modal Retrieval methods build similarity relations between vision and lan- guage modalities by jointly learning a common representation space. However, the predictions are often unreliable due the Aleatoric uncertainty, which is induced by low-quality data, e-g., corrupt images, fast-paced videos, and non-detailed texts. In this paper, propose novel Prototype-based Aleatoric Uncertainty Quan- tification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity. Concretely, first construct a set of various learnable prototypes for each modality to represent the entire semantics subspace. Then Dempster-Shafer Theory and Subjective Logic Theory are utilized to build an evidential theoretical framework by associating evidence with Dirichlet Distribution parameters. The PAU model induces accurate uncer- tainty and reliable predictions for cross-modal retrieval. Extensive experiments are performed four major benehmark datasets MSR- -TT, MSVD, DiDeMo, MS-COCO, demonstrating the effectiveness of our method. The code is accessible https:://see /ttp///gtttttttttt//ggg/ thub. www. wwm/leolee99//10
Introduction
Cross-modal Retrieval, devoted searching gallery of related samples in vision/language modality given a query in another, has attracted increasing interest as the wellspring development of multi- media platforms. However, the heterogeneity between modalities casts daunting challenges for this task. To tackle these challenges, most existing methods [9, 60, 19, 15, 4, 53, 49] map the query and retrieval items from distinct modalities into joint latent space make similarity calculation feasible.
powerful capabilities of deep neural networks to extract high-level features. Chen el al. [9] construct a hierarchical graph structure for videos and texts to build the relationships between the local and the global features. PVSE [60] utilizes the multi-head self- attention module to learn multiple and diverse representations of videos and texts for the polysemous problem. Recently, a large body of transformer-base pre-trat pre-training models [19, 15, 4, 53, 49] have shown remarkable superiority with excellent generalization and performance.
Some early works [9, 60] corroborate
However, prior approaches routinely assume the data qualities are absolutely stable and view them equally. They estimate the relevance of vision-language pairs relying only on the similarity scores
"Corresoodingg author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Video A
Semantic Smilirity Distribttion
Text A
Semantic Siilartty Disirbuttoon
Contin Camuun
Video A Video B
singing stage huge audicnee he is holding microphome.
Costree Smmterry
Text A Text B
0.
05
Video
K Semantics Catergory
Video Gallery
K Semantiss Catergory
Text
A man is singing
Mequatie
(a) Video modality
(b) Text modality
Figure 1: Illustration of confused matching in last-paced videos and non-detailed texts. Assuming the possible semantics of each modal subspace are finite with categories. (a) A single-scene Video A can only match one semantics of "talking". By contrast, multi-scene Video B can match semantics of "talking", "shadow", and "cave". (b) Text A can only match the left video, while Text with some details removed (in red) matches both videos.
generated by complex neural networks, but ignoring the confidence of these predictions, which should be taken seriously. Low-quality data like corrupt images [3 fast-paced videos [26], non-detailed sentences [34], etc., inevitably lead to unreliable results. The model therefore not only needs to predict the relevance of vision-language pairs, but also capably answer the question "How confident is the prediction?" ?" Namely, it is necessary to quantify the uncertainty of each data to guide more reliable vision-languaal pair similarity.
In the literature, uncertainty can be classified into Aleatoric Uncertainty and Epistemic Uncertainty based on causation [32]. The former is induced by the inherent data defects, while the latter is inherent to the model. This work mainly focuses on the aleatoric uncertainty (i.e., data uncertainty), defined as "the uncertainty caused by ambiguous multi-modal data, such fast-paced videos and non-detailed texts". Next, we will answer the question "Why do these ambiguous data lead to uncertainty?". We first assume the possible semantics of each modal subspace are finite with K categories. Afterward, the fast-paced videos (multi-scene videos) and the non- det-eeea texts are generally highly related to multiple semanties, misleading models into confused matching. shown in the Figure 1. These alternative matches will bring uncertainty, which can be explained through Information Entropy [59]. Concretely, Information Entropy is the average uncertainty level the random variable's 's possible outcomes, which can be denoted as Eq. with the possible semantics as the random variable X.
H(X) EE logg(o.))) = Σ p(r;) logp(ı;))
where ΣK p(z;) = 1, p(x;) is the ith semantics probability derived from semantic cosine similarity through a softmax layer. The ambiguous data highly associated with multiple semanties have low variance of the probability distribution, resulting in high information entropy and high uncertainty due "The Maximum Entropy Principle [33]" (proof in Appendix. E). Information entropy is seemingly reasonable criterion to estimate the data uncertainty. but unfortunately, it is powerless to distinguish the instances with similarity distribution [0.8, 0.8, 0.8, 0.8) and the distribution of [0.2, 0.2, 0.2, 0.2). After a softmax function, both distributions turn into [0.25, 0.25, 0.25, 0.25) and obtain the same information entropy. In practice, we hope the first one gets higher uncertainty score than the other because the front of the retrieval sequence with highly similar instances is more notable for the retrieval task. Thus, the Dempster-Sher Shafer Theory of Evidence (DST) [58], a generalization of the Bayesian theory subjective probabilities, which has developed into general framework for uncertainty modeling, is leveraged to quantify multi-modal aleatoric uncertainty.
To precisely quantify the aleatoric uncertainty, in this work, we propose an effective Prototype- based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from inherent data ambiguity. To be specific, we first construct series of diverse prototypes for each modality to capture the overall K semantics of the subspace. Each prototype represents individual semantics. Afterward, the variations of cosine similarities

---

between the instance and the prototypes from another modality are deployed as the belief masses used in DST [59] model uncertainty. Finally, re-ranking practice is utilized further strengthen prediction reliability by regarding each data uncertainty as a similarity weight.
The main contributions of our paper three folds. 1) We give clear definition of the inherent aleatoric uncertainty in multi-modal data. 2) A powerful aleatoric uncertainty quantification frame- work PAU is built to accurately estimate the data uncertainty and effectively mitigate the negative impact on retrieval. 3) We also verify the superiority of our proposed PAU on multiple challenging benchmarks. Massive insightful ablation studies further reveal its effectiveness and generalization.
2 Related work
2.1 Cross-modal Retrieval
Cross-modal retrieval [9, 25, 67, 39, 17, 60, 54, 49, 22, 18, 66, 26, 64, 37], one of the most popular
tasks in multi-modal learning, has attracted increasing attention with the aim of searching similar semantic samples from different modalities. Early works [25, 39] build joint embedding space to make query-positive pairs closer by canonical correlation analysis (CCA) [28]. Faghri et al. [17] first use triplet loss to improve the model performance by focusing only the hardest negative. PVSE [60] proposes a multi-candidate representation function to capture the diversity and build one-to-many relationships in different modalities. Recently, CLIP [54] has shown vital power and generalization in various multi-modal tasks. A rich body of CLIP-basee based approaches [49, 22, 18, 66, 26] have got significant achievements in cross-modal retrieval. CLIP4Clip [49] transfers the rich knowledge from image-text pre- trained model CLIP to video-text retrieval. CenterCLIP [66] proposes multi-segment token clustering algorithm keep the most representative tokens in consecutive frames. In this paper, we follow the setting of the above work, which inherits the knowledge from CLIP.
2.2 Uncertainty Quantification
Generally, there are two paradigms among existing uncertainty estimation approaches. The first one is the Bayesian-basee based paradigm [5, 51, 40], which estimates uncertainty by approximating the moments of the posterior predictive distribution. Subsequently, some algorithms focus on the variety of Bayesian such as Laplace approximation [50], Markov Chain Monte Carlo [52], and variational techniques [55, 6]. Recent work [21] leverages dropout [61] in the test phase, reducing the computational cost of uncertainty estimation. Nonetheless, limited by the increment of model parameters and poor convergence, these approaches always have expensive training costs and slow inference time. The second paradigm focuses the non-Bayesian approach [41, 27]. Lakshminarayanan el al. [41] propose an alternative to Bayesian Neural Networks (BNN) which simple to train and produce reliable uncertainty estimates. Wang el al. [57] replace the parameter set of a categorical distribution with the parameters of a Dirichlet density and represent the uncertainties with single abnormal class. Recently, uncertainty-be based algorithms have been used in multi-view classification for trusted decision decision-making [27]. Although uncertainty-basee methods have achieved impressive progress, they are limited to working with particular classification and segmentation tasks.
3 Method
In this section, present our Prototype-based Aleatoric Uncertainty Quantification (PAU) framework (Figure 2). We first define the cross-modal retrieval task and introduce a commonly used pipeline in Section 3.1. Next, we detail the uncertainty theory, as well as the approach to quantify the uncertainty in Section 3.2.
3.1 Task Definition
Let D = T) denote vision and language dataset, where V is a set of images or videos, and T a set of texts. The goal of cross-modal retrieval is to rank the relevance between a visual query V (respectively textual query / E T) and textual set T (respectively visual set V). Recent works [54, 49, 22, 18, 66, 26] have shown CLIP's strong performance and generalization in various downstream tasks, inspiring to employ CLIP as our backbone.
3

---

Textual Prototypes Z
(N)
Diversity Loss
Unnestaiidy Modeling
Uneertainty Loss
(K. D)
(N.
T
Visiual
Textual Encoder
The man is making sauce in the kitchen.
Encoder
(N, D)
(K, D)
Visual Instances
Textual Instances
Uncertainty Loss
Diversity Loss
Uncertiity
Modkling
(N)
Visual Prototypes Zy
Figure 2: The Framework PAU. The visual encoder &e and textual encoder Or separately map the visual and textual instances into joint embedding space calculate the similarity matrix M. dot product function is used to build a set of similarity vector E IR x between N instances and K prototypes, afterward modeling the uncertainty. An uncertainty loss forces the prototypes into learning the rich semantics of subspace to realize accurate uncertainty quantification. Besides, A diversity loss is introduced to keep prototypes diverse. means cosine similarity.
To be specific, given visual instance and text instance t, the CLIP encodes them into latent embedding space IR. through visual encoder (-)-) and a textual encoder ( respectively. Then the relevance between them can be computed using cosine similarity function as:
(v) [x)]
(t
sims(v, cos( (o),
[t)
3.2 Uncertainty Quantification
Prototype construction. Although the cosine similarity function has shown effectiveness in most existing works [25, 60, 54, 49], some unreliable results will inevitably produced due to the low inherent data quality. These ambiguous data are generally highly related to multiple semantics, leading to confusing matching (see Figure 1).
To assess these data uncertainty and provide trustworthy predictions, we first assume the semantic space is finite with K semantic categories. Then, K' learnable prototypes Z IR KxD are constructed to represent the overall latent semantic subspace for each modality. making data semantic ambiguity evaluable. Each prototype represents an individual semantics. Finally, DST [12] is employed to build the uncertainty quantification framework by assigning the variations of similarities between an instance and the prototypes from another modality as the belief masses.
The theory of uncertainty. The Dempster-Shafer Theory of Evidence (DST) [12], generalization of the Bayesian theory subjective probabilities [12], builds general uncertainty modeling framework by assigning belief masses to the set of mutually exclusive possible states with one state expressing "I am not sure". Following EDL [57], we adopt Subjective Logic (SL) [35] to associate the belief distribution in DST with the parameters of the Dirichlet distribution. More precisely, SL provides an evidential thcoretical framework quantify the belief masses by of different categories and overall uncertainty mass & for the K classification task. These K + mass values are all non-negative and sum up to one:
K
1b +
b
=
(3)
where by 2 and v 2 0 denote the belief mass the hthh semantic category (k 1,..., K) and overall uncertainty mass, respectively. In our setting, vi here means the uncertainty mass of the

---

event "the data is ambiguous", which is negatively related to the degree of data ambiguity. Thus, w -- Ex k= by is designed to express the aleatoric uncertainty defined in Section 1. In
the = 11-20 brief, the data with lower v is more ambiguous, which should be a high alcatoric uncertainty score as higher A belief mass by for the k*h semantic category is computed using the evidence for the category. We term the evidence as the amount of data supports for being classified into a certain category. Let [ei.-... .............................. ek be evidence vector derived from similarity vector p [p1, Pkk] between an instance and K prototypes (evidence generation shown in Appendix. B), then the belief by and the uncertainty
signified as: and u 3/ bk
Ek: 5
OCk
K
K
by
S
where 0g = + 1 denote the Dirichlet parameters associated with a belief mass assignment in SL, and S Σ i= Oę is referred as the Dirichlet strength. ex is the evidence of a visual (or textual) instance matching the hthh semantics category. Intuitively, massive evidence indicates considerably ambiguous data, resulting in high uncertainty, which brings out fallible predictions.
Uncertainty learning, After uncertainty modeling, how to force the model to learn uncertainty from data is worth considering (see Figure 2). The key is enriching prototypes' meaningful semantics to represent overall subspace rigorously. We first give normalized initialization proposed by Xavier [24] all prototypes. Subsequently, the cosine similarity matrix M IR N between N visual and textual instances is calculated by Eq. 2 within batch. Furthermore, N instances and K prototypes from different modalities build a set of similarity vectors P IR x to derive the evidence vector set and compute the uncertainty using Eq. 4. The data uncertainty sets of N visual and textual instances can be symbolized as U, = [ui, ... [%]] and U, [u4, ... w parallel.
Next, an objective function is required to force prototypes to learn rich semantic knowledge. As the prototypes are expected represent the overall semantics of each modality, the instance with high overall similarity to these prototypes should also have high overall similarity with other instances from another modality. According to Eq. 4, the aleatoric uncertainty u, denoted as the sum of belief masses, should be positively related to the mean similarities of each modality within batch. Thereby, we compute the mean of similarity matrix M along the row and the column, symbolized as H, 14y... h&] and H, = [hf..... ]. respectively. After that, prototypes can be optimized by employing Squa-Squared (MSE) Loss to keep the U, (U,) and H,, (H,) consistent, where is scaling coefficient to keep the same value range of u and h (Eq. 5).
Luct
(u A ..))22
(5)
N
Σ (14 - -
Diversity learning. To employ the DST uncertainty framework, the mutual exclusion among possible states needs to be satisfied. Precisely, prototypes should avoid attending to the same or similar semantics. Therefore, diversity loss is proposed to help the prototypes focus diversified independent semantics. The formula as:
K coo 2)
C =
K2
(6)
K
K
C2 = K2
(cos(z), 2)))²
i=1
where z and z denote ith prototypes of visual and textual modality. Intuitively, the semantic similarities between different prototypes are expected be
Re-ranking, After training, the uncertainty of each instance could precisely assessed using Eq. 4. To further alleviate the impact of the uncertainty prediction, we re-rank the retrieval sequences by weighting the predicted matrix based on the uncertainty vector U, and U, by Eq.
M" (exp(-3,UU)) exp(-32U,)) M'
(7)
5

---

where $1 and 2g are learnable parameters used to control the impact degree of re re-rannnkng on prediction. M' and M" separately signify the similarity matrices between all visual and textual instances in test processing before and after re-ranking.
Experiments
In this section, we first introduce the datasets and metrics used for our experiments, as well as the implementation details. Later, we compare PAU with the previous state of the arts several tasks and public benchmarks. the end, rich ablation studies are carried out to verify our method's effectiveness and generalization.
4.1 Datasets and Metrics
We report our experimental results on three public video-text retrieval datasets containing MSR- VTT [62], MSVD [7], DiDeMo [30], and public image-tex, retrieval dataset of MS-COCO [45]. Following [49], Recall @ (R@ Recall (R@5), Recall @ (R @ 10), Median Rank (MdR), and Mean Rank (MnR) are used to evaluate the performance.
MSR-VTT [62] commonly used benchmark in video-text retrieval, containing 10,000 Youtube videos with 20 captions for each. Following [20]. we employ 9,000 videos for training and 1,000 selected video-text pairs for testing.
MSVD [7] contains 1,970 videos ranging length from to 62 seconds, with approximately 40
sentences per video. There are 1,200, 100, and 670 videos used for training, validation, and testing in the standard split.
DiDeMo [30] consists of 10,611 Flickr videos with approximately 40,000 annotations. We follow [46, 42, and concatenate all captions of the same video into paragraph, which will be regarded as single item in paragraph-to-videe vide retrieval task.
MS-COCO [45] is widely used dataset for image-text retrieval, containing 123,287 images with five captions per image. We follow the split in [36] with 113,287 images for training, 5,000 for validation, and 5,000 for testing. The results reported 1K and 5K test sets.
Table 1: Retrieval Performance MSR- VTT
t2v
v21
Method
R@D
R@S
R@ 10
MdR
MnR
R@D
R@5
R@ 10
MdR
MnR
CLIP4Clip-meanP [49]
43.1
70.4
16.2
43.1
70.5
2.0
12.4
CLIP4Clip-seqTTaass [49]
44.5
71.4
81.6
2.0
15.3
42.7
70.9
80.6
2.0
11.6
CenterCLiP [66]
44.2
71.6
82.1
2.0
15.1
42.8
71.7
82.2
2.0
10.9
MILES [23]
44.3
71.1
80.2
2.0
14.7
CLIP2Video [18]
45.6
72.6
81.7
2.0
14.6
43.5
72.3
82.1
2.0
10.2
CLIP2TV [22]
46.1
72.5
82.9
2.0
15.2
43.9
73.0
82.8
2.0
11.1
XPool [26]
46.9
72.8
82.2
2.0
14.3
44.4
73.3
84.0
2.0
9.0
Baseline
45.6
72.9
81.5
2.0
14.5
45.2
71.6
81.5
2.0
10.9
ours
48.5
72.7
82.5
2.0
14.0
48.3
73.0
83.2
2.0
9.7
Table 2: Retrieval Performance MSVD
(2v
v2t
Method
R@1
R@5
R@ 10
MdR
MnR
R@1
R@5
R@ 10
MdR
MnR
CLIP4Clip-mest Peennl [49]
46.2
76.1
2.0
10.0
56.6
79.7
7.6
CLiP4Clip-seqTrans [49]
45.2
75.5
84.3
2.0
10.0
62.0
87.3
92.6
1.0
4.3
CenterCLiP [66]
47.3
76.9
86.0
2.0
9.7
63.5
86.4
92.6
1.0
3.8
CLiP2Video [18]
47.0
76.8
85.9
2.0
9.6
58.7
85,6
91.6
1.0
4.3
CLIP2TV [22]
47.0
76.5
85.1
2.0
10.1
XPool [22]
47.2
77.4
86.0
2.0
9.3
66.4
90.0
94.2
1.0
3.3
Baseline ours
46.3
76,8
84.9
2.0
10.0
60.1
82.8
87.3
1.0
7.5
47.3
77.4
85.5
2.0
9.6
68.9
93.1
97.1
1.0
2.4

---

Table 3: Retrieval Performance on DiDeMo
12v
v2t
Method
@@l
R@5
R@ 10
MdR
MnR
@@!
R@s
R@ 10
MdR
MnR
CLIP4Clip-meanP [49]
43.4
70.2
80.6
2.0
17.5
43.4
69.9
80.2
2.0
17.5
CLIP4Clip-sed LLI4CCiipseeTTrantt [49]
43.4
69.9
80.2
2.0
17.5
42.7
70,9
80.6
2.0
11.6
CLIP2TV [22]
45.5
69.7
80.6
2.0
17.1
Baseline
47.0
76.0
86.1
2.0
10.7
46.9
74.4
84.2
2.0
8.4
ours
48.6
76.0
84.5
2.0
12.9
48.1
74.2
85.7
2.0
9.8
Table 4: Comparison MS-COCO
MS-COCO IK
MS-COCO
i2t
12i
i2t
12i
Method
Backbone
R@1
R@S
R@ 10
R@1
R@5
R@
10
R@
R@5
R@
10
R@I
R@5
R@ 10
PVSE [60]
69.2
91.6
96.6
55.2
86.5
93.7
45.2
74.3
84.5
32.4
63.0
75.0
SGRAF [13]
79.6
96.2
98.5
63.2
96.1
57.8
91.6
41.9
81.3
101
NAAF [65]
80.5
96.5
98.8
64.1
90.7
96.5
58.9
85.2
92.0
42.5
70.9
81.4
DAA [43]
80.2
96.4
98.8
65.0
90.7
95.8
60.0
86.4
92.4
43.5
72.3
82.5
VSE [8]
82.0
97.2
98.9
69.0
92.6
96.8
62.3
87.1
93.3
48.2
76.7
85.5
PCME [1]]
80.1
96.6
98.7
67.6
92.1
96.9
59.9
85.8
92.3
46.1
75.0
84.6
PCME+ [10]
VIT-B/32
81.6
99.0
69.2
92.8
97.1
62.1
86.8
76.7
85.5
Baseline
80.1
95.7
98.2
67.1
91.4
62.9
84,9
73.8
82.9
ours
80.4
96.2
98.5
67.7
91.8
96.6
63.6
85.2
92.2
46.8
74.4
83.7
4.2
Implementation Details
We CLIP's [54] visual-textual encoder (ViT-B/32) the backbone and initialize all parameters from pre-trained CLIP weights. The max token and frame length are both set consistently with [49]. Specifically, the dimension of common embedding space
to 512. In video-text retrieval, the Adam optimizer [38] is employed with the initial learning rate of le-7 for the visual encoder and the textual encoder, well as the initial learning rate of le-4 for other layers. As for image-texi retrieval, the Adam W optimizer [48] updates the parameters of all modules with the initial learning rate of 1e-5. The learning rate decays complying with the cosine schedule strategy [47] 5, 3, 20, and epochs trained on MSR-V -TT,, MSVD, DiDeMo, and MS-COCO. Besides, the number of prototypes K is set as 8 according to Table 8. All experiments are conducted on to RTX3090.
4.3 Comparison with State of the Arts
Video-text retrieval. We compare our model with other state-of-the-ar methods [49, 66, 18, 22, 26, 23, 54] and baseline (model structure Appendix. A) on multiple datasets. For fair comparisons, all methods are based, symbolizing the most advanced techniques of vidco-text retrieval. The performance of baseline trained without PAU is also given. Table 1, show the results of video-text retrieval on MSR-VTT MSVD, and DiDeMo, respectively.
We can observe that PAU brings sizeable improvements on all benchmarks. For text-to-video retrieval (t2v), PAU boosts the @ performance of 2.9%. 1.0%, and 1.6% on MSR-VTT, MSVD, and DiDeMo. Moreover, 3.1 % 8.8%, and 1.2% improvements of R are obtained on video-to-text retrieval (v2t). Notably, we find that the performance improvement of v2t on MSVD more significant than other benchmarks. The reason is that MSVD is a small dataset with few videos (only 1,200 for training) but massive annotations (about 40 annotations per video). Mcanwhile, CLIP simply utilizes cross-entropy loss, which only regards the diagonal pairs the positives. Nonetheless, for batch with a typical size of 256 [49], the probability of any two texts coming from separate videos is approximately 3.49 x 10 (see Appendix. D), bringing innumerable noises and performance damage for the training on video-to-tex retrieval. Thus, the significant improvement indicates the ability of PAU to capture and rectify misleading data support robust learning.
Image-text retrieval. We also test our method on image-text retrieval task. To be specific, CLIP [54] is utilized as the baseline. We compare our model with several previous state-of-ther -ote----ttth--tttt- approaches, in- cluding PVSE [60], SGRAF [13], NAAF [65], DAA [43], VSEcc [8], PCME [1]], and PCME+ [10]. Table 4 shows the results of image-text retrieval on MS-COCO IK and MS-COCO 5K, respectively.

---

We can observe that the overall performance improvement of image-text retrieval is smaller than that of video-text retrieval. The phenomenon reflects that video information is more complex than image information, inducing immense uncertainty in video-text retrieval. Furthermore, we also validate the robustness of PAU training in noisy datasets. The detailed results can be found in Appendix. C.4.
4.4 Comparison with Probabilistic Model
PCME [11] is probabilistic-basee approach to represent instance diversity, which can also represent uncertainty. The more diverse the instance, the more uncertain the instance. Thus, an instance's uncertainty and diversity should be positively correlated. To verify the superiority of our SI.-based method over than probabilistic-uuse method, we conduct two experiments, pearson correlation analysis and uncertain data analysis.
Pearson correlation analysis. The pearson cor- relation coefficient (PCC) Tay (ray E [-1, 1)) a metric to estimate the strength of the corre- lation between variable x and variable y. Gen- erally, Tay > 0.5 means that two variables are strongly positively correlated. As mentioned before, the uncertainty and the diversity of an in- stance should be positively correlated for PCME. an instance is diverse, it will have high mean similarity h with the instances from the other modality. This means that the uncertainty score u and mean similarity i should also be positively correlated for an instance. Thus, we compute the pearson correlation coefficient between and h to explore whether PAU correctly estimates uncertainty. The results are shown in Table 5.
Table 5: PCCs between u and h
Method
PCCv
PCCr
Dataset
PCME
-0.071
-0.219
COCO COCO
PAU PAU
0.886
0.786
0.917
0.939
MSR-VTT
In this Table, we compute the PCCs of PCME and PAU on MSCOCO, well as PCCs of PAU on MSR-VTT. PCCv and PCCT separately indicate the instance's PCC in vision or textual modality. The low PCCs of PCME mean current probabilistic models have limited abilities to reflect uncertainty and diversity. We argue the reason arises from the strong prior distribution assumption, i.e., gaussian or other simple distributions are powerless to fit complex and diverse relationships in high-dimension space. By contrast, PAU obtains high PCCs in all settings, proving our approach indeed leads to proper uncertainty estimations.
Uncertain data analysis. To further prove the superiority of SL-based aleatoric uncertainty estimation method than probabilistic-basee method, we compare the changes of R@] after removing top-r instances with highest uncertainty scores predicted by PCME and PAU, respectively. To fairly compare, we employ removal on both predictions arised from PAU and PCME. The results with removed ratio from 0% to 20% on MS-COCO 5K is shown in Figure 3. We can observe that PAU outperforms PCME on all directions and predictions. This means that the uncertain data found by PAU is more precise than PCME, All experiments indicate the SL-based approach (PAU) works better than probabilistic-basee method (PCME) on representing aleatoric uncertainty.
MG
2.8
0
0 3.3
0.6
1.0
60
200
3001
100
6.
153
200
2003
400
(a) 12t on CLIP
(b) 12i CLIP
(c) 12t PCME
(d) (2i PCME
Figure 3: The performance changes comparison after removing top-r instances with the highest uncertainty scores quantified by PCME and PAU on MS-COCO. To fairly compare, we employ the removal on both predictions arising from CLIP and PCME. (a) and (b) show the performance changes on CLIP predictions. (c) and (d) show the performance changes on PCME predictions. In i2t, text instances are removed. In t2i, image instances are removed.
8

---

Table Ablation study to investigate the effect of different components on MSR- VTT
12v
V2t R@5
No
Luet
Ldie
Re-rank
R@5
R@D
R@ 10
1
45.66
72.9
81.5
45.2
71.6
81.5
2
45.9
72.8
81.5
46.0
72.4
81.7
3
47.0
72.4
82.8
47.4
71.8
82.2
4
47.8
72.7
82.5
47.9
72.2
83.0
5
48.4
72.4
82.2
47.7
72.6
82.6
6
48.5
72.7
82.5
48.3
73.0
83,2
4.5
Ablation Studies
The effect of different components. To understand the effect of each component, exhaustively ablate three main components PAU, including Uncertainty Loss, Diversity Loss, and Re-ranking. As shown in Table 6, all three components effectively improve the model performance. The compari- son between No. and No. 3 shows that uncertainty loss gives significant @ improvements of 1.4% 12y and 2.2% on v2t. The experiment No. outperforms the experiment No. 3 by clear margin, demonstrating the necessity of diversity loss. Furthermore, the experiment No. 2 re-ranks the baseline prediction using uncertainty vectors from PAU. It's worth noting that R @ performance still boosts on both 12v and v2t, reflecting the strong generalization of PAU.
The impact of uncertain data. To demonstrate the uncertain data found by PAU indeed affect the retrieval, a trial of the retrieval subsets with varying reliability levels is carried out on MSR-VTT. Specifically, the samples are first ranked according to the uncertainty calculated by PAU in descending order. Then, r samples with the highest uncertainty will be removed from the test set to produce an uncertainty- brasee removal subset. Mcanwhile, random samples will be removed from the test set to produce uundom-based removal subset. The baseline model is tested on both subsets above and reports the results in Figure 4.
It's worth noting that the model tested on the subsets removing uncertain data always outperforms the ones removing randomly. Furthermore, the performance gap reaches peak of 14.2 when r = 600
mevdore-ssed
mindom-based
unoo cal based
Lncertainy--aad
0.8
09
0.
08
(a) Text-to-vidco
(b) Vidco-to-text
Figure 4: The performance comparison against data removal number MSR-VTT.
0.45
200
1020
600
(a) Texd-to-video
(b) Video-to-text
Figure 5: The performance comparison against uncertain data number on MSR-VTI VTT.

---

Text Uncertainty
Video Uncertainty
person explaining
something. Uncertainty: 0.984
Uncertainty: 0.881
A man is driving a black car.
Uncertainty: 0.421
Uncertainty: 0.446
Multi colored horses in barn
and outside in the snow.
Uncertainty: 0.039
Uncertainty: 0.026
Figure 6: The visualization of data uncertainty on MSR-VTT
for texi-to-video retrieval and peak of 12.3 when 700 for video-to-text retrieval, respectively. The above phenomena prove PAU can indeed find uncertain data, which easily misleading the model into incorrect predictions, resulting in extremely unreliable retrieval results.
Is the prediction more trustworthy? To validate the effectiveness of PAU in providing trustworthy predictions, the performances of PAU and baseline are compared in the MSR-VTT's subsets under series of different uncertain degrees. Precisely, samples with the highest uncertainty are selected to form the uncertain test subsets. As shown in Figure 5, PAU consistently outperforms the baseline with the largest gap of 8.6 (5.8) while = 70 (r = 120) for 12v (v2t). The smaller the is, the higher the total uncertainty the subset so the results confirm the ability of PAU decrease the performance loss caused by uncertain data. Furthermore, PAU can provide trustworthy predictions even in high uncertainty sets.
Visualization of data uncertainty. Three captions and three videos with their uncertainty scores are shown in Figure 6. Obviously, the caption with high uncertainty is more semantically ambiguous, such as "A person is explaining something". By contrast, the caption with low uncertainty is more detailed, such "Multi colored horses in barn and outside in the snow". Moreover, the first video (complete video shown Appendix. F) is more difficult to summarize due to the varied scenes, inducing high uncertainty. By contrast, the third video with low uncertainty relatively simple.
Conclusion
In this work, we clearly define the inherent aleatoric uncertainty in multi-modal data. A Prototype- based Aleatoric Uncertainty Quantification (PAU) framework is designed excavate the uncertain data and further provide trustworthy predictions. Uncertainty loss and diversity loss are proposed to support the uncertainty training by encouraging the learnable prototypes to represent the semantic subspace of varied modalities. Plentiful empirical experiments benchmarks demonstrate the effectiveness and generalization PAU.
range of tasks and public
Broader impacts statement, Cross-modal matching has become a common paradigm in large acaee multi-modal pretraining, such as CLIP [54]. However, the enormous data demand brings huge challenge to pre-trained models. If we can accurately select high-quality data, which typically make greater contributions the performance of pre-trained models, the pretraining process would become much more efficient. This work could be one of the first work to be aware of the aleatoric uncertainty in cross-modal retrieval. By quantifying uncertainty, higher quality data and more reliable results are provided, which could perhaps be extended to other multi-modal tasks in the future [2, 44, 63].
scale
Acknowledgements
This study supported by grants from National Key R&D Program of China (D22XCC00093//2XX/00000
the National Natural Science Foundation of China (Grant No. 62122018, No. 62020106008, No. 61772116, No. 61872064), Fok Ying-Tong Education Foundation( 171 106), and SongShan Laboratory YYJC012022019.
10

---

References
[1] H. Akbari, L. Yuan, R. Qian, W. Chuang, S. Chang, Y. Cui, and B. Gong. VATT: transformers for multimodal supervise learning from raw video, audio and text. In NewrIPS, pages 24206-24221, 2021. 16, 17
[2] Anderson, X. Hc, C. Buchler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, pages 6077-6086, 2018. 10
[3] M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1708-1718, 2021. 6
[4] M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In /CCV, pages 1708-1718, 2021. 1, 17
[5] A. Blum, N. Haghtalab, and A. D. Procaccia. Variational dropout and the local reparameteriza- tion trick. In NeurlPS, pages 2575-2583, 2015. 3
[6] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In ICML, volume 37, pages 1613-1622, 2015.
[7] D. L. Chen and B. Dolan, Collecting highly parallel data for paraphrase evaluation. In ACL, pages 190-200, 2011. 6
[8] J. Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang. Learning the best pooling strategy for visual semantic embedding. In CVPR, pages 15789-15798, 2021. 7, 17
[9] Chen, Zhao, Q. Jin, and Q. Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR, pages 10635-10644, 2020. 1.3
[10] S. Chun. Improved probabilistic image-text representations, arXiv preprint arXiv:2305.18171,
2023. 7, 17
[11] Chun, Oh, R. S. de Rezende, Y. Kalantidis, and D. Larlus. Probabilistic embeddings for cross-modal retrieval. In CVPR, pages 8415-8424, 2021. 7, 8, 17
[12] A. P. Dempster. A generalization of bayesian inference. Journal of the Royal Statistical Society: Series B (Methodological), 30(2):205-22 2322, 1968. 4
[13] H. Diao, Y. Zhang, L. Ma, and Lu. Similarity reasoning and filtration for image-text matching.
In AAAI, pages 1218-1226, 2021. 7
[14] R. Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019.
18, 19
[15] M. Dzabraev, M. Kalashnikov, Komkov, and Petiushko. MDMMT: multidomain multi- modal transformer for video retrieval. In CVPR Workshops, pages 3354-3363 2021.
[16] D. M. Endres and J. E. Schindelin. A new metric for probability distributions. IEEE Trans. Inf. Theory, 49(7): 1868-1800, 2003. 18, 19
[17] Faghri, D. J. Fleet, R. Kiros, and S. Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In BMVC, page 12, 2018. 3
[18] H. Fang, P. Xiong, I.. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image CLIP. CoRR, abs/2106. 1007, 2021. 3, 6.
[19] V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In ECCV, volume 12349, pages 214-229, 2020.
[20] V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In ECCV, volume 12349, pages 214-229, 2020.
[21] Y. Gal and Z. Ghahramani. Dropout as bayesian approximation: Representing model uncer- tainty in deep learning. In ICML, volume 48, pages 1050-1059, 2016.
11

---

[22] Z. Gao, J. Liu, S. Chen, D. Chang, H. Zhang, and J. Yuan. CLIP2TV: an empirical study on transformer-basse methods for video-text retrieval. CoRR, abs/2111.05611 2021. 3, 6,
[23] Y. Ge. Y. Ge, X. Liu, J. Wang, J. Wu, Y. Shan, X. Qie, and Luo. MILES: visual BERT pre-training with injected language semantics for video-text retrieval. In ECCV, volume 13695, pages 691-708, 2022. 6, 7, 17
[24] X. Glorot and Y. Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, pages 249- 2010. 5, 16
(25) Y. Gong, L.. Wang, M. Hodosh, J. Hockenmaier, and S. Lazebnik. Improving image-sentence embeddings using large weakly annotated photo collections. In ECCV, volume 8692, pages 529-545, 2014. 3,4
[26] S. K. Gorti, N. Vouitsis, Ma, K. Golestan, M. Volkovs, A. Garg, and G. Yu. X-pool: Cross- modal language-video attention for text-video retrieval. In CVPR, pages 4996-5005, 2022. 2, 3, 6,7 7
[27] Z. Han, C. Zhang, H. Fu, and J. T. Zhou. Trusted multi-view classification. ICLR, 2021. 3
[28] D. R. Hardoon, S. Szedmák, and Chawe-Taylor, Canonical correlation analysis: An overview with application to learning methods. Neural Comput., 16(12):263-22644 2004.
[29] K. He, X. Zhang. S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, pages 1026-1034, 2015. 16
[30] I.. A. Hendricks, O. Wang, E. Shechtman, J. Sivic, T. Darrell, and B. C. Russell. Localizing moments video with natural language. ICCV. pages 5804-5813, 2017. 6
[31] D. Hendrycks and T. G. Dietterich. Benchmarking neural network robustness common corruptions and perturbations. In ICLR, 2019.
[32] S. C. Hora. Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management. Reliability Engineering & System Safety, 54(2-3):217-223, 1996.
2
[33] E. T. Jaynes. Information theory and statistical mechanics. Physical review, 106(4):620, 1957.
2
[34] J. Johnson, A. Karpathy, and I.. Fei-Fei. Densecap: Fully convolutional localization networks for dense captioning. In CVPR, pages 4565 2016. 2
[35] A. Jøsang. Subjective logic, volume 4. Springer. 2016. 4
[36] A. Karpathy and L. Fei-Fei. Deep sesual-semantic alignments for generating image descriptions.
IEEE Trans. Pattern Anal. Mach. Intell., 39(4):664-61) 2017. 6
[37] D. Kim, N. Kim, and S. Kwak. Improving cross-modal retrieval with set of diverse embeddings. In CVPR, pages 23422-23431, 2023. 3
[38] D. P. Kingma and Ba. Adam: A method for stochastic optimization. In Y. Bengio and Y. LeCun, editors, ICLR, 2015. 7
[39] B. E. Klein, G. Lev, G. Sadeh, and L. Wolf. Fisher vectors derived from hybrid gaussian- laplacian mixture models for image annotation. CoRR, abs/ 10111111.20 1.799,, 2014.
[40] S. Kohl, B. Romera-Paredes, C. Meyer, J. D. Fauw, J. R. Ledsam, K. H. Maier-Hein, S. M. A. Eslami, D. J. Rezende, and O. Ronneberger. A probabilistic u-net for segmentation of ambiguous images. In NeurlPS, pages 6965-6975, 2018.
[41] B. Lakshminarayanan, A. Pritzel, and C. Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurlPS, pages 6402 6411, 2017.
[42] Lei, I.. Li, I.. Zhou, Z. Gan, L.. Berg, M. Bansal, and Liu. Less is more: Clipbert for video-and- languuge learning via sparse sampling. In CVPR, pages 7331-7341, 2021. 6
12

---

[43] H. Li, J. Song, L. Gao, P. Zeng, H. Zhang, and G. Li. A differentiable semantic metric approximation probabilistic embedding for cross-modal retrieval. In NeurIPS, volume 35, pages 1194--119999 11946, 2022. 7
[44] Li, Ji Ji Y. Wu, M. Li, Y. Qin, L. Wei, and R. Zimmermann. Panoptic scene graph generation with semantics
semantics-poototyee prototype learning. CoRR, abs/2307. 155667 2023. 10
[45] T. Lin, M. Maire, S. J. Belongie, J. Hays, Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft COCO: common objects in context. In ECCV, volume 8693, pages 740-755, 2014.
[46] Y. Liu, S. Albanie, A. Nagrani, and A. Zisserman. Use what you have: Video retrieval using representations from collaborative experts. In BMVC, page 279, 2019. 6
[47] I. Loshchilov and F. Hutter. SGDR: stochastic gradient descent with warm restarts. In ICLR, 2017.
[48] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In ICLR, 2019.
[49] H. Luo, L. Ji, M. Zhong, Y. Chen, W. Lei, N. Duan, and T. Li. Clip4clip: An empirical study of CLIP for end to end video clip retrieval and captioning. Neurocompuling, 508:293-304, 2022.
1, 3, 4, 6, 7, 15
[50] D. C. Mackay. Bayesian methods for adaptive models. California Institute of Technology,
1992. 3
[51] D. Molchanov. A. Ashukha, and D. P. Vetrov. Variational dropout sparsifies deep neural networks. ICML, volume 70, pages 249822298222 2507, 2017.
[52] R. M. Neal. Bayesian learning for neural networks, volume 118. Springer Science & Business Media, 2012.
[53] J. A. Portillo- Quiiter, J. C. Ortiz-Bayliss, and H. Terashima-Marin A straightforward frame- work for video retrieval using CLIP. In MCPR, volume 12725 of Lecture Notes in Computer Science, pages 3-12, 2021.
[54] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In ICML, volume 139, pages 8748-8763, 2021. 3, 3, 7, 10, 15, 17
[55] R. Ranganath, S. Gerrish, and D. M. Blei. Black box variational inference. In AISTATS, volume 33, pages
2014.
[56] A. M. Saxe, J. L. McClelland, and S. Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In ICLR, 2014. 16
[57] M. Sensoy, M. Kaplan, and M. Kandemir. Evidential deep learning to quantify classification uncertainty. In NeurlPS, pages 3183-3193, 2018. 3, 3,4,7 15
[58] G. Shafer. A mathematical theory of evidence, volume 42. Princeton university press, 1976. 2
[59] C. E. Shannon. mathematical theory of communication. ACM SIGMOBILE Mob. Comput. Commun. Rev., 5(1):3-55, 2001. 2.
[60] Y. Song and M. Soleymani. Polysemous In CVPR, pages 1979-1988, 2019. 1,3,4,7
embedding for cross-modal retrieval.
[61] N. Srivastava, G. E. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting, J. Mach. Learn. Res., 15(1):1929-199) 2014.
[62] J. Xu, T. Mei, T. Yao, and Y. Rui. MSR-VTT. A large video description dataset for bridging video and language. In CVPR, pages 5288-5296, 2016. 6
[63] T. Xu, P. Zhang, Q. Huang, H. Zhang, Z. Gan, X. Huang, and X. He. Attngan: Fine-grained text to image generation with attentional generative adversarial networks. 1316-1324, 2018. 10
CVPR, pages
13

---

[64] P. Zeng, L. Gao. X. Lyu, S. Jing, and J. Song. Conceptual and syntactical cross-modal alignment with cross-level consistency for image-text matching. In ACM MM, pages
2021.
[65] K. Zhang, Z. Mao, Q. Wang, and Zhang. Negative-awand attention framework for image-text matching. In CVPR, pages 15640-15649, 2022. 7
[66] S. Zhao, L. Zhu, X. Wang, and Y. Yang. Centerclip: Token clustering for efficient text-video retrieval. In SIGIR, pages 970-98 9700-81,2220 2022. 6,7
[67] 7hu, P. Zeng. I.. Gao, G. Li, D. Liao, and J. Song. Complementariy--wware space learning for video-text retrieval, IEEE Trans, Circuits Syst. Video Technol., 33(8):4362-4374, 2023. 3
14

---

Appendix
A
The Model Structure of the Baselines
aggregated
Sjocal
Syloba
max
McanPool
Frume representation
Text
representation
Text Encoder (T (Trnnsormmr))
Video Encoder (ViT)
3
group
girls
mixing
puint
Figure 7: The model structure of baseline on video-text retrieval. The model takes video-text pair as imput. For the imput video, we encode it into J frame representations. For the input text, encode it into text representation. Then, mean pooling layer aggregates J frame representations global representation, which calculates the global similarity &global with the text representation. Meanwhile, the maximum value of the similarities between J frame representations and the text representation is selected as Stocot. Finally, we take the average of Sylobal and Stocol as the final similarity s, which will be constrained by cross-entropy loss in the training process. @ means cosine similarity. @ means average addition.
For image-text retrieval, we utilize CLIP [54] the baseline straightly. As for video-text retrieval, the model structure is shown in Figure 7. To be specific, we follow the model structure of the CLIP4CLIP [49] to generate frame features Z = {af li = .!, for each video and text feature 2 for each caption. Afterwards, two branches are designed to catch global and local information, respectively. In global branch, We first adopt mean pooling to aggregate J frame features to get an 'average global feature'
Next, the cosine similarity Sylobal between z" and 2t is calculated. In local branch, we first calculate a cosine similarities S RR between of and each frame feature z/- Then we take the maximum value in as Stocal. Finally, we take the average of globbll and Slocal as the final similarity score. This similarity score will be constrained by cross-entropy loss.
B Evidence Generation
How generate evidence from the similarity between the instance and the prototype crucial for uncertainty modeling. Following [57], three optional functions, including ReLU function, softplus function, and exponential function, are used to generate evidence as follows:
ReLU function:
E(s) e(s) -
otherwise
Softplus function. Let 0 denote threshold of reverting to a linear function. 7 and 0 are separately set to 20 in the experiments. Softplus function can formulate
log(1 + exp((rs)) .78 5
e(s)
otherwise
15

---

Exponential function. Let 7 be scale parameter of the exponential function (Eq. 10), indicated as our experiments.
= exp(s/T)
(10)
The performance comparison of three evidence-generaan functions is shown in Table 7. It's worth noting that the exponential function outperforms other methods on multiple metrics. We therefore utilize it the seddence-generttinn function in all experiments.
Table 7: Evidence generation function comparison on MSR-VTT
Method
12v
v2t
R@1
R@5
R 10
MdR
MnR
R@
R@5
R 10
MdR
MnR
Bascline
45.6
72.9
81.5
2.0
14.5
45.2
71.6
81.5
2.0
10.9
ReLU
46.6
71.9
82.9
2.0
14.4
47.0
72.9
82.4
2.0
10.1
Softplus
47.6
72.7
83.0
2.0
47.8
71.9
82.9
10.3
Exponential
48.5
72.7
82.5
2.0
14.0
48.3
73.0
83.2
2.0
9.7
Additional Experimental Results
C.1 The number of prototypes
Table 8: Ablation study of prototype number K on MSR-VTT
12v R@5
v2t R@5
Mean
Method
R@1
R 10
R@1
R@10
@
w/o PAU
45.6
72.9
81.5
45.2
71.6
81.5
45.4
K=1
47.2
72.2
81.4
46.2
72.7
81.3
46.7
K=4
47.6
72.8
82.0
46.7
71.6
82.5
47.2
K=8
48.5
72.7
82.5
48.3
73.0
83.2
48.4
K=12
48.6
72.4
82.4
46.9
72.6
82.2
47.8
K=16
47.8
73.0
82.0
47.0
72.8
82.1
47.4
Multiple prototypes are built represent the overall semantics of the subspace. To figure out the best number of prototypes for representing subspace, the performance variety against prototype number K shown Table 8. can be seen that PAU always performs better than the baseline. Furthermore, @ 1 of both t2v and v2t first increases to the peak against growing K (K 12 for 12v, K = 8 for v2t) and then decreases. To model the uncertainty consistently, is set for all modalities due to the maximum mean R@ when K 8.
C.2 Different Initialization Methods for Prototypes
Table 9: Comparison of different prototype initialization methods on MSR- VTT
12v
v2t
Method
R@
R@5 5
R@ 10
R@D
R@5
R@ 10
kaiming uniform [29]
47.2
72.8
81.9
46.3
73.2
81.6
kaiming _ateealooalllllaee [29]
46.8
72.5
82.2
46.6
73.0
82.3
orthogonal [56]
47.4
73.0
83.1
47.1
74.0
83.5
Xavier [24]
48.5
72.7
82.5
48.3
73.0
83.2
To explore the impact of different initial methods to prototypes, we employ several initialization methods for prototypes on MSR-VTT The methods contain Xavier [24], kaiming normal [29], kaiming uniform coimiingunnifommmm
[29], orthogonal [56] initialization. The results are shown below. We can observe that Xavier init methods can get the best @@ performance on both 12v and v2t.
C.3 Comparing Model Size and Complexity
The FLOPPs and parameter number of PAU on MSR-VTT are computed to explore the computational costs and model complexity. We compare PAU with some previous methods, such VATT [1],
16

---

Table 10: Comparison of model size and complexity on MSR- VTT.
Method
Params (M)
FLOPs (G)
VATT [1]
327.0
792.0
Frozen [4]
180.9
771.0
MILES [23]
180.9
771.0
PAU
84.2
36.0
Frozen [4], MILES [23]. The results are shown as below. can be seen that the FLOPs and the parameter number of PAU both significantly smaller than other methods, indicating that PAU is leght approach and easy to deploy.
C.4 Training with Noisy Correspondences
Table 11: Comparison on noisy MS-COCO-
MS-COCO IK
MS-COCO 5K
Noise ratio
Method
i2t
12i
R@1 1
R@5
R@10 R@
R@5
R@ 10
R@
R@5
R@
R@
R@5
R@
VSEoo
82.0
97.2
69.0
92.6
87.1
93.3
76.7
85.5
PCME [1]]
80.1
96.6
98.7
67.6
92.1
96.9
59.9
85.8
92.3
46.1
75.0
84.6
0%%
PCMH+ [10]
81.6
97.2
99.0
69.2
92.8
97.1
62.1
86.8
93.3
48.1
76.7
85.5
Baseline
80.1
95.7
98.2
67.1
91.4
96.6
62.9
84.9
91.6
46.5
73.8
82.9
ours
80.4
96.2
98.5
67.7
91.8
96.6
63.6
85.2
92.2
46.8
74.4
83.7
VSEoo [8]
78.4
94.3
97.0
65.5
89.3
94.
58.6
83.4
89.9
45.0
72.9
81.7
PCME [11]
76.8
95.4
98.2
63.8
90.5
81.7
90.0
41.8
71.4
81.7
20%
PCME+ [10]
78.4
95.9
98.4
64.9
90.8
96.1
57.7
83.9
91.0
43.2
72.3
82.4
Baseline
76.0
94.3
97.5
63.4
89.0
94.8
55.3
79.1
86.9
41.0
68.8
79.3
ours
78.2
95.2
98.1
64.5
90.0
95.4
59.3
82.9
90.4
44.2
71.3
81.3
VSHoo [8]
44.3
76.1
86.9
34.0
69.2
84.5
22.4
48.2
61.1
15.8
38.8
52.1
PCME [11]
73.4
94.0
97.6
59.8
88.3
94.8
50.9
78.8
87.3
37.9
67.3
78.4
50%
PCME+ [10]
74.8
94.3
97.7
60.4
88.7
95.0
52.5
79.6
88.4
38.6
68.0
79.0
Baseline
73.9
93.0
87.3
94.0
78.5
86.6
39.7
67.2
77.5
76.4
94.1
97.6
62.3
88.5
94.6
57.3
81.5
88.8
41.9
69.4
79.6
additionally compare PAU with some CLIP ViT-B/32 backbone-based methods [8, 11, 10, 54] fine-tuned noisy MS-COCE dataset. The results of training 0%, 20%, and 50% noise ratios are reported in the Table 11. We can observe that our method exhibits significant advantages while 50% noise ratio. Although the uncertainty-
training with several noisy correspondences, especially aware method, PAU, is not designed for tackling the noisy correspondences, still successfully handles the scenario, indicating that PAU possesses powerful robust learning ability even while training with noisy correspondences.
D The Explanation of High Boost on MSVD
In Table 2, we find the performance improvement MSVD on v2t is significantly larger than other benchmarks. Investigating its reason, the MSVD is tiny dataset with only 1,200 videos for the training set and approximately 40 captions per video (48,000 captions in total). In this case, utilizing cross-entropy loss, which views the diagonal pairs as positives, will lead into tremendous noise, especially with a large batchsize. In a batch with the size of 256, the probability of any texts coming from different videos is about 3.49 10 The solution is as follows.
B CH P
(11)
C
The probability of any texts coming from different videos can denote Eq. 11, where B is the batchsize of 256, and is the total caption number of 48,000. m, indicates the candidate number
Results come from: http::/gtttbb.coom/aaaaaaapppeet
17

---

of tth text in batch with m, 48000 - 40 x (i - 1) in MSVD. Then the probability P can be calculated by:
2
C
48000 (xx00-4
P
C256
48000
256
256
log P log 48000 x (i 1))
Σ
log (88000 - 1)
(12)
log P (log 48000 + log 47960 + log 47920
10g37800)
(log 48000 + log 47999 - log 47998 + ... + +og -log447740 47745)
log P 28.685
P ~ 3.49110-3,
The resultant noises mislead the model training, resulting in a terrible performance reduction. Our uncertainty-awe aware approach can mitigate the detriment caused by these data and improve the model robustness effectively.
E Proof of Ambiguous Data with High Information Entropy
In Section. 1, we propose theorem that "the ambiguous multi-modal data highly associated with multiple semantics, such fast-paced videos and non-detailed texts, lead to high information entropy, standing for high uncertainty." The mathematical Theorem and Proof are given below.
Theorem. Suppose E IR." is the vector of multi-modal data. Assuming the modality semantic subspace can be divided into K finite categories as X {oz E R =,,, ... The information entropy is denoted as Eq. 13:
K
1(X) = Σ p(z)) p(x.) s.t.
Σ [(.)))
(13)
where p(x;) the tth semantics probability derived from cosine similarity cos(2, (2) through softmax layer. Let Q (ax) be Discrete Uniform Distribution [14], then have:
J(P ())) x H(X)
(14)
where J(P Q) denotes the Lensen-Shannen Divergence (ISD) [16], metric to measure the distance between two probability distributions. Namely, Eq. 14 means that the closer the distribution P ~ p(z) is the Uniform Distribution q(x). the higher the information entropy H(X) is.
Proof. Let's first figure out what distribution of information entropy is the largest. Given an objective function (Eq. 15) and its constraint (Eq. 13):
max H(X) -p(x1) log p(22) logr(*2) - - p(œx) logp(xx). P
(15)
Let denote Lagrange multiplier. The Lagrangian function can constructed as:
((p, A) p(:)) log A
Σ p(xx) 1)-
(16)
Separately computing the partial derivatives for p(x1), pl (22) p(xx))
de(p. x) = À - logg(x;)) = O.
(17)
0p(x;)
From this:
logp(x;) = ((g2( = = log (ogg(TK) X.
(18)
Considering the constraints of Eq. 13, the p(æ) essentially satisfy:
p(z,) = p(æ2) = =.. - =p(xx)
(19)
18

---

which is the density of discrete uniform distribution. Therefore, the H(X gets the great value log K when the probability distribution conforms Discrete Uniform Distribution [14].
Then -ansen-Shannon Divergence (ISD) [16] is used to measure the distance between the probability distribution P p(z) and the Discrete Uniform Distribution Q (xx).
1
1 E (x; log( g(xA) p(x,)) (xx))
p(x;)
[(P Q1)
p(z;) log(
+
+
p(x+) q((s))
p(x;) log p(g) Σ poa))o(g(((a))) g(z)))
(20)
+ Σ (x:;) log ((x;)
Σ log(p(z;) + g(x;))) + 1.
Since q(x) conforms the Discrete Uniform Distribution, ((a)) k and E g(ar)) log q(xi)
log K. The JSD function (Eq. 20) can be converted to:
K
J(P = 2 Σ logp()))
Σ p(a;) log( + p(xi))
K
log K
log( + p(x:))) +1
(21)
K
K
1=1
K
K + p(z;)) log( 1
p(z,) log
Σ
K + p(x))) log K) + 1.
Note that there are two components affecting the J(P || Q). Σ p(x.)) log(; 1 + p(x.)) For convenience, let: K
and Σi
K
A(P) -H (X) Σ
p(x;) logp(z;)
1=1
(22)
B(P)
p(xx)) log( p(xx)).
It can be seen that [| Q) & (A(P) B(P)). When P ~ p(x) is closer to the Discrete Uniform Distribution Q g(z). i.e., [(P || Q) 4. three cases are possible for A(P) and B(P):
a. A(P) & and (PP) L, but the the former has larger reduction, i.e., AA(P) AB(P) < <.
b. A(P) + and ((P) t.
c. A(P) ? and (PP) 1, but the latter has larger growth, i.e., AA(P) - AB(P) <
+ 0 is the amount of function change. A(P) are both declining in cases a and b, reflecting the possibly induces the decrease of HQq (X) with the decrease of [(P || Q). If there is no case suit the theorem is proved.
increase of Hq~q(=)(X) (X) according to Eq. 22. Thus, only
Let f() x: log Its first-order derivative and second-order derivative f(x) log 1 and f(x)" 0, respectively. It is convex function obtaining the minimal value at 1
Due to ΣX1 p(x;)= = when p(x.) - p(xa) + A, there must a p(ws) - p(xs) A. Subject to A(P) t. Eq. 23 should be satisfied:
AA(P) = f(p(za)) ff(((z))) log p(Ja) p(x))
> p(xa) > p(z)).
(23)
19

---

When 0 < p(zs) - A < p(rb) < p(aa) < p(za) + A < 1, we have:
1 x f(((aa) )) -BPP) (f(v(za)) - ((o(xb)))) ((((z))) (s(p(za)
log p(faa p( + I log p((b)
(24)
K p((b) +
1
log(1 + Kp(zs) log(1 + Kp(xa))
0.
It indicates that case is impossible case when J(P || Q) Thus, only and are possible when J(P Q) 4. In these two cases,
[(P A(P) - Howpla) (X)
(X)T. f-
(25)
Therefore,
J(P
(26)
H(X)
demonstrating that the closer uniform distribution, the larger the information entropy is. It concludes the proof.
20

---

The Detailed Video Visualization
In Figure 6 of the main paper, we present several video fragments with their uncertainty, In Figure more detailed videos are displayed to illustrate the uncertainty degree of the videos.
Figure 8(a) is video with 0.881 uncertainty expressed in the video. Obviously, as the uncertainty diminishes, the number of scenes in the videos decreases, and the videos become easier understand.
Actually, is difficult understand the meaning
10
12
13
14
Uncertainty: 0.881
10
32
13
14
Uncertainty: 0.738
(b)
10
11
12
13
14
Uncertainty: 0.681
10
12
33
14
Uncertainty: 0.546
(d)
21

---

10
11
12
13
14
Uncertainly: 0.446
(c)
Uncertainty: 0.335
Uncertainly: 0.226
Uncertainty: 0.119
(h)
Uncertainty: 0.026
Figure 8: The detailed video visualization of data uncertainty on MSR- VTT.
22


Reviews:
[{"review_id": "6ks8WAUvrn", "reviewer": "Reviewer_qbVB", "summary": "This paper presents the Prototype-based Aleatoric Uncertainty Quantifications (PAU) framework, which enhances the reliability of a pre-trained vision-language model (CLIP) by leveraging evidential knowledge. The paper introduces learnable prototypes as semantic categories and maps images and texts to these categories for uncertainty quantification. The proposed method improves model training and ranking in image-text retrieval tasks.", "questions": "Please refer to my question in the Weaknesses section.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The paper addresses the issue of model reliability and provides a reasonable approach to quantify data uncertainty in image-text retrieval tasks.\n- The introduction of trainable prototypes complements contrastive learning-based vision-language alignment by incorporating evidence knowledge. This leads to improved reliability of the pre-trained model.\n- The paper provides extensive experimental results and a detailed ablation study, demonstrating the effectiveness of the proposed method compared to existing image-text retrieval approaches.", "weaknesses": "- The loss function defined in Equation 5 raises confusion. The uncertainty is computed in the K category domain, while the mean similarity is computed in the N category domain. Could the author explain why these two domains are mapped together?\n- The paper includes an ablation study on the number of trainable prototypes (K), showing that the best R@1 performance on MSR-VTT is achieved when K=8. Concerning the scalability of the method, would K vary on different datasets to achieve the best performance?\n- Figure 5 visualizes the uncertainty of videos and captions. However, the connection between the data pairs is not clear. Could the authors explain the rationale behind selecting these specific examples?\n- The paper lacks a clear introduction to the baseline models, including their training process and model structure. It would be helpful to provide more details in this regard. Also, the reporting of experimental results seems inconsistent or incomplete.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer qbVB for the valuable and constructive feedback. We will address the raised questions point-by-point as below.\n\n***Q1 : The loss function defined in Equation 5 raises confusion. Why uncertainty computed by K category domain and mean similarity computed by N category domain are mapped together?***\n\n**A1:** Thanks for the constructive question. In the Introduction, we explain why multi-scene videos and non-detailed texts have high uncertainty using information entropy. And in the histogram of Fig. 1(a), we can find that the multi-scene video B have a higher mean similarity with the text modality than Video A. Thus, we define that an instance with high uncertainty score $u$ should have high mean similarity $h$ with the instances from the other modality, i.e., $u$ and $h$ should be positively correlated. In the training process, given a specific video $q$, we use $K$ prototypes to compute $u_q$ and use $N$ captions to compute mean similarity $h_q$. According to our definition before, $u_q$ and $h_q$ should be positively correlated, and they have the same range $[0,1]$. Thus, we can adopt Eq. 5 to force $u_q$ closer to $h_q$ directly.\n\n***Q2 : Would K vary on different datasets to achieve the best performance?***\n\n**A2:** To verify the scalability of the method, we additionally conduct K-varied experiments on MSVD and DiDeMo datasets. All results of K ablation are shown below. \n|||MSR|VTT||||||DiDeMo||||||MSVD|||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|||**t2v**|**v2t**||**Mean**|||**t2v**|**v2t**||**Mean**|||**t2v**|**v2t**||**Mean**|\n||**R1**|**R5**|**R1**|**R5**|**R1**||**R1**|**R5**|**R1**|**R5**|**R1**||**R1**|**R5**|**R1**|**R5**|**R1**|\n|K=1|47.2|72.2|46.1|72.7|46.7||46.9|75.9|46.1|74.6|46.5||46.4|76.9|64.8|93.2|55.6|\n|K=4|47.6|72.8|46.7|71.6|47.2||48.0|75.8|47.3|74.8|47.65||46.6|77.0|67.9|92.9|57.25|\n|K=8|48.5|72.7|**48.3**|**73.0**|**48.4**||**48.6**|76.0|**48.1**|74.2|**48.35**||**47.3**|**77.4**|**68.9**|**93.1**|**58.1**|\n|K=12|**48.6**|72.4|46.9|72.6|47.8||48.1|75.8|46.2|75.4|47.15||46.9|77.1|68.3|92.2|57.6|\n|K=16|47.8|**73.0**|47.0|72.8|47.4||**48.6**|**76.9**|47.7|**75.5**|48.15||47.0|**77.4**|**68.9**|92.5|57.95|\n\nWe find that the best Mean R@1 scores are obtained when K=8 in all three datasets. Meanwhile, when K is greater than 8, the performance change caused by K growth is minimal.\n\n***Q3 : What is the rationale behind selecting the examples in Fig. 5?***\n\n**A3:** This paper mainly focuses on quantifying the uncertainty of a single video (image) or a single text, not vision-language pairs. Thus, the images and texts in Fig. 5 don't have a special connection. We want to show the samples with uncertainty scores sorted from large to small in text and video modalities, respectively. For example, the first row is the samples of a text and a video with high uncertainty. The third row is the samples of a text and a video with low uncertainty. \n\nIf we want to compute the confidence $c$ of a vision-language pair, we can utilize the equation like $c=(1-u_v)\\times(1-u_t)$. For example, assuming the text and the video in the first row of Fig. 5 make up a vision-language pair. The data quality confidence of this pair can be computed as $c=(1-0.984)\\times(1-0.881)=0.001904$. \n\n***Q4 : More details of the baseline training process and their model structure.***\n\n**A4:** For image-text retrieval, we utilize CLIP [1] as the baseline straightly. As for video-text retrieval, the training process and model structure can be found in **Fig. 9 of attached PDF file in common response**. We follow the model structure of the CLIP4CLIP [2] to generate $J$ frame representations $Z=\\\\{z^f_i|i=1,2,..,J\\\\}$ for each video and a text representation $z^t$ for each caption. Afterward, two branches are designed to separately catch global and local information. In the global branch, We first adopt a mean pooling to aggregate $J$ frame representations to get an \"average global representation\" $\\bar z^v$. Next, the cosine similarity $s_{global}$ between $\\bar z^v$ and $z^t$ is calculated. In the local branch, we first calculate a set of cosine similarities $S \\in \\mathbb R^J$ between $z^t$ and each frame representation $z^f_i$. Then we take the maximum value in $S$ as $s_{local}$. Finally, we take the average of $s_{global}$ and $s_{local}$ as the final similarity score, which will be constrained by a cross-entropy loss.\n\n***Q5 : The reporting of experimental results seems inconsistent or incomplete.***\n\n**A5:** Thanks for your comment. We supplement several experiments. We first compare PAU with some CLIP ViT-B/32 backbone-based methods on MS-COCO with different noise ratios. The results are shown in Table below. We can observe that our methods outperform all other methods with the same backbone. Furthermore, PAU emerges a powerful ability for robust learning even in 50% noise ratio.\n|||AVG R1||\n|-|-|-|-|\n|**Method**|**0% noise**|**20% noise**|**50% noise**|\n|VSE$\\infty$ [5]|**55.2**|51.4|18.4|\n|PCME [6]|53.0|48.1|43.0|\n|NCR [4]|-|48.8|45.5|\n|CLIP [1]|54.7|48.3|46.8|\n|PAU|**55.2**|**51.8**|**49.6**|\n\nBesides, We analyze the complexity of PAU and compare it with some famous previous methods. Concretely, We compute the FLOPs and parameter number of PAU on MSR-VTT. The results are shown in Table below. We find that the FLOPs and the parameter number of PAU are both significantly smaller than other methods, which means that PAU is a light-weight approach and easy to deploy.\n|Method|FLOPs (G)|Params (M)|\n|-|-|-|\n|VATT [10]|792|327|\n|Frozen [11]|771|180.9|\n|MILES [12]|771| 180.9|\n|PAU|**36**|**84.2**|\n\nWe also conducted several other experiments, such as pearson correlation analysis to prove the prototypes correctly represent the overall semantics, uncertain data analysis to explore the superiority of PAU over probabilistic methods, etc. Limited by the rebuttal character number, we cannot show more experiment results. But we will add all these experiment results in the final paper."}, {"review_id": "JmQ2dr3DIt", "reviewer": "Reviewer_VBVJ", "summary": "This paper proposes prototype-based aleatoric uncertainty quantification (PAU) to represent aleatoric uncertainty (i.e., data uncertainty) for cross-modal retrieval tasks. The proposed method uses the CLIP backbone, where the initial similarity matrix M is computed by the output of the CLIP encoders. Then, \"evidence\" $e$ is computed by the similarity between each modality embedding output and the prototype vectors. Using $e$, the belief $b$ and the uncertainty $u$ is computed. As far as the reviewer understood, the computed $u$ represents how the current embedding is *different from* the prototype vectors, i.e., if the current embedding is specifically different (in terms of the cosine similarity) from the prototype vectors, then its uncertainty becomes large. Using the learned uncertainty $u$, the proposed method re-rank the retrieved items by cosine similarity of the output of the CLIP encoders. Two losses are introduces to make the prototype vectors better, the uct loss (minimize the distance between average embedding vectors and prototypes) and the diversity loss (orthogonal regularization for prototype vectors). The effect of the proposed method is evaluated on four datasets.", "questions": "Please check my comments in the weakness section. My major concerns are (1) I am not sure whether applying SL to learnable prototypes can lead to proper uncertainty estimations (2) missing significant related works, such as [A] and [B] ([B] could be considered as a contemporary work because [B] is a CVPR 2023 paper, that is published after the NeurIPS deadline).", "limitations": "I don't think this paper has a potential negative societal impact.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- Measuring aleatoric uncertainty for multi-modal learning is an interesting and important problem.\n- The proposed method looks efficient and easily adaptable to various methods.\n- The proposed method is a somewhat straightforward extension of Subjective Logic (SL) [27] from the fixed class labels to the learnable prototypes. Hence, it may have the good theoretical properties shown by the SL paper [27], although it is not explicitly described in the paper.", "weaknesses": "My concerns can be summarized as:\n\n- It is not clear that replacing fixed class information of SL with learnable prototype embeddings will ensure a proper uncertainty estimation.\n- This paper misses the important previous work on the aleatoric uncertainty estimation method for cross-modal retrieval [A].\n- The comparisons of MS-COCO 5K would not be fair (only the proposed method uses the pre-trained CLIP model), and many recent state-of-the-art methods are missing.\n\n### Can the proposed method provide a proper uncertainty estimation?\n\nThe proposed method highly depends on SL [27], where SL uses fixed class information; hence, there is no learnable prototype. As this paper introduces new learnable prototypes, the proposed method requires two additional objective functions, Eq (5) and Eq (6). I am not sure how Eq (5) and Eq (6) will guarantee the quality of the proper uncertainty measure.\n\nEq (6) looks okay, but it enforces the orthogonality of the prototype embeddings, not the mutual exclusion among the embeddings. The latter one would be derived from independence, not orthogonality. Note that independence and orthogonality are highly related and somewhat exchangeable in many cases, I am not sure that smaller orthogonality regularization (i.e., Eq (6)) can ensure the weaker dependency between the prototype vectors. Moreover, I am not sure that Eq (6) is specifically better than widely-used orthogonality regularization, such as $\\| I - Z^\\top Z \\|_F^2$, where $I$ denotes the identity matrix, and $Z$ denotes the full prototype vectors. Another possible alternative is using Maximum Mean Discrepancy (MMD) or Hilbert-Schmidt Independence Criterion (HSIC), whose values directly induce the strength of the dependency between two probabilistic densities. In my opinion, Eq (6) could be better, but Eq (6) itself is okay and can be somewhat sound regularization.\n\nTo me, the meaning of Eq (5) is ambiguous. It seems that Eq (5) behaves as minimizing the distances between prototypes and the embeddings; thereby, it enforces to make the value of $e_K$ larger (i.e., make the uncertainty value $u$ smaller). I am not sure how Eq (5) helps the proper uncertainty estimation.\n\nI think both Eq (5) and (6) can be explained with some in-depth discussions with minimal changes in the manuscript, but as of now, I am not fully confident whether the proposed method can represent proper uncertainty estimations despite of using learnable prototypes rather than fixed class information.\n\n### Missing related works on uncertainty estimation\n\nThis paper misses many directly related works on uncertainty estimation and cross-modal retrieval. Especially this paper did not consider PCME [A], a probabilistic embedding approach to represent aleatoric uncertainty (i.e., the quality of the data) for cross-modal retrieval.\n\n- [A] Chun, Sanghyuk, et al. \"Probabilistic embeddings for cross-modal retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2021.\n\nI think [A] should be discussed in the related work section and in the experiment section if possible.\n\nThere are also more aleatoric uncertainty estimation methods based on probabilistic embeddings, such as:\n\n- Shi, Yichun, and Anil K. Jain. \"Probabilistic face embeddings.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2019.\n- Park, Jungin, et al. \"Probabilistic representations for video contrastive learning.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- Neculai, Andrei, Yanbei Chen, and Zeynep Akata. \"Probabilistic compositional embeddings for multimodal image retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n- Sun, Jennifer J., et al. \"View-invariant probabilistic embedding for human pose.\" Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16. Springer International Publishing, 2020.\n\nI would like to suggest adding discussions of the comparisons of the SL-based aleatoric uncertainty estimation and the probabilistic embedding-based aleatoric uncertainty estimation.\n\n### Missing comparison methods for MS-COCO 5K\n\nIt could be a minor issue, but I think Table 4 misses many relevant and state-of-the-art methods for cross-modal retrieval. For example, as my previous comment, PCME [A] is a highly related work to this paper. In terms of using multiple diverse embeddings, [B] could be another relevant work to the proposed method, which is more relevant to PVSE [49]. I also would like to suggest adding more recent state-of-the-art cross-modal retrieval works, such as VSE$\\infty$ [C].\n\n- [B] Kim, Dongwon, Namyup Kim, and Suha Kwak. \"Improving Cross-Modal Retrieval with Set of Diverse Embeddings.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n- [C] Chen, Jiacheng, et al. \"Learning the best pooling strategy for visual semantic embedding.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2021.\n\nEspecially, Table 4 can be unfair because all other comparison methods do not use the pre-trained CLIP backbone; hence their performances are much worse than a simple \"baseline\" of this paper. Therefore, in my opinion, Table 4 should contain the comparisons with other methods using the pre-trained CLIP backbone.\n\nHowever, I fully understand that it is not easy to compare with other cross-modal retrieval methods, especially when they apply different backbones, pre-extracted features, and optimization techniques. After a quick Google search, I found a very recent work, that uses the CLIP ViT-B/32 backbone for cross-modal retrieval:\n\n- [D] Chun, Sanghyuk. \"Improved Probabilistic Image-Text Representations.\" arXiv preprint arXiv:2305.18171 (2023).\n\nI think the experimental results of [D] can be directly comparable with Table 4 of this paper, including VSE $\\infty$, PCME, and InfoNCE. I would like to suggest adding the results of [D] to the main paper if the authors are willing to update their table to contain more recent papers. *Note that [D] is not a direct comparison method of the proposed method because it arxived after the NeurIPS submission deadline*; therefore, I don't argue that [D] is a significant related work that should be compared with the proposed method, but I bring [D] because [D] uses the same CLIP ViT-B/32 backbone with the experiments in this paper, and it reported [A] and [C] with the ViT-B/32 backbone, listed in my review.\n\n### Minor comments\n\n- The implementation details of the overall framework are somewhat hard to catch at first glance. I suggest adding the detailed process for computing $e$ and $u$ in Figure 2, and adding the details of $e$ (Table 7 in Appendix) to the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer VBVJ for the positive, patient, professional review and the valuable suggestions for improvement. Our responses to the raised questions are below:\n\n***Q1 : Compared with probabilistic-based uncertainty estimation.***\n\n**A1:** Thanks for your professional and valuable comments. PCME [6] is a probabilistic-based approach to represent instance diversity, which can also represent uncertainty. The more diverse the instance, the more uncertain the instance. Thus, an instance's uncertainty and diversity should be positively correlated. To verify the superiority of SL-based method over than probabilistic-based method, we conduct two experiments, pearson correlation analysis and uncertain data analysis.\n\n(1) **Pearson correlation analysis**. The pearson correlation coefficient (**PCC**) $r_{xy}$ ($r_{xy}\\in [-1,1]$) is a metric to estimate the strength of the correlation between variable $x$ and $y$. Generally, $r_{xy}>0.5$ means that two variables are strongly positively correlated. As mentioned, an instance's uncertainty and diversity should be positively correlated for PCME. A diverse instance will have a high mean similarity $h$ with the instances from the other modality. This means the uncertainty score $u$ and mean similarity $h$ should also be positively correlated. Thus, we compute the PCC between $u$ and $h$ to explore whether PAU correctly estimates uncertainty. The results are shown below.\n\n|Method|PCC-V|PCC-T|Dataset|\n|-|-|-|-|\n|PCME|-0.071|-0.219|MSCOCO|\n|PAU|0.886|0.786|MSCOCO|\n|PAU|0.917|0.939|MSR-VTT|\n\nIn this Table, we compute the PCCs of PCME and PAU on MSCOCO, and PCCs of PAU on MSR-VTT. PCC-V and PCC-T separately indicate the instance's PCC in vision or textual modality. The low PCCs of PCME mean current probabilistic models have limited abilities to reflect uncertainty and diversity. We argue the reason arises from the strong prior distribution assumption, i.e., gaussian or other simple distributions are powerless to fit complex and diverse relationships in high-dimension space. By contrast, PAU obtains high PCCs in all settings, proving our approach indeed leads to proper uncertainty estimations. \n\n(2) **Uncertain data analysis**. To further prove the superiority of the SL-based method over the probabilistic-based method, we compare the changes of R@1 after removing top-r instances with the highest uncertainty scores quantified by PCME and PAU, respectively. To fairly compare, we employ removal on both predictions arising from CLIP and PCME. The results with the removed ratio from 0% to 20% on MS-COCO 5K is shown in **Fig. 8 of the pdf in common response**. We can observe that PAU outperforms PCME in all directions and predictions, meaning the uncertain data found by PAU is more precise than PCME. All experiments indicate the SL-based approach (PAU) works better than the probabilistic-based method (PCME) to represent aleatoric uncertainty.\n\n***Q2 : Does PAU provide a proper uncertainty estimation?***\n\n**A2:** In A1, high PCCs of PAU mean the uncertainty score $u$ and mean similarity $h$ are strongly positively correlated, proving that our approach indeed leads to proper uncertainty estimations. \n\nBesides, in Fig. 3 of the main paper, we conduct experiments to explore the impact of uncertain data by comparing the retrieval results removing these uncertain data and removing random data. The results show that removing these uncertain data found by PAU can significantly improve retrieval performance. This can also prove that PAU possesses the ability to precisely estimate uncertainty.\n\n***Q3 : The discussion of Eq. (6)?***\n\n**A3:** Thanks for your valuable comments. To verify the effectiveness of Eq. 6, we randomly sample K embeddings with Xavier init, representing the prototypes before training. Then we compute their mean similarity $\\bar s$ between each other. Afterward, we compare it with the mean similarity between each prototype. We repeat the sampling trial 5 times and report the results below.\n\n||$t_1$|$t_2$|$t_3$|$t_4$|$t_5$|$P_V$|$P_T$|\n|-|-|-|-|-|-|-|-|\n|$\\bar s$|-1e-2|5e-3|-1e-2|-1e-2|-1e-2|9e-6|-2e-5|\n\n$P_V$ and $P_T$ denote the mean similarity of visual and textual prototypes, respectively. We can see that the mean similarity scales of $P_V$ and $P_T$ are about 0.001 to 0.0001 times the scale of random sampling, which means that Eq. 6 indeed pulls prototypes away from each other to a considerable distance.\n\nBesides, we also conduct the experiments on MSR-VTT using widely-used orthogonality regularization $|I-Z^\\top Z|_F^2$ and Maximum Mean Discrepancy (MMD) instead of Eq. 6. The results are shown in Table below. We can observe that original Eq. 6 obtain the best performance on both t2v and v2t.\n|||t2v|||v2t||\n|-|-|-|-|-|-|-|\n|Method|R1|R5|R10| R1|R5|R10|\n|orthogonality|47.7|72.4|82.4|47.5|72.4|82.3|\n|MMD|47.6|72.2|81.8|47.6|72.0|82.1|\n|original Eq. 6|48.5|72.7|82.5|48.3|73.0|83.2|\n\n***Q4 : The meaning of Eq. 5?***\n\n**A4:** In this paper, we define that an uncertain instance should have higher mean similarities with the instances from the other modality. i.e., as the A1(1) mentioned, the uncertainty score $u$ and mean similarity $h$ should also be positively correlated for an instance. Thus, Eq. 5 can be regarded as a constraint function to force $u$ closer to $h$. Then we can get a more precise uncertainty estimator.\n\n***Q5 : Missing comparision with ViT-B/32 backbone-based methods on MS-COCO 5K.***\n\n**A5:** Thanks to the nice reviewer for providing the necessary papers. Limited by the Rebuttal character number, we discuss these in **\"More Method Comparision in Different Noise Ratios\" of Common Response**.\n\n***Q6 : Adding some detailed processes for computing e and u in Fig. 2 and main paper.***\n\n**A6:** Thanks for your suggestion. We will add details of e, u computing process to Fig. 2. Meanwhile, we will transfer the e generation from Appendix to the main paper. Limited by the Rebuttal character number, we will update it in the final paper."}, {"review_id": "S6wBslJVJb", "reviewer": "Reviewer_tGbU", "summary": "This paper focuses on improving video-text retrieval by considering the uncertainty (or confidence) when computing image-text similarity. The high-level idea is to estimate the data uncertainty and then perform re-ranking to improve the performance of retrieval.\n\nThe authors designed a PAU framework, Prototype-based Aleatoric Uncertainty Quantification with CLIP encoders, to estimate the uncertainty score of the input instances. The authors conducted task-specific fine-tuning on the target datasets, and reported marginal performance improvements.", "questions": "see weakness", "limitations": "No potential negative societal impact", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The authors addressed an interesting problem of estimating the uncertainty of data, which is used to improve retrieval performance.\n- The proposed PAU framework is presented clearly.\n- Both the uncertainty loss and diversity loss are technically valid and straightforward.   \n- The authors conducted several experiments on multiple benchmarks, and demonstrate marginal performance improvements.", "weaknesses": "- The goal and task definition of this paper seem not clear. Both the introduction section and Figure 1 are somewhat confusing, which makes me difficult to get the main point. An image (or a single video frame) should contain rich semantics. Typically, an image is paired with a caption (or sentence), forming a image-text pair, to describe the semantics. However, the authors seem to argue that a single-scene video should match only a single word (i.e., talking), which is counter-intuitive. The multi-scene video may have richer semantics than single-scene video, but the authors argue that matching multiple semantics is not ideal. Overall, their key idea is counter-intuitive.   \n- The authors conducted experiments solely on downstream fine-tuning. The fine-tuning datasets are usually cleaner and with higher quality. A majarity of the noisy image-text paired data should be found in VL pre-training dataset. While downstream fine-tuning data (like youcook2 or MSRVTT) are human annotated/collected, the pre-training data is web-crawled noisy imgae-text pairs. To study the capability of uncertainty estimation for low-quality data, it would be much more interesting to experiment their proposed method in the pre-training stage. The proposed method may have potential to greatly improve the pre-training, but it remains unknown in current stage.\n- The paper is not written well, making it difficult to understand the motivation. This is probably due to unprecise or uncommon wording in the writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer tGbU for providing constructive feedback. In what follows, we wish to address the questions you raised as below.\n\n***Q1 : The authors seem to argue that a single-scene video should match only a single word (i.e., talking), which is counter-intuitive.***\n\n**A1:** We are sorry for causing some confusion to you. Actually, we don't argue that a single-scene video should match only a single word (i.e., talking). The example of **Video A in Figure 1(a)** matching the word **\"talking\"** is just one instance. It can also match the caption **\"two people are talking\"**. However, whether it is \"talking\" or \"two people are talking\", they both similarly convey the summary semantics of **\"talking\"**. But the multi-scene videos are hard to be summarized to a single semantics because the frames describing different scenes can match different semantics. The focus of this example is only to **illustrate the difference** in semantic richness between single-scene videos and multi-scene videos.\n\n***Q2 : The multi-scene video may have richer semantics than single-scene video, but the authors argue that matching multiple semantics is not ideal.***\n\n**A2:** Thanks for your comment. We don't argue that the multi-scene video matching multiple semantics is not ideal. By contrast, we claim that multi-scene videos are easy to match with multiple semantics. For example, in **Figure. 1(a) of the paper**, single-scene video A can be briefly summarized as the simple word **\"talking\"** or the brief caption **\"two people are talking\"**. In contrast, the multi-scene video B is hard to summarize briefly because of its rich semantics, leading to confusion in the retrieval process, i.e., it is hard to be described using a simple description. Thus, this confusion is the **uncertainty** we defined in our paper. The primary concept of this work is to identify these easily confused instances and mitigate their impact on the retrieval process. We use uncertain scores to **quantify the confusion degree these instances introduce** during the retrieval process. It is worth noting that our work is to quantify the uncertainty of individual instances (a video, an image,  or a text), not a vision-language pair. But we can compute the confidence $c$ of a vision-language pair using $c=(1-u_v)\\times(1-u_t)$. $u_v$ denotes the uncertainty score of the video in this pair. $u_t$ denotes the uncertainty score of the text in this pair.\n\n***Q3 : Training in the pre-training phase on web-crawled noisy dataset.***\n\n**A3:** Thanks for the constructive suggestions. However, limited by the high computation cost and time, it is hard to test our method in the pre-training phase. We will achieve it in the future work. Instead, we train our method on the noisy dataset and compare it with other **CLIP ViT-B/32 backbone-based methods**. We follow the noise split of NCR [4] with **20%, 50%** relationships randomly shuffled. It can be seen that PAU outperforms all other methods with a significant gap in 20% and 50% noise ratios. This clearly demonstrates that PAU exhibits robust learning capabilities as well.\n\n**Note:** CLIP is our baseline.\n\n|           |          | AVG R1    |            |\n|-----------|----------|-----------|------------|\n| **Method**| **0% noise**|**20% noise**| **50% noise**  |\n| VSE$\\infty$ [5]| **55.2**| 51.4      | 18.4       |\n| PCME [6]     | 53.0     | 48.1      | 43.0       |\n| NCR [4]      | -        | 48.8      | 45.5       |\n| CLIP [1]     | 54.7     | 48.3      | 46.8       |\n| PAU       | **55.2** | **51.8**  |**49.6**    |\n\n***Q4 : Some unprecise and uncommon wordings make motivation be understood hard.***\n\n**A4:** Thanks for your comment. We guess that some wording like **\"fickle video\"**, **\"non-detailed text\"**, and **\"ambiguous data\"** confuse you. **Fickle video** means that the video frame frequently changes in multiple scenes (e.g., **Figure. 6(a) in Appendix**). In our definition, a multi-scene video usually has richer semantics than a single-scene video. Therefore, it has high semantic relevance with more descriptions, and it is hard to summarize with a simple caption or word. **The non-detailed text** means text with general semantics, which can semantically match more videos or images than detailed text. Fickle videos and non-detailed texts jointly make up **ambiguous data**. Additionally, we are sorry for lacking the legend in Figure 1(b), which may cause confusion to you. Now, we have corrected it, which can be found at the **Figure 7 in the attched pdf file of Common Response**. If there are any other confusing wordings or further questions, feel free to let us know and we are more than happy to address them. All of these will be updated in the final paper. "}, {"review_id": "wcvePP13OQ", "reviewer": "Reviewer_b4gq", "summary": "This paper addresses the aleatoric uncertainty quantification in cross-modal retrieval. The authors point out that the model predictions are often unreliable due to the aleatoric uncertainty, which is induced by low-quality data. They first define the inherent aleatoric uncertainty in multi-modal data. Then K prototypes are constructed as the belief masses to employ DST and SL, two classical theories in uncertainty quantification. On top of this, the uncertainty loss and diversity loss are proposed to help prototypes represent the overall semantic space. Extensive experiments are conducted on multiple tasks and benchmarks.", "questions": "Refer to weaknesses.", "limitations": "Yes", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.\tThis work presents a novel insight for the uncertainty of cross-modal retrieval. They classify uncertainty into aleatoric uncertainty and epistemic uncertainty, and give a clear definition of aleatoric uncertainty for multi-modal data. I think it will inspire further uncertainty-aware work in cross-modal retrieval or multi-modal learning.\n2.\tIn Appendix, the proof of “the ambiguous multi-modal data highly associated with multiple semantics” is intuitive and reasonable.\n3.\tThe experiments greatly prove the claims, especially Fig. 3 and Fig. 4. These two experiments reflect that the uncertain data indeed have a significant negative impact on results, and PAU could provide more trustworthy predictions. This phenomenon may be able to give an inspiring insight for robust training.", "weaknesses": "1.\tIn the reranking process, beta_1 and beta_2 are two learnable parameters. But it is unclear how these parameters are trained in this paper. \n2.\tPrototypes are expected to represent the overall semantics space. Do they really represent the overall semantics space after training? \n3.\tThe paper only mentions uct loss and div loss, which seem to be proxy-based loss functions, and can not optimize the prediction directly like cross-entropy loss or triplet loss. I wonder if other basic losses are used in the training process.\n4.\tThe Table 7 in appendix shows that the exponential function is the best method to generate evidence. Why there is a large performance gap between ReLu function and exponential function? Or what is the advantages of exponential function.\n5.\tThe legend in Figure 1 (b) is empty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer b4gq for providing professional and valuable feedback, as well as encouraging comments. In what follows, we wish to address some of the questions you raised as below.\n\n***Q1 : How learnable parameters $\\beta_1$ and $\\beta_2$ are learned in PAU?***\n\n**A1:** Thanks for the valuable question. $\\beta_1$ and $\\beta_2$ are two learnable parameters to control the impact degree of re-ranking. After the model is trained, we first freeze other parameters of the model. Then, we utilize Eq. 7 to re-rank the similarity matrix. Finally, we re-trained the model for few steps constrained by a normal cross-entropy loss. After re-training, we can obtain the learned parameters $\\beta_1$ and $\\beta_2$, which can output the best re-ranked predictions.\n\n***Q2 : Prototypes are expected to represent the overall semantics space. Do they really represent the overall semantics space after training?***\n\n**A2:** Thanks for the constructive suggestions. To prove that, we carry out an experiment of pearson correlation analysis. The pearson correlation coefficient (**PCC**) $r_{xy}$ ($r_{xy} \\in [-1, 1]$) is a metric to estimate the strength of the correlation between variable $x$ and variable $y$. Generally, $r_{xy} > 0.5$ means that two variables are strongly positively correlated. Thus, given an instance, we compute the pearson correlation coefficient between its uncertainty score $u$ and its mean similarity $h$ with the instances from the other modality on two datasets. If $u$ and $h$ are strongly positively correlated, this means that the prototypes represent the overall semantics space well. The results are shown in the Table below.\n\n| Method | PCC-V  | PCC-T  | Dataset  |\n|:------:|:------:|:------:|:--------:|\n| PAU    | 0.886  | 0.786  | MSCOCO   |\n| PAU    | 0.917  | 0.939  | MSR-VTT  |\n\n It can be seen that $u$ and $h$ are all strongly positively correlated on both the image-text retrieval task and video-text retrieval task, which means that the prototypes in PAU indeed represent the overall semantics space.\n\n***Q3 : If other basic losses are used in the training process?***\n\n**A3:** Yes, uncertainty loss and diversity loss can be regarded as two proxy loss functions. We also utilize the cross-entropy loss to constrain the similarity matrix. The final loss formulation should be $\\mathcal L_{total}=\\mathcal L_{CE} + \\mathcal L_{uct} + \\mathcal L_{div}$.\n\n***Q4 : Why does the exponential function outperform than softplus function and ReLU function?***\n\n**A4:** The experiment results in **Table. 7 of the Appendix** show that the performance using exponential function > softplus function > ReLU function. The reason of exponential function outperforms than other two functions is that after similarity $s$ > $\\theta$, ReLU function and softplus function will convert to a linear function. For the linear function, the evidence change of $s$ boosted from 0.2 to 0.3 will be the same as the evidence change of $s$ boosted from 0.9 to 1. But the latter boost is harder than the former, and it should take a higher evidence boost. The reason of softplus function outperforms than ReLU function is that $e=0$ when $s < 0$ in ReLU function, which will discard half information from $s$.\n\n***Q5 : The legend in Figure 1(b) is empty.***\n\n**A5:** Thanks for the detailed review. We are sorry for the lack of legends in Figure 1(b). The top legend should be Text A, and the bottom legend should be Text B. You can find the correctly updated **Figure 7 in the attached pdf file of common response**."}, {"review_id": "bKGLcaPhTs", "reviewer": "Reviewer_hqQV", "summary": "The paper focuses on the unreliable prediction of cross-modal retrieval methods caused by low-quality data. The authors introduce evidential theoretical framework to quantify the uncertainty arising from inherent data ambiguity.  To provide trustworthy uncertainty predictions, they propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework. Extensive experiments demonstrate the effectiveness of their methods.", "questions": "1. What is the effect of using different initialization methods for all prototypes?\n2. What is the time complexity of the algorithm? How does it compare to other algorithms?\n3. The authors argue that the uncertainty loss forces the prototypes into learning the rich semantics of subspace to realize accurate uncertainty quantification. But I'm confused that the uncertainty loss only limits the consistency of U and H, why it can achieve accurate uncertainty quantification?", "limitations": "The authors point out the limitations of their work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- To the best of my knowledge, this paper is the first to combine the evidential framework with cross-modal retrieval methods, so this work is quite novel.\n- The idea of introducing uncertainty quantification to mitigate the negative impact of low-quality data on retrieval is interesting and sound. The experimental evaluations in the paper also provide strong evidence to support the claims made in the paper. These experiments are wide-ranging and cover several different datasets.\n- The paper is well written and easy to follow. However, there is room for improvements as I mention below.", "weaknesses": "1.  While the experiments are quite wide-ranging, the details provided in the paper are quite scarce, such as how many epochs are trained and how to choose hyperparameters. Code is also not provided.\n\n2.  Although the authors provide related work on cross-modal retrieval and uncertainty quantification, it is not clear how these works differ from their proposed method.\n\n3. There are some unclear points in the paper.\n- What is the meaning of the histogram in Figure 1(b)?\n- Does PAU only contain the component of $L_{uct}$, $L_{div}$ and Re-rank? How does it constrain whether $M''$ is predicted accurately? Providing pseudocode would help readers better understand the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer hqQV for the positive, valuable and detailed review, as well as the constructive suggestions for improvement. Our responses to the reviewer’s questions are below:\n\n***Q1 : Some details about the number of training epochs, how to choose parameters, and code are scarce.***\n\n**A1:** Thanks for the constructive suggestions. You can find the training epochs in Section 4.2 (Implementation details). PAU is trained with 5, 3, 20, and 5 epochs on MSR-VTT, MSVD, DiDeMo, and MS-COCO, respectively. The code can be found in the url link of the abstract. As for the parameters chosen, there are two important parameters for PAU, prototype number $K$ and re-ranking control parameter $\\beta$. For parameter $K$, we perform experiments on three datasets with different $K$ numbers. The results are shown below. We can observe that PAU obtains the best Mean R@1 performance when $K=8$ on all benchmarks. Thus, we select $K=8$. For control parameters $\\beta$, it is a learnable parameter. After the model is trained, we freeze other parameters of the model and only train $\\beta$ using the re-ranked similarity matrix constrained by a cross-entropy loss for few steps.\n|||MSR|VTT||||||DiDeMo||||||MSVD|||\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|||**t2v**|**v2t**||**Mean**|||**t2v**|**v2t**||**Mean**|||**t2v**|**v2t**||**Mean**|\n||**R1**|**R5**|**R1**|**R5**|**R1**||**R1**|**R5**|**R1**|**R5**|**R1**||**R1**|**R5**| **R1**|**R5**|**R1**|\n|K=1|47.2|72.2|46.1|72.7|46.7||46.9|75.9|46.1|74.6|46.5||46.4|76.9|64.8|93.2|55.6|\n|K=4|47.6|72.8|46.7|71.6|47.2||48.0|75.8|47.3|74.8|47.65||46.6|77.0|67.9|92.9|57.25|\n|K=8|48.5|72.7|**48.3**|**73.0**|**48.4**||**48.6**|76.0|**48.1**|74.2|**48.35**|| **47.3**|**77.4**|**68.9**|**93.1**|**58.1**|\n|K=12|**48.6**|72.4|46.9|72.6|47.8||48.1|75.8|46.2|75.4|47.15||46.9|77.1|68.3|92.2| 57.6   |\n|K=16|47.8|**73.0**|47.0|72.8|47.4||**48.6**|**76.9**|47.7|**75.5**|48.15||47.0|**77.4**|**68.9**|92.5|57.95|\n\n***Q2 : How do these related works differ from PAU?***\n\n**A2:** Previous cross-modal retrieval methods only focus on predicting the relevance of vision-language pairs. However, these methods don't consider the reliability of predictions, which will be decreased by low-quality input data. Previous uncertainty quantification methods mainly focus on single-modal tasks, such as classification and segmentation tasks. The gap between single-modal tasks and multi-modal tasks makes previous uncertainty quantification methods hardly transfer to multi-modal tasks.\n\n***Q3 : What is the meaning of the histogram in Figure 1(b)?***\n\n**A3:** We are sorry for the lack of legends in Figure 1(b). The top legend should be Text A, and the bottom legend should be Text B. The histogram means the non-detailed text B has high similarities with more semantics categories than detailed text A. You can find the correctly updated **Figure 7 attached in pdf file of common response**.\n\n***Q4 : How do PAU components constrain $M''$ to be predicted accurately?***\n\n**A4:** PAU only contains the components of $L_{uct}$, $L_{div}$, and re-rank. In the re-rank phase, there are two learnable parameters $\\beta_1$ and $\\beta_2$, which will constrain $M''$ predicted accurately. Concretely, after the model is trained, we freeze other parameters of the model and only train $\\beta$ using the re-ranked similarity matrix with the constraint by cross-entropy loss. The cross-entropy loss will make the $M''$ accurate by learning proper parameters $\\beta$. Limited by the rebuttal character number, we will add the pseudo code of $M''$ computing in our final paper.\n\n***Q5 : What is the effect of using different initialization methods for all prototypes?***\n\n**A5:** We test PAU with several initialization methods for prototypes on MSR-VTT. The methods contain Xavier [9], kaiming_normal [7], kaiming_uniform [7], and orthogonal [8] initialization. The results are shown in the Table below. We can observe that Xavier init method can get the best R@1 performance on both t2v and v2t.\n|||**t2v**|||**v2t**||\n|-|-|-|-|-|-|-|\n|**Method**|**R1**|**R5**|**R10**|**R1**|**R5**|**R10**|\n|kaiming_uniform|47.2|72.8|81.9|46.3|73.2|81.6|\n|kaiming_normal|46.8|72.5|82.2|46.6|73.0|82.3|\n|orthogonal|47.4|**73.0**|**83.1**|47.1|**74.0**|**83.5**|\n|Xavier|**48.5**|72.7| 82.5|**48.3**|73.0|83.2|\n\n***Q6 : What is the time complexity of the algorithm? How does it compare to other algorithms?***\n\n**A6:** We compute the FLOPs and parameter number of PAU on MSR-VTT. Meanwhile, we compare our methods with some famous previous methods. The results are shown in the Table below. We can observe that the FLOPs and the parameter number of PAU are both significantly smaller than other methods, which means that PAU is a light-weight approach and easy to deploy.\n|Methods|FLOPs (G)|Params (M) |\n|-|-|-|\n|VATT [10]|792|327|\n|Frozen [11]|771|180.9|\n|MILES [12]|771| 180.9|\n|PAU|**36**|**84.2**|\n\n***Q7 : The uncertainty loss only limits the consistency of U and H, why it can achieve accurate uncertainty quantification?***\n\n**A7:** In this paper, we define that an uncertain instance should have higher mean similarities with the instances from the other modality. For example, given a specific image $i$ with the uncertainty score $u_i$. $h_i$ denotes the mean similarity between $i$ and all texts. According to the definition before, **$u_i$ and $h_i$ should be positively correlated**. In the actual training process, it is hard to compute the mean similarity between an image and all texts in the datasets because of the high computational cost. Thus, we only compute the mean similarity between an image and the texts in a batch. Assuming $batchsize=N$, all images' uncertainty scores make up $U=[u_1, u_2, ..., u_N]$, and the mean similarity between these images and all texts make up $H=[h_1, h_2, ..., h_N]$. By limiting the consistency of $U$ and $H$, we can force the $u$ and $h$ to be positively correlated. Then we can obtain the precise uncertainty estimator after training."}]

Author rebuttal:
# Common Response #
We appreciate all reviewers for their kind reviews and recognizing our work is helpful to multi-modal learning.

According to reviews, **highlights of this paper include**:

- The problem this paper addressed is interesting, novel, and important. (**VBVJ, tGbU, b4gq, hqQV**)
- The proposed method is reasonable, technically valid, and theoretically good. (**qbVB, VBVJ, tGbU, hqQV**)
- The experimental evaluations in the paper provide strong evidence to support our claims. (**qbVB, b4gq, hqQV**)
- The paper is well-written and easy to follow. (**hqQV**)

Reviewers **qbVB** ,**VBVJ**, **b4gq**, **hqQV** mark our work as good soundness. Furthermore, Reviewers **VBVJ**, **b4gq**, **hqQV** give the positive comments. They suggest acceptance.

**Major weakness of this paper** contains:
- Some confusion in uncertainty loss.
- Some points should be further discussed.
- Lack of some related works.
- Lack of some experiments.

Based on the above summary, we answer each question raised by Reviewers.

**Note**: **A pdf file** is attached in the common response to help answer some questions. It contains **all the new figures** we used in the rebuttal phase.

For some common questions, we conduct the experiment as follows:

## More Method Comparision in Different Noise Ratios ##

To make the comparison more fair on the image-text retrieval task. We follow the report format of [3], showing the average R@1 of i2t and t2i on several CLIP ViT-B/32 backbone-based methods on MS-COCO 5K, including VSE$\infty$ [5], PCME [6], NCR [4], and CLIP [1]. CLIP is our baseline. We can observe that PAU shows powerful performance. 

Additionally, we explore the robustness of PAU training under noisy correspondences. We follow the noise split of NCR with 20%, 50% relationships randomly shuffled. It can be seen that PAU outperforms all other methods with a significant gap in 20% and 50% noise ratio, which shows that PAU also possesses strong robust learning ability.

|||AVG R1||
|-|-|-|-|
|**Method**|**0% noise**|**20% noise**|**50% noise**|
|VSE$\infty$ [5]|**55.2**|51.4|18.4|
|PCME [6]|53.0|48.1|43.0|
|NCR [4]|-|48.8|45.5|
|CLIP [1]|54.7|48.3|46.8|
|PAU|**55.2**|**51.8**|**49.6**|

All new experiments and supplemented contents will be updated to the final paper.

## Some References Mentioned in Our Responses. ##

[1] Radford, Alec  et. al. "Learning Transferable Visual Models From Natural Language Supervision." ICML, 2021.  
[2] Luo, Huaishao  et. al. "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval." Neurocomputing, 2022.  
[3] Chun, Sanghyuk. "Improved Probabilistic Image-Text Representations." arXiv preprint arXiv:2305.18171.  
[4] Huang, Zhenyu et. al. "Learning with noisy correspondence for cross-modal matching." NeurIPS, 2021.  
[5] Chen, Jiacheng, et al. "Learning the best pooling strategy for visual semantic embedding." CVPR. 2021.  
[6] Chun, Sanghyuk, et al. "Probabilistic embeddings for cross-modal retrieval." CVPR. 2021.  
[7] He, Kaiming, et al. "Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification." ICCV, 2015.  
[8] Andrew, Saxe, et al. "Exact solutions to the nonlinear dynamics of learning in deep linear neural networks." ICLR, 2014.  
[9] Glorot, Xavier, et al. "Understanding the difficulty of training deep feedforward neural networks." AISTATS, 2010.  
[10] Hassan, Akbari, et al. "VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text." NeurIPS, 2021.  
[11] Max, Bain, et al. "Frozen in time: A joint video and image encoder for
end-to-end retrieval." ICCV, 2021.  
[12] Ge, Yuying, et al. "MILES: Visual BERT Pre-training with Injected Language Semantics." ECCV, 2022.  

Meta-review: The paper provides a novel and meaningful approach to the data uncertainty in image-text retrieval tasks through an extension of Subjective Logic, inheriting its strong theoretical properties. There were initial issues with the presentation (e.g. equations 5,6) as well as missing discussion and comparison against prior approaches and potentially unfair comparisons (MS-COCO experiments). The authors have effectively addressed the concerns during the rebuttal phase. As such, we recommend accepting the paper. We *strongly* urge the authors to include *all* the additional results and references in the final draft.
Decision: Accept (poster)