SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Li
18th.leolee@gmail.com
Jingkuan Song*
jingkuan songwriter@gmail.com
Lianli Gao
lianli.gao@uestc.edu.cn
Xiaosu Zhu
xiaosu.zhu@outlook.com
Heng Tao Shen
shenhengtao@hotmail.com
Center for Future Media,
Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. In this paper, we propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity. Concretely, we first construct a set of various learnable prototypes for each modality to represent the entire semantics subspace. Then Dempster-Shafer Theory and Subjective Logic Theory are utilized to build an evidential theoretical framework by associating evidence with Dirichlet Distribution parameters. The PAU model induces accurate uncertainty and reliable predictions for cross-modal retrieval. Extensive experiments are performed on four major benchmark datasets of MSR-VTT, MSVD, DiDeMo, and MS-COCO, demonstrating the effectiveness of our method. The code is accessible at https://github.com/leolee99/PAU.
Cross-modal Retrieval, devoted to searching a gallery of related samples in vision/language modality given a query in another, has attracted increasing interest as the wellspring development of multimedia platforms. However, the heterogeneity between modalities casts daunting challenges for this task. To tackle these challenges, most existing methods [9, 60, 19, 15, 4, 53, 49] map the query and retrieval items from distinct modalities into a joint latent space to make similarity calculation feasible.
Some early works [9, 60] corroborate the powerful capabilities of deep neural networks to extract high-level features. Chen et al. [9] construct a hierarchical graph structure for videos and texts to build the relationships between the local and the global features. PVSE [60] utilizes the multi-head self-attention module to learn multiple and diverse representations of videos and texts for the polysemous problem. Recently, a large body of transformer-based pre-training models [19, 15, 4, 53, 49] have shown remarkable superiority with excellent generalization and performance.
However, prior approaches routinely assume the data qualities are absolutely stable and view them equally. They estimate the relevance of vision-language pairs relying only on the similarity scores
(a) Video modality
(b) Text modality
Figure 1: Illustration of confused matching in fast-paced videos and non-detailed texts. Assuming the possible semantics of each modal subspace are finite with  $K$  categories. (a) A single-scene Video  $A$  can only match one semantics of "talking". By contrast, a multi-scene Video  $B$  can match to 3 semantics of "talking", "shadow", and "cave". (b) Text  $A$  can only match the left video, while Text  $B$  with some details removed (in red) matches both videos.
generated by complex neural networks, but ignoring the confidence of these predictions, which should be taken seriously. Low-quality data like corrupt images [31], fast-paced videos [26], non-detailed sentences [34], etc., inevitably lead to unreliable results. The model therefore not only needs to predict the relevance of vision-language pairs, but also capably answer the question "How confident is the prediction?" Namely, it is necessary to quantify the uncertainty of each data to guide more reliable vision-language pair similarity.
In the literature, uncertainty can be classified into Aleatoric Uncertainty and Epistemic Uncertainty based on causation [32]. The former is induced by the inherent data defects, while the latter is inherent to the model. This work mainly focuses on the aleatoric uncertainty (i.e., data uncertainty), defined as "the uncertainty caused by ambiguous multi-modal data, such as fast-paced videos and non-detailed texts". Next, we will answer the question "Why do these ambiguous data lead to uncertainty?". We first assume the possible semantics of each modal subspace are finite with  $K$  categories. Afterward, the fast-paced videos (multi-scene videos) and the non-detailed texts are generally highly related to multiple semantics, misleading models into confused matching, shown in the Figure 1. These alternative matches will bring uncertainty, which can be explained through Information Entropy [59]. Concretely, Information Entropy is the average uncertainty level in the random variable's possible outcomes, which can be denoted as Eq. 1 with the possible semantics as the random variable  $X$ .
$$
\mathcal {H} (X) = \mathbb {E} [ - \log p \left(x _ {i}\right) ] = - \sum_ {i = 1} ^ {K} p \left(x _ {i}\right) \log p \left(x _ {i}\right), \tag {1}
$$
where  $\sum_{i=1}^{K} p(x_i) = 1$ ,  $p(x_i)$  is the  $i^{th}$  semantics probability derived from semantic cosine similarity through a softmax layer. The ambiguous data highly associated with multiple semantics have a low variance of the probability distribution, resulting in high information entropy and high uncertainty due to "The Maximum Entropy Principle [33]" (proof in Appendix E). Information entropy is a seemingly reasonable criterion to estimate the data uncertainty, but unfortunately, it is powerless to distinguish the instances with the similarity distribution of  $[0.8, 0.8, 0.8, 0.8]$  and the distribution of  $[0.2, 0.2, 0.2, 0.2]$ . After a softmax function, both distributions turn into  $[0.25, 0.25, 0.25, 0.25]$  and obtain the same information entropy. In practice, we hope the first one gets a higher uncertainty score than the other because the front of the retrieval sequence with highly similar instances is more notable for the retrieval task. Thus, the Dempster-Shafer Theory of Evidence (DST) [58], a generalization of the Bayesian theory to subjective probabilities, which has developed into a general framework for uncertainty modeling, is leveraged to quantify multi-modal aleatoric uncertainty.
To precisely quantify the aleatoric uncertainty, in this work, we propose an effective Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from inherent data ambiguity. To be specific, we first construct a series of diverse prototypes for each modality to capture the overall  $K$  semantics of the subspace. Each prototype represents an individual semantics. Afterward, the variations of cosine similarities
between the instance and the prototypes from another modality are deployed as the belief masses used in DST [59] to model uncertainty. Finally, a re-ranking practice is utilized to further strengthen prediction reliability by regarding each data uncertainty as a similarity weight.
The main contributions of our paper are three folds. 1) We give a clear definition of the inherent aleatoric uncertainty in multi-modal data. 2) A powerful aleatoric uncertainty quantification framework PAU is built to accurately estimate the data uncertainty and effectively mitigate the negative impact on retrieval. 3) We also verify the superiority of our proposed PAU on multiple challenging benchmarks. Massive insightful ablation studies further reveal its effectiveness and generalization.
2.1 Cross-modal Retrieval
Cross-modal retrieval [9, 25, 67, 39, 17, 60, 54, 49, 22, 18, 66, 26, 64, 37], as one of the most popular tasks in multi-modal learning, has attracted increasing attention with the aim of searching similar semantic samples from different modalities. Early works [25, 39] build a joint embedding space to make query-positive pairs closer by canonical correlation analysis (CCA) [28]. Faghri et al. [17] first use a triplet loss to improve the model performance by focusing only on the hardest negative. PVSE [60] proposes a multi-candidate representation function to capture the diversity and build one-to-many relationships in different modalities. Recently, CLIP [54] has shown vital power and generalization in various multi-modal tasks. A rich body of CLIP-based approaches [49, 22, 18, 66, 26] have got significant achievements in cross-modal retrieval. CLIP4Clip [49] transfers the rich knowledge from image-text pre-trained model CLIP to video-text retrieval. CenterCLIP [66] proposes a multi-segment token clustering algorithm to keep the most representative tokens in consecutive frames. In this paper, we follow the setting of the above work, which inherits the knowledge from CLIP.
2.2 Uncertainty Quantification
Generally, there are two paradigms among existing uncertainty estimation approaches. The first one is the Bayesian-based paradigm [5, 51, 40], which estimates uncertainty by approximating the moments of the posterior predictive distribution. Subsequently, some algorithms focus on the variety of Bayesian such as Laplace approximation [50], Markov Chain Monte Carlo [52], and variational techniques [55, 6]. Recent work [21] leverages dropout [61] in the test phase, reducing the computational cost of uncertainty estimation. Nonetheless, limited by the increment of model parameters and poor convergence, these approaches always have expensive training costs and slow inference time. The second paradigm focuses on the non-Bayesian approach [41, 57, 27]. Lakshminarayanan et al. [41] propose an alternative to Bayesian Neural Networks (BNN) which is simple to train and produce reliable uncertainty estimates. Wang et al. [57] replace the parameter set of a categorical distribution with the parameters of a Dirichlet density and represent the uncertainties with a single abnormal class. Recently, uncertainty-based algorithms have been used in multi-view classification for trusted decision-making [27]. Although uncertainty-based methods have achieved impressive progress, they are limited to working with particular classification and segmentation tasks.
In this section, we present our Prototype-based Aleatoric Uncertainty Quantification (PAU) framework (Figure 2). We first define the cross-modal retrieval task and introduce a commonly used pipeline in Section 3.1. Next, we detail the uncertainty theory, as well as the approach to quantify the uncertainty in Section 3.2.
Let  $\mathcal{D} = (\mathcal{V},\mathcal{T})$  denote a vision and language dataset, where  $\mathcal{V}$  is a set of images or videos, and  $\mathcal{T}$  is a set of texts. The goal of cross-modal retrieval is to rank the relevance between a visual query  $v\in \mathcal{V}$  (respectively a textual query  $t\in \mathcal{T}$ ) and textual set  $\mathcal{T}$  (respectively visual set  $\mathcal{V}$ ). Recent works [54, 49, 22, 18, 66, 26] have shown CLIP's strong performance and generalization in various downstream tasks, inspiring us to employ CLIP as our backbone.
Figure 2: The Framework of PAU. The visual encoder  $\phi_v$  and textual encoder  $\phi_t$  separately map the visual and textual instances into a joint embedding space to calculate the similarity matrix  $\mathbf{M}$ . A dot product function is used to build a set of similarity vector  $\mathbf{P} \in \mathbb{R}^{N \times K}$  between  $N$  instances and  $K$  prototypes, afterward modeling the uncertainty. An uncertainty loss forces the prototypes into learning the rich semantics of subspace to realize accurate uncertainty quantification. Besides, A diversity loss is introduced to keep prototypes diverse.  $\otimes$  means cosine similarity.
To be specific, given a visual instance  $v$  and a text instance  $t$ , the CLIP encodes them into a latent embedding space  $\mathbb{R}^D$  through a visual encoder  $\phi_v(\cdot)$  and a textual encoder  $\phi_t(\cdot)$ , respectively. Then the relevance between them can be computed using a cosine similarity function as:
$$
\operatorname {s i m s} (v, t) = \cos \left(\phi_ {v} (v), \phi_ {t} (t)\right) = \frac {\phi_ {v} (v)}{\| \phi_ {v} (v) \|} \cdot \frac {\phi_ {t} (t)}{\| \phi_ {t} (t) \|} \tag {2}
$$
3.2 Uncertainty Quantification
Prototype construction. Although the cosine similarity function has shown effectiveness in most existing works [25, 60, 54, 49], some unreliable results will inevitably be produced due to the low inherent data quality. These ambiguous data are generally highly related to multiple semantics, leading to confusing matching (see Figure 1).
To assess these data uncertainty and provide trustworthy predictions, we first assume the semantic space is finite with  $K$  semantic categories. Then,  $K$  learnable prototypes  $\mathbf{Z} \in \mathbb{R}^{K \times D}$  are constructed to represent the overall latent semantic subspace for each modality, making data semantic ambiguity evaluable. Each prototype represents an individual semantics. Finally, DST [12] is employed to build the uncertainty quantification framework by assigning the variations of similarities between an instance and the prototypes from another modality as the belief masses.
The theory of uncertainty. The Dempster-Shafer Theory of Evidence (DST) [12], a generalization of the Bayesian theory to subjective probabilities [12], builds a general uncertainty modeling framework by assigning belief masses to the set of mutually exclusive possible states with one state expressing "I am not sure". Following EDL [57], we adopt Subjective Logic (SL) [35] to associate the belief distribution in DST with the parameters of the Dirichlet distribution. More precisely, SL provides an evidential theoretical framework to quantify the belief masses  $b_{k}$  of different categories and overall uncertainty mass  $\psi$  for the  $K$  classification task. These  $K + 1$  mass values are all non-negative and sum up to one:
$$
\psi + \sum_ {k = 1} ^ {K} b _ {k} = 1, \tag {3}
$$
where  $b_{k} \geq 0$  and  $\psi \geq 0$  denote the belief mass of the  $k^{th}$  semantic category  $(k = 1, \dots, K)$  and overall uncertainty mass, respectively. In our setting,  $\psi$  here means the uncertainty mass of the
event "the data is ambiguous", which is negatively related to the degree of data ambiguity. Thus, the  $u = 1 - \psi = \sum_{k=1}^{K} b_k$  is designed to express the aleatoric uncertainty defined in Section 1. In brief, the data with lower  $\psi$  is more ambiguous, which should be a high aleatoric uncertainty score as higher  $u$ . A belief mass  $b_k$  for the  $k^{th}$  semantic category is computed using the evidence for the category. We term the evidence as the amount of data supports for being classified into a certain category. Let  $\mathbf{e} = [e_1, \dots, e_k]$  be an evidence vector derived from similarity vector  $\mathbf{p} = [p_1, \dots, p_k]$  between an instance and  $K$  prototypes (evidence generation shown in Appendix B), then the belief  $b_k$  and the uncertainty  $u$  are signified as:
$$
b _ {k} = \frac {e _ {k}}{S} = \frac {\alpha_ {k} - 1}{S} \quad \text {a n d} \quad u = 1 - \psi = \sum_ {k = 1} ^ {K} b _ {k} = 1 - \frac {K}{S}, \tag {4}
$$
where  $\alpha_{k} = e_{k} + 1$  denote the Dirichlet parameters associated with a belief mass assignment in SL, and  $S = \sum_{i=1}^{K} \alpha_{k}$  is referred to as the Dirichlet strength.  $e_{k}$  is the evidence of a visual (or textual) instance matching the  $k^{th}$  semantics category. Intuitively, massive evidence indicates considerably ambiguous data, resulting in high uncertainty, which brings out fallible predictions.
Uncertainty learning. After uncertainty modeling, how to force the model to learn uncertainty from data is worth considering (see Figure 2). The key is enriching prototypes' meaningful semantics to represent overall subspace rigorously. We first give a normalized initialization proposed by Xavier [24] to all prototypes. Subsequently, the cosine similarity matrix  $\mathbf{M} \in \mathbb{R}^{N \times N}$  between  $N$  visual and textual instances is calculated by Eq. 2 within a batch. Furthermore,  $N$  instances and  $K$  prototypes from different modalities build a set of similarity vectors  $\mathbf{P} \in \mathbb{R}^{N \times K}$  to derive the evidence vector set and compute the uncertainty using Eq. 4. The data uncertainty sets of  $N$  visual and textual instances can be symbolized as  $\mathbf{U}_v = [u_1^v, \dots, u_N^v]$  and  $\mathbf{U}_t = [u_1^t, \dots, u_N^t]$  in parallel.
Next, an objective function is required to force prototypes to learn rich semantic knowledge. As the prototypes are expected to represent the overall semantics of each modality, the instance with high overall similarity to these prototypes should also have high overall similarity with other instances from another modality. According to Eq. 4, the aleatoric uncertainty  $u$ , denoted as the sum of belief masses, should be positively related to the mean similarities of each modality within a batch. Thereby, we compute the mean of similarity matrix  $\mathbf{M}$  along the row and the column, symbolized as  $\mathbf{H}_v = [h_1^v,\dots ,h_N^v ]$  and  $\mathbf{H}_t = [h_1^t,\dots ,h_N^t ]$ , respectively. After that, prototypes can be optimized by employing Mean-Squared (MSE) Loss to keep the  $\mathbf{U}_v(\mathbf{U}_t)$  and  $\mathbf{H}_v(\mathbf{H}_t)$  consistent, where  $\lambda$  is a scaling coefficient to keep the same value range of  $u$  and  $h$  (Eq. 5).
$$
\mathcal {L} _ {u c t} ^ {v} = \frac {1}{N} \sum_ {\substack {i = 1 \\ N}} ^ {N} \left(u _ {i} ^ {v} - \lambda \cdot h _ {i} ^ {v}\right) ^ {2} \tag{5}
$$
$$
\mathcal {L} _ {u c t} ^ {t} = \frac {1}{N} \sum_ {i = 1} ^ {N} (u _ {i} ^ {t} - \lambda \cdot h _ {i} ^ {t}) ^ {2}
$$
Diversity learning. To employ the DST uncertainty framework, the mutual exclusion among possible states needs to be satisfied. Precisely, prototypes should avoid attending to the same or similar semantics. Therefore, a diversity loss is proposed to help the prototypes focus on diversified independent semantics. The formula is as:
$$
\mathcal {L} _ {d i v} ^ {v} = \frac {1}{K ^ {2}} \sum_ {\substack {i = 1 \\ K}} ^ {K} \sum_ {j = 1} ^ {K} (\cos \left(\mathbf {z} _ {i} ^ {v}, \mathbf {z} _ {j} ^ {v}\right)) ^ {2} \tag{6}
$$
$$
\mathcal {L} _ {d i v} ^ {t} = \frac {1}{K ^ {2}} \sum_ {i = 1} ^ {K} \sum_ {j = 1} ^ {K} (\cos (\mathbf {z} _ {i} ^ {t}, \mathbf {z} _ {j} ^ {t})) ^ {2}
$$
where  $\mathbf{z}_i^v$  and  $\mathbf{z}_i^t$  denote  $i^{th}$  prototypes of visual and textual modality. Intuitively, the semantic similarities between different prototypes are expected to be 0.
Re-ranking. After training, the uncertainty of each instance could be precisely assessed using Eq. 4. To further alleviate the impact of the uncertainty on prediction, we re-rank the retrieval sequences by weighting the predicted matrix based on the uncertainty vector  $\mathbf{U}_v$  and  $\mathbf{U}_t$  by Eq. 7.
$$
\mathbf {M} ^ {\prime \prime} = \left(\exp \left(- \beta_ {1} \mathbf {U} _ {v} ^ {T}\right) \cdot \exp \left(- \beta_ {2} \mathbf {U} _ {t}\right)\right) \circ \mathbf {M} ^ {\prime} \tag {7}
$$
where  $\beta_{1}$  and  $\beta_{2}$  are learnable parameters used to control the impact degree of re-ranking on prediction.  $\mathbf{M}'$  and  $\mathbf{M}''$  separately signify the similarity matrices between all visual and textual instances in test processing before and after re-ranking.
In this section, we first introduce the datasets and metrics used for our experiments, as well as the implementation details. Later, we compare PAU with the previous state of the arts on several tasks and public benchmarks. In the end, rich ablation studies are carried out to verify our method's effectiveness and generalization.
We report our experimental results on three public video-text retrieval datasets containing MSR-VTT [62], MSVD [7], DiDeMo [30], and a public image-text retrieval dataset of MS-COCO [45]. Following [49], Recall@1 (R@1), Recall@5 (R@5), Recall@10 (R@10), Median Rank (MdR), and Mean Rank (MnR) are used to evaluate the performance.
MSR-VTT [62] is a commonly used benchmark in video-text retrieval, containing 10,000 Youtube videos with 20 captions for each. Following [20], we employ 9,000 videos for training and 1,000 selected video-text pairs for testing.
MSVD [7] contains 1,970 videos ranging in length from 1 to 62 seconds, with approximately 40 sentences per video. There are 1,200, 100, and 670 videos used for training, validation, and testing in the standard split.
DiDeMo [30] consists of 10,611 Flickr videos with approximately 40,000 annotations. We follow [46, 42, 3] and concatenate all captions of the same video into a paragraph, which will be regarded as a single item in a paragraph-to-video retrieval task.
MS-COCO [45] is a widely used dataset for image-text retrieval, containing 123,287 images with five captions per image. We follow the split in [36] with 113,287 images for training, 5,000 for validation, and 5,000 for testing. The results are reported on 1K and 5K test sets.
Table 1: Retrieval Performance on MSR-VTT
Method	t2v	R@1	v2t
R@1	R@5	R@10	MdR	MnR	R@5	R@10	MdR	MnR
CLIP4Clip-meanP [49]	43.1	70.4	80.8	2.0	16.2	43.1	70.5	81.2	2.0	12.4
CLIP4Clip-seqTransf [49]	44.5	71.4	81.6	2.0	15.3	42.7	70.9	80.6	2.0	11.6
CenterCLIP [66]	44.2	71.6	82.1	2.0	15.1	42.8	71.7	82.2	2.0	10.9
MILES [23]	44.3	71.1	80.2	2.0	14.7	-	-	-	-	-
CLIP2Video [18]	45.6	72.6	81.7	2.0	14.6	43.5	72.3	82.1	2.0	10.2
CLIP2TV [22]	46.1	72.5	82.9	2.0	15.2	43.9	73.0	82.8	2.0	11.1
XPool [26]	46.9	72.8	82.2	2.0	14.3	44.4	73.3	84.0	2.0	9.0
Baseline	45.6	72.9	81.5	2.0	14.5	45.2	71.6	81.5	2.0	10.9
ours	48.5	72.7	82.5	2.0	14.0	48.3	73.0	83.2	2.0	9.7
Table 2: Retrieval Performance on MSVD
Method	t2v	R@1	R@5	v2t
R@1	R@5	R@10	MdR	MnR	R@10	R@10	MdR	MnR
CLIP4Clip-meanP [49]	46.2	76.1	84.6	2.0	10.0	56.6	79.7	84.3	1.0	7.6
CLIP4Clip-seqTransf [49]	45.2	75.5	84.3	2.0	10.0	62.0	87.3	92.6	1.0	4.3
CenterCLIP [66]	47.3	76.9	86.0	2.0	9.7	63.5	86.4	92.6	1.0	3.8
CLIP2Video [18]	47.0	76.8	85.9	2.0	9.6	58.7	85.6	91.6	1.0	4.3
CLIP2TV [22]	47.0	76.5	85.1	2.0	10.1	-	-	-	-	-
XPool [22]	47.2	77.4	86.0	2.0	9.3	66.4	90.0	94.2	1.0	3.3
Baseline	46.3	76.8	84.9	2.0	10.0	60.1	82.8	87.3	1.0	7.5
ours	47.3	77.4	85.5	2.0	9.6	68.9	93.1	97.1	1.0	2.4
Table 3: Retrieval Performance on DiDeMo
Method	t2v	R@1	R@5	v2t
R@1	R@5	R@10	MdR	MnR	R@10	R@10	MdR	MnR
CLIP4Clip-meanP [49]	43.4	70.2	80.6	2.0	17.5	43.4	69.9	80.2	2.0	17.5
CLIP4Clip-seqTransf [49]	43.4	69.9	80.2	2.0	17.5	42.7	70.9	80.6	2.0	11.6
CLIP2TV [22]	45.5	69.7	80.6	2.0	17.1	-	-	-	-	-
Baseline	47.0	76.0	86.1	2.0	10.7	46.9	74.4	84.2	2.0	8.4
ours	48.6	76.0	84.5	2.0	12.9	48.1	74.2	85.7	2.0	9.8
Table 4: Comparison on MS-COCO
Method	Backbone	MS-COCO 1K	MS-COCO 5K
R@1	i2t R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5	R@10
PVSE [60]	ResNet-101	69.2	91.6	96.6	55.2	86.5	93.7	45.2	74.3	84.5	32.4	63.0	75.0
SGRAF [13]	79.6	96.2	98.5	63.2	90.7	96.1	57.8	-	91.6	41.9	-	81.3
NAAF [65]	80.5	96.5	98.8	64.1	90.7	96.5	58.9	85.2	92.0	42.5	70.9	81.4
DAA [43]	80.2	96.4	98.8	65.0	90.7	95.8	60.0	86.4	92.4	43.5	72.3	82.5
VSE∞ [8]	ViT-B/32	82.0	97.2	98.9	69.0	92.6	96.8	62.3	87.1	93.3	48.2	76.7	85.5
PCME [11]	80.1	96.6	98.7	67.6	92.1	96.9	59.9	85.8	92.3	46.1	75.0	84.6
PCME++ [10]	81.6	97.2	99.0	69.2	92.8	97.1	62.1	86.8	93.3	48.1	76.7	85.5
Baseline	80.1	95.7	98.2	67.1	91.4	96.6	62.9	84.9	91.6	46.5	73.8	82.9
ours	80.4	96.2	98.5	67.7	91.8	96.6	63.6	85.2	92.2	46.8	74.4	83.7
We use CLIP's [54] visual-textual encoder (ViT-B/32) as the backbone and initialize all parameters from pre-trained CLIP weights. The max token and frame length are both set consistently with [49]. Specifically, the dimension of common embedding space is set to 512. In video-text retrieval, the Adam optimizer [38] is employed with the initial learning rate of 1e-7 for the visual encoder and the textual encoder, as well as the initial learning rate of 1e-4 for other layers. As for image-text retrieval, the AdamW optimizer [48] updates the parameters of all modules with the initial learning rate of 1e-5. The learning rate decays complying with the cosine schedule strategy [47] in 5, 3, 20, and 5 epochs trained on MSR-VTT, MSVD, DiDeMo, and MS-COCO. Besides, the number of prototypes  $K$  is set as 8 according to Table 8. All experiments are conducted on 1 to 4 RTX3090.
4.3 Comparison with State of the Arts