SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Video-text retrieval. We compare our model with other state-of-the-art methods [49, 66, 18, 22, 26, 23, 54] and our baseline (model structure see Appendix. A) on multiple datasets. For fair comparisons, all methods are CLIP-based, symbolizing the most advanced techniques of video-text retrieval. The performance of baseline trained without PAU is also given. Table 1, 2, 3 show the results of video-text retrieval on MSR-VTT, MSVD, and DiDeMo, respectively.
We can observe that PAU brings sizeable improvements on all benchmarks. For text-to-video retrieval (t2v), PAU boosts the R@1 performance of  $2.9\%$ ,  $1.0\%$ , and  $1.6\%$  on MSR-VTT, MSVD, and DiDeMo. Moreover,  $3.1\%$ ,  $8.8\%$ , and  $1.2\%$  improvements of R@1 are obtained on video-to-text retrieval (v2t). Notably, we find that the performance improvement of v2t on MSVD is more significant than other benchmarks. The reason is that MSVD is a small dataset with few videos (only 1,200 for training) but massive annotations (about 40 annotations per video). Meanwhile, CLIP simply utilizes cross-entropy loss, which only regards the diagonal pairs as the positives. Nonetheless, for a batch with a typical size of 256 [49], the probability of any two texts coming from separate videos is approximately  $3.49 \times 10^{-13}$  (see Appendix D), bringing innumerable noises and performance damage for the training on video-to-text retrieval. Thus, the significant improvement indicates the ability of PAU to capture and rectify misleading data to support robust learning.
Image-text retrieval. We also test our method on image-text retrieval task. To be specific, CLIP [54] is utilized as the baseline. We compare our model with several previous state-of-the-art approaches, including PVSE [60], SGRAF [13], NAAF [65], DAA [43], VSE $\infty$  [8], PCME [11], and PCME++ [10]. Table 4 shows the results of image-text retrieval on MS-COCO 1K and MS-COCO 5K, respectively.
We can observe that the overall performance improvement of image-text retrieval is smaller than that of video-text retrieval. The phenomenon reflects that video information is more complex than image information, inducing immense uncertainty in video-text retrieval. Furthermore, we also validate the robustness of PAU training in noisy datasets. The detailed results can be found in Appendix. C.4.
4.4 Comparison with Probabilistic Model
PCME [11] is a probabilistic-based approach to represent instance diversity, which can also represent uncertainty. The more diverse the instance, the more uncertain the instance. Thus, an instance's uncertainty and diversity should be positively correlated. To verify the superiority of our SL-based method over than probabilistic-based method, we conduct two experiments, pearson correlation analysis and uncertain data analysis.
Pearson correlation analysis. The Pearson correlation coefficient (PCC)  $r_{xy}$  ( $r_{xy} \in [-1, 1]$ ) is a metric to estimate the strength of the correlation between variable  $x$  and variable  $y$ . Generally,  $r_{xy} > 0.5$  means that two variables are strongly positively correlated. As mentioned before, the uncertainty and the diversity of an instance should be positively correlated for PCME. If an instance is diverse, it will have high mean
Table 5: PCCs between  $u$  and  $h$
Method	PCCV	PCCT	Dataset
PCME	-0.071	-0.219	COCO
PAU	0.886	0.786	COCO
PAU	0.917	0.939	MSR-VTT
similarity  $h$  with the instances from the other modality. This means that the uncertainty score  $u$  and mean similarity  $h$  should also be positively correlated for an instance. Thus, we compute the Pearson correlation coefficient between  $u$  and  $h$  to explore whether PAU correctly estimates uncertainty. The results are shown in Table 5.
In this Table, we compute the PCCs of PCME and PAU on MSCOCO, as well as PCCs of PAU on MSR-VTT.  $\mathrm{PCC}_V$  and  $\mathrm{PCC}_T$  separately indicate the instance's PCC in vision or textual modality. The low PCCs of PCME mean current probabilistic models have limited abilities to reflect uncertainty and diversity. We argue the reason arises from the strong prior distribution assumption, i.e., gaussian or other simple distributions are powerless to fit complex and diverse relationships in high-dimension space. By contrast, PAU obtains high PCCs in all settings, proving our approach indeed leads to proper uncertainty estimations.
Uncertain data analysis. To further prove the superiority of SL-based aleatoric uncertainty estimation method than probabilistic-based method, we compare the changes of R@1 after removing top-r instances with highest uncertainty scores predicted by PCME and PAU, respectively. To fairly compare, we employ removal on both predictions arises from PAU and PCME. The results with removed ratio from  $0\%$  to  $20\%$  on MS-COCO 5K is shown in Figure 3. We can observe that PAU outperforms PCME on all directions and predictions. This means that the uncertain data found by PAU is more precise than PCME. All experiments indicate the SL-based approach (PAU) works better than probabilistic-based method (PCME) on representing aleatoric uncertainty.
(a) i2t on CLIP
(b) t2i on CLIP
Figure 3: The performance changes comparison after removing top-r instances with the highest uncertainty scores quantified by PCME and PAU on MS-COCO. To fairly compare, we employ the removal on both predictions arising from CLIP and PCME. (a) and (b) show the performance changes on CLIP predictions. (c) and (d) show the performance changes on PCME predictions. In i2t, text instances are removed. In t2i, image instances are removed.
(c) i2t on PCME
(d) t2i on PCME
Table 6: Ablation study to investigate the effect of different components on MSR-VTT
No	w/ Luct	w/ Ldiv	Re-rank	t2v	R@1	v2t
R@1	R@5	R@10	R@1	R@5	R@10
1	✓	45.6	72.9	81.5	45.2	71.6	81.5
2	45.9	72.8	81.5	46.0	72.4	81.7
3	✓	✓	47.0	72.4	82.8	47.4	71.8	82.2
4	✓	47.8	72.7	82.5	47.9	72.2	83.0
5	✓	✓	48.4	72.4	82.2	47.7	72.6	82.6
6	✓	✓	✓	48.5	72.7	82.5	48.3	73.0	83.2
The effect of different components. To understand the effect of each component, we exhaustively ablate three main components of PAU, including Uncertainty Loss, Diversity Loss, and Re-ranking. As shown in Table 6, all three components effectively improve the model performance. The comparison between No. 1 and No. 3 shows that uncertainty loss gives significant R@1 improvements of  $1.4\%$  on t2v and  $2.2\%$  on v2t. The experiment No. 5 outperforms the experiment No. 3 by a clear margin, demonstrating the necessity of diversity loss. Furthermore, the experiment No. 2 re-ranks the baseline prediction using uncertainty vectors from PAU. It's worth noting that R@1 performance still boosts on both t2v and v2t, reflecting the strong generalization of PAU.
The impact of uncertain data. To demonstrate the uncertain data found by PAU indeed affect the retrieval, a trial of the retrieval subsets with varying reliability levels is carried out on MSR-VTT. Specifically, the samples are first ranked according to the uncertainty calculated by PAU in descending order. Then,  $r$  samples with the highest uncertainty will be removed from the test set to produce an uncertainty-based removal subset. Meanwhile,  $r$  random samples will be removed from the test set to produce a random-based removal subset. The baseline model is tested on both subsets above and reports the results in Figure 4.
It's worth noting that the model tested on the subsets removing uncertain data always outperforms the ones removing randomly. Furthermore, the performance gap reaches a peak of 14.2 when  $r = 600$
(a) Text-to-video
(b) Video-to-text
Figure 4: The performance comparison against data removal number on MSR-VTT.
(b) Video-to-text
Figure 5: The performance comparison against uncertain data number on MSR-VTT.
Text Uncertainty
A person is explaining something.
Uncertainty: 0.984
A man is driving a black car.
Uncertainty: 0.421
Multi colored horses in a barn and outside in the snow.
Uncertainty: 0.039
Video Uncertainty
Uncertainty: 0.881
Uncertainty: 0.446
Uncertainty: 0.026
Figure 6: The visualization of data uncertainty on MSR-VTT.
for text-to-video retrieval and a peak of 12.3 when  $r = 700$  for video-to-text retrieval, respectively. The above phenomena prove PAU can indeed find uncertain data, which easily misleading the model into incorrect predictions, resulting in extremely unreliable retrieval results.
Is the prediction more trustworthy? To validate the effectiveness of PAU in providing trustworthy predictions, the performances of PAU and baseline are compared in the MSR-VTT's subsets under a series of different uncertain degrees. Precisely,  $r$  samples with the highest uncertainty are selected to form the uncertain test subsets. As shown in Figure 5, PAU consistently outperforms the baseline with the largest gap of 8.6 (5.8) while  $r = 70$  ( $r = 120$ ) for t2v (v2t). The smaller the  $r$  is, the higher the total uncertainty of the subset is, so the results confirm the ability of PAU to decrease the performance loss caused by uncertain data. Furthermore, PAU can provide trustworthy predictions even in high uncertainty sets.
Visualization of data uncertainty. Three captions and three videos with their uncertainty scores are shown in Figure 6. Obviously, the caption with high uncertainty is more semantically ambiguous, such as "A person is explaining something". By contrast, the caption with low uncertainty is more detailed, such as "Multi colored horses in a barn and outside in the snow". Moreover, the first video (complete video shown in Appendix. F) is more difficult to summarize due to the varied scenes, inducing high uncertainty. By contrast, the third video with low uncertainty is relatively simple.
In this work, we clearly define the inherent aleatoric uncertainty in multi-modal data. A Prototype-based Aleatoric Uncertainty Quantification (PAU) framework is designed to excavate the uncertain data and further provide trustworthy predictions. Uncertainty loss and diversity loss are proposed to support the uncertainty training by encouraging the learnable prototypes to represent the semantic subspace of varied modalities. Plentiful empirical experiments on a range of tasks and public benchmarks demonstrate the effectiveness and generalization of PAU.
Broader impacts statement. Cross-modal matching has become a common paradigm in large-scale multi-modal pretraining, such as CLIP [54]. However, the enormous data demand brings a huge challenge to pre-trained models. If we can accurately select high-quality data, which typically make greater contributions to the performance of pre-trained models, the pretraining process would become much more efficient. This work could be one of the first work to be aware of the aleatoric uncertainty in cross-modal retrieval. By quantifying uncertainty, higher quality data and more reliable results are provided, which could perhaps be extended to other multi-modal tasks in the future [2, 44, 63].
This study is supported by grants from National Key R&D Program of China (2022YFC2009903/2022YFC2009900), the National Natural Science Foundation of China (Grant No. 62122018, No. 62020106008, No. 61772116, No. 61872064), Fok Ying-Tong Education Foundation(171106), and SongShan Laboratory YYJC012022019.
[1] H. Akbari, L. Yuan, R. Qian, W. Chuang, S. Chang, Y. Cui, and B. Gong. VATT: transformers for multimodal self-supervised learning from raw video, audio and text. In NeurIPS, pages 24206-24221, 2021. 16, 17
[2] P. Anderson, X. He, C. Buehler, D. Teney, M. Johnson, S. Gould, and L. Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, pages 6077-6086, 2018. 10
[3] M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1708-1718, 2021. 6
[4] M. Bain, A. Nagrani, G. Varol, and A. Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, pages 1708-1718, 2021. 1, 17
[5] A. Blum, N. Haghtalab, and A. D. Proscaccia. Variational dropout and the local reparameterization trick. In NeurIPS, pages 2575–2583, 2015. 3
[6] C. Blundell, J. Cornebise, K. Kavukcuoglu, and D. Wierstra. Weight uncertainty in neural network. In ICML, volume 37, pages 1613-1622, 2015. 3
[7] D. L. Chen and W. B. Dolan. Collecting highly parallel data for paraphrase evaluation. In ACL, pages 190–200, 2011. 6
[8] J. Chen, H. Hu, H. Wu, Y. Jiang, and C. Wang. Learning the best pooling strategy for visual semantic embedding. In CVPR, pages 15789-15798, 2021. 7, 17
[9] S. Chen, Y. Zhao, Q. Jin, and Q. Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR, pages 10635–10644, 2020. 1, 3
[10] S. Chun. Improved probabilistic image-text representations. arXiv preprint arXiv:2305.18171, 2023. 7, 17
[11] S. Chun, S. J. Oh, R. S. de Rezende, Y. Kalantidis, and D. Larlus. Probabilistic embeddings for cross-modal retrieval. In CVPR, pages 8415-8424, 2021. 7, 8, 17
[12] A. P. Dempster. A generalization of bayesian inference. Journal of the Royal Statistical Society: Series B (Methodological), 30(2):205-232, 1968. 4
[13] H. Diao, Y. Zhang, L. Ma, and H. Lu. Similarity reasoning and filtration for image-text matching. In AAAI, pages 1218–1226, 2021. 7
[14] R. Durrett. Probability: theory and examples, volume 49. Cambridge university press, 2019. 18, 19
[15] M. Dzabraev, M. Kalashnikov, S. Komkov, and A. Petiushko. MDMMT: multidomain multi-modal transformer for video retrieval. In CVPR Workshops, pages 3354-3363, 2021. 1
[16] D. M. Endres and J. E. Schindelin. A new metric for probability distributions. IEEE Trans. Inf. Theory, 49(7):1858-1860, 2003. 18, 19
[17] F. Faghri, D. J. Fleet, J. R. Kiros, and S. Fidler. VSE++: improving visual-semantic embeddings with hard negatives. In BMVC, page 12, 2018. 3
[18] H. Fang, P. Xiong, L. Xu, and Y. Chen. Clip2video: Mastering video-text retrieval via image CLIP. CoRR, abs/2106.11097, 2021. 3, 6, 7
[19] V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In ECCV, volume 12349, pages 214-229, 2020. 1
[20] V. Gabeur, C. Sun, K. Alahari, and C. Schmid. Multi-modal transformer for video retrieval. In ECCV, volume 12349, pages 214-229, 2020. 6
[21] Y. Gal and Z. Ghahramani. Dropout as a bayesian approximation: Representing model uncertainty in deep learning. In ICML, volume 48, pages 1050–1059, 2016. 3
[64] P. Zeng, L. Gao, X. Lyu, S. Jing, and J. Song. Conceptual and syntactical cross-modal alignment with cross-level consistency for image-text matching. In ACM MM, pages 2205–2213, 2021. 3
[65] K. Zhang, Z. Mao, Q. Wang, and Y. Zhang. Negative-aware attention framework for image-text matching. In CVPR, pages 15640-15649, 2022. 7
[66] S. Zhao, L. Zhu, X. Wang, and Y. Yang. Centerclip: Token clustering for efficient text-video retrieval. In SIGIR, pages 970–981, 2022. 3, 6, 7
[67] J. Zhu, P. Zeng, L. Gao, G. Li, D. Liao, and J. Song. Complementarity-aware space learning for video-text retrieval. IEEE Trans. Circuits Syst. Video Technol., 33(8):4362-4374, 2023. 3
A The Model Structure of the Baselines
Figure 7: The model structure of baseline on video-text retrieval. The model takes a video-text pair as input. For the input video, we encode it into  $J$  frame representations. For the input text, we encode it into a text representation. Then, a mean pooling layer aggregates  $J$  frame representations to a global representation, which calculates the global similarity  $s_{global}$  with the text representation. Meanwhile, the maximum value of the similarities between  $J$  frame representations and the text representation is selected as  $s_{local}$ . Finally, we take the average of  $s_{global}$  and  $s_{local}$  as the final similarity  $s$ , which will be constrained by a cross-entropy loss in the training process.  $\otimes$  means cosine similarity.  $\oplus$  means average addition.
For image-text retrieval, we utilize CLIP [54] as the baseline straightly. As for video-text retrieval, the model structure is shown in Figure 7. To be specific, we follow the model structure of the CLIP4CLIP [49] to generate  $J$  frame features  $\mathbf{Z} = \{z_i^f | i = 1,2,\dots,J\}$  for each video and a text feature  $z^t$  for each caption. Afterwards, two branches are designed to catch global and local information, respectively. In global branch, We first adopt a mean pooling to aggregate  $J$  frame features to get an 'average global feature'  $\bar{z}^v$ . Next, the cosine similarity  $s_{global}$  between  $\bar{z}^v$  and  $z^t$  is calculated. In local branch, we first calculate a set of cosine similarities  $S \in \mathbb{R}^{J}$  between  $z^t$  and each frame feature  $z_i^f$ . Then we take the maximum value in  $S$  as  $s_{local}$ . Finally, we take the average of  $s_{global}$  and  $s_{local}$  as the final similarity score. This similarity score will be constrained by a cross-entropy loss.
B Evidence Generation
How to generate evidence  $e$  from the similarity  $s$  between the instance and the prototype is crucial for uncertainty modeling. Following [57], three optional functions, including ReLU function, softplus function, and exponential function, are used to generate evidence as follows:
ReLU function:
$$
e (s) = \left\{ \begin{array}{l l} 0 & , s \leq 0 \\ s & , o t h e r w i s e \end{array} \right. \tag {8}
$$
Softplus function. Let  $\theta$  denote a threshold of reverting to a linear function.  $\gamma$  and  $\theta$  are separately set to 1, 20 in the experiments. Softplus function can formulate as:
$$
e (s) = \left\{ \begin{array}{l l} \frac {1}{\gamma} \log (1 + \exp (\gamma s)) & , \gamma s \leq \theta \\ s & , o t h e r w i s e \end{array} \right. \tag {9}
$$
Exponential function. Let  $\tau$  be a scale parameter of the exponential function (Eq. 10), indicated as 5 in our experiments.
$$
e (s) = \exp (s / \tau) \tag {10}
$$
The performance comparison of three evidence-generation functions is shown in Table 7. It's worth noting that the exponential function outperforms other methods on multiple metrics. We therefore utilize it as the evidence-generation function in all experiments.
Table 7: Evidence generation function comparison on MSR-VTT
Method	t2v	R@1	R@5	v2t
R@1	R@5	R@10	MdR	MnR	R@10	R@10	MdR	MnR
Baseline	45.6	72.9	81.5	2.0	14.5	45.2	71.6	81.5	2.0	10.9
ReLU	46.6	71.9	82.9	2.0	14.4	47.0	72.9	82.4	2.0	10.1
Softplus	47.6	72.7	83.0	2.0	14.4	47.8	71.9	82.9	2.0	10.3
Exponential	48.5	72.7	82.5	2.0	14.0	48.3	73.0	83.2	2.0	9.7
C.1 The number of prototypes
Table 8: Ablation study of prototype number  $K$  on MSR-VTT
Method	t2v	v2t	Mean
R@1	R@5	R@10	R@1	R@5	R@10	R@1
w/o PAU	45.6	72.9	81.5	45.2	71.6	81.5	45.4
K=1	47.2	72.2	81.4	46.2	72.7	81.3	46.7
K=4	47.6	72.8	82.0	46.7	71.6	82.5	47.2
K=8	48.5	72.7	82.5	48.3	73.0	83.2	48.4
K=12	48.6	72.4	82.4	46.9	72.6	82.2	47.8
K=16	47.8	73.0	82.0	47.0	72.8	82.1	47.4
Multiple prototypes are built to represent the overall semantics of the subspace. To figure out the best number of prototypes for representing subspace, the performance variety against prototype number  $K$  is shown in Table 8. It can be seen that PAU always performs better than the baseline. Furthermore, R@1 of both t2v and v2t first increases to the peak against growing  $K$  ( $K = 12$  for t2v,  $K = 8$  for v2t) and then decreases. To model the uncertainty consistently,  $K$  is set to 8 for all modalities due to the maximum mean R@1 when  $K = 8$ .
C.2 Different Initialization Methods for Prototypes
Table 9: Comparison of different prototype initialization methods on MSR-VTT
Method	t2v	v2t
R@1	R@5	R@10	R@1	R@5	R@10
kaiming.uniform [29]	47.2	72.8	81.9	46.3	73.2	81.6
kaiming_normal [29]	46.8	72.5	82.2	46.6	73.0	82.3
orthogonal [56]	47.4	73.0	83.1	47.1	74.0	83.5
Xavier [24]	48.5	72.7	82.5	48.3	73.0	83.2
To explore the impact of different initial methods to prototypes, we employ several initialization methods for prototypes on MSR-VTT. The methods contain Xavier [24], kaiming_normal [29], kaiming.uniform [29], orthogonal [56] initialization. The results are shown below. We can observe that Xavier init methods can get the best R@1 performance on both t2v and v2t.
C.3 Comparing Model Size and Complexity
The FLOPs and parameter number of PAU on MSR-VTT are computed to explore the computational costs and model complexity. We compare PAU with some previous methods, such as VATT [1],
Table 10: Comparison of model size and complexity on MSR-VTT.
Method	Params (M)	FLOPs (G)
VATT [1]	327.0	792.0
Frozen [4]	180.9	771.0
MILES [23]	180.9	771.0
PAU	84.2	36.0
Frozen [4], MILES [23]. The results are shown as below. It can be seen that the FLOPs and the parameter number of PAU are both significantly smaller than other methods, indicating that PAU is a light-weight approach and easy to deploy.
C.4 Training with Noisy Correspondences
Table 11: Comparison on noisy MS-COCO2
Noise ratio	Method	MS-COCO 1K	MS-COCO 5K
i2t	R@1	R@5	R@1	R@5	R@10	R@1	R@5	R@10	R@1	R@5
0%	VSE∞ [8]	82.0	97.2	98.9	69.0	92.6	96.8	62.3	87.1	93.3	48.2	76.7	85.5
PCME [11]	80.1	96.6	98.7	67.6	92.1	96.9	59.9	85.8	92.3	46.1	75.0	84.6
PCME++ [10]	81.6	97.2	99.0	69.2	92.8	97.1	62.1	86.8	93.3	48.1	76.7	85.5
Baseline	80.1	95.7	98.2	67.1	91.4	96.6	62.9	84.9	91.6	46.5	73.8	82.9
ours	80.4	96.2	98.5	67.7	91.8	96.6	63.6	85.2	92.2	46.8	74.4	83.7
20%	VSE∞ [8]	78.4	94.3	97.0	65.5	89.3	94.1	58.6	83.4	89.9	45.0	72.9	81.7
PCME [11]	76.8	95.4	98.2	63.8	90.5	96.0	55.7	81.7	90.0	41.8	71.4	81.7
PCME++ [10]	78.4	95.9	98.4	64.9	90.8	96.1	57.7	83.9	91.0	43.2	72.3	82.4
Baseline	76.0	94.3	97.5	63.4	89.0	94.8	55.3	79.1	86.9	41.0	68.8	79.3
ours	78.2	95.2	98.1	64.5	90.0	95.4	59.3	82.9	90.4	44.2	71.3	81.3
50%	VSE∞ [8]	44.3	76.1	86.9	34.0	69.2	84.5	22.4	48.2	61.1	15.8	38.8	52.1
PCME [11]	73.4	94.0	97.6	59.8	88.3	94.8	50.9	78.8	87.3	37.9	67.3	78.4
PCME++ [10]	74.8	94.3	97.7	60.4	88.7	95.0	52.5	79.6	88.4	38.6	68.0	79.0
Baseline	73.9	93.0	97.2	60.1	87.3	94.0	54.1	78.5	86.6	39.7	67.2	77.5
ours	76.4	94.1	97.6	62.3	88.5	94.6	57.3	81.5	88.8	41.9	69.4	79.6
We additionally compare PAU with some CLIP ViT-B/32 backbone-based methods [8, 11, 10, 54] fine-tuned on noisy MS-COCO dataset. The results of training on  $0\%$ ,  $20\%$ , and  $50\%$  noise ratios are reported in the Table 11. We can observe that our method exhibits significant advantages while training with several noisy correspondences, especially on  $50\%$  noise ratio. Although the uncertainty-aware method, PAU, is not designed for tackling the noisy correspondences, it still successfully handles the scenario, indicating that PAU possesses powerful robust learning ability even while training with noisy correspondences.
D The Explanation of High Boost on MSVD
In Table 2, we find the performance improvement of MSVD on v2t is significantly larger than other benchmarks. Investigating its reason, the MSVD is a tiny dataset with only 1,200 videos for the training set and approximately 40 captions per video (48,000 captions in total). In this case, utilizing cross-entropy loss, which views the diagonal pairs as positives, will lead into tremendous noise, especially with a large batchsize. In a batch with the size of 256, the probability of any texts coming from different videos is about  $3.49 \times 10^{-13}$ . The solution is as follows.
$$
P = \frac {\prod_ {i = 1} ^ {B} C _ {m _ {i}} ^ {1}}{C _ {N} ^ {B}}. \tag {11}
$$
The probability of any texts coming from different videos can denote as Eq. 11, where  $B$  is the batchsize of 256, and  $N$  is the total caption number of 48,000.  $m_{i}$  indicates the candidate number
2Results come from: https://github.com/naver-ai/pcmepp/issues/2
of  $i^{th}$  text in a batch with  $m_{i} = 48000 - 40 \times (i - 1)$  in MSVD. Then the probability  $P$  can be calculated by:
$$
P = \frac {\prod_ {i = 1} ^ {2 5 6} C _ {4 8 0 0 0 - 4 0 \times (i - 1)} ^ {1}}{C _ {4 8 0 0 0} ^ {2 5 6}}
$$
$$
\log P = \sum_ {i = 1} ^ {2 5 6} \log (4 8 0 0 0 - 4 0 \times (i - 1)) - \sum_ {i = 1} ^ {2 5 6} \log (4 8 0 0 0 - i) \tag {12}
$$
$$
\begin{array}{l} \log P = (\log 4 8 0 0 0 + \log 4 7 9 6 0 + \log 4 7 9 2 0 + \dots + \log 3 7 8 0 0) \\ - \left(\log 4 8 0 0 0 + \log 4 7 9 9 9 + \log 4 7 9 9 8 + \dots + \log 4 7 7 4 5\right) \\ \end{array}
$$
$$
\log P \approx - 2 8. 6 8 5
$$
$$
P \approx - 3. 4 9 1 0 ^ {- 1 3}.
$$