SlowGuess/NAACL_2022_OCR huggingface dataset:
Jiaheng Liu<sup>1,2</sup>, Tan Yu<sup>1</sup>, Hanyu Peng<sup>1</sup>, Mingming Sun<sup>1</sup>, Ping Li<sup>1</sup>
$^{1}$ Cognitive Computing Lab, Baidu Research
$^{2}$ State Key Lab of Software Development Environment, Beihang University
liujiaheng@buaa.edu.cn
{tanyu01,penghanyu,sunmingming01,liping11}@baidu.com
Existing multilingual video corpus moment retrieval (mVCMR) methods are mainly based on a two-stream structure. The visual stream utilizes the visual content in the video to estimate the query-visual similarity, and the subtitle stream exploits the query-subtitle similarity. The final query-video similarity ensembles similarities from two streams. In our work, we propose a simple and effective strategy termed as Cross-lingual Cross-modal Consolidation  $(\mathrm{C}^{3})$  to improve mVCMR accuracy. We adopt the ensemble similarity as the teacher to guide the training of each stream, leading to a more powerful ensemble similarity. Meanwhile, we use the teacher for a specific language to guide the student for another language to exploit the complementary knowledge across languages. Extensive experiments on mTVR dataset demonstrate the effectiveness of our  $\mathbf{C}^3$  method.
Video Corpus Moment Retrieval (VCMR) task has been proposed in (Escorcia et al., 2019; Lei et al., 2020), which aims to retrieve a short moment from a large video corpus given a natural language query. Recently, (Lei et al., 2021a) introduces a multilingual Video Corpus Moment Retrieval (mVCMR) task. Compared with VCMR, mVCMR supports queries in multiple languages. It is more useful in practice, especially in international applications.
To facilitate the research in mVCMR, (Lei et al., 2021a) builds a large-scale mTVR dataset, where queries are in two languages. Apart from the video's visual content, the video's textual subtitles are provided as auxiliary information to help the query-to-video retrieval. (Lei et al., 2021a) proposes an mXML model to generate the query-video similarity for video retrieval and the query-clip similarity for moment localization with a two-stream structure. The subtitle stream captures the similarity between the text query and the video's textual subtitles. In parallel, the visual stream describes the
Figure 1: Illustration of our Cross-lingual Cross-modal Consolidation  $(\mathrm{C}^3)$ . The left green and right yellow boxes contain similarities using English and Chinese queries, respectively. Query-visual similarity measures the relevance between the query and visual context, and query-subtitle similarity denotes that between the query and subtitle context. The final similarity is obtained by summing up the query-visual and query-subtitle similarities. The final similarity in a specific language serves as the teacher to guide the learning of query-subtitle/query-visual similarity in another language.
similarity between the text query and the video's visual content. The final text-video similarity is a summation of the similarities from two streams.
As the final similarity is obtained from summing up similarities of two streams, straightforwardly, it is reasonable to hypothesize that improving the effectiveness of the similarity from each stream is beneficial to enhancing the performance of the final similarity. Meanwhile, since the final similarity fuses the information from two modalities, it is also reasonable to hypothesize that the final similarity is more reliable than the similarity from each stream. Based on the above two hypotheses, we propose a simple approach to improve the performance of two-stream architecture for the mVCMR task. To be specific, we use the final similarity fusing two modalities as the teacher and the similarity from each stream as the student. We train the student through the guidance of the teacher's knowledge. Meanwhile, to exploit the natural compensation across languages, we devise a student in one language and the teacher in another language. We term our method as cross-lingual cross-modal con
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1854 - 1862
solidation  $(\mathrm{C}^{3})$ , as visualized in Figure 1. Comprehensive experimental results on the mTVR dataset demonstrate the effectiveness of our  $\mathrm{C}^{3}$  method.
Text-video retrieval. Traditionally, text-video retrieval (Rohrbach et al., 2015; Xu et al., 2016) is normally tackled through two mainstream methods: joint-embedding methods (Xu et al., 2015; Torabi et al., 2016; Pan et al., 2016; Plummer et al., 2017; Miech et al., 2019) and attention-based methods (Yu et al., 2017, 2018; Hori et al., 2017; Krishna et al., 2017). Recently, inspired by the great success of pre-training achieved by Transformer (Vaswani et al., 2017) and BERT (Devlin et al., 2019) in natural language processing, Transformer/BERT-based models emerge for solving text-video retrieval (Sun et al., 2019b,a; Li et al., 2020; Luo et al., 2020; Lei et al., 2021b).
Video corpus moment retrieval. Video corpus moment retrieval (Escorcia et al., 2019; Lei et al., 2020) aims to retrieve the ground-truth video from the whole corpus and predict the moment with high Intersection-of-Union (IoU) with the ground-truth moment using the natural language query. In practice, videos are often associated with other modalities (e.g., subtitles), so the multi-modal moment retrieval task with both visual and text contexts has been proposed. The recent work mTVR (Lei et al., 2021a) extends the monolingual moment retrieval task to the multilingual setting, and introduces a large-scale multilingual moment retrieval (i.e., mTVR) dataset, where the language queries and subtitles are in two languages (i.e., English and Chinese). Meanwhile, it proposes the mXML model jointly trained on the English and Chinese data for multilingual video moment retrieval.
Knowledge distillation. (Hinton et al., 2015) proposes knowledge distillation (KD), where the student network is trained by the soft output of the teacher network. Recently, knowledge distillation has demonstrated the effectiveness for many vision and language tasks (Tan et al., 2018; Hu et al., 2020; Fu et al., 2021; Wang et al., 2020; Fei et al., 2021; Sun et al., 2019c; Hou et al., 2020; Sanh et al., 2019; Peng et al., 2019; Jin et al., 2019; Liu et al., 2022, 2020, 2021). For example, (Sun et al., 2019c) proposes a Patient Knowledge Distillation method to compress an original large model into an equally-effective lightweight shallow network for pre-trained language models (Devlin et al., 2019).
Problem Definition. The current multilingual moment retrieval model mXML (Lei et al., 2021a) performs video retrieval at its shallow layers and moment localization at its deep layers. We denote a query by  $q_{g}$ , where  $g$  denotes the language type, e.g., English. A video  $v$  consists of  $L$  consecutive moments  $\{c_l\}_{l=1}^L$ . Each moment  $c_l$  is paired with subtitle  $s_{g,l}$ . mXML generates the query-video score  $S(q_{g},v)$  for video retrieval and the index of the start/end frame  $t_{\mathrm{st}} / t_{\mathrm{ed}}$  for moment localization:
$$
[ S (q _ {g}, v), t _ {\mathrm {s t}}, t _ {\mathrm {e d}} ] = \mathrm {m X M L} (q _ {g}, \{(c _ {l}, s _ {g, l}) \} _ {l = 1} ^ {L}).
$$
mXML supports English (en) and Chinese (zh), i.e.,  $g \in \{\mathrm{en},\mathrm{zh}\}$ . Below, we describe the video retrieval and moment localization in mXML briefly. You can refer to the Appendix A for more details.
Input Feature. ResNet-152 (He et al., 2016) and I3D (Carreira and Zisserman, 2017) extract the visual features of each video moment. The language features are extracted by RoBERTa-base (Liu et al., 2019) for English (Liu et al., 2019) and Chinese (Cui et al., 2020), respectively. Self-Encoder (SE) based on Transformer (Vaswani et al., 2017) and modular attention (Lei et al., 2020) are used to further encode the visual and text features.
Video Retrieval. The subtitle-based score for video retrieval  $S_{s}(q_{g},v)$  and the visual-based score  $S_{v}(q_{g},v)$  are obtained by two streams, respectively. The details of obtaining  $S_{s}(q_{g},v)$  and  $S_{v}(q_{g},v)$  are shown in Appendix A.3. The final video retrieval score  $S(q_{g},v)$  using both contexts is devised as
$$
S \left(q _ {g}, v\right) = S _ {s} \left(q _ {g}, v\right) + S _ {v} \left(q _ {g}, v\right). \tag {1}
$$
Moment Localization. The subtitle-based query-clip score  $S_{s}(q_{g},c_{l})$  and the visual-based query-clip score  $S_{v}(q_{g},c_{l})$  are computed by two streams, respectively. The details of computing  $S_{s}(q_{g},c_{l})$  and  $S_{v}(q_{g},c_{l})$  are shown in Appendix A.4. The final query-clip score is devised as
$$
S \left(q _ {g}, c _ {l}\right) = S _ {s} \left(q _ {g}, c _ {l}\right) + S _ {v} \left(q _ {g}, c _ {l}\right). \tag {2}
$$
Then, to produce moment localization predictions from  $S(q_{g}, c_{l})$ , mXML predicts the start and end probabilities  $\mathbf{p}_{g}^{\mathrm{st}}$ ,  $\mathbf{p}_{g}^{\mathrm{ed}} \in \mathbb{R}^{L}$  for each query.
For mXML, the video retrieval loss  $\mathcal{L}^{\mathrm{vr}}$ , moment localization loss  $\mathcal{L}^{\mathrm{loc}}$  and language neighborhood
constraint loss  $\mathcal{L}^{\mathrm{nc}}$  are illustrated in Appendix A.5. The final loss function of mXML is devised as
$$
\mathcal {L} _ {\mathrm {m X M L}} = \mathcal {L} ^ {\mathrm {v r}} + \lambda_ {1} \mathcal {L} ^ {\mathrm {l o c}} + \lambda_ {2} \mathcal {L} ^ {\mathrm {n c}}, \tag {3}
$$
where  $\lambda_{1}$  and  $\lambda_{2}$  are the loss weights.
3.2 C³ in Video Retrieval
In Eq. (1), the final video-level similarity  $S(q_{g}, v)$  is a summation of scores from two modalities,  $S_{v}(q_{g}, v)$  and  $S_{s}(q_{g}, v)$ . We use  $S(q_{g}, v)$  with knowledge of two modalities as the teacher and distill (Hinton et al., 2015) its knowledge to the score with information of only a single modality.
In the multilingual scenario, to exploit more complementary knowledge, a more effective approach is to distill the scores from a language  $g \in \{\mathrm{en},\mathrm{zh}\}$  to the scores from another language  $h \in \{\mathrm{en},\mathrm{zh}\}$ :
$$
S \left(q _ {h}, v\right) \xrightarrow [ \text {d i s t i l l} ]{\text {d i s t i l l}} S _ {v} \left(q _ {g}, v\right), \tag {4}
$$
$$
S (q _ {h}, v) \xrightarrow {\text {d i s t i l l}} S _ {s} (q _ {g}, v).
$$
Given a mini-batch of query-video pairs  $\{(q_g^i,v^i)\}_{i = 1}^n$ , where  $n$  is the batchsize,  $S(q_h^i,v^k)$  is the similarity score between  $q_{h}^{i}$  and  $v^{k}$  of the teacher model based on two modalities (i.e., visual and subtitle contexts), where  $k\in [1,n]$ .  $S_{v}(q_{g}^{i},v^{k})$  and  $S_{s}(q_{g}^{i},v^{k})$  are the corresponding similarity scores of the student model from visual and subtitle contexts, respectively. Then, for each query  $q_{h}^{i}$ , we can generate the teacher scores  $\{S(q_h^i,v^k)\}_{k = 1}^n$ , and perform the softmax function with temperature  $\tau_{\mathrm{vr}}$  on the scores to obtain the normalized score:
$$
\hat {S} \left(q _ {h} ^ {i}, v ^ {i}\right) = \frac {e ^ {S \left(q _ {h} ^ {i} , v ^ {i}\right) / \tau_ {\mathrm {v r}}}}{\sum_ {k = 1} ^ {n} e ^ {S \left(q _ {h} ^ {i} , v ^ {k}\right) / \tau_ {\mathrm {v r}}}}. \tag {5}
$$
In the same manner, we obtain the normalized student scores  $\hat{S}_v(q_h^i,v^i)$  and  $\hat{S}_s(q_h^i,v^i)$ . Finally, the  $\mathbf{C}^3$  loss for video retrieval is devised as
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C} ^ {3}} ^ {\mathrm {v r}} = \sum_ {i = 1} ^ {n} \sum_ {g \in \{\mathrm {e n}, \mathrm {z h} \}} \sum_ {h \neq g} \frac {- 1}{n} [ \hat {S} (q _ {h} ^ {i}, v ^ {i}) \log (\hat {S} _ {v} (q _ {h} ^ {i}, v ^ {i})) \\ \left. + \hat {S} \left(q _ {h} ^ {i}, v ^ {i}\right) \log \left(\hat {S} _ {s} \left(q _ {h} ^ {i}, v ^ {i}\right)\right) \right]. \tag {6} \\ \end{array}
$$
The loss for video retrieval is devised as
$$
\mathcal {L} _ {+} ^ {\mathrm {v r}} = \mathcal {L} ^ {\mathrm {v r}} + \alpha \mathcal {L} _ {\mathrm {C} ^ {3}} ^ {\mathrm {v r}}, \tag {7}
$$
where  $\alpha$  is a pre-defined positive constant.
3.3 C³ in Moment Localization
Similarly,  $\mathbf{C}^3$  can also be used on the moment localization. In Eq. (2), we generate the query-clip similarity score using two contexts, and then produce the start and end probabilities. In the same way, based on  $S_{s}(q_{g},c_{l})$  with single subtitle context, we can generate the start and end probabilities  $\mathbf{p}_{g,s}^{\mathrm{st}},\mathbf{p}_{g,s}^{\mathrm{ed}}\in \mathbb{R}^{L}$  , and based on  $S_{v}(q_{g},c_{l})$  with single visual context, we can generate  $\mathbf{p}_{g,v}^{\mathrm{st}},\mathbf{p}_{g,v}^{\mathrm{ed}}\in$ $\mathbb{R}^L$  . Note that we use the softmax function with temperature  $\tau_{\mathrm{loc}}$  to generate the start and end probabilities in the similar way of Eq. (5). We define the start and end probabilities of the teacher model from language  $h$  as  $\mathbf{p}_h^{\mathrm{st}}$  and  $\mathbf{p}_h^{\mathrm{ed}}$  , and the start and end probabilities of the student model from language  $g$  as  $\mathbf{p}_{g,v}^{\mathrm{st}},\mathbf{p}_{g,v}^{\mathrm{ed}}$  using visual context and  $\mathbf{p}_{g,s}^{\mathrm{st}},\mathbf{p}_{g,s}^{\mathrm{ed}}$  using subtitle context. Thus, the  $\mathbf{C}^3$  loss for moment localization is defined as follows:
$$
\begin{array}{l} \mathcal {L} _ {\mathrm {C} ^ {3}} ^ {\text {l o c}} = \sum_ {g \in \{\text {e n}, \text {z h} \}} \sum_ {h \neq g} \left[ \operatorname {C E} \left(\mathbf {p} _ {h} ^ {\mathrm {s t}}, \mathbf {p} _ {g, v} ^ {\mathrm {s t}}\right) + \operatorname {C E} \left(\mathbf {p} _ {h} ^ {\mathrm {s t}}, \mathbf {p} _ {g, s} ^ {\mathrm {s t}}\right) \right. \tag {8} \\ + \mathrm {C E} (\mathbf {p} _ {h} ^ {\mathrm {e d}}, \mathbf {p} _ {g, v} ^ {\mathrm {e d}}) + \mathrm {C E} (\mathbf {p} _ {h} ^ {\mathrm {e d}}, \mathbf {p} _ {g, s} ^ {\mathrm {e d}}) ], \\ \end{array}
$$
where  $\mathrm{CE}()$  is cross-entropy function defined as
$$
\operatorname {C E} (\mathbf {x}, \mathbf {y}) = - \sum_ {l = 1} ^ {L} \mathbf {x} [ l ] \log (\mathbf {y} [ l ]). \tag {9}
$$
The loss for moment localization is as follows:
$$
\mathcal {L} _ {+} ^ {\mathrm {l o c}} = \mathcal {L} ^ {\mathrm {l o c}} + \beta \mathcal {L} _ {\mathrm {C} ^ {3}} ^ {\mathrm {l o c}}, \tag {10}
$$
where  $\beta$  is a pre-defined positive constant.
Final loss function. Considering our proposed  $C^3$ , the final loss function is constructed as follows:
$$
\mathcal {L} _ {\mathrm {m X M L} +} = \mathcal {L} _ {+} ^ {\mathrm {v r}} + \lambda_ {1} \mathcal {L} _ {+} ^ {\mathrm {l o c}} + \lambda_ {2} \mathcal {L} ^ {\mathrm {n c}}, \tag {11}
$$
which is similar to the formulation of  $\mathcal{L}_{\mathrm{mXML}}$  defined in Eq. (3) but replaces  $\mathcal{L}^{\mathrm{vr}}$  by its counterpart  $\mathcal{L}_{+}^{\mathrm{vr}}$  and  $\mathcal{L}^{\mathrm{loc}}$  by  $\mathcal{L}_{+}^{\mathrm{loc}}$  based on our  $\mathbf{C}^3$ .
Training. The training consists of two stages. In the first stage, we train the standard mXML model using the loss function  $\mathcal{L}_{\mathrm{mXML}}$  in Eq. (3) to obtain the teacher model, which produces the query-video score  $S(q_{g},v)$  and the query-clip score  $S(q_{g},c_{l})$  with knowledge of two modalities. Then, in the second stage, we use the teacher mXML model to distill the training process of the randomly initialized student mXML model using the loss function  $\mathcal{L}_{\mathrm{mXML + }}$  in Eq. (11). After training, the student
Algorithm 1: The training process.
1 Train the teacher  $\mathrm{mXML}_{\mathrm{T}}$  , a standard mXML model, using  $\mathcal{L}_{\mathrm{mXML}}$  in Eq. (3).
2 for  $i\in [1,Q]$  do
3 Initialize a student model  $\mathrm{mXML_S}$
4  $\begin{array}{rl} & {\mathrm{mXML_T}\xrightarrow{\mathrm{distill}}\mathrm{mXML_S~using}}\\ & {\mathcal{L}_{\mathrm{mXML + }}\mathrm{in~Eq.~(11).}} \end{array}$
5  $\mathrm{mXML}_{\mathrm{T}} = \mathrm{mXML}_{\mathrm{S}}$
6 end
7 return mXMLs
mXML model will perform better than the teacher model. Thus, we utilize the trained student mXML model as the new teacher mXML model to guide the training of a new randomly initialized student mXML model from the beginning. We repeat the distillation process in the second stage for  $Q$  iterations until the performance saturates. For better clarification, we summarize the training process of our proposed  $C^3$  method as shown in Algorithm 1.
Inference. Since our method is orthogonal to the model, the inference process is same as the mXML.
Dataset. mTVR (Lei et al., 2021a) is a large-scale multilingual video moment retrieval dataset, which contains 218 thousand English and Chinese queries from 21.8 thousand TV show video clips. This dataset extends the TVR dataset (in English) (Lei et al., 2020) and adds Chinese queries and subtitles. (Lei et al., 2021a) proposed to split the mTVR dataset into  $80\%$  train,  $10\%$  val,  $5\%$  test-public and  $5\%$  test-private datasets. We use this dataset to validate the effectiveness of our method for research purpose. All experiments in our work are conducted on one NVIDIA Tesla V100 GPU.
Experimental setting. We report the average recall at K (i.e., R@K) for multilingual Video Corpus Moment Retrieval (mVCMR) task on the mTVR (Lei et al., 2021a) dataset, where the predicted moment is right when it has high Intersection-over-Union (IoU) with the ground-truth. We use the same training strategy and network architecture of mXML (Lei et al., 2021a).  $\lambda_{1}$  and  $\lambda_{2}$  of Eq. (3) and Eq. (11) are set as 0.01, 1, respectively. The loss weights  $\alpha$  and  $\beta$  of  $\mathcal{L}_{\mathrm{C}^3}^{\mathrm{vr}}$  and  $\mathcal{L}_{\mathrm{C}^3}^{\mathrm{loc}}$  for video retrieval and moment localization are set as 1.0, 100, respectively. The  $\tau_{\mathrm{vr}}$  is set as 0.02, and  $\tau_{\mathrm{loc}}$  is set as 1. The number of distillation iterations,  $Q$ ,
Table 1: R@1 on the test-public split of mTVR.
Methods	English	Chinese
IoU=0.5	IoU=0.7	IoU=0.5	IoU=0.7
MCN	0.02	0.00	0.13	0.02
CAL	0.09	0.04	0.11	0.04
MEE+MCN	0.92	0.42	1.43	0.64
MEE+CAL	0.97	0.39	1.51	0.62
MEE+ExCL	0.92	0.33	1.43	0.72
XML	7.25	3.25	5.91	2.57
mXML	8.30	3.82	6.76	3.20
C3	9.11	4.72	7.05	4.08
Table 2: R@10 on the test-public split of mTVR.
Methods	English	Chinese
IoU=0.5	IoU=0.7	IoU=0.5	IoU=0.7
XML	22.79	13.96	18.93	11.13
mXML	23.27	13.98	18.99	11.52
C3	24.72	16.00	20.62	13.30
is set as 2. We deploy our proposed  $\mathbf{C}^3$  in mXML and abbreviate the mXML with our  $\mathbf{C}^3$  to  $\mathbf{C}^3$ .
In Table 1, on the test-public split<sup>1</sup> of the mTVR dataset, we compare  $\mathrm{C}^3$  with existing methods, including proposal-based approaches (MCN (Anne Hendricks et al., 2017) and CAL (Es-corcia et al., 2019)), reranking-based methods (MEE (Miech et al., 2018)+MCN, MEE+CAL, MEE+ExCL (Ghosh et al., 2019) and XML (Lei et al., 2020)) and the state-of-the-art mXML (Lei et al., 2021a). We observe that our  $\mathrm{C}^3$  achieves consistently higher R@1 in both English and Chinese than the baseline methods. In Table 2,  $\mathrm{C}^3$  also considerably improves the R@10 of the XML and mXML on the test-public split of the mTVR dataset. Note that the R@10 results of XML is based on our re-implementation. In Table 3, we compare the performance on the val split of the mTVR dataset, and  $\mathrm{C}^3$  also outperforms the mXML a lot, which further shows the advantage of our method.
Effect of different components. By default, we exploit the  $C^3$  in both video retrieval  $(\mathcal{L}_{\mathrm{C}^3}^{\mathrm{vr}}$  in Eq. (7)) and moment localization  $(\mathcal{L}_{\mathrm{C}^3}^{\mathrm{loc}}$  in Eq. (10)). We conduct the experiments to evaluate the importance of  $\mathcal{L}_{\mathrm{C}^3}^{\mathrm{loc}}$  and  $\mathcal{L}_{\mathrm{C}^3}^{\mathrm{vr}}$  by removing one of them, respectively. From Table 4, we observe that, by removing  $\mathcal{L}_{\mathrm{C}^3}^{\mathrm{loc}}$  or  $\mathcal{L}_{\mathrm{C}^3}^{\mathrm{vr}}$ , the R@1 of video corpus moment retrieval considerably deteriorates. It validates the
'https://competitions.codalab.org/competitions/33493
Table 3: R@1 on the val split of mTVR.
Methods	English	Chinese
IoU=0.5	IoU=0.7	IoU=0.5	IoU=0.7
mXML	6.22	2.96	5.17	2.41
C3	7.44	3.85	5.70	2.86
Table 4: R@1 of the proposed C<sup>3</sup> and its alternative variants by removing one of the components in video corpus moment retrieval on the val split of mTVR.
Methods	English	Chinese
IoU=0.5	IoU=0.7	IoU=0.5	IoU=0.7
w/o LocC3	6.55	3.13	5.55	2.54
w/o LvrC3	6.71	3.54	5.53	2.72
C3	7.44	3.85	5.70	2.86
Table 5: R@1 and R@5 in the video retrieval (VR) of different methods on the val split of mTVR.
Method	English	Chinese
R@1	R@5	R@1	R@5
mXML	19.35	42.32	17.75	39.34
C3	21.51	45.30	19.42	41.68
Table 6: R@1 in the single video moment retrieval (SVMR) on the val split of mTVR.
Method	English	Chinese
IoU=0.5	IoU=0.7	IoU=0.5	IoU=0.7
mXML	29.05	13.20	26.31	11.46
C3	30.13	13.99	27.89	12.92
necessity of using both  $\mathcal{L}_{\mathrm{C}^3}^{\mathrm{loc}}$  and  $\mathcal{L}_{\mathrm{C}^3}^{\mathrm{vr}}$  in our  $\mathrm{C}^3$ .
Analysis on video retrieval. It is worth noting that we consider both the video retrieval and the moment localization in a single video for mVCMR task. To more comprehensively demonstrate the advantage of our  $\mathrm{C}^3$ , we also investigate its influence in the video retrieval task. From Table 5, we can observe that our  $\mathrm{C}^3$  outperforms the baseline model, mXML, by a large margin.
Analysis on single video moment retrieval. To further demonstrate the effectiveness of the proposed  $C^3$  in moment localization, we also report the single video moment retrieval (i.e., SVMR) results. From Table 6, we observe that our  $C^3$  achieves significant performance improvement compared with mXML. It indicates that our  $C^3$  predicts more accurate moment localization results, which shows the effectiveness of our  $C^3$  in moment localization.
Extension on monolingual video corpus moment retrieval. Despite that the proposed  $C^3$  method is devised to mVCMR, it is also naturally applicable to monolingual video corpus moment retrieval. Here, we evaluate the effectiveness of our  $C^3$  in the monolingual setting. TVR dataset (Lei et al., 2020)
Table 7: R@1 of different methods on both val and testpublic splits of the TVR dataset.
Method	val	test-public
IoU=0.5	IoU=0.7	IoU=0.5	IoU=0.7
XML	5.28	2.62	7.25	3.25
IC²	6.27	2.93	8.45	4.00
contains 109 thousand queries collected on 21.8 thousand videos from 6 TV shows of diverse genres, where each video is associated with subtitles and each query is associated with a tight temporal window. Specifically, we use the XML (Lei et al., 2020) model as the baseline method, and propose an alternative variant by applying the our  $\mathrm{C}^3$  loss functions between the outputs of the multi-modality and to the outputs of the single modality for both the video retrieval and moment localization tasks following the similar strategy of the Eq. (6) and Eq. (8). In other words, this alternative variant is called as the intra-lingual cross-modal consolidation  $(\mathrm{IC}^2)$  method. In Table 7, we compare our alternative variant  $\mathrm{IC}^2$  with the baseline model XML on both val and test-public splits of the TVR dataset for monolingual video corpus moment retrieval. As shown in Table 7, our  $\mathrm{IC}^2$  also achieves significant performance improvements on the TVR dataset, which further demonstrates the versatility of our proposed consolidation strategy.
5 Limitations and Potential risks
Although our  $\mathbf{C}^3$  has achieved substantial improvement based on mXML on the mTVR dataset, we find that there exists some hyper-parameters (e.g., the  $\tau_{\mathrm{vr}}$ ,  $\tau_{\mathrm{loc}}$ ) to tune in  $\mathbf{C}^3$ , which may be time-consuming. Besides, we develop the  $\mathbf{C}^3$  strategy to improve the performance of mVCMR task, and we have not seen the potential ricks in our paper.