SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Figure 3: Snapshot of video news coverage of the 2019 Notre Dame fire news cycle (an event in MultiVENT). The fire and the fallen spire were initially reported through first-person social media video uploads (at 10:02 AM and 10:51 AM, respectively) and then later broadcast by news organizations in more detail (10:38 AM, 11:10 AM, 12:20 PM, 12:26 PM). Some news coverage (12:20 PM) directly used first-person social media footage (10:51 AM). Hours later, news agencies uploaded more complete news stories with details and context (6:20 PM). This data suggests that it is important for models to learn from both first-person videos and official news coverage at various points in the news cycle to fully construct a factual model of the event, especially if the model is attempting to construct an event model while information develops online.
cycles for slowly unfolding events and for sudden, unexpected events that take time to assess. This is illustrated in Figure 3, which shows a snapshot of the 2019 Notre Dame fire news cycle and demonstrates how unedited and poorly curated footage, often first-person witness accounts on social media, can be instrumental in the construction of our collective understanding of events. So, we propose that teaching models to understand different video formats, despite clear discrepancies in the amount of information they present, is important for developing robust systems.
5.1 Approach
We consider the problem of teaching a model to map multilingual, natural language queries to multilingual video clips. Specifically, we consider a video set  $V$  and query set  $T$  with a indicator mapping function  $f$  that returns whether a query  $t \in T$  describes a video  $v \in V$ . The model  $h$  is provided with the full set of videos  $V$  and a text query  $t \in T$ , and for each video  $v \in V$  returns the probability that  $t$  describes  $v$ , or  $h(v, t) = \mathbb{P}[f(v, t) = 1]$ . When there is a bijective mapping between queries and videos (e.g., when using video descriptions as queries), the model is evaluated on its recall when considering the top 1, 5, and 10 ranked videos (R@1, R@5, and R@10), as well as the median rank (MedR). When a given query may describe multiple videos, (e.g., when using event descriptions as queries), we instead evaluate the model on its precision given the top 1, 5, and 10 ranked videos (P@1, P@5, and P@10). We define these metrics as:
$$
\text{Given} S:= \operatorname *{arg  max}_{V^{\prime}\subseteq V:|V^{\prime}| = k}\sum_{v\in V^{\prime}}h(v,t),
$$
$$
\mathrm {R} @ k = \frac {| \{s \in S : f (s , t) = 1 \} |}{| \{v \in V : f (v , t) = 1 \} |} \quad \text {a n d} \quad \mathrm {P} @ k = \frac {| \{s \in S : f (s , t) = 1 \} |}{k}.
$$
5.2 Model architecture and training
We introduce MultiCLIP, a multilingual baseline for video retrieval on MultiVENT. This model adopts a CLIP architecture with a multilingual text encoder and is trained on multilingual text-video pairs. In more detail:
We base our architecture on the pretrained LAION CLIP ViT-H/14 frozen XLM-Roberta-Large model [11], which jointly trains an image and text encoder on text-image data to learn to pair images with their captions. At test time, it produces a zero-shot linear layer based on the test input's visual features through which natural language captions can be passed in. The model architecture contains a vision encoder based on a ViT architecture [14] and a text encoder based on the multilingual XLM Roberta large model [12]. A full overview of the CLIP architecture and pretraining can be found in the original paper [34].
In experiments using MultiCLIP, we first tokenize text descriptions using the XLM-Roberta-Large tokenizer, containing a vocabulary of over 250,000 words, and pass the tokens into MultiCLIP which produces a text embedding of size 1024. Next, we uniformly sample videos at a rate of 12 frames per video with an input size of  $224 \times 224$ , which the model uses to create a frame embedding of size 1024. To incorporate multilinguality into the model's frame-level features, we use a ViT architecture trained with a contrastive objective over multilingual image-caption pairs from the LAION-5B dataset [44], which is constructed from the Common Crawl archive using images and their alt-text to produce a multilingual image-text dataset with over 100 languages. We mean pool the frame embeddings to produce a final video embedding, and use the text and video features to compute a similarity matrix of videos and descriptions.
5.3 Retrieval baselines
We first evaluate MultiCLIP on the existing video retrieval task MSR-VTT [53] using the recall metrics described in Sec. 5.1 alongside contemporary video retrieval approaches (FrozenInTime [4], Clip2Video [16], InternVideo [49], and MPLUG-2 [52]). Results on MSR-VTT's validation set are reported in Table 4. The results indicate MultiCLIP performs well on standard video retrieval tasks, matching performance of separate text/vision pipeline models released within the last two years. It performs better than existing models that use separate text and vision pipelines (FrozenInTime [4] and Clip2Video [16]), but not as well as models that use larger architectures involving multimodal encodings (InternVideo [49] and MPLUG-2 [52]).
Table 4: MultiCLIP evaluated alongside existing video retrieval approaches on the video retrieval benchmark MSR-VTT. Results indicate that MultiCLIP performs adequately on existing retrieval tasks, achieving comparable results to existing models. It does not perform as well as architectures that use multimodal transformers for joint encodings such as InternVideo and MPLUG-2.
Method	Year	Rank@1	Rank@5	Rank@10
FrozenInTime [4]	2021	32.5	61.5	71.2
Clip2Video [16]	2021	29.8	55.5	66.2
InternVideo [49]	2022	55.2	79.6	87.5
MPLUG-2 [52]	2023	53.1	77.6	84.7
MultiCLIP	2023	38.4	70.1	82.7
5.4 MultiVENT retrieval
We now evaluate MultiCLIP and related retrieval approaches on MultiVENT. We first use multilingual video descriptions as queries, and then we use English event summaries taken from the grounding text documents, meaning that one text query maps to up to ten videos. The event queries are selected by taking one to two sentences from each English event text document that describes the event most holistically. We exclusively use English queries for this section, as our annotators fluent in the other languages were not available for this task. In addition to MultiCLIP, we consider a set of contemporary video retrieval models with lightweight architectures (FrozenInTime [4], CLIP2Video [16], InternVideo [49], MPLUG [25], and a pooled CLIP model using the same setup as MultiCLIP without the additional multilingual pretraining). We argue that lightweight architectures are most appropriate for evaluating a full, pairwise set of similarity scores between text and video data of large multimodal corpora. Results are reported, partitioned on language, in Table 5.
We report the standard recall @ rank  $k$  metric for retrieval on individual video queries, and precision @ rank  $k$  for retrieval on event description queries. The results suggest that some existing video retrieval models may particularly struggle on this task, regardless of language. We hypothesize that this is due to a combination of the videos' length, complex semantic content, ambiguity, and frequent OCR content, as well as the long and often noisy video description queries.
While MultiVENTas a whole poses challenges to existing models, it is also clear that multilingual data may significantly impact performance on models trained primarily on English content - all models suffer a performance loss when evaluated on multilingual content (even when using English queries, as shown by the event description query results). While MultiCLIP suffers a performance loss on this data as well, comparing the standard pooled CLIP model against MultiCLIP shows that training on multilingual data does mitigate this multilingual performance loss: The two models perform comparably on English data, but MultiCLIP performs better on the multilingual content, especially when multilingual queries are used.
6 Limitations and Ethical Considerations
Data quality and demographic representation The events selected from Google Trends and Wikipedia were identified due to their popularity among speakers of the target language, but were identified by American English-speakers. Therefore, the domain of news items covered by MultiVENT may be slightly biased towards English-speaking audiences. Additionally, our dataset is limited to videos uploaded to primarily English-speaking video hosting forums, which limits its breadth by (1) excluding videos taken in countries with strict internet censorship laws and (2) skewing the video set towards those taken by people who choose English-centric websites over alternatives such as Bilibili or RuTube. We choose these websites because they are used globally and allow for multilingual data collection from a small set of sources. Then, by retrieving both keywords from Wikipedia and Google Trends for video retrieval, we aim to consider both official descriptions of current events as well as more colloquial terms across languages. However, this limits the videos retrieved to those associated with these terms, meaning that we omit relevant videos listed under different keywords. Finally, while we check the description, date of upload, and visual contents of videos included in the dataset, it is possible that human error led to a small number of mistrabeled or irrelevant videos being included in the dataset.
Table 5: Results showing the retrieval performance of video retrieval methods alongside MultiCLIP on MultiVENT. We use video descriptions and event descriptions as queries and partition results based on language. As shown, MultiVENT can be a difficult retrieval benchmark for video retrieval models even when considering only English, but the benefit of training on multilingual data is apparent when comparing MultiCLIP against the regular pooled CLIP model on non-English data.
Video description	Event description
Method	R@1	R@5	R@10	MedR	P@1	P@5	P@10
English
FrozenInTime [4]	6.5	20.0	28.4	53.0	42.3	34.6	26.9
MPLUG [25]	4.6	13.9	18.3	184.5	32.7	32.7	33.7
CLIP2Video [16]	41.3	71.8	80.4	2.0	96.2	96.9	73.3
InternVideo [49]	53.8	83.1	88.7	1.0	94.2	93.5	79.6
CLIP (pooled) [34]	55.9	83.9	91.3	1.0	98.1	94.6	80.6
MultiCLIP	55.9	84.5	92.3	1.0	100.0	96.9	80.6
Arabic + Chinese + Korean + Russian
FrozenInTime [4]	0.5	1.2	2.5	793.5	29.8	22.6	17.6
MPLUG [25]	4.4	12.2	16.4	315.0	10.6	10.4	10.5
CLIP2Video [16]	2.4	7.2	10.5	166.5	14.4	9.4	7.3
InternVideo [49]	5.7	13.9	19.8	91.0	79.3	71.0	55.7
CLIP (pooled) [34]	6.2	15.9	22.4	79.5	83.7	73.3	58.2
MultiCLIP	32.6	64.7	79.5	3.0	85.6	76.4	61.5
Privacy concerns The dataset consists of videos publicly uploaded to social media websites, but the videos are not directly available via the dataset download. We provide links to the videos which may be downloaded directly from their sources, and so these videos will automatically removed from the dataset if the video owners choose to revoke public visibility access. This may affect experiment replication if a video is removed from the dataset, but a small number of removed data points is unlikely to significantly impact experiment results.
Model misuse It is possible that models fine-tuned on MultiVENT without any safety or fairness methods considered may result in biased outputs or other fairness concerns. Therefore, we strongly urge researchers using this dataset as training data to employ ethical practices when training their models with this data.
Multilingularity and extensions Due to resource constraints, we focused on five diverse and widely-spoken languages that would have distinct spaces of current event coverage. However, including more languages would improve coverage, mitigate bias, and motivate more robust multilingual models. Using the methods described in Section 3, in the future the MultiVENT dataset can feasibly be scaled up to incorporate additional languages, events, and videos.
We introduce MultiVENT, a multimodal, multilingual dataset grounded in natural language documents for event-centric video retrieval and information acquisition consisting of 2,396 videos covering 260 current events in five target languages paired with multilingual text documents. We use this dataset to characterize online news coverage and how models can use this online content for information acquisition. We propose a multilingual video retrieval benchmark using MultiVENT and present MultiCLIP, multilingual video retrieval model to serve as a baseline for the task. We evaluate this model and related retrieval approaches on MSR-VTT and MultiVENT to illustrate the importance of pretraining on multilingual data for evaluation on MultiVENT. In future work, we aim to explore the effect that joint vision-OCR embeddings can have on video retrieval in text-heavy contexts. Additional tasks that could be explored for MultiVENT include cross-modal entity alignment, multilingual report generation, and multilingual video captioning. Also in future work, a PAQ-adjacent system [24] for automatically extracting question-answer pairs from video content and video-document pairs could be developed and applied to MultiVENT. Through this, a framework for teaching models to perform open-domain question-answering tasks with multimodal background corpora could be established, expanding the domain of questions a model can answer.
Table 6: All current events covered in MultiVENT, organized by language and event category. Each current event has up to ten videos and corresponding video descriptions, along with 1-2 full text documents.
A Current events
In Table 6 we list the full set of current events covered in MultiVENT.
Source	Arabic	Chinese	English	Korean	Russian
Mean tokens	123	114	95	108	170
Median tokens	25	53	38	53	54
Mean unique tokens	82	74	55	78	97
Total tokens	62820	60863	48072	56329	87493
Total unique tokens	17713	13824	6473	20820	19234
Num. videos w/o dialogue	9	13	15	20	0
Table 7: Audio information taken from Whisper transcripts of the MultiVENT video dataset. We partition statistics on language. As shown in the table, the dataset is rich in audio content and only approximately  $2\%$  of the dataset did not have detected spoken audio content.
FrameNet Template	Video Template
Disaster	Disaster
Duration	-
Place	Place
Responder	First responders
Response	-
Time	Time
Victim	Affected people
-	Disaster outcomes
Table 8: Original FrameNet disaster template compared against our revised template for video content.
B Audio content in MultiVENT
To assess the audio content present in the dataset, we pass the video set through OpenAI's Whisper model[35] to produce multilingual audio transcripts. We parse this data and present general statistics in Table 7. We use NLTK to tokenize all the transcripts except Chinese data which is tokenized with Jieba<sup>1</sup>.
C Disaster template modification
We modify the FrameNet "disaster scenario" template, which we use as a guide to annotate the videos in our Section 4 analysis. Specifically, we remove the "duration" and "response" fields from the FrameNet template, as duration is generally not depicted in video footage and the "first responders" field typically aligns with "response" content in video. We then add a "disaster outcomes" field as video content typically includes visually relevant depictions of the disaster aftermath. The difference between the two templates is shown in Table 8.
$^{1}$ https://github.com/fxsjy/jieba