MultiVENT: Multilingual Videos of Events and Aligned Natural Text
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: video retrieval
information retrieval
multimodal
dataset
multilingual
video
ocr
TL;DR: We introduce MultiVENT: A dataset of multilingual, event-centric videos grounded in natural language text documents covering five target languages.

Abstract: Everyday news coverage has shifted from traditional broadcasts towards a wide range of presentation formats such as first-hand, unedited video footage. Datasets that reflect the diverse array of multimodal, multilingual news sources available online could be used to teach models to benefit from this shift, but existing news video datasets focus on traditional news broadcasts produced for English-speaking audiences. We address this limitation by constructing MultiVENT, a dataset of multilingual, event-centric videos grounded in text documents across five target languages. MultiVENT includes both news broadcast videos and non-professional event footage, which we use to analyze the state of online news videos and how they can be leveraged to build robust, factually accurate models. Finally, we provide a  model for complex, multilingual video retrieval to serve as a baseline for information retrieval using MultiVENT.

MultiVENT:
Multilingual Videos of Events with Aligned Natural Text
Kate Sanders *
David Etter*
Reno Kriz*
Benjamin Van Durme
Johns Hopkins University
Human Language Technology Center of Excellence (ksande25, detter2, rkrizl, vandurme]@jhu.co edu
Abstract
Everyday news coverage has shifted from traditional broadcasts towards a wide range of presentation formats such as first-hand, unedited video footage. Datasets that reflect the diverse array of multimodal, multilingual news sources available online could be used to teach models to benefit from this shift, but existing news video datasets focus on traditional news broadcasts produced for English-speaking audiences. We address this limitation by constructing MultiVENT, dataset of multilingual, event-centric videos grounded text documents across five target languages. Multi VENT includes both news broadcast videos and non non-professional event footage, which we use to analyze the state online news videos and how they can be leveraged to build robust, factually accurate models. Finally, we provide a model for complex, multilingual video retrieval to serve as a baseline for information retrieval using MultiVENT.
1 Introduction
Information dissemination for current events has traditionally consisted of professionally collected and produced materials, leading large collections of well-writter news articles and high-quality videos. As a result, such materials form the basis for significant prior work in content analysis and
15 (50) Meanwhile, high volume of event-centric content today is generated by non-professionals, such as on-the-scene witnesses to events who hastily capture videos and upload them to the internet without further editing. We propose that this contemporary landscape of news content can be leveraged by models to produce a more comprehensive understanding of events. News agencies have adapted to this shift, often collecting and incorporating this online content into official broadcasts, but news video datasets still do not typically address this new domain of event coverage.
retrieval [5 20 2
In addition focusing traditional news sources, existing news video datasets predominantly consider content produced English. This is consistent with common practices in video dataset collection: Collected videos and captions recorded in English, and when multilinguality is Because this data is originally produced for English speaking audiences, these multilingual datasets contain unwanted content biases like "translationese" [6] [1] As event-centric video content produced in other languages makes up large portion of news videos online, we argue that including organic, multilingual content is necessary for a diverse and perspective-annoste sampling of event coverage.
considered, is achieved by directly translating captions and transcripts (48 22
40 19
With these ideas in mind, present MultiVENT, dataset of Multilingual Videos of Events with aligned Natural Text that contains 2,396 diverse, event-centric videos and text descriptions
Equal contribution.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

Query
للنسنت في افت لسطر رقكع الموممنية الملاتت B:le) عل 3 دلر الموويةة سن الاان ياب مات تمسن اتكلبيا مما وطانةة مغالل متها العور من حواتت
Pransfation After rounds of dinlogue sponsored by Baglidad and the Sultanate of Oman, China crowned it with the opening of n new door... The agrecment between Saudi Arabia and Iran to restore diplomatic rolations and open ombassics in the two countrics.
Documents
Grounding passage
Video
par))
26
Figure A sample video-text pair from VuITVVENT. Every event-centric video is paired with a corresponding video description and for-form text document describing the event, both in the same language as the video. the language is not English, the video is also paired with a corresponding English document.
that reflect the distribution of news content online. The videos are grounded in natural language video descriptions and long-form text documents, and the data spans 260 current events across over forty countries. The content in MultiVENT collected in five target languages: Arabic, Chinese, English, Korean, and Russian, and as the multilinguality is organic, the data is less likely to suffer from translation bias. We provide an illustration of the dataset's contents in Figure [ Each natural language query (describing video of current event) is paired with grounding text documents and a unique corresponding video. We use MultiVENT to explore and characterize the variety of event-centric videos available online and illustrate the importance of leveraging these different video types when building multimodal information systems.
Citizen journalism, the most notable example being Wikipedia [18] has emerged alongside other online news sources method for curating comprehensive summaries of events. Work in natural language processing has considered the problem of automating this process by training models to generate informative reports using online source materials (24 45 33). We use MultiVENT to explore how this process can be extended to incorporate multimodal sources of evidence. As first step this direction, we consider the task of video retrieval on MultiVENT, through which a model learns to retrieve multimodal source material given natural language event description. This task differs from prior video retrieval benchmarks [9 53 29 51 as the videos in MultiVENT vary widely in length and content presentation, are multilingual, and can involve significant amounts of on-screen text. In addition to multilingual natural language captions for each video, we provide full text documents that ground the events and serve more complex retrieval queries.
In summary, we address the lack of multilingual news video datasets that reflect the reality of our modern information landscape by proposing dataset of diverse web videos depicting current events across five languages. Enumerated, our contributions are:
1. We present MultVEET, a multimodal, multilingual information retrieval dataset of grounded videos depicting current events. The dataset targets five languages and covers a range of online video formats beyond traditional news broadcasts.
2. Using MultiVENT, we quantitatively illustrate the information presented by news videos and the differences in content between video formats, and qualitatively evaluate how multimodal coverage of an event can evolve over time.
3. We present MultiCLIP, model for multilingual, event-centric video retrieval that serves as a baseline for video retrieval approaches on the task.
2

---

2 Related Work
2.1 Video retrieval datasets
Early video datasets generally contained short clips spanning narrow ranges of topics, such as the Microsoft Research Video Description Corpus [2]. Video datasets spanning larger domains include MSR-VTT [53] and DiDeMo M. although the lengths of these videos were still relatively short. offered longer video lengths while still spanning wide range of topics such as news reports. shift towards massive video datasets was instigated by HowTo 100M [29]. which included over 130 million video clips belonging to one million narrated instructional videos. VaTeX 48 released in the same year, considered video retrieval from a multilingual context using caption translation. Additional multilingual video retrieval datasets include Rudder [13] consisting of instructional videos for making toys with multilingual captions, MTVR [22], which extended the TVR dataset [23] by adding Chinese subtitles and queries, and Multi- How
The V3C dataset 139 3
HoHTooooOOO To100M [19], which
extended HowTo 17000M by scraping Tubeube for subtitles in up to other languages. Recently, Chen et al. ( released the ChinaOpen dataset which contains wide range of video-caption pairs originally produced Chinese. Recent work has also considered the problem of interpreting text-heavy video content: Wu et al. [51] and Jahagirdar et al. [20) introduced datasets that focus on within-video text and OCR amnotations, including news broadcasts.
2.2 Video retrieval methods
The size of early video datasets allowed retrieval systems to rely on pre-extracted features from expert systems like action recognition models. As massive video datasets gained prominence, the video retrieval paradigm moved towards ad-hoe video-text feature extraction using large pretrained models. Dosovitskiy et al. [14] proposed using stand-alone transformer architectures for video understanding, and Bertasius et al. 77) showed that applying space- and time-based self-attention independently improved performance. Bain et al. applied findings directly to video retrieval, training and evaluating transformer architectures on WebVid-2NM 4 Radford et al. [34] introduced CLIP and showed that pretraining models to match captions to images can result in scalable models, and CLIP's applicability to video retrieval was demonstrated by Fang et al. [16] through their CLIP2Video model. More fine- grained modifications to CLIP were proposed. Wang al. 47 introduced "Object-aware Transformers" , which extended video-text transformers to incorporate object-level annotations within video footage, and Ge et al. [17 modified the pretraining task involve teaching a vision-text model to answer multiple choice questions about a video. Bain et al. [5] adapted large image-texi models to the task of long video retrieval by incorporating the weighted-mest of frame embeddings, and Wu et al. [51] incorporated independent optical character recognition and embeddings into the encoder pipeline to explicitly model in-video text.
2.3 Report generation using online sources
wide range of research has used online corpora for report generation tasks, including QA-pair generation [36] 30 (such as SQuAD (37 and HotPotQA (55) and knowledge graph generation [31]. Notably, Lewis al. 24 introduced method for automatically extracting question-anse answer pairs from large corpora of text documents, and applied this method to Wikipedia to produce the PAQ dataset. Some PAQ extensions have been multilingual - Pisare et [32] built the WikiOmnia QA dataset on Russian Wikipedia documents, and Rybak et al. [42] produced question- Diciiidn-iiipdiiiaa passage dataset in Polish. Recently, Qian et al. [33 extended the ideas in PAQ to construct WebBrain, task in which a model must generate factual articles with references given natural language query. In the multimodal domain, Reddy et al. and Chen et al. have considered the problem of open-domain QA for image-text data 38 [0], with Chen al. using Wikipedia generate multimodal dataset. In a similar vein, Li et al. propose a dataset for information extraction from multimedia articles 27 and extraction approach that can be used with text, image, and video content (26).
3 Dataset
In this section we outline the MultiVENT collection process. We draw from Davidsonian event semantics (28) to define event related set of particulars existing at a shared point in space
3

---

and time. We add the additional condition that an event must also be "newsworthy" in that it must have been reported on by respected news organizations. The dataset includes 2,396 videos and corresponding text descriptions covering 260 events grounded in 468 text documents, and includes content in Arabie, Chinese, English, Korean, and Russian. The average length per video over one minute (83.7 seconds) and the full dataset contains over 55 hours of video content. We first identify 260 visually salient current events spanning from 2013 to 2023, and assign a target language to each event. Then, for each event, we collect grounding text documents and set of videos in the event's s target language.
3.1 Current event curation
We consider four primary event categories for MultiVENT: Disasters, political events, social events, and technology events. We include thirteen current events per category for each target language. We use Google Trends statistics to select these events, based on its tracking of term popularity based on internet activity by country. We construct lists of the top five countries with the most speakers of each target language and review the top trending topics on Google in each of these countries over the last ten years. We record topics and search phrases that corresponded to current events that (I) align with one of the predefined event categories and (2) have sufficient online video coverage. For categories that did not amass a sufficient list of current events per language through this process, we consult Wikipedia's 's yearly summaries of events to fill the remaining slots. Detailed statistics characterizing this set of current events are shown in Figure 2 As shown, the majority of selected events take place in the last few years, with only three taking place before 2016.
Also shown in Figure 2 there is not a bijective mapping between the language used in event coverage and the country the event took place in. The language and country are often related, e.g., Russian news content in MultiVENT predominantly takes place in Russia, but this is not true of all events in the dataset. For example, we include data in Chinese pertaining to the 2023 ATP tennis circuit in Dallas, Texas: At this event, tennis player Wu Yibing became the highest-ranked Chinese player in the history of the ATP rankings, and so the event received substantial Chinese news coverage. In cases such this, news in multiple languages will heavily focus the same current event, such as sports events and international political relations. We do not include the same event in multiple languages in MultiVENT by design, in contrast with data collection procedures used for efforts such as AIDA 46 which aim to cover a small collection of current events in many languages.
Every current event in the dataset is grounded in an English natural language document and, if the event is tagged with a non- non-English language, an additional natural language document in that target language. First, check if full English Wikipedia article exists for the current event. If not, we manually find a Wikipedia article that includes a passage describing the event. If Wikipedia does not have a passage that appropriately grounds the event, then a news article in English is selected as a grounding document instead. This process is then repeated for the target language. The dataset includes 468 grounding articles in total: 313 are full Wikipedia articles, 104 are Wikipedia passages, and 51 are external articles.
3.2 Video collection
We aim to collect visually and semantically distinct videos for each current event with an even split between firsthand witness accounts (e.g.. first-person smartphone videos), amateur edited videos (e.g., vlogs), and professional news reports and compilations. Information regarding the resultant distribution of these categories and their semantic differences is included in Section 4.2 For each current event, we collect ten videos in the current event's target language. We search YouTube and Twitter for these videos using target keywords collected from the Google Trends search and Wikipedia. We check the upload date to ensure that it aligns with the time of the event's occurrence. Finally, we skim through the video content confirm that is relevant to the target event. After collecting the videos, we manually identify and remove duplicates, resulting in 2,396 videos in total. We do not include repeat videos, but sometimes professional news reports include firsthand footage that is already included as unedited footage in the dataset. In these cases, we keep both the news report and the original footage as the context and text metadata between the two are distinct. If the video has natural language description, we tag the video with this description. If it does not, we use the video title as the tagged natural language description. We report the distribution of videos by source in Figure 2

---

A2
Santi
Ambis
South Kenas
Totwon
# UK US
Ukzaate
CAA
Internniinaal
Orber
5015 2016 2017
2018
2019
2020
2021
2022
2023
YouTube Twitter YouTuhe Shors Other
Figure 2: A: Statistics illustrating the distribution of current events selected for the dataset. (A1) depicts the general breakdown of countries which each current event takes place. Many countries had single current event, particularly small countries in the middle cast and southeast Asia, and are consolidated into the "other" category for easier graph interpretation. (A2) shows the distribution of years during which the current events take place. B: Breakdown of data sources for the videos in the dataset. The majority of videos came from YouTube existed longer than You Tube Shorts.
has a larger international audience and has
4
Data analysis
We present analysis of MultiVENT to help characterize how online multimodal content contributes to our understanding of current events. We explore multimodal event coverage from three angles: (1) what kinds of information news videos contribute, (2) the differences in informative content provided by different types of news videos, and (3) how multimodal coverage of an event can evolve over time.
4.1 Semantic information in video
Visual data provide rich, semantically nuanced details of event that not captured in text documents due to reporting bias, limitations of text, and document length limits. To characterize the complexity of these videos and the information they provide, annotate of two hundred videos of disasters in MultiVENT to identify visual entities the videos that help answer common "who, what, where" -teee questions about the events they depict.
We present videos disaster footage to local annotators and provide them with a set of event-centric questions derived from FrameNet's "disaster scenario" template [41]. We modify this template, designed for text, to sufficiently cover visual content (further information included in the appendix). We instruct annotators to identify every on-screen entity (such people, scrolling news headline banners, etc.) that might help answer one of these event-centric questions. These annotators are professionals with experience in data analysis and compensated appropriately for their work. The annotations are for analysis only and are not part of the main dataset, and no personal information captured.
The template divides salient entities into six categories: The disaster itself ("what"), the location of the disaster ("where"), the time the disaster takes place ("when"), people affected by the disaster ("who") and first responders for the disaster, e.g., firefighters (also "who"), and any visible outcomes of the disaster. Not every category applies to both visual content and text: We exclude "where" and "when" from the set of categorics that visual content should be annotated for (because identifiable depictions of "where" outcomes from the set text annotation categories, as textual examples of this category tend to involve full clauses, which complicate the annotation process. We note that, while exclude visual "where" content, different video types are likely to contain different amounts of visual information informing viewer of where the event takes place. We omit this analysis for simplicity but do not model this facet of the videos' semantic richness by doing so.
present in almost every frame, and "when" in virtually none) and disaster
We present the number of event-relevant entities that appear on-screen in these annotated videos Table [ For each annotated entity, we additionally ask annotators to rate their certainty that the entity is directly related to the event described by the video's 's natural language description from 0%
5

---

Table I: Mean number of visual entities and in-scene text references (written text displayed within video) present per video in subset of 210 disaster videos from the current events dataset. We omit "where" and "when" entities from the visual content counts as "where" visual content technically appears in every frame and there are few types of visual evidence for "when" questions. We omit "outcomes" from the text references in text (this field omitted from the FrameNet event template analogue for text documents).
outcome by itself is full event that is difficult localize
Visual entities
Text references
Total
Disaster ("What")
1.25
1.37
2.62
Place of occurrence ("Where")
1.54
1.54
Time of occurrence ("When")
0.77
0.77
Affected people ("Who") First responders ("Who")
1.22
0.54
1.76
1.13
0.50
1.63
Disaster outcomes
1.00
1.00
Total
4.60
4.72
9.32
Table 2: Mean annotator certainty scores partitioned on entity type based on the annotations used for
Table [1 0.20 certainty indicates that the annotator is 20% sure that the annotated entity helps answer the tagged question about the described event, while 1.00 certainty indicates that the annotator is completely sure that the entity helps answer the tagged question about the event.
Disaster
Where
When
AP
FR
Outcomes
All
Visual content
.787
.716
.765
.798
.830
Text content
.931
.907
.929
.856
.836
.900
Average
.862
.907
.929
.759
.787
.798
.865
to 100%. We record these certainty scores in 20% intervals, i.c. 20%, 40%, 60%. 80%, 100%. The averages the linguists' confidence rankings by entity type are listed in Table
As shown in Table each video contains an average of 9.32 informative visual entities that pertain to the event in question. About half of these entities are purely visual, and half are within-vides text that can be identified with an optical character recognition model. As indicated by Table purely visual entities are more ambiguous than the text content shown onscreen alongside it, which aligns with past research that explores the difficulty humans have in interpreting visual content depicting complex events [33.
4.2 Video content by domain
As described Section 3 we collect three main types of videos: Official news broadcasts, edited video footage, and raw, unedited footage. Of the 210 videos in the annotation set reported in Table [. 53% are news broadcasts, 11% are edited footage, and 36% are raw footage. To quantify the difference in information presented by these different video types, we take the video annotations shown in Table I and partition these annotations by video type. We present the results in Table
As shown by the results, news broadcasts depict the most relevant semantic information, followed by edited footage. This is particularly apparent when considering text content alone. On average, news coverage contains almost 9 times as much relevant on-screen text content than raw footage, and over three times more than edited footage. Visual content differences were less drastic, but news content still had two times more visual content than raw footage and 1.3 times more than edited footage. The difference in visual content between news coverage and edited footage possibly due to average video length and the quality of the video curation oftentimes, unprofessionally edited footage only draws from one source whereas news coverage draws from many.
4.3 Information evolution
As shown Table first-person footage is often opaque compared to professional coverage. However, comprehensive coverage often builds on earlier, less informative coverage. This can be seen in news

---

Table 3: Mean number of visual entities and in-scene text references present per video, partitioned on
video type. Same 210 video subset used for analysis as that used for the analysis shown in Table
News coverage
Edited footage
Raw footage
Vis.
Text
Total
Vis.
Text
Total
Vis.
Text
Total
Disaster ("What")
1,42
2.38
3.80
1.14
0.41
1.55
1.05
0.17
1.22
Place ("Where")
2.51
2.51
0.59
0.59
0.39
0.39
Time ("When")
1.26
1.26
0.41
0.41
0.16
0.16
Affected people ("Who")
1.48
0.94
2.42
1.18
0.23
1.41
0.86
0.04
0.90
First responders ("Who")
1.73
0.78
2.51
1.36
0.45
1.81
0.17
0.12
0.29
Disaster outcomes
1.28
1.28
0.77
0.27
1.04
0.67
0.67
Total
5.91
7.87
13.78
4.45
2.36
6.81
2.75
0.88
3.63
CCCCNees
BCBS News
Firofightors struglinng
The blans Notre Dame broke
Mliaml d
Voor
Dame Cathedral Parts on
musivv
undergoing million
while cathedral
engulfal flmmes and famed
Moaday, The lastriic cburch was
renovation
brigado says work is "potentially linked" to the fire
spire collapaed
@H Miser
It's failing
10:02 AM
10:51 AM
12:20 PM
6:20 PM
10:38 AM
11:10 AM
12:26 PM
@CCSNeess
CCBNNews
Renters
BREAKING: Paris" hisorii Notre
Wrzees says 300-foot spire top the Notre Dome Cathedral has just cillepsed
Crowds survand Notre-Deme Cuthala spire cellogaes
Figure 3: Snapshot of video news coverage of the 2019 Notre Dame fire news cycle (an event in MultiEENT, The fire and the fallen spire were initially reported through first-person social media video uploads (at 10:02 AM and 10:51 AM, respectively) and then later broadcast by news organizations in more detail (10:38 AM, 11:10 AM, 12:20 PM, 12:26 PM). Some (12:20 PM) directly used first first-person social media footage (10:51 AM). Hours later, news agencies uploaded complete
coverage
stories with details and context (6:20 PM). This data suggests that important for models to learn from both first-person videos and official news coverage at various points in the news cycle fully construct factual model of the event, especially if the model is attempting to construct an event model while information develops online.
cycles for slowly unfolding events and for sudden, unexpected events that take time to assess. This is illustrated in Figure 3 which shows a snapshot of the 2019 Notre Dame fire news cycle and demonstrates how unedited and poorly curated footage, often first-person witness accounts on social media, can be instrumental in the construction of our collective understanding of events. So, we propose that teaching models to understand different video formats, despite clear discrepancies the amount of information they present, is important for developing robust systems.

---

Experiments
5.1 Approach
We consider the problem of teaching a model to map multilingual, natural language queries to multilingual video clips. Specifically, consider video set V and query set T with indicator mapping function f that returns whether query t E T describes a video v V. The model is V returns
provided with the full set of videos V and a text query t T, and for each video the probability that t describes U, or /12, t) = P[f(u, t) = 1]. When there a bijective mapping between queries and videos (e.g., when using video descriptions queries), the model is evaluated on its recall when considering the top 1, 5, and 10 ranked videos (R@ 1, R R@S, and R@ 10), as well as the median rank (MedR). When given query may describe multiple videos, (e.g., when using event descriptions as queries), we instead evaluate the model on its precision given the top 1, 5, and 10 ranked videos (P@) 1, P@5, and P@ 10). We define these metrics
Given S
arg max VEV. BEV"
Σ h(u,
| ( [(s,() 1}1 E V : f(o,t) = 1T
and P@k 1s ES f(s,1) 1}
R@k
5.2 Model architecture and training
We introduce MultiCLIP, multilingual baseline for video retrieval on MultiVENT. This model adopts a CLIP architecture with a multilingual text encoder and trained on multilingual text-video pairs. In more detail:
We base our architecture on the pretrained LAION CLIP VIT-H/I4 frozen XLMM-Robrrt--Large model [1I. which jointly trains an image and text encoder on text-image data to learn to pair images with their captions. At test time, it produces a zero-shot linear layer based on the test input's visual features through which natural language captions be passed in. The model architecture contains a vision encoder based on a ViT architecture [14] and a text encoder based on the the multilingual XLM Roberta large model [1]. A full overview of the CLIP architecture and pretraining can be
found in the original paper 34
In experiments using MultiCLIP, we first tokenize text descriptions using the tokenizer, containing vocabulary of over 250,000 words, and pass the tokens into MultiCLIP which produces text embedding of size 1024. Next, we uniformly sample videos at rate of 12 frames video with an input size of 224x224, which the model uses to create a frame embedding of size 1024. To incorporate multilinguality into the model's frame-level features, we use ViT architecture trained with contrastive objective over multilingual image-caption pairs from the LAION-5B dataset 44 which is constructed from the Common Crawl archive using images and their alt-text produce multilingual image-text dataset with over 100 languages. We mean pool the frame embeddings to produce final video embedding, and use the text and video features to compute similarity matrix of videos and descriptions.
5.3 Retrieval baselines
We first evaluate MultiCLIP on the existing video retrieval task MSR-VTT
533 using the recall metrics described in Sec. 5.1 alongside contemporary video retrieval approaches (FrozenInTime [4] Ciip2Video [16), Intern Video 49 and MPLUG-2 (52). Results on MSR- STT's validation set are reported in Table 4 The results indicate MultiCLIP performs well on standard video retrieval tasks, matching performance of separate text/vision pipeline models released within the last two years. performs better than existing models that use separate text and vision pipelines (FrozenInTime 4 and Clip2Video [16), but not as well as models that use larger architectures involving multimodal encodings (Intern Video
49 and MPLUG-2 (52)..

---

Table 4: MultiCLIP evaluated alongside existing video retrieval approaches the video re- trieval benchmark MSR-VTT. Results indicate that MultiCLIP performs adequately on existing retrieval tasks, achieving comparable results to existing models. It does not perform well as architectures that use multimodal transformers for joint encodings such as Intern Video and MPLUG-2.
Method 4
Year
Rank @ 1
Rank @5
Rank @ 10
FrozenInTiime
2021
32.5
61.5
71.2
Vidp2 [6
2021
29.8
55.5
66.2
Video [49
2022
55.2
79.6
87.5
MPLUG-2 [52]
2023
53.1
77.6
84.7
MultiCLIP
2023
38.4
70.1
82.7
5.4 MultiVENT retrieval
We now evaluate MultiCLIP and related retrieval approaches on MultiVENT. We first use multilingual video descriptions queries, and then we use English event summaries taken from the grounding text documents, meaning that one text query maps to up to ten videos. The event queries are selected by taking one to two sentences from each English event text document that describes the event most holistically. We exclusively use English queries for this section, as our annotators fluent in the other languages were not available for this task. In addition to MultiCLIP, we consider set of contemporary video retrieval models with lightweight architectures (FrozenInTime [4], CLIP2Video [16), Intern Intern [49]. MPLUG [25], and pooled CLIP model using the same setup MultiCLIP without the additional multilingual pretraining). We argue that lightweight architectures are most appropriate for evaluating full, pairwise set of similarity scores between text and video data of large multimodal corpora. Results are reported, partitioned on language, in Table
We report the standard recall @ rank l metric for retrieval on individual video queries, and precision rank & for retrieval on event description queries. The results suggest that some existing video retrieval models may particularly struggle on this task, regardless of language. We hypothesize that this is due to a combination the videos' length, complex semantic content, ambiguity, and frequent OCR content, well as the long and often noisy video description queries.
While Multi VENTS whole poses challenges existing models, it also clear that multilingual data may significantly impact performance on models trained primarily on English content all models suffer performance loss when evaluated on multilingual content (even when using English queries, as shown by the event description query results). While MultiCLIP suffers performance loss on this data as well, comparing the standard pooled CLIP model against MultiCLIP shows that training multilingual data does mitigate this multilingual performance loss: The two models perform comparably on English data, but MultiCLIPP performs better on the multilingual content, especially when multilingual queries are used.
Limitations and Ethical Considerations
Data quality and demographic representation The events selected from Google Trends and Wikipedia were identified due to their popularity among speakers of the target language, but were identified by American English-speakers. Therefore, the domain of news items covered by Mul- tiVENT may be slightly biased towards English-speaking audiences. Additionally, our dataset is limited to videos uploaded to primarily English-speaking video hosting forums, which limits its breadth by (1) excluding videos taken in countries with strict internet censorship laws and (2) skewing the video set towards those taken by people who choose English-centric websites over alternatives such Bilibili or RuTube. We choose these websites because they are used globally and allow for multilingual data collection from small set of sources. Then, by retrieving both keywords from Wikipedia and Google Trends for video retrieval, we aim to consider both official descriptions of current events as well as more colloquial terms across languages. However, this limits the videos retrieved to those associated with these terms, meaning that we omit relevant videos listed under different keywords. Finally, while we check the description, date of upload, and visual contents of videos included in the dataset, it possible that human error led to a small number of mislabeled or irrelevant videos being included in the datasct.

---

Table 5: Results showing the retrieval performance of video retrieval methods alongside MultiCLIP on MultiVENT. We use video deseriptions and event descriptions as queries and partition results based on language. As shown, MultiVENT can be difficult retrieval benchmark for video retrieval models even when considering only English, but the benefit of training on multilingual data is apparent when comparing MultiCI.IP against the regular pooled CLIPP model non-English data.
Video description
Event description
Method
@
R@5
R@ TO
MedR
P@T
P@5
P@
English
TimeeninTime
4
6.5
20.0
28.4
53.0
42.3
34,6
26.9
MPLUG 25
4.6
13.9
18.3
184.5
32.7
32.7
33.7
CLiP2Video
16
41.3
71.8
80.4
2.0
96.2
96.9
73.3
Intern InternVideo
449
53.8
83.1
88.7
1.0
94.2
93.5
79.6
CLIP (pooled)
34
55.9
83.9
91.3
1.0
98.1
94.6
80.6
MultiCLII
55.9
84.5
92.3
1.0
100.0
96.9
80.6
Arabic + Chinese + Korcan Russian
Frozenin Time
0.5
1.2
2.5
793.5
29.8
22.6
17.6
MPLUG 25
4.4
12.2
16.4
315.0
10.6
10.4
10.5
CLiP2Video
16
2.4
7.2
10.5
166.5
14.4
9.4
7.3
Intern
49
5.7
13.9
19.8
91.0
79.3
71.0
55.7
CLIP (pooled)
34
6.2
15.9
22.4
79.5
83.7
73.3
58.2
MultiCLIP
32.6
64.7
79.5
3.0
85.6
76.4
61.5
Privacy concerns The dataset consists of videos publicly uploaded to social media websites, but the videos directly available via the dataset download. We provide links the videos which may be downloaded directly from their sources, and so these videos will automatically removed from the dataset if the video owners choose to revoke public visibility access. This may affect experiment replication a video is removed from the dataset, but a small number of removed data points is unlikely to significantly impact experiment results.
Model misuse It is possible that models fine-tuned on MultiVENT without any safety or fairness methods considered may result in biased outputs other fairness concerns. Therefore, we strongly urge researchers using this dataset as training data to employ ethical practices when training their models with this data.
Multilingulaity and extensions Due to resource constraints, we focused on five diverse and widely- spoken languages that would have distinct spaces of current event coverage. However, including more languages would improve coverage, mitigate bias, and motivate more robust multilingual models. Using the methods described in Section 3, in the future the MultVEETT dataset can feasibly be scaled up to incorporate additional languages, events, and videos.
7 Conclusions and Future Work
We introduce MultiVENT, a multimodal, multilingual dataset grounded in natural language doc- uments for event-centric video retrieval and information acquisition consisting of 2,396 videos covering 260 current events in five target languages paired with multilingual text documents. We this dataset to characterize online news coverage and how models information acquisition. We propose a multilingual video retrieval benchmark using MultiVENT and present MultiCLIP, multilingual video retrieval model serve as baseline for the task. We evaluate this model and related retrieval approaches on MSR-VT VTT and MultiVENT to illustrate the importance of pretraining on multilingual data for evaluation on MultiVENT. In future work, we aim to explore the effect that joint vision-OCR embeddings can have video retrieval in text- heavy contexts. Additional tasks that could be explored for MultiVENT include cross-modal entity alignment, multilingual report generation, and multilingual video captioning. Also in future work, PAQ-adjacent system (24] for automatically extracting question-answer pairs from video content and video-document pairs could be developed and applied to MultVVENT. Through this, framework for teaching models to perform open-domain question-answerin tasks with multimodal background corpora could be established, expanding the domain of questions model can answer.
this online content for
10

---

References
[1] Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pages 5803 5812, 2017.
[2] André Araujo, Jason Chaves, David Chen, Roland Angst, and Bernd Girod. Stanford i2v: a news video dataset for query-by-image experiments. In Proceedings of the 6th ACM Multimedia Systems Conference, pages 237-242, 2015.
[3] George Awad, Asad A Butt, Keith Curtis, Yooyoung Lee, Jonathan Fiscus, Afzal Godil, Andrew Delgado, Jesse Zhang, Eliot Godard, Lukas Diduch, et al. Trecvid 2019: An evaluation campaign benchmark video activity detection, video captioning and matching, and video search & retrieval. arXiv preprint WwXiv:200.00996 2020.
[4] Max Bain, Arsha Nagrani, Güil Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728-1738, 2021.
[5] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. A clip-hitchhiker's guide to long vidco retrieval. arXiv preprint arXiv:2205.085555 2022.
[6] Marco Baroni and Silvia Bemardini. A new approach the study of translationese: Machine-learning the difference between original and translated text. Literary and Linguistic Computing. 21(3):259 274, 2006.
[7] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, volume 2. page 4, 2021.
[8] Aozhu Chen, Ziyuan Wang, Chengho Dong, Kaibin Tian, Ruixiang Zhao, Xun Liang, Zhanhui Kang, and Xirong Li. Chinaopen: A dataset for open-world multimodal learning, arXiv preprint arXiv:2305.05880 2023.
[9] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190-200, 2011.
[10] Wenhu Chen, Hexiang Hu, Xi Chen, Pat Verga, and William W Cohen. Murag: Multimodal retrieval- augmented generator for open question answering images and text. arXiv preprint ar arX1v:2210.02928, 2022.
[11] Mchdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. arXiv preprint arXiv:2212.0711 2022.
[12] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer. and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv prepriat arXiv:1911.00 19111111111.... 2019.
[13] Rishabh Dabral. Ganesh Ramakrishnan, Preethi Iyothi, et al. Rudder: A cross lingual video and text retrieval dataset. arXiv preprint arXiv:2103.055 05555557......... 2021.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 19999, 2020.
[15] Joseph G Ellis, Brendan Jou, and Shih-Fu Chang. Why we watch the news: a dataset for exploring sentiment in broadcast video news. In Proceedings of the 16th international conference mudtimodal interaction, pages 104-111, 2014.
[16] Han Fang, Pengfei Xiong, Luhui Xu, and Yu Chen. Clip2video: Mastering video-text retrieval via image clip, arXiv preprint arXi:: 206. 1007, 2021.
[17] Yuying Ge, Yixiao Ge, Xihui Liu, Dian Li, Ying Shan, Xiaohu Qie, and Ping Luo. Bridging video-text retrieval with multiple choice questions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 16167-16176, 2022.
[18] Ruediger Glott, Philipp Schmidt, and Rishab Ghosh. Wikipedia survey-overview of results. United Nations Universiry: Collaborative Creativity Group 8:1 8:1158-1178, 2010.
[19] Po- Yao Huang, Mandela Patrick, Junjic Hu, Graham Neubig, Florian Metze, and Alexander Hauptmann. Multilingual multimodal pre-training for zero-shot cross-lingual transfer of vision-language models. arXiv preprint 110::200300884: 2021.
[20] Soumya Jahagirdar, Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Watching the news: Towards videoqa models that can read. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4441 14504 2023.

---

[21] Moshe Koppel and Noam Ordan. Translationese and its dialects. In Proceedings of the 49th annual meeting of the asxociation for computational linguistics: Human language technologies, pages 1318-1326, 2011.
[22] Jie Lei, Tamara L. Berg, and Mohit Bansal, mivr: Multilingual moment retrieval in videos. arXiv preprint arXiv:2108,000 00060, 2021.
[23] Jie Lei, Licheng Yu, Tamara I. Berg, and Mohit Bansal, Tvr: A large-scale dataset for video-subtitle moment retrieval. In Computer Vision-ECCC 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXI 16, pages 447-463. Springer, 2020.
[24] Patrick Lewis, Yuxiang Wu, Linqing Liu, Pasquale Minervini, Heinrich Küttler, Aleksandra Piktus, Pontus Stenetorp, and Sebastian Riedel. Paq: 65 million probably-asked questions and what you can do with them. Transactions of the Association for Computational Linguistics, 9:1098-1115 11, 2021.
[25] Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hchong Chen, Guohai Xu, Zheng Cao, et al. mplug: Effective and efficient vision-language learning by cross-modal skip-connections. arXiv preprint arXiv:2205. /2005, 2022.
[26] Manling Li, Alireza Zareian. Ying Lin, Xiaoman Pan, Spencer Whitehead, Brian Chen, Bo Wu, Heng
Ji, Shih-Fu Chang, Clare Voss, ct al. Gaia: A fine-grained multimedia knowledge extraction system. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 77-86, 2020.
[27] Manling Li, Alireza Zareian, Qi Zeng, Spencer Whitehead, Di Lu, Heng Ji, and Shih-Fu Chang, Cross- media structured common space for multimedia event extraction. arXiv preprint arXiv:2005.02472. 2020.
[28] Claudia Maienborn. Event semantics. by C. Maienborn, K. von Heusinger. P. Portner.-Berte pages 232-266, 2011.
[29] Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac. Makaaand Tapaswi. Ivan Laptev, and Josef Sivic. Howtol nowto100m, 00m: Learning a text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 2630-2640, 2019,
[30] Xiangyang Mou, Chenghao Yang. Mo Yu, Bingsheng Yao, Xiaoxiao Guo, Saloni Potdar, and Hui Su. Narrative question answering with cutting- open-domain qa techniques: A comprehensive study. Transactions of the Association for Computational Linguistics, 9:1032- 1046, 2021.
[31] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne. Yacine Jernite. Vladimir Karpukhin, Jean Maillard, et al. Kilt: benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009. 02052,,022..2.2 2020.
[32] Dina Pisarevskaya and Tatiana Shavrina. Wikiomnia: generative qa corpus on the whole russian wikipedia. arXiv preprint 2xXiv2204.08000 2022.
[33] Hongjing Qian, Yutao 7hu, Zhicheng Dou, Haoqi Gu, Xinyu Zhang, Zheng Liu, Ruofei Lai, Zhao Cao, Jian-yun Nie, and Ji-Rong Wen. Webhrain: Learning to generate factually correct articles for queries by 2023.
grounding on large web corpus. arXiv preprint arXiv:2304.0433
[34] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[35] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. arxiy. arXiv preprint 2rXi::22/2.04356 2022.
[36] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't ton't know: Unanswerable questions for squad. arXiv preprint arXiv: 1806.03822. 2018
[37] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arxiv preprint arxiv 052500. 2016.
[38] Revant Gangi Reddy. Xilin Rui, Manling Li, Xudong Lin, Haoyang Wen, Jaemin Cho, Lifu Huang, Mohit Bansal, Avirup Sil. Shih-Fu Chang, et al. Mumuqa: Multimedia multi-ho hop news question answering via cross-medin knowledge extraction and grounding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 1 1200-1 1208, 2022.
[39] Luca Rossetto, Heiko Schuldt, George Awad, and Asad A Butt. V3c-a research video collection. In MultiMedia Modeling: 25th International Conference, MMM 2019, Thessaloniki, Greece, January 11, 2019, Proceedings, Part 1 25, pages 349-360. Springer, 2019.
[40] Andrew Rouditchenko, Yung- Sung Chuang, Nina Shvetsova, Samuel Thomas, Rogerio Feris, Brian Kingsbury, Leonid Karlinsky, David Harwath, Hilde Kuehne, and James Glass. C2kd: Cross-lingual cross cross-modal modal knowledge distillation for multilingual text-video retrieval. In ICASSP 2023-2023 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 1-5, IEEE, 2023.
12

---

[41] Josef' Ruppenhofer, Michael Ellsworth, Myriam Schwarzer-Petruee Petruck, Christopher R Johnson, and Jan Scheffezyk. Framenet ii: Extended theory and practice. Technical report, International Computer Science Institute, 2016.
[42] Piotr Rybak. Maupqa: Massive automatically-create polish question answering dataset. arXiv preprint arXiv:2305.00 005.0 2023.
[43] Kate Sanders, Reno Kriz, Anqi Liu, and Benjamin Van Durme. Ambiguous images with human judgments for robust visual event classification. In Thirry-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[44] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-Sb: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08400 2022.
[45] Christopher Sciavolino, Zexuan Zhong, Jinhyuk I Lee, and Danqi Chen. Simple entity-centric questions challenge dense retrievers, arXiv preprint arXiv:2 109.0853, 2021.
[46] Jennifer Tracey, Ann Bies, Jeremy Getman, Kira Griffitt, and Stephanie Strassel. A study in contradiction: Data and annotation for aida focusing on informational conffict in russia-ukraina relations. Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1831-1838, 2022.
[47] Jinpeng Wang. Yixiao Ge, Guanyu Cai, Rui Yan, Xudong Lin, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Object-aware video-language pre pre-training for retrieval. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3313-3322 2022.
[48] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang. Vatex: A large-scale, high-quality multilingual dataset for vidco-and-language research. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581 1581-40 14591, 2019.
[49] Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang. Zhiyu Zhao, Hongjie Zhang. Jilan Xu, Yi Liu, Zun Wang, al. Internvideo: General video foundation models via generative and discriminative learning. arXiv preprint wwXiv:2212.03//2 2022.
[50] Haoqian Wu, Keyu Chen, Haozhe Liu, Mingehen Zhuge, Bing Li, Ruizhi Qiao, Xiujun Shu, Bei Gan, Liangsheng Xu, Bo Ren, et al. Newsnet: A novel dataset for hierarchical temporal segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10669- 10680, 2023.
[51] Weijia Wu, Yuzhong Zhao, Zhuang Li. Jiahong Li, Hong Zhou, Mike Zheng Shou, and Xiang Bai. A large cross modal video retrieval dataset with reading comprehension. arXiv preprint arXiv:2305. 03347, 2023.
[52] Haiyang Xu, Qinghao Ye, Ming Yan, Yaya Shi. Jiabo Ye. Yuanhong Xu, Chenliang Li, Bin Bi, Qi Qian, Wei Wang, al. mplug-22 2: A modularized multi-modal foundation model across text, image and video. arXiv preprint 2r3iv:2.302. 00402, 2023.
[53] Jun Xu, Tao Mci, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5288-5296, 2016.
[54] Hui Yang, Lekha Chaisom, Yunlong Zhao, Shi-Yong Neo, and Tat-Seng Chua. Videoqa: question answering on news video. In Proceedings of the eleventh ACM international conference Mulrimedia, pages 632-641, 2003.
[55] Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpolqa: A dataset for diverse, explainable multi-hop question answering. arXiv preprint Xiv: 809,09600, 2018
13

---

Desaster
Social
Polncal
TechnologySScennccoo
Arabic
2020 Beant Explosion
2022 Dubai World Cup
2022 Sudan Sanctoos
2018 Hashish Sarcophagus Discovery
2022 Luszil Fire
2023 World Art Dubai
2020 Egypp Protests
2023 DIFC Metaverse Platfoom
2123 Saudi Arabia Flodss
2012 GITEX
2016 Tukey Coup 1'Etur Attempe
2018 Duhai Police Hike Demo
2022 Vielenee Kabul
2020 Dubai EXPO
2021 Palestine Ciisis
Sagurra Tomb Discovery
2022 Cypraa Fanthqukk
2018 Ramasdan
2021 Traqi Political Crisia
20251 1.FAP Conference
2012 Trag Dust Storm
2021 Saudi National Day
2017 Mann Al-Staca Anest
2017 NEOM Project Ammonsemmnn
2019 Ataba Fire
2025 Luubau Marathon
2023 SA Iran Relations Restocation
2022 Duhai Metaverse Forum
2016 EgyycAir
Crash
2021 Riyadh Senson
2022 Pakiitanii PM Removal
2013 SpaceX
Mission
2021 Soling Train Collision
AFCON
2023-23 Antini Protests
2019 FIRST Globai Challaage
2021 Cairo Factory Fize
2019 Cairo Him Festival
2018-19 Gaza Barder Protests
2013 Saudi Arabua MS Dutaeeater
2021 Cyclone Slasheen
2022 Shelkh Zayed Heringe Fostivll
2023 Trat-Irsq Security Deal
2022 Sundi Arahio Gilohil Af Summit
2023 Jeddal Floods
2019 Arahiza Gull Cup
King Abdullah Death
Breach
2020 Riywlh Drone Alback
2023 FIFA World Oup
2019 Bon Gurrion Airport Protest
2003 Kuwsit Saellite
Chinese
2021 Gorges Flood
2022 Olympics
2002 Hong Kong Election
Crung EE 4 LLanding
2018 Huilien Fartiquake
2021 "Solmon Chass"
3022 Revolution
2021 Raby Yingliang Disoweery
2021 Quuia Gold Mine Accident
2022 China Air Show
2022 Foxcour Factory Protests
2019 Bytedince Lawsuit
2014 Typhovn Rammasun
2021 Taippi Rose Festival
2019 Hong Koeg Extraliiin
2020 Changg 51 STanding
2022 Qinglai
2021 Universal Studios Reijing Opening
Spy Rallon
2023 Hunwel 5.5G
2022 Luding Eartiquakk
2025 Taiwan Lanten
2023 Hoaltheuse Reform Protests
2022 Google Bard Ansoneement
2019 Typhoon Lekima
2022 Macon Grand Pris
2020 Taiwan Election
2021 Pereonal Information Law
2020 Anshum Bus Crash
2018 Shonzhen Open
2023 Li Qiang Elected
2018 Jack Mr Alibaha Aldicaaton
2015 TTanjan Explusion
Lanar Year
2022 Bridge Protedt
2020 Semicundactur Shortage
2022 Shunghai Finshun Fire
2021 Hong Kong ACG
2019 EVA, Strike
2020 Thivaness Salf-Driving Bas
2021 Shiynn Pipeline Explosion
2022 Qingming Festival
2020 Chinn-India Skirmish
2022 Mengiian Modole Launch
2022 Changsha Fire
2021 Comic World Taiwan
2023 Missing Taiwanewe Soldier Chins-US Trade War
2020-200 EAST Developmen's
2022 Taitun Fanhquulees
2012 Reijing Intemutional Manshhnn
2021 Hualong One
English
2018 Lower Puna Eruption
2022 World Series
2022 Slarbucks Strike Catalan
2021 mRNA Vaecine Rollont
2021 Ouagon Fires
2018 Royal Wodkling
2020 CRISPR Nobel Prize
2019 Noure Dame Fire
2012 San Diego Comic-Con
2020 Brexit
2020 Oregon Psiloeybin Legulication
2022 Keystone Pigeline Oil Leak
2019 "Stamm Area, $1"
2022 UK Government Crisis
2019 Messier 874 Images
2020 California Fires
2017 Fyre Fost
2012 Glorgia Scrate Race
2012 Southwess Cancellation Crisis
2018 Anchorage Eanhquake
World Cup
2019 Government Shundowm
2021 Inspiration Launch
2019-30 Australia Fires
2018 Caschella
2001 Canadian Flection
2020 AlphaFoodd2
2019 Townsville Flood
2023 Superbowl
2022 Novak Djokovic Visa Controversy
2022 San Francisco Waymo Trials
2022 Las Vegas Floods
2019 NBA Fimils
2023 Black Sex Drune Incident
2020 NASA Mars Mission
2017 Hurricane
2021
2020 Election
3021 James Wiih Telescape Launch
2022 Ferdale Earthquake
2016 Olympics
2022 Queen Elizabeth's Funeral
2016 Pokemoo Go Launch
2020 Nachville Toraado
2002 Wimbledon Champiomships
2022 Canala Contony Protest
2017 Snupchal IPO
2018 Solowesi Tumnia
2019 Met Cula
201 9-39 Trump Empeschment
2018 7ucketberg Senate Hearing
Koress
2016 East Asia Cold Wive
2019 SEMICON
2016-17 17 South Korcon Protests
2022 Nuri Rocket Launch
2017 Pohung Earthquake
2022 Ulrra Karea
2016 Choi Soon Sil Hecuiing
2023 Danari Photos
2019 Typhoon Lingling
2022 COMEUP
2016 GSOMIA Pact
Tesla Optimas Demomitation
2018 Notheast Asia Heat Wave
2003 Korea Open Series
2018 Opinion Rigging Scandal
2023 Consumer Electroncss Show
2020 Korean Flouds
2022 Toekwondo
2023 Japan South Korea Trade Dispute
2022 [[wasoong 17 KBB Launch
2018 Typhoun Soulik
2023 Jeju Fire Festival
2022 BTS Militury Draft
2021 Anti-Google Law
2022 Typhoon Nanmadol
2012 Emmm Awards
2018-19 President Resignation Movement
2023 LG Security Breach
2022 South Koeean Floods
2022 Kimchi Fescival
2022 South Kocea Presidenial Election
201 Sewal Fenry Salvage
2022 Daegu Office Fire
2023 Araro Callery Recpening
2017 US TPP Withdrawal
2017 Lutte World Tower Opening
2020 lchecn. Fire
2022 Frieze Secul Furr
2019 DMZ Summit
Hyundai Mobed Demo
2002 Typhoon Ulinaamorr
2019 FIFA Women's is Cup
South Kurean Capital Attack
2022 Blockchuin Week
2021 East Asia Sandsturm
2013 Asian Rowing Chumpionship
2016 Geun-Tlye Impeachment
2022 Tech Lagolls
2016 Gyconeju Earthquake
2016 Speed Skating Championships
2017 South Korea Eleetion
2016 AlphaGO Macch
Russina
2017 Samaov Plane Crush
2019 Kussia Comac Con
2022 ZNPF Cupture
2022 Communiciion Exhabotion
2022 Kostroma Cafe Fire
2022 Grushinsky Festival
2022 Anti Mobilization Protests
2021 Paxlovid Lieensing
Soyuz MS-22 25--2215 Leak
2017 Moscow Film Festival
Ukmine Mobilizatoon
2022 Twitte Acquisiion
2017 Moscow Storm
2019 Winter Universade
2014 Revolution Dignity
RS-24 Yars Missile
2022 Crimeun Dridge Explosion
2019 Men's lce Hockey Champiomhip
2018 US Syria Withdrowal
2022 Digital Services Act
2018 Kemerova Fire
2012 Eurovision
2016 Russian Election
2022 FTX Hankcuptey
Norilsk Oil Spill
Mascow Marathon
Bout Griner Prisner Exchange
2020 Sputnik V Distribution
2020 Typhson Maysak
FIFA World Cup
Moldosa Government Resignation
Crimeun Bridge Opening
2023 St Petershurg Explosios
2022 Voice Of Nomads
2019 Ingustttiia Protests
2019 Edccunch Conference
2021 Yalta Flood
2016 World Chess Championshpp
2022 St. Peseriburg Eccnomic Forum
2022 Kamaz Jupiter 30 Annouzcement
2021 Linyyallagy Mine Disuster
2019 Afisli Piesic
20201-21 Khaharovsk Krai Protests
2023 Luna 25
Norilsk Avalanche
2016 Rodelecco Cup 2017 J.Pest
2019 Ukminn Ellction
2021 International Lunar Research Station 2023 Global Energy Crisis
2018 Magnitugonk Building Fire
2020 Third Kyrgyz Revolation
Table 6: All current events covered in MultiVENT, organized by language and event category. Each current event has up to ten videos and corresponding video descriptions, along with 1-2 full text documents.
A Current events
In Table 6 we list the full set of current events covered in MultiVENT.
14

---

Source
Arabic
Chinese
English
Korean
Russian
Mean tokens
123
114
95
108
170
Median tokens
25
53
38
53
54
Mean unique tokens
82
74
55
78
97
Total tokens
62820
60863
48072
56329
87493
Total unique tokens
17713
13824
6473
20820
19234
Num. videos w/o dialogue
13
15
20
Table 7: Audio information taken from Whisper transcripts of the MultVVENT video dataset. We
partition statistics on language. As shown in the table, the dataset rich audio content and only approximately 2% of the dataset did not have detected spoken audio content.
FrameNet Template
Video Template
Disaster
Disaster
Duration
Place
Place First responders
Responder
Response
Time
Time
Victim
Affected people
Disaster outcomes
Table 8: Original FrameNet disaster template compared against our revised template for video content.
B Audio content in MultiVENT
To assess the audio content present in the dataset, we pass the video set through OpenAl's Whisper model 351 to produce multilingual audio transcripts. We parse this data and present general statistics Table We use NLTK to tokenize the transcripts except Chinese data which tokenized with Jieba
C Disaster template modification
We modify the FrameNet "disaster scenario" template, which we use as guide to annotate the videos in our Section 4 analysis. Specifically, remove the "duration" and "response" fields from the FrameNet template, as duration is generally not depicted in video footage and the "first responders" field typically aligns with "response" content in video. We then add "disaster outcomes" field as video content typically includes visually relevant depictions of the disaster aftermath. The difference between the two templates shown in Table
hhapss//iithbb.om/sssssiiiii
15


Reviews:
[{"review_id": "c5dEJB447n", "reviewer": "Reviewer_ZCm2", "summary": "", "questions": "", "limitations": "No specific section on this in the paper. The authors can explain how the models using this dataset may be misused. what about the multilingual aspect of the dataset?", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The effort to create an organic multilingual dataset is appreciated and indeed important for the field.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "UqWgyQ5YTY", "reviewer": "Reviewer_uXbS", "summary": "", "questions": "", "limitations": "The authors do not seem to have addressed the limitations of their work and the potential societal impact.\nTheir development of multilingual data does go towards alleviating biases introduced by using only English and opens the way to larger data sets in the future that would include even more languages. Their overall technique is content neutral and is hence not affected significantly by any biases in the data. I think if the authors follow up on these and other points they will be fine. There are no serious ethical implications of their work.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Good literature review that is comprehensive and motivates the proposed work well.\n2. Interesting insights into content evolution over time.\n3. Decent baseline technique.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "E0jidlwvuX", "reviewer": "Reviewer_D4Di", "summary": "", "questions": "", "limitations": "1.\tMultiVENT covers 5 languages, but a larger diversity of languages could reveal more insights into multilingual modeling. For instance, other languages, like Japanese and Vietnamese, also have a large number of speakers.\n\n2.\tThe source of the data is too simple; most of it comes from YouTube and Twitter.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.\tThis is the first large-scale multilingual video retrieval dataset spanning 5 languages (Arabic, Chinese, English, Korean, Russian) and containing diverse video types beyond just news broadcasts.\n\n2.\tThe authors adopt CLIP to develop a simple yet effective method for multilingual, event-centric video retrieval. \n\n3.\tMethods and experiments are clearly presented, making them easy to follow.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "WFYFjv11Tw", "reviewer": "Reviewer_cNPz", "summary": "", "questions": "", "limitations": "The research motivation for this dataset needs to be further clarified, and the unique aspects of this dataset should be explained in more detail.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "Existing news video datasets primarily focus on content produced in English. However, the authors of this work have made a valuable contribution by providing a multilingual dataset consisting of videos capturing events and aligned natural text.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: Overall reviewers agree that the multilingual aspect and the diversity of news videos in this dataset represent valuable contributions to the field. Concerns about insufficient discussion of ethical considerations are well addresses in the new “Limitations and Ethical Considerations” section added during rebuttal. The authors are encouraged to incorporate the detailed suggestions provided by reviewers in the final version of the paper.
Decision: Accept (Poster)