SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Kate Sanders* David Etter* Reno Kriz* Benjamin Van Durme
Johns Hopkins University Human Language Technology Center of Excellence {ksande25, detter2, rkriz1, vandurme}@jhu.edu
Everyday news coverage has shifted from traditional broadcasts towards a wide range of presentation formats such as first-hand, unedited video footage. Datasets that reflect the diverse array of multimodal, multilingual news sources available online could be used to teach models to benefit from this shift, but existing news video datasets focus on traditional news broadcasts produced for English-speaking audiences. We address this limitation by constructing MultiVENT, a dataset of multilingual, event-centric videos grounded in text documents across five target languages. MultiVENT includes both news broadcast videos and non-professional event footage, which we use to analyze the state of online news videos and how they can be leveraged to build robust, factually accurate models. Finally, we provide a model for complex, multilingual video retrieval to serve as a baseline for information retrieval using MultiVENT.
Information dissemination for current events has traditionally consisted of professionally collected and produced materials, leading to large collections of well-written news articles and high-quality videos. As a result, such materials form the basis for significant prior work in content analysis and retrieval [54, 20, 2, 15, 50]. Meanwhile, a high volume of event-centric content today is generated by non-professionals, such as on-the-scene witnesses to events who hastily capture videos and upload them to the internet without further editing. We propose that this contemporary landscape of news content can be leveraged by models to produce a more comprehensive understanding of events. News agencies have adapted to this shift, often collecting and incorporating this online content into official broadcasts, but news video datasets still do not typically address this new domain of event coverage.
In addition to focusing on traditional news sources, existing news video datasets predominantly consider content produced in English. This is consistent with common practices in video dataset collection: Collected videos and captions are recorded in English, and when multilinguality is considered, it is achieved by directly translating captions and transcripts [48, 22, 40, 19]. Because this data is originally produced for English speaking audiences, these multilingual datasets can contain unwanted content biases like "translationese" [6, 21]. As event-centric video content produced in other languages makes up a large portion of news videos online, we argue that including organic, multilingual content is necessary for a diverse and perspective-agnostic sampling of event coverage.
With these ideas in mind, we present MultiVENT, a dataset of Multilingual Videos of Events with aligned Natural Text that contains 2,396 diverse, event-centric videos and text descriptions
Translation: After rounds of dialogue sponsored by Baghdad and the Sultanate of Oman, China crowned it with the opening of a new door... The agreement between Saudi Arabia and Iran to restore diplomatic relations and open embassies in the two countries.
Figure 1: A sample video-text pair from MultiVENT. Every event-centric video is paired with a corresponding video description and a long-form text document describing the event, both in the same language as the video. If the language is not English, the video is also paired with a corresponding English document.
that reflect the distribution of news content online. The videos are grounded in natural language video descriptions and long-form text documents, and the data spans 260 current events across over forty countries. The content in MultiVENT is collected in five target languages: Arabic, Chinese, English, Korean, and Russian, and as the multilinguality is organic, the data is less likely to suffer from translation bias. We provide an illustration of the dataset's contents in Figure 1: Each natural language query (describing a video of a current event) is paired with grounding text documents and a unique corresponding video. We use MultiVENT to explore and characterize the variety of event-centric videos available online and illustrate the importance of leveraging these different video types when building multimodal information systems.
Citizen journalism, the most notable example being Wikipedia [18], has emerged alongside other online news sources as a method for curating comprehensive summaries of events. Work in natural language processing has considered the problem of automating this process by training models to generate informative reports using online source materials [24, 45, 33]. We use MultiVENT to explore how this process can be extended to incorporate multimodal sources of evidence. As a first step in this direction, we consider the task of video retrieval on MultiVENT, through which a model learns to retrieve multimodal source material given a natural language event description. This task differs from prior video retrieval benchmarks [9, 53, 1, 29, 51] as the videos in MultiVENT vary widely in length and content presentation, are multilingual, and can involve significant amounts of on-screen text. In addition to multilingual natural language captions for each video, we provide full text documents that ground the events and serve as more complex retrieval queries.
In summary, we address the lack of multilingual news video datasets that reflect the reality of our modern information landscape by proposing a dataset of diverse web videos depicting current events across five languages. Enumerated, our contributions are:
1. We present MultiVENT, a multimodal, multilingual information retrieval dataset of grounded videos depicting current events. The dataset targets five languages and covers a range of online video formats beyond traditional news broadcasts.
2. Using MultiVENT, we quantitatively illustrate the information presented by news videos and the differences in content between video formats, and qualitatively evaluate how multimodal coverage of an event can evolve over time.
3. We present MultiCLIP, a model for multilingual, event-centric video retrieval that serves as a baseline for video retrieval approaches on the task.
2.1 Video retrieval datasets
Early video datasets generally contained short clips spanning narrow ranges of topics, such as the Microsoft Research Video Description Corpus [9]. Video datasets spanning larger domains include MSR-VTT [53] and DiDeMo [1], although the lengths of these videos were still relatively short. The V3C dataset [39, 3] offered longer video lengths while still spanning a wide range of topics such as news reports. A shift towards massive video datasets was instigated by HowTo100M [29], which included over 130 million video clips belonging to one million narrated instructional videos. VaTeX [48], released in the same year, considered video retrieval from a multilingual context using caption translation. Additional multilingual video retrieval datasets include Rudder [13], consisting of instructional videos for making toys with multilingual captions, MTVR [22], which extended the TVR dataset [23] by adding Chinese subtitles and queries, and Multi-HowTo100M [19], which extended HowTo100M by scraping YouTube for subtitles in up to 9 other languages. Recently, Chen et al. [8] released the ChinaOpen dataset which contains a wide range of video-caption pairs originally produced in Chinese. Recent work has also considered the problem of interpreting text-heavy video content: Wu et al. [51] and Jahagirdar et al. [20] introduced datasets that focus on within-video text and OCR annotations, including news broadcasts.
2.2 Video retrieval methods
The size of early video datasets allowed retrieval systems to rely on pre-extracted features from expert systems like action recognition models. As massive video datasets gained prominence, the video retrieval paradigm moved towards ad-hoc video-text feature extraction using large pretrained models. Dosovitskiy et al. [14] proposed using stand-alone transformer architectures for video understanding, and Bertasius et al. [7] showed that applying space- and time-based self-attention independently improved performance. Bain et al. applied findings directly to video retrieval, training and evaluating transformer architectures on WebVid-2M [4]. Radford et al. [34] introduced CLIP and showed that pretraining models to match captions to images can result in scalable models, and CLIP's applicability to video retrieval was demonstrated by Fang et al. [16] through their CLIP2Video model. More fine-grained modifications to CLIP were proposed. Wang et al. [47] introduced "Object-aware Transformers", which extended video-text transformers to incorporate object-level annotations within video footage, and Ge et al. [17] modified the pretraining task to involve teaching a vision-text model to answer multiple choice questions about a video. Bain et al. [5] adapted large image-text models to the task of long video retrieval by incorporating the weighted-mean of frame embeddings, and Wu et al. [51] incorporated independent optical character recognition and embeddings into the encoder pipeline to explicitly model in-video text.
2.3 Report generation using online sources
A wide range of research has used online corpora for report generation tasks, including QA-pair generation [36, 30] (such as SQuAD [37] and HotPotQA [55]) and knowledge graph generation [31]. Notably, Lewis et al. [24] introduced a method for automatically extracting question-answer pairs from large corpora of text documents, and applied this method to Wikipedia to produce the PAQ dataset. Some PAQ extensions have been multilingual — Pisare et al. [32] built the WikiOmnia QA dataset on Russian Wikipedia documents, and Rybak et al. [42] produced a question-Wikipedia passage dataset in Polish. Recently, Qian et al. [33] extended the ideas in PAQ to construct WebBrain, a task in which a model must generate factual articles with references given a natural language query. In the multimodal domain, Reddy et al. and Chen et al. have considered the problem of open-domain QA for image-text data [38, 10], with Chen et al. using Wikipedia to generate a multimodal dataset. In a similar vein, Li et al. propose a dataset for information extraction from multimedia articles [27] and an extraction approach that can be used with text, image, and video content [26].
In this section we outline the MultiVENT collection process. We draw from Davidsonian event semantics [28] to define an event as a related set of particulars existing at a shared point in space
and time. We add the additional condition that an event must also be "newsworthy" in that it must have been reported on by respected news organizations. The dataset includes 2,396 videos and corresponding text descriptions covering 260 events grounded in 468 text documents, and includes content in Arabic, Chinese, English, Korean, and Russian. The average length per video is over one minute (83.7 seconds) and the full dataset contains over 55 hours of video content. We first identify 260 visually salient current events spanning from 2013 to 2023, and assign a target language to each event. Then, for each event, we collect grounding text documents and a set of videos in the event's target language.
3.1 Current event curation
We consider four primary event categories for MultiVENT: Disasters, political events, social events, and technology events. We include thirteen current events per category for each target language. We use Google Trends statistics to select these events, based on its tracking of term popularity based on internet activity by country. We construct lists of the top five countries with the most speakers of each target language and review the top trending topics on Google in each of these countries over the last ten years. We record topics and search phrases that corresponded to current events that (1) align with one of the predefined event categories and (2) have sufficient online video coverage. For categories that did not amass a sufficient list of current events per language through this process, we consult Wikipedia's yearly summaries of events to fill the remaining slots. Detailed statistics characterizing this set of current events are shown in Figure 2. As shown, the majority of selected events take place in the last few years, with only three taking place before 2016.
Also shown in Figure 2, there is not a bijective mapping between the language used in event coverage and the country the event took place in. The language and country are often related, e.g., Russian news content in MultiVENT predominantly takes place in Russia, but this is not true of all events in the dataset. For example, we include data in Chinese pertaining to the 2023 ATP tennis circuit in Dallas, Texas: At this event, tennis player Wu Yibing became the highest-ranked Chinese player in the history of the ATP rankings, and so the event received substantial Chinese news coverage. In cases such as this, news in multiple languages will heavily focus on the same current event, such as sports events and international political relations. We do not include the same event in multiple languages in MultiVENT by design, in contrast with data collection procedures used for efforts such as AIDA [46] which aim to cover a small collection of current events in many languages.
Every current event in the dataset is grounded in an English natural language document and, if the event is tagged with a non-English language, an additional natural language document in that target language. First, we check if a full English Wikipedia article exists for the current event. If not, we manually find a Wikipedia article that includes a passage describing the event. If Wikipedia does not have a passage that appropriately grounds the event, then a news article in English is selected as a grounding document instead. This process is then repeated for the target language. The dataset includes 468 grounding articles in total: 313 are full Wikipedia articles, 104 are Wikipedia passages, and 51 are external articles.
3.2 Video collection
We aim to collect visually and semantically distinct videos for each current event with an even split between firsthand witness accounts (e.g., first-person smartphone videos), amateur edited videos (e.g., vlogs), and professional news reports and compilations. Information regarding the resultant distribution of these categories and their semantic differences is included in Section 4.2. For each current event, we collect ten videos in the current event's target language. We search YouTube and Twitter for these videos using target keywords collected from the Google Trends search and Wikipedia. We check the upload date to ensure that it aligns with the time of the event's occurrence. Finally, we skim through the video content to confirm that it is relevant to the target event. After collecting the videos, we manually identify and remove duplicates, resulting in 2,396 videos in total. We do not include repeat videos, but sometimes professional news reports include firsthand footage that is already included as unedited footage in the dataset. In these cases, we keep both the news report and the original footage as the context and text metadata between the two are distinct. If the video has a natural language description, we tag the video with this description. If it does not, we use the video title as the tagged natural language description. We report the distribution of videos by source in Figure 2.
Figure 2: A: Statistics illustrating the distribution of current events selected for the dataset. (A1) depicts the general breakdown of countries in which each current event takes place. Many countries had a single current event, particularly small countries in the middle east and southeast Asia, and are consolidated into the "other" category for easier graph interpretation. (A2) shows the distribution of years during which the current events take place. B: Breakdown of data sources for the videos in the dataset. The majority of videos came from YouTube as it has a larger international audience and has existed longer than YouTube Shorts.
4 Data analysis
We present an analysis of MultiVENT to help characterize how online multimodal content contributes to our understanding of current events. We explore multimodal event coverage from three angles: (1) what kinds of information news videos contribute, (2) the differences in informative content provided by different types of news videos, and (3) how multimodal coverage of an event can evolve over time.
4.1 Semantic information in video
Visual data can provide rich, semantically nuanced details of an event that are not captured in text documents due to reporting bias, limitations of text, and document length limits. To characterize the complexity of these videos and the information they provide, we annotate a set of two hundred videos of disasters in MultiVENT to identify visual entities in the videos that help answer common "who, what, where"-type questions about the events they depict.
We present videos of disaster footage to local annotators and provide them with a set of event-centric questions derived from FrameNet's "disaster scenario" template [41]. We modify this template, designed for text, to sufficiently cover visual content (further information is included in the appendix). We instruct annotators to identify every on-screen entity (such as people, scrolling news headline banners, etc.) that might help answer one of these event-centric questions. These annotators are professionals with experience in data analysis and are compensated appropriately for their work. The annotations are for analysis only and are not part of the main dataset, and no personal information was captured.
The template divides salient entities into six categories: The disaster itself ("what"), the location of the disaster ("where"), the time the disaster takes place ("when"), people affected by the disaster ("who") and first responders for the disaster, e.g., firefighters (also "who"), and any visible outcomes of the disaster. Not every category applies to both visual content and text: We exclude "where" and "when" from the set of categories that visual content should be annotated for (because identifiable depictions of "where" are present in almost every frame, and "when" in virtually none) and disaster outcomes from the set of text annotation categories, as textual examples of this category tend to involve full clauses, which complicate the annotation process. We note that, while we exclude visual "where" content, different video types are likely to contain different amounts of visual information informing a viewer of where the event takes place. We omit this analysis for simplicity but do not model this facet of the videos' semantic richness by doing so.
We present the number of event-relevant entities that appear on-screen in these annotated videos in Table 1. For each annotated entity, we additionally ask annotators to rate their certainty that the entity is directly related to the event described by the video's natural language description from  $0\%$
Table 1: Mean number of visual entities and in-scene text references (written text displayed within a video) present per video in a subset of 210 disaster videos from the current events dataset. We omit "where" and "when" entities from the visual content counts as "where" visual content technically appears in every frame and there are few types of visual evidence for "when" questions. We omit "outcomes" from the text references as an outcome by itself is a full event that is difficult to localize in text (this field is omitted from the FrameNet event template analogue for text documents).
Visual entities	Text references	Total
Disaster ("What")	1.25	1.37	2.62
Place of occurrence ("Where")	-	1.54	1.54
Time of occurrence ("When")	-	0.77	0.77
Affected people ("Who")	1.22	0.54	1.76
First responders ("Who")	1.13	0.50	1.63
Disaster outcomes	1.00	-	1.00
Total	4.60	4.72	9.32
Table 2: Mean annotator certainty scores partitioned on entity type based on the annotations used for Table 1. 0.20 certainty indicates that the annotator is  $20\%$  sure that the annotated entity helps answer the tagged question about the described event, while 1.00 certainty indicates that the annotator is completely sure that the entity helps answer the tagged question about the event.
Disaster	Where	When	AP	FR	Outcomes	All
Visual content	.787	-	-	.716	.765	.798	.830
Text content	.931	.907	.929	.856	.836	-	.900
Average	.862	.907	.929	.759	.787	.798	.865
to  $100\%$ . We record these certainty scores in  $20\%$  intervals, i.e. as  $20\%, 40\%, 60\%, 80\%,$  or  $100\%$ . The averages of the linguists' confidence rankings by entity type are listed in Table 2.
As shown in Table 1, each video contains an average of 9.32 informative visual entities that pertain to the event in question. About half of these entities are purely visual, and half are within-video text that can be identified with an optical character recognition model. As indicated by Table 2, purely visual entities are more ambiguous than the text content shown onscreen alongside it, which aligns with past research that explores the difficulty humans have in interpreting visual content depicting complex events [43].
4.2 Video content by domain
As described in Section 3, we collect three main types of videos: Official news broadcasts, edited video footage, and raw, unedited footage. Of the 210 videos in the annotation set reported in Table 1,  $53\%$  are news broadcasts,  $11\%$  are edited footage, and  $36\%$  are raw footage. To quantify the difference in information presented by these different video types, we take the video annotations shown in Table 1 and partition these annotations by video type. We present the results in Table 3.
As shown by the results, news broadcasts depict the most relevant semantic information, followed by edited footage. This is particularly apparent when considering text content alone. On average, news coverage contains almost 9 times as much relevant on-screen text content than raw footage, and over three times more than edited footage. Visual content differences were less drastic, but news content still had two times more visual content than raw footage and 1.3 times more than edited footage. The difference in visual content between news coverage and edited footage is possibly due to average video length and the quality of the video curation — oftentimes, unprofessionally edited footage only draws from one source whereas news coverage draws from many.
4.3 Information evolution
As shown in Table 3, first-person footage is often opaque compared to professional coverage. However, comprehensive coverage often builds on earlier, less informative coverage. This can be seen in news
Table 3: Mean number of visual entities and in-scene text references present per video, partitioned on video type. Same 210 video subset is used for analysis as that used for the analysis shown in Table 1.
News coverage	Edited footage	Raw footage
Vis.	Text	Total	Vis.	Text	Total	Vis.	Text	Total
Disaster ("What")	1.42	2.38	3.80	1.14	0.41	1.55	1.05	0.17	1.22
Place ("Where")	-	2.51	2.51	-	0.59	0.59	-	0.39	0.39
Time ("When")	-	1.26	1.26	-	0.41	0.41	-	0.16	0.16
Affected people ("Who")	1.48	0.94	2.42	1.18	0.23	1.41	0.86	0.04	0.90
First responders ("Who")	1.73	0.78	2.51	1.36	0.45	1.81	0.17	0.12	0.29
Disaster outcomes	1.28	-	1.28	0.77	0.27	1.04	0.67	-	0.67
Total	5.91	7.87	13.78	4.45	2.36	6.81	2.75	0.88	3.63