SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Meng Liu $^{1}$ , Mingda Zhang $^{2}$ , Jialu Liu $^{2}$ , Hanjun Dai $^{2}$ , Ming-Hsuan Yang $^{2}$ , Shuiwang Ji $^{1}$ , Zheyun Feng $^{2}$ , Boqing Gong $^{2}$
$^{1}$ Texas A&M University
2Google
{mengliu,sji}@tamu.edu,
{mingdaz, jialu, hadai, minghsuan, zfeng, bgong}@google.com
In this paper, we present a novel problem, namely video timeline modeling. Our objective is to create a video-associated timeline from a set of videos related to a specific topic, thereby facilitating the content and structure understanding of the story being told. This problem has significant potential in various real-world applications, for instance, news story summarization. To bootstrap research in this area, we curate a realistic benchmark dataset, YouTube-News-Timeline, consisting of over 12k timelines and 300k YouTube news videos. Additionally, we propose a set of quantitative metrics to comprehensively evaluate and compare methodologies. With such a testbed, we further develop and benchmark several deep learning approaches to tackling this problem. We anticipate that this exploratory work will pave the way for further research in video timeline modeling. The assets are available via https://github.com/google-research/google-research/tree/master/videoTimeline_modeling.
As the amount of online video content continues to grow rapidly, organizing and browsing through this massive collection of videos become increasingly challenging, especially for news videos. In the modern world, the news is not just delivered through text and images, but also through videos. With the increasing use of video streaming platforms and emerging short video applications, the number of news videos is growing at an unprecedented pace. However, such a vast number of news videos are usually unstructured. For example, there are tens of thousands of news videos about the topic "COVID-19 Developments in 2020" on YouTube, and their relationships are not well presented. It would be nearly impossible for users to watch such numerous unstructured videos, thereby hardly obtaining an in-depth understanding of the news story.
In this work, we propose a video timeline modeling problem to help analyze and comprehend news videos. In particular, as illustrated in Figure 1, our goal in this problem is to construct a video-associated timeline that represents the critical events and their evolutionary order in a news story. This is of great importance for a better, user-friendly understanding of news stories. To the best of our knowledge, this is the first work to define and formulate this problem.
To facilitate research into this problem, we present a benchmark dataset, namely YouTube-NewsTimeline, by crawling timelines from the web and sourcing videos from the YouTube news video corpus. YouTube-News-Timeline has over 12k timelines and covers over 300k YouTube news videos in total. To quantitatively evaluate methods on the proposed dataset, we further design several metrics to assess the complex prediction results. Aside from our contributions of the standard benchmark, we
Figure 1: Illustration of video timeline modeling.
further propose several novel deep learning approaches, based on Transformer [Vaswani et al., 2017], pointer networks [Vinvals et al., 2015], and knowledge distillation [Hinton et al., 2015], to tackle the problem. It is demonstrated through extensive experiments that our preliminary methods are effective and could serve as reference baselines for future research.
Timeline modeling for textual data is a well-studied problem in the field of natural language processing [Ghalandari and Ifrim, 2020]. The goal is to identify key events and provide short summaries from a collection of news articles or other text corpora. Existing work either extracts important sentences [Allan et al., 2001, Chieu and Lee, 2004, Radev et al., 2005, Yan et al., 2011, Nguyen et al., 2014, Tran et al., 2015, Martschat and Markert, 2018, Camacho Barranco et al., 2019] or generates abstractive summaries [Steen and Markert, 2019] to construct the desired timeline. These methods solely focus on using textual data and ignore visual signals which contain rich context that may help enhance the understanding of news stories. Wang et al. [2016] jointly learn from text and image data to generate timelines. Our work moves one important step forward by modeling timelines for videos.
While general video classification and representation tasks [Ji et al., 2012, Misra et al., 2016, Fernando et al., 2017, Han et al., 2019, Piergiovanni et al., 2020] have established importance in the field, they focus on understanding individual videos. In contrast, the video timeline modeling task considers the intricate relationships among multiple videos. We also note that the video timeline modeling problem is distinct from video summarization [Ngo et al., 2003, Khosla et al., 2013, Gong et al., 2014, Zhang et al., 2016, Li et al., 2018, Huang and Wang, 2019, Apostolidis et al., 2021]. Existing video summarization works mainly create a concise outline by selecting informative parts from a single video, while our task involves modeling the complex relationships among multiple videos. In addition, while video summarization typically relies on low-level visual features like appearance and motion, our video timeline modeling task requires a higher-level understanding of the events contained in the videos and the ability to align events across multiple videos.
Another relevant task is multi-video summarization [Panda et al., 2017, Wu et al., 2020], which primarily aims to create a concise representation of multiple videos by selecting keyframes or short segments, emphasizing the overlap and complementarity among the videos. On the other hand, while video timeline modeling also takes multiple videos as input, it seeks to identify, order, and represent significant events in an evolutionary sequence. Our task is also relevant to the recently proposed visual abductive reasoning (VAR) task [Liang et al., 2022, Hessel et al., 2022], which aims to infer the hypothesis that can best explain the premise, given an incomplete set of multiple visual events. Both VAR and our task involve multiple visual inputs, but there are significant differences. VAR reasons the causes of observed events, whereas we infer the evolutionary order of events within a set of videos. In addition, while VAR typically involves a single visual event, video timeline modeling deals with multiple videos that may contain various scenes and events.
3.1 Video Timeline Modeling Problem
Given a set of videos belonging to a specific news topic, the goal of the video timeline modeling problem is to yield a number of ordered nodes to form a timeline, where each node corresponds to a set of videos selected from the input video set. Formally, each training timeline sample consists of a video set  $V = \{v_{1}, v_{2}, \ldots, v_{N}\}$  and their corresponding labeled node IDs  $y = \{a_{1}, a_{2}, \ldots, a_{N}\}$ , where  $a_{i} \in \{1, 2, \ldots, K\}$  denotes the labeled node ID of the  $i$ -th video and  $K$  is the number of nodes on the timeline. The objective of video timeline modeling is to learn a function mapping:  $f: \{v_{1}, \ldots, v_{N}\} \mapsto \{a_{1}, \ldots, a_{N}\}$ . The  $k$ -th node can be represented as a set  $C_{k} = \{v_{i} | a_{i} = k, \forall i\}$ . For a concise problem definition, we assume that each video must be assigned to and can only be assigned to one node, i.e.,  $\bigcup_{k=1}^{K} C_{k} = V$  and  $\forall k \neq l, C_{k} \cap C_{l} = \emptyset$ .
As shown in Figure 1, intuitively, the objective of the proposed problem is to arrange a collection of unstructured news videos related to a specific news topic into a coherent timeline story. Particularly, each node on the obtained timeline represents a key event of the news topic, and the order among nodes denotes the evolution of the news story. Note that this work, as an exploratory attempt, focuses solely on assigning input video sets to ordered nodes. Once the ordered nodes are obtained, generating event information (in text, key frames, shorter videos, etc.) through video summarization becomes straightforward. To avoid any confusion, we consider video summarization as a separate step and do not include it in the problem we have defined.
Note that, in our problem setting, the textual timeline including the desired number of nodes is not given as an input. This setup aligns with the realistic scenario where a well-organized and timely timeline is typically unavailable for news, particularly for breaking news, but we are able to accurately retrieve the videos related to a news topic based on keyword filters or embedding guided retrieval [Revaud et al., 2013, Kordopatis-Zilos et al., 2019]. Another setting, where an authoritative timeline is provided as input and video assignments (or, recommendations) are conditional on that timeline, might also be practically useful. Both setups are valid, each stemming from distinct problem assumptions and targeting different real-world situations.
3.2 YouTube-News-Timeline Dataset
Challenges. Although the news video timeline modeling problem has significant potential in real-world applications, this area is unexplored. The main reason is likely that it is challenging to obtain such a video-associated timeline dataset. We summarize the main challenges as follows. i) News stories are often complex and multifaceted, with multiple events happening simultaneously or in rapid succession. As a result, creating a comprehensive and accurate timeline can be time-consuming and require a high level of expertise in the subject matter. ii) There does not exist a golden timeline for a news topic, since such a timeline is highly subjective, with different resolutions on the time scale and the event scale. iii) Assuming we have an authoritative reference timeline, it is still nearly impossible to use human labeling service to watch such an overwhelming number of videos and assign them to nodes on the reference timeline.
Data collection pipeline. Given such challenges, we consider using an automatic pipeline, instead of human labeling, to collect video-associated timeline samples. Our overall pipeline is illustrated in Figure 2. We start by crawling timelines from articles on the web that have mentioned the "timeline" keyword in the title or URL and have ordered dates in the content. We reasonably assume the crawled timelines can be considered authoritative and comprehensive for two reasons. i) Such articles are typically written by journalists or experts in their respective fields, who have access to reliable sources of information and are trained to ensure that the information presented is factual and of high quality. ii) The timelines themselves are presented in a structured and organized manner, with clear dates and descriptions of the events that occurred, thereby making them convincing and easy to be verified.
Using HTML parsing, for each crawled timeline, we then extract the chronological dates for the key events and their corresponding textual descriptions, specifically scraping the text descriptions from the corresponding leading paragraph. If the leading paragraph length is below 80 characters, we supplement it by appending additional words from subsequent paragraphs. This ensures we obtain a robust volume of information for the ordered nodes on the timeline. Next, we employ an embedding-based retrieval technique to identify relevant videos, from the in-house YouTube news
Figure 2: Illustration of our data collection pipeline.
video corpus, that correspond to the textual descriptions of nodes on the timeline. For both node text and video title text, we use the embedding given by the pre-trained NewsEmbed model [Liu et al., 2021], a universal text encoder for the news domain. By comparing the cosine similarity between these embeddings, we retrieve appropriate videos for each node text description on the timeline. This approach enables us to efficiently and effectively curate a collection of news videos that accurately reflect the context of the timeline nodes. Such assignments from videos to nodes are the desired labels for the video timeline modeling problem. Figure 3 illustrates an example of a crawled timeline and the corresponding video-associated timeline obtained via the above process.
Sanity assurance. To ensure that the retrieved videos are correct in terms of timeline node assignment, we prioritize high precision over high recall in the embedding-based retrieval process by setting a high embedding similarity threshold. In addition, we retain a maximum of the top 5 videos that surpass this stringent criterion, ensuring the fidelity of the node-video assignment. Furthermore, we take into account the possibility of consecutive events being semantically similar, which can result in similar embeddings, thus retrieving common videos. To mitigate this, we filter the videos associated with each node to only retain those with a release time, i.e., the time when the video is published on
Figure 3: Example of (a) a crawled timeline and (b) the obtained video-associated timeline.
YouTube, falling between the two consecutive timestamps indicated by the current node and the following node. This approach helps reduce any ambiguity between consecutive events and ensures a higher level of precision in the assigned videos. According to our manual check for sampled timelines, the above strategies are effective to ensure the collected labels are of high quality.
Dataset overview. For each timeline sample, we provide the following information. (1) The URL link of the webpage where we crawl the timeline. (2) The URL links of the retrieved
#Timelines	#Nodes	#Videos
9936/1255/1220	74886/9325/9171	242685/30369/29930
Table 1: Statistics of the training/validation/testing subsets.
YouTube news videos that are related to the specific news topic. (3) The corresponding assignment from videos to nodes. Such assignments are obtained by the described data collection pipeline. A data format example is included in the appendix.
Note that in the YouTube-News-Timeline dataset, we provide the Youtube URL link for each video. Thus, any public metadata that can be crawled from YouTube can be used as input features. In Section 4, we use the same set of input features when comparing different methods.
Dataset statistics. We crawl over 20k timelines and collect over 12k qualified video-associated timelines after filtering out certain failure cases during the data collection process. These timelines include over 300k YouTube news videos in total. The duration of these videos ranges from 3 seconds to 12 hours, and their average duration is around 10 minutes. We randomly split the timelines into
training, validation, and testing subsets with a ratio  $0.8 / 0.1 / 0.1$  before the data collection process. The number of timelines, nodes, and videos on each split in the final dataset are summarized in Table 1. Additionally, we present important distributions for each split in Figure 4, including the number of videos per node, the number of nodes per timeline, and the number of videos per timeline. Note that the number of nodes on a timeline in our dataset is between 2 and 24. Further details on the dataset's characteristics, such as news sources, topics, and chronological information about events, can be found in the appendix. Importantly, our reference timelines are sourced from a diverse set of publishers, cover a wide array of topics, and span an expansive temporal range, ensuring diversity and alleviating inherent biases.
3.3 Evaluation Protocol
To quantitatively evaluate methods on the proposed dataset, we further develop our evaluation protocol to cover several metrics. Following our previous notations, for a specific news topic, we have a set of input videos, denoted as  $V = \{v_{1}, v_{2}, \ldots, v_{N}\}$ , and their corresponding labeled node IDs are  $y = \{a_{1}, a_{2}, \ldots, a_{N}\}$ . We further denote the predicted node IDs as  $\hat{y} = \{\hat{a}_{1}, \hat{a}_{2}, \ldots, \hat{a}_{N}\}$ , where  $\hat{a}_{i} \in \{1, 2, \ldots, \hat{K}\}$  represents the predicted node ID for video  $v_{i}$ . Here,  $\hat{K}$  is the number of nodes produced by the prediction, which may be different from  $K$  since the desirable number nodes is not known as an input in our problem setting. Accordingly, let
Figure 4: Distributions of the number of videos per node, the number of nodes per timeline, and the number of videos per timeline in the training, validation, and testing subsets.
$C_k \subseteq V$  for  $k \in \{1, 2, \dots, K\}$  and  $\hat{C}_l \subseteq V$  for  $l \in \{1, 2, \dots, \hat{K}\}$  denotes the  $k$ -th node on the labeled timeline and the  $l$ -th node on the predicted timeline, respectively. Note that the number of input videos  $N$  and the desired number of nodes on the timeline  $K$  vary across samples, as depicted in Figure 4. For quantitative evaluation, we propose to measure two essential aspects according to the problem characteristics. In particular, we measure the correct identification of events on the timeline and the correct evolution ordering of videos on the timeline. We describe the proposed metrics in detail below.
Example. To enhance understanding, we use the following dummy example to help explain these metrics. In the example, we have 10 input videos belonging to a news topic, denoted as  $\{v_{1}, v_{2}, \ldots, v_{10}\}$ . The labeled node IDs are  $\{a_{1}, a_{2}, \ldots, a_{10}\} = \{1, 1, 1, 1, 1, 2, 2, 2, 2, 3\}$ . Suppose the predicted node IDs as  $\{\hat{a}_{1}, \hat{a}_{2}, \ldots, \hat{a}_{10}\} = \{1, 1, 1, 1, 3, 4, 2, 2, 4, 3\}$ . Obviously, there are 3 nodes on the labeled timeline, including  $C_{1} = \{v_{1}, v_{2}, v_{3}, v_{4}, v_{5}\}$ ,  $C_{2} = \{v_{6}, v_{7}, v_{8}, v_{9}\}$ , and  $C_{3} = \{v_{10}\}$ , while we have 4 nodes on the predicted timeline, including  $\hat{C}_{1} = \{v_{1}, v_{2}, v_{3}, v_{4}\}$ ,  $\hat{C}_{2} = \{v_{7}, v_{8}\}$ ,  $\hat{C}_{3} = \{v_{5}, v_{10}\}$ , and  $\hat{C}_{4} = \{v_{6}, v_{9}\}$ .
Metric 1: Node-level precision and recall. To measure how well the events on the constructed timeline match the ground truth events, we propose to compute precision and recall for nodes. First, we calculate the IoU scores between the ground truth nodes and the predicted nodes. Mathematically, the IoU score between node  $C_k$  and node  $\hat{C}_l$  is computed as  $\frac{|C_k \cap \hat{C}_l|}{|C_k \cup \hat{C}_l|}$ . Next, we construct a bipartite graph
Figure 5: Illustration of matching ground truth nodes and predicted nodes.
with the ground truth nodes and the predicted nodes as two separate sets of vertices. Each vertex is
connected to every vertex from the opposite set with edge weights equal to the corresponding IoU score. We further consider an edge to be a valid match only if its weight is greater than or equal to a threshold  $\sigma$ . All other edges are removed from the bipartite graph. Then, our goal is to find the maximum number of predicted nodes that match ground truth nodes, which is equivalent to a maximum bipartite matching problem, where the goal is to select a maximum number of edges in a bipartite graph in such a way that no two edges share a vertex. Such a problem can be converted to a maximum flow problem and solved via the Ford-Fulkerson algorithm [Ford and Fulkerson, 1956]. With the number of correctly matched nodes, we can compute the desired precision and recall as follows. The precision is the ratio of the number of correctly matched predicted nodes to the total number of predicted nodes, while the recall is the ratio of the number of correctly matched ground truth nodes to the total number of ground truth nodes. In our experiments, we select  $\sigma = 0.5$ . The node matching for our example is shown in Figure 5. The found matched node pairs are  $(C_1, \hat{C}_1)$ ,  $(C_2, \hat{C}_2)$ , and  $(C_3, \hat{C}_3)$ . Thus, the precision and recall for this timeline sample are  $\frac{3}{4}$  and 1, respectively.
Metric 2: Video-level Hamming distance. In order to evaluate the correctness of video ordering, we propose to calculate the Hamming distance between the predicted node IDs and ground truth node IDs. Specifically, Hamming distance between two sequences is the number of positions at which the corresponding symbols are different. This can directly reflect how many videos are assigned to a wrong node ID. Mathematically, the Hamming distance between a prediction and its corresponding ground truth is  $\sum_{i=1}^{N}[a_i \neq \hat{a}_i]$ . The average Hamming distance for our example is  $\frac{3}{10}$ , since there are three videos, i.e.,  $v_5$ ,  $v_6$ , and  $v_9$ , are assigned to the incorrect node IDs.
Metric 3: Video-level Euclidean distance. Although the above video-level Hamming Distance can show how many videos are wrongly assigned, it is still unclear how much the wrongly predicted node IDs deviate from the ground truth node IDs. Thus, we propose video-level Euclidean distance, which considers the absolute distance that a wrongly predicted video is shifted from its ground truth node IDs. It can capture the magnitude of errors in the video order. Formally, the Euclidean distance between a prediction and its corresponding ground truth is calculated as  $\sum_{i=1}^{N} |a_i - \hat{a}_i|$ . Obviously, the above video-level Hamming distance is a lower bound of this video-level Euclidean distance. In the given timeline example, the average Euclidean distance is  $\frac{3}{5}$ , because the absolute error distance of  $v_5$ ,  $v_6$ , and  $v_9$  are all 2.
Metric 4: Video pairwise agreement accuracy. To provide a more comprehensive evaluation of the video timeline modeling, we further include a video pairwise agreement accuracy metric. This metric is a straightforward extension of Kendall tau ranking distance [Kendall, 1938] by further considering the fact that multiple videos could have the same ranking. Specifically, we compute the number of pairwise video orders that are correctly predicted out of all possible pairwise orders. Note that for any pair of videos  $(v_{i}, v_{j})$ , there are three possible order relationships, including  $a_{i} < a_{j}$ ,  $a_{i} = a_{j}$ , and  $a_{i} > a_{j}$ . This metric complements the other metrics by focusing on the pairwise relationships between videos. In our example with 10 videos, there are 45 possible pairwise orders, and 32 of them are correctly predicted, so the pairwise agreement accuracy for this timeline is  $\frac{32}{45}$ .
For all the above four metrics, we can compute both macro and micro averages over the dataset. Macro averaging gives equal weight to each timeline, while micro averaging gives equal weight to each node (for the node-level precision and recall metric), each video (for video-level Hamming distance and Euclidean distance metrics), or each video pair (for video pairwise agreement accuracy).
4 Approaches
The benchmark proposed in Section 3 provides a standardized testbed for future research in this new direction. Here, we propose the following three exploratory approaches to tackle the defined video timeline modeling problem. We anticipate these models can serve as baselines for future research.
Given that each video in a timeline needs to be assigned to a specific node ID and the maximum number of nodes on the timelines in our YouTube-News-Timeline dataset is 24, in our following approaches, we formulate the model to perform a 24-class classification task to predict the node IDs. More specifically, the  $k$ -th class, for  $k = 1, 2, \ldots, 24$ , corresponds to the  $k$ -th node ID. Since learning from such a numerous number of raw videos is computationally expensive and raw video data often contains a large amount of noise and redundancy, we use an in-house feature extractor to obtain video embeddings for the following models. We anticipate that using other publicly available
(a) V-Transformer
(b) Tri-Transformer
(c) Tri-Transformer  $^+$  cross-modal distillation
Figure 6: Illustration of our proposed approaches, including (a) V-Transformer, (b) Tri-Transformer, and (c) Tri-Transformer + cross-modal distillation.