Video Timeline Modeling For News Story Understanding
Authors: Meng Liu
Mingda Zhang
Jialu Liu
Hanjun Dai
Ming-Hsuan Yang
Shuiwang Ji
Zheyun Feng
Boqing Gong
Venue: NeurIPS 2023 Datasets and Benchmarks Spotlight (2023)
Keywords: Video Timeline Modeling
News Story
Datasets
Benchmark
TL;DR: We introduce video timeline modeling, provide the YouTube-News-Timeline dataset, propose evaluation metrics, and benchmark baselines.

Abstract: In this paper, we present a novel problem, namely video timeline modeling. Our objective is to create a video-associated timeline from a set of videos related to a specific topic, thereby facilitating the content and structure understanding of the story being told. This problem has significant potential in various real-world applications, for instance, news story summarization. To bootstrap research in this area, we curate a realistic benchmark dataset, YouTube-News-Timeline, consisting of over $12$k timelines and $300$k YouTube news videos. Additionally, we propose a set of quantitative metrics to comprehensively evaluate and compare methodologies. With such a testbed, we further develop and benchmark several deep learning approaches to tackling this problem. We anticipate that this exploratory work will pave the way for further research in video timeline modeling. The assets are available via https://github.com/google-research/google-research/tree/master/video_timeline_modeling.

Video Timeline Modeling For News Story Understanding
Meng Liu', Mingda Zhang'. Jialu Liu?, Hanjun Dai Ming-Hsuan Yang', Shuiwang Ji! Zheyun Feng', Boqing Gong"
Texas A&M University
Google
tmenggiu,,sji@ttamuu edu,
baangdazjjial), hadai, minghsuang zf bgong bgoogle0goo.l..com
Abstract
In this paper, we present novel problem, namely video timeline modeling. Our objective is to create video-associated timeline from of videos related a specific topic, thereby facilitating the content and structure understanding of the story being told. This problem has significant potential in various real-world applications, for instance, news story summarization. To bootstrap research in this area, curate realistic benchmark dataset, BuuTube-News-Timllnn con- sisting of over 12k timelines and 300k YouTube news videos. Additionally, we propose a set of quantitative metrics comprehensively evaluate and compare methodologies. With such a testbed, we further develop and benchmark sev- eral deep learning approaches to tackling this problem. We anticipate that this exploratory work will pave the way for further research in video timeline mod- eling. The assets are available via htthttts://gtthiie reogaarresearee tree/ wwarre/mmateer/eee master/vider timel ine itteeiie_mmmeeeeeieeieeeeeeee
Introduction
As the amount of online video content continues to grow rapidly, organizing and browsing through this massive collection of videos become increasingly challenging, especially for news videos. In the modern world, the news is not just delivered through text and images, but also through videos. With the inereasing use of video streaming platforms and emerging short video applications, the number of news videos is growing at an unprecedented pace. However, such vast number of news videos are usually unstructured. For example, there are tens of thousands of news videos about the topic "COVID-19 19 Developments in 2020" on YouTube, and their relationships are not well presented. It would be nearly impossible for users to watch such numerous unstructured videos, thereby hardly obtaining an in-depth understanding of the news story.
In this work, propose video timeline modeling problem help analyze and comprehend news videos. In particular, as illustrated in Figure [ our goal in this problem is to construct a video- associated timeline that represents the critical events and their evolutionary order news story. This is of great importance for a better, user- friendly understanding of news stories. To the best of our knowledge, this is the first work define and formulate this problem.
To facilitate research into this problem, we present a benchmark dataset, namely You'ubeeNNwws. Timeline, by crawling timelines from the web and sourcing videos from the YouTube news video corpus. YouTTube-Nwws-TT News-Timeline has over 12k timelines and covers over 300k YouTube news videos in total. To quantitatively evaluate methods on the proposed dataset, further design several metrics to assess the complex prediction results. Aside from our contributions of the standard benchmark, we
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

"The US did achieve goal millon vacoinations by the end of the year."
"The AstraZeneca and Oxford CDVID-19 vaccine granted emergency authorization the UK."
"New COVID-19 variant Colorado"
Unstructured news videos about "COVID-19 developments in 2020"
Video-associated timeline
Figure I: Illustration of video timeline modeling.
further propose several novel deep learning approaches, based Transformer (aswann et al. 2017], pointer networks [Vinyals et al., 2015), and knowledge distillation [Hinton et al.) 22155] to tackle the problem. It is demonstrated through extensive experiments that our preliminary methods are effective and could serve as reference baselines for future research.
2 Related Work
Timeline modeling for textual data is a well-studied problem in the field of natural language process-
ing [Ghalandari and Ifrim 2020 The goal is identify key events and provide short summaries from collection of news articles or other text corpora. Existing work either extracts important
sentences Allan al. 2001 Chieu and Lee 2004 Radev al. 20055] Yan al., 2011 Nguyen et al.)
2014 Tran et al.) 2015 Martschat and Markert 2018|Camache Camacho Barranco et al. 2019 or generates
abstractive summaries Steen and Markert 2019 construct the desired timeline. These methods
solely focus on using textual data and ignore visual signals which contain rich context that may help enhance the understanding of news stories. |Wang et al. [2016) jointly learn from text and image data to generate timelines. Our work moves one important step forward by modeling timelines for videos.
While general video classification and representation tasks eJi al. 2012 Misra et al. 2016 Fernando eet al., 2017 Han et al. 2019 Piergiovanni et al., 2020] have established importance in the field, they focus on understanding individual videos. In contrast, the video timeline modeling task considers the intricate relationships among multiple videos. We also note that the video timeline modeling problem distinct from video summarization [Ngo et al. 2003 Khosla et al.) 2013 Gong al.
2014, Zhang et al. 2016 Li et al. 2018 Huang and Wang 2019 Apostolidis et al. 2021]. Existing video summarization works mainly create concise outline by selecting informative parts from single video, while our task involves modeling the complex relationships among multiple videos. In addition, while video summarization typically relies low-level visual features like appearance and motion, our video timeline modeling task requires a higher-level understanding of the events contained in the videos and the ability align events across multiple videos.
Another relevant task is multi-video summarization [Panda et al. 2017 Wu et al. 2020 which primarily aims create a concise representation of multiple videos by selecting keyframes or short segments, emphasizing the overlap and complementarity among the videos. On the other hand, while video timeline modeling also takes multiple videos as input, it seeks identify, order, and represent significant events in an evolutionary sequence. Our task is also relevant to the recently proposed visual abductive reasoning (VAR) task [Liang et al. [002]] [Hessel al.) [022], which aims infer the hypothesis that can best explain the premise, given an incomplete set of multiple visual events. Both VAR and our task involve multiple visual inputs, but there are significant differences. VAR reasons the causes of observed events, whereas we infer the evolutionary order of events within set of videos. addition, while VAR typically involves single visual event, video timeline modeling deals with multiple videos that may contain various scenes and events.

---

3 Proposed Benchmark
3.1 Video Timeline Modeling Problem
Given a set of videos belonging specifie news topic, the goal of the video timeline modeling problem is to yield a number of ordered nodes to form a timeline, where each node corresponds to a of videos selected from the input video set. Formally, each training timeline sample consists of a video set V = {1), U2, .2..... UN f and their corresponding labeled node IDs where a {1, 2, 2,.... K denotes the labeled node ID of the i-th video and K' is the number of nodes on the timeline. The objective of video timeline modeling is to learn a function mapping: f: {vi..... uN] + {a1..... aN). J. The k-th node can be represented as set C2 {Fr |a k, vi). For a concise problem definition, we assume that each video must be assigned to and can only be assigned to one node, i.e., U Cs V and Vk + 1, Ck n 0.
{a1, .2,..... aN J.
As shown in Figure [ intuitively, the objective of the proposed problem is arrange a collection of unstructured news videos related to specific news topic into coherent timeline story. Particularly. each node on the obtained timeline represents a key event of the news topic, and the order among nodes denotes the evolution of the news story. Note that this work, as an exploratory attempt, focuses solely on assigning input video sets to ordered nodes. Once the ordered nodes are obtained, generating event information (in text, key frames, shorter videos, etc.) through video summarization becomes straightforward. To avoid any confusion, we consider video summarization as a separate step and do not include it in the problem we have defined.
Note that, in our problem setting, the textual timeline including the desired number of nodes is not given as an input. This setup aligns with the realistic scenario where a well-organized and timely timeline is typically unavailable for news, particularly for breaking news, but we are able to accurately retrieve the vidcos related to news topic based on keyword filters or embedding guided retrieval [Revaud et al. 2013 Kordopatis-
et al.) 2019]. Another setting, where an authoritative timeline is provided as input and video assignments (or, recommendations) are conditional on that timeline, might also be practically useful. Both setups are valid, each stemming from distinct problem assumptions and targeting different real- world situations.
3.2 NouTTube-News-TTmelln Dataset
Challenges. Although the news video timeline modeling problem has significant potential in real- world applications, this area unexplored. The main reason is likely that is challenging to obtain such a addoo-assocatte timeline dataset. We summarize the main challenges as follows. i) News stories are often complex and multifaceted, with multiple events happening simultaneously or in rapid succession. As a result, creating a comprehensive and accurate timeline can be time-consuming and require a high level of expertise in the subject matter. ii) There does not exist a golden timeline for a news topic, since such timeline is highly subjective, with different resolutions on the time scale and the event scale. iii) Assuming we have an authoritative reference timeline, it is still nearly impossible to use human labeling service to watch such overwhelming number of videos and assign them to nodes on the reference timeline.
Data collection pipeline.
Given such challenges, we consider using an automatic pipeline, instead of human labeling, collect
timeline samples. Our overall pipeline is illustrated in Figure We start by crawling timelines from articles on the web that have mentioned the "timeline" keyword the title or URL and have ordered dates in the content. We reasonably assume the crawled timelines can be considered authoritative and comprehensive for two reasons. i) Such articles are typically written by journalists or experts in their respective fields, who have access to reliable sources of information and are trained to ensure that the information presented is factual and of high quality. úi) The timelines themselves are presented in a structured and organized manner, with clear dates and descriptions of the events that occurred, thereby making them convincing and easy to be verified.
Using HTML parsing, for each crawled timeline, we then extract the chronological dates for the key events and their corresponding textual descriptions, specifically scraping the text descriptions from the corresponding leading paragraph. If the leading paragraph length is below 80 characters, we supplement it by appending additional words from subsequent paragraphs. This ensures we obtain a robust volume of information for the ordered nodes on the timeline. Next, we employ an embedding- based retrieval technique to identify relevant videos, from the in-house Yube news
3

---

The assignments from videos to nodes
Timeline
YouTube news video corpus
Parse
Node text
Embedding based retrieval
Video title text
web
Figure Illustration of our data collection pipeline.
video corpus, that correspond to the textual descriptions of nodes on the timeline. For both node text and video title text, we use the embedding given by the pre-trained NewsEmbed model [Liu et al. 202T], universal text encoder for the news domain. By comparing the cosine similarity between these embeddings, we retrieve appropriate videos for each node text description approach enables us to efficiently and effectively curate a collection of news videos that accurately reflect the context of the timeline nodes. Such assignments from videos to nodes are the desired labels the video timeline modeling problem. Figure ( illustrates an example of a crawled timeline and the corresponding video-associated timeline obtained via the above process.
the timeline. This
Sanity assurance. To ensure that the retrieved videos are correct in terms of timeline node assignment, we pri- oritize high precision over high recall in the embedding- -band retrieval pro- cess by setting high embedding sim- ilarity threshold. In addition, we re-
A Timeline of COVID-19 Developments 2020
A Timeline of COVID-19 Developments 2020
December 31,
December 2020 "The US did not
"The US did not achieve its goal million vaccinations by end of the year."
December 30, 2020
December 2020
tain a maximum of the top 5 videos
'The AstraZeneca and Oxford
TThe AstraZeneca and
that surpass this stringent criterion, en- suring the fidelity of the node-video assignment. Furthermore, we take into account the possibility of consec- utive events being semantically simi- lar, which result in similar embed- dings, thus retrieving common videos. To mitigate this, filter the videos associated with each node to only re- tain those with release time, i.e., the time when the video is published on YouTube, falling between the two consecutive timestamps indicated by the current node and the following node. This approach helps reduce any ambiguity between consecutive events and ensures higher level of precision in the assigned videos. According to our manual check for sampled timelines. the above strategies are effective to ensure the collected labels are of high quality.
COVID-19 vaccine has granted emergency authorization in the UK."
December 29, 2020
December 29, 2020
"New COVID-19 variant found in Colorado."
"New COVID-19 variant...
(a) crawled timeline
(b) The labeled timeline
Figure 3: Example of (a) crawled timeline and (b) the obtained video-associated timeline.
Dataset overview. For each timeline sample, we provide the following in- formation. (1) The URL link of the
#Timelines
#Nodes
#Videos
9936 0336//125, 1255/1220 74886/9325/ 242685/30360 00555/33369//993/ /29930
webpage where we crawl the timeline. (2) The URL links of the retrieved Table 1: Statisties of the training/validaationnessnng subsets.
YouTube news videos that are related to the specific news topic. (3) The corresponding assign- ment from videos to nodes. Such assignments are obtained by the described data collection pipeline. A data format example is included in the appendix.
dataset, we provide the Youtube URL link for each video. Thus, any public metadata that can be crawled from YouTube can be used as input features. In Section 4 we use the same set of input features when comparing different methods.
Note that in the CeeTTTub-News-Tinne
Dataset statistics. We crawl over 20k timelines and collect over 12k qualified video-associated timelines after filtering out certain failure cases during the data collection process. These timelines include over 300k YouTube news videos total. The duration of these videos ranges from seconds to 12 hours, and their average duration is around 10 minutes. We randomly split the timelines into

---

training, validation, and testing subsets with ratio 0.8/0.1/0.1 before the data collection process. The number of timelines, nodes, and videos on each split the final dataset are summarized in Table [ Additionally, we present important distributions for each split in Figure 4 including the number of videos per node, the number of nodes per timeline, and the number of videos per timeline. Note that the number of nodes on timeline in our dataset is between 2 and 24. Further details on the dataset's characteristics, such as news sources, topics, and chronological information about events, can be found in the appendix. Importantly, our reference timelines sourced from diverse set of publishers, cover a wide array of topics, and span an expansive temporal range, ensuring diversity and alleviating inherent biases.
3.3 Evaluation Protocol
To quantitatively evaluate methods on the proposed dataset, we further de- velop our evaluation protocol to cover several metries. Following vious notations, for a specific news topic, have set of input videos, denoted as V = {u, .2..... UN h. and their corresponding labeled node IDs are = {α), 02, o2,....,ay aNN.. We further denote the predicted node IDs as 1 = {ā1. a2...., ........,; where à E (1.2..... K} represents the pre- dicted node ID for video u,. Here, K the number of nodes produced by the prediction, which may be differ- ent from K since the desirable num- ber nodes is not known an input in our problem setting. Accordingly, let
#Videos per node
#Nodes per timeline
ariidoss per timeline
Training
6122
8:8
0.00
0.00
pre-
0.8
3.00
10
15
0.00
20
03
10
320
0.1
Validation
8:12
0.00
0.2
9.00
5.00
Testing
0.00
3.00
0.50
0.0
Figure 4: Distributions of the number of videos per node, the number nodes per timeline, and the number of videos per timeline in the training, validation, and testing subsets.
Cn V for k E (1, K} and O₁ C for / 21, K} denotes the k-th node on the labeled timeline and the 1-th node on the predicted timeline. respectively. Note that the number of input videos N and the desired number of nodes on the timeline K vary across samples, as depicted in Figure 4 For quantitative evaluation, we propose to measure two essential aspects according the problem characteristics. In particular, we measure the correct identification of events on the timeline and the correct evolution ordering of videos on the timeline. We describe the proposed metries in detail below.
the following dummy example to help explain these met- rics. In the example, we have 10 input videos belonging to news topic, denoted as {U1. ....... } The labeled node IDs are {a1, .2...... 010} (1, 1,, 2, 2. Suppose the predicted node
Example. To enhance understanding,
IDs as {âı, ag..... ] 1,, timeline, including C1 = {U1, 02 U3, U4, U, C₂ {UG, U7, Us Ug and C₃ = {V1o), while we have 4 nodes on the predicted timeline, including C {u1, U2 M3 w. C2 = {17, vg C3 = (v3, v10 and C. = (16, vg).
4, 3. Obviously, there are nodes on the labeled
Metric 1: Node-level precision and recall. To measure how well the events on the constructed timeline match the ground truth events, we pro- pose compute precision and recall for nodes. First, we calculate the loU scores between the ground truth nodes and the predicted nodes. Mathemati- cally, the loU score between node Ck and node Ć is computed as [Candil |C,uC]]
score C2
C
C
C
Filter out
C₂
Maximum bipartile
edges
C2
C
C2
C₂
C.
C
Figure 5: Illustration of matching ground truth nodes and predicted nodes.
Next, we construct a bipartite graph with the ground truth nodes and the predicted nodes as two separate sets of vertices. Each vertex is
5

---

connected to every vertex from the opposite set with edge weights equal to the corresponding IoU score. We further consider an edge to be a valid match only if its weight greater than or equal to a threshold a. All other edges are removed from the bipartite graph. Then, our goal is to find the maxi- mum number of predicted nodes that match ground truth nodes, which is equivalent to a maximum bipartite matching problem, where the goal is to select maximum number of edges in a bipartite graph in such a way that no two edges share a vertex. Such a problem can be converted to a maximum flow problem and solved via the Ford-Fulkerson algorithm Ford and Fulkerson, 1956]. With the number of correctly matched nodes, we can compute the desired precision and recall as follows. The precision is the ratio of the number of correctly matched predicted nodes to the total number of predicted nodes, while the recall is the ratio of the number of correctly matched ground truth nodes to the total number of ground truth nodes. In our experiments, for our example is shown in Figure 5
select o = 0.5. The node matching The found matched node pairs are (Ci,Ca) (C2, C2). and (Cs, Cz). Thus, the precision and recall for this timeline sample are i and 1, respectively.
Metrie 2: Video-level Hamming distance. In order to evaluate the correctness of video ordering, we propose to calculate the Hamming distance between the predicted node IDss and ground truth node IDs. Specifically, Hamming distance between two sequences is the number of positions at which the corresponding symbols are different. This can directly reflect how many videos are assigned to wrong node ID. Mathematically, the Hamming distance between a prediction and its corresponding ground truth is [4; # â.]. The average Hamming distance for our example is 3 three videos, i.e., lts, Ug, and ng, are assigned to the incorrect node IDs.
TO since there are
Metric 3: Video-level Euclidean distance. Although the above video-level Hamming Distance can show how many videos are wrongly assigned, it is still unclear how much the wrongly predicted node IDs deviate from the ground truth node IDs. Thus, we propose video-level Euclidean distance, which considers the absolute distance that wrongly predicted video is shifted from its ground truth node IDs. It can capture the magnitude of errors in the video order. Formally, the Euclidean distance between prediction and its corresponding ground truth is calculated Σ la, â,G. Obviously, the above video-level Hamming distance is a lower bound of this video-level Euclidean distance. In the given timeline example, the average Euclidean distance is 3 of Uss Us. and iy are all 2.
Tt because the absolute error distance
Metric 4: Video pairwise agreement accuracy. To provide a more comprehensive evaluation of the video timeline modeling, we further include video pairwise agreement accuracy metric. This metric is a straightforward extension of Kendall tau ranking distance [Kendall] [938] by further considering the fact that multiple videos could have the same ranking. Specifically, we compute the number of pairwise video orders that are correctly predicted out of all possible pairwise orders. Note that for any pair of videos (i,, v.). there are three possible order relationships, including a; < aj. aj = aj. and a; > aj. This metric complements the other metrics by focusing the pairwise relationships between videos. In our example with 10 videos, there are 45 possible pairwise orders, and 32 of them correctly predicted, the pairwise agreement accuracy for this timeline is 32
For all the above four metrics, we can compute both macro and miero averages over the dataset. Macro averaging gives equal weight to each timeline, while micro averaging gives equal weight to each node (for the node-level precision and recall metric), each video (for video-level Hamming distance and Euclidean distance metrics), or each video pair (for video pairwise agreement accuracy).
4 Approaches
The benchmark proposed in Section Section Provides a standardized testbed for future research in this new direction. Here, we propose the following three exploratory approaches to tackle the defined video timeline modeling problem. We anticipate these models can serve as basclines for future research.
Given that each video in timeline needs to be assigned to specific node ID and the maximum number of nodes on the timelines in our YouTube-News -ew-u--------mmllnne Timeline dataset is 24, in our following approaches, we formulate the model to perform 24- 24-class classification task to predict the node IDs. More specifically, the k-th class, for k = 2, 24, corresponds to the k-th node ID. Since learning from such numerous number of raw videos is computationally expensive and raw video data often contains a large amount of noise and redundancy, we use an in-house feature extractor to obtain video embeddings for the following models. We anticipate that using other publicly available

---

Video spprasertation
Pre-trained teacher model
Student model
Node represontation
Aftend
Altend
Text embedding
Paua Einaal encoding
Atlend
Transformer encoder 2
Transtomer encoder 3
Transformer encoder
Transtormer encoder
Classificatio head
Transformer encoder
Transformer encoder $
Transformer encoder
Transformer encoder
Transformer enoodor
Transformer encoder
1
1
Distilation loss
(a) V-Trnsffrmmer
(6) Tri- Transformer
(c) Tri- Tn-Trannformer cross-modal distillation
Figure 6: Illustration of our proposed approaches, including (a) V-Transformer, (b) Tri- TrrTransformer, and (c) Tri- TriTTransformer + cross-modal distillation.
video feature extractors, such as MediaPipe YouTube-8Ke 8M feature extractor]] can achieve similar performance and does not affect the main trend of our experimental results. We encourage to use the same featurization strategy for model comparison. It is worth noting that the input videos for news topic can be considered as a set without any specific order. However, in our proposed models, we consider ordering the videos by their release time, as have found through ablation study that this prior knowledge is beneficial for improving performance significantly.
y-Transformer Our first model uses Transformer to encode input videos. Specifically, the em- beddings input videos are ordered by the release time of the videos and such order information incorporated into the model via positional encoding [Vaswani et al., 2017]. Such Transformer encoder aims to learn the evolution dependencies among videos. The resulting representations are then fed into a fully connected layer to perform 24-class classification, where each class corresponds to a specific node ID on the timeline. The whole model is trained by the cross entropy loss. An illustration of this model is shown in Figure G(a).
Trr-Transfommer The main limitation of V-Transformer that the 24 classes are treated indepen- dently. However, the nodes on timeline are ordered and their underlying dependencies should be explicitly captured. Motivated by this, we further propose to model nodes as learnable embed- dings, and they are ordered according to the node order. As shown in Figure bb), the randomly initialized and learnable node embeddings and video embeddings are concatenated into a single sequence, which is the input for the first Transformer encoder. Note that they two separate sets of positional encodings to indicate the node order and video release time order, respectively. Intuitively, the first Transformer encoder aims at capturing the interactions among all tokens, including node tokens and video tokens. With this module, the obtained node representations are dependent on the input video embeddings. In other words, the obtained node representations input video set, which is intuitively more effective than video-set-agnostic node embeddings for the subsequent node assignment process. We further use additional Transformer encoders to model the node-node interactions and video-video interactions, respectively. Next, we attend each video representation all node representations. The attention score indicates the probability that the video belongs to the corresponding node. Inspired by pointer networks [Vinyals et al.) 2015). such scores are then used by the cross entropy loss to train the whole model. Our model named as Tti-Tranfformer since it consists of three Transformer encoders.
customized for each
Tri- Tri-Transformer cross-modal distillation. In our labeled video-associated timelines, shown in Figure 3, we also have the text description for the nodes in addition to the assignments from videos to nodes. Such texts contain rich semanties of the nodes and supposed be informative for assigning videos to nodes. However, these privileged text descriptions are not available during inference but training only. These considerations raise natural question. Is it possible use the rich text information during training to improve inference performance?
httpe://reeearcchggoglee com.yluuueemmmotbeeeeen

---

Table 2: Performance of the proposed methods our benchmark in terms of node-level precision and recall, video-level Hamming distance, video-level Euclidean distance, and video pairwise agreement accuracy. 1 (J) denotes that higher (lower) value indicates better performance. The top two results in terms of each metric are highlighted as 1st and 2nd.
PreckionRRocaii
Hamminng distance?
Euclidean distance?
Agreement accurasy?
Method
Macm
Miceo
Macro
Micro
Macm
Micro
Macro
Mfleoo
V- T-Trnnformer Tn-TTrndfommer
68 89, /G8 1%%
64.4%
/63.00
U.477
0.518
0.765
0.316
84.2%
1038%%
71.4% 771.2%
66.6% /65.65
0.453
0.490
0.716
0.837
85.4%
94.3%
Transfomne cross-modal distillation
72.3% /T1.1%
68.4% 68.9%
0.436
0.470
0.684
0.804
85.2%
94.5%
A referenee optimum
79.8% 78. 3%
76.0% /72.9%
0,308
0.302
0,380
0.473
88.8%
95.5%
To answer this question, we propose to use knowledge distillation techniques (Hinton et al. 2015] Gou et al. 2021]. Specifically, as illustrated in Figure/[el ((), we first train a teacher model, i.e., Tri- Transformer model, that includes text embeddings as input to capture semantic knowledge contained in the node text descriptions. The text embeddings are still given by the pre-trained NewsEmbed model [Liu et al. 202T). and are added to the corresponding learnable node embeddings as input to the Tri-Transformer. The parameters of our teacher model are fixed after the training. Our student model is also Tri- Traassssffrmrr model, without text embeddings as input. To train the student model, in addition the regular cross entropy loss, we have another knowledge distillation loss defined as the L2 distance of the node and video representations between the student model and the pre-trained teacher model. We specifically focus on the node and video representations generated by the first Transformer encoder, as those in the teacher model contain the most essential information learned from the text embeddings and their interactions with videos. The final loss function is a weighted summation of the cross entropy loss and the distillation loss. After the training, only the student model is used for inference. The above knowledge distillation strategy transfers the learned text semantics knowledge from the teacher model to the student model, which does not receive text embeddings as input. Such distillation is cross-modal [Gupta et al.) 2016] since the text modality is not available during inference.
Since we convert the timeline modeling problem to a classification problem in our approaches, there might end up with empty classes in our prediction results, thus leading to intermediate empty nodes on the timeline. To address this issue, we apply a post-processing step to skip those intermediate empty nodes. For instance, if the predicted node IDs for a 4-video input are {1, 1, 1,4, 2), we refine it to {1, 1, 3, 2} by skipping the original third node, thereby forming a consecutive timeline. We leave the study of a more integrated approach for future work, as discussed in Section
Experiments
Setup. We use 2 Transformer layers for all Transformer encoders in our models. In our third method, we set the weight for the cross entropy loss and the distillation loss to be 1 and 0.1, respectively. We train all models for 100 epochs with the Adam optimizer [Kingma and Ba 2015) and a batch size of 32. We did not tune the training hyperparameters extensively. The key hyperparameters learning rate and dropout rate are respectively selected from {0.01, 0.001, 1 O. 0.0005) and {0, (0, 0.25, 0.5) based on the validation performance.
Overall results. The test performance of our proposed three approaches on Ceehuube--ew---mmelll
# #Videos per node
# #Nodes per timeline
Ground truth mediitiinn
Ground truh) mroikc.on on
is present in Table 2 Although these three mod- els are exploratory attempts at solv- ing the video timeline modeling prob- lem, they perform fairly well in terms of the correct identification of events and the correct evolution ordering of videos. For example, based on the mi- cro average results, the models were able to recall over 60% of event nodes and correctly infer over 90% of video
0.4
0.12
0.08
0.2
0.04
0.0
0.00
2
15
20
25
Figure 7: Comparison of distributions of the number of videos per node and the number of nodes per timeline be- tween ground truth and predictions given by Tri-Transformer over the testing subsets.
pairwise orders. In addition, the micro-average results of vidco-level Euclidean distance demonstrate
8

---

Table 3: Results of ablation studies. The table schema follows Table 2
Precision/Recal!
Hamming distance
Euclidean distance
Agreement accuracy"
Method
Macro
Miero
Macro
Micro
Mocro
Miero
Macro
Micro
Tri Thi-Transformer
71.4%% /11./11.99
66.8%
0.716
0.837
85.4%
94.3%
woo video PF:
50. 447
43. 4%%74.44%
0.621
0.647
1.523
2.105
58.3%
72.1%
w/o video PE and encoder 28.3
17.5%%/18.9
39.9% /37.%%
0.638
0.656
1,576
2.165
51.8%
72.0%
that, on average, the predicted node IDs deviate less than one absolute position from the ground truth node IDs. A comparison of the performance between V-Transformer and Tri-Transformer shows that explicitly modeling nodes and their underlying dependencies and interactions with videos can consistently achieve more effective performance. Furthermore, using knowledge distillation via Tri-Transformer + cross-model distillation can further boost performance by transferring the learned textual semantics from the teacher model to the student model.
Aside from quantitative results, we examine the distributions of the number of videos per node and the number of nodes per timeline between ground truth and predictions given by Tri-Transformer. As shown in Figure 7 such distributions of predicted results closely match those of the ground truth. It important to note that the dataset sets a limit of five videos per node, but we not enforce this constraint during prediction, and we rely solely on the model's raw predictions. This explains why our model sometimes assigns more than five videos to a single node. The visualizations of several predicted timelines are included in the appendix.
An optimum bound study. Our cross-modal distillation strategy involves using node texts as input in the teacher model and transferring the learned knowledge to the student model that does not have such input. This is because we assume that such textual node information not available during inference. Given that we do have such textual node information in our test set of YouTube-Newss Timeline Tine dataset, we can evaluate the test performance of the teacher model that includes text information as input and treat such performance as a reference optimum of our Tri- TriTTransformer model without using textual information. The results included in Table 2 reveal that, while our cross-modal distillation approach does improve the performance over Tri- TranTrassfrrrrr there is still an obvious performance gap compared to the reference optimum. Therefore, developing more effective ways to incorporate textual information during training is potential future direction, as discussed in Section 6
Ablation studies. We further perform ablation study experiments based on Tri-Transformer to evaluate two key designs. First, we analyze the effect of incorporating the prior knowledge of video release time. As described in Section 4 we use positional encodings (PE) to indicate the order of input vidcos according to their release time. In our ablation model, we remove such positional encodings and treat the input videos as set instead. The results shown in Table clearly demonstrate that this order information is a strong prior knowledge and significantly improves the performance. Additionally, we further remove Transformer encoders 2&:3 and observe a decrease in performance, highlighting the importance of modeling the node-node interactions and video-video interactions.
6 Conclusion and Outlook
In this work, we introduce a novel and important video timeline modeling task, which constructs a aadeo-associated timeline from a set of videos, to help understand unstructured news videos. We provide standardized testbed for this new task with a realistic dataset and comprehensive evaluation protocol. We further develop and evaluate several intuitive approaches as simple baselines for future studies. The proposed benchmark dataset and methods will open the door for future research in video timeline modeling.
Here, we highlight several promising future directions in this field, including more principle explo- ration for problems and methodologies. On one hand, more effective and principle methodologies are highly desired. First, as demonstrated in the experiments, we can develop more effective strategies to leverage textual information during training. Second, while our current method, as an exploratory work, treats the problem as multi-class classification problem, it would be more principle to consider the problem as ranking problem and use differentiable ranking models. Third, our algorithms assign a set of videos simultaneously, which may result in different assignments for the same video if we add or delete video in the set. It remains challenging to model video interactions while keeping the prediction of videos less dependent. This aspect presents potential avenue for enhancement.

---

In addition, future research can also be made from a problem-oriented oriented perspective. Given that our defined problem and evaluation standard concentrate solely on assigning imput video sets ordered nodes, one possible extension is to integrate the event summarization step with our defined problem construct timeline with both associated videos and event information. The event information can be in the form of text, key frames, etc. Although our You'Tuhe-News- dees-Timellnne dataset can be used to evaluate the generated event text descriptions, the evaluation protocol and algorithms need to be dedicatedly redesigned. In addition, extending the single linear timeline to more complex relationship modeling, such as multiple timelines (Yu et al. 2021] and graphs, can be promising future direction to enhance the understanding of news stories
Disclaimer
The reference timelines used to construct the dataset are crawled from the web, and the videos sourced from YouTube. The opinions expressed in the timelines and videos do not necessarily reflect our own, and we do not endorse or promote any specific viewpoint. The dataset is intended for research and educational purposes only, and users should exercise their own judgment when interpreting and using it.
Acknowledgments
We would like to express our gratitude to Anish Nair from Google who initiated and sponsored this project and as well as consistently provided constructive discussion throughout the delivery of the paper. We also thank the reviewers for their constructive feedback, which improved the quality of this work.
10

---

References
James Allan, Rahul Gupta, and Vikas Khandelwal. Temporal summaries of new topics. In Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 10-18, 2001.
Evlampios Apostolidis, Eleni Adamantidou, Alexandros I Metsai, Vasileios Mezaris, and Ioannis Patras. Video summarization using deep neural networks: A survey. Proceedings of the IEEE, 109 ((I):1838-180)) 2021.
Roberto Camacho Barranco, Amold Boedihardjo, and M Shahriar Hossain. Analyzing evolving stories in news articles. International Journal of Data Science and Analytics, 833):411226, 2019.
Hai Leong Chieu and Yoong Keok Lee. Query based event extraction along timeline. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 425-432, 2004.
Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould. Self-supervised video representation learning with odd-one-out networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3636-3645, 2017.
Lester Randolph Ford and Delbert R Fulkerson. Maximal flow through a network. Canadian journal of Mathematics, 8:399 1956.
Demian Gholipour Ghalandari and Georgiana Ifrim. Examining the state-of-the-arr in news timeline summarization. In Annual Meeting of the Association for Computational Linguistics, 2020.
Boqing Gong, Wei-Lun Chao, Kristen Grauman, and Fei Sha. Diverse sequential subset selection for supervised video summarization. Advances in Neural Information Processing Systems, 27, 2014.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision, 129: 29911789-8199 2021.
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2827-2836, 2016.
Tengda Han, Weidi Xie, and Andrew Zisserman. Video representation learning by dense predictive coding. In Proceedings of the IEEE/CVE International Conference on Computer Vision Workshops, pages 0-0, 2019.
Jack Hessel, Jena D Hwang. Jae Sung Park, Rowan Zellers, Chandra Bhagavatula, Anna Rohrbach, Kate Saenko, and Yejin Choi. The abduction of sherlock holmes: A dataset for visual abductive reasoning. European Conference on Computer Vision, pages 558-575. Springer, 2022.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv:1503.02531 2015.
Cheng Huang and Hongmei Wang. A novel key-frames selection framework for comprehensive video summarization. IEEE Transactions on Circuits and Systems for Video Technology, 30(2): 577-589, 2019.
Shuiwang Ji, Wei Xu, Ming Yang, and Kai Yu. 3d convolutional neural networks for human action recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(1):221-231, 2012.
Maurice G Kendall. A new measure of rank correlation. Biometrika, 30(1/2):81-93, 1938.
Aditya Khosla, Raffay Hamid, Chih-Jen Lin, and Neel Sundaresan. Large-scale video summarization using web-image priors. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2698-2705, 2013.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, In International Conference Learning Representations, 2015.

---

Giorgos Kordopatis- Zilos, Symeon Papadopoulos, Ioannis Patras, and Ioannis Kompatsiaris. Fivr: Fine-grained incident video retrieval. IEEE Transactions on Multimedia, 21(10):2638-663 2019.
Yandong Li, Liqiang Wang, Tianbao Yang, and Boqing Gong. How local is the local diversity? reinforcing sequential determinantal point processes with dynamic ground sets for supervised video summarization. In European Conference on Computer Vision, pages 151-167, 2018.
Chen Liang, Wenguan Wang, Tianfei Zhou, and Yi Yang. Visual abductive reasoning. In IEEE Conference on Computer Vision and Pattern Recognition, pages 15565-15575 2022.
Jialu Liu, Tianqi Liu, and Cong Yu. Newsembed: Modeling news through pre-trained document representations. In ACM SIGKDD Conference on Knowledge Discovery Data Mining, pages 1076-1086, 2021.
Sebastian Martschat and Katja Markert. A temporally sensitive submodularity framework for timeline summarization. In Proceedings of the 22nd Conference on Computational Natural Language Learning, pages 230 2400-240,200 2018.
Ishan Misra, C Lawrence Zitnick, and Martial Hebert. Shuffle and learn: unsupervised learning using temporal order verification. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part I 14, pages 527-544. Springer, 2016.
Chong-War Wah Ngo, Yu-Fei Ma, and Hong-Jiang Zhang. Automatic video summarization by graph modeling. In IEEE International Conference on Computer Vision, pages 104-109. IEEE, 2003.
Kiem-Hieu Nguyen, Xavier Tannier, and Véronique Moriceau. Ranking multidocument event descriptions for building thematic timelines. In COLING 2014, the 25th International Conference on Computational Linguistic, pages 1208-1217, 2014.
Rameswar Panda, Niluthpol Chowdhury Mithun, and Amit K Roy-Chowdhury. Diversity-aware multi-video summarization. IEEE Transactions on Image Processing, 26( 26(10):4712-4724, 2017.
AJ Piergiovanni, Anelia Angelova, and Michael Ryoo. Evolving losses for unsupervised video representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 133-144 2020.
Dragomir Radev, Jahna Otterbacher, Adam Winkel, and Sasha Blair-Goldensohn. Newsinessence: Summarizing online news topics. Communications of the ACM, 48(10))95-98, 2005.
Jérôme Revaud, Matthijs Douze, Cordelia Schmid, and Hervé Jégou. Event retrieval in large video collections with circulant temporal encoding. In IEEE Conference on Computer Vision and Pattern Recognition, pages 2459-24600 22699,2466622 2013.
Julius Steen and Katja Markert. Abstractive timeline summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 21-31, 2019.
Giang Tran, Mohammad Alrifai, and Eclco Herder. Timeline summarization from relevant headlines. In European Conference on Information Retrieval, pages 245-256. Springer, 2015.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances Neural Information Processing Systems, 30, 2017.
Oriol Vinyals, Meire Fortunato, and Navdeep Jaitly. Pointer networks. Advances in Neural Informa- tion Processing Systems, 28, 2015.
William Yang Wang, Yashar Mehdad, Dragomir Radev, and Amanda Stent. A low-rank approximation approach to learning joint embeddings of news stories and images for timeline summarization. In North American Chapter of the Association for Computational Linguistics, pages 58-68, 2016.
Jiaxin Wu, Shang-hua Zhong, and Yan Liu. Dynamic graph convolutional network for multi-video summarization. Pattern Recognition, 107:107382, 2020.
12

---

Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. Evolutionary timeline summarization: balanced optimization framework via iterative substitution. In Inter- national ACM SIGIR Conference on Research and Development in Information Retrieval, pages 745-754, 2011.
Yi Yu, Adam Jatowt, Antoine Doucet, Kazunari Sugiyama, and Masatoshi Yoshikawa. Multi-timeline summarization (mtls): Improving timeline summarization by generating multiple summaries. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 377-387, 2021.
Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman. Video summarization with long short-term memory. In European Conference Computer Vision, pages 766-782. Springer, 2016.
13

---

A License
The You/lube-News-Timelin dataset is under the CC BY 4.0 International license. Please refer to for license details.
B
Data Format Example
data format example is shown below.
Listing I: A data example
"'https://appee wttp:/ss...ca/rrrtaeeeee http://cccaaaeeeeeeeaeaeaaaaa -acppenn-ccideeeeeeee teceenttttttuute teunamis earthquakes -42
The URL link of the webpage
wwhere we crawl the timeline.
[
"OhEbGK4Pa2g";
"c119tin33h1",
"RO16zOHaUAM"
"5QhCsR-t-qk",
"e*3FBIOHMX8*
]
10
11
"psAuFreXeqs",
12
"BaRd7MQuBHc", "Dp "rp_81LL1
"him7GFPAq3o"
15
16
"f4TaKPKelgg", "DL1skd-QC2o"
19
20
"oclušiVhveg";
"vusthiUFx_O"
"vGHzuZQLYt""
24
"7XpLba4xpls"
25
"UsPFUzXisq4"
26
},
"БAзfNKOrΣCs"
29
]
30
#
The URL links of the retrieved YouTube news videos. Each list iin the nested list corresponds to one node the timeline. Theese nodes ordered in the neated list.
31
C More Dataset Characteristics
We include more details the dataset's characteristics here. As demonstrated in Figure the reference timelines are curated from a diverse set of more than 1, 0,000 publishers. The primary topics, each recurring more than 30 times, depicted in In Figure 9 The topic annotations given by an in-house proprietary entity linking algorithm. Moreover, the distribution of the event date, corresponding to each node timelines, is shown in Figure 10
D More Experimental Studies
To assess the impact of video duration on our model's performance, in Figure 11 present correlation between the video length and the corresponding video-level Fuclidean distance, based on our Tri- Sti-Transformer model over the test set. The video-level Euclidean distance is not sensitive
14

---

www.reutar's com
www.spp www.spoorrngeeeeeeo com
www. www.usmagazeee. com
www. www.uardian.n.mmmmmm com
www. billboard com
www. www.z/iaaaaa.aooooo .com
www. insider .com
abonewe, go.gg.ooo no.ccom
www. Du wwleeeeeeeeeeiieer.. .com
wwww unntoociyyyy com
www.wwahhingooppcalooo
people
nylimes
wwww vulltue com
www.chicagotrieoo com
blaccherssportoo comn
www. www.soomplxxcoo com
weww cosmopolllannnnoccoccmmmo com
Veew foueneee comm
weww man, com
Weww for .oom
weww bochop tairoview.com
more
Figure 8: Pie chart of the news publishers.
700
000
500
Papulentt
100
Scotts
Puliec
Amnrican Football
Music
Nastonal Fostbal Languu
Heuet
Roskettbll
Low
ecoropping
Univensty
Love
Technologg
Majer Loopee
Topics
Figure 9: Distribution of the covered main topics.
to video length. It can be observed that the variance becomes large for longer videos as the longer videos less in both the training and test sets.
E Examples of Predicted Timelines
present two examples of timelines predicted by our Tri- Trannssnffre model Table 4 and videos that have been assigned to incorrect nodes highlighted for clarity.
The
15

---

3400
120
3000
Figure 10: Distribution of the event date.
Number Videos
Mean Distance
5000
-5
Video-level Euclidean Distance
4000
Number of Videos
3000
2000
MW
1
20
80
100
120
Duration (minutes)
Figure 11: Number of videos and average video-level Fuclidean distances in relation video duration.
Appeadira antes dedeedlaad (08885//06 edoeiiel Hlmntty 0,10, viboolee Buskkics 6384 agrcecei 56.8%
S0
Greand truth tiMlel
Phodked Mew
Vakiggnnie Caaa
yoer
mokk Varping
Thew basi cenedr
UiCnC SA0CE MISSERI Sherdeak dolkng)
yoo reean pla
1preere (1))
Ta poncs in quee
GMTGG Caa Land Spwcess
4
(notera Tay TV teChe tendlees
Ioree Ciddd Tag Cschecten den Mow
Pactyppi
5
1 72alti Cas qusnndaly Trapee
Desppiiaan bedder
prmni
Chae'ss sour
C1po Car di Clae'ss
Cang' Taoe
Smogpp05 Dang
Prok
2
Cas
(yscr))
D CLM aallllaaaaa
We Caallande iad Man"
Cazinako
addoacess Man") Tamwnn
Batekkense terry Mart)
(0cohttm)) Dhe fradi
Table 4: A timeline of "Major milestones in Chinese space exploration".
16

---

aprond bech
apoande cocimmation adue die
Locil ststin marre peeggater prlkitio)
Stalisa))
aue Rocs
uchodd))
5
Mtgliggtr
J Woas hork Highigt
By Distoa
Maal full Ressem with Ray
4
Foetial lighipuu
Fix de
0
Suokicial akk Was Bilers Ed spwa liv Mack 2013
Phe SToakda in fide agas Oriee")
Srl Sgeez Soars
caugende rades Gures CAEEE
Penkentt SepeerCaattnn Contrreee one
30 U.S. voner star plecn munh 33 tne tehitercp po
MA womm ore
payun mach line 4igt naki
BAZ de
suth
524 edlli
Federalon))
Porraa US mome wete tn redi oh SDAMF)
8
( Motre Sesu. Inaurde
Thamt SCA hallc
Becrnce Chaggadd the NDAA
Roass
Themal NCAA
Hike
Aingiag Sadaare Whesk Bakkhhal Maic 26 2012 tode Soure (5T)
453 a dd tue
Aog Wee''s bedend
Warer's Roberic Mackk
NCAA
33
Sadi 110 Progue Not
11
Lrioonipe Sed Cuddavs Micer 2002 CCSCCAA vesee
12
Table 5: A timeline of "50 Years of Title IX: The Defining Moments of Women's Sports".
F
Societal Impacts
We expect that the proposed benchmark dataset and methods will facilitate future ad advancements in video timeline modeling. On the positive side, it offers a helpful tool for understanding and navigating large volumes of news video data, enabling more efficient news consumption and ensuring a more comprehensive understanding of events.
One the negative side, the crawled timelines might not always reflect absolute precision. They might be mistakenly used as evidence in some situations. This highlights the importance of using and interpreting these timelines carefully. In addition, while we have taken steps to diversify our data sources, news content can inherently carry biases based on various factors. We acknowledge this challenge and emphasize the future need for more diverse data sourcing to capture a broad spectrum of perspectives and reduce inherent biases. Also, there could be potential misuse if the technology were applied unethically. Specifically, the construction of timelines could be manipulated to present events in a way that supports a particular opinion, thereby distorting the truth. To alleviate this potential concern, regular validation and fact-checking mechanisms can help ensure the constructed timelines align with factual occurrences.
17


Reviews:
[{"review_id": "bCuLYsWdMY", "reviewer": "Reviewer_84mW", "summary": "", "questions": "", "limitations": "The supplementary material contains a brief discussion of the potential ethical considerations, but more information on how the use of the proposed video timeline modeling could impact user privacy, data security, or the spread of misinformation would be helpful. News content can be subject to various forms of bias, and this can carry over into the modeling process. The authors should also address how they plan to mitigate potential biases in the video timeline modeling.", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "- The paper addresses a novel and practical problem of video timeline modeling. The introduced YouTube-News-Timeline dataset can be a valuable resource for further research in this area. \n- The paper experiments with three proposed deep learning approaches that seem well-motivated and designed to capture dependencies among videos and nodes on timelines. \n- The paper is well-written and easy to follow. The presentation of the problem formulation, dataset creation, and methodology are well-described and clear.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "XGiOFQldjJ", "reviewer": "Reviewer_aTSU", "summary": "", "questions": "", "limitations": "The only limitation I see is about the time resolution; in the paper the time resolution has been set to 1 hour. The paper has not explore how the model can adapt by lowering this time length.", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "The novelty of the paper is presenting the problem which has a great importance given the abundant of video from various social network that can capture different time point of a story.   ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "5SRDyJvJhM", "reviewer": "Reviewer_eQ6E", "summary": "", "questions": "", "limitations": "The authors acknowledge that the data is crawled from the web and hence is not completely under their control. I think that is healthy. It is beyond the scope of the paper to carefully curate each timeline to find out if there are objectionable opinions etc. The focus is on the timelines and there is no endorsement of the underlying content, which is as it should be.\n\nThe underlying problem of timeline extraction is content-neutral so my view is that the authors have adequately addressed the potential negative social impact.", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. Clever automation of data collection leading to a satisfyingly large dataset.\n2. Convincing and comprehensive metrics.\n3. Reasonable baseline performance.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "XFRyTvyfYH", "reviewer": "Reviewer_xNvC", "summary": "", "questions": "", "limitations": "I did not see a discussion on the societal impact in the paper, other than one part that the authors made disclaimer about the model's timeline alignment and this might not be the actual timeline. The author present a section in the supplementary file about this. The authors could expand on this, maybe describing that because the groundtruth timelines were also obtained by web crawling, they may not be precise enough, and a potential wrong prediction of news alignment may indeed have negative societal impact. For example, it might be used as evidence, while it is not indeed a confident reference.", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "I read the paper with great interest and find it well-written and describes the objectives and methods clearly. It describes a new application/task on news video and also present the dataset that can be used for this purpose.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: This paper receives good reviews from multiple reviewers and the authors have managed to address most of the concerns. The main contribution is a dataset to a new task that is of value. While there is concern of the limited application of the dataset (e.g., news recommendation), researchers from the community may discover more interesting applications this the availability of the new dataset. The authors are encouraged to include more details about the dataset in the final version.
Decision: Accept (Spotlight)