SlowGuess/NeurIPS_2023_OCR huggingface dataset:
video feature extractors, such as MediaPipe YouTube-8M feature extractor<sup>1</sup>, can achieve similar performance and does not affect the main trend of our experimental results. We encourage to use the same featurization strategy for model comparison. It is worth noting that the input videos for a news topic can be considered as a set without any specific order. However, in our proposed models, we consider ordering the videos by their release time, as we have found through our ablation study that this prior knowledge is beneficial for improving performance significantly.
V-Transformer. Our first model uses Transformer to encode input videos. Specifically, the embeddings of input videos are ordered by the release time of the videos and such order information is incorporated into the model via positional encoding [Vaswani et al., 2017]. Such a Transformer encoder aims to learn the evolution dependencies among videos. The resulting representations are then fed into a fully connected layer to perform 24-class classification, where each class corresponds to a specific node ID on the timeline. The whole model is trained by the cross entropy loss. An illustration of this model is shown in Figure 6(a).
Tri-Transformer. The main limitation of V-Transformer is that the 24 classes are treated independently. However, the nodes on a timeline are ordered and their underlying dependencies should be explicitly captured. Motivated by this, we further propose to model nodes as learnable embeddings, and they are ordered according to the node order. As shown in Figure 6(b), the randomly initialized and learnable node embeddings and video embeddings are concatenated into a single sequence, which is the input for the first Transformer encoder. Note that they use two separate sets of positional encodings to indicate the node order and video release time order, respectively. Intuitively, the first Transformer encoder aims at capturing the interactions among all tokens, including node tokens and video tokens. With this module, the obtained node representations are dependent on the input video embeddings. In other words, the obtained node representations are customized for each input video set, which is intuitively more effective than video-set-agnostic node embeddings for the subsequent node assignment process. We further use two additional Transformer encoders to model the node-node interactions and video-video interactions, respectively. Next, we attend each video representation to all node representations. The attention score indicates the probability that the video belongs to the corresponding node. Inspired by pointer networks [Vinyals et al., 2015], such scores are then used by the cross entropy loss to train the whole model. Our model is named as Tri-Transformer since it consists of three Transformer encoders.
Tri-Transformer + cross-modal distillation. In our labeled video-associated timelines, as shown in Figure 3, we also have the text description for the nodes in addition to the assignments from videos to nodes. Such texts contain rich semantics of the nodes and are supposed to be informative for assigning videos to nodes. However, these privileged text descriptions are not available during inference but training only. These considerations raise a natural question. Is it possible to use the rich text information during training to improve inference performance?
$^{1}$ https://research.google.com/youtube8m
Table 2: Performance of the proposed methods on our benchmark in terms of node-level precision and recall, video-level Hamming distance, video-level Euclidean distance, and video pairwise agreement accuracy.  $\uparrow (\downarrow)$  denotes that higher (lower) value indicates better performance. The top two results in terms of each metric are highlighted as 1st and 2nd.
Method	Precision/Recall†	Hamming distance↓	Euclidean distance↓	Agreement accuracy†
Macro	Micro	Macro	Micro	Macro	Micro	Macro	Micro
V-Transformer	68.8%/68.9%	64.4%/63.0%	0.477	0.519	0.765	0.916	84.2%	93.8%
Tri-Transformer	71.4%/71.2%	66.6%/65.6%	0.453	0.490	0.716	0.837	85.4%	94.3%
Tri-Transformer + cross-modal distillation	72.5%/71.1%	68.4%/65.9%	0.436	0.470	0.684	0.804	85.2%	94.5%
A reference optimum	79.8%/78.3%	75.0%/72.9%	0.308	0.362	0.380	0.473	88.8%	95.5%
To answer this question, we propose to use knowledge distillation techniques [Hinton et al., 2015, Gou et al., 2021]. Specifically, as illustrated in Figure 6(c), we first train a teacher model, i.e., a Tri-Transformer model, that includes text embeddings as input to capture semantic knowledge contained in the node text descriptions. The text embeddings are still given by the pre-trained NewsEmbed model [Liu et al., 2021], and are added to the corresponding learnable node embeddings as input to the Tri-Transformer. The parameters of our teacher model are fixed after the training. Our student model is also a Tri-Transformer model, without text embeddings as input. To train the student model, in addition to the regular cross entropy loss, we have another knowledge distillation loss defined as the  $\mathcal{L}_2$  distance of the node and video representations between the student model and the pre-trained teacher model. We specifically focus on the node and video representations generated by the first Transformer encoder, as those in the teacher model contain the most essential information learned from the text embeddings and their interactions with videos. The final loss function is a weighted summation of the cross entropy loss and the distillation loss. After the training, only the student model is used for inference. The above knowledge distillation strategy transfers the learned text semantics knowledge from the teacher model to the student model, which does not receive text embeddings as input. Such distillation is cross-modal [Gupta et al., 2016] since the text modality is not available during inference.
Since we convert the timeline modeling problem to a classification problem in our approaches, there might end up with empty classes in our prediction results, thus leading to intermediate empty nodes on the timeline. To address this issue, we apply a post-processing step to skip those intermediate empty nodes. For instance, if the predicted node IDs for a 4-video input are  $\{1,1,4,2\}$ , we refine it to  $\{1,1,3,2\}$  by skipping the original third node, thereby forming a consecutive timeline. We leave the study of a more integrated approach for future work, as discussed in Section 6.
Setup. We use 2 Transformer layers for all Transformer encoders in our models. In our third method, we set the weight for the cross entropy loss and the distillation loss to be 1 and 0.1, respectively. We train all models for 100 epochs with the Adam optimizer [Kingma and Ba, 2015] and a batch size of 32. We did not tune the training hyperparameters extensively. The key hyperparameters learning rate and dropout rate are respectively selected from  $\{0.01, 0.001, 0.0005\}$  and  $\{0, 0.1, 0.25, 0.5\}$  based on the validation performance.
Overall results. The test performance of our proposed three approaches on YouTube-News-Timeline is present in Table 2. Although these three models are exploratory attempts at solving the video timeline modeling problem, they perform fairly well in terms of the correct identification of events and the correct evolution ordering of videos. For example, based on the micro average results, the models were able to recall over  $60\%$  of event nodes and correctly infer over  $90\%$  of video
Figure 7: Comparison of distributions of the number of videos per node and the number of nodes per timeline between ground truth and predictions given by Tri-Transformer over the testing subsets.
pairwise orders. In addition, the micro-average results of video-level Euclidean distance demonstrate
Table 3: Results of ablation studies. The table schema follows Table 2.
Method	Precision/Recall†	Hamming distance‡	Euclidean distance‡	Agreement accuracy†
Macro	Micro	Macro	Micro	Macro	Micro	Macro	Micro
Tri-Transformer	71.4%/71.2%	66.6%/65.6%	0.453	0.490	0.716	0.837	85.4%	94.3%
w/o video PE	50.1%/47.7%	43.4%/40.4%	0.621	0.647	1.523	2.105	58.3%	72.1%
w/o video PE and encoder 2&3	47.5%/43.3%	39.9%/37.7%	0.638	0.656	1.576	2.165	54.8%	72.0%
that, on average, the predicted node IDs deviate less than one absolute position from the ground truth node IDs. A comparison of the performance between V-Transformer and Tri-Transformer shows that explicitly modeling nodes and their underlying dependencies and interactions with videos can consistently achieve more effective performance. Furthermore, using knowledge distillation via Tri-Transformer + cross-model distillation can further boost performance by transferring the learned textual semantics from the teacher model to the student model.
Aside from quantitative results, we examine the distributions of the number of videos per node and the number of nodes per timeline between ground truth and predictions given by Tri-Transformer. As shown in Figure 7, such distributions of predicted results closely match those of the ground truth. It is important to note that the dataset sets a limit of five videos per node, but we do not enforce this constraint during prediction, and we rely solely on the model's raw predictions. This explains why our model sometimes assigns more than five videos to a single node. The visualizations of several predicted timelines are included in the appendix.
An optimum bound study. Our cross-modal distillation strategy involves using node texts as input in the teacher model and transferring the learned knowledge to the student model that does not have such input. This is because we assume that such textual node information is not available during inference. Given that we do have such textual node information in our test set of YouTube-News-Timeline dataset, we can evaluate the test performance of the teacher model that includes text information as input and treat such performance as a reference optimum of our Tri-Transformer model without using textual information. The results included in Table 2 reveal that, while our cross-modal distillation approach does improve the performance over Tri-Transformer, there is still an obvious performance gap compared to the reference optimum. Therefore, developing more effective ways to incorporate textual information during training is a potential future direction, as discussed in Section 6.
Ablation studies. We further perform ablation study experiments based on Tri-Transformer to evaluate two key designs. First, we analyze the effect of incorporating the prior knowledge of video release time. As described in Section 4, we use positional encodings (PE) to indicate the order of input videos according to their release time. In our ablation model, we remove such positional encodings and treat the input videos as a set instead. The results shown in Table 3 clearly demonstrate that this order information is a strong prior knowledge and significantly improves the performance. Additionally, we further remove Transformer encoders 2&3 and observe a decrease in performance, highlighting the importance of modeling the node-node interactions and video-video interactions.
In this work, we introduce a novel and important video timeline modeling task, which constructs a video-associated timeline from a set of videos, to help understand unstructured news videos. We provide a standardized testbed for this new task with a realistic dataset and a comprehensive evaluation protocol. We further develop and evaluate several intuitive approaches as simple baselines for future studies. The proposed benchmark dataset and methods will open the door for future research in video timeline modeling.
Here, we highlight several promising future directions in this field, including more principle exploration for problems and methodologies. On one hand, more effective and principle methodologies are highly desired. First, as demonstrated in the experiments, we can develop more effective strategies to leverage textual information during training. Second, while our current method, as an exploratory work, treats the problem as a multi-class classification problem, it would be more principle to consider the problem as a ranking problem and use differentiable ranking models. Third, our algorithms assign a set of videos simultaneously, which may result in different assignments for the same video if we add or delete a video in the set. It remains challenging to model video interactions while keeping the prediction of videos less dependent. This aspect presents a potential avenue for enhancement.
In addition, future research can also be made from a problem-oriented perspective. Given that our defined problem and evaluation standard concentrate solely on assigning input video sets to ordered nodes, one possible extension is to integrate the event summarization step with our defined problem to construct a timeline with both associated videos and event information. The event information can be in the form of text, key frames, etc. Although our YouTube-News-Timeline dataset can be used to evaluate the generated event text descriptions, the evaluation protocol and algorithms need to be dedicatedly redesigned. In addition, extending the single linear timeline to more complex relationship modeling, such as multiple timelines [Yu et al., 2021] and graphs, can be a promising future direction to enhance the understanding of news stories
The reference timelines used to construct the dataset are crawled from the web, and the videos are sourced from YouTube. The opinions expressed in the timelines and videos do not necessarily reflect our own, and we do not endorse or promote any specific viewpoint. The dataset is intended for research and educational purposes only, and users should exercise their own judgment when interpreting and using it.
We would like to express our gratitude to Anish Nair from Google who initiated and sponsored this project and as well as consistently provided constructive discussion throughout the delivery of the paper. We also thank the reviewers for their constructive feedback, which improved the quality of this work.
Rui Yan, Xiaojun Wan, Jahna Otterbacher, Liang Kong, Xiaoming Li, and Yan Zhang. Evolutionary timeline summarization: a balanced optimization framework via iterative substitution. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 745-754, 2011.
Yi Yu, Adam Jatowt, Antoine Doucet, Kazunari Sugiyama, and Masatoshi Yoshikawa. Multi-timeline summarization (mtls): Improving timeline summarization by generating multiple summaries. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 377-387, 2021.
Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman. Video summarization with long short-term memory. In European Conference on Computer Vision, pages 766-782. Springer, 2016.
The YouTube-News-Timeline dataset is under the CC BY 4.0 International license. Please refer to https://creativecommons.org/licenses/by/4.0/legalcode for license details.
B Data Format Example
A data format example is shown below.
Listing 1: A data example
"https://apnews.com/article/japan-accidents-tsunamis-earthquakes-42  $\rightarrow$  d4947609becd7f141e9524a8c98937": # The URL link of the webpage  $\rightarrow$  where we crawl the timeline.   
[   
[   
"OhEbGK4PnZg",   
"cl19tfn33hI",   
"R016z0HaUAM",   
"5QhCsR-t-qM",   
"ev3FBIoHMX8"   
],   
[   
"psAuFr8Xeqs",   
"BsRd7WQuBHc",   
"Dp_8rLL1Y18",   
"h1m7GFPAq3o"   
],   
[f4TaKPKe1gg",   
"DLlsKd-QC2o"   
],   
"ocluW1Vhvcg",   
"vusthiUFx_0",   
"vGHzuZQLYtg",   
"7XpLbhQxpLw",   
"UsPFUuzXisq4"   
],   
[hA3fNKOrxcs]   
] # The URL links of the retrieved YouTube news videos. Each list  $\leftrightarrow$  in the nested list corresponds to one node on the timeline.  $\leftrightarrow$  These nodes are ordered in the nested list.
C More Dataset Characteristics
We include more details on the dataset's characteristics here. As demonstrated in Figure 8, the reference timelines are curated from a diverse set of more than 1,000 publishers. The primary topics, each recurring more than 30 times, are depicted in In Figure 9. The topic annotations are given by an in-house proprietary entity linking algorithm. Moreover, the distribution of the event date, corresponding to each node on timelines, is shown in Figure 10.
D More Experimental Studies
To assess the impact of video duration on our model's performance, in Figure 11, we present a correlation between the video length and the corresponding video-level Euclidean distance, based on our Tri-Transformer model over the test set. The video-level Euclidean distance is not sensitive
Figure 8: Pie chart of the news publishers.
Figure 9: Distribution of the covered main topics.
to video length. It can be observed that the variance becomes large for longer videos as the longer videos are less in both the training and test sets.
E Examples of Predicted Timelines
We present two examples of timelines predicted by our Tri-Transformer model in Table 4 and 5. The videos that have been assigned to incorrect nodes are highlighted for clarity.
Figure 10: Distribution of the event date.
Figure 11: Number of videos and average video-level Euclidean distances in relation to video duration.
Table 4: A timeline of "Major milestones in Chinese space exploration".
Table 5: A timeline of "50 Years of Title IX: The Defining Moments of Women's Sports".
F Societal Impacts
We expect that the proposed benchmark dataset and methods will facilitate future advancements in video timeline modeling. On the positive side, it offers a helpful tool for understanding and navigating large volumes of news video data, enabling more efficient news consumption and ensuring a more comprehensive understanding of events.
One the negative side, the crawled timelines might not always reflect absolute precision. They might be mistakenly used as evidence in some situations. This highlights the importance of using and interpreting these timelines carefully. In addition, while we have taken steps to diversify our data sources, news content can inherently carry biases based on various factors. We acknowledge this challenge and emphasize the future need for more diverse data sourcing to capture a broad spectrum of perspectives and reduce inherent biases. Also, there could be potential misuse if the technology were applied unethically. Specifically, the construction of timelines could be manipulated to present events in a way that supports a particular opinion, thereby distorting the truth. To alleviate this potential concern, regular validation and fact-checking mechanisms can help ensure the constructed timelines align with factual occurrences.