Introduction
On-line eSports events provide a new setting for observing large-scale social interaction focused on a visual story that evolves over time—a video game. While watching sporting competitions has been a major source of entertainment for millennia, and is a significant part of today's culture, eSports brings this to a new level on several fronts. One is the global reach, the same games are played around the world and across cultures by speakers of several languages. Another is the scale of on-line text-based discourse during matches that is public and amendable to analysis. One of the most popular games, League of Legends, drew 43 million views for the 2016 world series final matches (broadcast in 18 languages) and a peak concurrent viewership of 14.7 million. Finally, players interact through what they see on screen while fans (and researchers) can see exactly the same views.
This paper builds on the wealth of interaction around eSports to develop predictive models for match video highlights based on the audience's online chat discourse as well as the visual recordings of matches themselves. ESports journalists and fans create highlight videos of important moments in matches. Using these as ground truth, we explore automatic prediction of highlights via multimodal CNN+RNN models for multiple languages. Appealingly this task is natural, as the community already produces the ground truth and is global, allowing multilingual multimodal grounding.
Highlight prediction is about capturing the exciting moments in a specific video (a game match in this case), and depends on the context, the state of play, and the players. This task of predicting the exciting moments is hence different from summarizing the entire match into a story summary. Hence, highlight prediction can benefit from the available real-time text commentary from fans, which is valuable in exposing more abstract background context, that may not be accessible with computer vision techniques that can easily identify some aspects of the state of play. As an example, computer vision may not understand why Michael Jordan's dunk is a highlight over that of another player, but concurrent fan commentary might reveal this.
We collect our dataset from Twitch.tv, one of the live-streaming platforms that integrates comments (see Fig. FIGREF2 ), and the largest live-streaming platform for video games. We record matches of the game League of Legends (LOL), one of the largest eSports game in two subsets, 1) the spring season of the North American League of Legends Championship Series (NALCS), and 2) the League of Legends Master Series (LMS) hosted in Taiwan/Macau/HongKong, with chat comments in English and traditional Chinese respectively. We use the community created highlights to label each frame of a match as highlight or not.
In addition to our new dataset, we present several experiments with multilingual character-based models, deep-learning based vision models either per-frame or tied together with a video-sequence LSTM-RNN, and combinations of language and vision models. Our results indicate that while surprisingly the visual models generally outperform language-based models, we can still build reasonably useful language models that help disambiguate difficult cases for vision models, and that combining the two sources is the most effective model (across multiple languages).
Related Work
We briefly discuss a small sample of the related work on language and vision datasets, summarization, and highlight prediction. There has been a surge of vision and language datasets focusing on captions over the last few years, BIBREF0 , BIBREF1 , BIBREF2 , followed by efforts to focus on more specific parts of images BIBREF3 , or referring expressions BIBREF4 , or on the broader context BIBREF5 . For video, similar efforts have collected descriptions BIBREF6 , while others use existing descriptive video service (DVS) sources BIBREF7 , BIBREF8 . Beyond descriptions, other datasets use questions to relate images and language BIBREF9 , BIBREF10 . This approach is extended to movies in MovieQA.
The related problem of visually summarizing videos (as opposed to finding the highlights) has produced datasets of holiday and sports events with multiple users making summary videos BIBREF11 and multiple users selecting summary key-frames BIBREF12 from short videos. For language-based summarization, Extractive models BIBREF13 , BIBREF14 generate summaries by selecting important sentences and then assembling these, while Abstractive models BIBREF15 , BIBREF16 , BIBREF17 , BIBREF18 generate/rewrite the summaries from scratch.
Closer to our setting, there has been work on highlight prediction in football (soccer) and basketball based on audio of broadcasts BIBREF19 BIBREF20 where commentators may have an outsized impact or visual features BIBREF21 . In the spirit of our study, there has been work looking at tweets during sporting events BIBREF22 , but the tweets are not as immediate or as well aligned with the games as the eSports comments. More closely related to our work, yahooesports collects videos for Heroes of the Storm, League of Legends, and Dota2 on online broadcasting websites of around 327 hours total. They also provide highlight labeling annotated by four annotators. Our method, on the other hand, has a similar scale of data, but we use existing highlights, and we also employ textual audience chat commentary, thus providing a new resource and task for Language and Vision research. In summary, we present the first language-vision dataset for video highlighting that contains audience reactions in chat format, in multiple languages. The community produced ground truth provides labels for each frame and can be used for supervised learning. The language side of this new dataset presents interesting challenges related to real-world Internet-style slang.
Data Collection
Our dataset covers 218 videos from NALCS and 103 from LMS for a total of 321 videos from week 1 to week 9 in 2017 spring series from each tournament. Each week there are 10 matches for NALCS and 6 matches for LMS. Matches are best of 3, so consist of two games or three games. The first and third games are used for training. The second games in the first 4 weeks are used as validation and the remainder of second games are used as test. Table TABREF3 lists the numbers of videos in train, validation, and test subsets.
Each game's video ranges from 30 to 50 minutes in length which contains image and chat data linked to the specific timestamp of the game. The average number of chats per video is 7490 with a standard deviation of 4922. The high value of standard deviation is mostly due to the fact that NALCS simultaneously broadcasts matches in two different channels (nalcs1 and nalcs2) which often leads to the majority of users watching the channel with a relatively more popular team causing an imbalance in the number of chats. If we only consider LMS which broadcasts with a single channel, the average number of chats are 7210 with standard deviation of 2719. The number of viewers for each game averages about 21526, and the number of unique users who type in chat is on average 2185, i.e., roughly 10% of the viewers.
Model
In this section, we explain the proposed models and components. We first describe the notation and definition of the problem, plus the evaluation metric used. Next, we explain our vision model V-CNN-LSTM and language model L-Char-LSTM. Finally, we describe the joint multimodal model INLINEFORM0 -LSTM.
Conclusion
We presented a new dataset and multimodal methods for highlight prediction, based on visual cues and textual audience chat reactions in multiple languages. We hope our new dataset can encourage further multilingual, multimodal research.
Acknowledgments
We thank Tamara Berg, Phil Ammirato, and the reviewers for their helpful suggestions, and we acknowledge support from NSF 1533771.

Question: What was the baseline?
Answer: Unanswerable

Question: What is the average length of the recordings?
Answer: 40 minutes

Question: How big was the dataset presented?
Answer: 321 videos

Question: What were their results?
Answer: Best model achieved F-score 74.7 on NALCS and F-score of 70.0 on LMS on test set