Venue: NAACL 2019

# Sub-Event Detection from Twitter Streams as

a Sequence Labeling Problem

 Giannis Bekoulis  Johannes Deleu   Thomas Demeester  Chris Develder

Ghent University - imec, IDLab

Department of Information Technology

firstname.lastname@ugent.be

###### Abstract

This paper introduces improved methods for sub-event detection in social media streams, by applying neural sequence models not only on the level of individual posts, but also directly on the stream level. Current approaches to identify sub-events within a given event, such as a goal during a soccer match, essentially do not exploit the sequential nature of social media streams. We address this shortcoming by framing the sub-event detection problem in social media streams as a sequence labeling task and adopt a neural sequence architecture that explicitly accounts for the chronological order of posts. Specifically, we (i) establish a neural baseline that outperforms a graph-based state-of-the-art method for binary sub-event detection (2.7% micro-F\({}_{1}\) improvement), as well as (ii) demonstrate superiority of a recurrent neural network model on the posts sequence level for labeled sub-events (2.4% bin-level F\({}_{1}\) improvement over non-sequential models).

## 1 Introduction

Social media allow users to communicate via real-time postings and interactions, with Twitter as a notable example. Twitter user posts, i.e., tweets, are often related to events. These can be social events (concerts, research conferences, sports events, etc.), emergency situations (e.g., terrorist attacks) Castillo (2016), etc. For a single event, multiple tweets are posted, by people with various personalities and social behavior. Hence, even more so than (typically more neutral) traditional media, this implies many different perspectives, offering an interesting aggregated description.

Given this continuous and large stream of (likely duplicated) information in Twitter streams, and their noisy nature, it is challenging to keep track of the main parts of an event, such as a soccer match. Automating such extraction of different sub-events within an evolving event is known as sub-event detection Nichols et al. (2012). For tracking each of the sub-events, the timing aspect is an important concept (i.e., consecutive tweets in time). Thus, a sequential model could successfully exploit chronological relations between the tweets in a Twitter stream as an informative feature for sub-event detection.

Several methods have been proposed for sub-event detection: clustering methods Pohl et al. (2012), graph-based approaches Meladianos et al. (2015), topic models Xing et al. (2016) and neural network architectures Wang and Zhang (2017). None of these studies exploits the chronological relation between consecutive tweets. In contrast, our work does take into account that chronological order and we predict the presence and the type of a sub-event exploiting information from previous tweets. Specifically, we (i) propose a new neural baseline model that outperforms the state-of-the-art performance on the binary classification problem of detecting the presence/absence of sub-events in a sports stream, (ii) establish a new reasonable baseline for predicting also the sub-event _types_, (iii) explicitly take into account chronological information, i.e., the relation among consecutive tweets, by framing sub-event detection as a sequence labeling problem on top of our baseline model, and (iv) perform an experimental study, indicating the benefit of sequence labeling for sub-event detection in sports Twitter streams.

## 2 Related Work

Twitter streams have been extensively studied in various contexts, such as sentiment analysis Kouloumpis et al. (2011), stock market prediction Nguyen and Shirai (2015) and traffic detection D'Andrea et al. (2015). Specifically, for sub-event detection in Twitter, several approaches

have been tried. _Unsupervised methods_ such as clustering aim to group similar tweets to detect specific sub-events (Pohl et al., 2012; Abhik and Toshniwal, 2013) and use simple representations such as tf-idf weighting combined with a similarity measure. Other unsupervised algorithms use topic modeling approaches, based on assumptions about the tweets' generation process (Xing et al., 2016; Srijith et al., 2017). Several methods (Zhao et al., 2011; Zubiaga et al., 2012; Nichols et al., 2012) assume that a sub-event happens when there is a 'burst', i.e., a sudden increase in the rate of tweets on the considered event, with many people commenting on it. Recently, neural network methods have used more complicated representations (Wang and Zhang, 2017; Chen et al., 2018). Also _supervised methods_ have been applied (Sakaki et al., 2010; Meladianos et al., 2018) for the sub-event detection task. These methods usually exploit graph-based structures or tf-idf weighting schemes. We believe to be the first to (i) exploit the chronological order of the Twitter stream and take into account its sequential nature, and (ii) frame the sub-event detection problem as a sequence labeling task.

## 3 Model

### Task Definition

The goal is, given a main event (i.e., soccer match), to identify its core sub-events (e.g., goals, kick-off, yellow cards) from Twitter streams. Specifically, we consider a _supervised_ setting, relying on annotated data (Meladianos et al., 2018).

### Word- vs Tweet-Level Representations

Similar to previous works, we split a data stream into time periods (Meladianos et al., 2018): we form bins of tweets posted during consecutive time intervals. E.g., for a soccer game, one-minute intervals (bins) lead to more than 90 bins, depending on the content before and after the game, halftime, stoppage time, and possibly some pregame and post-game buffer. Thus, for each bin, we predict either the presence/absence of a sub-event (Section 3.3) or the most probable sub-event type (Section 3.4), depending on the evaluation scenario.

We consider representing the content of each bin either on (i) a word-level or (ii) a tweet-level (see Fig. 1). Formally, we assume that we have a set of \(n\) bins \(b_{1},...,b_{n}\), where each bin \(b_{i}\) consists of \(m_{i}\) tweets and \(k_{i}\) words (i.e., all words of tweets in bin \(b_{i}\)). Then, the _tweet-level representation_ of bin \(b_{i}\) is symbolized as \(t_{i1},...,t_{im_{i}}\), where \(t_{im_{i}}\) is the \(m^{\text{th}}_{i}\) tweet of bin \(b_{i}\). In the _word-level representation_, we chronologically concatenate the words from the tweets in the bin: \(w_{i1},..,w_{ik_{i}}\), where \(w_{ik_{i}}\) is the \(k^{\text{th}}_{i}\) word of bin \(b_{i}\).

### Binary Classification Baseline

To compare with previous work (Meladianos et al., 2018), we establish a simple baseline for binary classification: presence/absence of a sub-event. For this case, we use as input the word-level representation of each bin. To do so, we use word embeddings (randomly initialized) with average (AVG) pooling (Iyyer et al., 2015) in combination with a multilayer perceptron (MLP) for binary classification, i.e., presence/absence of a

Figure 1: Our sub-event detection model comprises: (a) a bin layer, (b) a unit layer, (c) a word embeddings layer, (d) a representation layer and (e) a **chronological LSTM** layer to model the natural flow of the sub-events within the event. We represent each bin using either (i) a tweet- or (ii) a **word**-level representation. The AVG\({}^{*}\) represents an average pool operation, performed either directly on the embeddings or on the tweet’s LSTM representation.

 

sub-event. Note that we experimented with pre-trained embeddings as well as max-pooling, but those early experiments led to performance decrease compared to the presented baseline model. We found that training based on average bin representations works substantially better than with max-pooling, and we hypothesize that this is related to the noisy nature of the Twitter stream.

### Sequence Labeling Approach

Building on the baseline above, we establish a new architecture that is able to capture the sub-event types as well as their duration. We phrase sub-event detection in Twitter streams as a sequence labeling problem. This means we assume that the label of a bin is not independent of neighboring bin labels, given the chronological order of bins of the Twitter stream, as opposed to independent prediction for each bin in the binary classification baseline above. For instance, when a _goal_ is predicted as a label for bin \(b_{i}\), then it is probable that the label of the next bin \(b_{i+1}\) will also be _goal_. Although a sub-event may occur instantly, an identified sub-event in a Twitter stream can span consecutive bins, i.e., minutes: users may continue tweeting on a particular sub-event for relatively long time intervals. For this reason, we apply the well-known BIO tagging scheme Ramshaw and Marcus (1995) for the sub-event detection problem. For example, the beginning of a _goal_ sub-event is defined as B-_goal_, while I-_goal_ (inside) is assigned to every consecutive bin within the same sub-event, and the O tag (outside) to every bin that is not part of any sub-event. To propagate chronological information among bins, we adopt an LSTM on the sequence of bins as illustrated in Fig. 1, layer (e). Note that this tagging approach assumes that sub-events do not overlap in time, i.e., only at most one is ongoing in the Twitter stream at any point in time.

## 4 Experimental Setup

We evaluated our system1 on the dataset from Meladianos et al. (2018), with tweets on 20 soccer matches from the 2010 and 2014 FIFA World Cups, totalling over 2M pre-processed tweets filtered from 6.1M collected ones, comprising 185 events. The dataset includes a set of sub-events, such as _goal_, _kick-off_, _half-time_, etc. To compare our binary classification _baseline system_ to previous methods (Table 1), we use the same train/test splits as Meladianos et al. (2018), where 3 matches are used for training and 17 matches as test set. In this setting, we predict only the presence/absence of a sub-event. Similar to previous work, we count a sub-event as correct if at least one of its comprising bins has been classified as a sub-event. For the experimental study of our proposed _sequence labeling approach_ for sub-event detection, where sub-event types are predicted, we have randomly split the test set into test (10 matches) and development (7 matches) sets. We use the development set to optimize the F\({}_{1}\) score for tuning of the model parameters, i.e., the word/tweet embedding representation size, LSTM hidden state size, dropout probability. We adopt 2 evaluation strategies. The first one, referred to as _relaxed_ evaluation, is commonly used in entity classification tasks Adel and Schutze (2017); Bekoulis et al. (2018a, c) and similar to the binary classification baseline system evaluation: score a multi-bin sub-event as correct if at least one of its comprising bin types (e.g., _goal_) is correct, assuming that the boundaries are given. The second evaluation strategy, _bin-level_, is stricter: we count each bin individually, and check whether its sub-event type has been predicted correctly, similar to the token-based evaluation followed in Bekoulis et al. (2018b).

Footnote 1: [https://github.com/bekou/subevent_sequence_labeling](https://github.com/bekou/subevent_sequence_labeling)

## 5 Results

### Baseline Results

Table 1 shows the experimental results of our baseline model. The Burst baseline system is based on the tweeting rate in a specific time window (i.e., bin) and if a threshold is exceed, the system identifies that a sub-event has occurred. We report evaluation scores as presented in Meladianos et al. (2018). The second approach is the graph-based method of Meladianos et al. (2018). We observe that our baseline system (Section 3.3) has a 1.2% improvement in terms of macro-F\({}_{1}\) and 2.7% improvement in terms of micro-F\({}_{1}\), compared to the graph-based model from Meladianos et al. (2018), mainly due to increased precision, and despite the recall loss.

### Sequence Labeling Results

Table 2 illustrates the predictive performance of our proposed model (i.e., using the chronological



LSTM) compared to models making independent predictions per bin. The upper part of Table 2 contains models without the chronological LSTM. Our experiments study both _word-level_ and _tweet-level_ bin representations (see Fig. 1), as reflected in the 'Word' vs. 'Tweet' prefix, respectively, in the Model column of Table 2.

The simplest _word-level_ representation uses the tf-idf weighting scheme (as in Pohl et al. (2012)) followed by an MLP classifier. For the other word-level models, we exploit several architectures: AVG pooling (Iyyer et al., 2015), a CNN followed by AVG pooling (Kim, 2014) and hierarchical word-level attention (Yang et al., 2016).

For _tweet-level_ representations, we adopt similar architectures, where the AVG, CNNs and attention are performed on sentence level rather than on the word-level representation of the bin. In this scenario, we have also exploited the usage of sequential LSTMs to represent the tweets. When comparing models with and without tweet-level LSTMs, we report the strategy that yields the best results, indicated by and in the tweet-level LSTM (TL) columns of Table 2. We do not present results for applying sequential LSTMs on the word-level bin representation, because of slow training on the long word sequences.

**Benefit of Chronological LSTM:** The bottom part of Table 2 presents the results of the same models followed by a chronological LSTM to capture the natural flow of the stream as illustrated in Fig. 1. We report results as described in Section 4, using the micro F\({}_{1}\) score with the two evaluation strategies (_bin-level_ and _relaxed_). We observe that when using the chronological LSTM, the performance in terms of _bin-level_ F\({}_{1}\) score is substantially improved for almost every model. Note that the best model using the chronological LSTM (Tweet-AVG) achieves 2.4% better F\({}_{1}\) than the best performing model without the use of chronological LSTM (Word-CNN-AVG). In most cases there is also a consistent improvement for both the precision and the recall metrics, which is thanks to the sequential nature of the upper level LSTM capturing the flow of the text.

**Limitations of _Relaxed_ Evaluation:** On the other hand, using the _relaxed_ evaluation strategy, we observe that the best models are those without the chronological LSTM layer. Yet, we consider the _relaxed_ evaluation strategy flawed for our scenario, despite the fact that it has been used for entity classification tasks (Bekoulis et al., 2018; Adel and Schutze, 2017). Indeed, it is not able to properly capture sub-events which are characterized by duration: e.g., if a model assigns a different label to each of the bins that together constitute a single sub-event, then this sub-event counts as a true positive based on the _relaxed_ evaluation strategy (similar to the evaluation proposed by Meladianos et al. (2018) and followed in Table 1). Thus, in this work, we propose to use the _bin-level_ evaluation, since it is a more natural way to measure the duration of a sub-event in a supervised sequence labeling setting. Note that due to the noisy nature of Twitter streams, a tweet sequence spanning a particular sub-event is likely to contain also tweets that are not related to the given sub-event: a given bin inside the event may contain only a minority of tweets discussing the event. Therefore, we consider the standard sequence labeling evaluation (requiring to have types as well as boundaries correct) to be not applicable in sub-event detection.

**Performance Comparison of the Top-3 Models:** Figure 2 shows the performance of our three best performing models in terms of _bin-level_ F\({}_{1}\) score on the validation set. The best performing model is the Tweet-AVG model since it attains its maximum performance even from the first training epochs. The Word-AVG model performs well

\begin{table}
\begin{tabular}{c c r r r r r r} \hline  & \multicolumn{3}{c}{Micro} & \multicolumn{3}{c}{Macro} \\ Settings & \multicolumn{1}{l}{P} & R & F\({}_{1}\) & P & R & F\({}_{1}\) \\ \hline Burst & 78.00 & 54.00 & 64.00 & 72.00 & 54.00 & 62.00 \\ Melianos et al. (2018) & 76.00 & 75.00 & 75.00 & 73.00 & 74.00 & 73.00 \\
**Our binary density: baseline** & 89.70 & 69.99 & **76.16** & 33.65 & 69.05 & **75.65** \\ \hline \end{tabular}
\end{table}
Table 1: Comparing our neural network binary classification baseline model to state-of-the-art (P = precision, R = recall).

\begin{table}
\begin{tabular}{c l r r r r r r} \hline  & \multicolumn{3}{c}{Black-level} & \multicolumn{3}{c}{Related} \\ \cline{2-10}  & Model & TL & PE & R & F\({}_{1}\) & TL & PE & R & F\({}_{1}\) \\ \hline \multirow{5}{*}{Meliations} & Would-off & – & 49.40 & 52.06 & 50.69 & – & 56.10 & 56.10 \\  & Would-AVG & – & 54.10 & 45.96 & 48.53 & – & 56.10 & 56.10 \\  & Week-CNN & – & 56.93 & 56.01 & 56.47 & – & 75.00 & 75.60 \\  & Word-attention & – & 52.82 & 58.71 & 55.66 & – & 86.59 & 86.59 \\  & Tweet-AVG & ✓ & 49.94 & 45.96 & 47.45 & – & 61.92 & 61.92 & 61.19 \\  & Tweet-Attention & ✓ & 51.99 & 42.37 & 46.68 & ✗ & ✗ & ✗ & ✗ & ✗ \\  & Tweet-CNN & ✗ & 58.88 & 51.17 & 54.75 & ✗ & ✗ & ✗ & ✗ & ✗ \\ \hline \multirow{5}{*}{Wired-Vendor} & Would-Vendor & – & 58.14 & 58.53 & 58.24 & – & 71.95 & 71.95 & 71.95 \\  & Would-CNN-AVG & – & 60.93 & 56.19 & 58.45 & – & 60.97 & 60.97 & 60.97 \\  & Word-attention & – & 52.99 & 42.90 & 47.42 & – & 60.97 & 60.97 & 60.97 \\  & Tweet-AVG & ✓ & 57.43 & 60.32 & **68.84** & ✗ & ✗ & 64.63 & 64.63 \\  & Tweet-attention & ✓ & 48.26 & 52.24 & 50.17 & ✗ & 67.07 & 67.07 & 67.07 \\  & Tweet-CNN & ✗ & 65.33 & 49.73 & 56.47 & ✗ & ✗ & ✗ & ✗ & ✗ \\ \hline \end{tabular}
\end{table}
Table 2: Comparison of our baseline methods in terms of micro _bin-level_ and _relaxed_ F\({}_{1}\) score with and without chronological LSTM (see Fig. 1). The ✓and ✗ indicate whether the model uses a tweet-level LSTM (TL).

 

from the first epochs, showing similar behavior to the Tweet-AVG model. This can be explained by the similar nature of the two models. The word-level CNN model attains maximum performance compared to the other two models in later epochs. Overall, we propose the use of the chronological LSTM with the Tweet-AVG model since this model does not rely on complex architectures and it gives consistent results.

## 6 Conclusion

In this work, we frame the problem of sub-event detection in Twitter streams as a sequence labeling task. Specifically, we (i) propose a binary classification baseline model that outperforms state-of-the-art approaches for sub-event detection (presence/absence), (ii) establish a strong baseline that additionally predicts sub-event _types_, and then (iii) extend this baseline model with the idea of exchanging chronological information between sequential posts, and (iv) prove it to be beneficial in almost all examined architectures.

## Acknowledgements

We would like to thank the anonymous reviewers for their constructive feedback. Moreover, we would like to thank Christos Xypolopoulos and Giannis Nikolentzos for providing (i) the Twitter dataset (tweet ids) and (ii) instructions to reproduce the results of their graph-based approach.

## References

* Abhik and Toshniwal (2013) Dhekar Abhik and Durga Toshniwal. 2013. Sub-event detection during natural hazards using features of social media data. In _Proceedings of the 22nd International Conference on World Wide Web_, pages 783-788, New York, NY, USA. ACM.
* Adel and Schutze (2017) Heike Adel and Hinrich Schutze. 2017. Global normalization of convolutional neural networks for joint entity and relation classification. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 1723-1729. Association for Computational Linguistics.
* Bekoulis et al. (2018a) Giannis Bekoulis, Johannes Deleu, Thomas Demeester, and Chris Develder. 2018a. Adversarial training for multi-context joint entity and relation extraction. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2830-2836. Association for Computational Linguistics.

* Castillo (2016) Carlos Castillo. 2016. _Big Crisis Data: Social Media in Disasters and Time-Critical Situations_. Cambridge University Press.
* Chen et al. (2018) Guandan Chen, Nan Xu, and Weiji Mao. 2018. An encoder-memory-decoder framework for sub-event detection in social media. In _Proceedings of the 27th ACM International Conference on Information and Knowledge Management_, pages 1575-1578, New York, NY, USA. ACM.
* D'Andrea et al. (2015) Eleonora D'Andrea, Pietro Ducange, Beatrice Lazzerini, and Francesco Marcelloni. 2015. Real-time detection of traffic from Twitter stream analysis. _IEEE Transactions on Intelligent Transportation Systems_, 16(4):2269-2283.
* Iyyer et al. (2015) Mohit Iyyer, Varun Manjunatha, Jordan Boyd-Graber, and Hal Daume III. 2015. Deep unordered composition rivals syntactic methods for text classification. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1681-1691. Association for Computational Linguistics.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1746-1751. Association for Computational Linguistics.
* Kouloumpis et al. (2011) Efthymios Kouloumpis, Theresa Wilson, and Johanna Moore. 2011. Twitter sentiment analysis: The good

Figure 2: _Bin-level F\({}_{1}\)_ performance of the three best performing models on the validation set with respect to the number of epochs. The smoothed lines (obtained by LOWESS smoothing) model the trends and the 95% confidence intervals.

 

the bad and the omg! In _Proceedings of the Fifth International AAAI conference on weblogs and social media_, pages 538-541.
* Meladianos et al. (2015) Polykarpos Meladianos, Giannis Nikolentzos, Franois Rousseau, Yannis Stavrakas, and Michalis Vazirgiannis. 2015. Degeneracy-based real-time subevent detection in Twitter stream. In _Proceedings of the 9th International AAAI Conference on Web and Social Media_, pages 248-257. AAAI Press.
* Meladianos et al. (2018) Polykarpos Meladianos, Christos Xypolopoulos, Giannis Nikolentzos, and Michalis Vazirgiannis. 2018. An optimization approach for sub-event detection and summarization in Twitter. In _Proceedings of the 40th European Conference in Information Retrieval_, pages 481-493. Springer International Publishing.
* Nguyen and Shirai (2015) Thien Hai Nguyen and Kiyoaki Shirai. 2015. Topic modeling based sentiment analysis on social media for stock market prediction. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1354-1364. Association for Computational Linguistics.
* Nichols et al. (2012) Jeffrey Nichols, Jalal Mahmud, and Clemens Drews. 2012. Summarizing sporting events using Twitter. In _Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces_, pages 189-198, New York, NY, USA. ACM.
* Pohl et al. (2012) Daniela Pohl, Abdelhamid Bouchachia, and Hermann Hellwagner. 2012. Automatic sub-event detection in emergency management using social media. In _Proceedings of the 21st International Conference on World Wide Web_, pages 683-686, New York, NY, USA. ACM.
* Ramshaw and Marcus (1995) Lance Ramshaw and Mitch Marcus. 1995. Text chunking using transformation-based learning. In _Third Workshop on Very Large Corpora_.
* Sakaki et al. (2010) Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010. Earthquake shakes Twitter users: real-time event detection by social sensors. In _Proceedings of the 19th international conference on World wide web_, pages 851-860. ACM.

* Wang and Zhang (2017) Zhongqing Wang and Yue Zhang. 2017. A neural model for joint event detection and summarization. In _Proceedings of the 26th International Joint Conference on Artificial Intelligence_, pages 4158-4164. AAAI Press.
* Xing et al. (2016) Chen Xing, Yuan Wang, Jie Liu, Yadou Huang, and Wei-Ying Ma. 2016. Hashtag-based sub-event discovery using mutually generative LDA in Twitter. In _Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence_, pages 2666-2672. AAAI Press.
* Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1480-1489. Association for Computational Linguistics.
* Zhao et al. (2011) Siqi Zhao, Lin Zhong, Jehan Wickramasuriya, and Venu Vasudevan. 2011. Human as real-time sensors of social and physical events: A case study of Twitter and sports games. _arXiv preprint arXiv:1106.4300_.
* Zubiaga et al. (2012) Arkaitz Zubiaga, Damiano Spina, Enrique Amigo, and Julio Gonzalo. 2012. Towards real-time summarization of scheduled events from Twitter streams. In _Proceedings of the 23rd ACM Conference on Hypertext and Social Media_, pages 319-320, New York, NY, USA. ACM.