Venue: NAACL 2016

# Expectation-Regulated Neural Model for Event Mention Extraction

Ching-Yun Chang & Zhiyang Teng & Yue Zhang

Singapore University of Technology and Design

8 Somapah Road, Singapore 487372

{chingyun_chang, yue_zhang}@sutd.edu.sg

zhiyang_teng@mymail.sutd.edu.sg

###### Abstract

We tackle the task of extracting tweets that mention a specific event from all tweets that contain relevant keywords, for which the main challenges include unbalanced positive and negative cases, and the unavailability of manually labeled training data. Existing methods leverage a few manually given seed events and large unlabeled tweets to train a classifier, by using expectation regularization training with discrete ngram features. We propose a LSTM-based neural model that learns tweet-level features automatically. Compared with discrete ngram features, the neural model can potentially capture non-local dependencies and deep semantic information, which are more effective for disambiguating subtle semantic differences between true event mentions and false cases that use similar wording patterns. Results on both tweets and forum posts show that our neural model is more effective compared with a state-of-the-art discrete baseline.

## 1 Introduction

A Distributed Denial of Service (DDoS) attack employs multiple compromised systems to interrupt or suspend services of a host connected to the Internet. Victims are often high-profile web servers such as banks or credit card payment gateways, and therefore a single attack may cause considerable loss. The aim of this paper is to build an automatic system which can extract DDoS event mentions from social media, a timely information source for events taking place around the world, so that the mined emerging incidents can serve as early DDoS warnings or signs for Internet service providers.

Ritter et al. Ritter:2015 proposed the first work to extract cybersecurity event mentions from raw Twitter stream. They investigated three different event categories, namely _DDoS attacks_, _data breaches_ and _account hijacking_, by tracking the keywords _ddos_, _breach_ and _hacked_, respectively. Not all tweets containing the keywords describe events. For example, the tweet "give me paypall or i will tell my mum and _ddos_ u" shows a metaphor rather than a DDoS event. As a result, the event mention extraction task involves a classification task that filters out true events from all tweets that contain event keywords. Two main challenges exist for this task. First, the numbers of positive and negative examples are typically unbalanced. In our datasets, only about 22% of the tweets that contain the term _ddos_ are mentions to DDoS attack events. Second, there is typically little manual annotation available. Ritter et al. Ritter:2015 tackled the challenges by weakly supervising a classification model with a small number of human-provided seed events.

In particular, Ritter et al. exploit expectation regularization ER; Mann and McCallum (2007)) for semi-supervised learning from large amounts of raw tweets that contain the event keyword. They show that the ER approach outperforms semi-supervised expectation-maximization and one-class support vector machine on the task. They build a logistic regression classifier, using few human-labeled seed events and domain knowledge on the ratio between positive and negative examples for ER in training. Results show that the regulariza

tion method was effective on classifying unbalanced datasets.

Ritter et al. use manually-defined discrete features. However, the event mention extraction task is highly semantic-driven, and simple textual patterns may suffer limitations in representing subtle semantic differences between true event mentions and false cases with similar word patterns. Recently, deep learning received increasing research attention in the nlp community [13, 26, 45]. One important advantage of deep learning is automatic representation learning, which can effectively encodes syntactic and information about words, phrases and sentences in low-dimensional dense vectors.

In this paper we exploit a deep neural model for event mention extraction, using word embeddings and a novel LSTM-based neural network structure to automatically obtain features for a tweet. Results on two human-annotated datasets show that the proposed LSTM-based representation yields significant improvements over Ritter et al. ritter:2015:EMNLP.

## 2 Related Work

In terms of scope, our work falls into the area of information extraction from social media [8, 9]. The proposed event mention extraction system is domain-specific, similar to works that aim at detecting categorized events such as disaster outbreak [7, 12, 13] and cybersecurity events [14]. Such work typically trains semi-supervised classifiers to determine events of interest due to the limitation of annotated data. On the other hand, a few studies devote to open domain event extraction [13, 15, 16, 17, 18, 19], in which an event category is not predefined, and clustering models are applied to automatically induce event types.

In terms of method, the proposed model is in line with recent methods on deep learning for neural feature representations, which have seen success in some nlp tasks [15, 16, 17, 18]. Competitive results have been obtained in sentiment analysis [15, 16, 17, 18], semantic relation classification [15, 16], and question answering [13, 14]. In addition, deep learning models have shown promising results on syntactic parsing [14, 15] and machine translation [8]. Compared to syntactic problems, semantic tasks see relatively larger improvements by using neural architectures, possible because of the capability of neural features in better representing semantic information, which is relatively more difficult to capture by discrete indicator features. We consider event mention extraction as a semantic-heavy task and demonstrate that it can benefit significantly from neural feature representations.

## 3 Baseline

We take the method of Ritter et al. ritter:2015:EMNLP as a baseline. Given a tweet containing the keyword _ddos_, the task is to determine whether a DDoS attack event is mentioned in the tweet. A logistic regression classifier is used, which is trained by maximum-likelihood with ER on unlabeled tweets, and automatically generated positive examples from a few seed events.

### Seed Events

Ritter et al. ritter:2015:EMNLP manually pick seed events, represented as (entity, date) tuples, and treated tweets published on date referencing entity as positive training instances. For example, (_GitHub_, 2013 July 29)1 is defined as a seed DDoS event, and the tweet "@amosiu_GitHub_ is experiencing a large DDos [https://t.co/cqEIR6Rz6t](https://t.co/cqEIR6Rz6t)" posted on 2013 July 29 is seen as an event mention since it contains the Entity _GitHub_ as well as matches the Date 2013 July 29. Those tweets with the word _ddos_ but not matching any seed events are grouped as unlabeled data.

Footnote 1: [https://status.github.com/messages/2013-07-29](https://status.github.com/messages/2013-07-29)

### Sparse Feature Representation

Each tweet is represented by a sparse binary vector for feature extraction, where the features consist of bi- to five-grams containing a name entity or the event keyword. For 

words other than common nouns and verbs are replaced with their part-of-speech (POS) tags. Table 1 shows an example of contextual features extracted from the tweet "@amosie GitHub is experiencing a large DDos [https://t.co/cqEIR6Rz6t](https://t.co/cqEIR6Rz6t)". As can be seen from the table, the features contain shallow wording patterns from a tweet, which are local to a 5-word window. In contrast, the observed average tweet length is 16 words, with the longest tweet containing 48 words, which is difficult to fully represent using only a local window. Our neural model addresses the limitations by learning global tweet-level syntactic and semantic features automatically.

### Logistic Regression Classification with Expectation Regularization

With the feature vector \(\vec{f}_{s}\in\mathbb{R}^{d}\) defined for a given tweet \(s\), the probability of \(s\) being an event mention is defined as:

\[p_{\theta}(y=1|s)=\frac{1}{1+e^{-\vec{\theta}\vec{f}_{s}^{\prime}}} \tag{1}\]

where \(\vec{\theta}\in\mathbb{R}^{d}\) is a weight vector.

Given a set of event mentions \(M=\langle m_{1},m_{2},...,m_{j}\rangle\) and a set of unlabeled instances \(U=\langle u_{1},u_{2},...,u_{k}\rangle\), Ritter et al. (2015) train an ER model that maximizes the log-likelihood of positive data while keeping the conditional probabilities on unlabeled data consistent with the human-provided expectations. The objective function is defined as:

\[O(\theta;M,U) =\underbrace{\sum_{m\in M}\log p_{\theta}(y=1|m)}_{Log~{}Likelihood} \tag{2}\] \[-\underbrace{\lambda^{U}\Delta(\tilde{p},\hat{p}_{\theta}^{U})}_ {Expectation~{}Regularization}\] \[-\underbrace{\lambda^{L^{2}}\lVert\theta\rVert^{2}}_{L^{2}~{} Regularization}\]

The expectation regularization term \(\Delta(\tilde{p},\hat{p}_{\theta}^{U})\) is defined as the KL divergence between the model's posterior predictions on unlabeled data, \(\hat{p}_{\theta}^{U}\), and the human-provided label expectation priors, \(\tilde{p}\):

\[\begin{split}\Delta(\tilde{p},\hat{p}_{\theta}^{U})& =D(\tilde{p}||\hat{p}_{\theta}^{U})\\ &=\tilde{p}~{}\log\!\frac{\tilde{p}}{\hat{P}_{\theta}^{U}}+(1- \tilde{p})~{}\log\!\frac{1-\tilde{p}}{1-\hat{p}_{\theta}^{U}}\end{split} \tag{3}\]

## 4 Distant Seed Event Extraction

We follow Ritter et al. (2015), using a set of seed events and large raw tweets for ER. However, we take a fully-automated approach to find seed events, since manual listing of seed DDoS events can be a costly and time consuming process, and requires a certain level of expert knowledge.

We leverage news articles to collect seed events, representing events as (entity, date range) tuples. The entity in our seed events is defined as a name entity that appears in either the _assailant_ or _victim_ role of an _attack_ event labeled by frame-semantic parsing, and the date range is a date window around the news publication date. We use a date window rather than a definite news publication date because news articles are not always published on the day a DDoS attack happened. Some examples are given in Figure 1.

We parse DDoS attack news collected from [http://www.ddosattacks.net2](http://www.ddosattacks.net2) with a state-of-the-art frame-semantic parsing system (SEMAFOR; Das et al. (2010)). Tweets are gathered using the Twitter Streaming API3 with a case-insensitive track keyword _ddoS_. Name entities are extracted from both news articles and tweets using a Twitter-tuned NLP pipeline (Ritter et al., 2011).4

Footnote 2: Most of the articles are about DDoS attack events, while a smaller number discusses the nature of DDoS attacks and related issues.

Footnote 3: [https://dev.twitter.com/streaming/overview](https://dev.twitter.com/streaming/overview)

Footnote 4: [https://github.com/aritter/twitter](https://github.com/aritter/twitter) ̇nlp

Table 2 shows two example DDoS attack news, where the entity values are included in the victim roles, _RBS_, _Ulster Bank_, _GovCERT_ and _FBI_ in the first news, and _Essex_ in the second. It is worth noting that the DDoS attack on RBS, Ulster Bank and Natwest was actually on 2015 July 31. The correlation between tweet mentions and news reports are shown in Figure 1, where each bar indicates the

\begin{table}
\begin{tabular}{l l} ne: GitHub & keyword: DDoS \\ \hline \hline USRne & JJ DDoS \\ ineis & DDoS URL \\ USRne is & DT JJDoS \\ ne is experiencing & JJ DDoS URL \\ USRne is experiencing & experiencing DT JJ DDoS \\ ne is experiencing DT & UT JJDoS URL \\ USRne is experiencing DT & is experiencing DT JJ DDoS \\ UE is experiencing DT JJ & experiencing DT JJ DDoS URL \\ \hline \end{tabular}
\end{table}
Table 1: Features of a tweet by Ritter et al. (2015).

 

number of tweets (y-axis) containing a certain entity posted on a certain date (x-axis). According to these, we used a 11-day (-3,7) window centered at the news publication date for extracting positive training instances. Experiments show that our method can find seed events with 97% accuracy.

## 5 Neural Event Mention Extraction

The overall structure of our representation learning model is shown in Figure 2. Given a tweet, two LSTM models (Section 5.1) are used to capture its sequential semantic information in the left-to-right and right-to-left directions, respectively. For deep semantic representation, each LSTM model can include multiple stacked layers. Neural pooling (Section 5.2) is performed on each LSTM layer to extract rich features. Finally, features from the left-to-right and right-to-left components are combined using neural tensors (Section 5.3), and the resulting features are used as inputs to a feed-forward neural network for classification (Section 5.4).

### LSTM Models

The main goal of our neural model is to find dense vector representations for tweets, which are effective features for event mention extraction. Starting from word embeddings Mikolov et al. (2013); Pennington et al. (2014), a natural way of modeling a tweet is to treat it as a sequence and use a recurrent neural network (RNN) structure Pearlmutter (1989). LSTM Hochreiter and Schmidhuber (1997) is a variant of RNNs, which is better at exploiting long range context thanks to purpose-built units called _memory blocks_ to store history information.

\begin{table}
\begin{tabular}{l l} \hline News Title & DDoS Attacks Take Down RBS, Ulster Bank, and Natwest Online Systems \\ Date & 2015 August 02 \\ Sentences & But as can be seen from the attacks _against__RBS_, _NatWest_, _and___Ulster__Bank_, _and___the__warnings from__GovCERT_.ch__and___the__FBI_, these attacks are coming back into vogue again. \\ \hline News Title & Bored Brazilian skiddie claims DDoS against Essex Police \\ Date & 2015 September 04 \\ Sentences & A teenager from Brazil has claimed responsibility for a distributed denial of service (DDoS) attack _on__Essex__Police’s website_, following a similar attack on another force earlier this week. They added: “Officers investigating the suspected denial of service attack _on__the__Essex__Police website_ ... are laising with other law enforcement agencies to identify any investigative leads” \\ \hline \end{tabular}
\end{table}
Table 2: Example news sentences where victim roles are in italic and entity is in bold.

Figure 1: Visualization of the numbers of tweets mentioning Ulster bank (on the left) and Essex (on the right) around the news publication dates.

Figure 3: LSTM-based text embedding for word vectors \(x_{1},x_{2},\dots,x_{n}\).

 

LSTM has shown improvements over conventional RNN in many nlp tasks [16, 24].

A typical LSTM memory block consists of three gates (i.e. _input_, _forget_ and _output_), which control the flow of information, and a _memory cell_ to store the temporal state of the network [30]. While traditionally the values of gates are decided by the input and hidden states in a RNN, we take a variation with _people connections_[30], which allows gates in the same memory block to learn from the current cell state. In addition, to simplify model complexity, we use coupled _forget_ and _input_ gates [25].

Figure 3 illustrates the memory block used for our tweet representation. The network unit activations for input \(x_{t}\) at time step \(t\) are defined by the following set of equations:

Gates at step \(t\):

\[i_{t}=\sigma(W_{ix}x_{t}+W_{ih}h_{t-1}+W_{ic}c_{t-1}+b_{i}) \tag{4}\] \[f_{t}=1-i_{t}\] (5) \[o_{t}=\sigma(W_{ox}x_{t}+W_{oh}h_{t-1}+W_{oc}c_{t}+b_{o}) \tag{6}\]

Cell:

\[c_{t}=f_{t}\otimes c_{t-1}\] (7) \[~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{ }~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~ {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} ~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{} {}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}~{}  

The NTN combined features are concatenated, and fed into a _tanh_ hidden layer. The output of the layer, \(\vec{f}_{s}\), becomes the final representation of a tweet, and is used to compute the probability of the tweet being an event mention, as shown in Equation 1.

### Classification

The final classifier of the neural network model is Equation 1, consistent with the baseline model. As a result, ER is applied in the same way as Equation 2. The main difference between our model and the baseline is in the definition of \(\vec{f}_{s}\), the former being a deep neural network and the latter being manual features. Consequently, Equation 1 can be regarded as a softmax layer in our deep neural model, for which all the features and parameters are trained automatically and consistently.

For training, the parameters are initialized uniformly within the interval [\(-a\),\(a\)], where

\[a=\sqrt{\frac{6}{H_{k}+H_{k+1}}} \tag{11}\]

\(H_{k}\) and \(H_{k+1}\) are the numbers of rows and columns of the parameter, respectively Glorot and Bengio (2010). The parameters are learned using stochastic gradient descent with momentum Rumelhart et al. (1988). The model is trained by 500 iterations, in each of which unlabeled instances are randomly sampled so that the same numbers of positives and unlabeled data are used.

## 6 Experiments and Results

### Data

We streamed tweet with the track keyword _ddos_ for five months from April 13 to September 13, 2015. In addition, we extracted tweets containing the word _ddos_ from a tweet archive5 in the period from September 2011 to September 2014. Using the distant seed event extraction scheme described in Section 4, a total number of 930 mentions covering 45 entity were automatically derived. In order to examine whether the automatically-collected instances are true positives and hence form a useful training set, an author of this paper annotated 100 extracted mentions finding that that 3 are false positives, as listed in Table 3. The result suggests that the automatically extracted mentions are reliable.

Footnote 5: [https://archive.org/details/twitterstream](https://archive.org/details/twitterstream)

The remaining tweets were randomly split into a 200-instance development set, a 800-instance test set, and an unlabeled training set.6 Both the development and test sets were annotated by a human judge and an author of this paper. The inter-annotator agreement on the binary labeled 1000 instances was measured by using Fleiss' kappa Fleiss et al. (2013), and the score, which is 0.85 for the data, represents _almost perfect agreement_ according to Landis and Koch (1977). There were 47 out of the 1,000 tweets that received different labels, for which another human judge made the final decision.

Footnote 6: [http://people.sutd.edu.sg/~yue_zhang/pub/naacl16.cyc.zip](http://people.sutd.edu.sg/~yue_zhang/pub/naacl16.cyc.zip)

To test the applicability of the proposed mention extraction system on other domains, we collected 400 sentences containing the keyword _ddos_ from dark web. Again each sentence was annotated by two human judges, and the third person made the final decision on conflicting cases. The inter-annotator agreement kappa score on this dataset is 0.85, consistent with the tweet annotation. Table 4 presents the statistics of the datasets.

### Evaluation

We follow Ritter et al. (2015) and evaluate the performance by the area under the precision-recall curve (AUC), where _precision_ is the fraction of retrieved instances that are event mentions, and _re

\begin{table}
\begin{tabular}{r r r r r} \hline  & Training & Dev & Test & Dark Web Test \\ \hline \hline Positive & 930 & 43 & 160 & 82 \\ Negative & – & 157 & 640 & 318 \\ Unlabeled & 127,774 & – & – & – \\ \hline \end{tabular}
\end{table}
Table 4: Statistics of the datasets.

\begin{table}
\begin{tabular}{r c l} \hline  & Training & Dev & Test & Dark Web Test \\ \hline \hline Positive & 930 & 43 & 160 & 82 \\ Negative & – & 157 & 640 & 318 \\ Unlabeled & 127,774 & – & – & – \\ \hline \end{tabular}
\end{table}
Table 3: The three false positives in the 100 automatically extracted mentions, where event entities are in bold.

 

call_ is the fraction of gold event mention instances that are retrieved. Precision-recall (PR) curves offer informative pictures on the classification of unbalanced classes Davis and Goadrich (2006).

### Development Experiments

For the proposed model, we empirically set the LSTM output vector \(h_{t}\), the NTN output \(V\), and the size of the hidden layer to 32.7 For the ER model, the human-provided label expectation prior \(\tilde{p}\) is set to 0.22 since the percentage of positives in the development set is 22%, and the parameter \(\lambda^{U}\) is set to one-tenth of the positive training data.8

Footnote 7: The hidden layer size is chosen by comparing development test scores using the sizes of 16, 32 and 64.

Footnote 8: Mann and McCallum (2007) found that \(\lambda^{U}\) does not require tuning for different data set.

#### 6.3.1 Feature Combination

We first test whether using a NTN to combine the bi-directional representations can give a better performance compared to simply concatenating the two representation vectors. Table 5 gives AUCs of one-layer _basic_, _max_, _avg_ and _min_ pooling strategies tested on the tweet development set. We can see that all the four different pooling strategies perform better when the NTN combination is used. As a result, for the following experiments we only consider using NTNs to combine bi-directional representations.

Next we observe the effect of using different numbers of LSTM layers in our model. AUCs of _basic_, _max_, _avg_ and _min_ pooling strategies with respect to 1, 2 and 3 LSTM layers are presented in Table 5. In most of the cases, the performance of the model increases when the LSTM architecture goes deeper, and we build our final models using 3 LSTM layers.

#### 6.3.2 Pooling Strategies

In the previous experiments, _max_ pooling achieves the highest AUC with the architecture 3-LSTM-layer+NTN, we are interested in whether combining _max_ with other pooling strategies would further increase the performance. Table 6 summarizes the AUC of various combinations, according to which we choose _max+basic_ for final tests.

Finally, we test the performance of sparse feature representations as used in the model of Ritter et al. (2015). Figure 4 shows the PR curves of the sparse representation and the best setting _max+basic_ evaluated on the development set. The AUC of using sparse representation is 0.30 while that of the _max+basic_ model is 0.51. The runtime performances of training with sparse feature representations and neural feature representations are 276.17 and 1137.87 seconds, respectively, running on a single thread of an Intel Core i7-4790 3.60GHz CPU.

### Final Test

Figure 5 presents the PR curves of the baseline sparse feature representation and the final neural model evaluated on the datasets, and Table 7 gives the AUC for these test-set evaluations. From the curves we can see that the sparse representation is comparatively less efficient in picking out negative examples, since at a lower recall the model does not gain a higher precision. In contrast, LSTM-based representation demonstrates a better trade-off between recall and precision. We do not have a strong intuition on why the performance on dark web test set is better than that on tweet test set for the proposed model.

\begin{table}
\begin{tabular}{l c} \hline Pooling & AUC \\ \hline \hline _max+avg_ & 0.48 \\ _max+min_ & 0.50 \\ _max+basic_ & **0.51** \\ _max+basic+min_ & 0.50 \\ _max+basic+min+avg_ & 0.47 \\ \hline \end{tabular}
\end{table}
Table 6: AUCs of different pooling methods.

Figure 4: Development PR curves.

 



### Analysis

Table 8 shows the top 5 and bottom 5 ranked dark web sentences9 as scored by the baseline and the proposed LSTM-based model, respectively. For each sentence, the human judgment (\(P\) for event mentions and \(N\) for non-event mentions) is given, followed by the probability values output by the baseline and the proposed system.

Footnote 9: The sentence boundary was detected by NLTK PunktSentenceTokenizer.

Only one of the top five most probable eventmentioning sentences as decided by the baseline is true positive. On the other hand, all of the top five sentences indicated by the proposed model are true positives. We investigate the contextual features that contribute to the false positive case "They dealt with because they "fended off" a ddos attack faster than SRTM is super sketchy!

[0.7]1.0 what's the status?you find it in the first post i set it to GREEN/ORANGE as the site is still under DDOS attack but temporarily accessable.greets

[0.7]1.0 It seems their idea of a "hack" is a DDoS attack on the server (which does indeed go on right now, and as all DoS attacks, can result in denial of service) and a brute-force attack on the login system to try to find out users' passwords.

[0.6]1.0 One of the other markets (Nucleus) is paying some blackhat to DDOS most of the other markets, it's all over Reddit.Support here is alsoep, I don't know how you can run a market with a daily uptime of 25%, agree with OP.

[0.7]1.0 He also said he was involved in helping DPR hack into Tormarket's database and launch the DDoS against the Russian cyberattackers.

[0.7] Bottom 5:

[0.5]0.0 I only words I could understand were "DDoS" and "Bastard". N|0.5|0.9 In general, it seems like they have set the site up to accommodate all parties: escrow, vendor ratings, buyer ratings, quick wallet transactions, etc.Guess we'll see how they deal with the growing pains, DDOS, & hack attempts that will certainly come their way in the near future.

[0.5]0.0 Please doss him.

[0.5]0.0 Next fucking day, ddos dildos and damage...LEGs wares hit my drop while the market was still floundering like guppies on hot concrete, yeah, that’s why.

[0.5]0.2 child pornography, spamm, DDOS etc.

### Data analysis

Table 8 shows the top 5 and bottom 5 ranked dark web sentences9 as scored by the baseline and the proposed LSTM-based model, respectively. For each sentence, the human judgment (\(P\) for event mentions and \(N\) for non-event mentions) is given, followed by the probability values output by the baseline and the proposed system.

Footnote 9: The sentence boundary was detected by NLTK PunktSentenceTokenizer.

Only one of the top five most probable eventmentioning sentences as decided by the baseline is true positive. On the other hand, all of the top five sentences indicated by the proposed model are true positives. We investigate the contextual features that contribute to the false positive case "They dealt with the ddos attacks with grace and confidence." determined by the baseline, and find that the patterns "DT ddos", "ddos attack|NN", "DT ddos attack|NN IN" and "IN DT ddos" are ranked 2nd, 18th, 111th, 127th among the 15,355 contextual patterns, respectively, which have relatively high weights but only carry

Figure 5: Final PR curves.



limited information. In contrast, the LSTM-based model can capture global syntactic and semantic features other than words surrounding _ddos_ to distinguish mentions from non-mentions. From the table we can see that those high-confidence sentences determined by the LSTM-based model are more informative compared with those lower ranked sentences.

Figure 6 presents the probability distributions of positive and negative test cases as obtained by the baseline (x-axis) and the LSTM-based model (y-axis), respectively. It can be seen from the figures that the probabilities determined by the LSTM-based model are scattered between 0.0 and 1.0, while those by the baseline are gathered between 0.5 and 0.9, which shows that the proposed neural model can achieve better confidence on classifying event mentions. This demonstrates its stronger differentiating power as compared with discrete indicator features, as hypothesized in the introduction. In addition, for the proposed model a large portion of true positives (\(\blacktriangle\)) are close to the top in both test sets, while more negatives (\(\times\)) gather at the bottom of the dark web test set plot, compared to that in the tweet test set. As for the baseline model, many negatives locate around the horizontal centre, with a probability of 0.5, in the tweet test set, which explains why the baseline is relatively less effective on the precision-recall trade-off.

## 7 Conclusion

We investigated LSTM-based text representation for event mention extraction, finding that automatic features from the deep neural network largely improve the sparse representation method on the task. The model performance can further benefit by exploiting deep LSTM structures and tensor combination of bi-directional features. Results on tweets and dark web forum posts show the effectiveness of the method.

## Acknowledgments

We would like to thank Geoffrey Williams for data annotation, Lin Li for data processing, and anonymous reviewers for their informative comments. Yue Zhang is the corresponding author.

## References

* [Bengio2009] Yoshua Bengio. 2009. Learning deep architectures for AI. _Foundations and trends(r) in Machine Learning_, 2(1):1-127.
* [Benson et al.2011] Edward Benson, Aria Haghighi, and Regina Barzilay. 2011. Event discovery in social media feeds. In _Proceedings of the Annual Meeting of the ACL_, pages 389-398, Portland, Oregon.
* [Boureau et al.2011] Y-Lan Boureau, Nicolas Le Roux, Francis Bach, Jean Ponce, and Yann LeCun. 2011. Ask the locals: multiway local pooling for image recognition. In _ICCV, IEEE International Conference on_, pages 2651-2658.
* [Chen and Manning2014] Danqi Chen and Christopher D Manning. 2014. A fast and accurate dependency parser using neural networks. In _Proceedings of the Conference on EMNLP_, pages 740-750.
* [Chierichetti et al.2014] Flavio Chierichetti, Jon Kleinberg, Ravi Kumar, Mohammad Mahdian, and Sandeep Pandey. 2014. Event detection via communication pattern analysis. In _International AAAI Conference on Weblogs and Social Media_.
* [Cho et al.2014] Kyunghyun Cho, Bart Van Merrienboer, Calar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In _Proceedings of the Conference on EMNLP_, pages 1724-1734, Doha, Qatar.
* [Collobert and Weston2008] Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: Deep neural networks with multitask learning. In _Proceedings of the 25th ICML_, pages 160-167.
* [Collobert et al.2011] Ronan Collobert, Jason Weston, Leon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. _The Journal of Machine Learning Research_, 12:2493-2537.
* [Das et al.2010] Dipanjan Das, Nathan Schneider, Desai Chen, and Noah A. Smith. 2010. Probabilistic frame-semantic parsing. In _Human Language Technologies: The Annual Conference of NAACL_, pages 948-956, Los Angeles, California.

Figure 6: Probability distributions on the test sets.

 

Jesse Davis and Mark Goadrich. 2006. The relationship between precision-recall and ROC curves. In _Proceedings of the 23rd ICML_, pages 233-240, Pittsburgh, Pennsylvania.
* Diao et al. (2012) Qiming Diao, Jing Jiang, Feida Zhu, and Ee-Peng Lim. 2012. Finding bursty topics from microblogs. In _Proceedings of the Annual Meeting of the ACL_, pages 536-544, Jeju Island, South Korea.
* Dong et al. (2015) Li Dong, Furu Wei, Ming Zhou, and Ke Xu. 2015. Question answering over freebase with multi-column convolutional neural networks. In _Proceedings of the Annual Meeting of the ACL and the 7th International Joint Conference on NLP_, pages 260-269.
* Dyer et al. (2015) Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A Smith. 2015. Transition-based dependency parsing with stack long short-term memory. In _Proceedings of the Annual Meeting of the ACL and the 7th International Joint Conference on NLP_, pages 334-343.
* Fleiss et al. (2013) Joseph L Fleiss, Bruce Levin, and Myungheee Cho Paik. 2013. _Statistical methods for rates and proportions_. Wiley-Interscience, 3 edition.
* Gers and Schmidhuber (2000) Felix Gers and Jurgen Schmidhuber. 2000. Recurrent nets that time and count. In _Neural Networks, 2000. IJCNN 2000, Proceedings of the IEEE-INNS-ENNS International Joint Conference on_, pages 189-194.
* Gers et al. (2000) Felix A Gers, Jurgen Schmidhuber, and Fred Cummins. 2000. Learning to forget: Continual prediction with LSTM. _Neural computation_, 12(10):2451-2471.
* Glorot and Bengio (2010) Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural networks. In _International conference on artificial intelligence and statistics_, pages 249-256.
* Graves et al. (2013a) Alan Graves, Navdeep Jaitly, and Abdel-rahman Mohamed. 2013a. Hybrid speech recognition with deep bidirectional LSTM. In _ASRU, 2013 IEEE Workshop on_, pages 273-278.
* Graves et al. (2013b) Alan Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. 2013b. Speech recognition with deep recurrent neural networks. In _ICASSP, 2013 IEEE International Conference on_, pages 6645-6649.
* Guo et al. (2013) Weiwei Guo, Hao Li, Heng Ji, and Mona T Diab. 2013. Linking tweets to news: A framework to enrich short text data in social media. In _Proceedings of the Annual Meeting of the ACL_, pages 239-249.
* Hashimoto et al. (2013) Kazuma Hashimoto, Makoto Miwa, Yoshimasa Tsuruoka, and Takashi Chikayama. 2013. Simple customization of recursive neural networks for semantic relation classification. In _EMNLP_, pages 1372-1376.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Iyyer et al. (2014) Mohit Iyyer, Jordan Boyd-Graber, Leonardo Claudino, Richard Socher, and Hal Daume III. 2014. A neural network for factoid question answering over paragraphs. In _Proceedings of the Conference on EMNLP_, pages 633-644.
* Jozefowicz et al. (2015) Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015. An empirical exploration of recurrent network architectures. In _Proceedings of the 32nd ICML_.
* Kalchbrenner et al. (2014) Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. 2014. A convolutional neural network for modelling sentences. In _Proceedings of the 52nd Annual Meeting of the ACL_.
* Kim (2014) Yoon Kim. 2014. Convolutional neural networks for sentence classification. In _Proceedings of the Conference on EMNLP_, pages 1746-1751, Doha, Qatar.
* Landis and Koch (1977) J Richard Landis and Gary G Koch. 1977. The measurement of observer agreement for categorical data. _biometrics_, pages 159-174.
* Li and Cardie (2013) Jiwei Li and Claire Cardie. 2013. Early stage influenza detection from Twitter. _arXiv preprint arXiv:1309.7340_.
* Li et al. (2014) Jiwei Li, Alan Ritter, Claire Cardie, and Eduard Hovy. 2014. Major life event extraction from twitter based on congratulations/condolences speech acts. In _Proceedings of the Conference on EMNLP_, pages 1997-2007, Doha, Qatar.
* Li et al. (2015) Hao Li, Heng Ji, and Lin Zhao. 2015. Social event extraction: Task, challenges and techniques. In _Proceedings of the IEEE/ACM International Conference on ASONAM_, pages 526-532.
* Liu et al. (2015) Yang Liu, Furu Wei, Sujian Li, Heng Ji, Ming Zhou, and Houfeng Wang. 2015. A dependency-based neural network for relation classification. In _Proceedings of the Annual Meeting of the ACL and the 7th International Joint Conference on NLP_, pages 285-290, Beijing, China.
* Mann and McCallum (2007) Gideon S Mann and Andrew McCallum. 2007. Simple, robust, scalable semi-supervised learning via expectation regularization. In _Proceedings of the 24th ICML_, pages 593-600.
* Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient estimation of word representations in vector space. In _Proceedings of Workshop at ICLR_.
* Neubig et al. (2011) Graham Neubig, Masato Hagiwara, and Koji Murakami. 2011. Safety information mining -- what can NLP do in a disaster --. In _Proceedings of the 5th International Joint Conference on NLP_, pages 965-973.
* Owoputi et al. (2013) Olutobi Owoputi, Brendan O'Connor, Chris Dyer, Kevin Gimpel, Nathan Schneider, and Noah A Smith. 2013. Improved part-of-speech tagging for online conversational text with word clusters. In _Proceedings of the Conference of NAACL_ 

Barak A Pearlmutter. 1989. Learning state space trajectories in recurrent neural networks. _Neural Computation_, 1(2):263-269.
* Pennington et al. [2014] Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014. Glove: Global vectors for word representation. In _Proceedings of the Conference on EMNLP_, pages 1532-1543.
* Petrovic et al. [2010] Sasa Petrovic, Miles Osborne, and Victor Lavrenko. 2010. Streaming first story detection with application to twitter. In _Human Language Technologies: The Annual Conference of NAACL_.
* Qiu and Zhang [2014] Likun Qiu and Yue Zhang. 2014. ZORE: A syntax-based system for Chinese open relation extraction. In _Proceedings of the Conference on EMNLP_, pages 1870-1880, Doha, Qatar.
* Ritter et al. [2011] Alan Ritter, Sam Clark, Mausam, and Oren Etzioni. 2011. Named entity recognition in tweets: An experimental study. In _Proceedings of the Conference on EMNLP_, pages 1524-1534.
* Ritter et al. [2012] Alan Ritter, Matsam, Oren Etzioni, and Sam Clark. 2012. Open domain event extraction from twitter. In _Proceedings of ACM SIGKDD_, pages 1104-1112.
* Ritter et al. [2015] Alan Ritter, Evan Wright, William Casey, and Tom Mitchell. 2015. Weakly supervised extraction of computer security events from twitter. In _Proceedings of the 24th International Conference on World Wide Web_, pages 896-905.
* Rumelhart et al. [1988] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1988. Learning representations by backpropagating errors. _Cognitive modeling_, 5:3.
* Sak et al. [2014] Hasim Sak, Andrew Senior, and Francoise Beaufays. 2014. Long short-term memory recurrent neural network architectures for large scale acoustic modeling. In _Proceedings of the Annual Conference of INTERSPEECH_.
* Sak et al. [2015] Hasim Sak, Andrew Senior, Kanishka Rao, Ozan Irsoy, Alex Graves, Francoise Beaufays, and Johan Schalkwyk. 2015. Learning acoustic frame labeling for speech recognition with recurrent neural networks. In _ICASSP, 2015 IEEE International Conference on_, pages 4280-4284.
* Sakaki et al. [2010] Takeshi Sakaki, Makoto Okazaki, and Yutaka Matsuo. 2010. Earthquake shakes twitter users: real-time event detection by social sensors. In _Proceedings of the international conference on WWW_, pages 851-860. ACM.
* Socher et al. [2013a] Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. 2013a. Reasoning with neural tensor networks for knowledge base completion. In _NIPS_, pages 926-934.
* Socher et al. [2013b] Richard Socher, Alex Perelygin, Jean Y Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013b. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proceedings of the conference on EMNLP_.
* Vo and Zhang [2015] Duy-Tin Vo and Yue Zhang. 2015. Deep learning for event-driven stock prediction. In _Proceedings of IJCAI_, BueNos Aires, Argentina, August.
* Zhou et al. [2015] Hao Zhou, Yue Zhang, and Jiajun Chen. 2015. A neural probabilistic structured-prediction model for transition-based dependency parsing. In _Proceedings of the Annual Meeting of the ACL_, pages 1213-1222.