Venue: NAACL 2016

# Cross-genre Event Extraction with Knowledge Enrichment

 Hao Li and Heng Ji

Computer Science Department

Rensselaer Polytechnic Institute

jih@rpi.edu

###### Abstract

The goal of Event extraction is to extract structured information of events that are of interest from unstructured documents. Existing event extractors for social media suffer from two major problems: lack of context and informal nature. In this paper, instead of conducting event extraction solely on each social media message, we incorporate cross-genre knowledge to boost the event extractor performance. Experiment results demonstrate that without any additional annotations, our proposed approach is able to provide 15% absolute F-score improvement over the state-of-the-art.

## 1 Introduction

The rapid development of social media and social networks since 2000s has made it an important channel of information dissemination. Because of its real-time nature, social media can be used as a sensor to gather up-to-date information about the state of the world. Effective automatic detection and extraction of events from the media will be an extremely important contribution. Recently there has been increasing interests in event extraction from social media Yang et al. (1998); Kleinberg (2003); He et al. (2007); Weng and Lee (2011); Benson et al. (2011); Ritter et al. (2012).

Identifying and extracting events in social media is more challenging than traditional event extraction due to two major reasons: (1). **Lack of Context**: compared with traditional genres (e.g., new articles), social media context is usually short and incomplete (e.g., each tweet has a length limitation of 140 characters). Lacking of context, a single tweet itself usually cannot provide a complete picture of the corresponding events. For example, for the tweet _"Pray for Mali - the situation is coming to light, and it isn't pretty."_, an event extraction/discovery system (e.g. Ji and Grishman (2008)) fails to discover that it is about the same _war_ event in _Mali_ as mentioned in the news article _"State military forces on Friday retook a key town in northern Mali after intense fighting that included help from French military forces, a defense ministry spokesman said."_ (2). **Informal Nature**: social media messages are written in an informal style, which causes the poor performance of event extractors designed for formal genres. For example, the tweet _"#AaronSwartz, Dead @ 26, #CarmenOrtiz "pushed him to exhaustion" don't let her get away with this! #scandal."_ includes an _"Die"_ event with _Aaron Swartz_ as the _Vctim_. However, the person name "AaronSwartz" appears in the hashtag "#AaronSwartz" and "Dead at 26" is written as "Dead @ 26". Existing supervised name taggers and event extractors fail to identify the same "Die" event mentioned in the news article _"Internet activist Aaron Swartz dead at 26"_.

Based on the intuition that news articles contain more detailed and formal information than tweet messages, we apply an unsupervised knowledge enrichment algorithm to link each tweet to its most relevant news article. By incorporating the cross-genre knowledge to tweets, we are able to formulate the task of event extraction on tweets as the task of cross-genre extraction for tweets and news articles. Thus we can alleviate the previous mentioned challenges in single-genre event extraction for tweets to

some extent. To the best of our knowledge, it is the first work to conduct cross-genre event extraction through unsupervised knowledge enrichment.

## 2 Problem Definition

Given a tweet \(t_{i}\), our cross-genre event extraction framework first discovers its most relevant news article \(n_{i}\), then identifies event tuples (event phrases and event arguments) for the tweet (\(te_{i}\)) and the news article (\(ne_{i}\)) respectively, and finally conducts merging on the event extraction outputs from both genres to produce the cross-genre event extraction result (\(e_{i}\)). For example in Table 1, given the following tweet \(t_{1}\), \(n_{1}\) is retrieved as its most relevant news article. \(te_{1}\) and \(ne_{1}\) are the extracted event tuples for the tweet and the news article respectively and \(e_{1}\) is the final cross-genre event extraction output after merging. To evaluate the performance of an event extractor, the precision, recall and f-measure of the extracted event phrases and event arguments will be measured using the following criteria: an event phrase is correctly labeled if it matches a reference trigger; an argument is correctly labeled if it matches a reference argument.

## 3 Approach

### Baseline Event Extraction Systems

We use two state-of-the-art event extraction systems Ritter et al. (2012); Li et al. (2013) to extract events from tweets and news articles respectively. The tweet event extractor TwiCal-Event Ritter et al. (2012) is able to extract open-domain significant events from Twitter. It is a supervised system that identifies event phrases and event participants with tailored part-of-speech tagging and shallow parsing for tweets. In addition, it is also able to discover event categories and classify extracted events based on latent variable models. It takes tweets as input and outputs a four-tuple representation of events which includes event participants, event phrase, calendar date, and event type. The news event extractor Li et al. (2013) is a joint framework based on structured prediction which extracts triggers and arguments simultaneously while incorporating diverse lexical, syntactic, semantic and global features. It takes raw documents as input, distinguish events from non-events by classifying event triggers and identifying and classifying argument roles.

### Knowledge Enrichment Approach

To produce the latent vector representations for the whole dataset, we follow the same procedure in Guo et al. (2013): represent the dataset in a matrix X, where each cell stores the TF-IDF values of words. Word vectors \(P\) and tweet vectors \(Q\) are optimized by minimizing the following objective function:

\[\begin{split}&\sum_{i}\sum_{j}W_{ij}\left(P_{\cdot,i}\cdot Q_{ \cdot,j}-X_{ij}\right)^{2}+\lambda||P||_{2}^{2}+\lambda||Q||_{2}^{2}\\ +\delta\cdot(\frac{Q_{\cdot,j_{1}}\cdot Q_{\cdot,j_{2}}}{|Q_{ \cdot,j}||Q_{\cdot,j_{2}}|}-1)^{2}\\ W_{i,j}=\left\{\begin{array}{ll}1,&\text{if $X_{ ij}\neq 0$}\\ w_{m},&\text{if $X_{ij}=0$}\end{array}\right.\end{split} \tag{1}\]

where \(\lambda\) is a regularization term, \(Q_{\cdot,j_{1}}\) and \(Q_{\cdot,j_{2}}\) are linked pairs connected by text-to-text relations, \(|Q_{\cdot,j}|\) denotes the length of vector \(Q_{\cdot,j}\) and the coefficient \(\delta\) denotes the importance of the text-to-text links. we follow the same optimization procedure as Steck (2010) by alternating Least Square [ALS] is used for inference on \(P\) and \(Q\).

After obtaining the vector representations for the whole dataset, for each tweet, we retrieve its cross-genre knowledge by finding the news article with the highest cosine similarity.

\begin{table}
\begin{tabular}{l l} \hline \multicolumn{2}{c}{\(t_{1}\): Crowds rally in Belfast for flag protest@freturnh} \\ \hline \multicolumn{2}{c}{\(n_{1}\): Protesters march in Northern Ireland: Under a gray, overcast sky, more than 1,000 protesters gathered Saturday in the Northern Ireland city of Belfast carrying large Union flags, some wrapped around their shoulders.} \\ \hline \multicolumn{2}{c}{\(te_{1}\): \{_EventPhrase = [rally, protest], EventArgument_ = _Belfast, Time = 2013-01-14_\} \\ \hline \multicolumn{2}{c}{\(ne_{1}\): \{_EventType = Conflict, EventPhrase = march, EventArgument = [protesters, Northern Ireland, Belfasti]_\}} \\ \hline \multicolumn{2}{c}{\(e_{1}\): \{_EventType = Conflict, EventPhrase = [rally, protest, march], EventArgument = [protesters, Northern Ireland, Belfasti], Time = 2013-01-14_\}.} \\ \hline \end{tabular}
\end{table}
Table 1: Cross-genre Event Extraction Example 



## 4 Experiments

### Data Description

We use the same dataset as Guo et al. (2013) which contains 34,888 tweets and 12,704 news articles. For each tweet, we consider the url-referred news article as its gold standard cross-genre knowledge - the most relevant news document. As the news event extractor is designed for a closed set of 33 event types (ace, 2005) while the tweet event extractor is for open domain, in this paper we only focus on the tweet-news pairs that the news event extraction output is not empty. We randomly selected 50 tweet-news pairs for the cross-genre event extraction annotation and evaluation.

### Experiment Results

Figure 1, 2 and 3 present the Precision (P), Recall (R) and F-measure (F) of the overall event extraction performance for different settings respectively. From these experiment results, we have the following four observations:

1. Event Extraction solely on tweet messages achieves the lowest precision, recall and f-measure. It exactly confirms our motivation of conducting knowledge enrichment for event extraction in tweet messages. Because of the informal nature of tweet messages, the event extractor misidentified 54.16% of the events thus the precision is low. Take the ill-formatted tweet in Section 1 as an example, the named entity "AaronSwartz" is in the hashtag "#AaronSwartz" and "Dead at 26" is written as "Dead @ 26". It makes the automatic event extractor extremely difficult to identify the "Die" event for the person "Aaron Swartz".

The low recall is mainly caused by the "lack of context" problem. Due to the length limitation of tweets, users tend to use recapitulate languages to describe an event. For Example, the following tweet _"Well. That sucks. 'Deepening Crisis for the Boeing 787'"_ actually refers to an "emergency landing" event made by "All Nippon Airways" in "western Japan". The user only mentioned the summary "Deepening Crisis for the Boeing 787" to refer to the actual event. Therefore, the single-genre event extractor missed the event trigger "landing" and the event arguments "All Nippon Airways" and "western Japan".

2. Both single-genre event extractors can contribute to the cross-genre event extractor through the cross-genre linking process. Even with the automatic linking output, Figure 2 and Figure 3 show 29.9% recall and 14.0% f-measure improvement

Figure 1: Event extraction precision

Figure 3: Event extraction f-measure

Figure 2: Event extraction recall

over single-genre event extractor for tweets. It is because that in most cases, news articles cover more information than tweets as they are produced by professional news agencies while tweets are written by individuals with a 140-character length limitation. In the following example, although referring to the same events, the news article covers more event information than the tweet, thus the cross-genre event extractor can surpass the single-genre event extractor for tweets.

**News:**_Deepening Crisis for the Dreamliner The two largest Japanese airlines said they would ground their fleets of Boeing 787 aircraft after one operated by All Nippon Airways made an emergency landing in western Japan._

**Tweets:**_Well. That sucks. "Deepening Crisis for the Boeing 787"_

For some certain cases, the cross-genre event extractor is also able to benefit from single-genre event extractor for tweets. For the following example, the event extractor for news articles missed the "Attack" event as "Halt an ... Advance" is an unusual phrase to describe an "Attack" event. However, in the related tweet, "Battling" is a strong indicator of an "Attack" event and the event extractor for tweets is able to catch it. As a result, we are able to extract the overall event tuples {EventPhrase=[Advance,Battling], EventPaticipant=[France,Islamist,Mali]}.

**News:**_French Troops Help Mali Halt an Islamist Advance France answered an urgent plea from the government of its former colony to help blunt an advance into the center of the country by Islamic extremist militants._

**Tweets:**_France Battling Islamists in Mali firebels tarnish W. African #Islam Western #intervention_

3. Either improving single-genre event extractors or achieving better cross-genre linking performance is able to boost the overall event extraction performance. From Figure 1, 2 and 3, we can observe that higher quality of single-genre event extractors will significantly enhance the precision while better linking performance will mainly contribute to a higher recall.

4. Compared with cross-genre linking accuracy, the quality of single-genre event extractors is more important. Figure 3 shows that "Gold Event Extractor \(+\) System Linking" achieved 26.3% higher F-measure score than "System Event Extractor \(+\) Gold Linking". It is mainly because of two reasons: on one hand, the errors of single-genre event extractors will be propagated to the final event output; on the other hand, the current linking system is able to provide reasonable linking results thus the use of perfect linking will not have too much gain.

## 5 Remaining Challenges

* his return and his flippant attitude is what is insulting. Not his new placement."_ is about a "Movement" event that _Assemblyman_ returns to _Albany_ after scandal. However, the tweets express the event so implicitly that the automatic linking system is not able to discover its corresponding news article.

**Extraction Errors:** single-genre event extractors failed on both the tweet side and the news article side. For the following example, both single genre event extractors missed the "Threaten" event between the _vice president Hugo Chvez_ and _those questioning the legitimacy of Chavez's government_.

**Tweet:**_#Venezuela VP warns those questioning the legitimacy of #Chavez's government: "Watch your words and your actions."_

**News:**_The vice president threatened action against any who question the legality of delaying the swearing-in of President Hugo Chvez, who is still in Cuba._

## 6 Conclusions and Future Work

In this paper we study the bottlenecks of event extraction for tweets. We have two observations: (1). Because of the "lack of context" and "informal nature" characteristics of tweets, conducting event extraction solely on tweet messages cannot produce satisfactory results; (2) The events embedded in tweets and news articles are often complementary. Based on these observations, we proposed to link each tweet to its most relevant news article, and further incorporated this cross-genre knowledge to conduct cross-genre event extraction. Experiment results showed that without any additional annotation, our proposed cross-genre event extractor is able to outperform state-of-the-art tweet event extraction. Our future research will focus on joint modeling 

of cross-genre event extraction in the training stage through cross-genre knowledge enrichment.

## Acknowledgment

This work was supported by the U.S. DARPA DEFT Program No.FA8750-13-2-0041, ARL NSCTA No. W911NF-09-2-0053, NSF Award IIS-1523198, AFRL DREAM project, and a gift award from Bosch. The views and conclusions contained in this document are those of the authors and should not be interpreted as representing the official policies, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes notwithstanding any copyright notation here on.

## References

* [Ace (2005)] ACE (automatic content extraction) english annotation guidelines for events, Jan. [Online]. Available: [https://www.ldc.upenn.edu/collaborations/past-projects/ace](https://www.ldc.upenn.edu/collaborations/past-projects/ace), (Date Last Accessed, Nov 4th, 2015).
* [Benson et al. (2011)] Edward Benson, Aria Haghighi, and Regina Barzilay. 2011. Event discovery in social media feeds. In _ACL_, pages 389-398.
* [Guo et al. (2013)] Weiwei Guo, Hao Li, Heng Ji, and Mona T. Diab. 2013. Linking tweets to news: A framework to enrich short text data in social media. In _Proc. Annu. Meeting of the Assoc. for Comput. Linguist._, pages 239-249, Sofia, Bulgaria.
* [He et al. (2007)] Qi He, Kuiyu Chang, and Ee-Peng Lim. 2007. Analyzing feature trajectories for event detection. In _Proc. Int. ACM SIGIR Conf._, pages 207-214, Amsterdam, Netherland.
* [Ji and Grishman (2008)] Heng Ji and Ralph Grishman. 2008. Refining event extraction through cross-document inference. In _Proc. Annu. Meeting of the Assoc. for Comput. Linguist._, pages 254-262, Columbus, OH.
* [Kleinberg (2003)] Jon M. Kleinberg. 2003. Bursty and hierarchical structure in streams. _Data Min. Knowl. Discov._, 7(4):373-397.
* [Li et al. (2013)] Qi Li, Heng Ji, and Liang Huang. 2013. Joint event extraction via structured prediction with global features. In _Proc. Annu. Meeting of the Assoc. for Comput. Linguist._, pages 73-82, Sofia, Bulgaria.
* [Ritter et al. (2012)] Alan Ritter, Mausam, Oren Etzioni, and Sam Clark. 2012. Open domain event extraction from twitter. In _Proc. ACM SIGKDD Conf. on Knowl. Discovery and Data Mining_, pages 1104-1112, Beijing, China.
* [Steck (2010)] Harald Steck. 2010. Training and testing of recommender systems on data missing not at random. In _Proc. ACM SIGKDD Conf. on Knowl. Discovery and Data Mining_, pages 713-722, Washington, DC.
* [Weng and Lee (2011)] Jianshu Weng and Bu-Sung Lee. 2011. Event detection in twitter. In _Proc. Int. AAAI Conf. on Web and Social Media_, pages 401-408, Barcelona, Catalonia, Spain.
* [Yang et al. (1998)] Yiming Yang, Thomas Pierce, and Jaime G. Carbonell. 1998. A study of retrospective and on-line event detection. In _SIGIR_, pages 28-36.