# Presentation: 426121

## Information Extraction for New Event Detection 

- Xiaoqiang Luo

**Notes:**
My name is Xiaoqiang Luo, from IBM. Today I wiill be talking about Information Extraction for New Event Detection.

A little background: we did not participate last year’s TDT. Last couple of years, I was working in ACE. This year we want to see if we can make use of information from ACE this TDT task. 

## Acknowledgments

- Martin Franz
- Abe Ittycheriah
- Scott McCarley
- Salim Roukos
- Todd Ward

**Notes:**
I started to work in this task in mid August after this year’s ACE evaluation, and these are my colleagues helped me in this year’s evaluation. I’d like to take the opportunity to thank them.

## Outline

- NED systems
- tf-idf baseline
- MaxEnt model:  tf-idf + ACE annotation
- Errors and Observations
- Conclusions

**Notes:**
This year we compared two systems: one is a tf-idf-based baseline system; the other is a MaxEnt model that combines tf-idf and information from an ACE information extraction system. I will talk about the two system in detail. The second part of the talk is about error analysis, done on TDT3 and TDT4. Then I will make some conclusion remarks and the future direction.

## tf-idf Baseline

- Similarity score:

- Confidence:

- Decision:  New Event if

- DET Curve: Varying Threshold  Value

- d0: current doc
- d-: previous doc
- W: docs in the past window

**Notes:**
First a few words about New Event Detection: we are given a stream of news stories, or documents. We need to identify the first document covering an event. Example of events include: airplane crash and the following rescue efforts. Another event in one of the TDT corpora is Asia Economic Crisis. 

In our baseline system, we consider a current document d0, and compare it with each and every previous document d- in a time window.  For each such document pair, we use the first formula to compute their similarity score. It’s an inner product of normalized term freq. for all the stems words appearing in both documents, weighted by idf. Then we take the maximum similarity score, invert it to get a confidence, which represents how confident we are that the current document is a new event. Intuitively, if the current doc is not similar to any of the previous doc, the max sim score will be small, and we are more confident that it is a new event. 

## Baseline Performance

- *Newswire stories only

- Window size:

**Notes:**
The bar graph summarizes the performance of the tf-idf baseline on TDT2/3/4 with 4 diff window sizes: 1 mon, 2 mons, 3 months and all the docs in the past. 
On TDT3, performance saturates on and beyond 60 days. On TDT2/4, sometimes we see performance degradation as the window size increases. On TDT4, we get reasonably good performance with a 30 day window.  Later on we decided to go with a 30-day window because a smaller window means less computation.

30 less; reasonable performance; 

## Why Just Words?

- The tf-idf score: structure is ignored
- Example “structures”:
- PERSON, LOCATION, ORGANIZATION etc
- Coreference information
- Relation between entities

**Notes:**
The tf-idf treats documents as a bag of words, and structures are totally ignored. Example structures: named entities, 

## ACE: mention, entity, relation

- The _American Medical Association_ voted yesterday to install _Thomas R. Reardon_ as _its_ _president_-elect, rejecting a strong, upstart challenge by a _District_ _doctor_ who argued that the nation’s largest physicians’ _group_ needs stronger ethics and new leadership.
- In electing _Thomas R. Reardon_, an _Oregon_ general _practitioner_ who had been the _chairman_ of _its_ board, _members_ signified _they_ did not hold _him_ responsible for a costly gaffe last year, when the _group_ agreed to endorse a line of _Sunbeam Corp._ health care products. _Reardon_ had become _chairman_ of ...

**Notes:**
A mention is an instance of objects. Mentions are underlined words or phrases (click ...).  An entity is a physical object. For example, all mentions in green in this document refer to the organization “American Medical Association”; and all mentions in red refer to the person “Thomas R. Reardon.”  Note that mention includes NAME,  NOMINAL and PRONOUN phrases.
ACE also includes relations between entities. 

Imagine the 2 paragraphs are documents. If they share lots of entities and relations, we would think they’re talking about the same thing, and it should help us to identify new events. 

## ACE Entity and Relation Types

**RelType  Subtype ****	****	**
*AT **	**based-In **	**	*
*	**located **	**	*
*	**residence **	*
*NEAR **	**relative-location **	*
*PART **	**other **	**	*
*	**part-Of **	**	*
*	**subsidiary **	*
*ROLE **	**affiliate-partner **	*
*	**citizen-Of **	*
*	**client **	**	*
*	**founder **	**	*
*	**general-staff **	*
*	**management **	*
*	**member **	*
*	**other **	*
*	**owner **	*
*SOCIAL associate **	**	*
*	**grandparent **	*
*	**other-personal **	*
*	**other-professional **	*
*	**other-relative **	*
*	**parent **	**	*
*	**sibling **	**	*
*	**spouse **	**	*

- Entity Type:
-    PERSON
-    ORGANIZATION
-    FACILITY
-    LOCATION
-    GPE
- Mention Level:
-     NAME
-     NOMINAL
-     PRONOUN

**Notes:**
Here are a list of ACE ent types and rel types. There are 5 ACE ents: PER, ORG, FAC, LOC, GPE. 
5 main relatypes, ~2 dozens subtypes. 

Very rich information. So how to make use of this information?

## ME Model for NED

- Probability of “new”:
- MaxEnt Model
- Used to rescore a top-N set of candidate documents {d_}

**Notes:**
Novel information;  Example of a pair of documents; Detailed features

## Counting Common Entities

- Comm:  N(ent) =2

- Ratio: R1=N/N1
-            R2=N/N2
-            Rc=N/(N1+N2)

## Features in MaxEnt Model

- Example Features:
- tfidf:                   if
- R1:                     if
- R1&Rc:                   if
- Relation: similar

## ME Learning Curve

- Training Data: TDT3      #events: 2963 (180+)        #Features: 294

## ME Results Summary

|  | TDT3* | TDT2 | TDT4 |
| --- | --- | --- | --- |
| Baseline | .5503* | .6326 | .6061 |
| ME | .5215* | .6917 | .5480 |

- * Training on TDT3       Test=TDT3*,TDT2,TDT4

**Notes:**
Amount of training data; couldn’t do before

## ME Model: Easier to Pick A Good Operating Point

- tfidf system

- ME system

## Similar on TDT4

- tfidf system

- ME system

## ME Model on TDT5

## Analysis: Extra Information from ACE Entity?

**Analysis: Extra Information from ACE Entity?**

**Notes:**
Msg??

## Some Other Findings

- Entity not covered by ACE
- “Hurricane George”
- First Story or First Event?
- Feature or metrics computed at story-level
- Example follows

## Analysis: Want Event, Not Story

- TDT3 30033: Introduction of euro
- 1st Story: _[19981001_0635_0719_APW_ENG.tkn_RECID=4226.sent.htm](file:///C:/research/TDT/NED2004/19981001_0635_0719_APW_ENG.tkn_RECID=4226.sent.htm)_
- (FA) Doc: _[19981001_0931_1012_APW_ENG.tkn_RECID=5863.sent.htm](file:///C:/research/TDT/NED2004/19981001_0931_1012_APW_ENG.tkn_RECID=5863.sent.htm)_

## Events Are On the Way ...

## Conclusions

- ACE used in NED
- ME model useful for picking a good operating point
- Benefit of ACE features should be enhanced by:
- Training set with consistent annotation rules!
- More entity/relation types and Events
- Need for sub-document level analysis:
- Document-level features not good for detecting events!

## THE END