# Presentation: 376160

## New Event Detection at UMass Amherst

- Giridhar Kumaran and
- James Allan

## Preprocessing	

- Lemur Toolkit for tokenization, stopping, k-stemming
- http://www-2.cs.cmu.edu/~lemur/
- BBN IdentifinderTM for extracting named entities

- CIIR, UMass Amherst

## Systems fielded	

- Submitted four systems
- Didn’t include last year’s system
- Classification according to LDC categories and term – pruning
- Didn’t work on exclusively NW story corpus

- CIIR, UMass Amherst

## Primary system – UMass1

- Utility of named entities acknowledged
- Failure analysis indicates
- Large number of old stories have low confidence score (false alarms)
- Conflict with new story scores
- Reasons
- Stories on multiple topics
- Diffuse topics
- Varying document lengths

- CIIR, UMass Amherst

## Primary system – UMass1

- Focus
-  Identify old stories better – affects cost
- Clue
- Most old stories get low confidence scores as topics linked by
-  only named entities (large number)
-  only non-named entities (few)

- CIIR, UMass Amherst

## Primary system – UMass1

- Approach
- Look at the set of closest matching stories
- If consistently high named entity or non-named entity match modify confidence score

- CIIR, UMass Amherst

## Primary system – UMass1

- Procedure
- Double original confidence score if less than a threshold
- Gradually reduce score towards original score if set of closest stories match neither named entities nor non-named entities

- CIIR, UMass Amherst

## UMass1 – Examples from TDT3

- Russian Financial Crisis - Old Story

| APW19981020.0237 | AllSim | NESim | noNESim |
| --- | --- | --- | --- |
| APW19981015.0139  | 0.278 | 0.273 | 0.270 |
| APW19981009.0790 | 0.251 |  0.366 | 0.178 |
| APW19981016.0669 | 0.237 | 0.423 | 0.166 |
| APW19981006.0509 | 0.211 | 0.359 | 0.107 |
| APW19981013.0582 | 0.206 | 0.395 | 0.056 |
| APW19981006.0229 | 0.196 | 0.510 | 0.047 |

- CIIR, UMass Amherst

## UMass1 – Examples from TDT3

- Russian Financial Crisis - Old Story

| APW19981020.0237 | AllSim | NESim | noNESim |
| --- | --- | --- | --- |
| APW19981015.0139  | 0.278 | 0.273 | 0.270 |
| APW19981009.0790 | 0.251 |  0.366 | 0.178 |
| APW19981016.0669 | 0.237 | 0.423 | 0.166 |
| APW19981006.0509 | 0.211 | 0.359 | 0.107 |
| APW19981013.0582 | 0.206 | 0.395 | 0.056 |
| APW19981006.0229 | 0.196 | 0.510 | 0.047 |

- CIIR, UMass Amherst

## UMass1 – Examples from TDT3

- Russian Financial Crisis - Old Story

| APW19981020.0237 | AllSim | NESim | noNESim |
| --- | --- | --- | --- |
| APW19981015.0139  | 0.278 | 0.273 | 0.270 |
| APW19981009.0790 | 0.251 |  0.366 | 0.178 |
| APW19981016.0669 | 0.237 | 0.423 | 0.166 |
| APW19981006.0509 | 0.211 | 0.359 | 0.107 |
| APW19981013.0582 | 0.206 | 0.395 | 0.056 |
| APW19981006.0229 | 0.196 | 0.510 | 0.047 |

**Threshold = 0.1**

- CIIR, UMass Amherst

## UMass1 – Examples from TDT3

- Russian Financial Crisis - Old Story

| APW19981020.0237 | AllSim | NESim | noNESim |
| --- | --- | --- | --- |
| APW19981015.0139  | 0.278 | 0.273 | 0.270 |
| APW19981009.0790 | 0.251 |  0.366 | 0.178 |
| APW19981016.0669 | 0.237 | 0.423 | 0.166 |
| APW19981006.0509 | 0.211 | 0.359 | 0.107 |
| APW19981013.0582 | 0.206 | 0.395 | 0.056 |
| APW19981006.0229 | 0.196 | 0.510 | 0.047 |

**Threshold = 0.1**

- CIIR, UMass Amherst

## UMass1 – Examples from TDT3

- Russian Financial Crisis - Old Story

| APW19981020.0237 | AllSim | NESim | noNESim |
| --- | --- | --- | --- |
| APW19981015.0139  | 0.278*1.6 | 0.273 | 0.270 |
| APW19981009.0790 | 0.251 |  0.366 | 0.178 |
| APW19981016.0669 | 0.237 | 0.423 | 0.166 |
| APW19981006.0509 | 0.211 | 0.359 | 0.107 |
| APW19981013.0582 | 0.206 | 0.395 | 0.056 |
| APW19981006.0229 | 0.196 | 0.510 | 0.047 |

**Threshold = 0.1**

- CIIR, UMass Amherst

## UMass1 – Examples from TDT3

- Thai Airbus Crash   - New Story

| APW19981211.0623 | AllSim | NESim | noNESim |
| --- | --- | --- | --- |
| APW19981022.0205  | 0.250*1.2 | 0.154 | 0.341 |
| APW19981110.0229 | 0.184 |  0.052 | 0.282 |
| APW19981113.0905 | 0.155 | 0.003 | 0.228 |
| APW19981002.0557 | 0.152 | 0.234 | 0.012 |
| APW19981114.0396  | 0.149 | 0.042 | 0.245 |
| APW19981006.0511 |  0.143 | 0.031 | 0.251 |

- CIIR, UMass Amherst

## UMass1 on TDT3

- CIIR, UMass Amherst

## UMass1 on TDT3

- CIIR, UMass Amherst

## UMass2 

- Basic vector space model system
- Compare with all preceding stories
- Return highest cosine match

- CIIR, UMass Amherst

## UMass3	

- Same model as UMass2
- TDT5 – Very large collection
- Practical system
- Compare with a maximum of 25000 stories with highest coordination match
- Faster

- CIIR, UMass Amherst

## UMass4

- Similar to UMass1
- Rationale is the same
- Consider top five matches
- Use different formula for modifying confidence score

- CIIR, UMass Amherst

## Performance Summary 

| System | Topic weighted 
min. cost (TDT5) | Topic weighted 
min. cost (TDT4) |
| --- | --- | --- |
| UMass1 – Modify confidence score based on evidence | 0.8790 | 0.5055 |
| UMass2 – Basic vector space model | 0.8387 | 0.5404 |
| UMass3 – UMass2  + restriction on number of documents compared with | 0.8479 | 0.5404 |
| UMass4 – UMass1 with different formula | 0.9213 | -- |

- CIIR, UMass Amherst

## Summary

- Basic vector space model did the best
- Restricting number of stories to be compared with
- Improved system speed
- Didn’t improve performance
- Primary system did extremely well on training data, but failed on TDT5

- CIIR, UMass Amherst