# Presentation: 658350

## Overview of the TDT 2001 Evaluation and Results

- Jonathan Fiscus
- Gaithersburg Holiday Inn
- Gaithersburg, Maryland
- November 12-13, 2001

## Outline

- TDT Evaluation Overview
- 2001 TDT Evaluation Result Summaries
- First Story Detection (FSD)
- Topic Detection
- Topic Tracking
- Link Detection
- Other Investigations

## TDT 101

- 5 TDT Applications
- Story Segmentation
- Topic Tracking
- Topic Detection
- First Story Detection
- Link Detection

- “Applications for organizing text”

**Terabytes of Unorganized data**

## TDT’s Research Domain 

- Technology challenge
- Develop applications that organize and locate relevant stories from a continuous feed of news stories
- Research driven by evaluation tasks
- Composite applications built from
- Automatic Speech Recognition
- Story Segmentation
- Document Retrieval

## Definitions

**A ****topic**** ****is ...**
**	****	**a seminal **event** or activity, along with alldirectly related events and activities.
**A ****story**** ****is ...**
- 		a topically cohesive segment of news that includes two or more DECLARATIVE independent clauses about a single event.

## Example Topic

**Title:**  Mountain Hikers Lost
-  **WHAT:** 35 or 40 young mountain hikers were lost in an avalanche in France around the 20th of January.
-  **WHERE:** Orres, France
-  **WHEN:** January 1998
-  **RULES OF INTERPRETATION:**
- Rule #5. Accidents

## TDT 2001 Evaluation Corpus

- TDT3 + Supplemental Corpora used for the evaluation******* †
- TDT3 Corpus
- Third consecutive use for evaluations
- XXX stories, 4th Qtr. 1998
- Used for Tracking and Link Detection development test
- Supplement of 35K stories added to TDT3
- No annotations
- Data added from both 3rd and 4th Qtr. 1998
- Not used for FSD tests
- LDC Annotations †
- 120 fully annotated topics: divided into published and withheld sets
- 120 partially annotated topics
- FSD used all 240 topics
- Topic Detection used the 120 fully annotated topics
- Tracking and Link Detection used the 60 fully annotated withheld topics

******* see www.nist.gov/speech/tests/tdt/tdt2001 for details
- † see www.ldc.upenn.edu/Projects/TDT3/ for details

## TDT3 Topic Division

- Two topic sets:
- Published topics
- Withheld topics
- Selection criteria:
- 60 topics per set
- 30 of the 1999 topics
- 30 of the 2000 topics
- Balanced by number of on-topic stories

**TDT 2000  Systems**

## TDT Evaluation Methodology

- Evaluation tasks are cast as detection tasks:
**YES** there is a target, or **NO** there is not
- Performance is measured in terms of detection cost:
- “a weighted sum of ***missed detection ***and ***false alarm*** probabilities”     **C****De**t  =  CMiss • PMiss • Ptarget**    +    **CFA • PFA • (1- Ptarget)
- CMiss = 1 and CFA=0.1 are preset costs
- Ptarget = 0.02 is the a priori probability of a target
- Detection Cost is normalized to generally lie between 0 and 1:     **(C****Det****)****Norm** = **C****Det** **/** min{CMiss • Ptarget**,  **CFA • (1- Ptarget)}
- When based on the YES/NO decisions, it is referred to as the **actual decision **cost
- Detection Error Tradeoff (DET) curves graphically depict the performance tradeoff between PMiss and PFA
- Makes use of likelihood scores attached to the YES|NO decisions
- Minimum DET point is the best score a system could achieve with proper thresholds

## TDT: Experimental Control

- Good research requires experimental controls
- Conditions that affect performance in TDT
- Newswire vs. Broadcast News
- Manual vs. automatic transcription of Broadcast News
- Manual vs. automatic story segmentation
- Mono vs. multilingual language material
- Topic training amounts and languages
- Default automatic English translations of Mandarin vs.  native Mandarin orthography
- Decision deferral periods

## Outline

- TDT Evaluation Overview
- 2001 TDT Evaluation Result Summaries
- First Story Detection (FSD)
- Topic Detection
- Topic Tracking
- Link Detection
- Other Investigations

## First Story Detection Results

- System Goal:
- To detect the first story that discusses each topic
- Evaluating “part” of a Topic Detection system, i.e., when to start a new cluster

## 2001 TDT Primary FSD ResultsNewswire+BNews ASR, English texts,automatic story boundaries, 10 File Deferral

**Notes:**
Explain DET graph
	Lower left is good
	Red part of bar is based on the actual decisions
	Green part of bar is the best point on the DET curve
	Difference between the two indicates....
Convergence to same performance curve.

## TDT Topic Detection Task

- System Goal:
**To detect topics in terms of the (clusters of) stories****that discuss them.**
- “Unsupervised” topic training
- New topics must be detected as the incoming stories are processed.
- Input stories are then associated with one of the topics.

- Topic 1

- Topic 2

- Story Stream

## Primary Topic Detection Sys. Newswire+Bnasr, Multilingual, Auto Boundaries, Deferral=10

- Mandarin 	Native
- Translated	Mandarin

## Topic Tracking Task

- System Goal:
**To detect stories that discuss the target topic,****in multiple source streams.****	**
- Supervised Training
- Given Nt sample stories that discuss a given target topic
- Testing
- Find all subsequent stories that discuss the target topic

## Primary Tracking ResultsNewswire+BNman, English Training:1 Positive-0 Negative

## TDT Link Detection Task

- System Goal:
**To detect whether a pair of stories discuss the same topic.**
- (Can be thought of as a “primitive operator” to build a variety of applications)

**?**

## Primary Link Det. ResultsNewswire+BNasr, Deferral=10

- Mandarin 	Native
- Translated	Mandarin

- Native Mandarin

- NTU’s threshholding is unusual

## Outline

- TDT Evaluation Overview
- 2001 TDT Evaluation Result Summaries
- First Story Detection (FSD)
- Topic Detection
- Topic Tracking
- Link Detection
- Other Investigations

## Primary Topic Detection Sys. Newswire+Bnasr, Multilingual, Auto Boundaries, Deferral=10

## Topic Detection:False Alarm Visualization

- Topic Detection:False Alarm Visualization

- Outer Circle: Number of stories in a cluster
**Light** => cluster was mapped to a reference topic
**Blue** => unmapped cluster
- Inner Circle: Number of on-topic stories

- UMass1

- TNO1-late

- `

- Topic ID

- Topic ID

- System clusters, ordered by size

- System clusters, ordered by size

- Systems behave very differently
- IMHO a  user would not like to use a high FA rate system
- Perhaps False alarms should get more weight in the cost function

## Topic Detection:2000 vs. 2001 Index FilesMultilingual Text, Newswire + Broadcast News,Auto Boundaries, Deferral =10

- The 2000 test corpus covered 3 months
- The 2001 corpus covered 6 months
- 35K more stories
- Might affect performance, **BUT** appears not to.

## Topic Detection Evaluation via a Link-Style Metric

- Motivation:
- There is instability of measured performance during system tuning
- Likely to be a direct result of the need to map reference topic clusters to system-defined clusters
- We would like to avoid the assumption of independent topics

## Topic Detection Evaluation via a Link-Style Metric

- Evaluation Criterion: “Is this pair of stories discuss the same topic?”
- If a story pair is on the same topic
- A missed detection is declared if the system put the stories in different clusters
- Otherwise, it’s a correct detection
- If a pair of stories in not on the same topic
- A false alarm is declared if the system put the stories in the same cluster
- Otherwise, it’s a correct non-detection

## Link-Based vs. Topic Detection Metrics: Parameter Optimization Sweep

- The link curve is less erratic for System1
- Link curve is higher: What does this mean?

- System 1:
- 62K Test Stories
- 98 Topics

- System 2:
- 27K Test Stories
- 31 Topics

## 2001 Tracking:Published vs. Unpublished Topics

- Systematic effect: unpublished topics worse
- 2000 results showed no trend
- The absolute differences are comparable for both evaluations
- Not sure who used the TDT3 topics for training.

## Best Tracking Systems – ’00 - ’01Broadcast News Performance(By Language and Human Data Preparation)

- +50%

- +100%

## What can be learned?

- Are all the experimental controls necessary?
- Tracking performance degrades 50% going from manual to automatic transcription, and an additional 50% going to automatic boundaries
- Cross-language issues still not solved
- Most systems used only the required deferral period
- Progress was modest: did the lack of a new evaluation corpus impede research?

## Summary

- TDT Evaluation Overview
- 2001 TDT Evaluation Results
- Evaluating Topic Detection with the Link-based metric is feasible, but inconclusive
- The TDT3 corpus annotations are now public!