# Presentation: 385702

## Topic Tracking at Maryland:Lessons from the Johns Hopkins Mandarin-English Information  (MEI) Project

- Gina-Anne Levow and Douglas W.Oard
- Institute for Advanced Computer Studies
- University of Maryland, College Park

- TDT-2000 Workshop

- November 17, 2000

## Roadmap	

- MEI Overview (6 weeks in 5 minutes)
- MEI Results
- Adapting MEI to TDT
- TDT Results
- Conclusions

## The MEI Team

- Senior Members
- Students

- Helen Meng 		Chinese University of Hong Kong
- Erika Grams 		Advanced Analytic Tools
- Sanjeev Khudanpur 	Johns Hopkins University
- Gina-Anne Levow 	University of Maryland
- Douglas Oard 		University of Maryland
- Patrick Schone 		US Department of Defense
- Hsin-Min Wang 		Academia Sinica, Taiwan

- Berlin Chen 		National Taiwan University
- Wai-Kit Lo 		Chinese University of Hong Kong
- Karen Tang 		Princeton University
- Jianqiang Wang 	University of Maryland

## MEI: The Challenges

- Speech Recognition
- Tokenization
- Lexicon coverage
- Selection among alternatives
- Translation
- Tokenization
- Lexicon coverage
- Selection among alternatives

## Term Granularity Options

## MEI Evaluation Collections

- 2265
- manually
- segmented
- stories

- 3371
- manually segmented
- stories

**Development**
**Collection: TDT-2**

**Evaluation**
**Collection: TDT-3**

**Mar 98**

**Oct 98**

**Dec 98**

- 17 topics,
- variable number
- of exemplars

**Jun 98**

**Jan 98**

- English text
- topic exemplars:
- Associated Press
- New York Times

- Mandarin audio
- broadcast news:
- Voice of America

- 56 topics,
- variable number
- of exemplars

**Jun 98**

## Mandarin Audio

- Term
- Translation

- President Bill Clinton and...

- English Exemplar

- Term
- Selection

- Bilingual
- Term
- List

- Query
- Construction

- Mandarin
- IR System

- Story
- Boundaries

- Evaluation

- Named
- Entity
- Tagging

- Document
- Construction

- Speech
- Recognition

- Relevance Judgments

- Ranked
- List

- BBN

- U Mass

- LDC

- Cornell

- Dragon

- LDC

- LDC

- LDC

- 000100010000010100

- Mean
- Uninterpolated
- Average
- Precision

- LDC
- CETA

## Query Translation

- Dictionary inversion for phrase translation
- “Wall Street” “best interests”  “human rights”
- Lemmatize remaining words if necessary
- e.g. “televised” translates as “television
-      filtering for query term selection
- Compared to an English background model

## Evaluation Measure

- Able to characterize variation across exemplars!

## Balanced Translation Works Well

- Pirkola’s structured queries
- Treat translation alternatives as synonyms
- Inquery #syn() operator
- Balanced translation
- Distribute probability mass over translation alternatives
- Inquery #sum() operator

- TDT-2, phrase-based translation, word-based retrieval

## Phrase Translation Beats Words

- Phrases beat words
- Three sources
- Translation lexicon
- Named entities
- Numeric expressions

- Condition: TDT-2, 12 exemplars, word-based retrieval

## Character Bigram Indexing Wins

- Character _bigrams_ are best
- _Syllable_ bigrams do poorly

- TDT-2, single NYT exemplar, manual translation

## Untranslatable Terms

- _Term__	__	__Occurrences_
- suharto		97
- netanyahu	88
- starr		62
- arafat		50
- bjp		45
- vajpayee	44
- estrada		44
- ....
- hsu		19
- zemin		7

## Cross-Language Phonetic Matching

- Small improvement
- Not statistically significant
- Character bigrams are best
- Form a unified index
- Character _and_ syllable bigrams
- Translate words if possible
- Then form character bigrams
- Otherwise translate syllables
- Then form syllable bigrams

- TDT-2, phrase-based translation

## MEI: Comparing Collections

## MEI Conclusions

- ASR		Words
- Translation	Phrases, Words, Lemmas, Syllables
- Indexing		Character Bigrams

## TDT-2000: What’s New Since ’99?

- Key ideas from MEI:
- Dictionary inversion for phrase translation
- Balanced translation
- Post-translation resegmentation
- Adaptation to TDT:
- Exploit negative exemplars
- Improved Mandarin topic normalization
- Round-robin balanced translation

## TDT-2000

- Mandarin Audio

- Term
- Translation

- President Bill Clinton and...

- English Exemplars

- Term
- Selection

- Bilingual
- Term
- List

- Query
- Construction

- PRISE

- Story
- Boundaries

- Score
- Normalization

- Document
- Construction

- Speech
- Recognition

- Ranked
- List

- NIST

- Dragon

- LDC

- LDC

- LDC

- Scores

- LDC/
- CETA

- IDF
- Computation

- Training
- Epoch

## Topic Tracking Improvements

- Improved     filtering for query term selection
- First compare to background model
- Augment by comparison to negative exemplars
- Mandarin topic normalization (unofficial)
- Language-specific strategy
- Mandarin: 	Best single training epoch score
- English:	Average of exemplar scores
- Recomputed Mandarin source normalization

## Effect of Negative Exemplars

- Text Only DET Plots
- 1st 60 topics
- (self-scored)

- Mandarin Text
- Nn=0  & **Nn = 2**

- English Text
- Nn=0 & **Nn=2**

## Indexing Character Bigrams

- Mandarin Speech Only
- 1st 60 topics
- (unofficial renormalization)

**Character Bigrams**

- Words

## Round Robin 8-Best Translation

- TDT-1999
- 2-best translation

- Mandarin Text
- 1st 60 Topics
- (self-scored)

**TDT-2000**
**Round-robin 8 best**

## Conclusions

- Top-8 round robin translation to Mandarin wins
- Slightly outperforms top-2 translation to English
- Query translation is more efficient
- Better suited to a stream of stories
- Match term extent to purpose
- ASR, translation, indexing

## Closing Thoughts

- Thanks to Jon and LDC !
- Normalization limits our insight
- Need some way to see past it
- Availability of TDT-3 ground truth?