# Presentation: 474742

## Rapid and Accurate Spoken Term Detection

- Owen Kimball
- BBN Technologies
- 15 December 2006

## Overview of Talk

- BBN Levantine system description
- Evaluation results
- Diacritics
- Out-of-vocabulary issues

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## BBN Evaluation Team

- _Core Team_
- Chia-lin Kao
- Owen Kimball
- Michael Kleber
- David Miller

- _Additional assistance_
- Thomas Colthurst
- Herb Gish
- Steve Lowe
- Rich Schwartz

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## BBN System Overview

- Byblos STT

- indexer

- detector

- decider

- lattices
- phonetic-
- transcripts

- index

- scored
- detection
- lists

- final output
- with YES/NO
- decisions

- audio

- search
- terms

- ATWV cost
- parameters

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

**Notes:**
Walk through the 4 yellow boxes of the slide, noting inputs and outputs.

The Speech-to-text box was the only part of the system that was pre-existing. Indexer, detector, and decider are new technology.

## Levantine STT Configuration

- STT generates a lattice of hypotheses and a phonetic transcript for each input file.
- Word-based system:
- Orthography based on Modern Standard Arabic (MSA), no short vowel diacritics
- Acoustic: 57.3 hours LDC (noise words, no mixture exponents)
- Language: 250 hours of data, 1.3M words
- 38.5K dictionary, grapheme-as-phoneme based plus 100 manual pronunciations
- unknown short vowel (U), 39 phonemes
**42.32% WER on STD Dev06 CTS data**

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Levantine CTS Results

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## OOV Pipeline: Detector

- Word-based STT produces 1-best transcript: pronounce it  1-best phonetic transcript.
- Query is OOV if it contains any OOV word.
- OOV query detection:
- Pronounce query (grapheme-as-phoneme)
- Find minimal edit-distance alignments (agrep)
- Score = % error =

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## OOV Pipeline: Decider

- Need different Yes/No decision procedure:IV-decider requires posterior probabilities.
- Simple OOV decision procedure:
- Constant threshold on score (~ 0.7)
- Cap on maximum number of hits (0-3)
- Values set to maximize ATWV on Dev06 data.

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## OOV Pipeline: Results

- ATWV remained good:
- 0.3450 IV
- 0.3635 OOV
- Searches take longer:
- ~10-15x IV speed on Dev06 and DryRun06, with no attempt at indexing.

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## OOV Directions for Improvement

- Score substitutions using phoneme confusion matrix instead of flat edit distance
- Speed: indexing phonetic transcripts for approximate matching
- Search lattices beyond 1-best transcripts

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Levantine Diacritic Issues

- Originally looked at diacritized Levantine
- Trained STT engine using LDC 45 hour set
- Ran STD without knowing WER (no diacritized STT test set to measure WER).
- Found very high false alarm rate
- Examining FAs found hits that were legitimate alternate spellings

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Levantine Diacritics- Alternate Spellings

- Examining query words found more of same:
- In first 22 terms of dry run term list, 14 are “alternate diacritic” spellings of 5 underlying words, i.e. there were just 13 unique words in the first 22 terms
- Min~ahumo v Minohumo
- AlHayaApi v AlHayaAp
- Waliko  v  Walika
- qabilo  v  qabola  v qabolo
- LDC training and STD test set had additional pervasive differences

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## No-Diacritic Levantine Issues

- A quick look turned up a smaller number of  problems for no-diacritic Levantine
- Looking at 7 top-FA terms in dev set, found
- “bHky” vs “b>Hky” but no other spelling confusions
- One ref instance of term with 0 duration
- It would be interesting to QC test sets for inconsistent spellings and other issues

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06