# Presentation: 363461

## Rapid and Accurate Spoken Term Detection

- Michael Kleber
- BBN Technologies15 December 2006

## Overview of Talk

- BBN Mandarin system description
- Evaluation results
- Corpus size effects
- Detecting English terms
- Segmentation and transcript problems

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## BBN Evaluation Team

- _Core Team_
- Chia-lin Kao
- Owen Kimball
- Michael Kleber
- David Miller

- _Additional assistance_
- Thomas Colthurst
- Herb Gish
- Steve Lowe
- Rich Schwartz

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## BBN System Overview

- Byblos STT

- indexer

- detector

- decider

- lattices
- phonetic-
- transcripts

- index

- scored
- detection
- lists

- final output
- with YES/NO
- decisions

- audio

- search
- terms

- ATWV cost
- parameters

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

**Notes:**
Walk through the 4 yellow boxes of the slide, noting inputs and outputs.

The Speech-to-text box was the only part of the system that was pre-existing. Indexer, detector, and decider are new technology.

## Mandarin STT Configuration

- STT generates a lattice of hypotheses and a phonetic transcript for each input audio file.
- Word-based system:
- Acoustic:  HKUST (226hr), Callhome + Callfriend (35hr)
- Language:  Acoustic transcripts (2.2M words),                    UW-web (117M words), BN (123M words)
- PLP, pitch, prob. of voicing, features, no mixture exponents
- Word segmentation:
- 60k dictionary, longest-first algorithm to segment HKUST
- 23k dictionary to segment out-of-domain LM data
- Dictionaries:
- 26K word dictionary + 6.2K single characters
- Experiment: 1.6K English words spelled using Mandarin 82 phonemes
**58.37% WER on STD Dev06 CTS data  (44.91% CER)**
**40.94% WER on internal BBN-dev06     (30.97% CER)**

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## A Better STD Dev Set

- Original NIST Dev and Dryrun sets based on Eval03, which was drawn from CallFriend corpus
- Previous experience showed results not predictive of performance on  HKUST data to be used in Eval
- Created new BBN Dev set
- Used EARS Dev04 audio (HKUST data)
- Generated query terms using NIST software

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Mandarin CTS Results

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Why so much worse than English?

- Word error rate
- Effect of corpus size on ATWV
- Cross-language queries
- Word segmentation

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Corpus Size Effect

- ATWV sensitive to corpus size:
- Penalty for FA  ~  **.3 / #hours of speech**
- Value of a hit  =  **1 / #occurrences of term**
- Averaged over all **terms which occur**

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Corpus Size Effect

- ATWV sensitive to corpus size:
- Penalty for FA  ~  **.3 / #hours of speech**
- Value of a hit  =  **1 / #occurrences of term**
- Averaged over all **terms which occur**
- Actual effect on English DryRun06:

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Cross-Language Queries

- Many Mandarin query terms appearing in CTS contained English words/letters.(23% of Dev06, 12% of DryRun06)
- Training data mismatch:
- AM training: added English words, w/pronunciations
- Added  English to LM training but Dev set used to estimate LM weights contained no English
- Result: STT emitted no English words on Dev sets
- Mandarin decider logic fixed to give 0 hits for queries with English
- English is OOV, so ATWV on IV is immune.ATWV-IV = 0.4113  (vs. 0.3809 overall)

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Word Segmentation Problems

- If match by characters, many FAs result
- Character 后 appears in words like “afterwards” (之后 or 以后), “last” (最后), “then” (然后).
- Also can appear alone in abbreviated speech.
- Match by character: 1 true hit, 91 FAs.
- Match by words: miss due to disagreements with reference transcript segmentation.
- detection: **机械工程 **“mechanical engineering”
- reference: **机械 **“mechanical”
-                      **工程 **“engineering”
- Happens in English too: many FAs for “five” where transcript contained “twenty-five.”

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Word Segmentation Problems

- Can’t “just segment correctly” —Reference transcript is inconsistent.
- DryRun search term  **多**, “many”
- Transcription 1: **太 **“too”  **多 **“many”
- Transcription 2: **太多 **“too many”
- DryRun search term  **双 **“pair”** ****特 **“especially”
- Transcription 1: **几双 **“a few pair”
- Transcription 2: **几 **“a few”** ****双 **“pair”** ****特 **“especially”
- Happens in English too: “day care” vs “day-care.”

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06

## Directions for Improvement

- Cross-language queries, if important to task
- More robust word segmentation logic;Consistent transcripts or lenient scoring

- Rapid and Accurate Spoken Term Detection

- 15-Dec-06