# Presentation: 433614

## Rapid and Accurate Spoken Term Detection

- David R. H. Miller
- BBN Technolgies14 December 2006

## Overview of Talk

- BBN English system description
- Evaluation results
- Development experiments
- BBN explored STD across languages, but with limited evaluation resources we chose to field systems only in CTS for each language.

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

## BBN Evaluation Team

- _Core Team_
- Chia-lin Kao
- Owen Kimball
- Michael Kleber
- David Miller

- _Additional assistance_
- Thomas Colthurst
- Herb Gish
- Steve Lowe
- Rich Schwartz

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

## BBN System Overview

- Byblos STT

- indexer

- detector

- decider

- lattices
- phonetic-
- transcripts

- index

- scored
- detection
- lists

- final output
- with YES/NO
- decisions

- audio

- search
- terms

- ATWV cost
- parameters

- indexing

- searching

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Walk through the 4 yellow boxes of the slide, noting inputs and outputs.

The Speech-to-text box was the only part of the system that was pre-existing. Indexer, detector, and decider are new technology.

Now I’ll talk about each in turn

## BBN System Overview: STT

- Byblos STT

- indexer

- detector

- decider

- lattices
- phonetic-
- transcripts

- index

- scored
- detection
- lists

- final output
- with YES/NO
- decisions

- audio

- search
- terms

- ATWV cost
- parameters

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Walk through the 4 yellow boxes of the slide, noting inputs and outputs.

The Speech-to-text box was the only part of the system that was pre-existing. Indexer, detector, and decider are new technology.

## Primary STT configuration

- STT generates a lattice of hypotheses and a phonetic transcript for each input audio file.
- 2300-hour EARS RT04 CTS acoustic model training corpus
- 946M words language model training
**14.9% WER on Std.Dev06 CTS data**

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Fairly basic research system. No system combination, no fancy LM’s. 

AM training: Switchboard, Callhome, Fisher
LM Training: AM transcripts, UW web data, BN data, CNN&PBS archives 

StdDev06 = Fisher subset of EARS Eval03
15.5% WER on EARS Dev04. 
14.9 WER on Eval03 Fisher subset
These error rates using .glm based scoring. Exact orthography is 0.5 worse.

Operated at 44xRT – there 

## Primary STT English Architechture

- Segmentation
- +
- Feature
- Extraction

- Forward-
- Backward
- Decoding

- Lattice
- Rescoring

*Waveform*

*Fw SI STM AM,*
*bigram LM*
*Bw SI SCTM AM,*
*approx.trigram LM*

- RDLT Features

*Final Lattice*
*Final 1-best*

*SI crossword *
*SCTM AM, *
*trigram LM*

- Adaptation
- Parameters

- System described in detail in B. Zhang, et al. “Discriminatively trained region dependent feature transforms for speech recognition”. Proc. ICASSP 2006, Toulouse, France.

- N-best Hypothesis

- Trigram Lattice

- Speaker
- Adaptation

- Forward-
- Backward
- Decoding

- Lattice
- Rescoring

- Trigram Lattice

*Fw HLDA-SAT STM AM, *
*bigram LM*
*Bw HLDA-SAT SCTM AM,*
*approx.trigram LM*

*HLDA-SAT*
*crossword SCTM*
*AM, trigram LM*

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Actually, slight differences from Bing Z system. Slight difference in LM training with hyphenated words

Fairly basic research system. No system combination, no fancy LM’s.

FW-BW-RESC : Adapt : FW-BW-RESC
Automatic segmentation
VTLN, PLP Analysis, RDLT on 15frame super vectors --> 60 dimension.
STM forward pass, SCTM backward with approx 3-gram, SCTM x-word 3-gram lattice rescore.
Three pass decode, adapt all three models from N-best, full re-decode.

FW: 139k pdfs, 245 cbks, 504 Gauss/cbk
Triphone AM, nonx
BW: 126k pdfs, 6500 cbks, 127 Gauss/cbk
quinphone AM, nonx

RESC: 224k pdfs, 7000 cbks, 127 Gauss/cbk

Gauss/cbk is an average
Adaptation is from Nbest list

## BBN System Overview: Indexer

- Byblos STT

- indexer

- detector

- decider

- lattices
- phonetic-
- transcripts

- index

- scored
- detection
- lists

- final output
- with YES/NO
- decisions

- audio

- search
- terms

- ATWV cost
- parameters

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Walk through the 4 yellow boxes of the slide, noting inputs and outputs.

The Speech-to-text box was the only part of the system that was pre-existing. Indexer, detector, and decider are new technology.

## Indexer

- Indexer precomputes single-word detection records from lattices.
- Stores as hashed sorted lists for fast lookup.
- Computes fraction of likelihood that flows over each arc.
- Uses forward-backward algorithm.
- Optimistic posterior: ignores possibility true word is missing from lattice.
- Clusters detections with same word, close times, summing their scores

- WHICH [a=-205 l=-5]

- CAT [a=-170 l=-2]

- IS [a=-18 l=-2]

- THAT [a=-92 l=-3]

- A [a=-12 l=-2]

- WITCH [a=-200 l=-4]

- WITCH [a=-203 l=-4]

- CUT [a=-175 l=-3]

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**

Note existence of AM/LM overall scale knobs.
Variation of Word Consensus Networks
No global linear time scaffold.

## Index Structure

- phonetic
- transcripts

- CAT

- WITCH

- WHICH

- ...

- ...

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Each detection has a file reference, a begin time, a duration, and a score.

Bundles 1-best phone transcript (from Byblos lattice rescoring) into the index separately for OOV searches.

## BBN System Overview: Detector

- Byblos STT

- indexer

- detector

- decider

- lattices
- phonetic-
- transcripts

- index

- scored
- detection
- lists

- final output
- with YES/NO
- decisions

- audio

- search
- terms

- ATWV cost
- parameters

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Walk through the 4 yellow boxes of the slide, noting inputs and outputs.

The Speech-to-text box was the only part of the system that was pre-existing. Indexer, detector, and decider are new technology.

## Detector

- Detector generates a sorted, scored list of candidate detection records for each search term supplied.
- For single-word IV terms, performs trivial retrieval from index.
- For multi-word IV terms, looks for acceptable sequences of single-word detections
- Component detections must satisfy adjacency timing constraints
- Assigns minimum component score to the multi-word detection.
- OOV not a significant factor in English CTS – see Levantine talk.

| Audio File | Begin | Duration | Score |
| --- | --- | --- | --- |
| fsh_60262_exA | 83.1 | 0.23 | 0.93 |
| fsh_61228_exA | 29.7 | 0.18 | 0.85 |
| fsh_60844_exA | 101.5 | 0.28 | 0.47 |
| fsh_60650_exA | 2.71 | 0.30 | 0.13 |
| fsh_61228_exA | 55.9 | 0.21 | 0.01 |

- candidates for term “bombing”

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
No English Eval terms were both OOV and in the reference
We’ll talk about our OOV logic in Levantine
Note that our levantine OOV number is .364, better than our levantine IV number of 0.345
Not a significant factor for English Dev CTS (<1.5% OOV). 
Term declared OOV if any word is OOV.

## BBN System Overview: Decider

- Byblos STT

- indexer

- detector

- decider

- lattices
- phonetic-
- transcripts

- index

- scored
- detection
- lists

- final output
- with YES/NO
- decisions

- audio

- search
- terms

- ATWV cost
- parameters

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

## Decider

**Decider**

| Audio File | Begin | Duration | Score | YES/NO |
| --- | --- | --- | --- | --- |
| fsh_60262_exA | 83.1 | 0.23 | 0.93 | ? |
| fsh_61228_exA | 29.7 | 0.18 | 0.85 | ? |
| fsh_60844_exA | 101.5 | 0.28 | 0.47 | ? |
| fsh_60650_exA | 2.71 | 0.30 | 0.13 | ? |
| fsh_61228_exA | 55.9 | 0.21 | 0.01 | ? |

- Decider picks and applies a score threshold for each list to make YES/NO decisions.
- Processes each list of candidates independently
- Processes all detection records in a list jointly
- Aims to maximize ATWV metric

- candidates for term “bombing”

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

## Primary Evaluation Metric

- “Actual Term Weighted Value” is primary metric

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Those of us working in the eval know this quite well.
With apologies to them, wanted to review this for the rest of the audience.

Define Ncorrect, Ntrue, Nspurious.

N_terms is number of terms occuring once or more in the reference.
Otherwise Ntrue =0 causes division by 0 errors.

## Understanding ATWV

- Perfect ATWV = 1.0
- Mute detector has ATWV = 0.0
- Negative ATWV is possible.
- Motivated by application-based costs:

- All search terms are weighted equally
- False alarm cost is almost constant, but miss cost varies by term.
- Missing an instance of a rare term is expensive.
- Missing an instance of a frequent term cheap.

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Refer to next slide for cheap/expensive reasoning.

## Decider Theory

- Given unbiased, independent posterior probabilities on detections and known constant value/cost on outcome, optimal decision threshold ** satisfies

- In ATWV metric, if Ntrue(*term*) > 0

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
You start down the list of detections and declare YES whenever your expected to have postitive contribution to ATWV by doing so.
Expected impact = 0 is cutoff.

Vhit and Cfa are only constant within a term. So need per-term thresholding for optimal ATWV

Point out Ntrue has big effect on V, little effect on C

## Decider Approximations

- Ntrue(*term*) unknown, and detection scores biased.
- For each term, estimate from detections *D**i*:

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Start by adjusting for p(D) biases

P(D) is really estimate of P(D) given truth is in lattice at that time.
Scale by 1/(1-Pmissing from lattice)
fixed constant, optimized on Dev set

## 2006 STD Evaluation English Results 

- English CTS Results

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Note Indexing time on P and C different. Using “1xRT” STT system.

There’s a bug in the BBN speed/size numbers as reported by NIST on 1207. 
Here they are all scaled up by 8/3 since we only processed CTS audio.

Search speed a little misleading in precision: 
Our Linux system clock we were using had resolution only up to 0.01 sec 

## NIST English DET curves

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Pooled over all terms
Normalized raw scores after computing  so that =1 for each term.
Note no line represents optimal ATWV point since cost of Miss is not constant

## Effect of STT Error Rate

- Loss of 2.5 WER caused ATWV to drop 0.6-0.9
- Magnified effect because changes in lattice word posteriors don’t show up in WER
- WER affected by scoring conventions.
- Contraction, hyphenation normalization
- Rigorous match definition for this eval causes WER to increase by 0.5

| System | WER | Dev06
ATWV | DryRun06
ATWV |
| --- | --- | --- | --- |
| BBN primary | 18.0 | 0.786 | 0.766 |
| BBN contrast | 15.5 | 0.847 | 0.852 |

- STT WER has strong effect on ATWV:

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
Mention what Dev06 and DryRun06 are

Story 15 report
Byblos 370hr    1xRT    WER=24.7  Dev ATWV=?   DryRun06 ATWV=?

WERs quoted here are about 0.5 worse than on old evals
 current scoring is less forgiving on contractions

## Importance of  Lattice Output

- Lattice searching reduces *P**miss*
- 8-fold increase in number of candidate detections from STT
- Improves estimate of *N**true* for decisions
- Holds *P**FA* down

|  | Dev06 |  | DryRun06 |  |
| --- | --- | --- | --- | --- |
|  | 1-best | lattices | 1-best | lattices |
| primary | 0.787 | 0.847 | 0.735 | 0.852 |
| contrast | 0.740 | 0.786 | 0.704 | 0.766 |

- Search lattices is more accurate than searching 1-best transcripts

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
32,917   words in 1-best transcripts
259,266 detection clusters in lattices

## Effect of Multi-word Detection Logic

- Exact detection of multi-word search terms is possible:
- Store full lattice
- Search for words on adjacent edges
- Use fw-bw to get true posterior probability
- Approximate multi-word detection:
- Store only individual words, forget topology
- Search for words ordered & close in time
- Pr(phrase) = min Pr(words in phrase)

| Effect of Approximate Multi-word Detection |  |  |
| --- | --- | --- |
| Search time | Index size | ATWV |
| decreased by 99.5% | decreased by 97% | increased by 0.01 |

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
factor of 200 on speed
factor of 36 on size

## BBN STD Summary

- Accurate detection (83% of perfect ATWV)
- Fast search time
- Small index size
- Configurable indexing speed
- Fast index speed maintains good accuracy.
- Encapsulated decision logic
- Easy to tailor for cost metrics other than ATWV

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

## Contrast STT configuration 

- 2300hrs/800hrs/1500hrs AM training data (complementary MPE).
- Same LM training data as primary system
- Somewhat smaller model than primary
- 18.1 % WER on Std.Dev06 CTS data
- compared to 14.9% for primary

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06

**Notes:**
RT04 forward model disappeared.
Ran at about 3xRT
We used FW STM model from primary system
FW: 139k pdfs, 245 cbks, 504 Gauss/cbk
BW: 78k pdfs, 6500 cbks, 127 Gauss/cbk
RESC: 224k pdfs, 7000 ckbs, 52 Gauss/cbk

## Contrast STT English Architechture

- Segmentation
- +
- Feature
- Extraction

- Forward-
- Backward
- Decoding

- Speaker
- Adaptation

- Lattice
- Rescoring

*Waveform*

*Fw SI STM AM,*
*bigram LM*
*Bw SI SCTM AM,*
*approx.trigram LM*

- Cepstra + Energy

- Trigram Lattice

*Final Result*

*HLDA-SAT*
*crossword SCTM*
*AM, trigram LM*

- Cepstra + Energy

- 1-best Hypothesis

- Adaptation Parameters

- Architechture same as S. Matsoukas et al “The 2004 BBN 1xRT Recognition Systems for English Broadcast News and Conversational Telephone Speech”
- Proc. Interspeech 2005, Lisboa, Portugal.

- Rapid and Accurate Spoken Term Detection

- 14-Dec-06