# Presentation: 016434

## INFORMATION RETRIEVAL FROM VOICE:

**Joseph Picone****1**** and Company**
**Human Language Technology Program Management Office**
**National Security Agency**
**1.  Joseph Picone is on sabbatical from Mississippi State University where he conducts research into statistical modeling for speech recognition.**

**The Importance of Flexibility and Efficiency**

**INFORMATION RETRIEVAL FROM VOICE:**

## Abstract

**Successful applications of information retrieval on very large archives of streaming voice data pose some unique challenges that include:**
**hyper-real time indexing of data (1 xRT is no longer acceptable)**
**hyper-real time searches of indexed data (1,000,000 xfRT is desirable)**
**continuous updates of the audio archive**
**open vocabulary (no *****a priori***** constraints on search terms)**
**structured query languages that support operations on search terms****(e.g., multiple words, Boolean operations, regular expressions)**
**Though high accuracy speech to text engines configured for word recognition are an attractive choice for this task, their run-time characteristics are currently prohibitive for large-scale applications. Strategies for out of vocabulary words are required for such approaches to succeed. Alternatives to speech to text engines are gaining increased attention due to their speed and simplicity. **
**In this talk, we will discuss the implications of these constraints on the system architecture and underlying technology, and hopefully motivate future research into techniques that address computational efficiency as an integral component of the overall system design.**

## A Generic Spoken Term Detection Architecture

**Streaming data sources require real-time updates and incremental indexing**
**Two types of searches:**
**Real-time:**** data is indexed and searched as it arrives **
**Historical:**** all indexed data on disk is available for search**
**Computational considerations are crucial:**
**indexing at 100 xfRT in 500 Mbytes or less of memory on one CPU**
**search at 1,000,000 xfRT (processing seconds of indexed speech/second)**
**Indexes of 1 Mb/hour of speech or less are highly desirable**

## Context is Key to High Performance

**LVCSR systems use acoustic context****(e.g., context-dependent phones, phone n-grams)**
**LVCSR systems use linguistic context (e.g., trigrams)**
**Net result is high accuracy, but at a high computational cost**
**Linguistic context is key to search accuracy**
**Performance varies with the length of the search term**
**Performance improves dramatically with the number of search terms****(e.g., “spoken” and “term” and “detection” occurring in a 30 sec. window)**
**Intermediate information can play a big role**
**Deep lattices can improve performance but incur several risks****(e.g., false alarm rate, disk space, search time)**
**Confusion networks or pinched lattices can decrease search time but can potentially increase false alarm rates**

**Context is Key to High Performance**

**Phonetic recognition approaches:**
**   ****single-word search...****	****multi-word search...**

**Word recognition approaches:**
**	****simple LVCSR with phone backoffs...         N-gram LVCSR... **

## Flexibility of Search Terms

**Open vocabulary highly desirable**
**Important search terms are often less frequently occurring words****(e.g., proper nouns, surnames)**
**Flexibility of search terms often conflict with the linguistic constraints that ASR systems want to impose during search**
**Pronunciation modeling can be crucial:**
**Stress, dialect, accent, non-native speakers**
**Spelling to sound rules**
**Portability across a large number of languages**
**Structured queries are essential:**
**LVCSR focus on contiguous words **
**Multi-word phrases with optional silence or speech between words**
**Logical operators (e.g., “a and b but not c”)**
**Mapping semantics to actual spoken words**

**Flexibility of Search Terms**

## Metrics That Integrate Speed and Accuracy

## Technology Challenges

**Operational issues:**
**System must behave in an intuitive manner and be easy for users to optimize (e.g., false alarm problems should be easy to diagnose)**
**Accuracy vs. speed/power/volume**
**Disk I/O is very expensive**
**Language issues:**
**Do I need to know what language is being spoken? mixed languages?**
**Search terms:**
**What is a good search term?**
**Diagnostics and self-correction (e.g., pronunciation optimization)**
**Automatic derivation of search terms from other information sources**
**“****Find more words like this example” (e.g. time-marked waveform)**
**Semantic queries (e.g., “find all documents about banking”)**
**Score normalization across domains**
**Native orthography (in some languages, such as Arabic, this raises issues of written versus spoken forms)**

## Technology Challenges (Cont.)

**Languages and Domains:**
**Automatic adaptation to new domains and acoustic environments**
**Unseen channels and noise types; dynamic environments**
**Portability to less resourced languages (e.g. P. Schone, et al., Universal Phonetic Recognition)**
**Integration with other technologies to reduce the false alarm rate**
**“****Needle in a haystack”: the average audio cut has very few user ratings (Netflix and Amazon have an easier problem)**
**Topic Translation: cross-language queries and ****cross-language answers**
**Computational Complexity:**
**Hardware Acceleration: Are nonstandard hardware platforms such as DSPs viable for some approaches?**
**Algorithm Efficiency: Can research make progress at the pace needed to retain the interest of customers without careful consideration of application constraints such as computational resources?**