Abstract:
Systems, methods, and apparatus, including computer program products for accepting a predetermined vocabulary-dependent characterization of a set of audio signals, the predetermined characterization including an identification of putative occurrences of each of a plurality of vocabulary items in the set of audio signals, the plurality of vocabulary items included in the vocabulary; accepting a new vocabulary item not included in the vocabulary; accepting putative occurrences of the new vocabulary item in the set of audio signals; and generating, by an analysis engine of a speech processing system, an augmented characterization of the set of audio signals based on the identified putative occurrences of the new vocabulary item.

Description:
BACKGROUND 
     Speech processing systems analyze audio streams and can produce outputs such as a transcription or lattice indicating occurrences of phonemes, words, or phrases in the audio stream. Transcriptions are generally linear sequences of units (e.g., words or phonemes). A lattice generally indicates alternative units, each unit spanning an associated interval, allowing alternative transcription hypotheses to be determined from the lattice. Systems often generate multiple transcription hypotheses (e.g., for each word or phrase spoken in the audio stream), usually with some level of confidence attached to each hypothesis. 
     Speech processing systems generally have a closed set of units, such as a closed word vocabulary or phoneme set. Generally, a speech processing system processes an input and produces a transcript or lattice based on the input in terms of the units in the closed set known to the system. However, in practice, new words, acronyms, names, and other terms are generated in language continuously, and if these terms are not added to the vocabulary of the system, they will not be identified by the speech processing system even if they are spoken in the audio stream. The vocabulary of a speech processing system may also be limited and may not include words specific to a particular field or application (e.g., product names or technical terms). These field-specific words will also not be identified by a speech processing system unless its lexicon is augmented to include such terms. To search for new terms not originally present in the lexicon, the audio stream can be reprocessed by the speech processing system with the new terms added to the vocabulary of the system. As speech processing is a slow and computationally expensive procedure, this process is generally impractical. 
     SUMMARY 
     In general, in one aspect, the invention features a computer-implemented method that includes accepting a predetermined vocabulary-dependent characterization of a set of audio signals, the predetermined characterization including an identification of putative occurrences of each of a plurality of vocabulary items in the set of audio signals, the plurality of vocabulary items included in the vocabulary; accepting a new vocabulary item not included in the vocabulary; accepting putative occurrences of the new vocabulary item in the set of audio signals; and generating, by an analysis engine of a speech processing system, an augmented characterization of the set of audio signals based on the identified putative occurrences of the new vocabulary item. 
     Embodiments of the invention may include one or more of the following features. 
     The analysis engine may include a wordspotting engine. The method may further include generating, by a user interface engine, a visual representation of the augmented characterization and causing the visual representation to be displayed on a display terminal. The predetermined vocabulary-dependent characterization of the set of audio signals may include quantities representative of confidence scores associated with the putative occurrences of each vocabulary item. The predetermined vocabulary-dependent characterization of the set of audio signals may include a transcript that is generated by a speech recognition engine of the speech processing system. The predetermined vocabulary-dependent characterization of the set of audio signals may include a transcript that is manually generated by a human transcriptionist. 
     Accepting the new vocabulary item may include accepting a term that is present in a user-provided text corpus. Accepting the new vocabulary item may include accepting a term that is present in a result set of an automatic discovery action. The automatic discovery action may include a web crawler action. The automatic discovery action may include a speech recognition action that is performed on a subset of the set of audio signals or a disjoint set of the set of audio signals using a different vocabulary. 
     The method may further include identifying putative occurrences of the new vocabulary item. The putative occurrences of the new vocabulary item may be identified by the analysis engine. The putative occurrences of the new vocabulary item may be identified by a human user. Identifying the putative occurrences may include determining quantities representative of confidence scores associated with the putative occurrences of the new vocabulary item. Identifying the putative occurrences may include identifying a duration of each putative occurrence of the new vocabulary item. 
     Generating the augmented characterization may include comparing the confidence scores associated with the putative occurrences of the new vocabulary item with confidence scores associated with putative occurrences of each vocabulary item in the predetermined characterization. Generating the augmented characterization may includes updating the vocabulary to include the new vocabulary item; and using the updated vocabulary, processing at least a portion of the set of audio signals including the duration of each putative occurrence of the new vocabulary item. Generating the augmented characterization may include inserting the new vocabulary item into the predetermined characterization. Generating the augmented characterization may include deleting a vocabulary item occurring in the predetermined characterization. Generating the augmented characterization may include replacing a vocabulary item occurring in the predetermined characterization with the new vocabulary item. Generating the augmented characterization may include augmenting a word lattice representative of the set of audio signals. Generating the augmented characterization may include updating a transcription of the set of audio signals. 
     Accepting the new vocabulary item may include accepting the new vocabulary item from a user. The new vocabulary item may be in a language different that that of at least one of the plurality of vocabulary items included in the vocabulary. 
     The method may further include generating the predetermined vocabulary-dependent characterization of the set of audio signals, where the generating includes aligning, by a transcript alignment engine of the speech processing system, the set of audio signals and a transcript. 
     The set of audio signals may include one or more of the following: a live audio stream, a legal deposition, a telephone call, and broadcast media. At least a first audio signal of the set of audio signals may be associated with video. 
     Among other advantages, the methods described herein enable the use of existing fast and efficient searching of an audio stream for occurrences of a new word after the audio stream has already been transcribed using a vocabulary that does not include the new word. New words can be searched for quickly and with relatively little computational investment, enabling efficient retranscription or updated analysis of an audio stream. For instance, an audio file transcribed or otherwise analyzed using a standard dictionary may be re-evaluated to search for occurrences of a proper name, a word or phrase in a foreign language, a word or phrase related to a current event or recently identified technical problem, or the name of a product or promotion. 
     Other general aspects include other combinations of the aspects and features described above and other aspects and features expressed as methods, apparatus, systems, computer program products, and in other ways. 
     Other features and advantages of the invention are apparent from the following description and from the claims. 
    
    
     
       BRIEF DESCRIPTION OF DRAWINGS 
         FIG. 1  is a block diagram of a speech processing system. 
         FIG. 2  is a flow chart of speech processing. 
         FIGS. 3A and 3B  are exemplary word lattices. 
     
    
    
     DETAILED DESCRIPTION 
     Referring to  FIG. 1 , an audio stream  101  is received by a speech processing system  100  and transformed by an input processing engine  103  into audio data  102 . Audio stream  101  may be, for instance, an audio recording of a news broadcast, a song, a speech, or a series of telephone conversations to a customer service call center; multimedia documents (e.g., video) that include audio components; or real-time audio, such as a telephone call. The audio data  102  includes acoustically-based data which is derived from the audio stream  101  by sampling its waveform and optionally computing signal processing features or statistically based quantities based on the waveform. 
     Referring to  FIGS. 1 and 2 , a speech recognition engine  104  processes the audio data  102  using tools such as a lexicon (vocabulary)  112 , a grammar  114 , and an acoustic model  116  (step  200 ). Lexicon  112  includes a set of words known to the speech recognition engine  104  at the time of processing the audio stream  101 . In some embodiments, lexicon  112  includes specialized terms appropriate for a particular field or application, such as technical terms or names of products and promotions of a company. Grammar  114  includes statistics and/or constraints on word sequences imposed, for instance, by grammatical features (e.g., the part of speech) of the words in lexicon  112 . Acoustic model  116  includes statistical models for the acoustic form of the words in lexicon  112 , for example based on models of subword units such as phonemes, which are used to define acoustic realizations of the words in the lexicon. 
     In some embodiments, the processing of the audio data (step  200 ) produces a word-based lattice. For instance, speech recognition engine  104  first processes audio data  102  according to lexicon  112  to obtain a phonetic representation of the speech represented by the audio data (step  202 ). In some embodiments, the processed audio data takes the form of a phoneme lattice or a frame synchronous phoneme score file, which provides a searchable phonetic representation of the audio stream. The processed audio data may also include other descriptive information related to the audio stream. For instance, processed audio data corresponding to a log of telephone calls to a call center may contain information such as customer identifiers, customer characteristics (e.g., gender), agent identifiers, call durations, transfer records, date and time of a call, general categorization of calls (e.g., payment vs. technical support), agent notes, and customer-inputted dual-tone multi-frequency (DTMF; i.e., touch-tone) tones. The processed audio data is further processed according to grammar  114  and acoustic model  116  to obtain a transcription or a word lattice  108  (step  204 ). The following discussion uses the example of a word lattice as the output of speech recognition engine  104 ; however, the method described below applies to a transcription (e.g., a representation of a linear sequence of words or phonemes) in a similar manner. 
     Word lattice  108  represents one or more possibilities for words that may occur in audio stream  101  at particular times. Each possible word included in word lattice  108  is associated with a start time t 1 , an end time t 2 , and a confidence score representative of the probability that the word is a correct match to the word spoken between time t 1  and time t 2  in audio stream  101 . Word recognition engine  104  may identify multiple possible words for a given time period or for overlapping time periods, each possible word having a different confidence score. A database  118  stores the set of {word, t 1 , t 2 , score} for each word included in word lattice  108 . In some cases, word lattice  108  does not include the confidence score of each word. In some embodiments, a visual representation of word lattice  108  is output to a user via a user interface engine  122 , for example, with time on a first axis and score on a second axis. The word lattice is then used for any of a variety of tasks, such as searching for audio content according to its word content (e.g., searching for particular words, phrases, and their Boolean combinations). When a linear transcript is generated as the output of speech recognition, the transcript may contain only the start times t 1  of each word; the end times t 2  and scores can be determined after generation of the transcript. 
     In some examples, a new word not included in lexicon  112  becomes known after a lattice has already been generated. In some embodiments, some time after the audio data  102  have been transcribed or converted into a word lattice by speech recognition engine  104 , speech processing system  100  accepts a new word (step  206 ) input from by a user through user interface engine  122 . In other cases, a new word is obtained automatically, such as by an automated web crawler. For instance and without limitation, the new word may be a proper name, a word or phrase in a foreign language, a word or phrase related to a current event or recently identified technical problem, or the name of a product or promotion. In other examples, the pronunciation of a word that is already included in lexicon  112  is changed and the word with the new pronunciation is treated as the new word. For instance, pronunciation may be changed if the pronunciation in the lexicon is flawed or if a speaker in the audio stream has a heavy accent that was recognized only after the initial processing of the audio stream. 
     A wordspotting engine  124  processes audio data  102  in view of the new word (step  208 ). In general, a wordspotting engine searches a phonetically-based representation of audio data for phonetically-specified events, such as words or phrases, in the audio. Such a wordspotting engine accepts a search term as input and locates a set of putative instances of the search term in the data, providing a collection of results with a confidence score and a time offset for each result. One implementation of a suitable wordspotting engine is described in U.S. Pat. No. 7,263,484, “Phonetic Searching,” issued on Aug. 28, 2007, the contents of which are incorporated herein by reference. 
     In this case, wordspotting engine  124  searches audio data  102  to determine time intervals for putative occurrences  125  of the new word in audio stream  101  (step  210 ). 
     In one embodiment, for each putative occurrence of the new word identified in audio stream  101 , wordspotting engine  124  determines whether the original word lattice  108  or the putative occurrence better matches audio stream  101  (step  212 ), for instance by calculating a score for each putative occurrence of the new word. Wordspotting engine  124  then computes a score for the new word. The score of adjacent words may be affected by the insertion of new word  120 . In some cases, the new word replaces a previously existing word (or words) in the word lattice. In other instances, the insertion of the new word causes the deletion of another previously existing word or the insertion of another word in the vicinity of the new word. In some instances, wordspotting engine  124  may not identify any putative occurrences of new word  120  in audio data  102 . 
     Wordspotting engine  124  then compares the score of the new word with the scores of the words contained in the original word lattice  108 . If the score of the new word is higher than the score of the words contained in the original word lattice (i.e., the new word is a better match to the audio stream  101  than the existing lattice), an evaluation engine  128  merges the new word arc into the word lattice to generate an augmented lattice  126  (step  214 ). Each word included in augmented word lattice  126  is identified with a start time t 1 , an end time t 2 , and a confidence score representative of the probability that the word is a correct match to the word spoken between time t 1  and time t 2  in audio stream  101 . Often, augmented word lattice  126  includes multiple possible words for any given time period, each possible word having a different confidence score. Database  118  stores the set of {word, t 1 , t 2 , score} for each word included in augmented word lattice  126 . In the case of a transcription, evaluation engine  128  replaces the appropriate words or phrases in the transcript. 
     In another embodiment, speech recognition engine  104  performs local audio reprocessing in the vicinity of the putative occurrence of the new word, using an augmented lexicon that includes the new word. Reprocessing the audio data with speech recognition engine  104  allows word-order and grammatical constraints and other language rules to be applied when determining the quality of the match. If speech recognition engine  104  identifies the new word during reprocessing of the audio data  102 , evaluation engine  128  merges the new word arc into the original word lattice  108  to generate an augmented word lattice  126 . 
     Once one or more of the putative occurrences of the new word have been evaluated, augmented word lattice  126  or an updated transcript incorporating the new word(s) is outputted to a user via user interface engine  122 . 
     As an example, suppose that the phrase “the caterpillar ate” is spoken in an audio recording to be processed by speech processing system  100 , but the word “caterpillar” is not included in lexicon  112 . When speech recognition engine  104  processes the audio data corresponding to that recording, the phrase “the caterpillar ate” will not be represented in the lattice because speech recognition engine  104  is not aware of the word “caterpillar.” Instead, speech recognition engine  104  will process the audio data in view of the vocabulary that is included in lexicon  112 , outputting a word lattice as shown in  FIG. 3A . In this example, phrases such as “the cat or pillar ate” or “the cat or pill are eight,” as shown by the dashed lines connecting identified words in the word lattice, are identified as the most likely possibilities. Other, less likely words are also identified. The start time t 1  and end time t 2  (reflected on the x-axis) of each of the words in these phrases are determined and each word is assigned a score (reflected on the y-axis) indicating the likelihood of that word accurately reflecting the speech between time t 1  and time t 2  in the audio recording. 
     After the audio data are processed by speech recognition engine  104 , a user inputs the new word “caterpillar” into the speech processing system  100 . For instance, the user may have just become aware of the word “caterpillar” or may have just realized its relevance in a particular application of interest to the user. Wordspotting engine  124  evaluates the audio data  102  to identify putative occurrences of the new word “caterpillar” in audio stream  101 . Wordspotting engine  124  then generates an augmented word lattice, shown in  FIG. 3B . In the augmented word lattice, the phrase “the caterpillar ate” is identified as the most likely phrase to have been spoken in the audio recording, and the phrase “the caterpillar eight” is identified as a less likely phrase. The phrases previously identified in the original word lattice (“the cat or pillar ate” and “the cat or pill are eight”) can still be seen in the augmented word lattice but have significantly lower scores. 
     In some embodiments, audio stream  101  is initially processed manually by a human transcriptionist to generate a transcript. In these cases, the manually generated transcript is automatically aligned to the audio data  102  and reevaluated as described above to identify putative occurrences of new words that were not known to the person performing the transcription. 
     In some embodiments, wordspotting engine  124  operates on the entirety of the transcript or word lattice corresponding to audio data  102 . In other embodiments, wordspotting engine operates on a part of the transcript or word lattice corresponding to only a portion of audio data  102 . Focusing the wordspotting engine on a particular portion of audio data  102  may be useful if, for instance, a user is reasonably certain that the new word occurs during a particular time period. 
     In the implementation described above, the alteration of the word lattice is performed after speech recognition engine  104  has already completely processed audio data  102 . In other embodiments, the alteration of the word lattice by wordspotting engine  124  is performed concurrently with the processing of the audio data by speech recognition engine  104 . For instance, during the processing of a continuous audio stream by speech recognition engine  104 , the wordspotting engine  124  simultaneously augments an already-generated portion of the word lattice. 
     In the implementation described above, speech recognition engine  104  and wordspotting engine  124  are distinct entities within speech processing system  100 . In other implementations, the speech recognition and wordspotting functions are both performed by a single engine. 
     Speech processing system  100  may be implemented in software, in firmware, in digital electronic circuitry, in computer hardware, or in combinations thereof. The system may include a computer program product tangibly embodied in a machine-readable storage device for execution by a programmable processor. Method steps are performed by a programmable processor executing a program of instructions to perform functions by operating on input data and generating output. The system may be implemented in one or more computer programs that are executable on a programmable system including at least one programmable processor coupled to receive data and instructions from, and to transmit data and instructions to, a data storage system, at least one input device, and at least one output device. Each computer program may be implemented in a high-level procedural or object-oriented programming language, or in assembly or machine language; in any case, the language can be a compiled or interpreted language. Suitable processors include, by way of example, both general and special purpose microprocessors. 
     Generally, a processor receives instructions and data from a read-only memory and/or a random access memory. Generally, a computer includes one or more mass storage devices for storing data files; such devices include magnetic disks, such as internal hard disks and removable disks; magneto-optical disks; and optical disks. Storage devices suitable for tangibly embodying computer program instructions and data include all forms of non-volatile memory, including, for instance, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks such as internal hard disks and removable disks; magneto-optical disks; and CD-ROM disks. Any of the foregoing can be supplemented by, or incorporated in, ASICs (application-specific integrated circuits). 
     It is to be understood that the foregoing description is intended to illustrate and not to limit the scope of the invention, which is defined by the scope of the appended claims. Other embodiments are within the scope of the following claims.