Detecting segments of speech from an audio stream

The disclosure describes a speech detection system for detecting one or more desired speech segments in an audio stream. The speech detection system includes an audio stream input and a speech detection technique. The speech detection technique may be performed in various ways, such as using pattern matching and/or signal processing. The pattern matching implementation may extract features representing types of sounds as in phrases, words, syllables, phonemes and so on. The signal processing implementation may extract spectrally-localized frequency-based features, amplitude-based features, and combinations of the frequency-based and amplitude-based features. Metrics may be obtained and used to determine a desired word in the audio stream. In addition, a keypad stream having keypad entries may be used in determining the desired word.

BACKGROUND INFORMATION

The problem of entering text into devices having small form factors (like cellular phones, personal digital assistants (PDAs), RIM Blackberry, the Apple iPod, and others) using multimodal interfaces (especially using speech) has existed for a while now. This problem is of specific importance in many practical mobile applications that include text-messaging (short messaging service or SMS, multimedia messaging service or MMS, Email, instant messaging or IM), wireless Internet browsing, and wireless content search.

Although many attempts have been made to address the above problem using “Speech Recognition”, there has been limited practical success. These attempts rely on a push-to-speak configuration to initiate speech recognition. These push-to-speak configurations introduce a change in behavior for the user and reduce the overall through-put, especially when speech is used for input of text in a multimodal configuration. Typically, these configuration require a user to speak after some indicator provided by the system. For example, a user speaks “after” hearing a beep. The push-to-speak configurations also have impulse noise associated with the push of a button, which reduces speech recognition accuracies.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

The following disclosure describes a detection technique for detecting speech segments, and words, from an audio stream. The detection technique may be used for speech utterance detection in a traditional speech recognition system, a multimodal speech recognition system, and more generally in any system where detecting a desired speech segment from a continuous audio stream is desired. By way of background, speech recognition is the art of transforming audible sounds (e.g., speech) to text. “Multimodal” refers to a system that provides a user with multiple modes of interfacing with the system beyond traditional keyboard or keypad input and displays or voice outputs. Specifically for this invention, multimodal implies that the system is “intelligent” in that it has an option to combine inputs and outputs from the one or more non-speech input modes and output modes, with the speech input mode, during speech recognition and processing.

FIG. 1is a functional block diagram of a speech detection system100for determining desired speech segments in an audio stream. The speech detection system100includes an audio stream input102and a speech detection technique104. The speech detection technique104may be performed in various ways. In some embodiments, shown inFIGS. 2 and 3, the speech detection technique104may be based on pattern matching and may incorporate a traditional speech recognition system for a portion of the technique104. In other embodiments, shown inFIG. 8, the speech detection technique104may be based on signal processing. Additionally, a hybrid system that uses pattern matching and signal processing may be used.

In overview, the speech detection technique104in the speech detection system100includes several modules that perform different tasks. For convenience, the different tasks are separately identified inFIG. 1. However, one skilled in the art will appreciate that the functionality provided by some of the blocks inFIG. 1may be combined into one block and/or may be further split into several smaller blocks without departing from the present system. As shown, speech detection technique104includes an generate features110task, a obtain time-alignment112task, a process alignment114task, and a determine desired speech-segment116task. For the purpose of this discussion, the word phonemes refer to an audio feature that may or/or may not represent a word. Various embodiments for the speech detection system100are described below.

FIG. 2is one embodiment of the speech detection system ofFIG. 1where feature extraction is based on pattern matching. For this embodiment, the speech detection technique104may include an acoustic model202, a search206, a grammar or language model204, a module that obtains time-alignment210, a time alignment processor212, and a desired speech-segment detector214. Search106may be implemented using standard speech recognition methodologies. However, in contrast with standard speech recognition methodologies, search206attempts to generate features that are types of sounds (e.g. phonemes, noise, spikes, fricatives, voiced speech etc) and may not perform traditional speech recognition. In order to generate the features, search206accepts input from acoustic model202and grammar/language model204, which may be configured to identify types of speech like speech phonemes, noise phonemes, and so on. One illustrative grammar model is shown inFIG. 3and will be described later in conjunction withFIG. 3. Once the features have been identified, different time alignments corresponding to these features are obtained210.FIGS. 4-6illustrate example timing diagrams for a multimodal system, in combination with an input from a keypad stream, which may further aid identification of desired words. The use of the combination of the keypad stream and the timing diagrams may be applied in multimodal implementations for mobile applications, such as text-messaging, internet browsing, content searches, and the like, especially when the corresponding devices have small form factors. The time-alignments obtained from block210are then processed in block212to obtain the desired speech in block214. Details of the various processing that may be performed will now be described.

FIG. 3illustrates several grammar models represented as state diagrams302-308for generating features from the input audio stream. Because the desired speech may be a spoken word which may be accompanied by other audio, such as noise, background speech, and the like, the grammar models used by search206task may configure the search206task to simply output several types of sounds and their time alignments. For example, state diagram302includes three states: pause310, words312, and pause314. Transitions occur from pause310to words312, from words312back to words312, and from words312to pause314. Once the best matching word is determined by Search206, the corresponding time-alignment information may be output. Grammar304is similar to grammar302, but with word312state replaced with word/phonemes322state. The advantage of phoneme state is that there is no need to know the application's vocabulary and also phonemes give more detailed breakdown within words. Grammar306adds one additional state: noise-pause332state which may be transitioned to from pause310state. Once in noise-pause332state, a transition may occur to word/phonemes322state or back upon itself. This may be used in situations wherein the desired speech always occurs at the end of audio stream. In that case, the system may use the in noise-pause332state to match anything other than a desired spoken word and the word/Phonemes322state may be used to match the desired speech segment. Grammar308adds another state to grammar306. The additional state is another noise-pause342state which may be transition to from word/phonemes322. Once in noise-pause342state, a transition may occur to pause314or back upon itself. These phoneme/word grammar networks in conjunction with noise models may be used to build the search network for the search task in the traditional speech recognition system. One skilled in the art will recognize that grammars302-308may be extended to handle phrases, symbols, and other text depending on the specific application As will be shown inFIGS. 4-6, each time alignment, that is output, has a beginning and an end section that may or may not correspond to one of the desired speech segment. The process for performing time-alignment is now described.

FIG. 4is a timing diagram400illustrating an example of generated time alignments in combination with an input from a keypad stream when a user speaks a word first and then types a first letter. Audio410represents the extracted features from the search using the provided grammars. As shown, three time alignments402,404, and406are identified. Each time alignment has an associated begin time and end time. For example, time alignment402begins at t1and ends at t2. One implementation of the detection technique may be used in an “always listening” configuration within a multimodal system. In this implementation, a keypad stream420may be provided to further aid in determining desired words. Timing for the audio stream410and the keypad stream420are coupled so that key entries may be correlated with the identified features. In one embodiment, markers in the keypad stream may be a <space> key and <typing of letter> to indicate the beginning and end of any particular speech session. Using these markers allows consistency with present mobile interfaces that use text-prediction. However, other markers may be used based on the specific hardware device under consideration.

Timing diagram400represents an example of generated time alignments in combination with an input from keypad stream420when a user speaks a word first and then types a first letter. In this scenario, the speech segment corresponding to the last feature (e.g., the desired word406) before the first letter424is chosen as the desired speech-segment.

FIG. 5is a timing diagram500illustrating an example of generated time alignments in combination with an input from a keypad stream when a user types a letter first and then speaks a word. Audio510represents the extracted features from the search using the provided grammars. As shown, three time alignments502,504, and506are identified. Keypad stream520illustrates entry of a first letter522before time t1, a second letter524between time t1and t2, entry of a third letter526after time t3, and space key528after time t6to represent the turning of the microphone off. However, one will note, that the actual microphone is still on if in the “always listening” mode. For this timing diagram500, the time alignment502,504, or506that is the first to occur after entry of the first letter once the microphone has already been turned on is determined to be the desired speech segment (i.e., time alignment504). Thus, the desired word504occurs right after keypad entry524.

FIG. 6is a timing diagram600illustrating an example of generated time alignments in combination with an input from a keypad stream when a user types a first letter while speaking a word. Audio610represents the extracted features from the search using the provided grammars. As shown, three time alignments602,604, and606are identified. Keypad stream620illustrates keypad entry622and624. Keypad entry622corresponds to entry of a <space> key representing the turning on of a microphone. Keypad entry624corresponds to an entry of a letter after time t5. In this scenario, the desired word is chosen to be the last spoken word detected, before the first letter. The desired word may extend past the entry of the last letter as shown.

FIG. 7is a flow diagram illustrating one embodiment of a process700for processing time alignments suitable for use in the speech detection system shown inFIG. 1. Application specific knowledge may be incorporated into process700in order to process the start and end times in the alignments, to yield the audio segment corresponding to the desired speech. In addition, certain constraints may be introduced into process700. One example constraint may be a segment length. The segment length may be used to determine whether the segment is a valid word. For example, in the English language, words may be assumed to be of ½ second duration and hence if a specific audio segment is below a certain threshold (i.e., ¼ millisecond), then the audio segment may be ignored or combined with a neighboring nearby segment. Thus, the time-alignments are processed based on knowledge of an application's vocabulary (e.g., whether the vocabulary includes words, words with pauses, phrases, length of phrases, symbols, and the like). In addition, time-alignments may be processed based on a priori knowledge of starting letters, an average duration of words for a language being spoken, and others. For example, if the user spoke the word “Yesterday” and then typed the letter “Y”, then the knowledge that the desired speech segment has acoustics matching the phonemes that correspond to the pronunciation of words beginning with the letter “Y” may be additionally incorporated.

Those skilled in the art will appreciate that several variations of processing the alignments, based on the proposed framework, may be employed. For example, instead of starting from the last time-alignment, one could start form the first time-alignment. Another example may be to start at the time-alignment that indicates a word with the highest likelihood based on the V_rate and/or C_rate (where V_rate and C_rate will be explained below in conjunction withFIG. 8). In addition, traversing from one time-alignment to the next time-alignment may be performed in either direction.

Example process700begins at block702, where the last time alignment in a specified window is located. Processing continues at block704.

At block704, information about the time-alignment is obtained, such as the corresponding start and end time. Processing continues at decision block706, where a decision is made whether the current time-alignment is close to a previous time alignment. If the current time-alignment is close to the previous time alignment, the feature (recall this could be type of speech as in a word or syllable or phoneme) associated with the time alignment is marked as speech. Processing continues to block710to locate the previous time alignment and then back to block704. If it is determined that the time alignment is not close to a previous time alignment at decision block706, processing continues at block712.

At block712, properties of the time alignment are checked, such as the length, spikes, and other properties corresponding to any prosodic features. Processing continues at decision block714where a determination is made whether the time alignment represents the desired speech (in cases where desired speech corresponds to a spoken word, determination is made whether time alignment represents a word). The features may be specific to the application under consideration. If it is determined that the time alignment does not represent desired speech, processing continues to block716.

At block716, the time alignment is discarded and processing continues at block718to locate a previous time alignment and processing proceeds back to block704.

If the time alignment is determined to represent the desired speech at decision block714, processing continues at block720. At block720, the time alignment is marked as a desired speech that was detected. This desired speech may then be used for further processing such as speech recognition

FIG. 8is another embodiment of the speech detection system800ofFIG. 1where audio extraction is based on signal processing. For this embodiment, speech detection technique104may include an adaptive filter-bank802, a modulation feature extraction804component, and speech determination808component. The outcome of speech determination808component are time alignments, which may be processed by obtain time alignments block810, time alignment processor812, and desired speech-segment detector814as explained above for the speech detection technique104ofFIG. 2. By combining the time-alignment generated inFIG. 8with the processing discussed above inFIGS. 4-7, a more robust estimate may be achieved. The following discussion describes components802,804, and808.

Component802(i.e., adaptive filter bank) extracts modulation features from speech. One embodiment of an adaptive filter bank for extracting modulation features is described in an article entitled “On Decomposing Speech into Modulated Components”, by A. Rao and R. Kumaresan in IEEE Transactions on Speech and Audio Processing, Vol. 8, No. 3, May 2000. In overview, the adaptive filter bank uses a Linear Prediction (in spectral sub-band) spectral analysis to capture slowly varying gross details in the signal spectrum (or formants) and uses temporal analysis to extract other modulation around those gross details (or spectral formants). The output of component802is input to component804.

Component804(i.e., modulation feature extraction component) obtains individual features and/or features formed using linear combination of individual modulation features. In contrast with prior systems, which mostly use amplitude-based features, component804uses spectrally-localized frequency based features, amplitude based features, and combinations of frequency based and amplitude based features. By using frequency based features, the features are normalized due to the sampling frequency and their values may be correlated with phonetic information in sounds. For example, while the F2feature alone is known to be the second formant in speech that carries most of the intelligibility information, the inventors, by using combinations of the different features, have developed metrics that help distinguish different types of sounds and also separate them from noise. These metrics may then be used to better determine which time alignments correspond to the desired speech.

As shown inFIG. 8, component804obtains frequency-based features F0-F3, commonly referred to as formants. In addition, component804may use various combinations of these frequency-based features, such as F2-F1, F3-F2, and F3-2F2+F1, and the like. Each of this combination may also be a log, division, or the like. Component804may also obtain amplitude-based features, such as A0-A3. The inventors then combine the frequency-based features and the amplitude-base features to obtain other helpful features, such as A0*F0, A1*F1, A2*F2, and A3*F3. Those skilled in the art after reading the present application will appreciate that other linear and non-linear combinations may also be obtained and are envisioned by the present application. These features then code phonetic information in sounds. For example, F3-2F2+F1conveys information about the spacing between neighboring formats. By using these features, the present detection technique may capture the importance of spacing changes that occur over time during speech due to vocal cavity resonances that occur while speaking. Likewise, the feature distinguishes silence or relatively steady noise which has a more constant spacing. Further, component804processes the modulation features over time to generate metrics that indicate the variation of the amplitudes of these modulations (over time) and the frequency content in the modulations. Both metrics may be measured relative to the median of the specific modulation. The metrics are measured by processing overlapping windows of modulation features; the processing itself may be either done real-time or non-real-time. Those skilled in the art will appreciate that several variations of process700may be considered, including combination of features using discriminant analysis or other pattern recognition techniques; implementation using a sample-by-sample or a batch processing framework; using normalization techniques on the features, and the like. One example of process808will now be described.

At block820, a window of time may be used to determine the features. The duration of the window may be any time period.FIG. 8illustrates a window of 50 msecs. Each features specified in block804may be analyzed over this window. At block822, the number of times the feature is (consecutively or otherwise) greater than the median standard deviation (V_rate) is determined for each feature. At block824, the median crossing rate for each feature (C_rate) is determined. In other words, C_rate is the number of times the feature crosses the median. At block826, the results of block822and824are input to determine an indicator of speech/noise-silence using (V_rate>Vt) & (C_rate<Lt) where Vt and Ct are threshold values for the V_rate and C_rate, respectively. Those skilled in art will appreciate that several the median may be replaced by one of several other metrics including sample means, weighted averages, modes and so on. Likewise, the median crossing may be replaced by other level-crossing metrics. The threshold may be pre-determined and/or adapted during the application. The thresholds may be fixed for all features and/or may be different for some or all of the features. Based on the analysis, the results are either block828denoting speech or block830denoting noise. The outputs are stored and the process moves to the next block832where an overlapping window is obtained and proceeds to block820for processing as described above. Once the windowed audio segments have been processed, the stored indicators are combined with the time-location of the windows to yield a time-alignment of the audio. The alignment is then combined with other features and processed to yield the final begin and end of the desired speech segment as explained inFIGS. 4-7above.

Those skilled in the art will appreciate that several different ways of implementing the present detection technique may be done. In addition, the present detection technique may be generalized to address any text (phrases, symbols, and the like), and form of speech (discrete, continuous, conversational, spontaneous), any form of non-speech (background noise, background speakers, and the like) and any language (European, Mandarin, Korean, and the like).

Certain of the components described above may be implemented using general computing devices or mobile computing devices. To avoid confusion, the following discussion provides an overview of one implementation of such a general computing device that may be used to embody one or more components of the system described above.

FIG. 9is a functional block diagram representing a computing device for use in certain implementations of the disclosed embodiments or other embodiments of the word detection technique. The mobile device901may be any handheld computing device and not just a cellular phone. For instance, the mobile device901could also be a mobile messaging device, a personal digital assistant, a portable music player, a global positioning satellite (GPS) device, or the like. Although described here in the context of a handheld mobile phone, it should be appreciated that implementations of the invention could have equal applicability in other areas, such as conventional wired telephone systems and the like.

In this example, the mobile device901includes a processor unit904, a memory906, a storage medium913, an audio unit931, an input mechanism932, and a display930. The processor unit904advantageously includes a microprocessor or a special-purpose processor such as a digital signal processor (DSP), but may in the alternative be any conventional form of processor, controller, microcontroller, state machine, or the like.

The processor unit904is coupled to the memory906, which is advantageously implemented as RAM memory holding software instructions that are executed by the processor unit904. In this embodiment, the software instructions stored in the memory906include a speech detection technique911, a runtime environment or operating system910, and one or more other applications912. The memory906may be on-board RAM, or the processor unit904and the memory906could collectively reside in an ASIC. In an alternate embodiment, the memory906could be composed of firmware or flash memory.

The storage medium913may be implemented as any nonvolatile memory, such as ROM memory, flash memory, or a magnetic disk drive, just to name a few. The storage medium913could also be implemented as a combination of those or other technologies, such as a magnetic disk drive with cache (RAM) memory, or the like. In this particular embodiment, the storage medium913is used to store data during periods when the mobile device901is powered off or without power. The storage medium913could be used to store contact information, images, call announcements such as ringtones, and the like.

The mobile device901also includes a communications module921that enables bi-directional communication between the mobile device901and one or more other computing devices. The communications module921may include components to enable RF or other wireless communications, such as a cellular telephone network, Bluetooth connection, wireless local area network, or perhaps a wireless wide area network. Alternatively, the communications module921may include components to enable land line or hard wired network communications, such as an Ethernet connection, RJ-11 connection, universal serial bus connection, IEEE 1394 (Firewire) connection, or the like. These are intended as non-exhaustive lists and many other alternatives are possible.

The audio unit931is a component of the mobile device901that is configured to convert signals between analog and digital format. The audio unit931is used by the mobile device901to output sound using a speaker932and to receive input signals from a microphone933. The speaker932could also be used to announce incoming calls.

A display930is used to output data or information in a graphical form. The display could be any form of display technology, such as LCD, LED, OLED, or the like. The input mechanism932may be any keypad-style input mechanism. Alternatively, the input mechanism932could be incorporated with the display930, such as the case with a touch-sensitive display device. Other alternatives too numerous to mention are also possible.