Patent ID: 12223963

DETAILED DESCRIPTION

The following detailed description is made with reference to the figures. Exemplary embodiments are described to illustrate the subject matter of the disclosure, not to limit its scope, which is defined by the appended claims.

FIG.1Ais a block diagram of a system capable of performing dual mode speech recognition according to the present disclosure. The system includes two primary elements: a client device102and a server202. Client device102can be embedded in any of a wide range of mobile devices which incorporate some uses of speech recognition. A common example is voice control of a telephone dialing application, featured on various models of Apple Inc.'s iPhone®, or a number of implementations of Google Inc.'s Android® operating system, such as the MotoBlur® smart phones marketed by Motorola Inc. These applications accept telephone dialing commands as well as telephone numbers. Other applications on these and similar devices include mapping applications that accept location names as well as commands such as directions to find a restaurant of a given name, and the like. It will be understood that in this disclosure, client device102represents only the speech recognition portion of a mobile device, and not any other part of the functionality of such a system. Any of the wide variety of mobile systems employing some form of speech recognition could make use of embodiments of the present disclosure.

Client device102includes a communication module104, a control module106, a recognition module108driven by language context109, and an update module111. Communication module104sends and receives messages to/from the outside world, such as messages to/from the server, and issues commands to various parts of the host mobile device, such as, for example, a command to dial a particular telephone number. Control module106controls the operations of client device102, including data flow within the device as well as the communication interfaces to the host mobile device and the outside world. Recognition module108performs speech recognition functions, accepting audio input for a spoken query and producing as output results including the corresponding text. The language context109provides a description of the language accepted by the recognition module108, as explained more fully below. Update module111supports the adaptation of language context109to changing circumstances in the dialog.

Proprietary or Open Source embodiments exist for mobile recognition module108; for example, SphinxTiny, is an open source adaptation of Sphinx 3.x for the constraints of a mobile environment. Two general points should be noted about recognition module108. First, it will be readily understood that constraints on processor power, code size and supporting data sizes limit the coverage and accuracy of a speech recognition system hosted in a mobile device, compared to full-scale speech recognition systems that are generally extensive software structures with large vocabularies, depending on large data storage capabilities. Mobile systems with small vocabularies produce very good results when the spoken input uses the expected vocabulary but cannot generally match with the results of large server systems, particularly in terms of coverage.

Second, the results from speech recognition by module108include two parts: a transcription, which is its best estimate for the text of what the query said; and a score which measures its confidence in the accuracy of the recognition result. Confidence scoring is well-known in the art; the details of how such scores are typically generated are not essential here. More important is the fact that scores generated in the same manner from the same query can be compared to measure relative confidence between two transcription alternatives. U.S. Pat. No. 7,899,669 provides a recent example of a method that depends on performing such score comparisons.

In the illustrated embodiment, language context109consists of a set of words with their descriptions as text, and in terms of smaller components. The specific form of the vocabulary data structures depends on the specific embodiment of the recognition module108. Some embodiments include phonetic strings (one or more per pronunciation of the word) and phonetic lattices (one per word, more compactly encoding the set of all pronunciations). Other embodiments included other vocabulary elements. Those of ordinary skill in the art will understand the range of variations that may fall within this category without altering the scope of the invention. The language context109describes the words or phrases available to the recognition module108for transcribing a query.

FIG.1Bsets out an exemplary embodiment of a two-layer recognition module108that uses ASR techniques known in the art, supported with language context109. Recognition module108includes a phoneme recognizer module156and a word decoder module157. A given block of speech is processed by phoneme recognizer156that makes use of fixed acoustic models such as acoustic model152and creates as output a compact phoneme network. The phoneme network is passed as input to word decoder157, which uses the language context109to create a transcription and associated confidence score.

The language context109includes two modules, a vocabulary158and a language model170. Vocabulary158is set of words or phrases. It should be noted that a phrase (that is, a sequence of words such as “San Francisco” that is essentially used as a single word) will be treated as a word and be a vocabulary entry. As used below, “word” indicates “word or phrase.” Language model170consists of a set of constraints on word sequences. Language constraints are expressed differently in different embodiments; typical embodiments are N-grams and grammars.

Initially, language context109contains only a pre-determined vocabulary. As more recognitions are performed, as described below, update module111performs an adaptation of the client vocabulary, whereby one or more words are being added to the language context109, and other words might be removed to make room for the new words.

The new words, in a format intended for use by the client recognition module108, are sent by the server's module211. The actual extension of the client vocabulary is done by the update module111. Thereafter, the recognition module108will use the expanded vocabulary. The use of an expanded language context109may materially improve the response time of the ASR process. Inasmuch as language context109stores data for rapid access, this element may be viewed as a vocabulary cache.

In order to add words to the vocabulary, update module111may need to perform an additional function. When the available memory resources for client vocabulary data is about to run out, a garbage collection operation will be performed. In one exemplary embodiment, removal of a non-permanent word can be performed on a priority basis. The priority of a non-permanent word may be chosen using heuristic factors such as a word's importance (an assigned priority), how often it is used, or how recently it is used. Alternatively, frequency and recency of use may be combined as a word frequency amortized over time.

Another embodiment may map words to topics. The word to topic association can be downloaded by the update module111when it expands the language context109. In this case, word desirability can be based on topic desirability, using importance and frequency data for topics instead of words.

Server202designates a system operating remotely, in conjunction with client device102, to jointly support the desired speech recognition functionality; communications between the two may use one or more networks including the cellular telephone system, wireless networks, and the Internet. Any device fitting the term “server” as that term is generally known in the art can be employed as server202. The primary requirement here is robust computing power, supported by extensive data storage capacity. Those of skill in the art will recognize the minimum requirements for such a device.

It should further be noted that the present disclosure can be implemented using a number of commercially available speech recognition systems. The feature set out below do not depend upon the operation of any particular software, but rather the features set out here supplement the operation of any speech recognition architecture.

Recognition engine204can be any speech recognition system, proprietary or otherwise. In addition to the examples mentioned above; those in the art will appreciate the potential for the Sphinx 3 system or one of its derivatives, as disclosed in the Wikipedia webpage for “CMU Sphinx. As noted above, recognition engine204is a robust, full-featured speech recognition engine, capable of sophisticated, reliable, and accurate speech recognition; it offers considerable advantages in coverage and accuracy when compared with speech recognition module108. The vocabulary and language data available for local recognition by embedded recognition module108is much more limited. In some embodiments, recognition engine202and embedded recognition module108will be close algorithmic variants of each other, and thus highly compatible. A weaker compatibility is also an option, so long as the scores generated by the two systems are comparable, perhaps after some scaling. One must be able to compare a score generated by recognition module108with a score generated by recognition engine204, enabling the client control module106to make a relative confidence judgment between the results of the recognition engine204and those of recognition module108.

Server202and client device102are connected by way of one or more electronic communications networks. The connection path can be short and reliable, such as a Local Area Network. In other embodiments the connection between client device102and server202may have delays, or it may be available only intermittently.

FIG.2sets out a flowchart for a method disclosed in connection with the present disclosure. In general, the method begins with the receipt of a communication including a spoken query from a user and terminates by providing a text output (or transcription) that matches the content of the spoken query. The following discussion will include hardware references to specific elements of the exemplary embodiment ofFIG.1A, but it should be understood that those references do not limit the apparatus to be used in connection with the method ofFIG.2in any way.

Method400begins by receiving a spoken query at step402. The software in the host mobile device typically receives the spoken query directly from the user, speaking into the device's microphone. The user's spoken query is transmitted to a module such as communications module104(FIG.1A) and routed to the control module106which sends it to the speech recognition module108.

To perform the recognition both locally and remotely, the client forwards the spoken query at step404to both client device102and server202as part of a recognition request. Each system will return a transcription result with a certain score and latency. Note that the result text may be empty, indicating the absence of a result, indicating that the spoken query was not recognized.

In dual mode recognition, the client device102waits for a certain time for results from both the client's recognition module108and server and the server's recognition engine204. A timeout is chosen, as a latency that seems tolerable to the user, given the realities of the anticipated communication systems and networks. In one embodiment, the latency is chosen as 3 seconds. If both results are obtained within the latency threshold, the system will choose the one with the better score. Otherwise, the control module will choose the first result it receives. In either case, the chosen result will be returned as the basis for further processing of the response given to the user.

Simultaneously, with the performance of a local recognition operation in step502, recognition engine204also performs a recognition operation at step602. These two steps are completely asynchronous, and both operations return independent results to controller106, at step604. It is generally expected that the remote recognition result will be preferable to the local recognition result, the question being whether the remote recognition result can be produced with acceptable responsiveness.

At step406, the system waits for a recognition result to be received from the server, or for the timeout to expire, whichever occurs first. At the end of the latency period, four possible outcomes exist. Results may have been received from both the server and the client; from the client only; from the server only; or from neither.

In the last eventuality, where no result is returned before the latency timeout, the process shifts to step408, and the user is notified that no recognition was possible. Similarly, the cases of receiving the local result only or the server result only call for returning either the local result at step412or the server result at step414, respectively. The controller106compares the scores of two results and chooses the higher score as the successful recognition. Control then passes to step412if the local result score is higher, or to414if the server result score is higher. When a result is available from the server but not from the client recognition module, additional analysis and possible action is required beyond returning the result. It should be noted at the outset that some server systems may return an “empty” or “null” result upon failure to achieve a recognition. Those of skill in the art will understand that this situation may be dealt with in a number of ways. In one embodiment, the controller may determine whether a result was obtained from the local recognition module, and if so, it may reclassify the current outcome as either “No result” (if the local recognition module was unsuccessful) or as “Local result only” if a local result was achieved.

If the situation actually is classified as “server only”, the process first proceeds to step416, to determine whether the server's transcription contains any word or words that may be missing from the language context109. If such words are found, the client at step418issues a request to the server, employing update module111, to send descriptions of the missing words. After that request, or in the event that no missing words were found, control passes to step414, where the server result is returned. The server recognition engine204may apply contextual analysis or rules, employing the various linguistic models that may be incorporated into that engine, to identify useful responses to update requests. Upon seeing the missing word “Tuesday,” for example, it might send all the days of the week and months of the year. Upon seeing the word “rain” or “temperature” it can send a set of words that relate to the topic of weather.

CONCLUSION

The specification has described a method for performing dual mode speech recognition using both a client-based and server-based recognizers. Those of skill in the art will perceive a number of variations possible with the system and method set out above. These and other variations are possible within the scope of the claimed invention, which scope is defined solely by the claims set out below.