Speech recognition system loading different recognition engines for different applications

A speech recognition framework receives information about a recognized phrase from a speech engine. The framework identifies the application that is a focus of the recognized phrase. The framework then selects a handler function based on the recognized phrase and the application. The framework calls the handler function, which responds to the phrase.

TECHNICAL FIELD

This invention relates to a framework for speech recognition.

BACKGROUND

Sound enabled computers are equipped with a microphone and sound processing equipment, such as a soundcard, that allow the computers to receive sound input. Speech engine software may be installed on sound enabled computers to allow the computers to recognize speech contained within the sound input. The speech typically consists of words or phrases spoken by a user.

Speech-enabled applications, such as dictation software, are equipped to receive speech as input from a user, for example, through a speech engine. The applications typically receive the input through an application-programming interface (“API”) of the speech engine. All communication between the speech-enabled application and the speech engine must conform to a syntax that is specified by the speech API.

Different speech engines may have different APIs. Different versions of the same speech-enabled application are sometimes provided for the different APIs. Alternatively, some complex speech-enabled applications are equipped to communicate with more than one speech API.

DETAILED DESCRIPTION

As shown inFIG. 1, a computer10includes a microphone12and a sound card14, which allow the computer10to receive sound input from a user operating the computer. The computer also includes a processor16and a storage subsystem18. Storage subsystem18is a computer readable medium, such as a, computer memory, a DVDROM, a DVDRAM, a CDROM, a floppy disk, an optical disk, a hard disk, or a hard disk array.

Storage subsystem18stores computer software that is executed by processor16. The computer software includes an operating system20, a speech engine22that recognizes spoken phrases in the sound input, a speech service24, and two speech-enabled applications26and28. The operating system is typically a Windows™ operating system by Microsoft Inc., although any other operating system such as MACOS™ by Apple Inc. or a UNIX-like operating system (e.g. Linux, AIX™ or Solaris™) may be used instead.

As will be described in greater detail below, the speech service24receives recognized phrases from the speech engine22and directs them to the relevant one of the speech-enabled applications26,28. Speech service24selects a handler function that responds to the recognized phrase based on a speech handler file30,32associated with the speech-enabled application26,28. Speech service24also initializes and configures speech engine22based on the speech handler files30,32. The speech handler files30,32are described in greater detail below with reference toFIG. 5.

Thus speech enabled applications26,28interact with the speech engine22through the speech service24, which is configured to communicate with the API54of the speech engine22. As will be described below, the speech service24uses the same speech handler file30,32irrespective of the API54of the speech engine22. Consequently, the speech service24allows a single version of a speech-enabled application26,28to be used with any speech API that is supported by the speech service24.

Operating system20includes a sound card driver40that provides sound information to the software on the computer10. The operating system also includes a system registry42that contains information on the configuration of the computer10and the software installed on it. Certain operating systems may store the configuration information in configuration files instead. The operating system20provides an operating system API44through which other programs can interact and communicate with the operating system20.

The user interacts with the computer10through a user interface46provided by the operating system. User interface46is typically a graphical user interface, such as the windows user interface or the X-windows graphical interface, although the speech service may be used in systems with other user interfaces, such as a text user interface or a speech-driven user interface. A user selects a particular application or computer program as the target or “focus” of user input through the user interface46. The application in focus may, for example, be selected by clicking on an icon associated with the program, by typing a sequence of keys on a keyboard, or saying certain keywords.

Speech engine22recognizes spoken phrases based on a grammar provided by the speech service24. Speech engine22comprises code modules that perform the functions described below. Speech engine22includes a grammar compiler50, which compiles the grammar into a binary format that is easily loaded into the speech engine22. Speech engine22also includes a speech recognizer52that recognizes spoken phrases in sound information from the sound card driver40based on the grammar that has been loaded onto the speech engine. Other software can interact with speech engine22through a speech API54. The API54may, for example, be a speech application-programming interface (“SAPI”) developed by Microsoft Inc., an advanced speech API (“ASAPI”) by AT&T Corporation, or a JAVA™ speech application-programming interface (“JSAPI”) by Sun Microsystems.

The first speech-enabled application26is a speech-enabled video game. A user plays the video game using speech commands such as “jump” or “kick the monster in the stomach”. The user also performs other functions, such as sending mail to the technical support department of the vendor of the application, using audio commands. A user may, for example, say “send mail to technical support” to prepare a letter that is addressed to technical support. Speech-enabled application26is stored in a game directory60within storage subsystem18. The game directory60also includes the speech handler file30of the application26.

Handler functions62, which are contained within the first speech enabled application26, are executed whenever a phrase recognized by the speech engine22is directed at the application26, as described in greater detail below.

The second speech-enabled application28is a text editor. A user performs functions associated with entries in the menus of the text editor by audibly stating words corresponding to the functions. The user may, for example, say “save file” to save a file, “open file” to open a file, or “format in bold face” to format text in bold face font. A user also creates the body of a document by speaking the words he would like to include in the body. The user may, for example, begin a letter by saying “Dear Mother. How are you?” The computer10recognizes the spoken sounds and provides the text to the text editor as input.

Handler functions64, contained within the second speech enabled application28are executed whenever the speech engine22recognizes a phrase that is directed at the second speech enabled application. Unlike the first speech-enabled application26, the second speech-enabled application28has its speech handler file32embedded within the executable file of the application32as a resource file. The speech service24is configured to retrieve the speech handler file30from the application28, as will be described below.

Speech service24is a “service” or a “daemon” that does not does not interact with the user through the user interface46and is, typically, executed contemporaneously with other programs on the computer10. The speech service24is automatically started during the startup process of computer10. Speech service24comprises code modules that perform the functions described below.

Speech service24includes a speech engine launcher70that launches the speech engine22when the speech service24is first started. All speech-enabled applications26,28configured to use the speech service24that are later executed on the computer10use the same instance of the speech engine22through the speech server. Thus the speech service24eliminates additional processing that would otherwise be required to launch the speech engine22every time a speech application26is executed.

Speech service24also includes a speech handler file loader72that loads speech handler files30,32and a grammar generator74that generates a grammar file76and handling function information77afrom each speech handler file30,32. The grammar generator74, for example, generates the grammar file76as described in U.S. patent application Ser. No. 09/752,994, titled “Specifying Arbitrary Words In Rule-Based Grammars.” The handling function information77arelates different spoken phrases with corresponding functions that respond to the phrases. Each grammar file76informs the speech engine22of the different phrases to which a speech-enabled application26,28responds. The grammar is typically a context-free grammar with wild card support, although any grammar supported by the speech engine22may be used instead. A sample grammar file is attached as appendix A.

The grammar generator74also causes the grammar compiler50of the speech engine22to compile each grammar file76, producing compiled grammars78. Compiled grammars78are stored within storage subsystem18, preferably within a rapidly accessible memory, from where they are later retrieved, as described in greater below. The speech service24also includes a default grammar81that is loaded into the speech engine22when the speech service is first launched.

Speech service24further includes a focus module79, which is notified by the operating system20whenever a user changes focus from one application to another. Focus module79keeps track of the application that has focus. Speech service24includes a phrase parser84. Speech engine22notifies the phrase parser84of any phrases in the sound input that are recognized by the speech engine. As will be described in greater detail below, the phrase parser84parses the recognized phrases to determine any arguments that may be required by the handler function that corresponds to the phrase. A function caller86calls the handler function62,64with the arguments determined by the phrase parser86. The function caller calls the handler function62,64using an API, such as remote procedure call (RPC), or the component object model (COM) by Microsoft Inc.

As shown inFIG. 2, upon launching the speech service24, the speech engine launcher70queries (200) the operating system20to determine whether there is a speech engine22installed on the computer10. The speech engine launcher70may query (200) the operating system20through the operating system API44, the system registry42, or by checking operating system configuration files. The speech engine launcher70then queries (202) the operating system20to check if the speech engine22is running. If the speech engine22is not running, the speech engine launcher70starts (204) the speech engine22.

As shown inFIG. 3, when a user initiates (300) the execution of an application, for example, by clicking on an icon associated with the application, the operating system20notifies (302) the focus module79. The speech handler file loader72then checks (304) if a speech handler file30,32is embedded within the executable of the application as a resource file. If a speech handler file is not embedded within the executable, the speech handler file loader72checks (306) if a speech handler file is stored within the same directory as the executable. If a speech handler file is not stored within the directory, the speech handler file loader72checks (308) if speech handler information associated with the application is stored within the system registry42. If the speech handler information is not stored within the registry, the speech handler file loader72terminates the process.

The focus module79records (310) identity information associated with the initiated application to indicate that the application has focus. The grammar generator74generates (312) a grammar from the handler information extracted from either the speech handler file30,32or the registry42. The grammar generator74then directs (314) the speech engine22to compile the generated grammar into a compiled grammar78a, which the grammar generator74stores (316) within storage subsystem18and associates (318) with the application. Grammar generator74also generates and stores (320) handling function information77afrom the handler information. The handling function information associates spoken phrases with handler functions62,64that respond to them.

Grammar loader80unloads (322) the grammar on the speech engine22and loads (324) the compiled grammar78aonto the speech engine. Subsequently, all speech input is recognized by the speech engine22based on the compiled grammar78acorresponding to the initiated application. By loading the grammar78athat is tailored to the application, the speech service24causes speech input to be more accurately recognized by the speech engine22.

As shown inFIG. 4, when the user speaks (402) into microphone12, speech engine22recognizes (404) phrases contained within the speech and notifies (406) the speech service24of the recognized phrase. The speech service24identifies (408) the application that has focus by reading identification information previously recorded by the focus module79. The speech service24retrieves (410) handling function information77aof the application that is in focus and selects (412) a handler function from the information corresponding to the recognized phrase. The phrase parser84then parses (414) the phrase to determine any parameter required to call the selected function, as described below with reference toFIG. 5, and then the function caller86calls (416) the function.

As shown inFIG. 5, a speech handler file90corresponding to a speech-enabled application26(FIG. 1) includes phrases92to which the application26responds and corresponding handler functions93that are to be invoked whenever a corresponding phrase is matched.

The phrases92include a simple phrase92athat consists of a single word “jump.” The handler function93acorresponding to the simple phrase92ais called without any arguments whenever the simple phrase92ais recognized.

The phrases92also include a wildcard phrase92bthat consists of a sub-phrase94(i.e., “sendmail to”) and a wildcard portion96(i.e., *recipient). The speech engine22matches the wildcard phrase92bwith any collection of words that begins with the sub-phrase94. The handler function93bassociated with the wildcard phrase92bmust be called with an argument98bnamed “recipient”. The name of the argument98balso occurs in the wildcard portion96of the wildcard phrase92b.

As will be described below, speech engine22recognizes a collection of words that matches the wildcard phrase92b, the phrase parser84parses the matching collection of words, extracting any words after the sub-phrase94into a variable named “recipient”. The speech service then calls the corresponding handler function93bwith the “recipient” variable as the sole argument.

Phrases92further include a complex phrase92ccomprising a first part “kick”, a wild card for a variable named “person” a second part “in” and a wildcard for a variable named “bodypart.” The complex phrase92cis matched by any collection of spoken words that has both the words “kick” and “in,” in that order. The handling function93cassociated with the complex phrase92cmust be called with the variables named “person” and “bodypart.”

When the speech engine22recognizes any collection of spoken words that match the phrase92c, the phrase parser84parses the collection of words and assigns the words between “kick” and “in” to a variable named “person”. The phrase parser84also assigns all words after “in” to a variable named “bodypart”. The function caller86then calls the associated handler function93cwith the variables “person” and “bodypart.”

Since the speech handler file90does not contain any information that is specific to a particular speech API, the speech engine24allows a simple speech application to be used with any speech API version that is compatible with the speech service. The instructions for communicating with the different APIs are embedded in the speech service.

As shown inFIG. 6, when a user changes (600) focus from the first application26to the second application28, the operating system20notifies (602) the focus module79of the change. The grammar loader80then unloads (604) the grammar corresponding to the first application from the speech engine22and loads (606) the grammar corresponding to the second application. By loading a grammar that is tailored to the application in focus, the speech service24allows the speech engine22to recognize phrases in the spoken input more accurately.

For example, the speech service may use other means to detect the application that has the focus of the user interface, instead of registering with the operating system to receive notification of a change of focus from one application to another. The speech service may periodically poll the operating system, for example every half-second, to determine whether there has been a change of focus. The polling method may also be used to determine whether the execution of a new application has been initiated.

Alternatively, the speech service may use the compiled grammars to determine the application that is in focus. In this implementation, the speech service combines all the compiled grammars into a composite grammar, which it loads onto the speech engine. When the speech engine recognizes a phrase from the composite grammar, the speech engine notifies the speech service. The speech service in turn parses the recognized phrase and determines which application has a handler function that responds to the recognized phrase. The speech service infers that the determined application is the application that is in focus.