Patent Description:
Automatic speech recognition (ASR) has become increasingly prevalent for controlling personal electronic devices. Oftentimes, devices, such as televisions, mobile phones, smart speakers, etc., include a search interface that enables users to search for content using a voice input. As the system receives a voice input from the user, the system generates an interpretation of a user's intended command. Because of variations in the way users speak, background noise, similarities between words, limitations of the ASR recognition capabilities, or the like, the interpretations frequently include one or more errors. For example, a user may speak a command "Play my top music hits from iTunes," but the ASR system may interpret the command to be "Play me top music hits from show tunes. " When an error has occurred, the user must re-speak the query in its entirety, even if the interpretation error occurred in only a portion of the command. Upon interpreting the command for a second time, the ASR system may correct one error while introducing a new one. For example, the system may incorrectly recognize the command the second time as "Play Mike Top music hits from iTunes," thereby requiring the user to re-speak the command once again. In contrast, when two people are having a conversation, and one person misunderstands the other, the speaker may repeat a misunderstood word without repeating the entirety of what was previously said; oftentimes giving different emphasis to the misunderstood word to indicate that a misunderstanding took place. For example, if Bob tells Alice, "This food is great!" and Alice replies, "No, it wasn't late. " Bob may reply "No, GREAT!" Without Bob repeating the entire prior statement, Alice would understand that Bob previously stated, "This food is great," not "This food is late.

An exemplary approach for speech recognition error correction is disclosed in <CIT>.

In view of the foregoing, the present disclosure provides systems and related methods according to independent claims <NUM> and <NUM> defining the invention and dependent claims <NUM>-<NUM> defining preferred embodiments that, by interpreting a second voice query to determine whether it is intended to correct a previous voice query, are able to correct interpretation errors of user voice commands more accurately than possible with prior systems and methods. In some examples, the systems and related methods described herein allow for a user to correct an ASR interpretation error without requiring the user to repeat the entire command, much as in conversational speech between two people. In such examples, the users can correct an interpretation error by repeating a word that was misinterpreted. For example, when the user speaks the command "Show me Game of Thrones," and the system interprets the command to be "Show me Game of Homes," the user may repeat just the word "Thrones" to correct the interpretation error. Upon detecting the speech input "Thrones," the system will determine that it incorrectly interpreted the word "thrones" as "homes" and will correct the interpretation. In some embodiments, the system will execute the command "Show me Game of Thrones" subsequent to the correction.

When a user speaks, a speech segment is received by a user input component, such as a microphone, and is provided to control circuitry of a device (e.g., a smart speaker or server associated with a personal electronic device). The control circuitry processes the speech segment using an ASR algorithm and determines an interpretation for the received segment. For example, the control circuitry may identify properties of the speech segment (e.g., an acoustic envelope, an intensity, a pitch, a frequency, an amplitude, etc.) and, based on the identified properties, may identify one or more words corresponding to the speech segment (e.g., "Show me game of homes"). The interpretation may be output to a user (e.g., by displaying text of the interpretation on a display, by synthesizing an audio response based on the interpretation, or by executing a command corresponding to the interpretation).

If the user detects an interpretation error, the user may correct the error by repeating the word having the incorrect interpretation. For example, if the user determines that the last word was misinterpreted as "homes" instead of "thrones," the user may speak the word "thrones" to correct the error. The control circuitry, in response to detecting the second speech segment (e.g., the segment containing only "thrones"), determines whether the second speech segment is intended to correct an interpretation error of the first speech segment or if the second speech segment is intended to be a new command.

In some embodiments, the control circuitry determines that the user intends to correct an interpretation error when one or more sound properties of the second segment (e.g., a pitch, an acoustic envelope, a frequency, an amplitude, a correction expression, etc.) match one or more sound properties of the first segment. If, for instance, the control circuitry determines that the sound properties of the first segment and the second segment match, except an intensity, the control circuitry may determine that the user is attempting to correct an interpretation error. For example, when the user determines that the word "thrones" was misinterpreted, the user may shout "THRONES" in a higher pitch shortly after the error is made. When the control circuitry detects that the second segment was spoken in a higher intensity and pitch, the system may determine that the user is attempting to correct an interpretation error. In this manner, the control circuitry can identify a correction to a previous interpretation by detecting natural conversational variations in how a user speaks a query.

The control circuitry may detect that a user intends to correct an interpretation error when there is a correction word in the second segment. For example, when the second segment is "No, thrones," the control circuitry will determine that because the user stated "No" (e.g., the correction word), the user is attempting to correct an interpretation error. In such embodiments, the control circuitry can correct an error by identifying words spoken by a user indicating that an error has occurred.

In other embodiments, the control circuitry determines that the user intends to correct an interpretation error when a word in the second segment (e.g., "Thrones") is frequently misinterpreted for a word interpreted from the first segment (e.g., "Homes"). For example, control circuitry may access a database storing pairs of frequently misinterpreted words and may determine that "Thrones" and "Homes" are frequently misinterpreted for each other. The control circuitry corrects the interpretation by replacing the word "Homes" with the word "thrones. " By accessing associations between words having frequent misinterpretations, the control circuitry can locate and correct an interpretation error without requiring the user to restate an entire command.

In some aspects, if more than a threshold amount of time has passed between receiving the first and second segment, the control circuitry will determine that the user is issuing a new command instead of trying to correct an interpretation error. For example, if more than <NUM> seconds have passed since receiving the first segment, the control circuitry will interpret the second segment as a new command and will not try to determine whether the user is trying to correct an ASR error for the first segment.

The above and other objects and advantages of the disclosure will be apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings, in which:.

<FIG> illustrates an overview of a scenario in which system <NUM> corrects an ASR error, in accordance with some embodiments of the disclosure. System <NUM> includes device <NUM>, such as a smart phone, a smart speaker, a smart television, or the like, that has various user interfaces configured to interact with one or more users. In some examples, device <NUM> has a display <NUM>, which is configured to display information via a graphical user interface, and a microphone <NUM>, which is configured to receive natural language queries (e.g., queries <NUM> and <NUM>) as they are uttered by a user. In some embodiments, device <NUM> has an audio driver, such as a speaker <NUM>, configured to audibly provide information, such as responses (e.g., responses <NUM> and <NUM>) to the received natural language queries (e.g., queries <NUM> and <NUM>). In some embodiments, system <NUM> includes a network (not depicted in <FIG>), such as the Internet, one or more servers (not depicted in <FIG>), and one or more content databases (not depicted in <FIG>). Device <NUM> may execute the ASR process locally on device <NUM>, or some or all of the ASR process may be executed by the one or more servers.

Device <NUM> successively receives natural language queries <NUM> and <NUM> (e.g., "Show me Game of Thrones" and "THRONES!" respectively) via microphone <NUM>. In some embodiments, prior to receiving queries <NUM> and <NUM>, device <NUM> activates listening on microphone <NUM> in response to triggering by a specific keyword or selection of a user interface element. For example, device <NUM> may activate listening on microphone in response to a user selecting a voice input button (e.g., button <NUM>) or by detecting a wake word such as "Hey Siri. " Device <NUM> may activate listening on microphone <NUM> for a predetermined amount of time or until a detected end of a voice input.

In response to detecting the voice input, device <NUM> may perform an ASR process to identify text corresponding to the spoken query. For example, upon detecting query <NUM> ("Show me Game of Thrones"), device <NUM> performs an ASR process to detect words present in the voice query. In some embodiments, device <NUM> identifies text in the query by identifying sound properties of the query (such as an acoustic envelope, an intensity, a pitch, a frequency, amplitude, etc.) and compares the sound properties to an acoustic model or speech corpus. Based on the comparison, device <NUM> determines a textual interpretation of the received query (query <NUM>). For example, device <NUM> may compare portions of the voice input to entries in the acoustic model and determine how closely each portion matches an entry in the acoustic model.

Device <NUM> may assign weights or confidence values to each word in the textual representation based on how closely the speech segment matches an entry in an acoustic model or speech corpus. For example, device <NUM> may process query <NUM> and may determine that the words having the highest weights or confidence values for each portion are "Show me Game of Homes. " A high confidence value (e.g.,. <NUM>) is assigned to the word "Show" when the corresponding portion of the voice input closely matches the acoustic model. In contrast, a low confidence value is assigned to the word "Homes" (e.g.,. <NUM>) and to the word "Thrones" (e.g.,. <NUM>) when the corresponding portion of the voice input partially matches the acoustic model entry for either "Homes" or "Thrones. " The system displays the interpretation of the query "Show me Game of Homes" (response <NUM>) on display <NUM> of device <NUM>. In some embodiments, device <NUM> synthesizes the text of response <NUM> to an audio signal and outputs the audio signal via speaker <NUM>. In some embodiments, device <NUM> will re-activate microphone <NUM> upon presenting response <NUM> to detect whether the user provides a subsequent voice input (e.g., a subsequent voice input correcting an ASR error).

ASR interpretation errors (such as detecting "homes" instead of "thrones") may occur when an input audio is not clear, there are differences between how the user speaks and the voice model, the user speaks a word that sounds similar to other words, and/or in other similar circumstances. In instances where ASR errors occur, the systems and associated methods provided herein allow for a user to cause device <NUM> to correct the ASR error without requiring repetition of the previous query in its entirety. The systems and methods described herein detect natural conversational methods that users utilize to correct interpretation errors and make corrections to ASR errors based on the detection.

When the user detects an ASR error (e.g., in response to device <NUM> displaying response <NUM> "Show me Game of Homes"), the user may trigger correction of the error by repeating a part of the query where the detection error occurred in a louder voice or at a higher pitch. For example, the user may speak query <NUM> ("THRONES!") loudly to indicate to device <NUM> that an interpretation error occurred. In response to detecting the second query, device <NUM> determines whether query <NUM> is intended to correct an ASR error or whether query <NUM> is a new command. An exemplary process for detecting whether the query is intended to correct an ASR error is described further with respect to <FIG>.

When device <NUM> determines that query <NUM> is intended to correct an ASR error, device <NUM> determines where in the error occurred and corrects the error. For example, device <NUM> may perform an ASR process on query <NUM> and may assign a. <NUM> confidence value that the word is "Homes" and assign a. <NUM> confidence value that the word is "thrones. " However, because the user intends to correct an ASR error in response <NUM>, the system may generate response <NUM> by replacing the word "Homes" with the word "Thrones. " Accordingly, a word with a higher confidence value ("Homes") is replaced with a word having a lower confidence value ("Thrones") because the system detected the user's intent to correct an ASR error.

The corrected reply (e.g., reply <NUM>) is output to the user by displaying the corrected response (response <NUM>) on display <NUM>. In some embodiments, the device performs a command based on the corrected response. For example, device <NUM> may display program listings for the show Game of Thrones in response to correcting the ASR error.

<FIG> is an illustrative block diagram showing additional details of an example of system <NUM> for correcting an ASR error, in accordance with some embodiments of the disclosure. Although <FIG> shows system <NUM> including a number and configuration of individual components, in some embodiments, any number of the components of system <NUM> may be combined and/or integrated as one device, such as device <NUM> of system <NUM>. System <NUM> includes computing device <NUM>, server <NUM>, and database <NUM>, each of which is communicatively coupled to communication network <NUM>, which may be the Internet or any other suitable network or group of networks. In some embodiments, system <NUM> excludes server <NUM>, and functionality that would otherwise be implemented by server <NUM> is instead implemented by other components of system <NUM>, such as computing device <NUM>. In still other embodiments, server <NUM> works in conjunction with computing device <NUM> to implement certain functionality described herein in a distributed or cooperative manner.

Server <NUM> includes control circuitry <NUM> and input/output (hereinafter "I/O") path <NUM>. Control circuitry <NUM> includes storage <NUM> and processing circuitry <NUM>. Computing device <NUM>, which may be device <NUM> depicted in <FIG>, a personal computer, a laptop computer, a tablet computer, a smartphone, a smart television, a smart speaker, or any other type of computing device, includes control circuitry <NUM>, I/O path <NUM>, speaker <NUM> (e.g., speaker <NUM> of device <NUM>), display <NUM> (e.g., display <NUM> of device <NUM>), and user input interface <NUM> (e.g., a touchscreen of display <NUM> or microphone <NUM> of device <NUM>). In some embodiments user input interface <NUM> includes a voice-user interface (e.g., microphone <NUM> coupled with voice processing circuitry) configured to receive natural language queries uttered by users. Control circuitry <NUM> includes storage <NUM> and processing circuitry <NUM>. Control circuitry <NUM> and/or <NUM> may be based on any suitable processing circuitry such as processing circuitry <NUM> and/or <NUM>. As referred to herein, processing circuitry should be understood to mean circuitry based on one or more microprocessors, microcontrollers, digital signal processors, programmable logic devices, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), etc., and may include a multi-core processor (e.g., dual-core, quad-core, hexa-core, or any suitable number of cores). In some embodiments, processing circuitry may be distributed across multiple separate processors, for example, multiple of the same type of processors (e.g., two Intel Core i9 processors) or multiple different processors (e.g., an Intel Core i7 processor and an Intel Core i9 processor).

Each of storage <NUM>, storage <NUM>, and/or storages of other components of system <NUM> (e.g., storages of database <NUM>, and/or the like) may be an electronic storage device. As referred to herein, the phrase "electronic storage device" or "storage device" should be understood to mean any device for storing electronic data, computer software, or firmware, such as random-access memory, read-only memory, flash memory, hard drives, optical drives, digital video disc (DVD) recorders, compact disc (CD) recorders, BLU-RAY disc (BD) recorders, BLU-RAY 3D disc recorders, digital video recorders (DVRs, sometimes called personal video recorders, or PVRs), solid state devices, quantum storage devices, gaming consoles, gaming media, or any other suitable fixed or removable storage devices, and/or any combination of the same. Each of storage <NUM>, storage <NUM>, and/or storages of other components of system <NUM> may be used to store various types of content, metadata, and or other types of data. Non-volatile memory may also be used (e.g., to launch a boot-up routine and other instructions). Cloud-based storage may be used to supplement storages <NUM>, <NUM> or instead of storages <NUM>, <NUM>. In some embodiments, control circuitry <NUM> and/or <NUM> executes instructions for an application stored in memory (e.g., storage <NUM> and/or <NUM>). Specifically, control circuitry <NUM> and/or <NUM> may be instructed by the application to perform the functions discussed herein. In some implementations, any action performed by control circuitry <NUM> and/or <NUM> may be based on instructions received from the application. For example, the application may be implemented as software or a set of executable instructions that may be stored in storage <NUM> and/or <NUM> and executed by control circuitry <NUM> and/or <NUM>. In some embodiments, the application may be a client/server application where only a client application resides on computing device <NUM>, and a server application resides on server <NUM>.

The application may be implemented using any suitable architecture. For example, it may be a stand-alone application wholly implemented on computing device <NUM>. In such an approach, instructions for the application are stored locally (e.g., in storage <NUM>), and data for use by the application is downloaded on a periodic basis (e.g., from an out-of-band feed, from an Internet resource, or using another suitable approach). Control circuitry <NUM> may retrieve instructions for the application from storage <NUM> and process the instructions to perform the functionality described herein. Based on the processed instructions, control circuitry <NUM> may determine what action to perform when input is received from user input interface <NUM>.

In client/server-based embodiments, control circuitry <NUM> may include communication circuitry suitable for communicating with an application server (e.g., server <NUM>) or other networks or servers. The instructions for carrying out the functionality described herein may be stored on the application server. Communication circuitry may include a cable modem, an Ethernet card, or a wireless modem for communication with other equipment, or any other suitable communication circuitry. Such communication may involve the Internet or any other suitable communication networks or paths (e.g., communication network <NUM>). In another example of a client/server-based application, control circuitry <NUM> runs a web browser that interprets web pages provided by a remote server (e.g., server <NUM>). For example, the remote server may store the instructions for the application in a storage device. The remote server may process the stored instructions using circuitry (e.g., control circuitry <NUM>) and/or generate displays. Computing device <NUM> may receive the displays generated by the remote server and may display the content of the displays locally via display <NUM>. This way, the processing of the instructions is performed remotely (e.g., by server <NUM>) while the resulting displays, such as the display windows described elsewhere herein, are provided locally on computing device <NUM>. Computing device <NUM> may receive inputs from the user via input interface <NUM> and transmit those inputs to the remote server for processing and generating the corresponding displays.

A user may send instructions to control circuitry <NUM> and/or <NUM> using user input interface <NUM>. User input interface <NUM> may be any suitable user interface, such as a remote control, trackball, keypad, keyboard, touchscreen, touchpad, stylus input, joystick, voice recognition interface, gaming controller, or other user input interfaces. User input interface <NUM> may be integrated with or combined with display <NUM>, which may be a projector, a liquid crystal display (LCD), a light emitting diode (LED) display, an electronic ink display, or any other equipment suitable for displaying visual images.

Server <NUM> and computing device <NUM> may transmit and receive content and data via I/O path <NUM> and <NUM>, respectively. For instance, I/O path <NUM> and/or I/O path <NUM> may include a communication port configured to transmit and/or receive (for instance to and/or from content database <NUM>), via communication network <NUM>, content item identifiers, natural language queries, responses to natural language queries, and/or other data. Control circuitry <NUM>, <NUM> may be used to send and receive commands, requests, and other suitable data using I/O paths <NUM>, <NUM>.

Having described system <NUM>, reference is now made to <FIG>, which depicts an illustrative flowchart for correcting an ASR error, which may be implemented by system <NUM> or <NUM>, in accordance with some embodiments of the disclosure. In various embodiments, individual steps of process <NUM>, or any process described herein, may be implemented by one or more components of system <NUM> or <NUM>. Although the present disclosure may describe certain steps of process <NUM> (and of other processes described herein) as being implemented by certain components of system <NUM>, this is for purposes of illustration only, and other components of system <NUM> may implement those steps instead. At <NUM>, control circuitry <NUM> detects a second voice input, such as query <NUM> and query <NUM>. As described above, control circuitry <NUM> receives the voice input by way of user input interface <NUM>. In some embodiments, detection of the second voice input includes detecting a wake word and then recording a second voice input received subsequent to detection of the wake word. For example, control circuitry <NUM> may determine that subsequent to the wake word "Hey Siri," a voice input stating "THRONES!".

At <NUM>, control circuitry <NUM> determines, based at least in part on a sound property of the second voice input (e.g., query <NUM>), whether the second voice input is directed to correcting a text string (e.g., response <NUM>) previously generated based on a first voice input (e.g., query <NUM>). A sound property of the second voice input may be a spectral/acoustic envelope, an intensity, a pitch, a frequency, an amplitude, a correction expression, etc. Control circuitry <NUM> may compare the sound properties for the first voice input (e.g., query <NUM>) to the sound properties of the second voice input (e.g., query <NUM>) and may determine that the user is attempting to correct the text string (e.g., response <NUM>) when at least a threshold value of the sound properties of the first and second voice inputs match but at least one differs. For example, the spectral envelope for the same words spoken by a user in both voice inputs (e.g., queries <NUM> and <NUM>) will approximately match. For example, control circuitry <NUM> may detect that the spectral envelope for the term "Thrones" in query <NUM> approximately matches the spectral envelope for the term "THRONES" in query <NUM>, and may detect that the intensity or pitch varies between the utterances of the term. Because the words sound the same, but they were stated in different pitches/intensities control circuitry <NUM> will determine that the user is attempting to correct response <NUM>.

At <NUM>, control circuitry <NUM>, in response to determining that the second voice input is directed to correcting the text string, modifies the text string based on the second voice input. For example, control circuitry <NUM> may modify the text string by locating the word that the user intends to correct (e.g., the word "Homes" in response <NUM>) based on the second voice input (e.g., the word "thrones" in query <NUM>). Control circuitry <NUM> may output the corrected text string to the user by displaying the modified text string (e.g., response <NUM>) on a display (e.g., display <NUM> or display <NUM>), converting the modified text string to audio and outputting via an audio device (e.g., speaker <NUM> or <NUM>) or by performing an action in response to the modified text string (e.g., presenting program listings for Game of Thrones).

<FIG> depicts an additional example process <NUM> for determining whether a user intends to correct an ASR error, in accordance with some embodiments of the present disclosure. At <NUM>, control circuitry <NUM> detects a voice input (e.g., query <NUM>) via a user input device such as microphone <NUM> or user input interface <NUM>. At <NUM>, control circuitry <NUM> performs a fast Fourier transform (FFT) on the voice input to identify the frequency components of the voice input. At <NUM>, control circuitry <NUM> extracts sound parameters based on the voice input and on the FFT. For example, control circuitry <NUM> may extract sound parameters such as an acoustic envelope, an intensity, a pitch, frequency, amplitude, or the presence of a correction expression.

At <NUM>, control circuitry <NUM> determines whether a time threshold is satisfied for detected voice input. For example, control circuitry <NUM> may retrieve a threshold time limit for estimating whether the user intends to correct a response or not. The time limit may be based on a computed average amount of time that it takes for a user to attempt to correct an interpretation error by a voice control device. For example, if, on average, a user will attempt to correct an error by a voice control device within <NUM> seconds, the threshold time limit may be set as <NUM> seconds. Therefore, any voice inputs received more than <NUM> seconds after outputting a response will be interpreted by control circuitry <NUM> as a new query. Any voice inputs received less than <NUM> seconds after outputting a response will be analyzed further to determine whether the user intends to correct a response by the system.

At <NUM>, control circuitry <NUM> determines whether there is an acoustic similarity between the second voice input and the first voice input. By determining whether there is an acoustic similarity between the second voice input and the first voice input, control circuitry <NUM> may more accurately predict whether the user intends to make a correction and if so, and may determine the location of the ASR error. <FIG> depicts exemplary process <NUM>, which may be employed at <NUM> to determine whether the second voice input is acoustically similar to the first voice input. At <NUM> control circuitry <NUM> identifies a first plurality of sound properties corresponding to the first voice input. For example, control circuitry <NUM> may identify properties of the sound wave and perform a FFT on the first voice input (e.g., query <NUM>) to identify an acoustic envelope, intensity, pitch, frequency, and/or amplitude. At <NUM>, control circuitry <NUM> identifies a second plurality of sound properties (e.g., acoustic envelope, intensity, pitch, frequency, and/or amplitude) corresponding to the second voice input (e.g., query <NUM>).

At <NUM>, control circuitry <NUM> determines whether at least a portion of the first plurality of sound properties match at least a portion of the second plurality of sound properties. For example, control circuitry <NUM> may compare the acoustic envelope of query <NUM> to the acoustic envelope of query <NUM>. When at least a first portion of the acoustic envelope for the first voice input (e.g., the portion of query <NUM> corresponding to the word "thrones") matches at least a second portion of the acoustic envelope for the second voice input (e.g., the portion of query <NUM> corresponding to the word "THRONES"), control circuitry <NUM> may determine that the user is attempting to perform a correction to the corresponding portion of response <NUM> (e.g., the word "Homes" in response <NUM>).

At <NUM>, control circuitry <NUM> determines whether both the time threshold is satisfied and whether there was an acoustic similarity between the first voice input (e.g., query <NUM>) and the second voice input (e.g., query <NUM>). When both conditions are met, the user has uttered a word in the second query (e.g., query <NUM>) that sounds similar to a word in the first query (e.g., query <NUM>) within a short amount of time, it is possible that the user is attempting to correct a previously generated text string (e.g., response <NUM>). However, these two conditions alone may not be enough to determine whether the user is correcting a previous query or issuing a new command. For example, sometimes the user may use two words that sound the same in temporally proximate queries when the user is trying to change a context of the conversation. In an example, a first query "Show me Game of Thrones episodes" and the second query "Show me Game of Thrones soundtrack," the second query does not attempt to correct the first query. Control circuitry <NUM> determines whether there are any additional linguistic elements that indicate whether the user is attempting to correct a previously generated response, such as an intensity spike (e.g., user speaks louder), pitch change (e.g., user increases pitch of voice), or the presence of a corrective expression (e.g., "No," "Wrong," etc.).

At <NUM>, control circuitry <NUM> determines whether there is an intensity spike between the first voice input and the second voice input. When two people are having a conversation and one person misunderstands the other, one may raise the volume of their voice so that the other person can better understand. Accordingly, when control circuitry <NUM> detects such an occurrence, control circuitry <NUM> may determine that the user is trying to correct an ASR error. For example, control circuitry <NUM> may measure an intensity of query <NUM> (e.g., in dB) and may measure an intensity of query <NUM>. Control circuitry <NUM> may compare the two measured intensities and may determine whether the intensity of the second voice input (e.g., query <NUM>) surpasses the intensity of the first voice input (e.g., query <NUM>) by a threshold amount. Control circuitry <NUM> may retrieve the threshold from a database, such as database <NUM>. The threshold may vary based on the intensity level or may be a static value. For example, the threshold may be 1dB or may be 1dB when the intensity is less than 50dB, but may be 2dB when the intensity is greater than 50dB. As depicted in <FIG>, query <NUM> is depicted having a shouting volume (all capital letters) whereas query <NUM> is depicted having a normal volume (sentence case). If the intensity of the second voice input (e.g., query <NUM>) surpasses that of the first voice input (e.g., query <NUM>) by the threshold amount, and both the time threshold is satisfied and there is acoustic similarity between the two voice inputs, control circuitry <NUM> determines that the user is inputting a correction.

At <NUM>, control circuitry <NUM> determines whether there is a pitch change between the first voice input (e.g., query <NUM>) and the second voice input (e.g., query <NUM>). Because in natural conversations a person may indicate that a misunderstanding occurred by changing the pitch of their voice, control circuitry <NUM> determines whether there is a pitch change between the first voice input and the second voice input to determine whether the second voice input is a correction. For example, control circuitry <NUM> may detect the sound properties for both the first voice input and the second voice input as discussed above. Control circuitry <NUM> may compare the pitch of the first voice input with the pitch of the second voice input and may determine that there was a pitch change when a difference between the pitch values is greater than a threshold amount. In some instances, the threshold value may be stored on and retrieved from a database, such as database <NUM>. The threshold value may be a static value or may be based on the measured pitch (e.g., a percentage of the measured pitch). When the difference between the pitch values exceeds the threshold value, control circuitry <NUM> determines that there is a pitch change. By detecting the pitch change, control circuitry <NUM> can more accurately determine whether the user intends to correct an ASR error or the user is issuing a new command.

At <NUM>, control circuitry <NUM> determines whether the second voice input (e.g., query <NUM>) contains a corrective expression. When two people are speaking, one may indicate to the other that there is a misunderstanding by stating a corrective expression such as "No," "Wrong," "Not," etc. In one example, control circuitry <NUM> determines whether the second voice input comprises a corrective expression by converting the second voice input to text and determining whether any words in the text match entries in a database of corrective expressions (e.g., database <NUM>). In another example, control circuitry <NUM> may compare sound phones detected in a speech signal to a database of corrective expression phones (e.g., database <NUM>) and determine whether the phones detected in the speech signal match phones in the database. As an example, control circuitry <NUM> would not detect the presence of a corrective word in query <NUM> ("THRONES") but would detect the presence of a corrective expression when the second voice input is "No, Thrones.

At <NUM>, control circuitry <NUM> determines whether any of the conditions at <NUM>, <NUM>, and <NUM> are true. Because a positive identification of an intensity spike, pitch change, or corrective expression alone is not enough to indicate that the user intends to correct an ASR error, at <NUM>, control circuitry <NUM> determines whether the results at <NUM> and <NUM> are both true. At <NUM>, control circuitry <NUM> determines that the user intends to correct an ASR error when the time threshold is satisfied (e.g., the second voice inputs was received in close temporal proximity to response <NUM> or was received in close temporal proximity to query <NUM>), there is acoustic similarity between the voice inputs (e.g., a sound in the first voice input partially matches a sound in the second voice input), and at least one of the following is true: <NUM>) an intensity spike occurred between the first and second voice input; <NUM>) a pitch change occurred between the first and second voice input; or <NUM>) the second voice input contained a corrective expression. If the results at <NUM> and <NUM> are not both true, then control circuitry <NUM> determines that the user is not making a correction and may accordingly process the second voice input as a new query.

<FIG> depicts an additional example process <NUM> for correcting an ASR error based on words in a voice input, in accordance with some embodiments of the present disclosure. At <NUM>, control circuitry <NUM> converts the second voice input to a second text string. For example, control circuitry <NUM> may perform an ASR process on query <NUM> to identify a word in the query - "Thrones. " At <NUM>, control circuitry <NUM> determines, based on a database of misrecognized words, that a first word in the first test string and a second word in the second text string are misrecognized for one another. For example, control circuitry <NUM> may look up an entry for "thrones" in database <NUM> and may retrieve words that are commonly misrecognized as thrones. Entries in the database may be determined based on how closely phones for the word match another. For example, the database entry for "thrones" may list the words "homes" and "throws. " When the first text string (e.g., response <NUM>) contains a first word matching the database entry, control circuitry <NUM> determines that the user intends to correct the first word ("homes") for the second word in the second voice input ("thrones").

At <NUM>, control circuitry <NUM> modifies the first text string based on the second voice input by replacing the first word in the first text string with the second word. For example, control circuitry <NUM> modifies the text string by replacing the word "homes" with the word "thrones" and generates the modified text string "Show me Game of Thrones" (response <NUM>).

Because in conventional conversational speech a user may vary the pitch of a word to indicate that a misunderstanding occurred, control circuitry <NUM> may additionally determine whether there is a variation in the pitch between the first portion of the first query and the second portion of the second query. When there is no variation in the pitch, control circuitry <NUM> may determine that the user is issuing a new command having a word that was also in the previous query. When there is a variation in the pitch, control circuitry <NUM> may determine that the user is varying the pitch to correct an error in a previous response.

Claim 1:
A method for correcting a speech recognition error, the method comprising:
detecting a second voice input;
determining, based at least in part on a sound property of the second voice input, whether the second voice input is directed to correcting a text string previously generated based on a first voice input; and
in response to the determining that the second voice input is directed to correcting the text string, modifying the text string based on the second voice input,
wherein determining whether the second voice input is directed to correcting the text string comprises:
determining that the second voice input was received within a predetermined amount of time since the generation of the text string; and
determining that at least one of an acoustic envelope, an intensity, a pitch, a frequency, or an amplitude of the second voice input exceeds a threshold.