Patent Description:
Speech recognition is often used to enable an electronic device to interpret spoken questions or commands from users. Such spoken questions or commands can be identified by analyzing an audio signal, such as a microphone input, at an automatic speech recognition (ASR) engine that generates a textual output of the spoken questions or commands. An "always-on" ASR system enables the electronic device to continually scan audio input to detect user commands or questions in the audio input. However, continual operation of the ASR system results in relatively high power consumption, which reduces battery life when implemented in a mobile device.

In some devices, a spoken voice command will not be recognized unless it is preceded by a spoken activation keyword. Recognition of the activation keyword enables such devices to activate the ASR engine to process the voice command. However, speaking an activation keyword before every command uses additional time and requires the speaker to use correct pronunciation and proper intonation. In other devices, a dedicated button is provided for the user to press to initiate speech recognition. However, in some circumstances, such as when operating a vehicle, locating and precisely pressing the button can result in a diversion of the user's attention from other tasks. <CIT> describes a mobile terminal including an input unit configured to receive an input to activate a voice recognition function on the mobile terminal and a memory configured to store multiple domains related to menus and operations of the mobile terminal. The voice recognition function may be activated by interpreting body movements of the user. It further includes a controller configured to access a specific domain among the multiple domains included in the memory based on the received input to activate the voice recognition function, to recognize user speech based on a language model of the accessed domain, and to determine at least one menu and operation of mobile terminal based on the accessed specific domain and the recognized user speech. <CIT> describes an apparatus for speech recognition including a plurality of trigger detection units, each of which being configured to detect a start trigger for recognizing a command utterance for controlling the device. The trigger detection units consist of a gesture-trigger detection unit, a handclap-trigger detection unit, and a voice-trigger detection unit. <CIT> describes a control apparatus to be connected to a route guidance apparatus, comprising a hand information detection part for detecting information on a hand of a user from a taken image. <CIT> describes a vehicle rear-view mirror with an infrared sensor that may be used to activate a voice recognition function module.

The scope of the present invention is defined by the appended independent claims, with the dependent claims providing further preferred embodiments.

Devices and methods to activate a speech recognition system are disclosed. Because an always-on ASR system that continually scans audio input to detect user commands or questions in the audio input results in relatively high power consumption, battery life is reduced when the ASR engine is implemented in a mobile device. In an attempt to reduce power consumption, some systems may use a reduced-capacity speech recognition processor that consumes less power than a full-power ASR engine to perform keyword detection on the audio input. When an activation keyword is detected, the full-power ASR engine can be activated to process a voice command that follows the activation keyword. However, requiring a user to speak an activation keyword before every command is time consuming and requires the speaker to use correct pronunciation and proper intonation. Devices that require the user to press a dedicated button to initiate speech recognition can result in an unsafe diversion of the user's attention, such as when operating a vehicle.

As described herein, speech recognition is activated in response to detecting a hand over a portion of a device, such as a user's hand hovering over a screen of the device. The user can activate speech recognition for a voice command by positioning the user's hand over the device and without having to speak an activation keyword or having to precisely locate and press a dedicated button. Removal of the user's hand from over the device can signal that the user has finished speaking the voice command. As a result, speech recognition can be activated conveniently and safely, such as when the user is operating a vehicle. Further, because positioning the user's hand over the device can signal the device to initiate the speech recognition and removing the user's hand from over the device signals an end of the user's voice command, improper activation of the speech recognition and inaccurate detection of the end of voice commands can both be reduced.

Unless expressly limited by its context, the term "producing" is used to indicate any of its ordinary meanings, such as calculating, generating, and/or providing. Unless expressly limited by its context, the term "providing" is used to indicate any of its ordinary meanings, such as calculating, generating, and/or producing. Unless expressly limited by its context, the term "coupled" is used to indicate a direct or indirect electrical or physical connection. If the connection is indirect, there may be other blocks or components between the structures being "coupled". For example, a loudspeaker may be acoustically coupled to a nearby wall via an intervening medium (e.g., air) that enables propagation of waves (e.g., sound) from the loudspeaker to the wall (or vice-versa).

The term "configuration" may be used in reference to a method, apparatus, device, system, or any combination thereof, as indicated by its particular context. Where the term "comprising" is used in the present description and claims, it does not exclude other elements or operations. The term "based on" (as in "A is based on B") is used to indicate any of its ordinary meanings, including the cases (i) "based on at least" (e.g., "A is based on at least B") and, if appropriate in the particular context, (ii) "equal to" (e.g., "A is equal to B"). In the case (i) where A is based on B includes based on at least, this may include the configuration where A is coupled to B. Similarly, the term "in response to" is used to indicate any of its ordinary meanings, including "in response to at least. " The term "at least one" is used to indicate any of its ordinary meanings, including "one or more". The term "at least two" is used to indicate any of its ordinary meanings, including "two or more.

The terms "apparatus" and "device" are used generically and interchangeably unless otherwise indicated by the particular context. Unless indicated otherwise, any disclosure of an operation of an apparatus having a particular feature is also expressly intended to disclose a method having an analogous feature (and vice versa), and any disclosure of an operation of an apparatus according to a particular configuration is also expressly intended to disclose a method according to an analogous configuration (and vice versa). The terms "method," "process," "procedure," and "technique" are used generically and interchangeably unless otherwise indicated by the particular context. The terms "element" and "module" may be used to indicate a portion of a greater configuration. The term "packet" may correspond to a unit of data that includes a header portion and a payload portion. Any incorporation by reference of a portion of a document shall also be understood to incorporate definitions of terms or variables that are referenced within the portion, where such definitions appear elsewhere in the document, as well as any figures referenced in the incorporated portion.

As used herein, the term "communication device" refers to an electronic device that may be used for voice and/or data communication over a wireless communication network. Examples of communication devices include smart speakers, speaker bars, cellular phones, personal digital assistants (PDAs), handheld devices, headsets, wireless modems, laptop computers, personal computers, etc..

<FIG> depicts a system <NUM> that includes a device <NUM> that is configured to activate an ASR system <NUM> to process an input sound <NUM>, such as a voice command, when at least a portion of a hand <NUM> is positioned over the device <NUM>. The device <NUM> includes one or more microphones, represented as a microphone <NUM>, a screen <NUM>, one or more sensors <NUM>, a hand detector <NUM>, and the ASR system <NUM>. A perspective view <NUM> illustrates the hand <NUM> positioned over the device <NUM>, and a block diagram <NUM> illustrates components of the device <NUM>. In some implementations, the device <NUM> can include a portable communication device (e.g., a "smart phone"), a vehicle system (e.g., a speech interface for an automobile entertainment system, navigation system, or self-driving control system), a virtual reality or augmented reality headset, or a wireless speaker and voice command device with an integrated assistant application (e.g., a "smart speaker" device), as illustrative, non-limiting examples.

The microphone <NUM> is configured to generate an audio signal <NUM> responsive to the input sound <NUM>. In some implementations, the microphone <NUM> is configured to be activated, responsive to an indication <NUM>, to generate the audio signal <NUM>, as described further with reference to <FIG>.

The one or more sensors <NUM> are coupled to the hand detector <NUM> and configured to provide sensor data <NUM> to the hand detector <NUM>. For example, the sensor(s) <NUM> can include one or more cameras, such as a low-power ambient light sensor or a main camera, an infrared sensor, an ultrasound sensor, one or more other sensors, or any combination thereof, such as described further with reference to <FIG>.

The hand detector <NUM> is configured to generate the indication <NUM> responsive to detection of at least a portion of a hand being positioned within a range of <NUM> to <NUM> from the one or more sensors <NUM>, such as over the screen <NUM>. As used herein, "at least a portion of a hand" can correspond to any part of a hand (e.g., one or more fingers, a thumb, a palm or a back of the hand, or any portion thereof, or any combination thereof) or can correspond to an entire hand, as illustrative, non-limiting examples. As used herein, "detecting a hand" is equivalent to "detecting at least a portion of a hand" and can include detecting two or more fingers, detecting at least one finger connected to a portion of a palm, detecting a thumb and at least one finger, detecting a thumb connected to at least a portion of a palm, or detecting an entire hand (e.g., four fingers, a thumb, and a palm), as illustrative, non-limiting examples.

Although the hand <NUM> is described as being detected "over" the device <NUM>, "over" the device <NUM> refers to being located at a specified relative position (or within a specified range of positions) relative to the position and orientation of the one or more sensors <NUM>. In an example in which the device <NUM> is oriented so that the sensor(s) <NUM> face upward, such as illustrated in <FIG>, detecting the hand <NUM> over the device <NUM> indicates that the hand <NUM> is above the device <NUM>. In an example in which the device <NUM> is oriented so that the sensor(s) <NUM> face downward, detecting the hand <NUM> over the device <NUM> indicates that the hand <NUM> is below the device <NUM>.

The hand detector <NUM> is configured to process the sensor data <NUM> to determine whether the hand <NUM> is detected over the device <NUM>. For example, as described further with reference to <FIG>, in some implementations the hand detector <NUM> processes image data to determine whether a hand shape has been captured by a camera, processes infrared data to determine whether a detected temperature of the hand <NUM> corresponds to a hand temperature, processes ultrasound data to determine whether a distance between the hand <NUM> and the device <NUM> is within a specified range, or a combination thereof.

The device <NUM> is configured to generate a notification for a user of the device <NUM> to indicate, responsive to detecting the hand <NUM> being positioned within a range of <NUM> to <NUM> from the one or more sensors <NUM>, that speech recognition has been activated, and is further configured to generate a second notification to indicate, responsive to no longer detecting the hand <NUM> over the device <NUM>, that voice input for speech recognition is deactivated. For example, the device <NUM> may be configured to generate an audio signal such as a chime or a voice message such as "ready," a visual signal such as an illuminated or blinking light, a digital signal to be played out by another device, such as by a car entertainment system in communication with the device, or any combination thereof. Generating the notification(s) enables the user to confirm that the device <NUM> is ready to receive a voice command and may further enable the user to detect and prevent false activations (e.g., caused by another object that may be misidentified as the hand <NUM>) and missed activations due to improper positioning of the hand <NUM>. Because each activation of the ASR system <NUM> consumes power and uses processing resources, reducing false activations results in reduced power consumption and processing resource usage.

The ASR system <NUM> is configured to be activated, responsive to the indication <NUM>, to process the audio signal <NUM>. In an illustrative example, a specific bit of a control register represents the presence or absence of the indication <NUM> and a control circuit within or coupled to the ASR system <NUM> is configured to read the specific bit. A "<NUM>" value of the bit corresponds to the indication <NUM> and causes the ASR system <NUM> to activate. In other implementations, the indication <NUM> is instead implemented as a digital or analog signal on a bus or a control line, an interrupt flag at an interrupt controller, or an optical or mechanical signal, as illustrative, non-limiting examples.

When activated, the ASR system <NUM> is configured to process one or more portions (e.g., frames) of the audio signal <NUM> that include the input sound <NUM>. For example, the device <NUM> can buffer a series of frames of the audio signal <NUM> as the sensor data <NUM> being processed by the hand detector <NUM> so that, upon the indication <NUM> being generated, the ASR system <NUM> can process the buffered series of frames and generate an output indicative of the user's speech. The ASR system <NUM> can provide recognized speech <NUM> as a text output of the speech content of the input sound <NUM> to another component of the device <NUM>, such as a "virtual assistant" application or other application as described with reference to <FIG>, to initiate an action based on the speech content.

When deactivated, the ASR system <NUM> does not process the audio signal <NUM> and consumes less power than when activated. For example, deactivation of the ASR system <NUM> can include gating an input circuit of the ASR system <NUM> to prevent the audio signal <NUM> from being input to the ASR system <NUM>, gating a clock signal to prevent circuit switching within the ASR system <NUM>, or both, to reduce dynamic power consumption. As another example, deactivation of the ASR system <NUM> can include reducing a power supply to the ASR system <NUM> to reduce static power consumption without losing the state of the circuit elements, removing power from at least a portion of the ASR system <NUM>, or a combination thereof.

In some implementations, the hand detector <NUM>, the ASR system <NUM>, or any combination thereof, are implemented using dedicated circuitry or hardware. In some implementations, the hand detector <NUM>, the ASR system <NUM>, or any combination thereof, are implemented via execution of firmware or software. To illustrate, the device <NUM> can include a memory configured to store instructions and one or more processors configured to execute the instructions to implement the hand detector <NUM> and the ASR system <NUM>, such as described further with reference to <FIG>.

During operation, a user can position the user's hand <NUM> over the device <NUM> prior to speaking a voice command. The hand detector <NUM> processes the sensor data <NUM> to determine that the hand <NUM> is positioned within a range of <NUM> to <NUM> from the one or more sensors <NUM>. In response to detecting the hand <NUM> being positioned within a range of <NUM> to <NUM> from the one or more sensors <NUM>, the hand detector <NUM> generates the indication <NUM>, which causes activation of the ASR system <NUM>. After receiving the voice command at the microphone <NUM>, the ASR system <NUM> processes the corresponding portion(s) of the audio signal <NUM> to generate the recognized speech <NUM> indicating the voice command.

Activation of the ASR system <NUM> when a hand is detected over the device <NUM> enables a user of the device <NUM> to activate speech recognition for a voice command by positioning the user's hand <NUM> over the device, without the user having to speak an activation keyword or having to precisely locate and press a dedicated button. As a result, speech recognition can be activated conveniently and safely, such as when the user is operating a vehicle. Further, because positioning the user's hand over the device signals the device to initiate speech recognition, improper activation of the speech recognition can both be reduced as compared to a system that instead uses keyword detection to activate speech recognition.

<FIG> depicts an example <NUM> showing further aspects of components that can be implemented in the device <NUM> of <FIG>. As illustrated in <FIG>, the sensors <NUM> include one or more cameras <NUM> configured to provide image data <NUM> to the hand detector <NUM>, an infrared (IR) sensor <NUM> configured to provide infrared sensor data <NUM> to the hand detector <NUM>, and an ultrasound sensor <NUM> configured to provide ultrasound sensor data <NUM> to the hand detector. The image data <NUM>, the infrared sensor data <NUM>, and the ultrasound sensor data <NUM> are included in the sensor data <NUM>. The cameras <NUM> include a low-power ambient light sensor <NUM> configured to generate at least part of the image data <NUM>, a main camera <NUM> configured to generate at least part of the image data <NUM>, or both. Although the main camera <NUM> can capture image data having a higher resolution than the ambient light sensor <NUM>, the ambient light sensor <NUM> can generate image data having sufficient resolution to perform hand detection and operates using less power than the main camera <NUM>.

The hand detector <NUM> includes a hand pattern detector <NUM>, a hand temperature detector <NUM>, a hand distance detector <NUM>, and an activation signal unit <NUM>. The hand pattern detector <NUM> is configured to process the image data <NUM> to determine whether the image data <NUM> includes a hand pattern <NUM>. In an example implementation, the hand pattern detector <NUM> processes the image data <NUM> using a neural network trained to recognize the hand pattern <NUM>. In another example implementation, the hand pattern detector <NUM> applies one or more filters to the image data <NUM> to identify the hand pattern <NUM>. The hand pattern detector <NUM> is configured to send a first signal <NUM> to the activation signal unit <NUM> that indicates whether the hand pattern <NUM> is detected. Although a single hand pattern <NUM> is depicted, in other implementations multiple hand patterns may be included that represent differing aspects of a hand, such as a fingers-together pattern, a fingers-spread pattern, a partial hand pattern, etc..

The hand temperature detector <NUM> is configured to process the infrared sensor data <NUM> from the infrared sensor <NUM> and to send a second signal <NUM> to the activation signal unit <NUM> that indicates whether the infrared sensor data <NUM> indicates a temperature source having a temperature indicative of a human hand. In some implementations, the hand temperature detector <NUM> is configured to determine whether at least a portion of a field of view of the infrared sensor <NUM> has a temperature sources in a temperature range indicative of a human hand. In some implementations, the hand temperature detector <NUM> is configured to receive data indicating a location of a hand from the hand pattern detector <NUM> to determine whether a temperature source at the hand location matches the temperature range of a human hand.

The hand distance detector <NUM> is configured to determine a distance <NUM> between the hand <NUM> and at least a portion of the device <NUM>. In an example, the hand distance detector <NUM> processes the ultrasound sensor data <NUM> and generates a third signal <NUM> that indicates whether the hand <NUM> is within specified a range <NUM> of distances. In some implementations the hand distance detector <NUM> receives data from the hand pattern detector <NUM>, from the hand temperature detector <NUM>, or both, that indicates a location of the hand <NUM> and uses the hand location data to determine a region in the field of view of the ultrasound sensor <NUM> that corresponds to the hand <NUM>. In other implementations, the hand distance detector <NUM> identifies the hand <NUM> by locating a nearest object to the screen <NUM> that exceeds a specified portion (e.g., <NUM>%) of the field of view of the ultrasound sensor <NUM>.

The range <NUM> has a lower bound of <NUM> centimeters (cm) and an upper bound of <NUM> (i.e., the range <NUM> includes distances that are greater than or equal to <NUM> and less than or equal to <NUM>). In other implementations, the range <NUM> is adjustable. For example, the device <NUM> may be configured to perform an update operation in which the user positions the hand <NUM> in a preferred position relative to the device <NUM> so that the distance <NUM> can be detected and used to generate the range <NUM> (e.g., by applying a lower offset from the detected distance <NUM> to set a lower bound and applying an upper offset from the detected distance <NUM> to set an upper bound).

The activation signal unit <NUM> is configured to generate the indication <NUM> responsive to the first signal <NUM> indicating detection of the hand pattern <NUM> in the image data <NUM>, the second signal <NUM> indicating detection of a hand temperature within a human hand temperature range, and the third signal <NUM> indicating detection that the hand <NUM> is within the range <NUM> (e.g., the hand <NUM> is at a distance <NUM> of <NUM> centimeters to <NUM> centimeters from the screen <NUM>). For example, in an implementation in which each of the signals <NUM>, <NUM>, and <NUM> has a binary "<NUM>" value indicating detection and a binary "<NUM>" value indicating no detection, the activation signal unit <NUM> can generate the indication <NUM> as a logical AND of the signals <NUM>, <NUM>, and <NUM> (e.g., the indication <NUM> has a <NUM> value in response to all three signals <NUM>, <NUM>, <NUM> having a <NUM> value). In another example, the activation signal unit <NUM> is also configured to generate the indication <NUM> having a <NUM> value in response to any two of the signals <NUM>, <NUM>, <NUM> having a <NUM> value.

In other implementations, one or more of the signals <NUM>, <NUM>, and <NUM> has a multi-bit value indicating a likelihood that the corresponding hand detection criterion is satisfied. For example, the first signal <NUM> may have a multi-bit value that indicates a confidence that a hand pattern is detected, the second signal <NUM> may have a multi-bit value that indicates a confidence that a hand temperature is detected, and the third signal <NUM> may have a multi-bit value that indicates a confidence that the distance of the hand <NUM> from the device <NUM> is within the range <NUM>. The activation signal unit <NUM> can combine the signals <NUM>, <NUM>, and <NUM> and compare the combined result to a threshold to generate the indication <NUM>. For example, the activation signal unit <NUM> may apply a set of weights to determine a weighted sum of the signals <NUM>, <NUM>, and <NUM>. The activation signal unit <NUM> may output the indication <NUM> having a value indicating hand detection responsive to the weighted sum exceeding the threshold. Values of weights and thresholds can be hardcoded or, alternatively, can be dynamically or periodically adjusted based on user feedback regarding false positives and false negatives, as described further below.

The hand detector <NUM> is further configured to generate a second indication <NUM> in response to detection that the hand <NUM> is no longer over the device <NUM>. For example, hand detector may output the second indication <NUM> as having a <NUM> value (indicating that hand removal is not detected) responsive to detecting the hand <NUM>, and may update the second indication <NUM> to have a <NUM> value in response to determining that the hand is no longer detected (e.g., to indicate a transition from a "hand detected" state to a "hand not detected" state). The second indication <NUM> can correspond to an end-of-utterance signal for the ASR system <NUM>, as explained further with reference to <FIG>.

Although <FIG> depicts multiple sensors including the ambient light sensor <NUM>, the main camera <NUM>, the infrared sensor <NUM>, and the ultrasound sensor <NUM>, in other implementations one or more of the ambient light sensor <NUM>, the main camera <NUM>, the infrared sensor <NUM>, or the ultrasound sensor <NUM> is omitted. For example, although the ambient light sensor <NUM> enables generation of at least part of the image data <NUM> to detect hand shapes using reduced power as compared to using the main camera <NUM>, in some implementations the ambient light sensor <NUM> is omitted and the main camera <NUM> is used to generate the image data. For power reduction, the main camera <NUM> can be operated according to an on/off duty cycle, such as at quarter-second intervals, for hand detection. As another example, although the main camera <NUM> enables generation of at least part of the image data <NUM> to detect hand shapes with higher resolution, and therefore higher accuracy, as compared to using the ambient light sensor <NUM>, in some implementations the main camera <NUM> is omitted and the ambient light sensor <NUM> is used to generate the image data <NUM>.

As another example, although the infrared sensor <NUM> enables generation of at the infrared sensor data <NUM> to detect whether an object has a temperature matching a human hand temperature, in other implementations the infrared sensor <NUM> is omitted and the device <NUM> performs hand detection without regard to temperature. As another example, although the ultrasound sensor <NUM> enables generation of the ultrasound sensor data <NUM> to detect whether a distance to an object is within the range <NUM>, in other implementations the ultrasound sensor <NUM> is omitted and the device <NUM> performs hand detection without regard to distance from the device <NUM>. Alternatively, one or more other mechanisms can be implemented for distance detection, such as by comparing object locations in image data from multiple cameras of the device <NUM> (e.g., parallax) o multiple cameras of a different device (e.g., a vehicle in which the device <NUM> is located) to estimate the distance <NUM>, by using a size of a detected hand in the image data <NUM> or in the infrared sensor data <NUM> to estimate the distance <NUM>, or by projecting structured light or other electromagnetic signals estimate object distance, as illustrative, non-limiting examples.

Although increasing a number of sensors and a variety of sensor types generally enhances accuracy of hand detection, in some implementations two sensors or a single sensor provides sufficient accuracy for hand detection. As a non-limiting example, in some implementations the only sensor data used for hand detection is the image data <NUM> from the ambient light sensor <NUM>. Although in some implementations the sensors <NUM> are concurrently active, in other implementations one or more of the sensors <NUM> are controlled according to a "cascade" operation in which power is conserved by having one or more of the sensor <NUM> remain inactive until a hand detection criterion is satisfied based on sensor data from another of the sensors <NUM>. To illustrate, the main camera <NUM>, the infrared sensor <NUM>, and the ultrasound sensor <NUM> may remain inactive until the hand pattern detector <NUM> detects the hand pattern <NUM> in the image data <NUM> generated by the ambient light sensor <NUM>, in response to which one or more of the main camera <NUM>, the infrared sensor <NUM>, and the ultrasound sensor <NUM> is activated to provide additional sensor data for enhanced accuracy of hand detection.

<FIG> depicts an example <NUM> showing further aspects of components that can be implemented in the device <NUM>. As illustrated in <FIG>, activation circuitry <NUM> is coupled to the hand detector <NUM> and to the ASR system <NUM>, and the ASR system includes a buffer <NUM> that is accessible to an ASR engine <NUM>. The device <NUM> also includes a virtual assistant application <NUM> and a speaker <NUM> (e.g., the device <NUM> implemented as a wireless speaker and voice command device).

The activation circuitry <NUM> is configured to activate the automatic speech recognition system <NUM> in response to receiving the indication <NUM>. For example, the activation circuitry <NUM> is configured to generate an activation signal <NUM> in response to the indication <NUM> transitioning to a state that indicates hand detection (e.g., the indication <NUM> transitions from a <NUM> value indicating no hand detection to a <NUM> value indicating hand detection). The activation signal <NUM> is provided to the ASR system <NUM> via a signal <NUM> to activate the ASR system <NUM>. Activating the ASR system <NUM> includes initiating buffering of the audio signal <NUM> at the buffer <NUM> to generate buffered audio data <NUM>. The activation signal <NUM> is also provided to the microphone <NUM> via a signal <NUM> that activates the microphone <NUM>, enabling the microphone to generate the audio signal <NUM>.

The activation circuitry <NUM> is also configured to generate an end-of-utterance signal <NUM>. For example, the activation circuitry <NUM> is configured to generate the end-of-utterance signal <NUM> in response to the second indication <NUM> transitioning to a state that indicates an end of hand detection (e.g., the second indication <NUM> transitions from a <NUM> value (indicating no change in hand detection) to a <NUM> value (indicating that a detected hand is no longer detected)). The end-of-utterance signal <NUM> is provided to the ASR system <NUM> via a signal <NUM> to cause the ASR engine <NUM> to begin processing of the buffered audio data <NUM>.

The activation circuitry <NUM> is configured to selectively activate one or more components of the ASR system <NUM>. For example, the activation circuitry <NUM> may include or be coupled to power management circuitry, clock circuitry, head switch or foot switch circuitry, buffer control circuitry, or any combination thereof. The activation circuitry <NUM> may be configured to initiate powering-on of the buffer <NUM>, the ASR engine <NUM>, or both, such as by selectively applying or raising a voltage of a power supply of the buffer <NUM>, the ASR engine <NUM>, or both. As another example, the activation circuitry <NUM> may be configured to selectively gate or un-gate a clock signal to the buffer <NUM>, the ASR engine <NUM>, or both, such as to prevent circuit operation without removing a power supply.

The recognized speech <NUM> output by the ASR system <NUM> is provided to the virtual assistant application <NUM>. For example, the virtual assistant application <NUM> may be implemented by one or more processors executing instructions, such as described in further detail with reference to <FIG>. The virtual assistant application <NUM> may be configured to perform one or more search queries, such as via wireless connection to an internet gateway, search server, or other resource, searching a local storage of the device <NUM>, or a combination thereof.

To illustrate, the audio signal <NUM> may represent the spoken question "what the weather like today?" The virtual assistant application <NUM> may generate a query to access an Internet-based weather service to obtain a weather forecast for a geographic region in which the device <NUM> is located. The virtual assistant application <NUM> is configured to generate an output, such as an output audio signal <NUM> that causes the speaker <NUM> to generate an auditory output, such as in a voice interface implementation. In other implementations, the virtual assistant application <NUM> generates another mode of output, such as a visual output signal that may be displayed by a screen or display that is integrated in the device <NUM> or coupled to the device <NUM>.

In some implementations, values of parameters, such as weights and thresholds used by the device <NUM> (e.g., in the hand detector <NUM>) can be set by a manufacturer or provider of the device <NUM>. In some implementations, the device <NUM> is configured to adjust one or more such values during the life of the device <NUM> based on detected false negatives, false activations, or a combination thereof, associated with the ASR system <NUM>. For example, a history of false activations can be maintained by the device <NUM> so that the characteristics of sensor data <NUM> that triggered the false activations can be periodically used to automatically adjust one or more weights or thresholds, such as to emphasize the relative reliability of one sensor over another for use in hand detection, to reduce a likelihood of future false activations.

Although particular values are included in the descriptions of <FIG>, such as a "<NUM>" value to indicate a positive result (e.g., hand detection) and a "<NUM>" value to indicate a negative result, it will be understood that such values are provided for explanatory purposes only and are not limitations. To illustrate, in some implementations the indication <NUM> is indicated by a "<NUM>" value. As another example, in some implementations a "<NUM>" value of the first signal <NUM> indicates a high likelihood that the hand pattern <NUM> is in the image data <NUM>, while in other implementations a "<NUM>" value of the first signal <NUM> indicates a low likelihood the hand pattern <NUM> is in the image data <NUM>. Similarly, in some implementations a "<NUM>" value of the second signal <NUM>, the third signal <NUM>, or both, indicates a high likelihood that a hand detection criterion is satisfied, in other implementations the "<NUM>" value of the second signal <NUM>, the third signal <NUM>, or both, indicates a high likelihood that a hand detection criterion is not satisfied.

<FIG> depicts an implementation <NUM> of a device <NUM> that includes the hand detector <NUM> and the ASR system <NUM> integrated in a discrete component, such as a semiconductor chip or package as described further with reference to <FIG>. The device <NUM> includes an audio signal input <NUM>, such as a first bus interface, to enable the audio signal <NUM> to be received from a microphone external to the device <NUM>. The device <NUM> also includes a sensor data input <NUM>, such as a second bus interface, to enable the sensor data <NUM>, to be received from one or more sensors external to the device <NUM>. The device <NUM> may further include one or more outputs to provide processing results (e.g., the recognized speech <NUM> or the output audio signal <NUM>) to one or more external components (e.g., the speaker <NUM>). The device <NUM> enables implementation of hand detection and voice recognition activation as a component in a system that includes a microphone and other sensors, such as in a vehicle as depicted in <FIG>, a virtual reality or augmented reality headset as depicted in <FIG>, or a wireless communication device as depicted in <FIG>.

Referring to <FIG>, a particular implementation of a method <NUM> of processing an audio signal representing input sound is depicted that may be performed by the device <NUM> or the device <NUM>. The method begins at <NUM> and includes determining whether a hand is over a screen of the device, at <NUM>, such as by the hand detector <NUM> processing the sensor data <NUM>. In response to detecting a hand over the screen, a microphone and buffer are activated, at <NUM>. For example, the microphone <NUM> and the buffer <NUM> of <FIG> are activated by the activation circuitry <NUM> via the signals <NUM> and <NUM>.

In response to determining that the hand has been removed from over the screen, at <NUM>, the method <NUM> includes activating an ASR engine to process the buffered data, at <NUM>. For example, the ASR engine <NUM> is activated by the signal <NUM> generated by the activation circuitry <NUM> to process the buffered audio data <NUM>.

Activating ASR when a hand is detected over the screen enables a user to activate speech recognition for a voice command by positioning of the user's hand without having to speak an activation keyword or locate and press a dedicated button. As a result, speech recognition can be activated conveniently and safely, such as when the user is operating a vehicle. Further, because positioning the user's hand over the screen initiates activation of components to receive a voice command for speech recognition and removing the user's hand from over the screen initiates processing of the received voice command, improper activation, deactivation, or both, of speech recognition can both be reduced as compared to a system that instead uses keyword detection to activate speech recognition.

Referring to <FIG>, a particular implementation of a method <NUM> of processing an audio signal representing input sound is depicted that may be performed by the device <NUM> or the device <NUM>, as illustrative, non-limiting examples.

The method <NUM> starts at <NUM> and includes detecting, at a device, at least a portion of a hand over at least a portion of the device, at <NUM>. For example, the hand detector <NUM> detects the hand <NUM> via processing the sensor data <NUM> received from the one or more sensors <NUM>. In some implementations, detecting the portion of the hand over the portion of the device includes processing image data (e.g., the image data <NUM>) to determine whether the image data includes a hand pattern (e.g., the hand pattern <NUM>). In an example, the image data is generated at a low-power ambient light sensor of the device, such as the ambient light sensor <NUM>. Detecting the portion of the hand over the portion of the device may further include processing infrared sensor data from an infrared sensor of the device, such as the infrared sensor data <NUM>. Detecting the portion of the hand over the portion of the device may also include processing ultrasound sensor data from an ultrasound sensor of the device, such as the ultrasound sensor data <NUM>.

The method <NUM> includes, responsive to detecting the portion of the hand over the portion of the device, activating an automatic speech recognition system to process the audio signal, at <NUM>. For example, the device <NUM> activates the ASR system <NUM> in response to the indication <NUM>. In some implementations, activating the automatic speech recognition system includes initiating buffering of the audio signal, such as the device <NUM> (e.g., the activation circuitry <NUM>) activating the buffer <NUM> via the signal <NUM>. In some examples, responsive to detecting the portion of the hand over the portion of the device, such as over a screen of the device, the method <NUM> further includes activating a microphone to generate the audio signal based on the input sound, such as the device <NUM> (e.g., the activation circuitry <NUM>) activating the microphone <NUM> via the signal <NUM>.

In some implementations, the method <NUM> includes detecting that the portion of the hand is no longer over the portion of the device, at <NUM>, and responsive to detecting that the portion of the hand is no longer over the portion of the device, providing an end-of-utterance signal to the automatic speech recognition system, at <NUM>. In an example, the hand detector <NUM> detects that the hand is no longer over the portion of the device, and the activation circuitry <NUM> provides the end-of-utterance signal <NUM> to the ASR engine <NUM> responsive to the second indication <NUM>.

By activating the ASR system responsive to detecting a hand over a portion of the device, the method <NUM> enables a user to activate speech recognition for a voice command without having to speak an activation keyword or locate and press a dedicated button. As a result, speech recognition can be activated conveniently and safely, such as when the user is operating a vehicle. In addition, false activation of the ASR system can both be reduced as compared to a system that instead uses keyword detection to activate speech recognition.

The method <NUM> of <FIG>, the method <NUM> of <FIG>, or both, may be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a DSP, a controller, another hardware device, firmware device, or any combination thereof. As an example, the method <NUM> of <FIG>, the method <NUM> of <FIG>, or both, may be performed by a processor that executes instructions, such as described with reference to <FIG>.

<FIG> depicts an example of an implementation <NUM> of the hand detector <NUM> and the ASR system <NUM> integrated into a vehicle dashboard device, such as a car dashboard device <NUM>. A visual interface device, such as the screen <NUM> (e.g., a touchscreen display), is mounted within the car dashboard device <NUM> to be visible to a driver of the car. The microphone <NUM> and one or more sensors <NUM> are also mounted in the car dashboard device <NUM>, although in other implementations one or more of the microphone <NUM> and the sensor(s) <NUM> can be located elsewhere in the vehicle, such as the microphone <NUM> in a steering wheel or near the driver's head. The hand detector <NUM> and the ASR system <NUM> as illustrated with dashed borders to indicate that the hand detector <NUM> and the ASR system <NUM> are not visible to occupants of the vehicle. The hand detector <NUM> and the ASR system <NUM> may be implemented in a device that also includes the microphone <NUM> and sensor(s) <NUM> such as in the device <NUM> of <FIG>, or may be separate from and coupled to the microphone <NUM> and sensor(s) <NUM>, such as in the device <NUM> of <FIG>.

In some implementations, multiple microphones <NUM> and sets of sensors <NUM> are integrated into the vehicle. For example, a microphone and set of sensors can be positioned at each passenger seat, such as at an armrest control panel or seat-back display device, to enable each passenger to enter voice commands using hand-over-device detection. In some implementations, each passenger's voice command may be routed to a common ASR system <NUM>; in other implementations, the vehicle includes multiple ASR systems <NUM> to enable concurrent processing of voice commands from multiple occupants of the vehicle.

<FIG> depicts an example of an implementation <NUM> of the hand detector <NUM> and the ASR system <NUM> integrated into a headset <NUM>, such as a virtual reality or augmented reality headset. The screen <NUM> is positioned in front of the user's eyes to enable display of augmented reality or virtual reality images or scenes to the user while the headset <NUM> is worn, and the sensor(s) <NUM> are positioned to detect when the user's hand is over (e.g., in front of) the screen <NUM> to initiate ASR recognition. The microphone <NUM> is located to receive the user's voice while the headset <NUM> is worn. While wearing the headset <NUM>, the user can lift a hand in front of the screen <NUM> to indicate to the headset <NUM> that the user is about to speak a voice command to activate ASR, and can lower the hand to indicate that the user has finished speaking the voice command.

<FIG> depicts a block diagram of a particular illustrative implementation of a device <NUM> that includes the hand detector <NUM> and the ASR engine <NUM>, such as in a wireless communication device implementation (e.g., a smartphone). In various implementations, the device <NUM> may have more or fewer components than illustrated in <FIG>. In an illustrative implementation, the device <NUM> may correspond to the device <NUM>. In an illustrative implementation, the device <NUM> may perform one or more operations described with reference to <FIG>.

In a particular implementation, the device <NUM> includes a processor <NUM> (e.g., a central processing unit (CPU)). The device <NUM> may include one or more additional processors <NUM> (e.g., one or more DSPs). The processors <NUM> may include a speech and music coder-decoder (CODEC) <NUM> and the hand detector <NUM>. The speech and music codec <NUM> may include a voice coder ("vocoder") encoder <NUM>, a vocoder decoder <NUM>, or both.

The device <NUM> may include a memory <NUM> and a CODEC <NUM>. The memory <NUM> may include instructions <NUM>, that are executable by the one or more additional processors <NUM> (or the processor <NUM>) to implement the functionality described with reference to the hand detector <NUM>, the ASR engine <NUM>, the ASR system <NUM> of <FIG>, the activation circuitry <NUM>, or any combination thereof. The device <NUM> may include a wireless controller <NUM> coupled, via a transceiver <NUM>, to an antenna <NUM>.

The device <NUM> may include a display <NUM> (e.g., the screen <NUM>) coupled to a display controller <NUM>. The speaker <NUM> and the microphone <NUM> may be coupled to the CODEC <NUM>. The CODEC <NUM> may include a digital-to-analog converter <NUM> and an analog-to-digital converter <NUM>. In a particular implementation, the CODEC <NUM> may receive analog signals from the microphone <NUM>, convert the analog signals to digital signals using the analog-to-digital converter <NUM>, and provide the digital signals to the speech and music codec <NUM>. The speech and music codec <NUM> may process the digital signals, and the digital signals may further be processed by the ASR engine <NUM>. In a particular implementation, the speech and music codec <NUM> may provide digital signals to the CODEC <NUM>. The CODEC <NUM> may convert the digital signals to analog signals using the digital-to-analog converter <NUM> and may provide the analog signals to the speaker <NUM>.

In a particular implementation, the device <NUM> may be included in a system-in-package or system-on-chip device <NUM>. In a particular implementation, the memory <NUM>, the processor <NUM>, the processors <NUM>, the display controller <NUM>, the CODEC <NUM>, and the wireless controller <NUM> are included in a system-in-package or system-on-chip device <NUM>. In a particular implementation, an input device <NUM> (e.g., one or more of the sensor(s) <NUM>) and a power supply <NUM> are coupled to the system-on-chip device <NUM>. Moreover, in a particular implementation, as illustrated in <FIG>, the display <NUM>, the input device <NUM>, the speaker <NUM>, the microphone <NUM>, the antenna <NUM>, and the power supply <NUM> are external to the system-on-chip device <NUM>. In a particular implementation, each of the display <NUM>, the input device <NUM>, the speaker <NUM>, the microphone <NUM>, the antenna <NUM>, and the power supply <NUM> may be coupled to a component of the system-on-chip device <NUM>, such as an interface or a controller.

The device <NUM> may include a smart speaker (e.g., the processor <NUM> may execute the instructions <NUM> to run the voice-controlled digital assistant application <NUM>), a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) or Blu-ray disc player, a tuner, a camera, a navigation device, a virtual reality of augmented reality headset, a vehicle console device, or any combination thereof.

In conjunction with the described implementations, an apparatus to process an audio signal representing input sound includes means for detecting at least a portion of a hand over at least a portion of a device. For example, the means for detecting the portion of the hand can correspond to the hand detector <NUM>, the hand pattern detector <NUM>, the hand temperature detector <NUM>, the hand distance detector <NUM>, one or more other circuits or components configured to detect at least a portion of hand over at least a portion of a device, or any combination thereof.

The apparatus also includes means for processing the audio signal. The means for processing is configured to be activated responsive to detection of the portion of a hand over at the portion of the device. For example, the means for processing the audio signal can correspond to the ASR system <NUM>, the ASR engine <NUM>, the microphone <NUM>, the CODEC <NUM>, the speech and music codec <NUM>, one or more other circuits or components configured to process the audio signal and activated responsive to detection of the portion of a hand over at the portion of the device, or any combination thereof.

In some implementations, the apparatus includes means for displaying information, and the means for detecting is configured to detect the portion of the hand over the means for displaying information. For example, the means for displaying information can include the screen <NUM>, the display <NUM>, the display controller <NUM>, one or more other circuits or components configured to display information, or any combination thereof.

The apparatus may also include means for generating the audio signal based on the input sound, the means for generating configured to be activated responsive to detect the portion of the hand over the means for displaying information. For example, the means for generating the audio signal can correspond to the microphone <NUM>, a microphone array, the CODEC <NUM>, the speech and music codec <NUM>, one or more other circuits or components configured to generate the audio signal based on the input sound and to be activated responsive to the first indication, or any combination thereof.

In some implementations, the apparatus includes means for generating image data, and the means for detecting is configured to determine whether the image data includes a hand pattern, such as the hand pattern detector <NUM>. In some implementations, the apparatus includes at least one of: means for detecting a temperature associated with the portion of the hand (e.g., the hand temperature detector <NUM>, the infrared sensor <NUM>, or a combination thereof), and means for detecting a distance of the portion of the hand from the device (e.g., the hand distance detector <NUM>, the ultrasound sensor <NUM>, a camera array, a structured light projector, one or more other mechanism for detecting a distance of the portion of the hand from the device, or any combination thereof).

In some implementations, non-transitory computer-readable medium (e.g., the memory <NUM>) includes instructions (e.g., the instructions <NUM>) that, when executed by one or more processors of a device (e.g., the processor <NUM>, the processor(s) <NUM>, or any combination thereof), cause the one or more processors to perform operations for processing an audio signal representing input sound. The operations include detecting at least a portion of a hand over at least a portion of the device (e.g., at the hand detector <NUM>). For example, detecting the portion of the hand over the portion of the device can include receiving the sensor data <NUM>, processing the sensor data <NUM> using one or more detectors (e.g., the hand pattern detector <NUM>, the hand temperature detector <NUM>, or the hand distance detector <NUM>) to determine whether one or more detection criteria are met, and generating the indication <NUM> at least partially in response to detection that the one or more criteria are met (e.g., as described with reference to the activation signal unit <NUM>). For example, in some implementations, processing the sensor data <NUM> to determine whether a detection criterion is met includes applying a neural network classifier (e.g., as described with reference to the hand pattern detector <NUM>) that is trained to recognize the hand pattern <NUM> to process the image data <NUM> or applying one or more filters to the image data <NUM> to detect the hand pattern <NUM>.

The operations also include, responsive to detecting the portion of the hand over the portion of the device, activating an automatic speech recognition system to process the audio signal. For example, activating the automatic speech recognition can include detecting the indication <NUM> at an input to the ASR system <NUM> and, in response to detecting the indication <NUM>, performing at least one of a power-up or clock activation for at least one component (e.g., the buffer <NUM>, the ASR engine <NUM>) of the ASR system <NUM>.

Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, such implementation decisions are not to be interpreted as causing a departure from the scope of the present disclosure.

The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor may read information from, and write information to, the storage medium. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or user terminal.

Claim 1:
A device (<NUM>) to process an audio signal representing input sound (<NUM>), the device (<NUM>) comprising:
one or more sensors (<NUM>) coupled to a hand detector and configured to provide sensor data to the hand detector;
a hand detector (<NUM>);
and an automatic speech recognition system (<NUM>) including a buffer and an automatic speech recognition engine;
the hand detector (<NUM>) being configured to:
generate a first indication responsive to detection, via the one or more sensors (<NUM>), of at least a portion of a hand being positioned within a range of <NUM> to <NUM> from the one or more sensors (<NUM>); and
generate a second indication in response to detection that the portion of the hand is no longer positioned within a range of <NUM> to <NUM> from the one or more sensors (<NUM>), wherein the second indication corresponds to an end-of-utterance signal that causes the automatic speech recognition engine (<NUM>) to begin processing audio data from the buffer (<NUM>); and
the automatic speech recognition system (<NUM>) configured to initiate buffering of the audio signal responsive to the first indication, and to process the audio signal, wherein the audio signal comprises the input sound (<NUM>) received between the generation of the first indication and the generation of the second indication.