Patent Description:
Many audio headsets, such as true wireless stereo (TWS) headphones and gaming headsets, detect a user's voice in addition to playing audio. For example, many TWS headphones function as both a telephone headset for phone calls and an audio playback device for playing media (e.g., music).

When detecting the user's voice, the quality of the user's voice often suffers from environmental noise. In order to minimize the impact of environmental noise on the user's voice, many headsets include voice activity detection (VAD) to detect the presence or absence of human speech, and perform noise cancelling or suppression techniques in response to detecting the presence of human speech. As a result, the quality of the user's voice is improved during, for example, telephone calls.

Audio headsets typically implement VAD using a microphone and/or a bone conduction accelerometer. For example, a user's voice may be detected by a microphone as acoustic signals propagating through air, and by a bone conduction accelerometer as bone vibration signals propagating through the human body (e.g., bone and tissue vibrations through the jaw or ear canal). The detected acoustic signals and bone vibration signals are synchronized using, for example, time-division multiplexing (TDM). The audio headset then detects the presence or absence of the user's voice based on the synchronized acoustic signals and bone vibration signals. A hardware setup utilizing a bone vibration sensor for earphone conversation scenarios is demonstrated, for example, in <CIT>. An accelerometer used as bone conduction sensor for voice activity detection, in an in-ear communication device is disclosed in <CIT>.

VAD techniques that utilize a microphone and/or a bone conduction accelerometer typically involve processing large amounts of data for detected acoustic signals and bone vibration signals at a high rate. For example, each of the microphone and the bone conduction accelerometer may have a data rate between <NUM> kilohertz and <NUM> kilohertz, and the microcontroller may process data every <NUM> millisecond. Consequently, VAD techniques consume large amounts of power, and are, thus, often unsuitable for portable audio devices, such as true wireless stereo (TWS) headphones, that have a limited power supply. For example, a microphone typically consumes between <NUM> microamps and <NUM> microamps, a bone conduction accelerometer typically consumes between <NUM> milliamps and <NUM> milliamps, and a microcontroller for controlling the microphone and the bone conduction accelerometer typically consumes between <NUM> milliamps and <NUM> milliamps.

An aim of the invention is to provide voice activity detection that solves problems of prior solutions.

Accordingly, a device and a method for performing voice activity detection (VAD) are provided, as defined in the attached claims.

For the understanding of the present invention, embodiments thereof are now described, purely as non-limitative examples, with reference to the attached drawings, wherein:.

In the drawings, identical reference numbers identify similar features or elements. The size and relative positions of features in the drawings are not necessarily drawn to scale.

The present invention is directed to a device and method for performing VAD. The presence or absence of human speech is detected using a low-cost, low-power accelerometer, instead of a microphone and/or a bone conduction accelerometer used in current audio headsets. As a result, the overall current consumption of the device disclosed herein is greatly reduced compared to devices that utilize a microphone and/or a bone conduction accelerometer for VAD. As such, the device and method disclosed herein are ideal for portable audio devices, such as TWS headphones, that have a limited power supply.

In particular, the device and method generate an acceleration signal using the accelerometer, filter the acceleration signal with a band pass filter or a high pass filter, determine at least one calculation of the filtered acceleration signal, and detect a presence or absence of a voice based on the at least one calculation.

<FIG> is a block diagram of a device <NUM> for performing VAD. The device <NUM> may be any type of audio headset that detects a user's voice. For example, the device <NUM> may be TWS headphones, a gaming headset, a telephone headset, etc..

The device <NUM> includes a processing unit <NUM>, an accelerometer <NUM>, and a bone conduction accelerometer <NUM>. The device <NUM> may also include other components, such as a microphone for capturing voice signals.

The processing unit <NUM> is, for example, a processor, controller, signal processor, or microcontroller that controls and processes various functions of the device <NUM>. The processing unit <NUM> controls and coordinates the hardware components (e.g., the accelerometer <NUM> and the bone conduction accelerometer <NUM>) of the device <NUM>, and any features or applications of the device <NUM> (e.g., a pedometer, gesture recognition, activity recognition, tap detection, etc.). The processing unit <NUM> also gathers and processes data from the hardware components of the device <NUM> (e.g., acoustic signals generated by a microphone, bone vibration signals generated by the bone conduction accelerometer <NUM>, and acceleration signals generated by the accelerometer <NUM>).

The accelerometer <NUM> is communicatively coupled to the processing unit <NUM>. The accelerometer <NUM> measures acceleration of the device <NUM>, and generates an acceleration signal that indicates measured accelerations. The accelerometer <NUM> includes sensing circuitry configured to measure acceleration of the device <NUM> along at least one axis. The accelerometer may measure acceleration along three axes.

As will be discussed in further detail below, the accelerometer <NUM> also includes control or processing circuitry configured to detect a user's voice as bone vibration signals propagating through the human body (e.g., bone and tissue vibrations through the jaw or ear canal) for VAD.

The bone conduction accelerometer <NUM> may be communicatively coupled to the processing unit <NUM>. In the alternative, the bone conduction accelerometer <NUM> may be communicatively coupled to the accelerometer <NUM>.

The bone conduction accelerometer <NUM> is similar to the accelerometer <NUM>. For example, the bone conduction accelerometer <NUM> measures acceleration of the device <NUM>, and generates an acceleration signal that indicates measured accelerations. However, in contrast to the accelerometer <NUM>, the bone conduction accelerometer <NUM> is specialized to detect a user's voice as bone vibration signals propagating through the human body.

The bone conduction accelerometer <NUM> processes data at a high rate (e.g., between <NUM> kilohertz and <NUM> kilohertz), and typically includes a TDM interface to, for example, synchronize with acoustic signals detected by a microphone. Consequently, as discussed above, the bone conduction accelerometer <NUM> consumes large amounts of power (e.g., consumes between <NUM> milliamps and <NUM> milliamps).

In contrast, the accelerometer <NUM> is a conventional accelerometer that is both low-cost and low-power. The accelerometer <NUM> is not a bone conduction accelerometer that is specialized to detect a user's voice as bone vibration signals propagating through the human body. Rather, the accelerometer <NUM> is used to implement other applications of the device <NUM>, such as a pedometer, gesture recognition, activity recognition, and tap detection.

Compared to the bone conduction accelerometer <NUM>, the accelerometer <NUM> has a low data rate and does not consume large amounts of power. For example, the accelerometer <NUM> can have a data rate between <NUM> hertz and <NUM> hertz, and consume between <NUM> microamps and <NUM> microamps.

The accelerometer <NUM>, itself, detects the presence or absence of human speech. Stated differently, the steps or operations to perform VAD are implemented directly in hardware (e.g., control or processing circuitry) of the accelerometer <NUM>. As such, a separate, dedicated microcontroller to perform VAD is unnecessary. In addition, in some circumstances, the bone conduction accelerometer <NUM> may be turned off when not in use or even be removed from the device <NUM>.

<FIG> is a flow diagram of a method <NUM> of detecting a presence or absence of speech.

In block <NUM>, the accelerometer <NUM> measures acceleration of the device <NUM>, and generates an acceleration signal that indicates the measured accelerations.

The acceleration signal is indicative of bone vibration signals propagating through the human body (e.g., bone and tissue vibrations through the jaw or ear canal) that are caused by a user's voice. As discussed above, the accelerometer <NUM> may measure acceleration along a single axis or multiple axes.

In block <NUM>, the accelerometer <NUM> applies a filter to the acceleration signal generated in block <NUM>, and generates a filtered acceleration signal.

The accelerometer <NUM> applies the filter to the acceleration signal in order to remove frequencies outside of the frequency range of voiced speech. A voiced speech of a typical adult male has a fundamental frequency between <NUM> hertz and <NUM> hertz, and that of a typical adult female has fundamental frequency between <NUM> hertz and <NUM> hertz. Thus, the accelerometer <NUM> may apply a high pass filter with for example, a cutoff frequency at <NUM> hertz to remove frequencies outside of, e.g., below, the frequency range of voiced speech. Alternatively, the accelerometer <NUM> may apply a band pass filter with, for example, cutoff frequencies at <NUM> hertz and <NUM> hertz. As a result, the filtered acceleration signal generated in block <NUM> is indicative of acceleration measurements of vibrations signals caused by the user's voice, rather than vibrations signals caused by a surrounding environment or a user's movement.

<FIG> shows an acceleration signal <NUM> and a filtered acceleration signal <NUM> In <FIG>, a user is walking and talking concurrently. The horizontal axis represents time (seconds), and the vertical axis represents acceleration (milli-g).

The acceleration signal <NUM> is, for example, the acceleration signal generated in block <NUM>, and the filtered acceleration signal <NUM> is the acceleration signal <NUM> after applying a filter in block <NUM>. In <FIG>, a high pass filter having a cutoff frequency at <NUM> hertz is applied to the acceleration signal <NUM>. As discussed above, the filtered acceleration signal <NUM> represents acceleration measurements of vibrations signals caused by the user's voice because frequencies outside of the frequency range of voiced speech, which in this example are frequencies below <NUM> hertz, is removed from the acceleration signal <NUM> by the filter. As shown in <FIG>, the filtered acceleration signal <NUM> includes many features (e.g., peaks, zero crossings, etc.) that are indicative of voiced speech.

Returning to <FIG>, in block <NUM>, the accelerometer <NUM> extracts features from the filtered acceleration signal generated in block <NUM> (e.g., the filtered acceleration signal <NUM> in <FIG>).

The extracted features are distinguishing characteristics of the filtered acceleration measurements that are indicative of human speech. For instance, features are extracted from the filtered acceleration signal in the time domain. For example, the accelerometer <NUM> determines at least one of the following calculations: a peak-to-peak calculation (e.g., a difference between the maximum amplitude and the minimum amplitude of the filtered acceleration signal in a period of time), a zero crossing calculation (e.g., a number of times the filtered acceleration signal crosses zero in a period of time), a peak count calculation (e.g., a total number of peaks in the filtered acceleration signal in a period of time), or a variance calculation (e.g., a variance of the filtered acceleration signal in a period of time). Other types of calculations are also possible.

The features may be extracted within a time window of the filtered acceleration signal that is defined based on a desired minimum latency of the VAD algorithm (e.g., the method <NUM>). For example, the features may be calculated within a <NUM> seconds time window of the filtered acceleration signal.

In block <NUM>, the accelerometer <NUM> classifies the filtered acceleration signal as either human speech or not human speech based on the features extracted in block <NUM>. Stated differently, the accelerometer <NUM> detects the presence or absence of a user's voice by detecting whether the filtered acceleration signal, and in turn the acceleration signal, is a speech signal.

The accelerometer <NUM> uses a machine learning approach to classify the filtered acceleration signal as either human speech or not human speech. The accelerometer <NUM> may classify the filtered acceleration signal as either human speech or not human speech using at least one of a decision tree, a neural network, and a support vector machine. Other machine learning techniques are also possible.

Learning/inference machines may fall under the technological titles of machine learning, artificial intelligence, artificial neural networks (ANN), probabilistic inference engines, accelerators, and the like. Classification problems, such as VAD and other signal processing applications, benefit from the use of learning/inference machines, such as deep convolutional neural networks (DCNN), fuzzy-logic machines, etc. For example, a DCNN is a computer-based tool that processes large quantities of data and adaptively "learns" by conflating proximally related features within the data, making broad predictions about the data, and refining the predictions based on reliable conclusions and new conflations. The DCNN is arranged in a plurality of "layers," and different types of predictions are made at each layer.

The accelerometer <NUM> may extract different features along different axes of the device <NUM>, and utilize a decision tree to classify the filtered acceleration signal as either human speech or not human speech based on the extracted features along the different axes. For example, in block <NUM>, the accelerometer <NUM> determines, within a selected time window (e.g., <NUM> milliseconds), a first peak-to-peak calculation of the filtered acceleration signal along a z-axis of the device <NUM>, a second peak-to-peak calculation of the filtered acceleration signal along a x-axis of the device <NUM>, and a zero crossing calculation of the filtered acceleration signal along a y-axis of the device <NUM>.

Subsequently, in block <NUM>, the accelerometer classifies the filtered acceleration signal as either human speech or not human speech based on the first peak-to-peak calculation, the second peak-to-peak calculation, and the zero crossing calculation using a decision tree. <FIG> is a decision tree <NUM> that may be used to this end.

In block <NUM> of <FIG>, the accelerometer <NUM> determines whether the first peak-to-peak calculation is greater than a first threshold value. If the first peak-to-peak calculation is greater than the first threshold value, the decision tree <NUM> moves to block <NUM>. If the first peak-to-peak calculation is not greater than (i.e., is less than or equal to) the first threshold value, the decision tree <NUM> moves to block <NUM>.

In block <NUM>, the accelerometer <NUM> determines that the filtered acceleration signal is not human speech.

In block <NUM>, the accelerometer <NUM> determines whether the second peak-to-peak calculation is greater than a second threshold value. If the second peak-to-peak calculation is greater than the second threshold value, the decision tree <NUM> moves to block <NUM>. If the second peak-to-peak calculation is not greater than (i.e., is less than or equal to) the second threshold value, the decision tree <NUM> moves to block <NUM>.

In block <NUM>, the accelerometer <NUM> determines whether the zero crossing calculation is greater than a third threshold value. If the zero crossing calculation is greater than the third threshold value, the decision tree <NUM> moves to block <NUM>. If the zero crossing calculation is not greater than (i.e., is less than or equal to) the third threshold value, the decision tree <NUM> moves to block <NUM>.

In block <NUM>, the accelerometer <NUM> determines that the filtered acceleration signal is human speech.

The first, second, and third threshold values may be set to any values. Further, although peak-to-peak calculations and a zero crossing calculation are used along three axes in the decision tree <NUM>, any type of feature extracted in block <NUM> may be used (e.g., a peak-to-peak calculation, a zero crossing calculation, a peak count calculation, a variance calculation, etc.) along any number of axes.

In practice, human speech is detected based on the directionality of the features, by detecting different features along the monitored axes.

Here, human speech is detected if the filtered acceleration signal on a first axis (Z) is higher than a threshold, if the filtered acceleration signal on a second axis (X) is lower than another threshold and if the frequency (measured by counting the zero-crossings) on a third axis (Y) is higher than a still another threshold.

Returning to <FIG>, in block <NUM>, the accelerometer <NUM> uses a meta-classifier to filter classifications of the filtered acceleration signal in block <NUM>. The accelerometer <NUM> processes classifications of the filtered acceleration signal to remove or reduce false positives or false negatives.

For example, the accelerometer <NUM> reduces false detections of the filtered acceleration signal being human speech by maintaining a first count value. The first count value is a total number of times the accelerometer <NUM> classified the filtered acceleration signal as human speech. When the total number is equal to or greater than a first threshold count value, the accelerometer <NUM> determines that the filtered acceleration signal is human speech.

As an alternative, the accelerometer <NUM> reduces false detections of the filtered acceleration signal not being human speech by maintaining a second count value. The second count value is a total number of times the accelerometer <NUM> classified the filtered acceleration signal as not being human speech. When the total number is equal to or greater than a second threshold count value, the accelerometer <NUM> determines that the filtered acceleration signal is not human speech.

Block <NUM> may also be removed from the method <NUM> (i.e., not performed) to reduce latency of the method <NUM>.

In block <NUM>, the accelerometer <NUM> outputs the detection results of the method <NUM>. For example, the accelerometer <NUM> outputs, to the processing unit <NUM>, a detection signal indicating either that human speech is present (i.e., the filtered acceleration signal is a speech signal) or human speech is not present (i.e., the filtered acceleration signal is not a speech signal).

If the method <NUM> includes block <NUM>, the accelerometer <NUM> outputs a detection signal indicating that human speech is present in a case where the total number times the accelerometer <NUM> classified the filtered acceleration signal as human speech is equal to or greater than the first threshold count value. Conversely, the accelerometer <NUM> outputs a detection signal indicating that human speech is not present in a case where the total number of times the accelerometer <NUM> classified the filtered acceleration signal as not being human speech is equal to or greater than the second threshold count value.

If the method <NUM> does not include block <NUM>, the accelerometer <NUM> outputs a detection signal indicating that human speech is present in a case where the accelerometer <NUM> classifies the filtered acceleration signal as human speech in block <NUM>. Conversely, the accelerometer <NUM> outputs a detection signal indicating that human speech is not present in a case where the accelerometer <NUM> does not classify the filtered acceleration signal as human speech in block <NUM>.

The processing unit <NUM> may activate or deactivate the bone conduction accelerometer <NUM> based on the detection signal received from the accelerometer <NUM>. For example, the processing unit <NUM> may activate the bone conduction accelerometer <NUM> in a case where detection signal indicates that human speech is present, and deactivate the bone conduction accelerometer <NUM> in a case where detection signal indicates that human speech is not present. Accordingly, power consumption of the device <NUM> may be reduced as the bone conduction accelerometer <NUM> is activated when human speech is detected, rather than being continuously on. In the alternative, the accelerometer <NUM> directly activates or deactivates the bone conduction accelerometer <NUM>, without intervention from the processing unit <NUM>.

In the above description, the program or algorithm to perform the method <NUM> of detecting a presence or absence of human speech is implemented directly in hardware of the accelerometer <NUM>. However, the program or algorithm to perform the method <NUM> may be implemented in several different locations within the device <NUM>. For example, the program or algorithm to perform the method <NUM> may be implemented in the processing unit <NUM> instead of the accelerometer <NUM>. In this case, the processing unit <NUM> is configured to detect a presence or absence of speech as described above with respect to <FIG>. For example, the processing unit <NUM> may receive the acceleration signal generated in block <NUM> of the method <NUM>, and subsequently perform blocks <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. The processing unit <NUM> is also configured to receive and process an acceleration signal from the bone conduction accelerometer <NUM> and any other components included in the device <NUM>.

The device and method utilize a low-power, low-cost accelerometer to perform VAD. In addition, the intervention of the processing unit <NUM> is reduced. As a result, the overall current consumption of the device disclosed herein is greatly reduced compared to devices that utilize a microphone and/or a bone conduction accelerometer for VAD. As such, the device and method disclosed herein are well suited for portable audio devices, such as TWS headphones, that have a limited power supply.

Finally, it is clear that numerous modifications may be made to the device and method described and illustrated herein, all falling within the scope of the invention as defined in the attached claims. For example, the device may include only the conventional accelerometer <NUM>.

Claim 1:
A device (<NUM>) for performing voice activity detection, comprising:
an accelerometer (<NUM>) configured to:
measure a first acceleration of the device, and generate a first acceleration signal based on the measured first acceleration;
apply a filter to the first acceleration signal;
determine at least one characteristic of the filtered first acceleration signal;
detect whether the first acceleration signal is a speech signal based on the at least one characteristic; and
output a detection signal that indicates whether the first acceleration signal is a speech signal;
an operating system layer (<NUM>; <NUM>) configured to receive the detection signal; and
a bone conduction accelerometer (<NUM>) configured to measure a second acceleration of the device, and generate a second acceleration signal based on the measured second acceleration,
wherein the operating system layer (<NUM>; <NUM>) is configured to activate or deactivate the bone conduction accelerometer (<NUM>) based on the detection signal.