Voice Activity Detection Using Active Acoustic Sensing

Techniques and apparatuses are described that perform voice activity detection using active acoustic sensing. By transmitting and receiving acoustic signals, a hearable can recognize changes in an acoustic circuit to perform voice activity detection. With active acoustic sensing, the hearable can detect a vocalization made by a user in a noisy and/or loud environment. As such, the hearable can support a voice user interface (VUI) by providing an indication of when the user is speaking. The hearable can also support multi-factor voice authentication to enhance security and provide robust protection from voice attacks. In addition to being relatively unobtrusive, some hearables can be configured to support voice activity detection using active acoustic sensing without the need for additional hardware.

BACKGROUND

Wireless technology has become prevalent in everyday life, making communication and data readily accessible to users. One type of wireless technology are wireless hearables, examples of which include wireless earbuds and wireless headphones. Wireless hearables have allowed users freedom of movement while listening to audio content from music, audio books, podcasts, and videos. With the prevalence of wireless hearables, there is a market for adding additional features to existing hearables without introducing hardware changes.

SUMMARY

Techniques and apparatuses are described for performing voice activity detection using active acoustic sensing. By transmitting and receiving acoustic signals, a hearable can recognize changes in an acoustic circuit to perform voice activity detection. Voice activity detection involves detecting vocalizations made by a user. With active acoustic sensing, the hearable can detect a vocalization in a noisy and/or loud environment. As such, the hearable can support a voice user interface (VUI) by providing an indication of when the user is speaking. The hearable can also support multi-factor voice authentication to enhance security and provide robust protection from voice attacks. In addition to being relatively unobtrusive, some hearables can be configured to support voice activity detection using active acoustic sensing without the need for additional hardware. As such, the size, cost, and power usage of the hearable can help make voice activity detection accessible to a larger group of people and improve the user experience with hearables.

Aspects described below include a method for performing voice activity detection using active acoustic sensing. The method includes transmitting, during a first time period, an ultrasound transmit signal that propagates within at least a portion of an ear canal of a user. The method also includes receiving, during the first time period, an ultrasound receive signal. The ultrasound receive signal represents a version of the ultrasound transmit signal with one or more characteristics modified based on the propagation within the ear canal and based on a vocalization made by the user during the first time period. The method additionally includes detecting the vocalization based on the one or more modified characteristics of the ultrasound receive signal.

Aspects described below include a computer-readable storage medium comprising instructions that, responsive to execution by a processor, cause a hearable to perform any one of the methods described herein.

Aspects described below include a device with at least one transducer and at least one processor. The device is configured to perform, using the at least one transducer and the at least one processor, any one of the methods described herein.

Aspects described below include a system with means for performing voice activity detection using active acoustic sensing.

DETAILED DESCRIPTION

As electronic devices become more ubiquitous, users incorporate them into everyday life. A user, for example, may use an electronic device to get daily weather and traffic information, control a temperature of a home, answer a doorbell, turn on or off a light, and/or play background music. Interacting with some electronic devices, however, can be cumbersome and inefficient. An electronic device, for instance, can have a physical user interface that may require a user to navigate through one or more prompts by physically touching the electronic device. In this case, the user has to devote attention away from other primary tasks to interact with the electronic device, which can be inconvenient and disruptive.

To address this problem, some electronic devices support voice control, which enables a user to interact with the electronic device in a non-physical and less cognitively demanding way compared to other interfaces that require physical touch and/or the user's visual attention. With voice control, the electronic device seamlessly exists in the surrounding environment and provides the user access to information and services while the user performs a primary task, such as cooking, cleaning, driving, talking with people, or reading a book.

While voice control can provide a convenient means of interacting with an electronic device, there are several challenges associated with voice control. In a noisy environment, for instance, the user's voice can be made imperceptible by the other external noise. Consequently, it can be challenging for voice control to detect and/or recognize voice commands spoken by the user. Also, sometimes the noisy environment can cause the voice control to incorrectly respond to a voice of another person who is not authorized to use the electronic device.

Some devices address these challenges by integrating a voice accelerometer (VA) into earbuds. The voice accelerometer can detect a user speaking based on sound that travels by means of bone conduction. The voice accelerometer, however, can be bulky and expensive. To improve aesthetics and reduce encumbrance, it can be desirable to design hearables with smaller sizes. As space becomes limited, it can be challenging to integrate additional components, such as the voice accelerometer, within the hearables. With the prevalence of hearables, there is a market for adding additional features to existing hearables without introducing hardware changes.

Provided according to one or more preferred embodiments is a hearable, such as an earbud, that is capable of performing a novel physiological monitoring process termed herein audioplethysmography. Audioplethysmography is an active acoustic method capable of sensing subtle physiologically-related changes observable at a user's outer and middle ear. Instead of relying on other auxiliary sensors, such as optical or electrical sensors, audioplethysmography involves transmitting and receiving acoustic signals that at least partially propagate within a user's ear canal. To perform audioplethysmography, the hearable forms at least a partial seal in or around the user's outer ear. This seal enables formation of an acoustic circuit, which includes the seal, the hearable, the ear canal, and an ear drum of the ear. By transmitting and receiving acoustic signals, the hearable can recognize changes in the acoustic circuit to perform voice activity detection. Voice activity detection involves detecting vocalizations of the user. Vocalizations can include any sound that is produced using the user's lung's, vocal cords, and/or mouth. Example types of vocalizations can involve the user speaking, whispering, shouting, humming, whistling, singing, or making other utterances.

With active acoustic sensing, the hearable can detect a vocalization made by the user in a noisy and/or loud environment. As such, the hearable can support a voice user interface (VUI) by providing an indication of when the user is speaking. The hearable can also support multi-factor voice authentication to enhance security and provide robust protection from voice attacks. In addition to being relatively unobtrusive, some hearables can be configured to support voice activity detection using active acoustic sensing without the need for additional hardware. As such, the size, cost, and power usage of the hearable can help make voice activity detection accessible to a larger group of people and improve the user experience with hearables.

Active acoustic sensing can improve the performance of voice activity detection relative to other sensing techniques. Techniques that involve an electronic device observing the user's jaw movement using ultrasound, for instance, may not be as sensitive or as accurate compared to active acoustic sensing. This is in part because the hearable is worn by the user and active acoustic sensing can directly measure the user's vocalization based on a pressure wave that propagates to the user's ear. In contrast, observing the user's jaw movement with ultrasound may only work in limited circumstances in which the user is properly oriented relative to the electronic device and the electronic device has an unobstructed line-of-sight to the user's face to observe the jaw movement.

Operating Environment

FIG.1-1is an illustration of an example environment100in which active acoustic sensing can be implemented. In the example environment100, a hearable102is connected to a computing device104using a physical or wireless interface. The hearable102is a device that can play audible content provided by the computing device104and direct the audible content into a user106's ear108. In this example, the hearable102operates together with the computing device104. In other examples, the hearable102can operate or be implemented as a stand-alone device. Although depicted as a smartphone, the computing device104can include other types of devices, including those described with respect toFIG.6.

The hearable102is capable of performing audioplethysmography110, which is an active acoustic method of sensing that occurs at the ear108. The hearable102can perform this sensing without the use of other auxiliary sensors, such as an optical sensor or an electrical sensor. Through audioplethysmography110, the hearable102can perform voice activity detection112.

Voice activity detection112enables the hearable102to detect vocalizations made by the user106. To perform voice activity detection112, the hearable102uses audioplethysmography110to detect subtle pressure waves that propagate to the user106's ear canal114. These pressure waves modify characteristics of ultrasound signals that are transmitted and received by the hearable102and propagate through the ear canal114. With voice activity detection112, the hearable102can enhance performance of a voice user interface or provide multi-factor authentication.

To use audioplethysmography110, the user106positions the hearable102in a manner that creates at least a partial seal116around or in the ear108. Some parts of the ear108are shown inFIG.1-1, including the ear canal114and an ear drum118(or tympanic membrane). Due to the seal116, the hearable102, the ear canal114, and the ear drum118couple together to form an acoustic circuit. Audioplethysmography110involves, at least in part, measuring properties associated with this acoustic circuit. The properties of the acoustic circuit can change due to a variety of different situations or actions.

For example, considerFIG.1-2in which a change occurs in a physical structure of the ear108. Example changes to the physical structure include a change in a geometric shape of the ear canal114and/or a change in a volume of the ear canal114. This change can be caused, at least in part, by bone conduction and/or a pressure wave associated with a vocalization made by the user106.

At120, for instance, the tissue around the ear canal114and the ear drum118itself are slightly “squeezed” due to the bone conduction and/or the pressure wave. This squeeze causes a volume of the ear canal114to be slightly reduced at120. At122, however, the squeezing subsides and the volume of the ear canal114is slightly increased relative to120. The physical changes within the ear108can modulate an amplitude and/or phase of an ultrasound signal that propagates through the ear canal114, as further described with respect toFIG.15.

The techniques for audioplethysmography110can be performed while the hearable102is playing audible content to the user106and/or while the user106is actively moving or performing an activity. As such, active acoustic sensing enables the hearable102to perform voice activity detection112in a variety of different situations. One such situation is further described with respect toFIG.2.

FIG.2illustrates an example environment200in which voice activity detection112using active acoustic sensing can be implemented. The environment200represents a noisy and/or loud environment that includes a variety of audible signals. These audible signals propagate over-the-air and can make it challenging for the user106to utilize a voice user interface202of the computing device104. Example noise sources include environmental noise204, music206played by a speaker208, a vocalization210made by another person, or some combination thereof.

The noise sources can make it challenging for the voice user interface202to detect and/or recognize a vocalization212made by the user106. In some cases, the vocalization212can be a voiceprint phrase or a voice command. The voiceprint phrase can be a unique phrase that enables the user106to be identified and authenticated for voice-control access, as further described with respect toFIG.3.

To address this problem, the user106makes the vocalization212while wearing the hearable102. With voice activity detection112, the hearable102can detect the vocalization212and indicate this to the voice user interface202of the computing device104. This indication can assist the voice user interface202in detecting and/or recognizing the vocalization212in the noisy environment200. The voice activity detection112can also be used to enhance security of the computing device104, as further described with respect toFIG.3.

FIG.3illustrates other example environments in which voice activity detection112using active acoustic sensing can be implemented. In environment300-1, the user106speaks a voiceprint phrase302while wearing at least one hearable102. In some implementations, the voiceprint phrase302can enable the user106to activate certain features of the computing device104, such as providing hands-free control of the computing device104through spoken commands. During a same time interval that the computing device104detects the voiceprint phrase302, the hearable102performs voice activity detection112using audioplethysmography110and determines that the user106is speaking.

The computing device104applies a multi-factor authentication (MFA) approach to voice authentication that requires the computing device104to recognize the voiceprint phrase302and the hearable102to detect voice activity during a same time interval that the voiceprint phrase302is received by the computing device104. In environment300-1, the computing device104determines that the voice authentication is successful, as shown at304, based on the recognized voiceprint phrase302and the voice detected by the hearable102.

Unbeknownst to the user106, another person306in the environment300-1is recording308the user106's voiceprint phrase302with a recording device310. This person306is not an authorized user of the computing device104. Without the techniques for performing multi-factor voice authentication using the active acoustic sensing of the hearable102, the computing device104's security can be vulnerable to hacking with this recorded voiceprint phrase312.

In environment300-2, for instance, the person306is proximate to or in possession of the computing device104. In this situation, the user106may have accidentally walked away from the computing device104or the person306may have stolen the computing device104from the user106. The person306, however, does not have control over the hearable102.

To access the computing device104, the person306plays the recorded voiceprint phrase312through speakers of the recording device310. In this case, the hearable102does not detect the user106speaking. Consequently, the computing device104determines that voice authentication failed, as shown at314. In this manner, the computing device104denies the person306access to the features of the computing device104(e.g., the computing device104does not authenticate the person306).

In some situations, the voice authentication fails in the environment300-2because the hearable102is no longer in communication with the computing device104and is unable to provide data associated with voice activity detection112. For example, the hearable102can be too far from the computing device104(e.g., outside a communication range) or powered down. In other situations, the voice authentication fails because the hearable102does not detect speech coming from the user106during the time interval that the computing device104receives the recorded voiceprint phrase312. In this case, the hearable102can be in communication with the computing device104. As shown in environment300-2, multi-factor voice authentication can enhance security of the computing device104and provide robust protection from voice attacks. The techniques for utilizing active acoustic sensing for voice activity detection112are further described with respect toFIGS.4and5.

FIG.4illustrates example signals that can be detected by the hearable102. In the environment400, the hearable102is being worn by the user106. During operation, the hearable102can receive a variety of different signals. In one aspect, the hearable102receives at least one over-the-air (OTA) signal402(OTA signal402). The over-the-air signal402can include a voice component404and/or a noise component406. The voice component404can include a vocalization212made by the user106, such as the voiceprint phrase302ofFIG.3. The noise component406can represent any undesired audible sound that can mask or interfere with detection of the vocalization212. The noise component406can represent the environmental noise204, the music206, and/or the vocalization210ofFIG.2. In general, the over-the-air signal402includes audible frequencies.

In another aspect, the hearable102receives at least one bone-conduction signal408. The bone-conduction signal408represents sound that travels, via bone conduction, to the user106's ear108. The bone-conduction signal408also includes the voice component404. In most circumstances, the bone-conduction signal408does not include the noise component406.

To perform active acoustic sensing, the hearable102transmits and receives at least one ultrasound signal410. The ultrasound signal410propagates within the ear canal114. A vocalization212of the user106can cause a physical structure of the ear108to change. As such, the ultrasound signal410can also include the voice component404.

The ultrasound signal410may not directly include the noise component406. However, some designs of the hearable102can cause the noise component406associated with the over-the-air signal402to interfere with the detection of the voice component404within the ultrasound signal410, as further explained below.

In some example implementations, the hearable102may be designed to minimize interference between the over-the-air signal402and the ultrasound signal410. The hearable102, for instance, can utilize different microphones to receive these signals. In this case, the hearable102can directly process the ultrasound signal410to detect the vocalization212, as described with respect toFIG.15. In other example implementations, the hearable102utilizes a same microphone to receive the over-the-air signal402and the ultrasound signal410. While this may be beneficial for meeting size constraints of the hearable102, it can cause a version of the noise component406to be present within the ultrasound signal410. As the hearable102receives the over-the-air signal402, the bone-conduction signal408, and the ultrasound signal410, these signals can interact with each other and make it challenging to utilize audioplethysmography110to detect the voice component404, as further described with respect toFIG.5.

FIG.5illustrates an example operation of a microphone502of the hearable102. During an operation, the microphone502receives the over-the-air signal402, the bone-conduction signal408, and the ultrasound signal410. The microphone502can include a filter module, which can generate separate signals associated with different frequency ranges. In this example, the microphone502generates a received audible signal504and a received ultrasound signal506, which are further depicted in a graph508at the bottom ofFIG.5. These signals can be downconverted to baseband frequencies. In general, the received audible signal504and the received ultrasound signal506represent electrical signals that can be processed by other components of the hearable102.

In the graph508, the received audible signal504and the received ultrasound signal506are shown to include different frequencies. The received audible signal504can include frequencies associated with the audible frequency spectrum (e.g., frequencies between approximately 20 hertz (Hz) and 20 kilohertz (kHz)). In contrast, the received ultrasound signal506can include frequencies associated with the ultrasound frequency spectrum (e.g., frequencies between approximately 20 kHz and 2 megahertz (MHz)).

The received audible signal504represents a convolution of the over-the-air signal402and the bone-conduction signal408. As such, the received audible signal504includes the voice component404associated with the vocalization212and the noise component406. The voice component404is provided by both the over-the-air signal402and the bone-conduction signal408. The received audible signal504can be represented by Equation 1:

where YRASrepresents the received audible signal504, hBCrepresents a bone-conduction channel, S represents the vocalization212, hOTArepresents an over-the-air channel, and N represents noise (e.g., the environmental noise204, the music206, and/or the vocalization210). The hBC·S term represents a bone-conduction component512. The hOTA·S term represents the voice component404of the received audible signal504. The hOTA·N term represents the noise component406of the received audible signal504.

During reception, the received audible signal504can be modulated onto or mixed with the received ultrasound signal506. This interference between the received audible signal504and the received ultrasound signal506can be due to non-linearities in the microphone502, intermodulation distortion, harmonics, a mixing operation performed by the hearable102, or some other component and/or operation of the hearable102. The impact of the received audible signal504on the received ultrasound signal506is represented by a modulation component510. The modulation component510represents portions of the received ultrasound signal506where an amplitude, phase, and/or frequency is affected due to the interference associated with the received audible signal504. Generally speaking, the modulation component510represents a version of the received audible signal504that is shifted to the ultrasound frequencies. The modulation component510is linearly modulated into the ultrasound frequency spectrum. The received ultrasound signal506can be represented by Equation 2:

where YRUSrepresents the received ultrasound signal506, hUSrepresents an ultrasound channel, S represents the vocalization212, hMCrepresents a modulation channel, and YRASrepresents the received audible signal504. The hUS·S term represents the voice component404of the received ultrasound signal506. The hMC·YRASterm represents the modulation component510. Due to the modulation channel, the received ultrasound signal506includes a linearly modulated version of the received audible signal504. As such, the noise component406within the modulated version of the received audible signal504can make it challenging to directly detect the voice component404within the received ultrasound signal506.

Generally speaking, the voice component404is superimposed onto the ultrasound signal410and is correlated with the vocalization212. The voice component404is not correlated with the noise component406. The voice component404modulates the received ultrasound signal506in a different manner than the modulation component510due to the bone conduction and change in the physical structure within the ear108.

The voice component404of the received ultrasound signal506is frequency dependent. In other words, different ultrasound frequencies modulate the vocalization212differently. The bone-conduction component512, however, does not have frequency selectivity. In other words, the bone conduction component510is associated with a fixed channel. The techniques for voice activity detection112utilize the received audible signal504to extract the voice component404from the received ultrasound signal506, as further described with respect toFIGS.12and13.

FIG.6illustrates an example implementation of the computing device104. The computing device104is illustrated with various non-limiting example devices including a desktop computer104-1, a tablet104-2, a laptop104-3, a television104-4, a computing watch104-5, computing glasses104-6, a gaming system104-7, a microwave104-8, and a vehicle104-9. Other devices may also be used, such as an augmented and/or virtual reality headset, a home service device, a smart speaker, a smart thermostat, a baby monitor, a Wi-Fi™ router, a drone, a trackpad, a drawing pad, a netbook, an e-reader, a home automation and control system, a wall display, and another home appliance. Note that the computing device104can be wearable, non-wearable but mobile, or relatively immobile (e.g., desktops and appliances).

The computing device104includes one or more computer processors602and at least one computer-readable medium604, which includes memory media and storage media. Applications and/or an operating system (not shown) embodied as computer-readable instructions on the computer-readable medium604can be executed by the computer processor602to provide some of the functionalities described herein. The computer-readable medium604can optionally include an application606, the voice user interface202, and/or a voice authenticator608. The application606can use information provided by the hearable102to perform an action. Example actions can include displaying data associated with audioplethysmography110to the user106. For voice activity detection112, the application606can indicate whether or not the vocalization212is detected. The voice user interface202can enable the user106to control the computing device104via voice commands, as described with respect toFIG.2. The voice authenticator608can authenticate the user106and enable use of the voice user interface202upon successful authentication. The application606, the voice user interface202, and/or the voice authenticator608can utilize aspects of voice activity detection112to improve performance and/or enhance security of the computing device104.

The computing device104can also include a network interface610for communicating data over wired, wireless, or optical networks. For example, the network interface610may communicate data over a local-area-network (LAN), a wireless local-area-network (WLAN), a personal-area-network (PAN), a wire-area-network (WAN), an intranet, the Internet, a peer-to-peer network, point-to-point network, a mesh network, Bluetooth®, and the like. The computing device104may also include the display612. Although not explicitly shown, the hearable102can be integrated within the computing device104, or can connect physically or wirelessly to the computing device104. The hearable102is further described with respect toFIG.7.

FIG.7illustrates an example hearable102. The hearable102is illustrated with various non-limiting example devices, including wireless earbuds702-1, wired earbuds702-2, and headphones702-3. The earbuds702-1and702-2are a type of in-ear device that fits into the ear canal114. Each earbud702-1or702-2can represent a hearable102. Headphones702-3can rest on top of or over the ears108. The headphones702-3can represent closed-back headphones, open-back headphones, on-ear headphones, or over-ear headphones. Each headphone702-2includes two hearables102, which are physically packaged together. In general, there is one hearable102for each ear108.

The hearable102includes a communication interface704to communicate with the computing device104, though this need not be used when the hearable102is integrated within the computing device104. The communication interface704can be a wired interface or a wireless interface, in which audio content is passed from the computing device104to the hearable102. The hearable102can also use the communication interface704to pass data associated with audioplethysmography110to the computing device104. In general, the data provided by the communication interface704is in a format usable by the application606, the voice user interface202, and/or the voice authenticator608.

The communication interface704also enables the hearable102to communicate with another hearable102. During bistatic sensing, for instance, the hearable102can use the communication interface704to coordinate with the other hearable102to support two-ear audioplethysmography110, as further described with respect toFIG.8. In particular, the transmitting hearable102can communicate timing and waveform information to the receiving hearable102to enable the receiving hearable102to appropriately demodulate a received ultrasound signal506.

The hearable102includes at least one transducer706that can convert electrical signals into sound waves. The transducer706can also detect and convert sound waves into electrical signals. These sound waves may include ultrasonic frequencies and/or audible frequencies, either of which may be used for audioplethysmography110. In particular, a frequency spectrum (e.g., range of frequencies) that the transducer706uses to generate an acoustic signal can include frequencies from a low-end of the audible range to a high-end of the ultrasonic range, e.g., between 20 hertz (Hz) to 2 megahertz (MHz). Other example frequency spectrums for audioplethysmography110can encompass frequencies between 20 Hz and 20 kilohertz (kHz), between 20 kHz and 2 MHz, between 20 and 60 kHz, or between 30 and 40 kHz.

In an example implementation, the transducer706has a monostatic topology. With this topology, the transducer706can convert the electrical signals into sound waves and convert sound waves into electrical signals (e.g., can transmit or receive acoustic signals). Example monostatic transducers may include piezoelectric transducers, capacitive transducers, and micro-machined ultrasonic transducers (MUTs) that use microelectromechanical systems (MEMS) technology.

Alternatively, the transducer706can be implemented with a bistatic topology, which includes multiple transducers that are physically separate. In this case, a first transducer converts the electrical signal into sound waves (e.g., transmits acoustic signals), and a second transducer converts sound waves into an electrical signal (e.g., receives the acoustic signals). An example bistatic topology can be implemented using at least one speaker708and at least one microphone710. The speaker708and the microphone710can be dedicated for audioplethysmography110or can be used for both audioplethysmography110and other functions of the computing device104(e.g., presenting audible content to the user106, capturing the user106's voice for a phone call, or for voice control). The microphone710can represent the microphone502ofFIG.5.

In general, the speaker708and the microphone710are directed towards the ear canal114(e.g., oriented towards the ear canal114). Accordingly, the speaker708can direct ultrasound signals towards the ear canal114, and the microphone710is responsive to receiving ultrasound signals from the direction associated with the ear canal114. In some cases, the hearable102includes another microphone710that is directed away from the ear canal114towards an external environment (e.g., oriented away from the ear canal114). This other microphone can be used to receive the over-the-air signal402.

The hearable102includes at least one analog circuit712, which includes circuitry and logic for conditioning electrical signals in an analog domain. The analog circuit712can include analog-to-digital converters, digital-to-analog converters, amplifiers, filters, mixers, and switches for generating and modifying electrical signals. In some implementations, the analog circuit712includes other hardware circuitry associated with the speaker708or microphone710.

The hearable102also includes at least one system processor714and at least one system medium716(e.g., one or more computer-readable storage media). In the depicted configuration, the system medium716includes a pre-processing module718and a measurement module720. The system medium716also optionally includes a calibration module722. The pre-processing module718, the measurement module720, and the calibration module722can be implemented using hardware, software, firmware, or a combination thereof. In this example, the system processor714implements the pre-processing module718, the measurement module720, and the calibration module722. In an alternative example, the computer processor602of the computing device104can implement at least a portion of the pre-processing module718, the measurement module720, and/or the calibration module722. In this case, the hearable102can communicate digital samples of the acoustic signals to the computing device104using the communication interface704.

Operations of the pre-processing module718, the measurement module720, and the calibration module722are further described with respect toFIGS.9to13. Aspects of voice activity detection112using active acoustic sensing can be performed, at least partially, by the measurement module720, as further described with respect toFIGS.12and13.

Some hearables102include an active-noise-cancellation circuit724, which enables the hearables102to reduce background or environmental noise. In this case, the microphone710used for audioplethysmography110can be implemented using a feedback microphone of the active-noise-cancellation circuit724. During active noise cancellation, the feedback microphone provides feedback information regarding the performance of the active noise cancellation. During audioplethysmography110, the feedback microphone receives an ultrasound signal, which is provided to the pre-processing module718. In some situations, active noise cancellation and audioplethysmography110are performed simultaneously using the feedback microphone. In this case, the ultrasound signal received by the feedback microphone can be provided to the pre-processing module718and the feedback signal for active noise cancellation can be provided to the active-noise-cancellation circuit724. Other implementations are also possible in which the microphone710is implemented using a feedforward microphone of the active-noise-cancellation circuit724.

Although not explicitly shown inFIG.7, the system medium716can also include a voice user interface202and/or a voice authenticator608. In this case, the voice user interface202enables the user106to use voice controls to control an operation of the hearable102. The voice authenticator608can authenticate the user106and enable the voice user interface202for the hearable102. Different types of audioplethysmography110are further described with respect toFIG.8.

Active Acoustic Sensing

FIG.8illustrates example operations of two hearables102-1and102-2. In a first example operation, the hearables102-1and102-2perform single-ear audioplethysmography110. This means that the hearables102-1and102-2independently perform audioplethysmography110on different ears108of the user106. In this case, the first hearable102-1is proximate to the user106's right ear108, and the second hearable102-2is proximate to the user106's left ear108. Each hearable102-1and102-2includes a speaker708and a microphone710. The hearables102-1and102-2can operate in a monostatic manner during the same time period or during different time periods. In other words, each hearable102-1and102-2can independently transmit and receive ultrasound signals.

For example, the first hearable102-1uses the speaker708to transmit a first ultrasound transmit802-1, which propagates within at least a portion of the user106's right ear canal114. The first hearable102-1uses the microphone710to receive a first ultrasound receive signal804-1. The first ultrasound receive signal804-1represents a version of the first ultrasound transmit signal802-1that is modified, at least in part, by the acoustic circuit associated with the right ear canal114. This modification can change an amplitude, phase, and/or frequency of the first ultrasound receive signal804-1relative to the first ultrasound transmit signal802-1.

Similarly, the second hearable102-2uses the speaker708to transmit a second ultrasound transmit signal802-2, which propagates within at least a portion of the user106's left ear canal114. The second hearable102-2uses the microphone710to receive a second ultrasound receive signal804-2. The second ultrasound receive signal804-2represents a version of the second ultrasound transmit signal802-2that is modified by the acoustic circuit associated with the left ear canal114. This modification can change an amplitude, phase, and/or frequency of the second ultrasound receive signal804-2relative to the second ultrasound transmit signal802-2.

The techniques of single-ear audioplethysmography110can be particularly beneficial as it enables the computing device104to compile information from both hearables102-1and102-2, which can further improve measurement confidence. For some aspects of audioplethysmography110, it can be beneficial to analyze the acoustic channel between two ears108, as further described below.

In a second example operation, the two hearables102-1and102-2perform two-ear audioplethysmography110. This means that the hearables102-1and102-2jointly perform audioplethysmography110across two ears108of the user106. In this case, at least one of the hearables102(e.g., the first hearable102-1) includes the speaker708, and at least one of the other hearables102(e.g., the second hearable102-2) includes the microphone710. The hearables102-1and102-2operate together in a bistatic manner during the same time period.

During operation, the first hearable102-1transmits a third ultrasound transmit802-3using the speaker708. The third ultrasound transmit signal802-3propagates through the user106's right ear canal114. The third ultrasound transmit signal802-3also propagates through an acoustic channel that exists between the right and left ears108. In the left ear108, the third ultrasound transmit signal802-3propagates through the user106's left ear canal114and is represented as a third ultrasound receive signal804-3. The second hearable102-2receives the third ultrasound receive signal804-3using the microphone710. The third ultrasound receive signal804-3represents a version of the third ultrasound transmit signal802-3that is modified by the acoustic circuit associated with the right ear canal114, modified by the acoustic channel associated with the user106's face, and modified by the acoustic circuit associated with the left ear canal114. This modification can change an amplitude, phase, and/or frequency of the third ultrasound receive signal804-3relative to the third ultrasound transmit signal802-3. In some cases, the hearable102-2measures the time-of-flight (ToF) associated with the propagation from the first hearable102-1to the second hearable102-2. Sometimes a combination of single-ear and two-ear audioplethysmography110are applied to further improve measurement confidence.

The ultrasound transmit signals802ofFIG.8can represent a variety of different types of signals as described above with respect toFIG.7. In example implementations, the ultrasound transmit signal802can be the ultrasound signal410ofFIGS.4and5. Also, the ultrasound transmit signal802can be a continuous-wave signal (e.g., a sinusoidal signal) or a pulsed signal. Some ultrasound transmit signals802can have a particular tone (or frequency). Other ultrasound transmit signals802can have multiple tones (or multiple frequencies). A variety of modulations can be applied to generate the ultrasound transmit signal802. Example modulations include linear frequency modulations, triangular frequency modulations, stepped frequency modulations, phase modulations, or amplitude modulations. The ultrasound transmit signal802can be transmitted as part of a calibration procedure or a measurement procedure, as further described as part ofFIG.9.

FIG.9illustrates an example implementation of the hearable102for performing voice activity detection112. In the depicted configuration, the hearable102includes the speaker708, the microphone710, the analog circuit712, the pre-processing module718, the measurement module720, and the calibration module722. Other implementations of the hearable102, however, are also possible in which the hearable102does not include the calibration module722to reduce processing power requirements. In this case, the pre-processing module718can perform aspects of frequency selection as further described with respect toFIG.12to improve the signal-to-noise ratio for audioplethysmography110.

Outputs of the speaker708and the microphone710are coupled to inputs of the analog circuit712. The pre-processing module718has inputs that are coupled to outputs of the analog circuit712. The pre-processing module718also has outputs that are coupled to inputs of the measurement module720and the calibration module722. The measurement module720has another input that is coupled to the microphone710. The calibration module722has an output that is coupled to the speaker708.

Consider an example operation of the hearable102in accordance with single-ear audioplethysmography110. In the case that the hearable102includes the calibration module722, the hearable102can perform a calibration process prior to performing a measurement process. The calibration process and the measurement process are further described with respect toFIG.10.

During both the calibration process and the measurement process, the speaker708transmits the ultrasound transmit signal802and the microphone710receives the ultrasound receive signal804. During the calibration process, the ultrasound transmit signal802and the ultrasound receive signal804can have tones902-1to902-M, where M represents a positive integer. During the measurement process, the ultrasound transmit signal802and the ultrasound receive signal804can have selected tones904-1to904-N, where N represents a positive integer that is less than or equal to M. The selected tones904-1to904-N can represent a subset (sometimes a proper subset) of the tones902-1to902-M. The microphone710can also receive the over-the-air signal402and the bone conduction signal408during the measurement process.

The analog circuit712performs analog-to-digital conversion to generate a digital transmit signal906and a digital receive signal908based on the ultrasound transmit signal802and the received ultrasound signal506, respectively. The pre-processing module718performs frequency downconversion and demodulation to generate at least one pre-processed signal910based on the digital transmit signal906and the digital receive signal908. The pre-processing module718can also apply filtering to generate the pre-processed signal910.

As part of the calibration procedure, the calibration module722processes the pre-processed signal910to determine the selected tones904-1to904-N. The selected tones904-1to904-N can improve performance of audioplethysmography110during the measurement procedure. The calibration module722communicates the selected tones904-1to904-N to the speaker708using a control signal. The speaker708accepts the control signal that identifies the selected tones904-1to904-N and can transmit a subsequent ultrasound transmit signal802for the measurement procedure using the selected tones904-1to904-N.

As part of the measurement procedure, the measurement module720can perform aspects of voice activity detection112using the pre-processed signal910to generate a voice activity indicator912(VA indicator912). In cases in which the environment is noisy, the measurement module720can also utilize the received audible signal504provided by the microphone710to further process the pre-processed signal910for voice activity detection112. The voice activity indicator912can be communicated to the application606, the voice user interface202, and/or the voice authenticator608. Additionally or alternatively, the voice activity indicator912can include a control signal for controlling operation of the hearable102and/or the computing device104. The calibration procedure and the measurement procedure are further described with respect toFIG.10.

FIG.10illustrates an example flow diagram1000for operating a hearable102. InFIG.10, the hearable102can optionally perform a calibration procedure at1002using the calibration module722. The calibration procedure can determine appropriate characteristics (e.g., waveform or signal characteristics) of ultrasound transmit signals802to improve audioplethysmography110(e.g., to enhance the performance of voice activity detection112). The calibration procedure enables audioplethysmography110to take into account the wear of the hearable102(e.g., the position of the hearable102relative to the ear canal114) and the physical structure of the ear canal114to determine a transmission frequency that can increase sensitivity. With the calibration procedure, the hearable102can dynamically adjust the transmission frequency (e.g., one or more carrier frequencies) each time the seal116is formed (e.g., based on the wear of the hearable102) and based on the unique physical structure of the ear108. Through this calibration procedure, the hearables102on different ears108may operate with one or more different ultrasound frequencies. Steps of the calibration procedure are further described below.

In some circumstances, the hearable102can perform on-head detection (or in-ear detection) by detecting the presence of the seal116and initiating the calibration procedure based on a determination that on-head detection is “true.” In other circumstances, the hearable102can initiate the calibration procedure based on a specified schedule or a timer, which can be controlled by the user106via the computing device104.

At1004, the hearable102executes the calibration procedure by transmitting and receiving a first ultrasound signal. The first ultrasound signal propagates within at least a portion of the ear canal114of the user106and has multiple tones902-1to902-M (or multiple carrier frequencies). The multiple tones902-1to902-M are transmitted in parallel or in series over a given time interval. The first ultrasound transmit signal802can have a particular bandwidth on the order of several kilohertz. For example, the ultrasound transmit signal802can have a bandwidth of approximately 4, 5, 6, 8, 10, 16, or 20 kHz. In example implementations, the first ultrasound transmit signal802is transmitted over multiple seconds, such as 2, 3, 4, 6, or more seconds. A duration of each tone902can be evenly divided over a total duration of the first ultrasound transmit signal802.

In an example implementation, the ultrasound transmit signal902has seven tones902(e.g., M equals 7). In some cases, the tones902are evenly distributed across an interval. For example, the tones902can be in 1 kHz increments between 32 kHz and 38 kHz (e.g., at approximately 32, 33, 34, 35, 36, 37, and 38 kHz). The term “approximately” means that the tones902can be within 5% of a given value or less (e.g., within 3%, 2%, or 1% of the given value).

An amplitude of the ultrasound transmit signal802can be approximately the same across the tones902-1to902-M. In this manner, power is evenly distributed across each tone902. The quantity of tones902(e.g., M) can be determined based on an output power of the speaker708. Increasing the quantity of tones902can increase a likelihood that the hearable102can support voice activity detection112across various conditions including user wear and a physical structure of the user106's ear canal114. However, an amplitude of the ultrasound transmit signal802can be limited across these tones902based on the output power of the speaker708. ThUS, the quantity of tones902can be optimized based on an amount of output power that is available for audioplethysmography110.

At1006, the calibration procedure selects one or more tones904-1to904-N to be used for a measurement procedure based on one or more modified characteristics of the ultrasound receive signal804. The process for selecting the tones904is further described with respect toFIG.11. In general, the calibration procedure determines that the selected tones904improve a signal-to-noise ratio for audioplethysmography110(or more specifically for voice activity detection112).

At1008, the hearable102performs a measurement procedure using the measurement module720. In accordance with the measurement procedure, the hearable102transmits a second ultrasound transmit signal802that propagates within at least the portion of the ear canal114of the user106. If the calibration procedure was performed, the second ultrasound transmit signal802can have the selected tones904-1to904-M that were determined by the calibration procedure. The selected tones904can be transmitted in parallel or in series over a given time interval.

An amplitude of the second ultrasound transmit signal802can be approximately the same across the selected tones904-1to904-N. In this manner, power is evenly distributed across each selected tone. The amplitude of the second ultrasound transmit signal802can be higher than the amplitude of the first ultrasound transmit signal802because the available output power is distributed across fewer tones. Additionally or alternatively, a duration of each of the selected tones904of the second ultrasound transmit signal802can be longer than the duration of the tones902of the first ultrasound transmit signal802. The higher amplitude and/or the longer duration can further improve the signal-to-noise ratio performance of the hearable102for audioplethysmography110. By using a few selected tones904that were determined to improve signal-to-noise ratio performance, the measurement procedure can achieve a higher accuracy for voice activity detection112.

At1012, the hearable102performs voice activity detection112using the second ultrasound signal (e.g., the second ultrasound receive signal804). An example process for performing voice activity detection112is further described with respect toFIG.13. The calibration module722is further described with respect toFIG.11.

FIG.11illustrates an example scheme implemented by the calibration module722. In the depicted configuration, the calibration module722implements a frequency selector, which selects one or more tones904for the measurement procedure. In the example implementation, the calibration module722includes at least one amplitude detector1102, at least one phase detector1104, at least one quality detector1106, and at least one comparator1108. The operations of these components are further described below.

During the calibration procedure, the calibration module722accepts the pre-processed signal910from the pre-processing module718, as previously described with respect toFIG.9. The pre-processed signal910can include amplitude and/or phase information associated with the multiple tones902-1to902-M, which were used to transmit the first ultrasound signal described at1002inFIG.10.

In this example, the calibration module722extracts an amplitude1110of the pre-processed signal910using the amplitude detector1102and extracts a phase1112of the pre-processed signal910using the phase detector1104. Alternatively, if in-phase and quadrature components of the pre-processed signal910are received separately, the amplitude detector1102and the phase detector1104can respectively measure the amplitude1110and phase1112based on the in-phase and quadrature components.

The quality detector1106measures quality metrics1114-1to1114-2M for each of the tones902-1to902-M and for each of the characteristics (e.g., amplitude1110and phase1112). In general, the quality metrics1114can represent a variety of different metrics, including peak-to-average ratios and/or signal-to-noise ratios. The peak-to-average ratio represents a peak intensity within a frequency range of interest divided by an average intensity within this frequency range. A higher quality metric1114indicates a higher-quality signal, or more generally, better performance for audioplethysmography110.

In one aspect, the comparator1108can evaluate the quality metrics1114-1to1114-2M with respect to a threshold1116. The threshold1116can be set, for example, to a particular value. In other cases, the calibration module722can dynamically determine the threshold1116and update it over time based on the observed quality metrics1114-1of1114-2M. In an example implementation, the comparator1108determines the selected tones904-1to904-N for a subsequent measurement procedure based on the frequencies associated with the quality metrics1114-1to1114-M that are greater than or equal to the threshold1116.

Additionally or alternatively, the comparator1108can evaluate the quality metrics1114-1to1114-2M with respect to each other. In an example implementation, the comparator1108determines one of the selected tones904based on a frequency with the highest quality metric1114across the amplitude1110. Also, the comparator1108can determine one of the selected tones904based on a frequency with the highest quality metric1114across the phase1112. In other implementations, the comparator1108can determine a single selected tone904based on a frequency having the highest quality metric1114associated with either the amplitude1110or the phase1112.

In general, the calibration module722enables the selected tones904-1to904-N to be dynamically adjusted prior to the measurement procedure based on a current environment, which can account for a wear of the hearable102(e.g., a current insertion depth and/or rotation), a physical structure of the user106's ear canal114, and a response characteristic of the hearable102(e.g., speaker, microphone, and/or housing). In this manner, the calibration module722can improve the signal-to-noise ratio performance of the hearable102for the measurement procedure. The calibration module722can also determine which tones904generate ultrasound receive signals804with desired characteristics for voice activity detection112. In general, the calibration procedure can be performed whether or not the user106is speaking.

InFIGS.9to11, the calibration procedure and the measurement procedure are described as individual procedures that occur at different time intervals. In particular, the calibration procedure occurs before the measurement procedure. This enables the ultrasound transmit signal802for the measurement procedure to be transmitted with fewer tones than the ultrasound transmit signal802used for the calibration procedure, which can increase signal-to-noise ratio performance for audioplethysmography110. In some implementations, however, the hearable102can have sufficient output power to perform the measurement procedure with the multiple tones902-1to902-M using a single ultrasound transmit signal802. In this case, aspects of the calibration module can be integrated within the pre-processing module718as a frequency selector, which is further described with respect toFIG.12. This frequency selector can effectively pass the selected tones904-1to904-N for further processing. Aspects of the measurement procedure are further described with respect toFIG.12.

Voice Activity Detection

FIG.12illustrates an example implementation of the pre-processing module718for performing aspects of voice activity detection112using active acoustic sensing. In the depicted configuration, the hearable102includes the pre-processing module718, which is coupled to the measurement module720and the calibration module722. The measurement module720is also coupled to the microphone710(not shown).

The pre-processing module718includes at least one in-phase and quadrature mixer1202(I/Q mixer1202) and at least one filter1204. The in-phase and quadrature mixer1202performs frequency down-conversion. In an example implementation, the in-phase and quadrature mixer1202includes at least two mixers, at least one phase shifter, and at least one combiner (e.g., a summation circuit). The filter1204attenuates intermodulation products that are generated by the in-phase and quadrature mixer1202. In an example implementation, the filter1204is implemented using a low-pass filter.

The pre-processing module718can optionally include at least one frequency selector1206. The frequency selector1206can identify and select one or more tones904(or carrier frequencies) that provide a high-quality signal for later processing. The frequency selector1206can further pass the selected tones904to other processing modules and filter (or attenuate) other tones that are not selected. The frequency selector1206can be implemented in a similar manner as the calibration module722ofFIG.11. For example, the frequency selector1206, can include the amplitude detector1102, the phase detector1104, the quality detector1106, and the comparator1108.

During an operation, the in-phase and quadrature mixer1202uses the phase shifter and the two mixers to generate in-phase and quadrature components associated with the digital receive signal908. In particular, the in-phase and quadrature mixer1202mixes the digital receive signal908with a first version of the digital transmit signal906that has a zero-degree phase shift to generate the in-phase component. Additionally, the in-phase and quadrature mixer1202mixes the digital receive signal908with a second version of the digital transmit signal906that has a 180-degree phase shift to generate the quadrature signal. This mixing operation downconverts the digital receive signal908from acoustic frequencies to baseband frequencies. Using the combiner, the in-phase and quadrature mixer1202combines the in-phase and quadrature components of the digital receive signal908to generate a down-converted signal1208. Use of the in-phase and quadrature mixer1202can further improve the signal-to-noise ratio of the down-converted signal1208compared to other mixing techniques.

In this example, the down-converted signal1208represents a combination of the in-phase and quadrature components of the mixed-down digital receive signal908. In alternative implementations, the in-phase and quadrature mixer1202doesn't include the combiner and passes the in-phase and quadrature components separately to the filter1204. In this manner, the in-phase and quadrature components individually propagate through the filter1204.

The filter1204generates a filtered signal1210based on the down-converted signal1208. In particular, the filter1204filters the down-converted signal1208to attenuate spurious or undesired frequencies (e.g., intermodulation products), some of which can be associated with an operation of the in-phase and quadrature mixer1202. In this example, the filtered signal1210represents a combination of the in-phase and quadrature components of the down-converted signal1208. Alternatively, the filtered signal1210can represent separate or distinct in-phase and quadrature components, which are individually passed to the frequency selector1206, the calibration module722, or the measurement module720.

During the measurement procedure, the pre-processing module718can optionally apply the frequency selector1206. The frequency selector1206passes tones that meet a quality threshold level of performance for audioplethysmography110. For example, the frequency selector1206passes tones904having an amplitude1110and/or phase1112with a quality metric1114that is greater than or equal to a threshold1116. The resulting signal outputted by the frequency selector1206is represented by signal1212. In some implementations, this signal1212is passed to the measurement module720as the pre-processed signal910. In other implementations in which the frequency selector1206is not implemented, the filtered signal1210can be passed to the measurement module720and/or the calibration module722as the pre-processed signal910.

In general, the measurement module720can generate the voice activity indicator912based on the pre-processed signal910. In example implementations, the measurement module720can be implemented using a machine-learned model or another model that performs signal and/or data processing. In this case, the measurement module720can analyze the changes in the amplitude1110and/or phase1112of the pre-processed signal910to determine whether or not the voice component404is present. This processing technique can be utilized in implementations of the hearable102that have minimal (if any) interference between the over-the-air signal402and the ultrasound signal410or in situations in which voice activity detection112is performed in a relatively quiet environment.

To handle noisy environments, however, the measurement module720can generate the voice activity indicator912based on the pre-processed signal910and the received audible signal504. More specifically, the measurement module720can utilize the received audible signal504as a reference to filter (e.g., attenuate) the modulation component510within the pre-processed signal910and detect the voice component404. An example implementation of the measurement module720, which can perform voice activity detection112in noisy environments, is further described with respect toFIG.13.

FIG.13illustrates an example implementation of the measurement module720for performing voice activity detection112using active acoustic sensing. In the depicted configuration, the measurement module720includes at least one filter module1302, at least one vocalization enhancer1304, and at least one vocalization detector1306. The filter module1302can be implemented using at least one adaptive filter1308or at least one blind-source separator1310. The adaptive filter1308performs adaptive filtering using the received audible signal504as a reference to filter the modulation component510from the pre-processed signal910. The blind-source separator1310performs blind-source separation (BSS) using the received audible signal504as a reference to filter the modulation component510from the pre-processed signal910. Explained another way, adaptive filtering and/or blind-source separation utilize the received audible signal504to separate the voice component404from the modulation component510(or from the modulated noise component406) within the pre-processed signal910. In general, the adaptive filter1308and the blind-source separator1310can utilize the received audible signal504to significantly attenuate the modulation component510(or the modulated noise component406) within the pre-processed signal910. To perform adaptive filtering or blind-source separation, the pre-processed signal910represents a primary reference (e.g., the primary channel or the signal to be filtered) and the received audible signal504represents a secondary or a noise reference (e.g., the reference channel).

The vocalization enhancer1304enhances (e.g., amplifies relative to a noise level) the voice component404within an output signal provided by the filter module1302. In this way, the vocalization enhancer1304can increase sensitivity for performing voice activity detection112. In an example implementation, the vocalization enhancer1304is implemented using a wiener filter1312.

The vocalization detector1306can detect the voice component404within an output signal provided by the vocalization enhancer1304. The vocalization detector1306generates the voice activity indicator912to indicate whether or not the voice component404is detected. In some examples, the vocalization detector1306can perform a signal-to-noise ratio detection process, which determines whether or not an amplitude of an input signal exceeds a detection threshold. If the amplitude exceeds the detection threshold, the vocalization detector1306generates the voice activity indicator912to indicate that the vocalization212is detected. Otherwise, if the amplitude does not exceed the detection threshold, the vocalization detector1306generates the voice activity indicator912to indicate that a vocalization212is not detected. Voice activity detection112can be utilized in a variety of different ways to control an operation of the hearable102and/or the computing device104, as further described with respect toFIG.14.

FIG.14illustrates an example scheme1400for applying voice activity detection112. In this example, voice activity detection112can be used to enhance performance of a voice control interface202or enhance security for voice authentication. At1402, the hearable102receives the ultrasound receive signal804. At1404, the hearable102detects or does not detect a vocalization212made by the user106based on the ultrasound receive signal804. In particular, the hearable102uses the measurement module720to analyze the pre-processed signal910and generate the voice activity indicator912, which indicates whether or not the vocalization212is detected.

If the hearable102does not detect a vocalization (e.g., determines that a vocalization did not occur or determines the absence of a vocalization), the hearable102disables voice authentication at1406and/or disables voice control at1408. More specifically, the hearable102can generate a voice activity indicator912that indicates vocalization is not detected and communicate this information to the voice authenticator608. The voice activity indicator912can also include a time period associated with the ultrasound receive signal804. This information causes the voice authenticator608to disable voice authentication or cause voice authentication to fail for at least the time period associated with the ultrasound receive signal804. In this manner, even if the voice authenticator608recognizes the voiceprint phrase302during the same time period that is associated with the ultrasound receive signal804, voice authentication fails. Additionally or alternatively, the voice activity indicator912causes the voice user interface202to disable voice control1408or otherwise ignore a recognized command that is received during the same time period associated with the ultrasound receive signal804.

If the hearable102detects a vocalization at1404, the hearable102can enable voice authentication at1410and/or enable voice control at1412. More specifically, the hearable102can generate a voice activity indicator912that indicates a vocalization212is detected and communicate this information to the voice authenticator608. With this information, the voice authenticator608can enable voice authentication. As such, voice authentication can succeed if other aspects of the multi-factor voice authentication are met. For example, voice authentication can succeed if the computing device104also recognizes the voiceprint phrase302during the same time period that is associated with the ultrasound receive signal804.

Additionally or alternatively, the voice user interface202can enable voice control based on the voice activity indicator912indicating that the vocalization212is detected. As such, the user106can utilize the voice user interface202to control the computing device104and/or the hearable102provided that the voice command is recognized by the voice user interface202. In some cases, the voice activity indicator912provides a timing reference, which can make it easier for the voice user interface202to detect a voice command. With voice activity detection112using active acoustic sensing, the hearable102can enhance voice control and/or enhance security of the computing device104against voice attacks. Example pre-processed signals910that capture the user106's vocalizations are further described with respect toFIG.15.

FIG.15illustrates the impact of a vocalization212on an ultrasound receive signal804. More specifically,FIG.15depicts example amplitudes1110and phases1112of pre-processed signals910generated by different hearables102-1and102-2. As shown below, the pressure wave caused by a vocalization212can significantly impact the amplitude1110and/or the phase1112of the pre-processed signals910. In some instances, the change in the amplitude1110and/or the phase1112can be relative to a previous state or relative to a previous trend in the amplitude1110and/or the phase1112. The previous state can refer to values of the amplitude1110and/or the phase1112during which the user106does not vocalize.

In general, the term “significantly” can mean that the values of the amplitude1110and/or the phase1112can change by 20% or more relative to a previous value (e.g., relative to an average of a set of previous values). Additionally or alternatively, a slope of the amplitude1110and/or the phase1112can vary significantly. Sometimes the slope of the amplitude1110and/or the phase1112can change signs (e.g., from a positive slope to a negative slope, or vice versa). A magnitude of the slope of the amplitude1110and/or the phase1112can sometimes change by approximately 10% or more.

In some implementations, the measurement module720can detect and recognize the vocalization212based on the amplitude1110of the pre-processed signal910provided by the hearable102-1, the phase1112of the pre-processed signal910provided by the hearable102-1, the amplitude1110of the pre-processed signal910provided by the hearable102-2, the phase1112of the pre-processed signal910provided by the hearable102-2, or some combination thereof. Generally speaking, processing a larger quantity of signals and/or tones904that are sensitive to the pressure wave caused by the vocalization212provides more information to the measurement module720. This can make it easier for the measurement module720to accurately detect the vocalization212.

Graphs1500-1and1500-2depict amplitudes1110and phases1112of pre-processed signals910that are respectively generated by the hearables102-1and102-2. Time is depicted along the horizontal axes of the graphs1500-1and1500-2.

During the time interval indicated at1502, the user106makes a vocalization212(e.g., speaks, hums, whistles, sings, or makes other utterances). This causes the amplitude1110and/or the phase1112of the ultrasound receive signal804to change significantly relative to a previous state. With audioplethysmography110, the measurement module720can detect and recognize the vocalization212based on the change in the amplitude1110and/or phase1112of the pre-processed signals910provided by the hearable102-1and/or the hearable102-2.

If the microphone710that detects the ultrasound receive signal804is also used to receive the over-the-air signal402, the measurement module720can perform additional processing to separate the vocalization212from the modulation component510. In particular, the measurement module720can utilize the received audible signal504as a reference signal to perform adaptive filtering and/or blind-source separation to detect the vocalization212for voice activity detection112.

Aspects of voice activity detection112can be performed using one hearable102(e.g., the hearable102-1or102-2) or multiple hearables102(e.g., the hearables102-1and102-2). With multiple hearables102performing voice activity detection112, the computing device104can have higher confidence that the user106's vocalization is detected. In general, the hearable102can detect a vocalization by analyzing changes in the amplitude1110of the ultrasound receive signal804, changes in the phase1112of the ultrasound receive signal804, or changes in both the amplitude1110and phase1112of the ultrasound receive signal804.

Example Methods

FIGS.16and17depict example methods1600and1700for implementing aspects of voice activity detection112using active acoustic sensing. Methods1600and1700are shown as sets of operations (or acts) performed but not necessarily limited to the order or combinations in which the operations are shown herein. Further, any of one or more of the operations may be repeated, combined, reorganized, or linked to provide a wide array of additional and/or alternate methods. In portions of the following discussion, reference may be made to the environments100,200,300-1, and300-2ofFIGS.1,2, and3, and entities detailed inFIGS.6and7, reference to which is made for example only. The techniques are not limited to performance by one entity or multiple entities operating on one device.

At1602, an ultrasound transmit signal is transmitted during a first time period. The ultrasound transmit signal propagates within at least a portion of an ear canal of a user. For example, the transducer706(or speaker708) of the hearable102transmits the ultrasound transmit signal802. The ultrasound transmit signal802propagates within at least a portion of the ear canal114of the user106, as described with respect toFIGS.4and8.

At1604, an ultrasound receive signal is received. The ultrasound receive signal represents a version of the ultrasound transmit signal with one or more waveform characteristics modified based on the propagation within the ear canal and based on a vocalization made by the user during the first time period. For example, the transducer706(or the microphone710) of the hearable102receives the ultrasound receive signal804. The ultrasound receive signal804represents a version of the ultrasound transmit signal802with one or more waveform characteristics modified based on the propagation within the ear canal114and based on a vocalization212made by the user106during the first time period. The hearable102that receives the ultrasound receive signal804can be a same hearable102that transmitted the ultrasound transmit signal802(e.g., the hearable102-1or102-2inFIG.8), or another hearable102that did not transmit the ultrasound transmit signal802(e.g., the hearable102-2inFIG.8). Example waveform characteristics include amplitude, phase, and/or frequency. In some implementations, a feedback microphone of an active-noise-cancellation circuit724can receive the ultrasound receive signal804.

At1606, the vocalization is detected based on the one or more modified characteristics of the ultrasound receive signal. For example, the hearable102uses the measurement module720to analyze the one or more modified characteristics of the ultrasound receive signal804and detect the vocalization212. The hearable102can generate a voice activity indicator912, which can be used to control an operation of the hearable102and/or an operation of the computing device104. The voice activity indicator912indicates whether or not the hearable102detected the vocalization212.

In one aspect, the voice activity indicator912can enable or disable the voice control interface202. This can reduce a probability of the voice control interface202incorrectly processing a voice command provided by another person and enhance performance of the voice control interface202. In another aspect, the voice activity indicator912can enable or disable voice authentication. With voice activity detection112, the hearable102can provide multi-factor voice authentication that enhances security and provides robust protection from voice attacks.

At1702inFIG.17, active acoustic sensing is performed to detect a pressure wave that propagates within an ear canal of a user and is associated with a vocalization of the user. For example, the hearable102performs active acoustic sensing to detect a pressure wave that propagates within an ear canal114of a user106and is associated with a vocalization212of the user106. To perform active acoustic sensing, the hearable102transmits and receives an ultrasound signal410(e.g., the ultrasound transmit signal802and the ultrasound receive signal804). The received ultrasound signal410includes the voice component404, which enables audioplethysmography110to perform voice activity detection112.

At1704, voice activity detection is performed based on the active acoustic sensing. For example, the hearable102performs voice activity detection112based on the active acoustic sensing. More specifically, the hearable102analyzes the amplitude1110and/or phase1112of the ultrasound receive signal804to detect the vocalization212. In some implementations, the hearable102can utilize a received audible signal504to process the ultrasound receive signal804and attenuate a modulation component510to enhance voice activity detection112.

At1706, a signal that controls an operation of at least one of a hearable or a computing device that is coupled to the hearable is generated. For example, the measurement module720generates a voice activity indicator912, which can be used to control an operation of the hearable102and/or the computing device104.

Example Computing System

FIG.18illustrates various components of an example computing system1800that can be implemented as any type of client, server, and/or computing device as described with reference to the previousFIGS.6and7to implement aspects of active acoustic sensing using a hearable.

The computing system1800includes communication devices1802that enable wired and/or wireless communication of device data1804(e.g., received data, data that is being received, data scheduled for broadcast, or data packets of the data). The communication devices1802or the computing system1800can include one or more hearables102. The device data1804or other device content can include configuration settings of the device, media content stored on the device, and/or information associated with a user of the device. Media content stored on the computing system1800can include any type of audio, video, and/or image data. The computing system1800includes one or more data inputs1806via which any type of data, media content, and/or inputs can be received, such as human utterances, user-selectable inputs (explicit or implicit), messages, music, television media content, recorded video content, and any other type of audio, video, and/or image data received from any content and/or data source.

The computing system1800also includes communication interfaces1808, which can be implemented as any one or more of a serial and/or parallel interface, a wireless interface, any type of network interface, a modem, and as any other type of communication interface. The communication interfaces1808provide a connection and/or communication links between the computing system1800and a communication network by which other electronic, computing, and communication devices communicate data with the computing system1800.

The computing system1800includes one or more processors1810(e.g., any of microprocessors, controllers, and the like), which process various computer-executable instructions to control the operation of the computing system1800. Alternatively or in addition, the computing system1800can be implemented with any one or combination of hardware, firmware, or fixed logic circuitry that is implemented in connection with processing and control circuits which are generally identified at1812. Although not shown, the computing system1800can include a system bus or data transfer system that couples the various components within the device. A system bus can include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and/or a processor or local bus that utilizes any of a variety of bus architectures.

The computing system1800also includes a computer-readable medium1814, such as one or more memory devices that enable persistent and/or non-transitory data storage (i.e., in contrast to mere signal transmission), examples of which include random access memory (RAM), non-volatile memory (e.g., any one or more of a read-only memory (ROM), flash memory, EPROM, EEPROM, etc.), and a disk storage device. The disk storage device may be implemented as any type of magnetic or optical storage device, such as a hard disk drive, a recordable and/or rewriteable compact disc (CD), any type of a digital versatile disc (DVD), and the like. The computing system1800can also include a mass storage medium device (storage medium)1816.

The computer-readable medium1814provides data storage mechanisms to store the device data1804, as well as various device applications1818and any other types of information and/or data related to operational aspects of the computing system1800. For example, an operating system1820can be maintained as a computer application with the computer-readable medium1814and executed on the processors1810. The device applications1818may include a device manager, such as any form of a control application, software application, signal-processing and control module, code that is native to a particular device, a hardware abstraction layer for a particular device, and so on.

The device applications1818also include any system components, engines, or managers to implement audioplethysmography110for voice activity detection112. In this example, the device applications1818include the pre-processing module718, the measurement module720, and optionally the calibration module722. Although not explicitly shown, the device applications1818can also include the application606, the voice user interface202, and/or the voice authenticator608.

Throughout this disclosure, examples are described where a computing system1800(e.g., the hearable102, the computing device104, a client device, a server device, a computer, or another type of computing system) may analyze information (e.g., various audible and/or ultrasound signals) associated with a user, for example, the vocalization212mentioned with respect toFIG.2. Further to the descriptions above, a user106may be provided with controls allowing the user106to make an election as to both if and when systems, programs, and/or features described herein may enable collection of information (e.g., information about a user's social network, social actions, social activities, profession, a user's preferences, a user's current location), and if the user is sent content or communications from a server. The computing system1800can be configured to only use the information after the computing system1800receives explicit permission from the user106to use the data. For example, in situations where the hearable102analyzes signals to authenticate the user106, individual users106may be provided with an opportunity to provide input to control whether programs or features of the computing system1800can collect and make use of the data. Further, individual users106may have constant control over what programs can or cannot do with the information.

In addition, information collected may be pre-treated in one or more ways before it is transferred, stored, or otherwise used, so that personally-identifiable information is removed. For example, before the computing system1800shares data with another device, a user106's identity may be treated so that no personally identifiable information can be determined for the user106. Thus, the user106may have control over whether information is collected about the user106and the user106's device, and how such information, if collected, may be used by the computing system1800and/or a remote computing system.

CONCLUSION

Although techniques using, and apparatuses including, performing voice activity detection using active acoustic sensing have been described in language specific to features and/or methods, it is to be understood that the subject of the appended examples is not necessarily limited to the specific features or methods described. Rather, the specific features and methods are disclosed as example implementations of performing voice activity detection using active acoustic sensing.

Some examples are provided below.

Example 1: A method comprising:transmitting, during a first time period, an ultrasound transmit signal that propagates within at least a portion of an ear canal of a user;receiving, during the first time period, an ultrasound receive signal, the ultrasound receive signal representing a version of the ultrasound transmit signal with one or more characteristics modified based on the propagation within the ear canal and based on a vocalization made by the user during the first time period; anddetecting the vocalization based on the one or more modified characteristics of the ultrasound receive signal.

Example 2: The method of example 1, wherein the detecting the vocalization comprises detecting a change in an amplitude and/or a phase of the ultrasound receive signal.

Example 3: The method of example 1, further comprising:generating, during the first time period, a received audible signal comprising audible frequencies, the audible frequencies comprising the vocalization made by the user during the first time period, wherein:the receiving of the ultrasound receive signal comprises generating a received ultrasound signal, the received ultrasound signal having a modulation component associated with the received audible signal; andthe detecting the vocalization comprises attenuating the modulation component within the received ultrasound signal based on the received audible signal.

Example 4: The method of example 3, wherein the modulation component represents a version of the received audible signal that is linearly modulated onto the received ultrasound signal.

Example 5: The method of example 3 or 4, wherein the attenuating of the modulation component comprises performing adaptive filtering with the received ultrasound signal representing a primary reference and the received audible signal representing a secondary reference.

Example 6: The method of any one of examples 3 to 5, wherein the generating of the received audible signal and the generating of the received ultrasound signal comprises generating the received audible signal and the received ultrasound signal using a same microphone.

Example 7: The method of any previous example, wherein the vocalization comprises speech, humming, whistling, or singing.

Example 8: The method of any previous example, further comprising:communicating to a computing device that the vocalization is detected to enable a voice control interface and/or to enable voice authentication.

Example 9: The method of any previous example, further comprising:transmitting, during a second time period, another ultrasound transmit signal that propagates within at least a portion of the ear canal of the user;receiving, during the second time period, another ultrasound receive signal, the other ultrasound receive signal representing a version of the other ultrasound transmit signal; anddetermining that the user did not make another vocalization during the second time period based on the other ultrasound receive signal.

Example 10: The method of example 9, further comprising:communicating to a computing device that the other vocalization is not detected to disable a voice control interface and/or to disable voice authentication.

Example 11: The method of any previous example, wherein the transmitting of the ultrasound transmit signal comprising transmitting the ultrasound transmit signal with at least two tones.

Example 12: The method of example 11, further comprising:transmitting, prior to the transmitting of the ultrasound transmit signal, another ultrasound transmit signal that propagates within at least the portion of the ear canal of the user, the other ultrasound transmit signal having multiple tones, the multiple tones including the at least two tones and at least one other tone;receiving, prior to the transmitting of the ultrasound transmit signal, another ultrasound receive signal, the other ultrasound receive signal representing a version of the other ultrasound transmit signal with one or more characteristics modified due to the propagation within the ear canal; andselecting the at least two tones from the multiple tones based on the other ultrasound receive signal.

Example 13: The method of example 12, wherein transmitting the ultrasound transmit signal comprises at least one of the following:transmitting the ultrasound transmit signal such that the ultrasound transmit signal has a higher amplitude at the at least two tones compared to an amplitude of the other ultrasound transmit signal at the multiple tones; ortransmitting the ultrasound signal such that a duration of ultrasound transmit signal at each of the at least two tones is longer compared to a duration of the other ultrasound transmit signal at each of the multiple tones.

Example 14: A computer-readable storage medium comprising instructions that, responsive to execution by a processor, cause a hearable to perform any one of the methods of examples 1 to 13.

Example 15: A device comprising:at least one transducer; andat least one processor, the device configured to perform, using the at least one transducer and the at least one processor, any one of the methods of examples 1 to 13.

Example 16: The device of example 17, further comprising:a speaker; andan active-noise-cancellation circuit comprising a feedback microphone,wherein the at least one transducer comprises the speaker and the feedback microphone.

Example 17: The device of example 17, wherein:the at least one transducer comprises a speaker and a microphone;the speaker is configured to be positioned proximate to a first ear of a user; andthe microphone is configured to be positioned proximate to a second ear of the user.

Example 18: The device of any one of examples 17 to 19, wherein the device comprises:at least one earbud; orheadphones.

Example 19: A method comprising:transmitting, during a first time period, an ultrasound transmit signal that propagates within at least a portion of the ear canal of the user;receiving, during the first time period, an ultrasound receive signal, the ultrasound receive signal representing a version of the ultrasound transmit signal with one or more characteristics modified based on the propagation within the ear canal and based on a vocalization made by the user during the first time period;detecting the vocalization based on the ultrasound receive signal; andgenerating a control signal that controls an operation of a device based on the detected vocalization.

Example 20: The method of example 19, wherein:the device comprises a hearable;the transmitting of the ultrasound transmit signal comprises transmitting the ultrasound transmit signal using the hearable; andthe receiving of the ultrasound receive signal comprises receiving the ultrasound receive signal using the hearable.

Example 21: The method of example 19, wherein:the device comprises a computing device that is coupled to a hearable;the transmitting of the ultrasound transmit signal comprises transmitting the ultrasound transmit signal using the hearable;the receiving of the ultrasound receive signal comprises receiving the ultrasound receive signal using the hearable.

Example 22: The method of example 20 or 21, further comprising:enabling voice control and/or voice authentication based on the control signal.

Example 23: The method of any one of examples 19 to 22, wherein the detecting the vocalization comprises detecting a change in an amplitude and/or a phase of the ultrasound receive signal.

Example 24: The method of any one of examples 19 to 23, further comprising:transmitting, during a second time period, another ultrasound transmit signal that propagates within at least a portion of the ear canal of the user;receiving, during the second time period, another ultrasound receive signal, the other ultrasound receive signal representing a version of the other ultrasound transmit signal;determining an absence of another vocalization during the second time period based on the other ultrasound receive signal; andgenerating another control signal that controls the operation of the device based on the determined absence of the other vocalization.

Example 25: The method of example 24, further comprising:disabling a voice control interface and/or voice authentication based on the other control signal.

Example 26: The method of any one of examples 19 to 25, further comprising:receiving, during the first time period, an over-the-air signal comprising the vocalization made by the user during the first time period and a noise component, wherein:the received ultrasound receive signal has a modulation component associated with the noise component of the received over-the-air signal; andthe detecting of the vocalization comprises attenuating the modulation component within the received ultrasound receive signal based on the received audible signal.

Example 27: The method of any one of examples 19 to 26, wherein the vocalization comprises speech, humming, whistling, or singing.

Example 28: The method of any one of examples 19 to 27, further comprising:transmitting, prior to the transmitting of the ultrasound transmit signal, another ultrasound transmit signal that propagates within at least the portion of the ear canal of the user, the other ultrasound transmit signal having multiple tones;receiving, prior to the transmitting of the ultrasound transmit signal, another ultrasound receive signal, the other ultrasound receive signal representing a version of the other ultrasound transmit signal with one or more characteristics modified due to the propagation within the ear canal;generating quality metrics that respectively correspond to the multiple tones, the quality metrics based on amplitudes and/or phases of the multiple tones; andselecting at least two tones from the multiple tones based on the quality metrics corresponding to the at least two tones being greater than a threshold,wherein the transmitting of the ultrasound transmit signal comprises transmitting the ultrasound transmit signal having the at least two tones.

Example 29: The method of example 28, wherein the transmitting of the ultrasound transmit signal comprises at least one of the following:transmitting the ultrasound transmit signal such that the ultrasound transmit signal has a higher amplitude at the at least two tones compared to an amplitude of the other ultrasound transmit signal at the multiple tones; ortransmitting the ultrasound signal such that a duration of ultrasound transmit signal at each of the at least two tones is longer compared to a duration of the other ultrasound transmit signal at each of the multiple tones.

Example 30: A computer-readable storage medium comprising instructions that, responsive to execution by a processor, cause a hearable to perform any one of the methods of example 19 to 29.

Example 31: A device comprising:at least one transducer; andat least one processor, the device configured to perform, using the at least one transducer and the at least one processor, any one of the methods of examples 19 to 29.

Example 32: The device of example 31, further comprising:a speaker; andan active-noise-cancellation circuit comprising a feedback microphone,wherein the at least one transducer comprises the speaker and the feedback microphone.

Example 33: The device of example 31, wherein:the at least one transducer comprises a speaker and a microphone;the speaker is configured to be positioned proximate to a first ear of a user; andthe microphone is configured to be positioned proximate to a second ear of the user.

Example 34: The device of any one of examples 31 to 33, wherein the device comprises:at least one earbud; orheadphones.