Patent Description:
This disclosure generally relates to wearable hearing assist devices. More particularly, the disclosure relates to a machine learning based approach for removing user voice signals in wearable hearing assist devices.

Wearable hearing assist devices can significantly improve the hearing experience for a user. For instance, such devices typically employ one or more microphones and amplification components to amplify desirable sounds such as the voice or voices of others speaking to the user. Additionally, such devices may employ technologies such as active noise reduction (ANR) for countering unwanted environmental noise. Wearable hearing assist devices may come in various form factors, e.g., headphones, earbuds, audio glasses, etc. However, removing unwanted acoustic signals, such as a user's own voice signals, continues to present various technical challenges.

<CIT>, which was not published at the priority date of the present application, discloses a hearing device comprising a neural network for separating a user voice signal from the input signal. It is further known from the patent applications <CIT> and <CIT>, techniques for removing user voice signals, i.e. self-speech, of a user wearing a hearing assist device.

The present invention relates to a method of removing user speech for a hearing assist device and a system according to the independent claims. Advantageous embodiments are set forth in the dependent claims.

All examples and features mentioned below can be combined in any technically possible way.

Systems and approaches are disclosed that employ a wearable hearing assist device that removes a user's speech. Some implementations are discussed in the detailed description.

Two or more features described in this disclosure, may be combined to form implementations not specifically described herein.

Other features, objects and benefits will be apparent from the description and drawings, and from the claims.

It is noted that the drawings of the various implementations are not necessarily to scale. The drawings are intended to depict only typical aspects of the disclosure, and therefore should not be considered as limiting the scope of the implementations.

Various implementations describe solutions for removing a user's own voice in a wearable hearing assist device ("self-speech removal"). In general, when using a hearing assist device, the user can be annoyed or otherwise irritated by amplification of the user's own voice. However, amplification of others' voices is critical for audibility.

In a hearing assist device, such as a hearing aid, an audio augmented reality system, a system utilizing a remote microphone (e.g., from a phone or other device) that streams to a headphone, etc., sounds are transmitted to the ear via two different paths. The first path is the "direct path" where sound travels around the device or headphone and directly into the ear canal. In the second, "processed path," the audio travels through the hearing assist device or headphone, is processed, and is then delivered to the ear canal through the driver (i.e., electrostatic transducer or speaker).

A key factor in the performance of any hearing assist device when processing real time audio for human consumption is latency created by the algorithm processing signals along the processed path. Latency is defined as the delay (usually measured in milliseconds) between the time the audio enters the device to when it emerges. Excessive latency in a system can significantly degrade the perceived quality of the algorithm, as well as the user experience.

If the overall system latency is above a certain threshold (for example on the order of <NUM> to <NUM> milliseconds), it can be extremely disconcerting and annoying to users. More precisely, this threshold is driven by human sensitivity to latency on their own voice in the processed path. However, if the user's own voice is not present in the processed path, users can tolerate latency much better, and the overall system latency threshold increases significantly (for example, up to approximately <NUM>-<NUM>.

This disclosure describes implementations to remove the user's own voice (i.e., self-speech) from the processed path using machine learning based processing, thus effectively increasing the latency budget for audio processing.

Although generally described with reference to hearing assist devices, the solutions disclosed herein are intended to be applicable to a wide variety of wearable audio devices, i.e., devices that are structured to be at least partly worn by a user in the vicinity of at least one of the user's ears to provide amplified audio for at least that one ear. Other such implementations may include headphones, two-way communications headsets, earphones, earbuds, hearing aids, audio eyeglasses, wireless headsets (also known as "earsets') and ear protectors. Presentation of specific implementations are intended to facilitate understanding through the use of examples, and should not be taken as limiting either the scope of disclosure or the scope of claim coverage.

Additionally, the solutions disclosed herein are applicable to wearable audio devices that provide two-way audio communications, one-way audio communications (i.e., acoustic output of audio electronically provided by another device), or no communications, at all. Further, what is disclosed herein is applicable to wearable audio devices that are wirelessly connected to other devices, that are connected to other devices through electrically and/or optically conductive cabling, or that are not connected to any other device, at all. These teachings are applicable to wearable audio devices having physical configurations structured to be worn in the vicinity of either one or both ears of a user, including and not limited to, headphones with either one or two earpieces, over-the-head headphones, behind-the neck headphones, headsets with communications microphones (e.g., boom microphones), in-the-ear or behind-the-ear hearing aids, wireless headsets (i.e., earsets), audio eyeglasses, single earphones or pairs of earphones, as well as hats, helmets, clothing or any other physical configuration incorporating one or two earpieces to enable audio communications and/or ear protection.

In the illustrative implementations, the processed audio may include any natural or manmade sounds (or, acoustic signals) and the microphones may include one or more microphones capable of capturing and converting the sounds into electronic signals.

In various implementations, the wearable audio devices (e.g., hearing assist devices) described herein may incorporate active noise reduction (ANR) functionality that may include either or both feedback-based ANR and feedforward-based ANR, in addition to possibly further providing pass-through audio and audio processed through typical hearing aid signal processing such as dynamic range compression.

Additionally, the solutions disclosed herein are intended to be applicable to a wide variety of accessory devices, i.e., devices that can communicate with a wearable audio device and assist in the processing of audio signals. Illustrative accessory devices include smartphones, Internet of Things (IoT) devices, computing devices, specialized electronics, vehicles, computerized agents, carrying cases, charging cases, smart watches, other wearable devices, etc..

In various implementations, the wearable audio device (e.g., hearing assist device) and accessory device communicate wirelessly, e.g., using Bluetooth, or other wireless protocols. In certain implementations, the wearable audio device and accessory device reside within several meters of each other.

<FIG> depicts an illustrative implementation of a wearable hearing assist device <NUM> that utilizes a machine learning (ML) based approach to output a processed audio signal <NUM> in which self-speech <NUM> is removed or substantially removed. As shown, device <NUM> includes a set of microphones <NUM> configured to receive a mixed acoustic input <NUM> that includes a mixture of self-speech <NUM> and noise <NUM>. Noise <NUM> generally includes all other acoustic inputs other than self-speech <NUM>, e.g., other speech, background voices, environmental sounds, music, etc. Microphone inputs <NUM> receive mixed audio signals from the microphones <NUM> and pass the mixed audio signals <NUM> to audio processing system <NUM>.

Audio processing system <NUM> includes a self-speech filter <NUM> that includes a trained machine learning (ML) model configured to process the mixed audio signals <NUM>. More specifically, the self-speech filter <NUM> uses an intrinsic user vector <NUM>, created during an enrollment process (or, phase) <NUM> based on the user's voice, to remove self-speech <NUM> of the user. In certain implementations, the intrinsic user vector <NUM>, which includes an intrinsic compressed representation of speech patterns of the user, is determined during the enrollment process <NUM>. In other implementations (not encompassed by the wording of the claims), the intrinsic user vector <NUM> can be calculated by device <NUM>. The ML process for generating the intrinsic user vector <NUM> and self-speech filter <NUM> is described further herein, e.g., with reference to <FIG>. Once the mixed audio signals <NUM> are processed with the self-speech filter <NUM> to remove self-speech of the user (i.e., the user of the wearable hearing assist device <NUM>), post processing algorithms <NUM> can further enhance the filtered signal, e.g., by providing speech enhancement, signal-to-noise (SNR) improvement, beamforming, active noise reduction, etc. The resulting signal is then amplified by amplifier system <NUM> and output over electro-acoustic transducer <NUM>.

<FIG> depicts an illustrative overview of the machine learning based process utilized to implement a self-speech filter <NUM> and calculate the intrinsic user vector <NUM>, such as that shown with device <NUM> (<FIG>). The process may be implemented in three phases, including: an offline training phase, an enrollment phase, and an inference phase. An implementation of the inference phase was described in <FIG>, in which device <NUM> filters self-speech of the user during operation of the device <NUM> by the user. The enrollment phase can for example be implemented as a one-time or as-needed operation when a new user begins using the device <NUM>. In various implementations, the offline training phase is responsible for training two ML based models required for the other phases, namely, a speech characterization system <NUM> and the self-speech filter <NUM>.

In certain implementations, speech characterization system <NUM> (first model) is trained with a user utterance database (DB) <NUM> and an associated intrinsic user vector DB <NUM>, e.g., using a neural network or the like. For example, the first model can be trained with user utterance/ expected intrinsic user vector pairs. User utterances are inputted into the speech characterization system <NUM> to generate an intrinsic user vector. The generated intrinsic user vector can be compared to an expected intrinsic user vector from the intrinsic user vector DB <NUM>, with results being fed back to adjust the first model. The training can continue until each user utterance from the user utterance database (DB) <NUM> entered into the model reliably outputs its expected intrinsic user vector. Once trained, the resulting speech characterization system <NUM> can be used in the enrollment phase to map a new (or enrolling) user's utterance to an intrinsic user vector <NUM> that is explicitly associated with the new user. The intrinsic user vector <NUM> generally consists of an intrinsic compressed representation of the user's speech patterns. The intrinsic compressed representation includes at least one of: a d-vector representation or an i-vector representation. As is readily understood in the art, various algorithms exist to extract d-vectors and i-vectors based on a voice sample of the user.

Self-speech filter <NUM> (second model), which is likewise trained offline using a neural network or the like, is trained to remove a person's speech from a mixed audio signal (i.e., an audio signal containing both self-speech <NUM> and noise <NUM>) using an intrinsic user vector of the person. For example, this model can be trained by inputting a mixed audio signal from a mixed audio signal DB <NUM> along with an associated intrinsic user vector to obtain a self-speech filtered signal, which can be compared to an expected signal in the self-speech filtered signal database <NUM>. Results can be fed back to adjust and tune the second model. Once trained, the resulting self-speech filter <NUM> can be deployed in the inference phase (e.g., on device <NUM>).

During the enrollment phase, the user is, for example, instructed to record a short audio clip of themselves speaking a predefined set of utterances. This can be done on the device <NUM> itself or on an accessory device such as a mobile phone or coprocessor. Systems or guidance may be utilized to ensure the recording contains only the user's voice and no other interfering audio or speech. For example, the user may be instructed to speak the utterances in a quiet space, and/or the resulting audio clips can be analyzed to ensure there is no unwanted noise. The recorded enrollment utterance is then passed through the speech characterization system <NUM> (either on the device <NUM> itself, on a companion device, in the cloud, etc.), and the intrinsic user vector <NUM> for the enrolling user is extracted.

Once the intrinsic user vector <NUM> is extracted, it is loaded and stored, e.g., on device <NUM>, and the inference phase can be implemented. During this phase, mixed audio containing the user's self-speech <NUM> is first sent to the self-speech filter <NUM> along with the user's intrinsic vector <NUM> to remove the user's own voice from the processed path. The resulting self-speech filtered audio <NUM> is then passed to any downstream post processing algorithms <NUM>, such as speech enhancement, SNR improvement, beamforming, ANR, etc. Finally, the self-speech filtered, enhanced audio <NUM> is sent to the driver to be reproduced in the ear canal of the user.

Due to the fact that the user's own voice has been removed from the processed path during the inference phase, the user is able to tolerate much higher processing latency than they would with an audio signal that contains their self-speech. This higher tolerance to latency enables more complex downstream post processing algorithms <NUM>, such as algorithms implemented on an accessory, such as a smart phone or wireless coprocessor device, which accumulate latency not only from algorithmic processing but also from the wireless transmission of audio.

<FIG> depicts an alternative embodiment in which the self-speech filter <NUM> and post processing algorithms <NUM> of <FIG> are combined into a single system, i.e., self-speech filter and post processing system <NUM>. In this manner, processing enhancements, such as speech enhancement, SNR improvement, beamforming, ANR, etc., are handled by a single system or machine learning trained model.

<FIG> depicts a further implementation (not encompassed by the claimed wording) in which a similar machine learning platform is utilized to provide speech enhanced audio <NUM>, either as a stand-alone or in addition to the herein described self-speech filtering platform. In this case, the platform creates an intrinsic user vector designed to pass speech from one or more non-device users (i.e., targeted speech). Instead of training a model to remove the speech corresponding to the intrinsic user vector (i.e., self-speech filter <NUM> of <FIG>), a model, in this case targeted speech enhancer <NUM>, is trained to keep the targeted speech of a non-device user. Targeted speech enhancer <NUM> is trained by processing mixed audio signals (that include targeted speech and noise) with associated intrinsic user vectors to dependably generate speech enhanced audio <NUM> that only includes targeted speech. Speech characterization system <NUM> is trained in the same manner as with the self-speech filtering platform.

Accordingly, a user of device <NUM> could choose to only hear speech from their partner by enrolling their partner with speech characterization system <NUM>, and loading their partner's intrinsic user vector <NUM> into targeted speech enhancer <NUM>. When a mixed audio <NUM> containing their partner's speech, i.e., target speech, is presented to the device <NUM>, targeted speech enhancer <NUM> generates speech enhanced audio <NUM> that substantially only includes their partner's speech. Processed enhanced audio <NUM> may thereafter be generated from post processing algorithms <NUM>. In an alternative embodiment, post processing algorithms <NUM> may be incorporated into target speech enhancer <NUM>.

The process may be expanded to handle multiple targets whose speech is allowed to pass simply by enrolling multiple users that will each generate an intrinsic user vector <NUM>. Multiple intrinsic user vectors <NUM> can then be inputted to targeted speech enhancer <NUM>.

It is understood that the devices <NUM>, <NUM> (<FIG>) shown and described according to various implementations may be structured to be worn by a user to provide an audio output to a vicinity of at least one of the user's ears. The devices <NUM>, <NUM> may have any of a number of form factors, including configurations that incorporate a single earpiece to provide audio to only one of the user's ears, others that incorporate a pair of earpieces to provide audio to both of the user's ears, and others that incorporate one or more standalone speakers to provide audio to the environment around the user. Example wearable audio devices are illustrated and described in further detail in <CIT>), which are hereby incorporated by reference in its entirety.

In the illustrative implementations, the acoustic input <NUM> (<FIG>) may include any ambient acoustic signals, including acoustic signals generated by the user of the wearable hearing assist device, including, e.g., natural or manmade sounds (or, acoustic signals). The microphones <NUM> may include one or more microphones (e.g., one or more microphone arrays including a feedforward and/or feedback microphone) capable of capturing and converting the sounds into electronic signals.

It is appreciated that while a few examples have been provided herein relating to removing a user's voice in a hearing assist device, other approaches, or combinations of described approaches can be used.

<FIG> is a schematic depiction of an illustrative wearable hearing assist device <NUM> (in one example form factor) that includes a self-speech removal system contained in housing <NUM>. It is understood that the example wearable hearing assist device <NUM> can include some or all of the components and functionality described with respect to devices <NUM>, <NUM> depicted and described with reference to <FIG>. The self-speech removal system and/or target speech enhancer can be one of, or can be executed on one or more of a variety of electronics <NUM> in the housing <NUM>. In certain embodiments, some or all of the self-speech or enhancement algorithms may be implemented in an accessory <NUM> that is configured to communicate with the wearable hearing assist device <NUM>. In this example, the wearable audio device <NUM> includes an audio headset that includes two earphones (for example, in-ear headphones, also called "earbuds") <NUM>, <NUM>. While the earphones <NUM>, <NUM> are tethered to a housing <NUM> (e.g., neckband) that is configured to rest on a user's neck, other configurations, including wireless configurations can also be utilized. Each earphone <NUM>, <NUM> is shown including a body <NUM>, which can include a casing formed of one or more plastics or composite materials. The body <NUM> can include a nozzle <NUM> for insertion into a user's ear canal entrance and a support member <NUM> for retaining the nozzle <NUM> in a resting position within the user's ear. In addition to the self-speech removal system, the control unit <NUM> can include other electronics <NUM>, e.g., an amplifier, batteries, user controls, a voice activity detection (VAD) device, etc..

In certain implementations, as noted above, a separate accessory <NUM> can include a communication system <NUM> to, e.g., wirelessly communicate with device <NUM> and includes remote processing <NUM> to provide some or all of the functionality described herein, e.g., the self-speech filter <NUM>, the enrollment process <NUM>, post processing algorithms <NUM>, etc. Accessory <NUM> can be implemented in many embodiments. In one embodiment, the accessory <NUM> comprises a stand-alone device. In another embodiment, the accessory <NUM> comprises a user-supplied smartphone utilizing a software application to enable remote processing <NUM> while using the smartphone hardware for communication system <NUM>. In another embodiment, the accessory <NUM> could be implemented within a charging case for the device <NUM>. In another embodiment, the accessory <NUM> could be implemented within a companion microphone accessory, which also performs other functions such as off-head beamforming and wireless streaming of the beamformed audio to device <NUM>. Additionally, other wearable device forms could likewise be implemented, including around-the-ear headphones, over-the-ear headphones, audio eyeglasses, open-ear audio devices etc..

Referring to <FIG>, the set of microphones <NUM> may include an in-ear microphone. Referring to <FIG>, such an in-ear microphone could be integrated into the earbud body <NUM>, for example in nozzle <NUM>. The in-ear microphone can also be used for performing feedback active noise reduction (ANR) and voice pickup for communication, which may be performed within other electronics <NUM>. The user's own voice within the ear canal is substantially different than that received from microphones placed outside of the ear canal due to bone and tissue conduction of own voice. The in-ear microphone can provide unique characteristics, in conjunction with other microphones in the set <NUM>, to the mixed audio signal DB <NUM>, user enrollment utterance <NUM>, and mixed audio containing user's self-speech <NUM>, referring to <FIG>. In conjunction, in-ear and out-ear microphones provide more information to the speech characterization system <NUM>, improving performance of the self-speech filter <NUM>.

According to various implementations, a hearing assist device is provided that will filter the user's self-speech in order to enhance performance. In particular, a self-speech filter <NUM> that utilizes an intrinsic user vector <NUM> will remove the user's voice from a mixed audio signal.

It is understood that one or more of the functions of the described systems may be implemented as hardware and/or software, and the various components may include communications pathways that connect components by any conventional means (e.g., hard-wired and/or wireless connection). For example, one or more non-volatile devices (e.g., centralized or distributed devices such as flash memory device(s)) can store and/or execute programs, algorithms and/or parameters for one or more described devices. Additionally, the functionality described herein, or portions thereof, and its various modifications (hereinafter "the functions") can be implemented, at least in part, via a computer program product, e.g., a computer program tangibly embodied in an information carrier, such as one or more non-transitory machine-readable media, for execution by, or to control the operation of, one or more data processing apparatus, e.g., a programmable processor, a computer, multiple computers, and/or programmable logic components.

Actions associated with implementing all or part of the functions can be performed by one or more programmable processors executing one or more computer programs to perform the functions. All or part of the functions can be implemented as, special purpose logic circuitry, e.g., an FPGA (field programmable gate array) and/or an ASIC (application-specific integrated circuit). Generally, a processor may receive instructions and data from a read-only memory or a random access memory or both. Components of a computer include a processor for executing instructions and one or more memory devices for storing instructions and data.

It is noted that while the implementations described herein utilize microphone systems to collect input signals, it is understood that any type of sensor can be utilized separately or in addition to a microphone system to collect input signals, e.g., accelerometers, thermometers, optical sensors, cameras, etc..

Additionally, actions associated with implementing all or part of the functions described herein can be performed by one or more networked computing devices. Networked computing devices can be connected over a network, e.g., one or more wired and/or wireless networks such as a local area network (LAN), wide area network (WAN), personal area network (PAN), Internet-connected devices and/or networks and/or a cloud-based computing (e.g., cloud-based servers).

In various implementations, electronic components described as being "coupled" can be linked via conventional hard-wired and/or wireless means such that these electronic components can communicate data with one another. Additionally, sub-components within a given component can be considered to be linked via conventional pathways, which may not necessarily be illustrated.

Claim 1:
A method of removing user speech for a hearing assist device (<NUM>), comprising:
receiving an audio signal (<NUM>, <NUM>), wherein the audio signal includes a speech component (<NUM>) of the user and a noise component (<NUM>);
filtering the audio signal with a self-speech filter (<NUM>) that utilizes an intrinsic user vector (<NUM>) to filter the speech component, wherein the intrinsic user vector is determined based on a voice input of the user during an enrollment process and comprises an intrinsic compressed representation of speech patterns (<NUM>, <NUM>) of the user, wherein the intrinsic compressed representation comprises one of: a d-vector representation or an i-vector representation; and
outputting a filtered audio signal (<NUM>, <NUM>) in which the speech component of the user has been substantially removed from the audio signal.