Patent Description:
An automated assistant can convert audio data, corresponding to a spoken utterance of a user, into corresponding text (or other semantic representation). For example, audio data can be generated based on the detection of a spoken utterance of a user via one or more microphones of a client device that includes the automated assistant. The automated assistant can include a speech recognition engine that attempts to recognize various characteristics of the spoken utterance captured in the audio data, such as the sounds produced (e.g., phonemes) by the spoken utterance, the order of the pronounced sounds, rhythm of speech, intonation, etc. Further, the speech recognition engine can identify text words or phrases represented by such characteristics. The text can then be further processed by the automated assistant (e.g., using a natural language understanding engine and/or a dialog state engine) in determining responsive content for the spoken utterance. The speech recognition engine can be implemented by the client device and/or by one or more automated assistant component(s) that are remote from, but in network communication with, the client device.

According to its abstract, <CIT> describes a system configured to process speech commands that may classify incoming audio as desired speech, undesired speech, or non-speech. Desired speech is speech that is from a same speaker as reference speech. The reference speech may be obtained from a configuration session or from a first portion of input speech that includes a wakeword. The reference speech may be encoded using a recurrent neural network (RNN) encoder to create a reference feature vector. The reference feature vector and incoming audio data may be processed by a trained neural network classifier to label the incoming audio data (for example, frame-by-frame) as to whether each frame is spoken by the same speaker as the reference speech. The labels may be passed to an automatic speech recognition (ASR) component which may allow the ASR component to focus its processing on the desired speech.

According to its abstract, <NPL>) describes a speech extraction method that utilizes an inventory of voice snippets of possible interfering speakers, or speaker enrollment data, in addition to that of the target speaker. Furthermore, an attention-based network architecture is proposed to form time-varying masks for both the target and other speakers during the separation process.

Techniques described herein are directed to isolating a human voice from a frequency representation of an audio signal by generating a predicted mask using a trained voice filter model, wherein the frequency representation is generated using an automatic speech recognition (ASR) engine, and where processing the frequency representation with the predicted mask can isolate portion(s) of the frequency representation corresponding to the human voice. The revised frequency representation generated by processing the frequency representation using the predicted mask can be processed using additional portions of the ASR engine to, for example, generate a text representation (e.g., text, a symbolic representation of text, etc.) of utterance(s) spoken by the isolated human voice. In other words, a voice filter model can be used in processing acoustic features (e.g., the frequency representation) and generating revised acoustic features (e.g., the revised frequency representation) isolating portion(s) of the acoustic features corresponding to utterance(s) of a single human for use in an automatic speech recognition without reconstruction of the audio signal(s) from the features.

For example, assume a sequence of audio data that includes first utterance(s) from a first human speaker, second utterance(s) from a second human speaker, and various occurrences of background noise. Implementations disclosed herein can utilize a frequency representation of the sequence of audio data generated using an ASR engine to generate a revised frequency representation that includes portion(s) corresponding to the utterance(s) from the first human speaker, and excludes portion(s) corresponding to the second utterance(s) and the background noise, where the revised frequency representation can be further utilized by an ASR engine without reconstruction of an additional audio signal corresponding to the revised frequency representation.

The voice filter model can be used to process a speaker embedding corresponding to the human speaker in addition to the frequency representation of the audio data to generate the predicted mask. For instance, a speaker embedding corresponding to the first human speaker can be processed in addition to the sequence of audio data to generate a predicted mask which can be utilized to isolate utterance(s) of the first human speaker. In many implementations, a speaker embedding can be generated prior to processing the sequence of audio data during an enrollment process (i.e., a pre-generated speaker embedding). The sequence of audio data can be associated with the pre-generated speaker embedding after verification of the first human speaker (e.g., using voice fingerprinting and/or other biometric verification(s)). Utilizing a pre-generated speaker embedding can enable real-time automatic speech recognition of the sequence of audio data.

In some additional or alternative implementations, a speaker embedding utilized in generating a revised frequency representation can be based on one or more instances of the sequence of audio data itself. For example, a voice activity detector (VAD) can be utilized to determine a first instance of voice activity in the audio data, and portion(s) of the first instance can be utilized in generating a first speaker embedding for a first human speaker. For example, the first speaker embedding can be generated based on processing, using a speaker embedding model, features of the first X (e.g., <NUM>, <NUM>, <NUM>, <NUM>) second(s) of the first instance of voice activity (the first instance of voice activity can be assumed to be from a single speaker). The first speaker embedding can then be utilized to generate a first revised frequency representation that isolates utterance(s) of the first speaker as described herein.

A speech recognition model portion of the ASR engine can be utilized to process acoustic features such as the frequency representation of audio data to generate the predicted text representation of the audio data. In many implementations, the audio data can be processed using a frequency transformation such as a Fourier transform (e.g., a fast Fourier transform, a discrete Fourier transform, and/or additional Fourier transform(s)) to generate the frequency representation of the audio data. In many implementations, a filterbank representation (e.g., a filterbank energy representation of the amount of energy captured in each frequency band of the frequency representation) can be processed using the ASR model to generate the predicted text representation. In some such implementations, the frequency representation can be processed using a filterbank transformation to generate the filterbank representation.

Voice filter models disclosed herein can be utilized in generating a revised frequency representation of a Fourier transform frequency representation, where the revised Fourier transform frequency representation can be processed using the filterbank transform to generate a filterbank representation, and where the filterbank representation can be processed using the ASR model to generate the predicted text representation. Additionally or alternatively, voice filter models can be utilized in generating a revised frequency representation of the filterbank representation, where a Fourier transform representation of the audio data is processed using the filterbank transformation to generate the filterbank frequency representation prior to processing the frequency representation using the voice filter model, and where the revised filterbank frequency representation is processed using the ASR model to generate the predicted text representation.

Modern ASR systems frequently have an inherent robustness to background noise. As such, integrating a voice filter model with an ASR system can cause an over suppression by the voice filter model. In some such implementations, the voice filter model can be trained using an asymmetric loss function, where the asymmetric loss function is used in training a voice filter model more tolerant to under suppression errors and less tolerant to over suppression errors. Additionally or alternatively, an adaptive suppression strength can be utilize at run time to reduce over suppression by the voice filter model, where the adaptive suppression strength can dynamically suppress how much of the frequency representation is filtered based on the amount, type, etc. of additional noise in the audio data (e.g., the type of background noise, the number of additional speaker(s), etc.).

Accordingly, various implementations set forth techniques for isolating utterance(s) of a human speaker in captured audio data utilizing a trained voice filter model which can be integrated with an ASR system. In many implementations, the voice filter model can be used in processing acoustic features of the captured audio generated using the ASR system (i.e., processing a frequency representation of the audio data generated using the ASR system). Moreover, a revised version of the acoustic features, which isolate the utterance(s) of the human speaker generated using the voice filter model (i.e., the revised frequency representation generated using the voice filter model), can be additionally processed using the ASR system to generate predicted text of the utterance(s) without the need to reconstruct the audio signal. Put another way, the revised version of the acoustic features can be directly processed using the ASR system, without first converting the acoustic features back to an audio waveform. Further, the revised version of the acoustic features can be generated based on acoustic features generated using the ASR system, and utilizing a voice filter model that is integrated as part of the ASR system. Processing ASR acoustic features to generate revised acoustic features can enable ASR to be performed more quickly and efficiently as the revised acoustic features are already in a format that is directly utilizable by the ASR system to generate predicted speech. Put another way, the revised acoustic features need not be converted back to a time domain representation and then subsequently processed to convert the time domain representation to a format acceptable to the ASR system. In these and other manners, voice filtering techniques can be integrated with an on-device ASR system to enable quick speech recognition.

Some implementations disclosed herein utilize one or more techniques to mitigate suppression, in audio data, of a target human speaker. Mitigating suppression of the target human speaker in the audio data can directly result in improved speech recognition performance by an ASR system the processes the audio data to generate the speech recognition. This can be due to, for example, the ASR system being trained based on non-suppressed human speech audio data, and having high error rates when processing suppressed human speech audio data. Such techniques that can mitigate suppression can include utilizing an asymmetric loss training function as described herein, utilizing an adaptive suppression strength at inference as described herein, and/or other technique(s) described herein. Absent these technique(s), portion(s) of the audio data corresponding to the targeted human speaker can be suppressed when filtering the audio data using the trained voice filter model. Again, such suppression can lead to an increase error rate in generating text corresponding to the targeted human speaker.

Utilizing an asymmetric loss when training the voice filter model in place of a conventional loss function (e.g., a L2 loss, a mean squared error loss, a cross-entropy loss, etc.) can result in a trained model which gives more tolerance to under suppression errors and less tolerance to over suppression errors. For example, an ASR system trained to have inherent robustness against background noise can filter out background noise in a sequence of audio data as part of generating predicted text. Integrating a voice filter model trained using a conventional loss function into this ASR system can result in an over suppression of the targeted human speaker. However, a voice filter model trained using an asymmetric loss function is trained to have less tolerance to these over suppression errors. In these and other manners, the integration of the voice filter model trained using the asymmetric loss function into the ASR system will lead to an increased accuracy of generated text.

Additionally or alternatively, an adaptive suppression strength utilized at run time can provide additional and/or alternative compensation for over suppression errors. For example, the ASR system can process a compensated frequency representation using an ASR model to generate the predicted text of the targeted human speaker, where the compensated frequency representation includes a dynamic weight of suppression. Utilizing the dynamic suppression strength can decrease the error rate from over suppression when integrated with a voice filter model in the ASR system compared to a voice filter model integrated in the ASR system without a dynamic suppression strength.

Furthermore, parameter(s) of the voice filter model can be quantized (e.g., quantized to <NUM> bit integers) which can reduce the consumption of computing resources (e.g., memory, battery power, processor cycle(s), etc.) when utilizing the voice filter model on a client device when compared to utilizing voice filter model parameter(s) which have not been quantized. This reduction in computing resources can aid in real time and/or on device ASR processing of audio data. For example, quantized voice filter models can enable the voice filter techniques described herein to be utilized on mobile phones or other client devices that have constrained processor and/or memory resources. As another example, quantized voice filter models can enable the voice filter techniques described herein to be utilized on mobile phones or other client devices even when resources of such devices are throttled due to overheating, low battery charge, and/or other condition(s).

Additionally or alternatively, a predicted text representation generated using the ASR system based on revised frequency representation can improve the accuracy of the voice-to-text relative to processing the frequency representation due to, for example, the revised frequency representation lacking background noise, utterance(s) of other user(s) (e.g., overlapping utterances), etc. The improved accuracy of text can increase accuracy of one or more downstream components that rely on the resulting text (e.g., natural language processor(s), module(s) that generate a response based on intent(s) and parameter(s) determined based on natural language processing of the text, etc.). Also, for example, when implemented in combination with an automated assistant and/or other interactive dialog system, the improved accuracy of text can lessen the chance that the interactive dialog system will incorrectly convert the spoken utterance to text, thereby reducing the likelihood of an erroneous response to the utterance by the dialog system. This can less and quantity of dialog turns that would otherwise be needed for a user to again provide the spoken utterance and/or other clarification(s) to the interactive dialog system.

In addition, some implementations include one or more processors (e.g., central processing unit(s) (CPU(s)), graphics processing unit(s) (GPU(s), and/or tensor processing unit(s) (TPU(s)) of one or more computing devices, where the one or more processors are operable to execute instructions stored in associated memory, and where the instructions are configured to cause performance of any of the methods described herein. Some implementations also include one or more non-transitory computer readable storage media storing computer instructions executable by one or more processors to perform any of the methods described herein.

The invention is set forth by the appended claims.

Techniques disclosed herein include training and utilizing a voice filter model, a single-channel source separation model that can run on a client device to preserve only the speech signals from a target user, as part of a streaming speech recognition system. Delivering such a model presents numerous challenges: It should improve the performance when the input signal consists of overlapped speech, and must not hurt the speech recognition performance under all other acoustic conditions. Additionally or alternatively, the voice filter model should be tiny, fast, and perform inference in a streaming fashion, in order to have minimal impact on CPU, memory, battery and latency. In many implementations, training and/or using the voice filter model includes using an asymmetric loss, adopting adaptive runtime suppression strength, and/or quantizing the model.

Integrating a speaker-conditioned speech separation model to production environments, especially on-device automatic speech recognition (ASR) systems presents a number of challenges. Quality wise, the model should not only improve the ASR performance when there are multiple voices, but also should be harmless to the recognition performance under other scenarios, e.g. when the speech is only from the target speaker, or when there are non-speech noise such as music present in the background. For streaming systems, to have minimal latency, bi-directional recurrent layers or temporal convolutional layers shall not be used in the model. For on-device systems, the model must be tiny and fast to add minimal budget in CPU and memory.

A voice filter model in accordance with many implementations can be used in achieving these goals. In some implementations, the voice filter model can operate as a frame-by-frame frontend signal processor to enhance the features consumed by a speech recognizer, without reconstructing audio signals from the features. During training, an asymmetric loss function can be utilized to penalize over-suppression which can make the model harmless under more acoustic environments. At runtime, an adaptive suppression strength can be utilized to dynamically adapt to different noise conditions. In many implementations, the voice filter model parameters can be quantized to <NUM>-bit integers for faster inference.

In a variety of implementations, the focus of utilizing voice filter models can be improving ASR performance. Instead of reconstructing any audio waveform from a frequency representation of the audio data via an inverse frequency transformation, voice filtering can operate as a frame-by-frame frontend signal processor that directly takes acoustic features as input, and outputs enhanced acoustic features. Here the acoustic features are exactly the same features being consumed by ASR models, such that voice filtering doesn't even have its own frequency transformation operator. Since ASR models can take stacked log Mel filterbank energies as input, a voice filter model has two integration options: (<NUM>) taking FFT magnitudes as input, and outputs enhanced FFT magnitudes, which will be used to compute filterbank energies; (<NUM>) taking filterbank energies as input, and outputs enhanced filterbank energies.

As an illustrative example, FFT magnitudes are used as the acoustic features for a voice filter system. First, the FFT magnitudes for both noisy audio and clean audio can be power-law compressed with, for example, y = x<NUM> before being fed to the neural network or used to compute the loss function. The purpose of power-law compression is to equalize and/or partially equalize the importance of quieter sounds relative to loud ones.

Second, both the d-vectors (also referred to herein as speaker embeddings) and the power-law compressed FFT magnitudes are globally normalized using the means and standard deviations computed on a small subset of the training data before they are fed into the neural network. Performing global normalization of the network inputs as a separate operation will largely reduce quantization errors when the neural network parameters are quantized.

The neural network topology of the voice filter model is designed for minimal latency: (<NUM>) The convolutional layers can be 1D instead of 2D, meaning the convolutional kernels are for the frequency dimension only; (<NUM>) The LSTM layers can be uni-directional and able to take streaming inputs. In practice, since frequency-only 1D-CNN is not as powerful as 2D-CNN, many implementations of voice filter models remove these CNN layers and purely consist of LSTM layers.

In many implementations, it can be assumed the d-vector is available at runtime. Users are usually prompted to follow an enrollment process before enabling speaker verification or voice filtering. During this enrollment process, d-vectors are computed from the target user's recordings, and stored on the device. The enrollment is usually a one-off experience.

At training time, noisy audio can be generated by adding the waveforms from one or several interference audios to the waveform of the clean audio, with certain probability. For example, the interference audios can be either speech from other speakers, or non-speech noise from ambient recordings.

Modern ASR models are usually trained with intensively augmented data, such as with different types of background noise and under different reverberance conditions. Such ASR models already inherently have some robustness against noise. When adding a voice filtering component to an existing ASR system, it should be guaranteed that performance does not degrade under all noise conditions. In some instances, ASR performance can degrade when the voice filter model is enabled and, for example, non-speech noise is present. The majority part of the degraded Word Error Rate (WER) is from false deletions, which indicates significant over suppression by the voice filtering model.

To overcome the over suppression problem, a new loss function for spectrogram masking based speech separation/enhancement named asymmetric loss can be utilized. Let Scln(t,f) and Senh(t,f) denote the clean and enhanced time-frequency FFT magnitudes, respectively. A conventional L2 loss with compression of power <NUM> is defined as:
<MAT>.

In many implementations, it is desirable to be more tolerate to under-suppression errors, and less tolerate to over-suppression errors. Thus, an asymmetric penalty function gasym can be defined with penalty factor α > <NUM> for a system that can be more tolerant to under-suppression errors and less tolerant to over-suppression errors:
<MAT>.

Then the asymmetric L2 loss function can be defined as:
<MAT>.

As mentioned before, modern ASR models are usually already robust against non-speech noise. Having a voice filtering model that performs an additional step of feature masking can harm the ASR performance.

One way to mitigate the performance degradation is to have an additional compensation to the over-suppression at inference time. Let Sin and Senh denote the input and enhanced FFT magnitudes, respectively, the final compensated output would be:
<MAT>.

Here w is the weight of suppression, and p is the compression power. When w = <NUM>, voice filtering is completely disabled; and when w = <NUM>, there is no compensation.

In practice, a larger w should be used when the voice filtering model improves ASR, and a smaller w should be used when it hurts ASR. For multi-talker speech, the FFT magnitudes usually have better separable energy bands than speech with non-speech noise. The above average rate (AAR), which is defined as the percentage of values that's larger than the mean value, is a good statistic that reflects this property. In many implementations, speech under non-speech noise tends to have larger AAR than speech under speech noise. Thus, the adaptive weight at time t can be defined as
<MAT>
where <NUM> ≤ β < <NUM> is the moving average coefficient, a and b define a linear transform from AAR to weight, and <MAT> is a rectification function that clips values above wmax or below wmin.

Raw TensorFlow graphs can store voice filter network parameters in <NUM>-bit floating point values, and may not be well optimized for on-device inference. In many implementations, the voice filtering models can be serialized to the FlatBufferTensorFlow Lite format. Additionally or alternatively the network parameters can be quantized to <NUM>-bit integers. This can reduce memory cost, and/or make use of optimized hardware instructions for integer arithmetic.

Turning initially to <FIG>, an example environment is illustrated in which various implementations can be performed. <FIG> includes a client computing device <NUM>. In many implementations, client device <NUM> can execute an instance of an automated assistant (not depicted).

The client computing device <NUM> may be, for example: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, and in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart television, and/or a wearable apparatus of a user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided.

The example environment illustrated in <FIG> includes voice filter engine <NUM>, masking engine <NUM>, power compression engine <NUM>, normalization engine <NUM>, automatic speech recognition (ASR) engine <NUM>, speaker embedding engine <NUM>, voice filter model <NUM>, and/or additional engine(s) and/or additional model(s) (not depicted). Voice filter engine <NUM>, masking engine <NUM>, power compression engine <NUM>, normalization engine <NUM>, ASR engine <NUM>, and speaker embedding engine <NUM> are example components in which techniques described herein may interface. The operations performed by one or more engines <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> of <FIG> may be distributed across multiple computing engines. In some implementations, one or more aspects of engines <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> may be combined into a single system and/or more aspects may be implemented on the client device <NUM>. For example, in some of those implementations, aspects of voice filter engine <NUM> may be combined with aspects of speaker embedding engine <NUM>. Engines in accordance with many implementations may each be implemented in one or more computing devices that communicate, for example, through a communication network. A communication network may include a wide area network such as the Internet one or more local area networks ("LAN"s) such as Wi-Fi LANs, mesh networks, etc., and/or one or more bus subsystems. A communication network may optionally utilize one or more standard communication technologies, protocols, and/or inter-process communication techniques.

Voice filter model <NUM> can be trained to process a frequency representation of an audio signal as well as a speaker embedding corresponding to a human speaker to generate a predicted mask, where the frequency representation can be processed with the predicted mask to generate a revised frequency representation isolating utterance(s) of the human speaker. In various implementations, voice filter model <NUM> can be trained as described herein in accordance with <FIG> and/or <FIG>. In a variety of implementations, ASR engine <NUM> can process captured audio to determine a frequency representation of the audio data. In many implementations, the audio data can be captured in real time using one or more microphones of client device <NUM> (not depicted). For example, ASR engine <NUM> can generate a FFT frequency representation of the audio data, a filterbank representation of the audio data, and/or additional frequency representation(s) of the audio data.

In some implementations, the frequency representation generated using ASR engine <NUM> can be additionally processed using power compression engine <NUM> and/or normalization engine <NUM>. For example, power compression engine <NUM> can be utilized to process the frequency representation using a power compression process to equalize (or partially equalize) the importance of quieter sounds relative to loud sounds in the audio data. Additionally or alternatively, normalization engine <NUM> can be utilized to normalize the frequency representation.

Speaker embedding engine <NUM> can determine a speaker embedding corresponding to a human speaker. In some implementations, speaker embedding engine <NUM> can process portion(s) of the captured audio data using a speaker embedding model (not depicted) to generate the speaker embedding. Additionally or alternatively, speaker embedding engine <NUM> can select a pre-generated speaker embedding (e.g., a speaker embedding previously generated using an enrollment process) using voice fingerprinting, image recognition, a passcode, and/or other verification techniques to determine the human speaker currently active and, as a result, the speaker embedding for the currently active human speaker. In many implementations, normalization engine <NUM> can normalize the selected speaker embedding.

Voice filter engine <NUM> can generate a revised frequency representation which isolates utterance(s) of the human speaker corresponding to the speaker embedding selected using speaker embedding engine <NUM>. In many implementations, masking engine <NUM> can process the speaker embedding selected using speaker embedding engine <NUM> and the frequency representation generated using ASR engine <NUM> using voice filter model <NUM> to generate a predicted mask. Voice filter engine <NUM> can process the frequency representation using the predicted mask to generate the revised frequency representation. For example, voice filter engine <NUM> can convolve the frequency representation with the predicted mask to generate the revised frequency representation. Additionally or alternatively, ASR engine <NUM> can process the revised frequency representation generated using voice filter engine <NUM> to generate a predicted text representation of the isolated utterance(s) of the human speaker.

Turning to <FIG>, an example of training a voice filter model <NUM> is illustrated. The voice filter model <NUM> can be a neural network model and can include a convolutional neural network portion, a recurrent neural network portion, a fully connected feed forward neural network portion, and/or additional neural network layers. The voice filter model <NUM> is trained to be used to generate, based on processing a frequency domain representation of audio data and a speaker embedding of a target human speaker, a revised frequency representation of the audio data that isolates utterance(s) (if any) of the target speaker. As described herein, the voice filter model <NUM> can be trained to accept as input, a frequency representation of the audio data (i.e., a frequency representation generated by processing the audio data with a frequency transformation portion of an ASR engine). As further described herein, the output is also generated using the speaker embedding of the target speaker. For example, the speaker embedding of the target speaker can be applied as input to one or more portions of the voice filter model. Accordingly, voice filter model <NUM>, once trained, can be used to generate, as output of the voice filter model <NUM>, a predicted mask which can be convolved with the frequency representation to generate a revised frequency representation. The revised frequency representation can be processed using additional and/or alternative portions of the ASR engine to, for example, generate a predicted text representation of the audio data.

Also illustrated is an training instance engine <NUM> that generates a plurality of training instances 206A-N, that are stored in training instances database <NUM> for use in training the voice filter model <NUM>. Training instance 206A is illustrated in detail in <FIG>. The training instance 206A includes a mixed instance of audio data 210A, an embedding of a given speaker 212A, and ground truth instance of audio data 210A that includes only utterance(s) from the given speaker corresponding to the embedding 212A. In many implementations, voice filter engine <NUM> can utilize a frequency transformation portion of an ASR engine to generate a mixed frequency representation based on the mixed instance of audio data 210A. Similarly, voice filter engine <NUM> can be utilized to generate a ground truth frequency representation of ground truth audio 214A using the frequency transformation portion of the ASR engine. Additionally or alternatively, training instance 206A can include a mixed frequency representation in addition to or in place of mixed instance of audio data 210A and/or a ground truth frequency representation in addition to or in place of ground truth audio data 214A.

The training instance engine <NUM> can generate the training instances 206A-N based on instances of audio data from instances of audio data database <NUM>, and through interaction with speaker embedding engine <NUM>. For example, the training instance engine <NUM> can retrieve the ground truth audio data 214A from the instances of audio data database <NUM> - and use it as the ground truth audio data for the training instance 206A.

Further, the training instance engine <NUM> can provide an instance of audio data from the target speaker to the speaker embedding engine <NUM> to receive, from the speaker embedding engine <NUM>, the embedding of the given speaker 212A. In many implementations, the speaker embedding engine <NUM> can process one or more segments of the ground truth audio data 214A using the speaker embedding model <NUM> to generate the embedding of the given speaker 212A. For example, the speaker embedding engine <NUM> can utilize a voice activity detector (VAD) to determine one or more segments of the ground truth audio data 214A that include voice activity, and determine the embedding of the given speaker 212A based on processing one or more of those segments using the speaker embedding model <NUM>. For instance, all of the segments can be processed using the speaker embedding model <NUM>, and the resulting final output generated based on the processing can be used as the embedding of the given speaker 212A. Also, for instance, a first segment can be processed using the speaker embedding model <NUM> to generate a first output, a second segment can be processed using the speaker embedding model <NUM> to generate a second output, etc. - and a centroid of the outputs utilized as the embedding of the given speaker 212A. Additionally or alternatively, speaker embedding engine <NUM> can determine a speaker embedding generated through an enrollment process which corresponds to a user profile associated with the client device.

The training instance engine <NUM> generates the mixed instance of audio data 210A by combining the ground truth audio data 214A with an additional instance of audio data from the instances of audio data database <NUM>. For example, the additional instance of audio data can be one that includes one or more other human speaker(s) and/or background noise.

In training the voice filter model <NUM> based on the training instance 206A, the voice filter engine <NUM> applies a frequency representation of the mixed instance of audio data 210A as input to the CNN portion of the voice filter model to generate CNN output. Additionally or alternatively, voice filter engine <NUM> applies the embedding of the given speaker 212A as well as the CNN output as input to the RNN portion of voice filter model <NUM> to generate RNN output. Furthermore, the voice filter engine <NUM> applies the RNN output as input to a fully connected feed-forward portion of voice filter model <NUM> to generate a predicted mask, which voice filter engine <NUM> can utilize in processing the mixed frequency representation (i.e., the frequency representation of the mixed instance of audio data 210A) to generate a predicted refined frequency representation 218A that isolates utterance(s) of the human speaker.

The loss module <NUM> generates a loss 220A as a function of: the predicted frequency representation 218A (i.e., a frequency representation of the audio data isolating utterance(s) of the human speaker) and a frequency representation of ground truth audio data 214A. The loss 220A is provided to update module <NUM>, which updates voice filter model <NUM> based on the loss. For example, the update module <NUM> can update one or more weights of the voice filter model using backpropagation (e.g., gradient descent). In many implementations, loss module <NUM> can generate an asymmetric loss 220A to mitigate over suppression by voice filter model <NUM> when non-speech noise is present.

While <FIG> only illustrates a single training instance 206A in detail, it is understood that training instances database <NUM> can include a large quantity of additional training instances. The additional training instances can include training instances of various lengths (e.g., based on various durations of audio data), training instances with various ground truth audio data and speaker embeddings, and training instances with various additional sounds in the respective mixed instances of audio data. Moreover, it is understood that a large quantity of the additional training instances will be utilized to train voice filter model <NUM>.

<FIG> illustrates an example of generating a revised frequency representation of audio data using the audio data, a speaker embedding, and a voice filter model. The voice filter model <NUM> can be the same as voice filter model <NUM> of <FIG>, but has been trained (e.g., utilizing process <NUM> of <FIG> as described herein).

In <FIG>, the voice filter engine (e.g., voice filter engine <NUM> as illustrated in <FIG> and/or <FIG>) can receive a frequency representation <NUM> generated using a frequency transformation portion of an ASR engine. The frequency representation <NUM> can be, for example, streaming audio data that is processed in an online manner (e.g., in real-time or in near real-time) or non-streaming audio data that has been previously recorded and provided to the voice filter engine. The voice filter engine also receives a speaker embedding <NUM> form a speaker embedding engine (e.g., speaker embedding engine <NUM> as illustrated in <FIG> and/or <FIG>). The speaker embedding <NUM> is an embedding for a given human speaker, and can be generated based on processing one or more instances of audio data, from the given speaker, using a speaker embedding model. As described herein, in some implementations, the speaker embedding <NUM> is previously generated by the speaker embedding engine based on previous instance(s) of audio data from the given speaker. In some of those implementations, the speaker embedding <NUM> is associated with an account of the given speaker and/or a client device of the given speaker, and the speaker embedding <NUM> can be provided for utilization with the frequency representation <NUM> based on the frequency representation <NUM> coming from the client device and/or the digital system where the account has been authorized.

In many implementations, the voice filter engine can optionally process frequency representation <NUM> using a power compression process to generate power compression <NUM>. In many implementations, the power compression process can equalize (or partially equalize) the importance of quieter sounds relative to loud sounds in the audio data. Additionally or alternatively, the voice filter engine can optionally process frequency representation <NUM> using a normalization process to generate normalization <NUM>, and can optionally process speaker embedding <NUM> using the normalization process to generate normalization <NUM>.

Frequency representation <NUM> can be applied as input to a convolutional neural network (CNN) portion <NUM> of voice filter model <NUM>. In many implementations, CNN portion <NUM> is a one dimensional convolutional neural network. In many implementations, convolutional output generated by the CNN portion <NUM>, as well as speaker embedding <NUM>, can be applied as input to a recurrent neural network (RNN) portion <NUM> of voice filter model <NUM>. In many implementations, RNN portion <NUM> can include uni-directional memory units (e.g., long short term memory units (LSTM), gated recurrent units (GRU), and/or additional memory unit(s)). Additionally or alternatively, RNN output generated by the RNN portion <NUM> can be applied as input to a fully connected feed-forward neural network portion <NUM> of voice filter model <NUM> to generate predicted mask <NUM>. In many implementations, CNN portion <NUM> can be omitted and frequency representation <NUM> and speaker embedding <NUM> can both be applied as input to RNN <NUM>.

Frequency representation <NUM> can be processed with predicted mask <NUM> to generate revised frequency representation <NUM>. For example, frequency representation <NUM> can be convolved <NUM> with predicted mask <NUM> to generate revised frequency representation <NUM>. In many implementations, text representation <NUM> can be generated by processing revised frequency representation <NUM> using a speech recognition process <NUM> such as processing the revised frequency representation <NUM> using an ASR model of the ASR engine (not depicted).

In many implementations, the revised frequency representation <NUM> can: be the same as frequency representation <NUM> when frequency representation <NUM> includes only utterance(s) from the speaker corresponding to the speaker embedding <NUM>; be null/zero when the frequency representation <NUM> lacks any utterances from the speaker corresponding to the speaker embedding <NUM>; or exclude additional sound(s) while isolating utterance(s) from the speaker corresponding to the speaker embedding <NUM>, when the frequency representation <NUM> includes utterance(s) from the speaker and additional sound(s) (e.g., overlapping utterance(s) of other human speaker(s) and/or additional background noise).

In many implementations, the text representation <NUM> can be provided to one or more additional components by a voice filter engine, by an ASR engine, and/or by additional engine(s). Although <FIG> illustrates generating a single instance of text representation <NUM> based on a single speaker embedding <NUM>, it is understood that in various implementations, multiple instances of text representations can be generated, with each instance being based on frequency representation <NUM> and a unique speaker embedding for a unique human speaker.

Turning now to <FIG>, an example process <NUM> is illustrated of generating training instances for training a voice filter model according to implementations disclosed herein. For convenience, the operations of certain aspects of the flowchart of <FIG> are described with reference to ground truth audio data <NUM>, additional audio data <NUM>, and mixed audio data <NUM> that are schematically represented in <FIG>. Also, for convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as training instance engine <NUM> and/or one or more of GPU(s), CPU(s), and/or TPU(s). Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block <NUM>, the system selects a ground truth instance of audio data that includes spoken input from a single human speaker. For example, the system can select ground truth audio data <NUM> of <FIG>. In <FIG>, the arrow illustrates time and the three diagonal shading areas in the ground truth audio data <NUM> represent segments of the audio data where "Speaker A" is providing a respective utterance. Notably, the ground truth audio data <NUM> includes no (or de minimis) additional sounds.

At block <NUM>, the system determines a speaker embedding for the single human speaker. For example, the speaker embedding can be generated by processing one or more segments of the ground truth instance of audio data <NUM> of <FIG>, using a speaker embedding model. Additionally or alternatively, the system can determine a speaker embedding associated with the single human speaker of the ground truth instance of audio data.

At block <NUM>, the system selects an additional instance of audio data that lack spoken input from the single human speaker. The additional instance of audio data can include spoken input from other speaker(s) and/or background noise (e.g., music, sirens, air conditioning noise, etc.). For example, the system can select additional instance of audio data <NUM> schematically illustrated in <FIG> which includes an utterance from "Speaker B" (crosshatch shading) and "background noise" (stippled shading). Notably, "Speaker B" is different from "Speaker A".

At block <NUM>, the system generates a mixed instance of audio data that combines the ground truth instance of audio data and the additional instance of audio data. For example, mixed audio data <NUM> of <FIG> is generated by combining ground truth audio data <NUM> and additional audio data <NUM>. Accordingly, mixed audio data <NUM> includes the shaded areas from ground truth audio data <NUM> (diagonal shading) and the shaded areas from additional audio data <NUM> (crosshatch shading and stippled shading). Accordingly, in mixed audio data <NUM>, both "Speaker A" and "Speaker B" utterance are included, as well as "background noise". Further, parts of "Speaker A" utterances overlap with parts of the "background noise" and with part of "Speaker B" utterances.

At block <NUM>, the system generates and stores a training instance that includes: the mixed instance of audio data, the speaker embedding, and the ground truth instance of audio data. For example, the system can generate and store a training instance that includes: mixed instance of audio data <NUM>, the ground truth instance of audio data <NUM>, and a speaker embedding generated using the ground truth instance of audio data <NUM>.

At block <NUM>, the system determines whether to generate an additional training instance using the same ground truth instance of audio data and the same speaker embedding, but a different mixed instance of audio data that is based on another additional instance. If so, the system proceeds back to block <NUM> and selects a different additional instance, proceeds to block <NUM> and generates another mixed instance of audio data that combines the same ground truth instance of audio data and the different additional instance, then proceeds to block <NUM> and generates and stores a corresponding training instance.

If, if at an iteration of block <NUM>, the system determines not to generate an additional training instance using the same ground truth instance of audio data and the same speaker embedding, the system proceeds to block <NUM> and determines whether to generate an additional training instance using another ground truth instance of training data. If so, the system performs another iteration of blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> utilizing a different ground truth instance of audio data with a different human speaker, utilizing a different speaker embedding, and optionally utilizing a different additional instance of audio data.

If, at an iteration of block <NUM>, the system determines not to generate an additional training instance using another ground truth instance of audio data, the system proceeds to block <NUM> and generating of training instances ends.

Turning now to <FIG>, an example process <NUM> is illustrated of training a voice filter model according to various implementations disclosed herein. For convenience, the operations of the flowchart are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as voice filter engine <NUM> and/or one or more GPU(s), CPU(s), and/or TPU(s). Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block <NUM>, the system selects a training instance that includes a mixed instance of audio data, a speaker embedding, and ground truth audio data. For example, the system can select a training instance generated according to process <NUM> of <FIG>.

At block <NUM>, the system processes the mixed instance of audio data using a frequency transformation portion of an ASR engine to generate a mixed frequency representation. In a variety of implementations, the frequency transformation portion of the ASR engine includes a Fourier transformation, such as a fast Fourier transformation. Additionally or alternatively, the frequency transformation can include a filterbank process, which can convert a frequency representation of audio data into filterbank energies.

At block <NUM>, the system processes the mixed frequency representation and the speaker embedding using a voice filter model (e.g., voice filter model <NUM>) to generate a predicted mask.

At block <NUM>, the system processes the mixed frequency representation using the predicted mask to generate a revised frequency representation. In many implementations, the predicted mask is convolved with the mixed frequency representation to generate the revised frequency representation. In various implementations, the predicted mask isolates frequency representations of utterance(s) of the human speaker in the frequency representation of the mixed audio data.

At block <NUM>, the system processes the ground truth instance of audio data using the frequency transformation portion of the ASR engine to generate a ground truth frequency representation <NUM>.

At block <NUM>, the system generates an asymmetric loss based on comparing the revised frequency representation to the ground truth frequency representation. In many implementations, an existing ASR system inherently has some robustness against background noise, which can lead to over suppression of utterance(s) captured in audio data. An asymmetric loss can be more tolerant to under-suppression errors, and can additionally and/or alternatively be used to penalize over suppression by improving speech recognition when speech is overlapped with additional noise(s). In many implementations, the system generates additional and/or alternative loss(es) which are not asymmetric.

At block <NUM>, the system updates one or more weights of the voice filter model based on the generated loss (i.e., backpropagation).

At block <NUM>, the system determines whether to perform more training of the voice filter model. If so, the system proceeds back to block <NUM>, selects an additional training instance, then performs an iteration of blocks <NUM>, <NUM>, <NUM>, <NUM>, <NUM>, and <NUM> based on the additional training instance, and then performs an additional iteration of block <NUM>. In some implementations, the system can determine to perform more if there are one or more additional unprocessed training instances and/or if other criterion/criteria are not yet satisfied. The other criterion/criteria can include, for example, whether a threshold number of epochs have occurred and/or a threshold duration of training has occurred. Although process <NUM> is described with respect to a non-batch learning technique, batch learning may additional and/or alternatively be utilized.

If, at an iteration of block <NUM>, the system determines not to perform more training, the system can proceed to block <NUM> where the system considers the voice filter model trained, and provides the voice filter model for use. For example, the system can provide the trained voice filter model for use in process <NUM> (<FIG>) as described herein.

Turning now to <FIG>, a process <NUM> of generating refined audio data using the audio data, a speaker embedding, and a voice filter model, according to various implementations disclosed herein. For convenience, the operations of certain aspects of the flowchart of <FIG> are described with reference to audio data corresponding to a mixed frequency representation <NUM> and audio data corresponding to a revised frequency representation <NUM> that are schematically represented in <FIG>. Also, for convenience, the operations of the flowchart of <FIG> are described with reference to a system that performs the operations. This system may include various components of various computer systems, such as voice filter engine <NUM>, and/or one or more GPU(s), CPU(s), and/or TPU(s). In various implementations, one or more blocks of <FIG> may be performed by a client device using a speaker embedding model and a machine learning model stored locally at the client device. Moreover, while operations of process <NUM> are shown in a particular order, this is not meant to be limiting. One or more operations may be reordered, omitted, and/or added.

At block <NUM>, the system receives a mixed instance of audio data that captures utterance(s) of a human speaker and additional sound(s) that are not from the human speaker. In some implementations, the audio data is streaming audio data. As one example, at block <NUM> the system can receive the audio data corresponding to the mixed frequency representation <NUM> of <FIG>, which includes utterances from "Speaker A" (diagonal shading), as well as utterances from "Speaker B" (stippled shading) and "background noise" (crosshatch shading).

At block <NUM>, the system identifies a previously generated speaker embedding for the human speaker. For example, the system can select a previously generated speaker embedding for "Speaker A". For instance, the speaker embedding could have been previously generated based on an immediately preceding utterance from "Speaker A" that was received at the client device that generated the audio data - and can be selected based on "Speaker A" being the speaker of the immediately preceding utterance. Also, for instance, the speaker embedding could have been previously generated during an enrollment process performed by "Speaker A" for an automated assistant, client device, and/or other digital system. In such an instance, the speaker embedding can be selected based on the audio data being generated by the client device and/or via an account of "Speaker A" for the digital system. As one particular instance, audio data received at block <NUM> can be determined to be from "Speaker A" based on "Speaker A" being recently verified as an active user for the digital system. For example, voice fingerprinting, image recognition, a passcode, and/or other verification may have been utilized to determine "Speaker A" is currently active and, as a result, the speaker embedding for "Speaker A" can be selected.

At block <NUM>, the system generates a mixed frequency representation by processing the audio data with a frequency transformation portion of an ASR engine. For example, the frequency transformation portion of the ASR engine can include a Fast Fourier Transformation (FFT), where the audio data is transformed using the FFT to generate a FFT representation of the audio data, and where the FFT representation of the audio data is utilized by the system as the mixed frequency representation. In many implementations, the frequency transformation portion of the ASR engine can include a FFT as well as a filterbank transformation, where the audio data is transformed using the FFT to generate a FFT representation of the audio data, the FFT representation is transformed using the filterbank transformation to generate a filterbank representation, and where the filterbank representation of the audio data is utilized by the system as the mixed frequency representation.

At block <NUM>, the system generates a predicted mask by processing the mixed frequency representation and the speaker embedding using a voice filter model. In many implementations, the voice filter model can be trained in accordance with process <NUM> of <FIG> as described herein.

At block <NUM>, the system generates a revised frequency representation by processing the mixed frequency representation with the predicted mask. For example, the mixed frequency representation can be convolved with the predicted mask to generate the revised frequency representation. For example, the system can generate a revised frequency representation corresponding to audio data <NUM> schematically illustrated in <FIG>, in which only utterances of "Speaker A" remain.

At block <NUM>, the system generates a text representation of the utterance(s) of the human speaker by processing the revised frequency representation using the ASR engine. In many implementations, the text representation can be text, a symbolic representation of text, and/or additional representation(s) of the text. For example, the system can generate a text representation of the utterance(s) of "Speaker A" by processing the revised frequency representation <NUM> using the ASR engine.

In many implementations, when the mixed frequency representation is a FFT representation (as described herein with respect to block <NUM>), the system can process the revised frequency representation (i.e., a revised FFT representation of the audio data) using a filterbank transformation to generate a filterbank representation. The filterbank representation can be processed using an ASR model portion of the ASR engine to generate the text representation of the utterance(s) of the human speaker. In additional and/or alternative implementations, when the mixed frequency representation is a filterbank representation (as described herein with respect to block <NUM>), the system can process the revised frequency representation (i.e., a revised filterbank representation of the audio data) using the ASR model portion of the ASR engine to generate the text representation of the utterance(s) of the human speaker.

Turning now to <FIG>, an example environment is illustrated where various implementations can be performed. <FIG> is described initially, and includes a client computing device <NUM>, which executes an instance of an automated assistant client <NUM>. One or more cloud-based automated assistant components <NUM> can be implemented on one or more computing systems (collectively referred to as a "cloud" computing system) that are communicatively coupled to client device <NUM> via one or more local and/or wide area networks (e.g., the Internet) indicated generally at <NUM>.

An instance of an automated assistant client <NUM>, by way of its interactions with one or more cloud-based automated assistant components <NUM>, may form what appears to be, from the user's perspective, a logical instance of an automated assistant <NUM> with which the user may engage in a human-to-computer dialog. An instance of such an automated assistant <NUM> is depicted in <FIG>. It thus should be understood that in some implementations, a user that engages with an automated assistant client <NUM> executing on client device <NUM> may, in effect, engage with his or her own logical instance of an automated assistant <NUM>. For the sakes of brevity and simplicity, the term "automated assistant" as used herein as "serving" a particular user will often refer to the combination of an automated assistant client <NUM> executing on a client device <NUM> operated by the user and one or more cloud-based automated assistant components <NUM> (which may be shared amongst multiple automated assistant clients of multiple client computing devices). It should also be understood that in some implementations, automated assistant <NUM> may respond to a request from any user regardless of whether the user is actually "served" by that particular instance of automated assistant <NUM>.

The client computing device <NUM> may be, for example: a desktop computing device, a laptop computing device, a tablet computing device, a mobile phone computing device, a computing device of a vehicle of the user (e.g., an in-vehicle communications system, an in-vehicle entertainment system, an in-vehicle navigation system), a standalone interactive speaker, a smart appliance such as a smart television, and/or a wearable apparatus of the user that includes a computing device (e.g., a watch of the user having a computing device, glasses of the user having a computing device, a virtual or augmented reality computing device). Additional and/or alternative client computing devices may be provided. In various implementations, the client computing device <NUM> may optionally operate one or more other applications that are in addition to automated assistant client <NUM>, such as a message exchange client (e.g., SMS, MMS, online chat), a browser, and so forth. In some of those various implementations, one or more of the other applications can optionally interface (e.g., via an application programming interface) with the automated assistant <NUM>, or include their own instance of an automated assistant application (that may also interface with the cloud-based automated assistant component(s) <NUM>).

Automated assistant <NUM> engages in human-to-computer dialog sessions with a user via user interface input and output devices of the client device <NUM>. To preserve user privacy and/or to conserve resources, in many situations a user must often explicitly invoke the automated assistant <NUM> before the automated assistant will fully process a spoken utterance. The explicit invocation of the automated assistant <NUM> can occur in response to certain user interface input received at the client device <NUM>. For example, user interface inputs that can invoke the automated assistant <NUM> via the client device <NUM> can optionally include actuations of a hardware and/or virtual button of the client device <NUM>. Moreover, the automated assistant client can include one or more local engines <NUM>, such as an invocation engine that is operable to detect the presence of one or more spoken invocation phrases. The invocation engine can invoke the automated assistant <NUM> in response to detection of one of the spoken invocation phrases. For example, the invocation engine can invoke the automated assistant <NUM> in response to detecting a spoken invocation phrase such as "Hey Assistant," "OK Assistant", and/or "Assistant". The invocation engine can continuously process (e.g., if not in an "inactive" mode) a stream of audio data frames that are based on output from one or more microphones of the client device <NUM>, to monitor for an occurrence of a spoken invocation phrase. While monitoring for the occurrence of the spoken invocation phrase, the invocation engine discards (e.g., after temporary storage in a buffer) any audio data frames that do not include the spoken invocation phrase. However, when the invocation engine detects an occurrence of a spoken invocation phrase in processed audio data frames, the invocation engine can invoke the automated assistant <NUM>. As used herein, "invoking" the automated assistant <NUM> can include causing one or more previously inactive functions of the automated assistant <NUM> to be activated. For example, invoking the automated assistant <NUM> can include causing one or more local engines <NUM> and/or cloud-based automated assistant components <NUM> to further process audio data frames based on which the invocation phrase was detected, and/or one or more following audio data frames (whereas prior to invoking no further processing of audio data frames was occurring). For instance, local and/or cloud-based components can generate refined versions of audio data and/or perform other processing in response to invocation of the automated assistant <NUM>. In some implementations, the spoken invocation phrase can be processed to generate a speaker embedding that is used in generating a refined version of audio data that follows the spoken invocation phrase. In some implementations, the spoken invocation phrase can be processed to identify an account associated with a speaker of the spoken invocation phrase, and a stored speaker embedding associated with the account utilized in generating a refined version of audio data that follows the spoken invocation phrase.

The one or more local engine(s) <NUM> of automated assistant <NUM> are optional, and can include, for example, the invocation engine described above, a local voice-to-text ("STT") engine (that converts captured audio to text), a local text-to-speech ("TTS") engine (that converts text to speech), a local natural language processor (that determines semantic meaning of audio and/or text converted from audio), and/or other local components. Because the client device <NUM> is relatively constrained in terms of computing resources (e.g., processor cycles, memory, battery, etc.), the local engines <NUM> may have limited functionality relative to any counterparts that are included in cloud-based automated assistant components <NUM>.

Cloud-based automated assistant components <NUM> leverage the virtually limitless resources of the cloud to perform more robust and/or more accurate processing of audio data, and/or other user interface input, relative to any counterparts of the local engine(s) <NUM>. Again, in various implementations, the client device <NUM> can provide audio data and/or other data to the cloud-based automated assistant components <NUM> in response to the invocation engine detecting a spoken invocation phrase, or detecting some other explicit invocation of the automated assistant <NUM>.

The illustrated cloud-based automated assistant components <NUM> include a cloud-based TTS module <NUM>, a cloud-based STT module <NUM>, a natural language processor <NUM>, a dialog state tracker <NUM>, and a dialog manager <NUM>. In some implementations, one or more of the engines and/or modules of automated assistant <NUM> may be omitted, combined, and/or implemented in a component that is separate from automated assistant <NUM>. Further, in some implementations automated assistant <NUM> can include additional and/or alternative engines and/or modules. Cloud-based STT module <NUM> can convert audio data into text, which may then be provided to natural language processor <NUM>. In various implementations, the cloud-based STT module <NUM> can covert audio data into text based at least in part on revised version(s) of frequency representation that are provided by the voice filter engine <NUM>.

Cloud-based TTS module <NUM> can convert textual data (e.g., natural language responses formulated by automated assistant <NUM>) into computer-generated speech output. In some implementations, TTS module <NUM> may provide the computer-generated speech output to client device <NUM> to be output directly, e.g., using one or more speakers. In other implementations, textual data (e.g., natural language responses) generated by automated assistant <NUM> may be provided to one of the local engine(s) <NUM>, which may then convert the textual data into computer-generated speech that is output locally.

Natural language processor <NUM> of automated assistant <NUM> processes free form natural language input and generates, based on the natural language input, annotated output for use by one or more other components of the automated assistant <NUM>. For example, the natural language processor <NUM> can process natural language free-form input that is textual input that is a conversion, by STT module <NUM>, of audio data provided by a user via client device <NUM>. The generated annotated output may include one or more annotations of the natural language input and optionally one or more (e.g., all) of the terms of the natural language input.

In some implementations, the natural language processor <NUM> is configured to identify and annotate various types of grammatical information in natural language input. In some implementations, the natural language processor <NUM> may additionally and/or alternatively include an entity tagger (not depicted) configured to annotate entity references in one or more segments such as references to people (including, for instance, literary characters, celebrities, public figures, etc.), organizations, locations (real and imaginary), and so forth. In some implementations, the natural language processor <NUM> may additionally and/or alternatively include a coreference resolver (not depicted) configured to group, or "cluster," references to the same entity based on one or more contextual cues. For example, the coreference resolver may be utilized to resolve the term "there" to "Hypothetical Café" in the natural language input "I liked Hypothetical Café last time we ate there. " In some implementations, one or more components of the natural language processor <NUM> may rely on annotations from one or more other components of the natural language processor <NUM>. In some implementations, in processing a particular natural language input, one or more components of the natural language processor <NUM> may use related prior input and/or other related data outside of the particular natural language input to determine one or more annotations.

In some implementations, dialog state tracker <NUM> may be configured to keep track of a "dialog state" that includes, for instance, a belief state of a one or more users' goals (or "intents") over the course of a human-to-computer dialog session and/or across multiple dialog sessions. In determining a dialog state, some dialog state trackers may seek to determine, based on user and system utterances in a dialog session, the most likely value(s) for slot(s) that are instantiated in the dialog. Some techniques utilize a fixed ontology that defines a set of slots and the set of values associated with those slots. Some techniques additionally or alternatively may be tailored to individual slots and/or domains. For example, some techniques may require training a model for each slot type in each domain.

Dialog manager <NUM> may be configured to map a current dialog state, e.g., provided by dialog state tracker <NUM>, to one or more "responsive actions" of a plurality of candidate responsive actions that are then performed by automated assistant <NUM>. Responsive actions may come in a variety of forms, depending on the current dialog state. For example, initial and midstream dialog states that correspond to turns of a dialog session that occur prior to a last turn (e.g., when the ultimate user-desired task is performed) may be mapped to various responsive actions that include automated assistant <NUM> outputting additional natural language dialog. This responsive dialog may include, for instance, requests that the user provide parameters for some action (i.e., fill slots) that dialog state tracker <NUM> believes the user intends to perform. In some implementations, responsive actions may include actions such as "request" (e.g., seek parameters for slot filling), "offer" (e.g., suggest an action or course of action for the user), "select," "inform" (e.g., provide the user with requested information), "no match" (e.g., notify the user that the user's last input is not understood), a command to a peripheral device (e.g., to turn off a light bulb), and so forth.

Storage subsystem <NUM> stores programming and data constructs that provide the functionality of some or all of the modules described herein. For example, the storage subsystem <NUM> may include the logic to perform selected aspects of one or more of the processes of <FIG>, <FIG>, and/or <FIG>, as well as to implement various components depicted in <FIG> and/or <FIG>.

In some implementations, a method implemented by one or more processors is provided that includes receiving audio data that captures an utterance of a human speaker and that also captures one or more additional sounds that are not from the human speaker. The method further includes processing the audio data with a frequency transformation portion of an automatic speech recognition ("ASR") engine to generate a frequency representation of the audio data. The method further includes generating a revised version of the frequency representation that includes one or more portions of the frequency representation corresponding to the utterance and that excludes one or more other portions of the frequency representation that capture the one or more additional sounds, and wherein generating the revised version of the frequency representation includes processing, using a voice filter model, the frequency representation and a speaker embedding to generate a predicted mask that is conditioned on both the frequency representation and the speaker embedding, wherein the speaker embedding corresponds to the human speaker. The method further includes generating the revised version of the frequency representation by processing the frequency representation using the predicted mask. The method further includes processing the revised version of the frequency representation using a speech recognition portion of the ASR engine to generate a text representation of the utterance of the human speaker.

In some implementations, processing the audio data with the frequency transformation portion of the ASR engine to generate the frequency representation of the audio data further includes processing the audio data using the frequency transformation portion of the ASR engine to generate output. In some implementations, the method further includes processing the output using a power law compression process to generate the frequency representation.

In some implementations, prior to generating the revised version of the frequency representation, the method further includes normalizing the frequency representation using a normalization process and normalizing the speaker embedding using a normalization process.

In some implementations, processing, using the voice filter model, the frequency representation and the speaker embedding to generate the predicted mask that is conditioned on both the frequency representation and the speaker embedding further includes processing the frequency representation using a one dimensional convolutional neural network portion of the voice filter model to generate convolutional output. In some implementations, the method further includes processing the convolutional output and the speaker embedding using a unidirectional long short term memory model portion of the voice filter output to generate recurrent output. In some implementations, the method further includes processing the recurrent output using a feed-forward neural network portion of the voice filter model to generate the predicted mask.

In some implementations, generating the revised version of the frequency representation by processing the frequency representation using the predicted mask includes processing the frequency representation using the predicted mask by convolving the frequency representation using the predicted mask.

In some implementations, the speaker embedding is generated by processing one or more instances of speaker audio data corresponding to the human speaker using a speaker embedding model. In some versions of those implementations, the speaker audio data processed in generating the speaker embedding includes one or more enrollment utterances spoken by a human speaker during enrollment with a digital system. In some versions of those implementations, the speaker embedding is stored locally at a client device during the enrollment with the digital system, and wherein the speaker embedding is used in generating the revised frequency representation of the audio data.

In some implementations, the one or more additional sounds that are not from the human speaker captured in the received audio data comprises background noise not from an additional human speaker.

In some implementations, the one or more additional sounds that are not from the human speaker captured in the received audio data comprises an utterance of an additional human speaker.

In some implementations, the audio data is captured via one or more microphones of a client device, and wherein the speaker embedding corresponding to the human speaker is generated after at least part of the audio data is captured via the one or more microphones of the client device.

In some implementations, the audio data wherein the audio data is captured via one or more microphones of a client device, and wherein the speaker embedding corresponding to the human speaker is generated prior to the audio data being captured via the one or more microphones of the client device. In some versions of those implementations, the speaker embedding is selected based on sensor data captured at the client device indicating that the human speaker is currently interfacing with the client device.

In some implementations, a method of training a voice filter machine learning model to generate revised versions of frequency representations of audio data that isolate any utterances of a target human speaker is provided, the method implemented by one or more processors and including identifying an instance of audio data that includes spoken input from only a first human speaker. The method further includes identifying a speaker embedding for the first human speaker. The method further includes identifying an additional instance of audio data that lacks any spoken input from the first human speaker, and that includes spoken input from at least one additional human speaker. The method further includes generating a mixed instance of audio data that combines the instance of audio data and the additional instance of audio data. The method further includes processing the instance of audio data using a frequency transformation portion of an automatic speech recognition ("ASR") engine to generate a frequency representation of the instance of audio data. The method further includes processing the mixed instance of audio data using the frequency transformation portion of the ASR engine to generate a mixed frequency representation of the mixed audio data. The method further includes processing the mixed frequency representation and the speaker embedding using the voice filter model to generate a predicted mask. The method further includes processing the mixed frequency representation using the predicted mask to generate a predicted frequency representation. The method further includes determining an asymmetric loss based on comparison of the frequency representation and the predicted frequency representation. The method further includes updating one or more weights of the voice filter model based on the loss.

Claim 1:
A method implemented by one or more processors, the method comprising:
receiving audio data that captures an utterance of a human speaker and that also captures one or more additional sounds that are not from the human speaker;
processing the audio data with a frequency transformation portion of an automatic speech recognition, "ASR", engine (<NUM>) to generate a frequency representation (<NUM>) of the audio data;
generating a revised version of the frequency representation (<NUM>) that includes one or more portions of the frequency representation (<NUM>) corresponding to the utterance and that excludes one or more other portions of the frequency representation (<NUM>) that capture the one or more additional sounds, and wherein generating the revised version of the frequency representation (<NUM>) comprises:
processing, using a voice filter model (<NUM>), the frequency representation (<NUM>) and a speaker embedding (<NUM>) to generate a predicted mask (<NUM>) that is conditioned on both the frequency representation (<NUM>) and the speaker embedding (<NUM>), wherein the speaker embedding (<NUM>) corresponds to the human speaker;
generating the revised version of the frequency representation (<NUM>) by processing the frequency representation (<NUM>) using the predicted mask (<NUM>), wherein the processing comprises convolving (<NUM>) the frequency representation (<NUM>) with the predicted mask (<NUM>); and
processing the revised version of the frequency representation (<NUM>) using a speech recognition portion (<NUM>) of the ASR engine (<NUM>) to generate a text representation (<NUM>) of the utterance of the human speaker.