NEURAL NETWORK TRAINING FOR SPEECH ENHANCEMENT

A method of training neural networks may include receiving a sequence of audio frames, and mapping a first audio frame in the sequence of audio frames to a first output frame based on a neural network. The first output frame may represent a noise-invariant component of the first audio frame. The method may also include determining a first loss value based on differences between the first output frame and a first ground truth frame. The method may include mapping the first audio frame to a second output frame based on the neural network. The second output frame may represent a noise-variant component of the first audio frame. The method may further include determining a second loss value based on differences between the second output frame and a second ground truth frame, and updating the neural network based at least in part on the first and second loss values.

TECHNICAL FIELD

The present embodiments relate generally to neural networks, and specifically to training neural networks for speech enhancement.

BACKGROUND OF RELATED ART

Many machine learning approaches for enhancing speech in an audio signal are based on signal-to-noise ratio (SNR) information. These approaches often require a significant amount of training data to train a machine learning model (e.g., a neural network model) to account for large variations in noise. Consequently, once the machine model is trained, the machine learning model may be too large to be implemented on resource-constrained platforms, such as mobile devices, wearable technologies, or Internet of Things (IOT) devices. Moreover, many machine learning approaches for enhancing speech in an audio signal are not well-suited for far-field applications, where a speech source is located a distance from one or more audio capture devices.

SUMMARY

One innovative aspect of the subject matter of this disclosure can be implemented in a method of training neural networks. The method may include receiving a sequence of audio frames, and mapping a first audio frame in the sequence of audio frames to a first output frame based on a neural network. The first output frame may represent a noise-invariant component of the first audio frame. The method may further include determining a first loss value based on differences between the first output frame and a first ground truth frame. The method may also include mapping the first audio frame to a second output frame based on the neural network, where the second output frame represents a noise-variant component of the first audio frame. In addition, the method may include determining a second loss value based on differences between the second output frame and a second ground truth frame, and updating the neural network based at least in part on the first loss value and the second loss value.

Another innovative aspect of the subject matter of this disclosure can be implemented in a machine learning system including a processing system and a memory. The memory may store instructions that, when executed by the processing system, cause the machine learning system to receive a sequence of audio frames, and map a first audio frame in the sequence of audio frames to a first output frame based on a neural network. The first output frame may represent a noise-invariant component of the first audio frame. Execution of the instructions may further cause the machine learning system to determine a first loss value based on differences between the first output frame and a first ground truth frame, and map the first audio frame to a second output frame based on the neural network. The second output frame may represent a noise-variant component of the first audio frame. In addition, execution of the instructions may cause the machine learning system to determine a second loss value based on differences between the second output frame and a second ground truth frame, and update the neural network based at least in part on the first loss value and the second loss value.

Another innovative aspect of the subject matter of this disclosure can be implemented in a method of training neural networks. The method may include receiving a sequence of audio frames, and mapping a first audio frame in the sequence of audio frames to a first output frame based on a first neural network. The first output frame may represent a noise-invariant component of the first audio frame. The method may also include determining a first loss value based on differences between the first output frame and a first ground truth frame, and updating the first neural network based at least in part on the first loss value.

DETAILED DESCRIPTION

In the following description, numerous specific details are set forth such as examples of specific components, circuits, and processes to provide a thorough understanding of the present disclosure. The term “coupled” as used herein means connected directly to or connected through one or more intervening components or circuits. The terms “electronic system” and “electronic device” may be used interchangeably to refer to any system capable of electronically processing information. Also, in the following description and for purposes of explanation, specific nomenclature is set forth to provide a thorough understanding of the aspects of the disclosure. However, it will be apparent to one skilled in the art that these specific details may not be required to practice the example embodiments. In other instances, well-known circuits and devices are shown in block diagram form to avoid obscuring the present disclosure. Some portions of the detailed descriptions which follow are presented in terms of procedures, logic blocks, processing and other symbolic representations of operations on data bits within a computer memory.

These descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. In the present disclosure, a procedure, logic block, process, or the like, is conceived to be a self-consistent sequence of steps or instructions leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated in a computer system. It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities.

The various illustrative logical blocks, modules, circuits and instructions described in connection with the embodiments disclosed herein may be executed by one or more processors (or a processing system). The term “processor,” as used herein may refer to any general-purpose processor, special-purpose processor, conventional processor, controller, microcontroller, and/or state machine capable of executing scripts or instructions of one or more software programs stored in memory.

Aspects of the disclosure provide systems and methods for enhancing speech in an audio signal. In some embodiments, a neural network (also referred to as a “neural network algorithm” or “neural network model”) may be trained using multi-tasking to infer a clean representation of speech in an audio signal that includes speech and noise. In some embodiments, the neural network may be trained using a network structure that includes a backbone (e.g., an encoder) and multiple heads (e.g., decoders), where each head is trained to perform a different task. That is, one head may be trained to infer a speech (or noise-invariant) component of an audio signal, and the other head may be trained to infer a SNR-related (or noise-variant) component of the audio signal.

More generally, to train the neural network, a machine learning system may receive a sequence of audio frames. In some aspects, the sequence of audio frames may represent an audio signal that includes speech and noise. The machine learning system may also receive ground truth frames, which are reference frames of different components of the audio signal (or sequence of audio frames). In some embodiments, the machine learning system may receive a first ground truth frame that includes the speech (or noise-invariant) component of a first audio frame in the sequence of audio frames, with little or no noise. The machine learning system may also receive a second ground truth frame that includes a SNR-related (or noise-variant) component of the first audio frame.

In some embodiments, the machine learning system may map the first audio frame in the sequence of audio frames to a first output frame based on the neural network. In some aspects, the first output frame may represent an estimate of the speech (or noise-invariant) component of the first audio frame. The machine learning system may also determine a first loss value based on differences between the first output frame and the first ground truth frame. Further, the machine learning system may map the first audio frame in the sequence of audio frames to a second output frame based on the neural network. In some aspects, the second output frame may represent an estimate of a SNR-related (or noise-variant) component of the first audio frame. The machine learning system may further determine a second loss value based on differences between the second output frame and the second ground truth frame.

In some embodiments, the machine learning system may update the neural network based at least in part on the first and second loss values at, for example, the end of each iteration of the training process. In some aspects, the machine learning system may repeat the training process until certain convergence criteria are met. In some embodiments, the trained neural network model may be deployed in, for example, an audio capture and output system, and used to infer, in real time, enhanced representations of speech in audio frames. Unlike conventional techniques for speech enhancement, where a machine learning system is trained to model only the SNR of training data (often using a substantial amount of training data to account for large variations in noise), embodiments described herein use multi-tasking to train a machine learning system to model both a clean speech (or noise-invariant) component and SNR-related (or noise-variant) component of training data. Because variations in clean speech tend to be smaller than variations in noise, and because the embodiments described herein train a machine learning system to primarily model the clean speech component of training data, the techniques described herein may require less training data than conventional techniques for speech enhancement. Accordingly, a machine learning model trained using the systems and methods described herein may be smaller in size and better suited for resource-constrained platforms (e.g., mobile phones, wearable technologies, or IOT devices) relative to conventional machine learning models for speech enhancement.

FIG.1shows a block diagram of an example audio capture and output system100that may employ a neural network model, according to some embodiments. The system100includes an audio capture device110, an audio processor120, and an audio output device130. The audio capture device110(e.g., a microphone or other acoustic sensor) captures (or records) a sequence of audio frames of an audio signal101and converts the audio frames to digital audio capture data102(also referred to as “audio capture data102”). The audio output device130(e.g., wearable technology, a smartphone, or IoT device) may play the digital data by reproducing the captured audio signal using one or more speakers.

The audio processor120converts the audio capture data102to enhanced audio data103that, when played by the audio output device130, may reflect the original audio signal captured by the audio capture device110as audio with little or no noise. For example, the audio processor120may be configured to enhance speech included in the audio capture data102. Further, in some embodiments, the audio processor120may be configured to recognize speech in the enhanced audio data103. Although depicted as an independent block inFIG.1, in actual embodiments, the audio processor120may be incorporated or otherwise included in the audio capture device110, the audio output device130, or a combination thereof.

In some embodiments, the audio processor120may create enhanced representations of the audio capture data102(e.g., enhanced audio data103) based on a neural network model123that is trained through machine learning. Machine learning is a technique for improving the ability of a computer system or application to perform a certain task. During a training phase, a machine learning system may be provided with multiple “answers” and one or more sets of raw data to be mapped to each answer. For example, the machine learning system may be trained to perform enhancement operations on a sequence of audio frames by providing the system with a sequence of audio frames captured of an audio signal that includes speech and noise (which represents the raw data). The machine learning system may also be provided one or more representative (or ground truth) frames of the same speech that contain little or no noise (which represents part of the answer), and one or more representative (or ground truth) frames of SNR-related information based on same speech and noise (which represents the rest of the answer). The machine learning system may then analyze the raw data to “learn” a set of rules that can be used to reproduce the answers. For example, the machine learning system may perform statistical analysis on the raw data to determine a common set of features (also referred to as “rules”) related to speech and SNR-related information in the raw data.

In some aspects, the machine learning system may employ deep learning, which is a particular form of machine learning in which the model being trained is a multi-layer neural network. Deep learning architectures are often referred to as artificial neural networks due to the way in which information is processed (similar to a biological nervous system). For example, each layer of the deep learning architecture may be composed of a number of artificial neurons. The neurons may be interconnected across the various layers so that input data (or the raw data) may be passed from one layer to another. More specifically, each layer of neurons may perform a different type of transformation on the input data that will ultimately result in one or more desired outputs. The interconnected framework of neurons may be referred to as a neural network model. Thus, the neural network model123may include a set of rules that can be used to “infer” enhanced representations of the audio capture data102. As such, the audio processor120may use the neural network model123to enhance speech in the audio capture data102.

FIG.2shows a block diagram of an example machine learning system200, according to some embodiments. The machine learning system200includes a neural network222, a loss calculator224, and a network optimizer226. In some aspects, the machine learning system200may be used to train the neural network222to produce a neural network model223. The neural network model223may be an embodiment of the neural network model123ofFIG.1.

As shown inFIG.2, the machine learning system200may receive a sequence of input audio frames202. In some aspects, the input audio frames202may represent an audio signal that includes speech and noise. The machine learning system200may also receive a sequence of ground truth frames225A, where each ground truth frame225A includes the same speech as the audio signal, but little or no noise. The machine learning system200may further receive a sequence of ground truth frames225B, where each ground truth frame225B includes SNR-related information based on the same speech and noise as the audio signal. In some embodiments, the SNR-related information may represent a gain based on the total energy of speech in the audio signal relative to the total energy of the audio signal. The machine learning system200may also receive a sequence of input audio frames202′, where each input audio frame202′ includes the audio signal and/or information related to the audio signal.

In some embodiments, the neural network222may be configured to receive an input audio frame202and produce an output frame203A and an output frame203B, based on the received input audio frame202. In some aspects, the output frame203A may represent an estimate of the speech (or noise-invariant) component of the input audio frame202, and the output frame203B may represent an estimate of a SNR-related (or noise-variant) component of the input audio frame202. Further, in some aspects, the neural network222may produce enhanced representations of the sequence of input audio frames202by attempting to recreate the ground truth frames225A and225B.

In some embodiments, the neural network222may be a multi-head neural network (e.g., a neural network that includes multiple heads) configured to learn the characteristics of speech and SNR-related information in the input audio frames202. In some aspects, the neural network222may form a network of weighted connections across multiple layers of artificial neurons (e.g., layers in a backbone and multiple heads) that begin with the sequence of input audio frames202and lead to a sequence of output frames203A and a sequence of output frames203B. In some embodiments, the neural network222may be configured to provide each sequence of output frames203A and203B to the loss calculator224.

As shown inFIG.2, the loss calculator224may be configured to receive each of the output frames203A and203B from the neural network222. The loss calculator224may also be configured to receive each of the ground truth frames225A and225B, and each of the input audio frames202′. In some aspects, the loss calculator224may be configured to determine an amount of loss (or error) in each output frame203A relative to a respective ground truth frame225A. For example, to determine a first loss, the loss calculator224may compare an output frame203A (which represents an estimate of the speech or noise-invariant component of an input audio frame202) to a ground truth frame225A (which represents the speech or noise-invariant component of the same input audio frame202, but with little or no noise). The loss calculator224may also be configured to determine an amount of loss (or error) in each output frame203B relative to a respective ground truth frame225B. For example, to determine a second loss, the loss calculator224may compare an output frame203B (which represents an estimate of a SNR-related or noise-variant component of an input audio frame202) to a ground truth frame225B (which represents the SNR-related or noise-variant component of the same input audio frame202). The loss calculator224may also be configured to determine additional losses based on the output frames203A and203B, ground truth frames225A and225B, and the input audio frames202′. In some aspects, the loss calculator224may be configured to combine at least the first and second losses to determine a total loss204. The loss calculator224may be further configured to output the total loss204to the network optimizer226.

In some aspects, the network optimizer226may be configured to receive the total loss204and adjust one or more weights (also referred to as “parameters”)205of the neural network222based on the total loss204. More specifically, the network optimizer226may adjust the weights205in a manner that reduces the total loss204. The machine learning system200may repeat the training process described above over one or more iterations until certain convergence criteria are met. For example, responsive to each new sequence of audio frames provided as inputs to the neural network222, the loss calculator224may determine a total loss204based on outputs of the neural network222, and the network optimizer226may further update the weights205based on the total loss204.

FIG.3Ashows a block diagram of an example neural network300A, according to some embodiments. The neural network300A may be an embodiment of the neural network222ofFIG.2. As illustrated inFIG.3A, the neural network300A includes a backbone network327A, a noise-invariant head328A, and a noise-variant head329A. In some embodiments, the neural network300A may include more than one noise-variant head329A. Further, in some embodiments, each of the backbone network327A, the noise-invariant head328A, and the noise-variant head329A may represent a separate neural network. In some aspects, each of the backbone network327A, the noise-invariant head328A, and the noise-variant head329A may include multiple layers of artificial neurons (e.g., convolutional layers).

As shown inFIG.3A, the backbone network327A may be configured to receive a sequence of input audio frames302. In some aspects, the input audio frames302may represent an audio signal that includes speech and noise. Further, in some embodiments, the input audio frames302may include the audio signal and/or various features related to the audio signal.

In some embodiments, the audio signal may be represented in the frequency domain as a sequence of audio frames (e.g., the sequence of input audio frames302, the sequence of input audio frames202ofFIG.2and/or the sequence of input audio frames202′ ofFIG.2). In such embodiments, the audio signal may be expressed as X(l,k), where l represents the frame index and k represents the frequency index.

In some aspects, the audio signal X(l,k) may be expressed (or modeled) as follows:

In Equation 1, C(l,k) represents the speech component (also referred to as a “clean speech signal” or “speech signal”) of the audio signal X(l,k), and N(l,k) represents the noise component (also referred to as a “noise signal”) of the audio signal X(l,k). The frame index l may range from 0 to L−1, where L represents the total number of frames in the sequence of audio frames. The frequency index k may range from 0 to K−1, where K represents the total number of frequency bins. In some embodiments, the audio signal X(l,k), but not the clean speech signal C(l,k) (in isolation) or the noise signal N(l,k) (in isolation), may be identified (or represented) in the input audio frames302. Further, in some embodiments, the clean speech signal C(l,k) (and/or a variation of C(l,k) such as a normalized version of C(l,k)) may be included in a sequence of ground truth frames (e.g., the sequence of ground truth frames225A ofFIG.2). Moreover, in some embodiments, the clean speech signal C(l,k) may be determined by a loss calculator (e.g., the loss calculator224ofFIG.2) using, for example, (i) a normalized version of C(l,k), which may be included in a sequence of ground truth frames (e.g., the sequence of ground truth frames225A ofFIG.2); and (ii) the total energy of speech in the audio signal X(l,k), which may be included in or derived from a sequence of ground truth frames (e.g., the sequence of ground truth frames225B ofFIG.2).

In some aspects, each of the clean speech signal C(l,k) and the audio signal X(l,k) may be formulated as follows:

In Equation 2, |C(l,k)| represents the magnitude of the clean speech signal C(l,k), and Cθ(l,k) represents the phase of the clean speech signal C(l,k). In Equation 3, |X(l,k)| represents the magnitude of the audio signal X(l,k), and Xθ(l,k) represents the phase of the audio signal X(l,k). In some embodiments, Cθ(l,k) and Xθ(l,k) may be used to determine ρ(l,k), a SNR-related phase of X(l,k) (e.g., a phase based on a difference between Cθ(l,k) and Xθ(l,k)), which may be included in a sequence of ground truth frames (e.g., the sequence of ground truth frames225B ofFIG.2).

In some aspects, each of the clean speech signal C(l,k) and the audio signal X(l,k) may be normalized using the L1 norm as follows:

In Equation 4, Cn(l,k) represents the normalized clean speech signal C(l,k), and ec(l) represents the total energy of the clean speech signal C(l,k) at frame l. In Equation 5, Xn(l,k) represents the normalized audio signal X(l,k), and ex(l) represents the total energy of the audio signal X(l,k) at frame l. In some embodiments, Cn(l,k) may be included in a sequence of ground truth frames (e.g., the sequence of ground truth frames225A ofFIG.2). Further, in some embodiments, Xn(l,k) and ex(l) may be included in, for example, the sequence of input audio frames202′ ofFIG.2.

In some embodiments, ec(l) and ex(l) may be used to define a SNR-related gain (or ratio) α(l) of the audio signal X(l,k) as shown below:

In some embodiments, the SNR-related gain α(l) may be included in a sequence of ground truth frames (e.g., the sequence of ground truth frames225B ofFIG.2).

As described above, in some embodiments, the input audio frames302may include the audio signal X(l,k). In some other embodiments, in addition to or in lieu of X(l,k), the input audio frames302may include I(l,k), a feature which represents the log-normalized frequency spectrum magnitude (also referred to as the “log-normalized spectrum magnitude”) of the audio signal X(l,k), as shown below:

By training the neural network300A using I(l,k), the neural network300A may be better-suited (e.g., able to more precisely enhance speech) for far-field applications, where a speech source is located a distance from a microphone. In some embodiments, in addition to or as an alternative to X(l,k) and/or I(l,k), the input audio frames302may include any one or more of the following features:

In Equation 8, d(l,k) represents a first derivative of the log-normalized spectrum magnitude I(l,k). In Equation 9, dd(l,k) represents a second derivative of the log-normalized spectrum magnitude I(I,k), and in Equation 10, de(l) represents a change in the log domain of ex(l), the total energy of the audio signal X(l,k) for a frame (l). In some embodiments, two or more of the features, I(l,k), d(l,k), dd(l,k), and de(l), may be concatenated and input to the backbone network327A as the input audio frames302. By training the neural network300A using two or more of the features, I(l,k), d(l,k), dd(l,k), and de(l), the neural network300A may be better suited for far-field applications relative to using just one of the features.

Upon receiving the input audio frames302, the backbone network327A may analyze the input audio frames302using multiple layers of artificial neurons. In some embodiments, the backbone network327A may extract features (e.g., rules) from the input audio frames302, and output the extracted features as, for example, a feature map to the noise-invariant head328A and the noise-variant head329A.

As shown inFIG.3A, the noise-invariant head328A may use the feature map to generate the output frames303A, which may include a normalized clean speech frequency spectrum magnitude (also referred to as a “normalized clean speech spectrum magnitude”) Ĉn(l,k). The normalized clean speech spectrum magnitude Ĉn(l,k) may represent an estimate of Cn(l,k), the normalized clean speech signal (defined above in Equation 4).

In some aspects, when the noise-invariant head328A analyzes a frame l that contains only speech, the sum of values associated with the output frames303A is 1, as shown below:

In contrast, when the noise-invariant head328A analyzes a frame l that includes only noise, the sum of the values associated with the output frames303A is 0, as shown below:

In some embodiments, the noise-invariant head328A may include a softmax layer that generates K+1 outputs (e.g., Ĉn(l,0), . . . , Ĉn(l,K−1), and Ĉn(l,K), as shown inFIG.3A), where the additional output (the +1 output) is used to detect noise-only frames.

As shown inFIG.3A, the noise-variant head329A may use the feature map from the backbone network327A to generate output frames303B, which include a SNR-related gain {circumflex over (α)}(l). The SNR-related gain {circumflex over (α)}(l) may represent an estimate of the SNR-related gain α(l) (defined above in Equation 6). In some embodiments, the noise-variant head329A may include a sigmoid function, and output values of {circumflex over (α)}(l) between, for example, 0 and 1.

In some aspects, the neural network300A may be based on frequency spectrum magnitude (also referred to as “spectrum magnitude”), but not frequency spectrum phase (also referred to as “spectrum phase”). That is, the neural network300A (e.g., the backbone network327A, noise-invariant head328A, and noise-variant head329A) may analyze the spectrum magnitude (e.g. |X(l,k)|), but not the spectrum phase (e.g., Xθ(l,k)), of the input audio frames302. Consequently, the neural network300A may output Ĉn(l,k), which is based on spectrum magnitude, and {circumflex over (α)}(l), which is also based on spectrum magnitude. Accordingly, the neural network300A may be configured to generate K+2 outputs for each frame l (e.g., Ĉn(l,0), . . . , Ĉn(l,K−1), Ĉn(l,K), and {circumflex over (α)}(l), as shown inFIG.3A. Once the neural network300A has generated the K+2 outputs for each frame l (or output frames303A and303B), the neural network300A may provide the output frames303A and303B to a loss calculator, such as the loss calculator224described above with respect toFIG.2.

FIG.3Bshows a block diagram of an example neural network300B, according to some embodiments. The neural network300B may be an embodiment of the neural network222and/or300A ofFIGS.2and3A, respectively. As illustrated inFIG.3B, the neural network300B includes a backbone network327B, a noise-invariant head328B, and a noise-variant head329B. In some embodiments, the neural network300B may include more than one noise-variant head329B. Further, in some embodiments, each of the backbone327B, the noise-invariant head328B, and the noise-variant head329B may represent a separate neural network. Further, in some aspects, each of the backbone network327B, the noise-invariant head328B, and the noise-variant head329B may include multiple layers of artificial neurons (e.g., convolutional layers).

As shown inFIG.3B, the backbone network327B may be configured to receive a sequence of input audio frames302. As discussed above with respect toFIG.3A, the input audio frames302may represent the audio signal X(l,k), which includes speech and noise. In some embodiments, the input audio frames302may include X(l,k) and/or various features of X(l,k). With reference to Equation 1 above, the audio signal X(l,k) may be modeled as the sum of a clean speech signal C(l,k) and a noise signal N(l,k); and with reference to Equations 2 and 3 above, the audio signal X(l,k) may be formulated to include a magnitude |X(l,k)| and phase Xθ(l,k), and the clean speech signal C(l,k) may be formulated to include a magnitude |C(l,k)| and phase Cθ(l,k). In some embodiments, the phases, Xθ(l,k) and Cθ(l), and the SNR-related gain α(l,k) (defined above in Equation 6), may be used to define the following expressions for the audio signal X(l,k):

In Equation 13, ρ(l,k) may represent a SNR-related phase of the audio signal X(l,k), and range from 0 to 1. In Equation 14, β(l,k) may represent a SNR-related phase-sensitive gain (also referred to as “phase sensitive gain”) of the audio signal X(l,k). In some embodiments, (i) α(l), (ii) α(l) and ρ(l,k), and/or (iii) β(l,k) may be included in a sequence of ground truth frames (e.g., the sequence of ground truth frames225B ofFIG.2).

Upon receiving the input audio frames302, the backbone network327B may analyze the input audio frames302using multiple layers of artificial neurons. In some embodiments, the backbone network327B may extract features from the input audio frames302, and output the extracted features as, for example, a feature map to the noise-invariant head328B and the noise-variant head329B.

The noise-invariant head328B may use the feature map to generate output frames303A′. In some embodiments, the output frames303A′ may include normalized clean speech spectrum magnitude Ĉn(l,k), which may represent an estimate of the normalized clean speech signal Cn(l,k) (defined above in Equation 4). In some embodiments, the noise-invariant head328B may include a softmax layer that generates K+1 outputs (e.g., Ĉn(l,0), . . . , Ĉn(l,K−1), and Ĉn(l,K), as shown inFIG.3B), where the additional output (the +1 output) is used to detect noise-only frames.

As shown inFIG.3B, the noise-variant head329B may use the feature map to generate output frames303B′. In some embodiments, the output frames303B′ may include phase-sensitive gain {circumflex over (β)}(l,k), which may represent an estimate of the phase-sensitive gain β(l,k) (defined above in Equation 14). In such embodiments, the noise-variant head329B may generate K outputs (e.g., {circumflex over (β)}(l,0), . . . , {circumflex over (β)}(l,K−2), {circumflex over (β)}(l,K−1)), as shown inFIG.3B.

In some other embodiments, the output frames303B′ may include SNR-related phase {circumflex over (ρ)}(l,k) and SNR-related gain {circumflex over (α)}(l). The SNR-related phase {circumflex over (ρ)}(l,k) may represent an estimate of the SNR-related phase ρ(l,k) (defined above in Equation 13). The SNR-related gain {circumflex over (α)}(l) may represent an estimate of the SNR-related gain α(l) (defined above in Equation 6). In some embodiments, the noise-variant head329B may include a sigmoid function, and output values of {circumflex over (α)}(l) between, for example, 0 and 1. In embodiments where the output frames303B′ include {circumflex over (ρ)}(l,k) and {circumflex over (α)}(l), the noise-variant head329B may generate K+1 outputs for a frame l (e.g., {circumflex over (α)}(l), {circumflex over (ρ)}(l,0), . . . , {circumflex over (ρ)}(l,K−2), {circumflex over (ρ)}(l,K−1)).

In some aspects, the neural network300B may be based on both spectrum magnitude and spectrum phase (unlike the neural network300A ofFIG.3A, which may be based on spectrum magnitude but not spectrum phase). That is, the neural network300B (e.g., the backbone network327B, noise-invariant head328B and noise-variant head329B) may analyze both the spectrum magnitude (e.g. |X(l,k)|) and the spectrum phase (e.g., Xθ(l,k)) of the input audio frames302. Consequently, the neural network300B may output Ĉn(l,k), which is based on spectrum magnitude, and either (i) {circumflex over (β)}(l,k), which is based on spectrum phase and spectrum magnitude, or (ii) {circumflex over (ρ)}(l,k), which is based on spectrum phase and {circumflex over (α)}(l), which is based on spectrum magnitude.

In embodiments where the noise-variant head329B is configured to generate {circumflex over (β)}(l,k), the neural network300B may generate 2K+1 outputs (e.g., Ĉn(l,0), . . . , Ĉn(l,K−1), Ĉn(l,K), and {circumflex over (β)}(l,0), . . . , {circumflex over (β)}(l,K−2), {circumflex over (β)}(l,K−1)), as shown inFIG.3B. In other embodiments where the noise-variant head329B is configured to generate {circumflex over (ρ)}(l,k) and {circumflex over (α)}(l), the neural network300B may generate 2K+2 outputs (e.g., Ĉn(l,0), . . . , Ĉn(l,K−1), Ĉn(l,K), {circumflex over (α)}(l), and {circumflex over (ρ)}(l,0), . . . , {circumflex over (ρ)}(l,K−2), {circumflex over (β)}(l,K−1)). Once the neural network300B has generated the outputs (or output frames303A′ and303B′), the neural network300B may provide the output frames303A′ and303B′ to a loss calculator, such as the loss calculator224described above with respect toFIG.2.

FIG.4shows a block diagram of an example loss calculator400, according to some embodiments. The loss calculator400may be an embodiment of the loss calculator224ofFIG.2. As shown inFIG.4, the loss calculator400includes a noise-variant loss calculator432, a noise-invariant loss calculator434, a joint loss calculator436, and a total loss calculator438.

The Noise-Variant Loss Calculator

The noise-variant loss calculator432may be configured to receive a sequence of noise-variant output frames403B from a neural network, such as the neural network222,300A, or300B ofFIGS.2,3A and3B, respectively. In some embodiments, the noise-variant output frames403B may be an embodiment of the output frames303B ofFIG.3A, and each noise-variant output frame403B may include SNR-related gain {circumflex over (α)}(l). As shown inFIG.4, the noise-variant loss calculator432may also be configured to receive a sequence of noise-variant ground truth frames425B. In some embodiments, the noise-variant ground truth frames425B may be an embodiment of the ground truth frames225B ofFIG.2, and each noise-variant ground truth frame425B may include SNR-related gain α(l) (as defined above in Equation 6).

In some embodiments, the noise-variant loss calculator432may be configured to use the noise-variant output frames403B ({circumflex over (α)}(l)), and noise-variant ground truth frames425B (α(l)) to determine a noise-variant loss433(also referred to as “LNV”) as follows:

In Equation 15.1, the noise-variant loss LNVis a mean squared error (MSE) loss that represents an amount of error in the noise-variant output frames403B ({circumflex over (α)}(l)) relative to the noise-variant ground truth frames425B (α(l)). wv(l) is a frame-based weight that may be set to a value of 1 or ex(l) (described above with reference to Equation 5). p may be set to a value of 1 or 2.

In some other embodiments, the noise-variant loss calculator432may be configured to determine the noise-variant loss433(LNV) as follows:

In Equation 15.2, the noise-variant loss LNVis a Kullback-Leibler (KL) loss that represents an amount of error in the noise-variant output frames403B ({circumflex over (α)}(l)) relative to the noise-variant ground truth frames425B (α(l)). wv(l) is a frame-based weight that may be set to a value of 1 or ex(l) (described above with reference to Equation 5). ε is a parameter used for numerical stability, and is based on the smallest value that may be handled by hardware and software. In some embodiments, ε may be set to 1×10−12. Further, α0(l) represents the difference between 1 and {circumflex over (α)}(l); and {circumflex over (α)}0(l) represents the difference between 1 and {circumflex over (α)}(l).

In some other embodiments, the noise-variant output frames403B may be an embodiment of the output frames303B′ ofFIG.3Band include {circumflex over (β)}(l,k). In such embodiments, the noise-variant ground truth frames425B may be an embodiment of the ground truth frames225B ofFIG.2and include {circumflex over (β)}(l,k) (defined above in Equation 14). Further, the noise-variant loss calculator432may be configured to use the noise-variant output frames403B (e.g., {circumflex over (β)}(l,k)), and noise-variant ground truth frames425B (β(l,k)) to determine a noise-variant loss433(LNV) as follows:

In Equation 15.3, the noise-variant loss LNVis a MSE loss that represents an amount of error in the noise-variant output frames403B ({circumflex over (β)}(l,k)) relative to the noise-variant ground truth frames425B (β(l,k)). wv1(l,k) is a frame and frequency-based weight that may set to a value of 1 or (λi+ex(l))*(Xn(l,k)), where λiis a hyperparameter that may be used for tuning and set to a value greater than or equal to 0, ex(l) represents a total energy of the audio signal X(l,k) for a frame l (as discussed above with reference to Equation 5), and Xn(l,k) represents the normalized audio signal X(l,k) (defined above in Equation 5). p may be set to a value of 1 or 2.

In some other embodiments, the noise-variant loss calculator432may be configured to determine the noise-variant loss433(LNV) as follows:

In Equation 15.4, the noise-variant loss LNVis a KL loss that represents an amount of error in the noise-variant output frames403B ({circumflex over (β)}(l,k)) relative to the noise-variant ground truth frames425B (β(l,k)). wv1(l,k) is a frame and frequency-based weight that may set to a value of 1 or (λi+ex(l))*(Xn(l,k)), as described above with respect to Equation 15.3. ε is a parameter that may be set to 1×10−12, as described above with respect to Equation 15.2. Further, β0(l,k) represents the difference between 1 and β(l,k); and {circumflex over (β)}0(l,k) represents the difference between 1 and {circumflex over (β)}(l,k).

The Noise-Invariant Loss Calculator

The noise-invariant loss calculator434may be configured to receive a sequence of noise-invariant output frames403A from a neural network, such as the neural network222,300A, or300B ofFIGS.2,3A, and3B, respectively. In some embodiments, the noise-invariant output frames403A may be an embodiment of the output frames303A or303A′ ofFIGS.3A and3B, respectively, and include Ĉn(l,k). As shown inFIG.4, the noise-invariant loss calculator434may be configured to receive a sequence of noise-invariant ground truth frames425A. In some embodiments, the noise-invariant ground truth frames425A may be an embodiment of the ground truth frames225A ofFIG.2and include Cn(l,k) (defined above in Equation 4).

In some embodiments, the noise-invariant loss calculator434may be configured to use the noise-invariant output frames403A (Ĉn(l,k)), and noise-invariant ground truth frames425A (Cn(l,k)) to determine a noise-invariant loss (also referred to as “LNI”)435as follows:

In Equation 16.1, the noise-invariant loss LNIis a MSE loss that represents an amount of error in the noise-invariant output frames403A (Ĉn(l,k)) relative to the noise-invariant ground truth frames425A (Cn(l,k)). p may be set to a value of 1 or 2.

In some other embodiments, the noise-invariant loss calculator434may be configured to determine the noise-invariant loss435(LNI) as follows:

In Equation 16.2, the noise-invariant loss LNIis a KL loss that represents an amount of error in the noise-invariant output frames403A (Ĉn(l,k)) relative to the noise-invariant ground truth frames425A (Cn(l,k)). E is a parameter that may be set to 1×10−12, as described above with respect to Equation 15.2.

Joint Loss Calculator

In some embodiments, the joint loss calculator436may be configured to receive the noise-invariant output frames403A (e.g., Ĉn(l,k)) and the noise-variant output frames403B (e.g., {circumflex over (α)}(l,k); {circumflex over (α)}(l,k) and {circumflex over (ρ)}(l,k); or {circumflex over (β)}(l,k)). The joint loss calculator436may also be configured to receive the noise-invariant ground truth frames425A (e.g., C(l,k)) and the noise-variant ground truth frames425B (e.g., α(l); α(l) and ρ(l,k); or β(l,k)). Moreover, in some embodiments, the joint loss calculator436may be configured to receive a sequence of input audio frames402′, which may be an embodiment of the input audio frames202′ ofFIG.2, and include, for example, X(l,k), Xn(l,k), and/or ex(l). Further, in some embodiments, the joint loss calculator436may be configured to use the noise-invariant output frames403A, the noise-variant output frames403B, the noise-invariant ground truth frames425A, the noise-variant ground truth frames425B, and the input audio frames402′, to determine an estimated gain Ĝ(l,k) and ground truth gain G(l,k), and in turn, a joint loss437.

In some embodiments, the joint loss calculator436may be configured to use the noise-invariant output frames403A (Ĉn(l,k)), the noise-variant output frames403B ({circumflex over (α)}(l,k)), and the input audio frames402′ (e.g., (Xn(l,k)) to determine an estimated gain Ĝ(l,k), as follows:

In such embodiments, the joint loss calculator436may also be configured to use the noise-invariant ground truth frames425A (C(l,k)) and input audio frames402′ (X(l,k)) to determine a ground truth gain G(l,k), as follows:

In some aspects, the estimated gain Ĝ(l,k) and ground truth gain G(l,k) defined above in Equations 17.1 and 18.1, respectively, are each based on spectrum magnitude, and may each take on values between 0 and 1.

In some other embodiments, the joint loss calculator436may be configured to use the noise-invariant output frames403A (Ĉn(l,k)), the noise-variant output frames403B ({circumflex over (α)}(l,k) and {circumflex over (ρ)}(l,k)), and the input audio frames402′ (e.g., (Xn(l,k)) to determine an estimated gain Ĝ(l,k), as follows:

In such embodiments, the joint loss calculator436may further be configured to use the noise-invariant ground truth frames425A (C(l,k))), the noise-variant ground truth frames425B (ρ(l,k)), and input audio frames402′ (X(l,k)) to determine a ground truth gain G(l,k), as follows:

In some aspects, the estimated gain Ĝ(l,k) and ground truth gain G(l,k) defined above in Equations 17.2 and 18.2, respectively, are each based on spectrum magnitude and spectrum phase, and may each take on values between 0 and 1.

Once the joint loss calculator436has determined Ĉ(l,k) and G(l,k) (using Equations 17.1 and 18.1, respectively; or Equations 17.2 and 18.2, respectively), the joint loss calculator436may determine the difference between G(l,k) and Ĉ(l,k) (also referred to as the gain difference “δ(l,k)”), as follows:

In some embodiments, to determine the gain difference δ(l,k) in Equation 19, the joint loss calculator436may use Ĝ(l,k) and G(l,k), as defined above in Equations 17.1 and 18.1, respectively; or in Equations 17.2 and 18.2, respectively.

In some embodiments, once the joint loss calculator436has determined the gain difference δ(l,k) using Equation 19, the joint loss calculator436may normalize the gain difference δ(l,k) using the L2 norm to determine a gain difference loss (also referred to as “Δ(l,k)”), as follows:

In some aspects, the gain difference loss Δ(l,k), as defined in Equation 20.1, is related to MSE loss, and represents a loss that is symmetric (e.g., a loss in which the amount of distortion introduced to speech by a neural network, or the amount of noise suppression resulting from a neural network, is symmetric).

In some other embodiments, the joint loss calculator436may normalize the gain difference δ(l,k) using the L1 norm to determine the gain difference loss Δ(l,k), as follows:

In some aspects, the gain difference loss Δ(l,k), as defined in Equation 20.2, is related to MSE loss and symmetric.

In some embodiments, the joint loss calculator436may determine a gain difference loss Δ(l,k) that is related to MSE loss, and non-symmetric (e.g., a loss in which the amount of distortion introduced to speech by a neural network, or the amount of noise suppression resulting from a neural network, is not symmetric), when calculated as follows:

In Equation 20.3, γmserepresents a hyperparameter that may be tuned to emphasize (or de-emphasize) speech distortion, and may be set to a value greater than or equal to 1. In some embodiments, to determine the gain difference loss Δ(l,k) using Equation 20.3, the joint loss calculator436may use Ĝ(l,k) and G(l,k), as defined above in Equations 17.1 and 18.1, respectively, or as defined above in Equations 17.2 and 18.2, respectively.

Further, in some other embodiments, the joint loss calculator436may calculate a gain difference loss Δ(l,k) that is related to KL loss and symmetric, as follows:

In some embodiments, to determine the gain difference loss Δ(l,k) using Equation 20.4, the joint loss calculator436may use Ĝ(l,k) and G(l,k), as defined above in Equations 17.1 and 18.1, respectively. In such embodiments, the joint loss calculator436may calculate G0(l,k) as the difference between 1 and G(l,k) (defined above in Equation 18.1), and the joint loss calculator436may calculate Ĝ0(l,k) as the difference between 1 and Ĝ(l,k) (defined above in Equation 17.1). In some other embodiments, to determine the gain difference loss Δ(l,k) using Equation 20.4, the joint loss calculator436may use Ĝ(l,k) and G(l,k) as defined above in Equations 17.2 and 18.2, respectively. In such embodiments, the joint loss calculator436may calculate G0(l,k) as the difference between 1 and G(l,k) (defined above in Equation 18.2), and the joint loss calculator436may calculate Ĝ0(l,k) as the difference between 1 and Ĝ(l,k) (defined above in Equation 17.2). Further, in some embodiments, εmay be set to 1×10−12(as described above with respect to Equation 15.2).

In some other embodiments, the joint loss calculator436may calculate a gain difference loss Δ(l,k) that is related to KL loss and non-symmetric, as follows:

In some embodiments, to determine the gain difference loss Δ(l,k) of Equation 20.5, the joint loss calculator436may use G(l,k), as defined above in Equation 18.1. In such embodiments, the joint loss calculator436may calculate G0(l,k) as the difference between 1 and G(l,k) (defined above in Equation 18.1), and the joint loss calculator436may calculate Ĝ0(l,k) as the difference between 1 and Ĝ(l,k) (defined above in Equation 17.1). In some other embodiments, to determine the gain difference loss Δ(l,k) of Equation 20.5, the joint loss calculator436may use G(l,k) as defined above in Equation 18.2. In such embodiments, the joint loss calculator436may calculate G0(l,k) as the difference between 1 and G(l,k) (defined above in Equation 18.2), and the joint loss calculator436may calculate Ĝ0(l,k) as the difference between 1 and Ĝ(l,k) (defined above in Equation 17.2). Further, in some embodiments, ε may be set to 1×10−12(as described above with respect to Equation 15.2), and σ(l,k) may be determined as follows:

In Equation 20.5.1, γKLrepresents a hyperparameter that may be tuned to emphasize (or de-emphasize) speech distortion, and may be set to a value less than or equal to 1. ε may be set to 1×10−12, as described above with respect to Equation 15.2. Further, in some embodiments, the joint loss calculator436may determine Ĝ(l,k) and G(l,k) using Equations 17.1 and 18.1, respectively; and in some other embodiments, the joint loss calculator436may determine Ĝ(l,k) and G(l,k) using Equations 17.2 and 18.2, respectively.

Once the joint loss calculator436has determined the gain difference loss Δ(l,k) using any of Equations 20.1-20.5, the joint loss calculator436may determine a weight to apply to the gain difference loss Δ(l,k). For example, in some embodiments, the joint loss calculator436may determine a weight w(l,k) using the input audio frames402′ (e.g., X(l,k) and Xn(l,k)) as follows:

In Equation 26.1, λjis a hyperparameter that may be used for tuning.

Further, in some embodiments, the joint loss calculator436may determine the weight w(l,k) using the input audio frames402′ (e.g., |X(l,k)|, Xn(l,k), and ex(l)) and the hyperparameter λjas follows:

In some aspects, the weight w(l,k) defined above in Equation 26.1 is equivalent to the weight w(l,k) defined above in Equation 26.2. That is, w(l,k)=(λjXn(l,k)+|X(l,k)|)=(λj+ex(l))Xn(l,k).

Further, in some embodiments, the joint loss calculator436may set the weight w(l,k) to a value of 1. Once the joint loss calculator436has determined the weight w(l,k) using Equation 26.1 or 26.2, or by setting w(l,k) to a value of 1, the joint loss calculator436may apply the weight w(l,k) to the gain difference loss Δ(l,k) to determine a weighted gain difference loss (Δ(l,k)) as follows:

After determining the weighted gain difference lossΔ(l,k), the joint loss calculator436may calculate the joint loss437(also referred to as “Ljoint”) by summing the weighted gain difference lossΔ(l,k) across each of the frequency bins and frames, as follows:

In some embodiments, the joint loss calculator436may apply a filterbank (e.g., a Mel, equivalent rectangular bandwidth, or Gammatone filterbank) to the weighted gain difference lossΔ(Lk), where the filterbank may be represented as a matrix M with dimensions of Q bands×K frequency bins, as follows:

In Equation 27.2, q represents a band index and ranges from 0 to Q, and k represents a frequency index and ranges from 0 to K−1. After applying the filterbank M to the weighted gain difference lossΔ(l,k), the joint loss calculator436may calculate the joint loss437as follows:

Total Loss Calculator

In some embodiments, the total loss calculator438may receive each of the noise-variant loss433(LNV), the noise-invariant loss435(LNI), and the joint loss437(Ljoint), as shown inFIG.4. Further, the total loss calculator438may combine each of the noise-variant loss433, the noise-invariant loss435, and the joint loss437to determine a total loss439(Ltotal) as follows:

In Equation 29, Lindividualrepresents each of the noise-variant loss433and the noise-invariant loss435, and λtrepresents a hyperparameter that may be used to balance Ljointand Lindividual. It is noted that each of the total loss439, the joint loss437, the noise-variant loss433, and the noise-invariant loss435may be defined differently from the embodiments described herein.

Once the total loss439(Ltotal) is determined using Equation 29, a network optimizer (such as the network optimizer226ofFIG.2) may use the total loss439to determine whether certain convergence criteria are met. For example, where the total loss439falls below a threshold level, and/or where a predetermined number of training iterations has been completed, the network optimizer may determine that a neural network (e.g., the neural network222ofFIG.2, or the neural network300A or300B ofFIGS.3A and3B, respectively) is optimized. Accordingly, the network optimizer may not update the weights (e.g., the weights205ofFIG.2) of the neural network. As another example, where the total loss439is above a threshold level, and/or where a predetermined number of training iterations has not yet been completed, the network optimizer may determine that the neural network is not yet optimized. Thus, the network optimizer may determine one or more weights that minimize the total loss439in order to update the weights of the neural network.

FIG.5shows an example machine learning system500, according to some embodiments. In some embodiments, the machine learning system500may be one example of the machine learning system200ofFIG.2. Thus, the machine learning system500may be configured to produce a neural network model523based on a sequence of input audio frames502, a sequence of input audio frames502′, a sequence of ground truth frames525A and a sequence of ground truth frames525B. In some aspects, the input audio frames502may be an embodiment of the input audio frames202ofFIG.2and/or the input audio frames302ofFIGS.3A and3B; and the ground truth frames525A and525B may be embodiments of the ground truth frames225A and225B, respectively, ofFIG.2, or the ground truth frames425A and425B, respectively, ofFIG.4. In some embodiments, the machine learning system500may include a processing system540and a memory550.

The memory550may include a non-transitory computer-readable medium (including one or more nonvolatile memory elements, such as EPROM, EEPROM, Flash memory, a hard drive, and the like) that may store at least the following software (SW) modules:a neural network SW module551configured to create representations of the noise-invariant and noise-variant components of the input audio frames502(e.g., a sequence of noise-invariant output frames and a sequence of noise-variant output frames) by attempting to recreate the ground truth frames525A and the ground truth frames525B, to train the neural network model523, the neural network SW module551including:a backbone sub-module552to determine features (or a feature map) of the input audio frames502;a noise-invariant head sub-module553to determine an estimate of the noise-invariant component of the input audio frames502based on the feature map; anda noise-variant head sub-module554to determine an estimate of the noise-variant component of the input audio frames502based on the feature map.a loss calculator SW module555configured to determine a total amount of loss based on the sequence of noise-invariant output frames, the sequence of noise-variant output frames, the ground truth frames525A and525B, and the input audio frames502′, the loss calculator SW module555further including:a noise-invariant loss sub-module556to determine a noise-invariant loss based on the sequence of noise-invariant output frames and the ground truth frames525A;a noise-variant loss sub-module557to determine a noise-variant loss based on the sequence of noise-variant output frames and the ground truth frames525B;a joint loss sub-module558to determine a joint loss based on the sequence of noise-invariant output frames, the sequence of noise-variant output frames, the ground truth frames525A and525B, and the input audio frames502′; anda total loss sub-module559to determine a total loss based on the joint loss, the noise-invariant loss, and the noise-variant loss; anda network optimizer SW module560configured to determine one or more updated weights of the neural network SW module551based on the total loss.

Each software module includes instructions that, when executed by the processing system540, cause the machine learning system500to perform the corresponding functions.

The processing system540may include any suitable one or more processors capable of executing scripts or instructions of one or more software programs stored in the machine learning system500(such as in memory550). For example, the processing system540may execute the neural network SW module551to create representations of speech (e.g., a clean speech signal) and SNR-related information of the sequence of input audio frames502by attempting to recreate the ground truth frames525A and525B, respectively. Put differently, the processing system540may execute the neural network SW module551to map the sequence of input audio frames502to both a sequence of noise-invariant output frames and a sequence of noise-variant output frames.

In executing the neural network SW module551, the processing system540may further execute the backbone sub-module552, the noise-invariant head sub-module553, and the noise-variant head sub-module554. For example, the processing system540may execute the backbone sub-module552to determine a feature map of the input audio frames502. The processing system540may also execute the noise-invariant head sub-module553to determine the sequence of noise-invariant output frames based on the feature map. The processing system540may further execute the noise-variant head sub-module554to determine the sequence of noise-variant output frames based on the feature map.

The processing system540may also execute the loss calculator SW module555to determine a total loss. In executing the loss calculator SW module555, the processing system540may further execute the noise-invariant loss sub-module556, the noise-variant loss sub-module557, the joint loss sub-module558, and the total loss sub-module559. For example, the processing system540may execute the noise-invariant loss sub-module556to determine a noise-invariant loss. The processing system540may also execute the noise-variant loss sub-module557to determine a noise-variant loss. Further, the processing system540may execute the joint loss sub-module558to determine a joint loss. The processing system540may also execute the total loss sub-module559to determine a total loss. In some embodiments, the processing system540may further execute the network optimizer SW module560to determine updated weights (or parameters) of the neural network SW module551based on the total loss.

FIG.6shows an illustrative flowchart depicting an example operation600for training neural networks, according to some embodiments. The example operation600may be performed by a machine learning system (such as the machine learning system200and/or500ofFIGS.2and5, respectively) to train a neural network to infer representations of speech (e.g., a clean speech signal) and SNR-related information in a sequence of audio frames.

As shown inFIG.6, the machine learning system may receive a sequence of audio frames (610). In some embodiments, the sequence of audio frames may represent, in the frequency domain, an audio signal that includes a noise-invariant component and a noise-variant component. The machine learning system may map a first audio frame in the sequence of audio frames to a first output frame based on a neural network, where the first output frame may represent the noise-invariant component of the first audio frame (620). In some embodiments, the first audio frame may include multiple sub-frames, where each of the sub-frames is associated with a respective frequency bin. Further, in some embodiments, the first audio frame may be normalized based on a spectrum magnitude.

The machine learning system may further determine a first loss value based on differences between the first output frame and a first ground truth frame (630). In some embodiments, the first ground truth frame may represent the noise-invariant component of the first audio frame.

Further, the machine learning system may map the first audio frame to a second output frame based on the neural network, where the second output frame may represent a noise-variant component of the first audio frame (640). In some embodiments, the mapping of the first audio frame to the second output frame may be further based on a phase and/or a magnitude of the first audio frame. Further, in some embodiments, the second output frame may represent a ratio of an amount of energy in the noise-invariant component of the first audio frame relative to an amount of energy of the first audio frame (e.g., a SNR-related gain). Further, in such embodiments, the mapping of the first audio frame to the second output frame may be further based on a phase of the first audio frame.

The machine learning system may further determine a second loss value based on differences between the second output frame and a second ground truth frame (650). In some embodiments, the second ground truth frame may represent the noise-variant component of the first audio frame.

The machine learning system may also update the neural network based at least in part on the first loss value and the second loss value (660). In some embodiments, updating the neural network may include summing at least the first and second loss values to determine a total loss value. Further, in some embodiments, updating the neural network may further include determining a set of parameters for the neural network that minimizes the total loss value.

FIG.7shows an illustrative flowchart depicting an example operation700for training neural networks, according to some embodiments. The example operation700may be performed by a machine learning system (such as the machine learning system200and/or500ofFIGS.2and5, respectively) to train neural networks to infer representations of speech (e.g., a clean speech signal) and SNR-related information in a sequence of audio frames. In some embodiments, the machine learning system may include a neural network that represents an encoder, and two neural networks that each represent a decoder. In some other embodiments, the machine learning system may include two neural networks, where each of the two neural networks represents an encoder and decoder.

As shown inFIG.7, the machine learning system may receive a sequence of audio frames (710). In some embodiments, the sequence of audio frames may be a frequency-domain representation of an audio signal that includes a noise-invariant component and a noise-variant component. In some other embodiments, the sequence of audio input frames may include a feature map of an audio signal that includes a noise-invariant component and noise-variant component. The machine learning system may map a first audio frame in the sequence of audio frames to a first output frame based on a first neural network, where the first output frame represents a noise-invariant component of the first audio frame (720). In some embodiments, the first neural network may represent a decoder, or both an encoder and a decoder.

The machine learning system may further determine a first loss value based on differences between the first output frame and a first ground truth frame (730). In some embodiments, the first ground truth frame may represent the noise-invariant component of the first audio frame. The machine learning system may also update the first neural network based at least in part on the first loss value (740).

In some embodiments, the machine learning system may further map the first audio frame in the sequence of audio frames to a second output frame based on a second neural network, where the second output frame may represent a noise-variant component of the first audio frame. In some embodiments, the second neural network may represent a decoder, or both an encoder and decoder.

The machine learning system may further determine a second loss value based on differences between the second output frame and a second ground truth frame. In some embodiments, the second ground truth frame may represent the noise-variant component of the first audio frame. The machine learning system may also update the second neural network based at least in part on the second loss value.

In the foregoing specification, embodiments have been described with reference to specific examples thereof. It will, however, be evident that various modifications and changes may be made thereto without departing from the broader scope of the disclosure as set forth in the appended claims. The specification and drawings are, accordingly, to be regarded in an illustrative sense rather than a restrictive sense.