Patent Description:
Audio codecs can be used for compressing speech in a communications application (e.g., as executed on a mobile computing device). However, these audio codecs may not compress speech to a desirable size for minimizing bandwidth usage (e.g., bit rate) in a communications network. The possibility of using a machine-learning based generative model as an audio codec suggests further compression is possible resulting in a reduction in size and bandwidth usage (e.g., bit rate) for speech applications. However, successfully training machine-learning based generative models that can handle speech from a wide variety of speakers and speech uttered in a wide variety of environments can be difficult.

<CIT> relates to a method of low-bitrate coding of audio data and generating enhancement metadata for controlling audio enhancement of the low-bitrate coded audio data at a decoder side.

relates to source coding of audio signals with the help of generative models.

Example embodiments will become more fully understood from the detailed description given herein below and the accompanying drawings, wherein like elements are represented by like reference numerals, which are given by way of illustration only and thus are not limiting of the example embodiments and wherein:.

It should be noted that these Figures are intended to illustrate the general characteristics of methods, structure and/or materials utilized in certain example embodiments and to supplement the written description provided below.

The possibility of using a machine-learning based generative model as an audio codec suggests further compression is possible resulting in a reduction in size and bandwidth usage (e.g., bit rate) for speech applications. However, the performance of generative models can deteriorate with distortions that can exist in real-world input signals. For example, a technical problem with generative synthesis is that it can be sensitive to the quality of the data used for training and to the conditioning sequences used for training and inference. This deterioration can be caused by the sensitivity of the maximum likelihood criterion to outliers in the training data (e.g., noise events that are uncorrelated with the speech to be modeled), resulting in poor synthesized speech quality.

Implementations of the techniques described herein relate to a technical solution in which predictive-variance regularization is used to reduce the sensitivity of the maximum likelihood criterion to outliers. The use of predictive-variance regularization can result in an increase in the performance of one or more generative models as an audio codec (e.g., the synthesis of an utterance by the generative model can better represent the original speech). Example implementations can use noise reduction to remove unwanted signals (e.g., during model training) to increase performance of generative models as an audio codec for bit rates as low as, for example, <NUM> kb/s for real-world speech signals at reasonable computational complexity (e.g., to minimize processor usage).

<FIG> illustrates a pictorial representation and a block diagram of a portion of a communication system according to at least one example embodiment. As shown in <FIG>, a communication system <NUM> includes a first computing device <NUM> operated by a first user <NUM> and a second computing device <NUM> operated by a second user <NUM>. The first computing device <NUM> and the second computing device <NUM> can be communicatively coupled via, for example, an Internet.

The first computing device <NUM> can include an application <NUM> including an associated encoder <NUM> and an associated decoder <NUM>, and the second computing device <NUM> can include an application <NUM> including an associated encoder <NUM> and an associated decoder <NUM>. The encoder <NUM>, <NUM> and the decoder <NUM>, <NUM> can be pre-installed on the corresponding computing device <NUM>, <NUM> and/or installed on the corresponding computing device <NUM>, <NUM> with installation of the associated application <NUM>, <NUM>. For example, the encoder <NUM>, <NUM> and the decoder <NUM>, <NUM> can include an audio and/or video codec that is pre-installed on the corresponding computing device <NUM>, <NUM> and/or installed on the corresponding computing device <NUM>, <NUM> with installation of the associated application <NUM>, <NUM>.

The application <NUM> can be communicatively coupled with the application via the coupling of the first computing device <NUM> and the second computing device <NUM>. For example, data generated by the encoder <NUM> (e.g., a bitstream) can be communicated to the decoder <NUM> and data generated by the encoder <NUM> (e.g., an audio bitstream) can be communicated to the decoder <NUM>. The encoder <NUM>, <NUM> can be configured to compress (e.g., encode) an utterance by the corresponding user <NUM>, <NUM> as captured using, for example, a microphone of the computing device <NUM>, <NUM>. The compressed utterance can have a small size (e.g., memory usage) such that low bit rates (e.g., small bandwidth utilization) can be achieved between the computing device <NUM> and the computing device <NUM>.

The application <NUM>, <NUM> can include a user interface <NUM>, <NUM>. The user interface <NUM>, <NUM> can be configured to provide an interface with user <NUM>, <NUM>. For example, the user interface <NUM>, <NUM> can be configured to initiate and control the communication between computing device <NUM> and computing device <NUM>. The user interface <NUM>, <NUM> can be configured to capture and display audio and/or video as captured using components of computing device <NUM>, <NUM>. The user interface <NUM>, <NUM> can be configured to control audio and/or video (e.g., mute, background modification, camera on/off, and the like). The user interface <NUM>, <NUM> can be configured to provide other communication operations (e.g., chat). The user interface <NUM>, <NUM> can be configured to be an audio communication (e.g., phone call, short range private communication (e.g., walkie-talkie), and the like) application. The user interface <NUM>, <NUM> can be configured to be an audio device (e.g., streaming music, broadcast audio, podcast, and the like), a video device (e.g., streaming video), and/or the like. Any of the functions of the user interface <NUM>, <NUM> can include the use of the techniques/implementations described below.

The decoder <NUM>, <NUM> can include an audio codec including a generative model for decoding an audio stream. Accordingly, the encoder <NUM>, <NUM> can include an audio codec to compress audio such that the compressed audio can be decompressed (e.g., decoded, synthesized) using a generative model. The generative model can be a machine learned (e.g., trained) generative model. The generative model can be trained using a noise reduction technique to remove unwanted signals in audio before and/or during the training of the generative model. Training the model using predictive variance regularization is described in more detail below.

An example implementation can include the identification of causes of the sensitivity (e.g., audio noise) to distortion when training the model and techniques to reduce this sensitivity (e.g., reduce the impact of the audio noise on the performance of the trained model). A cause of the sensitivity can be associated with an attribute of the log-likelihood (LL) objective function. The LL objective function can incur a relatively high penalty if the model assigns a low probability to observed data. Therefore, in the context of an autoregressive structure, the LL objective function can encourage an overly broad predictive distribution when at least some training data are difficult to predict accurately from the past signal and conditioning, which is the case for real-world training data that include random or unusual noise events. This effect can be mitigated (e.g., reduced) by including predictive variance regularization in the overall objective function used to train the machine learning model. To prevent the need for simultaneous modeling of independent signals, example implementations can, at lower signal-to-noise ratios, apply noise reduction techniques prior to extracting the features that are used for conditioning.

To understand how an autoregressive model is used to model a process, consider a random process {Xi} that consists of real-valued random samples Xi, with a time index i ∈ Z. The joint distribution of a finite sequence, p(xi, ···, xi-N), can be expressed as a product of conditional distributions: <MAT> where β represents conditioning information.

It follows from eqn. (<NUM>) that an approximate realization of a random process can be created by recursively sampling from a model of the predictive distribution p(xi|xi-<NUM>, ··· , xi-N, β) for sufficiently large N. A standard-form distribution q(xi|α) with parameters α can be used as a model predictive distribution. The standard-form distribution can be, for example, a Gaussian or a logistic mixture. This formulation can enable prediction of the model parameters with a deterministic neural network φ: (xi-<NUM>, ···, xi-N, β, W) ↦ α where W is a vector of network parameters. Thus, the predictive distribution for sample xi can be q(xi|(φ(xi-<NUM>, ··· , xi-N, β, W)).

To find the parameters W, a reasonable objective can be to minimize the Kullback-Leibler divergence between the ground-truth joint distribution p(xi, ···, xi-N) and the model distribution q(xi,···, xi-N), or, equivalently, the cross-entropy between these distributions. The latter measure can be tractable even though p may only be available as an empirical distribution. It follows from eqn. (<NUM>) and the formulation of q(xi|α) that cross-entropy based estimation of the parameters of φ can be implemented using maximum-likelihood based teacher forcing. For M signal samples, the maximum-likelihood estimate of W can be written as: <MAT> Note that eqn. (<NUM>) can lead to rapid training as it facilitates parallel implementation.

For sufficiently large N and M, the LL objective can provide an upper bound on the differential entropy rate as: <MAT> where, for notational convenience, the unconditioned case is considered.

Conversely, eqn (<NUM>) can be interpreted as a lower bound on a measure of uncertainty associated with the model predictive distribution. This lower bound is associated with the process itself and not with the model. Although the differential entropy rate can be subadditive for summed signals, predictive models may not work well for summed signals. In general, a model of summed signals can be multiplicative in the required model configurations. The sum of finite-order linear autoregressive models may not be a finite-order autoregressive model. This problem can be reduced with noise suppression.

A challenging problem relates to drawbacks of the Kullback-Leibler divergence, and, hence, the LL objective of eqn (<NUM>). When the model distribution q vanishes in the support region of the ground-truth p, the Kullback-Leibler divergence can diverge. In eqn (<NUM>) this divergence can manifest as a penalty for training data xi that have a low model probability q(xi|φ(xi-<NUM>, ···, xi-N, β, W)). Hence, a few nonrepresentative outliers in the training data may lead the training procedure to equip the predictive model distribution with heavy tails (e.g., data outside a primary frequency). Such tails can lead to signal synthesis with a relatively high entropy rate during inference. In audio synthesis, the relatively high entropy rate can correspond to a noisy synthesized signal. Therefore, it may be desirable to counter the severity of the penalty for low probability training data.

There can be a second relevant drawback to the machine learned (ML) objective. When the ML objective function is used, the model distribution should converge to the ground-truth distribution with an increasing data sample size. However, in practice, the stochastic nature of the training data and the training method can result in inaccuracies. Therefore, the method can attempt to minimize the impact of such errors. For example, the implicit description of pitch by the predictive distribution may be inaccurate. A predictive model distribution with heavy tails for voiced speech then increases the likelihood of training data as it reduces the impact of the model pitch deviating from the ground-truth pitch. From this reasoning, accounting for the audibility (perception) of distortions leading to empirically motivated refinements of the objective function may be desirable.

Two related techniques can modify the maximum likelihood criterion to obtain improved performance. Both techniques can reduce the impact of data points in the training set that are difficult to predict, and the techniques can remove the need for heuristic modifications during inference.

A first technique for modifying the maximum likelihood criterion can be to add a term to the objective function that encourages low-variance predictive distributions. In this approach the overall objective function can be defined for the weights W given a database {x} as: <MAT> where the log likelihood over the database, <MAT> is combined with a variance regularization term Jvar({x}, W) that is defined below and where v is a constant that can be tuned.

The variance of the predictive distribution can be an instantaneous parameter that varies over a set of data, and Juar({x}, W) can be an average over the predictive distributions. The predictive distribution of each sample can have a distinct variance, and the averaging method can be selected to have properties that can be advantageous for the specific application. As discussed above, the predictive distribution can be a standard-form distribution q(x|α).

The predictive distribution q(x|α) can be a mixture distribution. Therefore, an expression for the variance of a mixture distribution should be determined. The mean of a mixture distribution can be: <MAT> where Eq is the expectation over q and qk = <IMG>(·; µk, sk), with <IMG> a mixture component.

The variance of the mixture distribution can be: <MAT>.

Considering the specific case of a mixture of logistics in more detail, the logistic distribution for component k can be expressed as: <MAT> where sk is the scale and µk is an offset.

The logistic distribution can be symmetric around µ. , and therefore, µ is the distribution mean. The variance of the logistic distribution can be expressed as: <MAT>.

The variance of the mixture of logistics model can be a combination of eqn. (<NUM>) and eqn. (<NUM>): <MAT>.

A technique for reducing the prediction variance can be to use the prediction variance eqn. (<NUM>) directly as variance regularization in the objective function eqn. (<NUM>): <MAT> where Edata indicates averaging over the set of data.

Weights W of the network φ that minimize <MAT> can be selected. Optimization of eqn. (<NUM>) over a set of data may result in the prediction variance being reduced for signal regions where the conditional differential entropy eqn. (<NUM>) is large. The conditional differential entropy can be decomposed into the sum of a scale-independent term and a logarithmic scale (signal variance) dependency. For speech, the scale independent term can be large for an unvoiced segment, while the scale-dependent term can be large for voiced speech.

For signals that have uniform overall signal variance, setting low predictive variance for regions that have relatively low conditional differential entropy may be desirable (e.g., for speech that would correspond to encouraging low variance for voiced speech only). This can be accomplished by a monotonically increasing concave function of the predictive variance. The logarithm can be used for this purpose because the logarithm can be invariant with scale. The effect of a small variance getting smaller can equal that of a large variance getting smaller by the same proportion. Therefore: <MAT> can be used, with a providing a floor.

A second technique for modifying the maximum likelihood criterion to prevent the vanishing support problem of the Kullback-Leibler divergence can be to use a baseline distribution. For example, using a mixture distribution of the form: <MAT> where the parameters α<NUM> are set by the designer and where the first term is omitted during inference (the other terms can be renormalized by a factor γk/(<NUM> - γ<NUM>).

By selecting α<NUM> to provide an overly broad distribution, the distribution used for inference can be of a low variance.

In an example implementation, consider an input signal with a sampling rate S Hz. To avoid the need for modeling summed independent signals, the input can be pre-processed with a real-time TasNet at inference. An encoder can convert the signal into a sequence of log mel-spectra. A set of subsequent log mel-spectra can be stacked into a supervector that can be subjected to a Karhunen-Loève transform (KLT). The transformed stacked log mel-spectra can be encoded using split-vector quantization with a small number of coefficients per split. In the example implementation, no other information may be encoded.

A decoder can decode the bitstream into a sequence of quantized log mel-spectra. These spectra can form the input to a conditioning stack, including a set of one-dimensional (1D) convolutional layers. The 1D convolutional layers can include dilation except for the first convolutional layer. The output can be a vector sequence with a dimensionality equal to a gated recurring unit (GRU) state and a sampling rate can be equal to that of the mel-spectra of the encoder.

An autoregressive network can include a multi-band WaveGRU that is based on gated recurring units. For an N-band WaveGRU, N samples can be generated simultaneously at an update rate of S/N Hz, one sample for each frequency band. For each update, the state of the GRU network can be projected onto an N × K × <NUM> dimensional space that defines N parameter sets. Each of the parameter sets can correspond to a mixture of logistics for a band. The value of a next signal sample for each band can be drawn by first selecting the mixture component (e.g., a logistics distribution) according to the bands probability and then drawing the sample from this logistics distribution by transforming a sample from a uniform distribution. For each set of N samples, a synthesis filter-bank can generate N subsequent time-domain samples. The time-domain samples can result in an output with sampling rate S Hz.

The input to the WaveGRU can include the addition of autoregressive and conditioning components. The autoregressive component can be a projection of the last N frequency-band samples projected onto a vector of the dimensionality of the WaveGRU state. The second component can be the output of the conditioning stack (e.g., at a dimensionality of the WaveGRU state), repeated in time to obtain the correct sampling rate of S/N Hz.

The training of the GRU network and the conditioning stack can be performed simultaneously using teacher forcing. In other words, the past signal samples that are provided as input to the GRU can be ground-truth signal samples. The objective function eqn. (<NUM>), combining log likelihood (cross entropy) and variance regularization, can be used for each subsequent signal sample.

<FIG> and <FIG> illustrate a block diagram of an audio processing system according to at least one example embodiment. As shown in <FIG> and <FIG>, an audio processing system <NUM> includes at least one processor <NUM>, at least one memory <NUM>, and a controller <NUM>. <FIG> illustrates components of the system <NUM> used to encode an audio signal into a bitstream at a computing device for transmission from the computing device to another computing device. <FIG> illustrates components of the system <NUM> used to decode a bitstream received at a computing device from another computing device and to reconstruct/synthesize an audio signal from the decoded bitstream. The components of system <NUM> can be included in computing devices <NUM> and <NUM>, such that devices <NUM> and <NUM> can both process audio signals for transmission as bitstreams to another device and can receive bitstreams from another device and process the received bitstreams to synthesize an audio signal for rendering to a user. As shown in <FIG>, the audio processing system <NUM> can include a sample <NUM> block, a time domain separator <NUM> block, a converter <NUM> block, a transformer <NUM> block, and a quantizer <NUM> block. As shown in <FIG>, the audio processing system <NUM> can include a power <NUM> block, a convolve <NUM> block, and a reconstruct <NUM> block. The at least one processor <NUM>, the at least one memory <NUM>, the controller <NUM>, the sample <NUM> block, the time domain separator <NUM> block, the converter <NUM> block, the transformer <NUM> block, the quantizer <NUM> block, the power <NUM> block, the convolve <NUM> block, and the reconstruct <NUM> block are communicatively coupled via bus <NUM>. In <FIG> and <FIG> solid lines represent control links and dashed lines represent data links.

The at least one processor <NUM> may be utilized to execute instructions stored on the at least one memory <NUM>, so as to thereby implement the various features and functions described herein, or additional or alternative features and functions. The at least one processor <NUM> may be a general-purpose processor. The at least one processor <NUM> may be a graphics processing unit (GPU) and/or an audio processing unit (APU). The at least one processor <NUM> and the at least one memory <NUM> may be utilized for various other purposes. In particular, the at least one memory <NUM> can represent an example of various types of memory and related hardware and software which might be used to implement any one of the modules described herein.

The at least one memory <NUM> may be configured to store data and/or information associated with the audio processing system <NUM>. For example, the at least one memory <NUM> may be configured to store code associated with implementing a communications application including a codec that uses a regressive (e.g., autoregressive) model. For example, the at least one memory <NUM> may be configured to store code associated with encoding/decoding audio data (e.g., an audio stream or utterance) using a regressive (e.g., autoregressive) model. The at least one memory <NUM> may be a non-transitory computer readable medium with code that when executed by the processor <NUM> causes the processer <NUM> to implement one or more of the techniques described herein. The at least one memory <NUM> may be a shared resource. For example, the audio processing system <NUM> may be an element of a larger system (e.g., a server, a personal computer, a mobile device, and the like). Therefore, the at least one memory <NUM> may be configured to store data and/or information associated with other elements (e.g., web browsing, camera, games, and/or the like) within the larger system.

The controller <NUM> may be configured to generate various control signals and communicate the control signals to various blocks in the audio processing system <NUM>. The controller <NUM> may be configured to generate the control signals to implement the techniques described herein. The controller <NUM> may be configured to control data for a codec based on a regressive model.

Referring to <FIG>, an audio stream <NUM> is input to the audio processing system <NUM>. The audio processing system <NUM> can be configured to generate a bitstream <NUM> (e.g., a compressed audio bitstream) based on the audio stream <NUM>. The bitstream <NUM> can be communicated as an operation of a communication application. For example, the bitstream <NUM> can be communicated from device <NUM> to device <NUM> using application <NUM>, <NUM>. For example, the bitstream <NUM> can be communicated from device <NUM> to device <NUM> using application <NUM>, <NUM>.

The sample <NUM> block can be configured to sample the audio stream <NUM>. Sampling can include using a sampling rate of S Hz to generate a discrete-time signal representing the audio stream <NUM>. The sampling rate can be directly proportional to the bit rate. Therefore, the lower the sampling rate the lower the bit rate. Therefore, the sampling rate can cause a compression of the audio stream <NUM>. In other words, the lower the sampling rate S, the more compressed (holding the bit depth or the number of bits used to store the sampled audio stream <NUM> constant) the audio stream <NUM> may be. Therefore, the sampling rate S can be selected to enable (or help enable) the bit rate of the audio stream to be as low as, for example, <NUM> kb/s.

The time domain separator <NUM> block can be configured to separate speech, so as to avoid the need for modeling summed independent signals. In some implementations, the sampled audio stream <NUM> can be processed to separate independent speech signals in the sampled audio stream <NUM> and/or to suppress noise in the separated independent speech signals. In some implementations, a real-time TasNet (time-domain audio separation network) can apply a set of weighting functions (or masks) to the sampled audio stream <NUM> to separate speech and to perform noise suppression. In other words, to separate each speaker that may be heard in an utterance represented by the sampled audio stream <NUM>, a weighted mask can be applied to the sampled audio stream. In some implementations, a TasNet with only one output channel can be used to suppress noise and to remove all but one speech channel by masking out by multiplying components (of a learned representation) that are not attributed to the one speech signal. The result can be that the user (e.g., user <NUM>, <NUM>) can be the focus of the audio processing system <NUM> by using a mask to isolate the user and filter out any background voices.

The converter <NUM> block can be configured to parameterize the separated audio stream. In example implementations, the separated audio stream can be converted into a sequence of log mel-spectra, and these spectra can form the input to a conditioning stack, including a set of one-dimensional (1D) convolutional layers. A log mel-spectra can be a representation of the short-term power spectrum of the separated audio stream <NUM>, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency. A set of the resultant log mel-spectra can be stacked into a supervector. To quantify speech (e.g., an utterance) associated with the user (e.g., user <NUM>, <NUM>) the log mel-spectra can be adapted to a Gaussian mixture model to fit the user effectively shifting the means to some direction. The adaptation direction is a real valued vector that characterizes the user. This is sometimes called a supervector.

The transformer <NUM> block can be configured to apply a Karhunen- Loève transform (KLT) to the log mel-spectra (e.g., the supervector) corresponding to the audio stream <NUM>. The KLT is a linear transform where the basis functions are taken from the statistics of the signal (e.g., the log mel-spectra or supervector). The KLT can be an adaptive transform (e.g., adapted to the audio power). The KLT can be an optimal transform as relates to energy compaction. In other words, the KLT can place as much energy as possible into as few coefficients as possible.

The quantizer <NUM> block can be configured to quantize the energy (e.g., power or coefficients) associated with the transformed audio stream <NUM>. For example, the transformed stacked log mel-spectra are encoded using split-vector quantization with a small number of coefficients per split. Quantization of energy can refer to assigning the energy (e.g., the transformed stacked log mel-spectra) to discrete energy levels. In split-vector quantization coefficients of the log mel-spectra or supervector can be split into p equal size groups. If the groups cannot be of equal size, then either the remaining coefficients can be added to one of the groups or the remaining coefficients can be put in their own smaller group.

Referring to <FIG>, a bitstream <NUM> is input to the audio processing system <NUM>. The audio processing system <NUM> can be configured to synthesize the audio stream <NUM> (e.g., as a reconstructed audio stream) based on the received bitstream <NUM>. The audio stream <NUM> can be synthesized (e.g., reconstructed) based on the bitstream <NUM> using a generative model. The bitstream <NUM> can be received as an operation of a communication application. For example, the bitstream <NUM> can be received by device <NUM> from device <NUM> using application <NUM>, <NUM>. For example, the bitstream <NUM> can be received by device <NUM> from device <NUM> using application <NUM>, <NUM>.

The power <NUM> block can be configured to generate a sequence of quantized log mel-spectra based on the bitstream <NUM>. The sequence of quantized log mel-spectra should substantially match the transformed stacked log mel-spectra input to the quantizer <NUM> for at least a portion of bitstream <NUM> corresponding to at least a portion of audio stream <NUM>.

The convolve <NUM> block can be configured to generate a vector sequence based on the quantized log-mel spectra. The sequence of quantized log-mel spectra can be the input to a conditioning stack, including a set of one-dimensional (1D) convolutional layers. The 1D convolutional layers can include dilation except for the first convolutional layer. The output can be the vector sequence with a dimensionality equal to a gated recurring unit (GRU) state and a sampling rate can be equal to that of the mel-spectra of the encoder.

The reconstruct <NUM> block can be configured to generate an audio stream reconstruction of a corresponding encoded audio stream (e.g., a bitstream). The reconstruct <NUM> block can be configured to use a regressive (e.g., autoregressive) network to generate the audio stream reconstruction (e.g., reconstructed audio stream <NUM>). The regressive network can include a multi-band WaveGRU that is based on gated recurring units. For an N-band WaveGRU, N samples can be generated simultaneously at an update rate of S/N Hz, one sample for each frequency band. For each update, the state of the GRU network can be projected onto an N × K × <NUM> dimensional space that defines N parameter sets. Each of the parameter sets can correspond to a mixture of logistics for a band. The value of a next signal sample for each band can be drawn by first selecting the mixture component (e.g., a logistics distribution) according to the bands probability and then drawing the sample from this logistics distribution by transforming a sample from a uniform distribution. For each set of N samples, a synthesis filter-bank can generate N subsequent time-domain samples. The time-domain samples can result in an output (e.g., reconstructed audio stream <NUM> or an utterance) with sampling rate S Hz.

<FIG> illustrates a block diagram of a model training system according to at least one example embodiment. As shown in <FIG>, a model training system <NUM> includes at least one processor <NUM>, at least one memory <NUM>, a controller <NUM>, a sample <NUM> block, a time domain separator <NUM> block, an evaluator and modifier <NUM> block, a transformer <NUM> block, a quantizer <NUM> block, a reconstruct <NUM> block, and a compare <NUM> block. The at least one processor <NUM>, the at least one memory <NUM>, the controller <NUM>, the sample <NUM> block, the time domain separator <NUM> block, the converter <NUM> block, the transformer <NUM> block, the quantizer <NUM> block, the reconstruct <NUM> block, and the evaluator and modifier <NUM> block are communicatively coupled via bus <NUM>. In <FIG> solid lines represent control links and dashed lines represent data links.

The at least one memory <NUM> may be configured to store data and/or information associated with the audio processing system <NUM>. For example, the at least one memory <NUM> may be configured to store code associated with training a regressive (e.g., autoregressive) model for encoding audio (e.g., speech). For example, the at least one memory <NUM> may be configured to store code associated with teacher forcing regressive model training and removing noise from audio data used in the teacher forcing. The at least one memory <NUM> may be a non-transitory computer readable medium with code that when executed by the processor <NUM> cause the processer <NUM> to implement one or more of the techniques described herein. The at least one memory <NUM> may be a shared resource. For example, the model training system <NUM> may be an element of a larger system (e.g., a server, a personal computer, a mobile device, and the like). Therefore, the at least one memory <NUM> may be configured to store data and/or information associated with other elements within the larger system.

The controller <NUM> may be configured to generate various control signals and communicate the control signals to various blocks in the model training system <NUM>. The controller <NUM> may be configured to generate the control signals to implement the techniques described herein. The controller <NUM> may be configured to control the evaluator and modifier <NUM> block to evaluate an objective function associated with a recursive model and to modify weights associated with the objective function in response to the evaluation. Modifying weights can be performed in response to determining whether the objection function eqn. <NUM> has been minimized during the training process.

As shown in <FIG>, an audio stream <NUM> is input to the model training system <NUM>. The sample <NUM> block can be configured to sample the audio stream <NUM>. Sampling can include using a sampling rate S Hz to generate a discrete-time signal representing the audio stream <NUM>.

The time domain separator <NUM> block can be configured to separate speech. In order to avoid the need for modeling summed independent signals, the sampled audio stream <NUM> can be processed with a real-time TasNet (time-domain audio separation network) in order to separate speech and/or suppress noise. A TasNet can apply a set of weighting functions (or masks) to separate the speech. In other words, to separate each speaker that may be heard in an utterance represented by the sampled audio stream <NUM>, a weighted mask can be applied to the sampled audio stream. The result can be that the user (e.g., <NUM>, <NUM>) can be the focus of the audio processing system <NUM> by using a mask to isolate the user and filter out any background voices.

The converter <NUM> block can be configured to convert the separated audio stream into a sequence of log mel-spectra. These spectra can form the input to a conditioning stack, including a set of one-dimensional (1D) convolutional layers. A log mel-spectra can be a representation of the short-term power spectrum of the separated audio stream, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency. A set of the resultant log mel-spectra can be stacked into a supervector. To quantify speech (e.g., an utterance) associated with the user (e.g., user <NUM>, <NUM>) the log mel-spectra can be adapted to a Gaussian mixture model to fit the user effectively shifting the means to some direction. The adaptation direction is a real valued vector that characterizes the user. This is sometimes called a supervector.

The transformer <NUM> block can be configured to apply a Karhunen- Loève transform (KLT) to the log mel-spectra (e.g., the supervector) corresponding to the audio stream <NUM>. The KLT is a linear transform where the basis functions are taken from the statistics of the signal (e.g., the log mel-spectra or supervector). The KLT can be an adaptive transform (e.g., adapt to the audio power). The KLT can be an optimal transform as relates to energy compaction. In other words, the KLT can place as much energy as possible in as few coefficients as possible.

Implementations can include two techniques that can reduce the impact of data points in the training set that are difficult to predict (e.g., low probability data). In the invention, a first technique can modify the maximum likelihood criterion by adding a term to the objective function that encourages low-variance predictive distributions. In other words, a regularization term is used in the objective function to de-emphasize the influence of the outlier data (e.g., high differential entropy data) during the training of the model.

As discussed above with regard to equations (<NUM>)-(<NUM>), for signals that have uniform overall signal variance, setting a low predictive variance for regions that have relatively low conditional differential entropy may be desirable (e.g., for speech this would correspond to encouraging low variance for voiced speech only). This can be accomplished by a monotonically increasing concave function of the predictive variance. The logarithm can be used for this purpose because the logarithm can be invariant with scale.

A second technique for modifying the maximum likelihood criterion can be to prevent the vanishing support problem of the Kullback-Leibler divergence by using a baseline distribution as discussed above with regard to equation <NUM>. In an example implementation, the distribution used for inference can be selected to have a low variance.

The reconstruct <NUM> block can be configured to generate an audio stream reconstruction of a corresponding encoded audio stream (e.g., a bitstream). The reconstruct <NUM> block can be configured to use a machine learned (ML) model to reconstruct an audio stream. The reconstruct <NUM> block can be configured to communicate the results from executing the objection function eqn. <NUM> to the evaluator and modifier <NUM> block.

The model training system <NUM> can be configured to train the ML model. Training the ML model can include training the ML model to generate a high-fidelity audio bitstream from a low bitrate input bitstream. The training of the ML model can include de-emphasizing the influence of low-probability distortion events in the sampled audio data on the trained ML model. For example, audio stream <NUM> can include noise. This noise can cause the low-probability distortion events in the sampled audio data. The de-emphasizing of the distortion events (e.g., minimizing the effect or impact of noise on the training of the ML model) is achieved by the inclusion of a term (e.g., a regularization term) in an objective function of the ML model. The term can encourage low-variance predictive distributions of a next sample in the sampled audio data. Accordingly, the ML model used in the reconstruct <NUM> block can include a modification to an objective function associated with the ML model. The objective function can include a regularization term that can reduce the effect of noise on the ML model.

Reducing the effect of noise when training a generative model can reduce the sensitivity of the model to distortion when the model is used to decompress an audio stream that includes some noise. For example, a cause of the sensitivity can be associated with an attribute of the log-likelihood (LL) objective function. The LL objective function can incur a penalty if the model assigns a low probability to observed data. Therefore, in the context of an autoregressive structure, assigning a low probability to observed data can encourage an overly broad predictive distribution when at least some training data are difficult to predict accurately from the past signal and conditioning. Therefore, the reconstruct <NUM> block can be configured to mitigate the effect of assigning a low probability to observed data on the LL objective function during the training of the ML model.

The ML model can include a regressive (e.g., autoregressive) network to generate the audio stream reconstruction of the input audio stream (e.g., reconstructed audio stream <NUM>). The regressive network can include a multi-band WaveGRU that is based on gated recurring units. For an N-band WaveGRU, N samples can be generated simultaneously at an update rate of S/N Hz, one sample for each frequency band. For each update, the state of the gated recurring unit (GRU) network can be projected onto an N × K × <NUM> dimensional space that defines N parameter sets. Each of the parameter sets can correspond to a mixture of logistics for a band. The value of a next signal sample for each band can be drawn by first selecting the mixture component (e.g., a logistics distribution) according to the bands probability and then drawing the sample from this logistics distribution by transforming a sample from a uniform distribution. For each set of N samples, a synthesis filter-bank can generate N subsequent time-domain samples. The time-domain samples can result in an output with sampling rate S Hz.

The evaluator and modifier <NUM> block can be configured to implement the training of the regressive model. In an example implementation, the training of the regressive model (e.g., as the GRU network and the conditioning stack) can be performed simultaneously using teacher forcing, in which past signal samples are provided as input to the model as ground-truth signal samples. The objective function of eqn. <NUM> can be used for each subsequent signal sample. The evaluator and modifier <NUM> block can evaluate the result of the objective function eqn. <NUM> from the current iteration against the result of the objective function eqn. <NUM> of at least one previous iteration of the training cycle. The evaluator and modifier <NUM> block can be configured to minimize the objective function including the regularization term eqn. <NUM>, for example, with stochastic gradient descent or related methods, which can maximize the likelihood of the training data as indicated by eqn.

If the result of the evaluation passes a criterion, the training can end. For example, the criterion can be based on minimizing the objective function eqn. Should a modification of the weights result in a change in the result of the objective function being below a threshold, the evaluation can be deemed to pass the criterion. If the result of the evaluation fails the criterion, the regressive model can be modified (e.g., weights associated with the recursive model or GRU network can be changed), for example, with stochastic gradient descent or related methods, and the training can continue.

<FIG> and <FIG> are flowcharts of methods according to example embodiments. The methods described with regard to <FIG> and <FIG> may be performed due to the execution of software code stored in a memory (e.g., a non-transitory computer readable storage medium) associated with an apparatus and executed by at least one processor associated with the apparatus.

However, alternative embodiments are contemplated such as a system embodied as a special purpose processor. The special purpose processor can be a graphics processing unit (GPU) and/or an audio processing unit (APU). A GPU can be a component of a graphics card. An APU can be a component of a sound card. The graphics card and/or sound card can also include video/audio memory, random access memory digital-to-analogue converter (RAMDAC) and driver software. The video/audio memory can be a frame buffer that stores digital data representing an image, a frame of a video, audio data associated with the frame, and/or streaming audio. A RAMDAC can be configured to read the contents of the video/audio memory, convert the content into an analogue signal and sends analog signal to a display/monitor and/or a speaker. The driver software can be the software code stored in the memory referred to above. The software code can be configured to implement the method described herein.

Although the methods described below are described as being executed by a processor and/or a special purpose processor, the methods are not necessarily executed by a same processor. In other words, at least one processor and/or at least one special purpose processor may execute the method described below with regard to <FIG> and <FIG>.

<FIG> illustrates a flowchart of a method for communicating audio according to at least one example embodiment. As shown in <FIG>, in step S405 a sampled audio data corresponding to a first utterance is received at a first device. For example, an audio stream <NUM> can be sensed by a computing device <NUM>, <NUM> executing a communications application <NUM>, <NUM>. The audio stream <NUM> can be sampled. For example, sampling can include using a sampling rate S Hz to generate a discrete-time signal representing the audio stream <NUM>. The sampling rate can be directly proportional to the bit rate. Therefore, the lower the sampling rate the lower the bit rate. Therefore, the sampling rate can cause a compression of the audio stream <NUM>. In other words, the lower the sampling rate S, the more compressed (holding the bit depth or the number of bits used to store the sampled audio stream <NUM> constant) the audio stream <NUM> may be. Therefore, the sampling rate S can be selected to enable (or help enable) the bit rate of the audio stream to be as low as, for example, <NUM> kb/s.

In step S410 the sampled audio signal is separated in the time domain and noise in the signal is suppressed. For example, the sampled audio stream <NUM> can be pre-processed with a real-time TasNet (time-domain audio separation network) in order to separate speech and to reduce noise. A TasNet can apply a set of weighting functions (or masks) to separate the speech and/or to suppress noise. In other words, to separate each speaker that may be heard and to suppress noise in an utterance represented by the sampled audio stream <NUM>, a weighted mask can be applied to the sampled audio stream. The result can be that the user (e.g., user <NUM>, <NUM>) can be the focus of the audio processing by using a mask to isolate the user and filter out any background voices.

In step S415 the separated sampled audio signal is converted to parameters that characterize the speech signals. For example, the parameters that characterize the speech signals can be a sequence of vectors. For example, the separated audio stream can be converted into a sequence of log mel-spectra. These spectra can form the input to a conditioning stack, including a set of one-dimensional (1D) convolutional layers. A log mel-spectra can be a representation of the short-term power spectrum of the separated audio stream, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency. A set of the resultant log mel-spectra can be stacked into a supervector. To quantify speech (e.g., an utterance) associated with the user (e.g., user <NUM>, <NUM>) the log mel-spectra can be adapted to a Gaussian mixture model to fit the user effectively shifting the means to some direction. The adaptation direction is a real valued vector that characterizes the user. This is sometimes called a supervector.

In step S420 a first bitstream representing the utterance is generated by quantizing transformed parameters that characterize the speech signals. For example, the sequence of vectors can be transformed. In an example implementation, a Karhunen-Loève transform (KLT) can be applied to the log mel-spectra (e.g., the supervector) corresponding to the audio stream <NUM>. The KLT is a linear transform where the basis functions are taken from the statistics of the signal (e.g., the log mel-spectra or supervector). The KLT can be an adaptive transform (e.g., adapt to the audio power). The KLT can be an optimal transform as relates to energy compaction. In other words, the KLT can place as much energy as possible in as few coefficients as possible. For example, the energy (e.g., power or coefficients) associated with the transformed audio stream <NUM> can be quantized. The transformed stacked log mel-spectra can be encoded using split-vector quantization with a small number of coefficients per split. Quantization of energy can refer to assigning the energy (e.g., the transformed stacked log mel-spectra) to discrete energy levels. In split-vector quantization coefficients of the log mel-spectra or supervector can be split into p equal size groups. If the groups cannot be of equal size, then either the remaining coefficients can be added to one of the groups or the remaining coefficients can be put in their own smaller group.

In step S425 the first bitstream is communicated from the first device to a second device, for the second device to decode the bitstream and to synthesize an audio signal based on the decoded bitstream. For example, bitstream <NUM> as communicated from computing device <NUM> to computing device <NUM> can be the first bitstream.

In step S430 a second bitstream is received by the first device from the second device, the second bitstream representing speech signals of a second utterance. For example, a bitstream received by computing device <NUM> from computing device <NUM> can be the second bitstream, which then can be decoded and used to synthesize an audio signal using a trained machine learning model at the receiving computing device.

In step S435 a sequence of quantized parameters that characterize the speech signals is created based on the second bitstream. For example, a sequence of quantized log mel-spectra can be created based on the received bitstream. The sequence of quantized log mel-spectra should substantially match the transformed stacked log mel-spectra input to the quantizer of the computing device that generated the received bitstream for at least a portion of received bitstream.

In step S440 a vector sequence is generated based on inversely transforming the quantized parameters that characterize the speech signals. For example, a vector sequence can be created based on inversely transforming the quantized log-mel spectra. Then, the sequence of quantized log-mel spectra can be the input to a conditioning stack, including a set of one-dimensional (1D) convolutional layers. The 1D convolutional layers can include dilation except for the first convolutional layer. The output can be the vector sequence with a dimensionality equal to a gated recurring unit (GRU) state and a sampling rate can be equal to that of the mel-spectra of the encoder.

In step S445 the second utterance is regenerated based on the vector sequence using a trained regressive (e.g., autoregressive) network. For example, a regressive network can be used to create the audio stream reconstruction (e.g., reconstructed audio stream <NUM>). The regressive network can include a multi-band WaveGRU that is based on gated recurring units. For an N-band WaveGRU, N samples can be created simultaneously at an update rate of S/N Hz, one sample for each frequency band. For each update, the state of the GRU network can be projected onto an N × K × <NUM> dimensional space that defines N parameter sets. Each of the parameter sets can correspond to a mixture of logistics for a band. The value of a next signal sample for each band can be drawn by first selecting the mixture component (e.g., a logistics distribution) according to the bands probability and then drawing the sample from this logistics distribution by transforming a sample from a uniform distribution. For each set of N samples, a synthesis filter-bank can createN subsequent time-domain samples. The time-domain samples can result in an output (e.g., a reconstructed audio signal or utterance) with a sampling rate of S Hz. The reconstructed audio can be converted (e.g., using a digital to analog converter) to an analog audio signal for play-back on a speaker of the first device.

<FIG> illustrates a flowchart of a method for training a model according to at least one example embodiment. As shown in <FIG>, in step S505 sampled audio data corresponding to an utterance is received. For example, a database including at least one dataset of speech segments (e.g., utterances) can be used as the audio data. Sampling can include using a sampling rate S Hz to create a discrete-time signal representing the audio data (e.g., audio stream <NUM>). The sampling rate can be directly proportional to the bit rate. The database can be a publicly available database.

In step S510 a machine learning (ML) model is trained based on the sampled audio data by including a regularization term in an objective function of the ML model. Example implementations can include training the ML model to generate a high-fidelity audio bitstream from a low bitrate input bitstream. Training of the ML model can include de-emphasizing (e.g., minimizing the effect of) the influence of low-probability distortion events in the sampled audio data on the trained ML model. In other words, the effect of noise in the audio on the training of the ML model can be minimized. The de-emphasizing of the distortion events is achieved by the inclusion of a term (e.g., the regularization term) in the objective function of the ML model. The term (e.g., the regularization term) encourages the low-variance predictive distributions of a next sample in the sampled audio data.

Example implementations can include two techniques that can reduce the impact of data points in the training set that are difficult to predict (e.g., low probability data). A first technique can modify the maximum likelihood criterion by adding a term to the objective function that encourages low-variance predictive distributions.

As discussed above with regard to equations (<NUM>)-(<NUM>), signals that have uniform overall signal variance, setting low predictive variance for regions that have relatively low conditional differential entropy may be desirable (e.g., for speech that would correspond to encouraging low variance for voiced speech only). This can be accomplished by a monotonically increasing concave function of the predictive variance. The logarithm can be used for this purpose because the logarithm can be invariant with scale.

A second technique for modifying the maximum likelihood criterion can include preventing the vanishing support problem of the Kullback-Leibler divergence by using a baseline distribution as discussed above with regard to equation <NUM>. In an example implementation, the distribution used for inference can be selected to have a low variance.

Prior to training the ML model, the sampled audio signal can be separated in the time domain. In an example implementation, separating speech can include noise suppression. For example, the sampled audio stream <NUM> can be pre-processed with a real-time TasNet (time-domain audio separation network) in order to separate speech and suppress noise. A TasNet can apply a set of weighting functions (or masks) to separate the speech and suppress noise. In other words, to separate each speaker that may be heard and suppress noise in an utterance represented by the sampled audio stream <NUM>, a weighted mask can be applied to the sampled audio stream. The result can be that the user (e.g., user <NUM>, <NUM>) can be the focus of the audio processing by using a mask to isolate the user and filter out any background voices.

The separated sampled audio signal can be converted to parameters that characterize the speech signals. For example, the separated audio stream can be converted into a sequence of log mel-spectra. These spectra can form the input to a conditioning stack, including a set of one-dimensional (1D) convolutional layers. A log mel-spectra can be a representation of the short-term power spectrum of the separated audio stream, based on a linear cosine transform of a log power spectrum on a nonlinear mel scale of frequency. A set of the resultant log mel-spectra can be stacked into a supervector. To quantify speech (e.g., an utterance) associated with the user (e.g., user <NUM>, <NUM>) the log mel-spectra can be adapted to a Gaussian mixture model to fit the user effectively shifting the means to some direction. The adaptation direction is a real valued vector that characterizes the user. This is sometimes called a supervector.

The parameters that characterize the speech signals can be transformed. For example, a Karhunen- Loève transform (KLT) can be applied to the log mel-spectra (e.g., the supervector) corresponding to the audio stream <NUM>. The KLT is a linear transform where the basis functions are taken from the statistics of the signal (e.g., the log mel-spectra or supervector). The KLT can be an adaptive transform (e.g., adapt to the audio power). The KLT can be an optimal transform as relates to energy compaction. In other words, the KLT can place as much energy as possible in as few coefficients as possible.

The energy (e.g., power or coefficients) associated with the transformed audio stream <NUM> can be quantized. For example, the transformed stacked log mel-spectra are encoded using split-vector quantization with a small number of coefficients per split. Quantization of energy can refer to assigning the energy (e.g., the transformed stacked log mel-spectra) to discrete energy levels. In split-vector quantization coefficients of the log mel-spectra or supervector can be split into p equal size groups. If the groups cannot be of equal size, then either the remaining coefficients can be added to one of the groups or the remaining coefficients can be put in their own smaller group.

Training the ML model can include reconstructing the utterance by inputting the quantized audio data (e.g., as compressed audio data or a low bitrate input bitstream) into a regressive (e.g., autoregressive) network, the regressive network can be based on gated recurring units. For example, a regressive (e.g., autoregressive) network can be used to generate the audio stream reconstruction (e.g., reconstructed audio stream <NUM>). The regressive network can include a multi-band WaveGRU that is based on gated recurring units. For an N-band WaveGRU, N samples can be generated simultaneously at an update rate of S/N Hz, one sample for each frequency band. For each update, the state of the GRU network can be projected onto an N × K × <NUM> dimensional space that defines N parameter sets. Each of the parameter sets can correspond to a mixture of logistics for a band. The value of a next signal sample for each band can be drawn by first selecting the mixture component (e.g., a logistics distribution) according to the bands probability and then drawing the sample from this logistics distribution by transforming a sample from a uniform distribution. For each set of N samples, a synthesis filter-bank can generate N subsequent time-domain samples. The time-domain samples can result in an output with sampling rate S Hz.

Training the ML model can further include modifying the regressive network based on comparing the reconstructed utterance to a ground-truth sample. For example, the regressive network can be modified based on the training of the regressive model. In an example implementation, the training of the regressive model (e.g., as the GRU network and the conditioning stack) can be performed simultaneously using teacher forcing, in which past signal samples are provided as input to the model as ground truth signal samples, and the model is trained to maximize the likelihood of the observed signal data, assuming the past signal samples represent ground truth data. Evaluation of the training of the model can be based on a cost or loss function. The cost or loss function can be used to evaluate the comparison against a predefined criterion. The cost or loss function can be, for example, a maximum likelihood estimation (MLE), a MLE and cross-entropy, a mean squared error (MSE), a logarithmic loss, and the like.

If the result of the loss function passes the criterion, the training can end. If the result of the loss function fails the criterion, the regressive model can be modified (e.g., weights associated with the recursive model or GRU network can be changed) and the training can continue. Accordingly, the past signal samples (e.g., sampled audio stream <NUM>) can be ground-truth signal samples in a teacher forcing algorithm. The objective function eqn. (<NUM>), combining log likelihood (cross entropy) and variance regularization, can be used for each subsequent signal sample.

Training a machine learning (ML) model can include training the ML model to generate a high-fidelity audio bitstream from a low bitrate input bitstream. The training of the ML model can include de-emphasizing the influence of low-probability distortion events in the sampled audio data on the trained ML model. For example, utterances with noise can include noise that can cause the low-probability distortion events in the sampled audio data. The de-emphasizing of the distortion events (e.g., minimizing the effect or impact of noise on the training of the ML model) is achieved by the inclusion of a term (e.g., a regularization term) in an objective function of the ML model. The term encourages low-variance predictive distributions of a next sample in the sampled audio data. Accordingly, the objective function with a regularization term includes a modification to an objective function associated with the ML model. The objective function includes a regularization term that can reduce the effect of noise on the ML model, and reducing the effect of noise can reduce the sensitivity of a generative model to distortion.

An example illustrating the performance of the trained ML model based on predictive variance regularization and noise suppression follows. This example uses eight systems, all being variants based on a single baseline system that operates with <NUM> sampled signals. The baseline system was conditioned using a sequence of <NUM>-dimensional log mel spectra computed from <NUM> windows at an update rate of <NUM>. The system used four frequency bands, each band sampled at <NUM>. The conditioning stack consisted of a single non-causal input layer (giving a <NUM> lookahead and expanding from <NUM> channels to <NUM> channels), three dilated causal convolutional layers with kernel size two, and three upsampling transpose convolutional layers (kernel size two). The conditioning outputs were tiled to match the GRU update frequency. The GRU state dimensionality was <NUM>. The eight systems differed in mixture-of-logistics components used for the predictive distribution per band.

The systems were trained from randomly initialized weights W for <NUM> million steps, using a mini-batch size of <NUM>. The target signal and teacher forced autoregressive input audio from a combination of clean and noisy sources. In some cases, additional noise was added, with random signal-to-noise (SNR) ratios between <NUM> and <NUM> dB SNR.

Table <NUM> shows the combinations of coder attributes (i.e., variance regularization (v), noise suppression (t), quantization (q), and pruning) that were used for the eight systems (i.e., the systems labeled as "b" for baseline, and as "v", "t", "vt", "q", "qv", "qt", and "qvt"), and each attribute is discussed briefly below. The variance regularization was applied to the first two frequency bands only, and the parameter v in eqn. <NUM> was made proportional to a voicing score. The noise suppression was applied using a version of a fully-convolutional time-domain audio separation network (Conv-TasNet). The system was quantized with <NUM> bits per supervector, each supervector containing two log mel spectra, which implies a rate of <NUM> kb/s. The quantization was a two-dimensional vector-quantization of the KLT coefficients. The weight pruning attribute was selected to enable implementation of the model on consumer devices. For the three GRU matrices, block-diagonal matrices with <NUM> blocks were used, which uses <NUM>% fewer weights than a fully connected model. For other hidden layers, iterative magnitude pruning was used to remove <NUM>% of the model weights. The pruning makes the codec run reliably on user devices, such as a Pixel <NUM> phone, in single-threaded mode.

In this example, to evaluate the absolute quality of the different systems on different SNRs, a Mean Opinion Score (MOS) listening test was performed following the ITU-TP. <NUM> recommendation. The data was collected using a crowd-sourcing platform with the requirements on listeners being native English speakers and using headphones. The evaluation dataset was composed of <NUM> samples from the Noisy VCTK dataset from the Centre for Speech Technology's Voice Cloning Toolkit, with <NUM> clean samples (i.e., samples without noise) and <NUM> samples augmented with additive noise at SNRs of <NUM> dB, <NUM> dB, and <NUM> dB. Each utterance for each system was rated about <NUM> times and the average and <NUM>% confidence interval were calculated per SNR.

<FIG> and <FIG> are graphs showing the quality, based on MOS values, for the systems of Table <NUM>.

<FIG> displays the effect of predictive variance regularization and noise suppression without weight pruning and quantization. For each of the clean samples, and the samples with SNRs of <NUM> dB, <NUM> dB, and <NUM> dB MOS values are presented for: the baseline system (with and without noise suppression), the system with predictive variance regularization with and without noise suppression), and unprocessed signals as a reference (with and without noise suppression). As seen from <FIG>, predictive variance regularization results in a significant quality improvement and reduces the sensitivity to noise in the input signal. Noise suppression aids performance when noise is present.

<FIG> shows the quality for implementations that include pruning and quantization that may be used in consumer devices having finite processing power. For each of the clean samples, and the samples with SNRs of <NUM> dB, <NUM> dB, and <NUM> dB MOS values are presented for: the Opus codec operating at <NUM> kb/s; the generative model system at <NUM> kb/se with quantization; the generative model system at <NUM> kb/s with quantization and noise suppression; the generative model system at <NUM> kb/s with quantization and predictive variance regularization; the generative model system at <NUM> kb/s with quantization, predictive variance regularization, and noise suppression; and the EVS codec operating at <NUM>. As seen from <FIG>, the improvement of the generative model due to variance regularization is particularly large for clean signals. The effect of noise suppression varies in an unexpected manner with SNR, and likely results from an interaction between noise suppression and quantization, which may be related to noise suppression reducing signal variability and quantization reducing noise on its own. <FIG> illustrates that a <NUM> kb/s WaveGRU coder implementation (e.g., the generative model system at <NUM> kb/s with predictive variance regularization) performs significantly better than Opus at <NUM> kb/s and similarly to the EVS codec operating at <NUM> kb/s.

<FIG> shows an example of a computer device <NUM> and a mobile computer device <NUM>, which may be used with the techniques described here. Computing device <NUM> is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. Computing device <NUM> is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, and other similar computing devices.

Computing device <NUM> includes a processor <NUM>, memory <NUM>, a storage device <NUM>, a high-speed interface <NUM> connecting to memory <NUM> and high-speed expansion ports <NUM>, and a low-speed interface <NUM> connecting to low-speed bus <NUM> and storage device <NUM>. The processor <NUM> can process instructions for execution within the computing device <NUM>, including instructions stored in the memory <NUM> or on the storage device <NUM> to display graphical information for a GUI on an external input/output device, such as display <NUM> coupled to high-speed interface <NUM>.

The high-speed controller <NUM> manages bandwidth-intensive operations for the computing device <NUM>, while the low-speed controller <NUM> manages lower bandwidth-intensive operations.

In addition, an external interface <NUM> may be provide in communication with processor <NUM>, to enable near area communication of device <NUM> with other devices.

Such expansion memory <NUM> may provide extra storage space for device <NUM> or may also store applications or other information for device <NUM>. Specifically, expansion memory <NUM> may include instructions to carry out or supplement the processes described above and may include secure information also. Thus, for example, expansion memory <NUM> may be provide as a security module for device <NUM> and may be programmed with instructions that permit secure use of device <NUM>.

In addition, short-range communication may occur, such as using a Bluetooth, Wi-Fi, or other such transceiver (not shown).

Various implementations of the systems and techniques described here can be realized as and/or generally be referred to herein as a circuit, a module, a block, or a system that can combine software and hardware aspects. For example, a module may include the functions/acts/computer program instructions executing on a processor (e.g., a processor formed on a silicon substrate, a GaAs substrate, and the like) or some other programmable data processing apparatus.

Some of the above example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations as sequential processes, many of the operations may be performed in parallel, concurrently or simultaneously. In addition, the order of operations may be re-arranged. The processes may be terminated when their operations are completed but may also have additional steps not included in the figure. The processes may correspond to methods, functions, procedures, subroutines, subprograms, etc..

Methods discussed above, some of which are illustrated by the flow charts, may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine or computer readable medium such as a storage medium.

Specific structural and functional details disclosed herein are merely representative for purposes of describing example embodiments. Example embodiments, however, be embodied in many alternate forms and should not be construed as limited to only the embodiments set forth herein.

For example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term and/or includes any and all combinations of one or more of the associated listed items.

It will be understood that when an element is referred to as being connected or coupled to another element, it can be directly connected or coupled to the other element or intervening elements may be present. In contrast, when an element is referred to as being directly connected or directly coupled to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted in a like fashion (e.g., between versus directly between, adjacent versus directly adjacent, etc.).

As used herein, the singular forms a, an and the are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms comprises, comprising, includes and/or including, when used herein, specify the presence of stated features, integers, steps, operations, elements and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components and/or groups thereof.

For example, two figures shown in succession may in fact be executed concurrently or may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

Portions of the above example embodiments and corresponding detailed description are presented in terms of software, or algorithms and symbolic representations of operation on data bits within a computer memory. These descriptions and representations are the ones by which those of ordinary skill in the art effectively convey the substance of their work to others of ordinary skill in the art. An algorithm, as the term is used here, and as it is used generally, is conceived to be a self-consistent sequence of steps leading to a desired result. The steps are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of optical, electrical, or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated.

In the above illustrative embodiments, reference to acts and symbolic representations of operations (e.g., in the form of flowcharts) that may be implemented as program modules or functional processes include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types and may be described and/or implemented using existing hardware at existing structural elements. Such existing hardware may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific-integrated-circuits, field programmable gate arrays (FPGAs) computers or the like.

Unless specifically stated otherwise, or as is apparent from the discussion, terms such as processing or computing or calculating or determining of displaying or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical, electronic quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices.

Claim 1:
A method comprising:
receiving (S505) sampled audio data corresponding to utterances; and
training (S510) a machine learning, ML, model, using the sampled audio data, to generate an audio stream from an input bitstream, wherein the bitstream is generated based on the sampled audio data, wherein the training of the ML model includes de-emphasizing the influence of distortion events in the sampled audio data on the trained ML model,
the method being characterized in that
the de-emphasizing of the distortion events is achieved by the inclusion of a term in an objective function of the ML model, which term encourages low-variance predictive distributions of a next sample in the sampled audio data, based on previous samples of the audio data, wherein the term that encourages low-variance predictive distributions of a next sample in the sampled audio data includes a regularization term.