Patent Description:
Examples of machine learning models include support vector machines, decision forests, linear models, and neural networks.

<NPL>, discloses training a bank of complex filters that operates on a raw waveform and is fed into a convolutional neural network for end-to-end phone recognition. These time-domain filterbanks (TD-filterbanks) are initialized as an approximation of mel-filterbanks, and then fine-tuned jointly with the remaining convolutional architecture.

<NPL>, discloses a frontend called per-channel energy normalization (PCEN). PCEN uses an automatic gain control based dynamic compression to replace static (such as log or root) compression.

This specification describes a system implemented as computer programs on one or more computers in one or more locations that processes an audio signal, i.e., an audio waveform that includes a sequence of audio samples, e.g., amplitude values, using a machine learning model to generate an output for an audio understanding task.

The audio understanding task can be any task that requires processing an audio waveform to generate a prediction that characterizes the audio waveform.

Many existing neural network systems that perform audio understanding tasks use mel-filterbank representations of input audio waveforms. Mel-filterbanks are fixed, engineered audio features which emulate human perception. However, mel-filterbanks are limited by the fundamental limitations of manually engineered representations. This specification describes a single, universal learnable frontend that outperforms mel-filterbanks over a wide range of audio domains, including speech, music, audio events and animal sounds, providing a general purpose learned frontend for audio. In particular, this specification describes a lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-filterbanks. In other words, this learned audio frontened model has few parameters and, because these parameters are learned, can result in improved performance with minimal computational overhead, both during training and at inference.

For example, if the input is an audio sequence, i.e., an audio waveform, representing a spoken utterance, the audio understanding task can be speech recognition, e.g., with an output that defines a score for each of a set of pieces of text, each score representing an estimated likelihood that the piece of text is the correct transcript for the utterance.

As another example, the audio understanding task may be a keyword spotting task where, if the input is an audio sequence representing a spoken utterance, the output can indicate whether a particular word or phrase ("hotword") was spoken in the utterance.

As another example, the audio understanding task can be a language identification task where, if the input is an audio sequence representing a spoken utterance, the output generated by the trainee neural network can identify the natural language in which the utterance was spoken.

The audio understanding task can also be a task that is performed on audio that is not speech. Examples of such audio understanding tasks include music recognition tasks that receive an audio sequence as input and generate a prediction of the name or other property of a piece of music that is represented in the input, animal classification tasks that receive an audio input that includes one or more animal sounds and generate a prediction of a category of animal that is making the animal sounds, audio event classification tasks in which the input is an audio sequence is a prediction of, for each of multiple different sounds, at what temporal instances the sound is active within the audio sequence, and so on.

More generally, while this specification describes a frontend that can replace a mel-filterbank representation when processing an audio signal, the frontend can also instead replace a manually engineered or other learned representation of a different kind of input signal, e.g., seismic data or physiological recordings.

<FIG> shows an example audio processing system <NUM>. The audio processing system <NUM> is an example of a system implemented as computer programs on one or more computers in one or more locations, in which the systems, components, and techniques described below can be implemented.

The audio processing system <NUM> obtains an audio waveform <NUM> as input. The audio waveform <NUM> is a sequence of audio samples, e.g., amplitude values, at a first frequency ("sampling frequency").

The audio processing system <NUM> processes the audio waveform <NUM> using a learned audio frontend model <NUM> to generate a feature representation <NUM> of the audio waveform.

Generally, the audio frontend model <NUM> is a machine learning model that is configured to apply a learned filtering operation that has a plurality of filtering parameters, a learned pooling operation that has a plurality of pooling parameters, and a learned normalization operation that has a plurality of normalization parameters to generate the feature representation <NUM> of the audio waveform <NUM>.

The operations performed by the audio frontend model <NUM> to generate the feature representation <NUM> are described in more detail below with reference to <FIG>.

The system <NUM> processes the feature representation <NUM> using an audio understanding machine learning model <NUM>. The audio understanding machine learning model <NUM> is a machine learning model that has parameters ("audio understanding parameters") that is configured to process the feature representation <NUM> to generate a respective output <NUM> for each of one or more audio understanding tasks, e.g., for one or more of the tasks described above or for a different task that requires making a prediction about the content of the audio waveform <NUM>.

The audio understanding model <NUM> can be any appropriate model, e.g., one that was previously configured to receive a mel-filterbank representation of an audio signal as input. That is, as a particular example, the audio frontend model <NUM> can replace, in an audio processing pipeline, a system that maps an audio waveform into a mel-filterbank representation of the audio waveform that is provided as input to the audio understanding model <NUM>.

Particular examples of audio understanding models that can receive as input the feature representation <NUM> include convolutional neural networks, e.g., those having an EfficientNet architecture, fully-connected neural networks, e.g., a multi-task neural network that has a respective set of linear layers for each of the multiple tasks, recurrent neural networks, e.g., long-short term memory (LSTM) or gated recurrent unit (GRU) based neural networks, or self-attention neural networks, e.g., Transformer neural networks.

The audio frontend model <NUM> is referred to as a "learned" audio frontend model because the values of the parameters of the learned audio frontend model, i.e., the values of the filtering, pooling, and normalization parameters, are learned end-to-end with the audio understanding model <NUM>. In other words, the operations performed by the audio frontend model <NUM> are entirely differentiable, allowing the audio frontend model <NUM> to be trained jointly with a "backend" model through gradient descent. This is unlike other prominent representations, e.g., mel-filterbank representations, that are hard-coded and are therefore not able to be fine-tuned to improve the performance of a given model <NUM> on a given set of one or more audio processing tasks.

In particular, the system <NUM> includes a training engine <NUM> that trains the audio frontend model <NUM> and the audio understanding model <NUM> on respective training data for each of the one or more audio understanding tasks.

The training data for a given task includes a set of training audio inputs and, for each audio input, a target output for the given task that should be generated by the audio understanding model <NUM> by processing a feature representation for the training audio input that is generated by the audio frontend model <NUM>.

The training engine <NUM> trains the models <NUM> and <NUM> on the training data for a given task through gradient descent and, in particular, trains the audio frontend model <NUM> by backpropagating gradients of a loss function for the given task through the audio understanding model <NUM> and into the audio frontend model <NUM>. The loss function can be any appropriate loss function for the task that measures the performance of the by the audio understanding model <NUM> on the given task given the target outputs in the training data for the given task, e.g., a classification loss if the audio understanding task is a classification task or a regression loss if the audio understanding task is a regression task. In particular, the system can train the models <NUM> and <NUM> to determine trained values of the parameters θ of the audio understanding model <NUM> and the parameters ψ of the audio frontend model <NUM> through any appropriate gradient descent based technique, e.g., stochastic gradient descent, Adam optimization, rmsProp optimization, and so on, to minimize the following: <MAT> where x is a training audio input sampled from the training data set D, y is the target output for the training input, Fψ(x) is the representation generated by the model <NUM> by processing the training audio input x, gθ(Fψ(x)) is the output generated by the model <NUM> by processing the representation, and L is the loss function for the given task.

In some cases, the training engine <NUM> can pre-train the audio frontend model <NUM> on one or more tasks jointly with an original backend model and then train the audio frontend model <NUM> with a new backend model on a different task, e.g., while keeping the pre-trained values of the parameters of the model <NUM> fixed and updating the parameters of the new backend model or updating the parameters of the new backend model while also fine-tuning the pre-trained values of the parameters of the model <NUM>. For example, this can be done in cases where there is a large amount of training data available for the one or more tasks that the original backend model performs while only a limited amount of training data is available for the task(s) that the new backend model performs.

<FIG> is a flow diagram of an example process <NUM> for processing an audio input. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, an audio processing system, e.g., the audio processing system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system obtains an audio waveform (step <NUM>). The audio waveform is a sequence of audio samples at a first frequency. That is, the audio waveform is a sequence that has a respective scalar audio sample at each of T first time steps. In other words, the audio waveform is a one-dimensional waveform of T samples available at a sampling frequency Fs Hz.

The system processes the audio waveform using a learned audio frontend model to generate a feature representation of the audio waveform (step <NUM>).

The feature representation includes a sequence of features at a second frequency. Generally, the second frequency is lower than the first frequency.

More specifically, the operations of the learned audio frontend can be represented as a function with parameters that have been learned through the training of the learned audio frontend and that maps the audio waveform from a one-dimensional space to a <NUM>-dimensional feature space, i.e., that maps the T scalar values in the audio waveform into a <NUM>-dimensional M x N representation using learned parameters, where M denotes the number of temporal frames ("features") and N denotes the number of feature channels in each temporal frame ("feature") of the representation. Because the second frequency will generally be lower than the first frequency, the number of temporal frames M will generally be less than the number T of audio samples in the audio waveform.

Processing the audio waveform using the learned audio frontend model to generate the feature representation is described in more detail below with reference to <FIG> and <FIG>.

The system processes the feature representation using an audio understanding machine learning model having a plurality of audio understanding parameters to generate a respective output for each of one or more audio understanding tasks as described above (step <NUM>).

<FIG> is a flow diagram of an example process <NUM> for generating a feature representation of an audio waveform using a learned audio frontend model. For convenience, the process <NUM> will be described as being performed by a system of one or more computers located in one or more locations. For example, an audio processing system, e.g., the audio processing system <NUM> of <FIG>, appropriately programmed, can perform the process <NUM>.

The system applies a learned filtering operation having a plurality of filtering parameters to the audio waveform to generate a filtered representation that includes a sequence of filtered features at the first frequency (step <NUM>).

At a high level, the learned filtering operation passes the audio waveform through a bank of bandpass filters followed by a non-linearity, operating at the original sampling frequency Fs.

In other words, the sequence of filtered features includes a respective filtered feature at each of the first time steps in the audio waveform and, therefore, each filtered feature in the sequence has a corresponding audio sample in the audio waveform.

Generally, each filtered feature includes a respective value for each of a plurality of channels. That is, the learned filtering operation maps the T scalar audio samples in the input waveform to T filtered features that each have N feature channels, where N is greater than one.

To generate the values for any given channel of the filtered features, the system applies one or more one-dimensional convolutional filters to the audio waveform to generate the values of the filtered features for the channel.

In some implementations, for each channel, the system can apply a single one-dimensional convolutional filter with stride <NUM> to the audio waveform to generate the values of the filtered features for the channel.

As a particular example, each convolutional filter can be a complex-valued filter and the system can generate the T values fn of the filtered features for the n-th channel by performing operations that satisfy: <MAT> where x is the audio waveform, φn is the complex-valued filter for the channel n, and the * operation denotes convolution with stride <NUM>. Applying the squared modulus operator brings the output of the convolution back to the real-valued domain.

In some other implementations, for each channel, the system can apply multiple one-dimensional convolutional filters with stride <NUM> to the audio waveform to generate the values of the filtered features for the channel.

As a particular example, for each channel, the system can apply a plurality of one-dimensional convolutional filters to the audio waveform to generate a plurality of convolved values and then combine the convolved values to generate the values for the channel.

More specifically, the system can apply <NUM>N total real-valued one-dimensional filters to generate the T values fn of the filtered features for the n-th channel by performing operations that satisfy: <MAT> where φ̃<NUM>n-<NUM> and φ̃<NUM>n are two of the 2N total real-valued one-dimensional filters that are used for computing the values of the n-th channel. For example, φ̃<NUM>n-<NUM> and φ̃<NUM>n can be the real and imaginary parts of a single complex-valued one-dimensional filter. Applying the <NUM>N total filters in this manner can generate an output that is equivalent to the output generated using the complex-valued filters above, without requiring the system to explicitly manipulate complex numbers.

In some implementations, the system does not place any constraints on the values ("coefficients") of the one-dimensional filters that are learned during training, i.e., the one-dimensional filters are standard 1D convolutions. For example, the system can initialize the coefficients of the filters to approximate the computation of a mel-filterbank and then adjust the filters during training through backpropagation without placing any constraints on any of the coefficients in the filters.

In some of these implementations, to prevent instability during training, the system applies normalization, e.g., l2 normalization, to the filter coefficients before computing the convolutions.

In some implementations, the system does place constrains on the coefficients of the one-dimensional filters. In particular, the system can require that each of the plurality of one-dimensional convolutional filters be a Gabor filter.

Generally, Gabor filters are produced by modulating a Gaussian kernel with a sinusoidal signal. Formally, a Gabor filter φn having length W is parametrized by its center frequency ηn and inverse bandwidth σn as follows: <MAT>.

That is, the equation above defines the W coefficients of a Gabor filter with length W. Thus, the frequency response of the Gabor filter φn is a Gaussian centered at frequency ηn and of bandwidth <NUM>/σn, both expressed in normalized frequency units in [-<NUM>/<NUM>,<NUM>/<NUM>].

For each of the N Gabor filters, the system only needs to learn two parameter values (the center frequency and inverse bandwidth) and learning these two parameter values for each filter allows learning a bank of smooth, quasi-analytic bandpass filters, with controllable center frequency and bandwidth.

Thus, Gabor filters have significantly fewer parameters than normalized 1D-convolutions. For example, N Gabor filters of length W are fully specified by 2N parameters, i.e., N for the center frequencies and N for the bandwidths, against W x N for a standard 1D-convolution. As a particular example, when using a window length of <NUM> and operating at a sampling rate of <NUM>, then W = <NUM> samples, and Gabor-based filtering accounts for <NUM> times fewer parameters than their unconstrained alternatives. This can make training significantly less computationally intensive and significantly reduce the memory footprint of the learned frontend without decreasing the performance of the frontend.

To apply the Gabor filters, the system obtains the impulse response of each of the N Gabor filters over the range t= -W/<NUM>,. , to W/<NUM> and then convolves these impulse responses to the input waveform.

Alternatively, the system can apply the Gabor filters using 2N real valued filters rather than N complex valued Gabor filters as described above, i.e., by splitting real and imaginary part of the Gabor filter, convolving inputs with each one independently, and then merging the result using squared l2-pooling.

In some cases, to assist with stabilizing training, the system clips the center frequencies of the N Gabor filters to be in a predefined range, e.g., [<NUM>, ½] so that they lie in the positive part of the frequency range. The system can optionally also constrain the center frequency to be in a predefined range, e.g., the range <MAT> such that the full-width at half-maximum of the frequency response is within <NUM>/W and ½.

The system applies a learned pooling operation having a plurality of pooling parameters to the filtered representation to generate a pooled representation that includes a sequence of pooled features at the second frequency (step <NUM>). Each pooled feature has a respective value for each of the plurality of channels.

At a high level, the learned pooling operation decimates the signal represented by the filtered representation to reduce the temporal resolution of the filtered representation.

More specifically, the system implements the learned pooling operation through depthwise convolution with lowpass filters. Thus, each input channel is associated with one lowpass filter.

In other words, for each channel, the system performs the learned pooling operation by applying, with stride greater than one, a respective learned lowpass filter for the channel to the values in the filtered features for the channel to generate a set of pooled values for the channel that have the second frequency. By having the stride be greater than one, the system reduces the number of values in each channel, thereby reducing the temporal resolution of the filtered representation.

The system parameterizes the respective learned lowpass filter for each channel to have a Gaussian impulse response. That is, the learned lowpass filter φn having length W for channel n can satisfy: <MAT> Thus, the system can learn per-channel lowpass pooling functions while adding only N parameters to the frontend model.

The system applies a learned normalization operation having a plurality of normalization parameters to the pooled representation to generate the feature representation (step <NUM>).

At a high level, the learned normalization operation normalizes the pooled representation and then applies a non-linearity to the pooled representation to reduce the dynamic range of the pooled representation.

In particular, for each respective value for each of the plurality of channels of each pooled feature, the system normalizes the respective value in accordance with one or more of the normalization parameters and then applies a non-linearity to the normalized respective value in accordance with one or more of the normalization parameters to generate the corresponding value in the feature representation.

As a particular example, the system can apply a non-linearity to the normalized respective value by adding a learned offset to the normalized value to generate a sum and applying compression to the sum using a learned exponent. For example, the system can learn a respective offset and a respective exponent for each channel.

As another particular example, for each respective value for each of the plurality of channels of each pooled feature, the system can normalize the respective value using an exponential moving average of the values for the channel in any pooled features that are before the pooled feature in the sequence. The exponential moving average is controlled by a respective smoothing coefficient.

For example, the exponential moving average M(t,n) for a value F(t,n) of channel n in the feature with index t can be recursively defined as: <MAT> where sn is the smoothing coefficient for channel n.

Thus, a particular example of the learned normalization operation can satisfy <MAT> where ε is a constant value, αn is a learned exponent for the normalization for the channel n, δn is the learned offset for channel n, and rn is the learned exponent for channel n. Thus, in this particular example, the system learns a respective value of sn, αn, δn and rn for each of the N channels.

<FIG> illustrates an example of the operation of the learned audio frontend model.

As shown in <FIG>, the audio waveform <NUM> that has T audio samples is first processed using a learned filtering operation to generate a filtered representation, i.e., a sequence of filtered features that includes T filtered features that each have N channels. The learned filtering operation first applies N learned Gabor filters <NUM> of length W to the input audio waveform <NUM> and then applies a squared modulus operator <NUM> to the output generate a filtered representation.

The filtered representation is then processed using a learned pooling operation to generate a pooled representation. The learned pooling operation applies a respective Gaussian lowpass filter <NUM> to each channel of the filtered representation to reduce the temporal resolution of the filtered representation and generate the pooled representation.

The pooled representation is then processed using a learned normalization operation to generate the feature representation. The learned normalization operation applies the sPCEN operations <NUM> described above to generate the feature representation.

The feature representation can then be provided to a downstream audio processing neural network for processing, as described above.

Claim 1:
A method performed by one or more computers, the method comprising:
obtaining (<NUM>) an audio waveform comprising a sequence of audio samples at a first frequency;
processing (<NUM>) the audio waveform using a machine-learned audio frontend model to generate a feature representation of the audio waveform, wherein the feature representation comprises a sequence of features at a second frequency, and wherein the learned audio frontend model is configured to:
apply (<NUM>) a machine-learned filtering operation having a plurality of filtering parameters to the audio waveform to generate a filtered representation comprising a sequence of filtered features at the first frequency;
apply (<NUM>) a machine-learned pooling operation having a plurality of pooling parameters to the filtered representation to generate a pooled representation comprising a sequence of pooled features at the second frequency; and
apply (<NUM>) a machine-learned normalization operation having a plurality of normalization parameters to the pooled representation to generate the feature representation; and
processing (<NUM>) the feature representation using an audio understanding machine learning model having a plurality of audio understanding parameters to generate a respective output for each of one or more audio understanding tasks;
wherein each filtered feature has a respective value for each of a plurality of channels, and wherein applying a machine-learned pooling operation having a plurality of pooling parameters to the filtered representation to generate a pooled representation comprising a sequence of pooled features at the second frequency comprises:
for each channel, applying, with stride greater than one, a respective learned lowpass filter for the channel to the values in the filtered features for the channel to generate a set of pooled values for the channel that have the second frequency; and
wherein the respective learned lowpass filter for each channel has a Gaussian impulse response.