Patent Description:
Herein, we use the expression "smart audio device" to denote a smart device which is either a single purpose audio device or a virtual assistant (e.g., a connected virtual assistant). A single purpose audio device is a device (e.g., a TV or a mobile phone) including or coupled to at least one microphone (and optionally also including or coupled to at least one speaker), and/or at least one speaker (and optionally also including or coupled to at least one microphone), and which is designed largely or primarily to achieve a single purpose. Although a TV typically can play (and is thought of as being capable of playing) audio from program material, in most instances a modern TV runs some operating system on which applications run locally, including the application of watching television. Similarly, the audio input and output in a mobile phone may do many things, but these are serviced by the applications running on the phone. In this sense, a single purpose audio device having speaker(s) and microphone(s) is often configured to run a local application and/or service to use the speaker(s) and microphone(s) directly. Some single purpose audio devices may be configured to group together to achieve playing of audio over a zone or user configured area.

A virtual assistant (e.g., a connected virtual assistant) is a device (e.g., a smart speaker or voice assistant integrated device) including or coupled to at least one microphone (and optionally also including or coupled to at least one speaker) and which may provide an ability to utilize multiple devices (distinct from the virtual assistant) for applications that are in a sense cloud enabled or otherwise not implemented in or on the virtual assistant itself. Virtual assistants may sometimes work together, e.g., in a very discrete and conditionally defined way. For example, two or more virtual assistants may work together in the sense that one of them, i.e., the one which is most confident that it has heard a wakeword, responds to the word. Connected devices may form a sort of constellation, which may be managed by one main application which may be (or include or implement) a virtual assistant.

Herein, "wakeword" is used in a broad sense to denote any sound (e.g., a word uttered by a human, or some other sound), where a smart audio device is configured to awake in response to detection of ("hearing") the sound (using at least one microphone included in or coupled to the smart audio device, or at least one other microphone). In this context, to "awake" denotes that the device enters a state in which it awaits (i.e., is listening for) a sound command.

Herein, the expression "wakeword detector" denotes a device configured (or software, e.g., a lightweight piece of code, for configuring a device) to search (e.g., continuously) for alignment between realtime sound (e.g., speech) features and a pretrained model. Typically, a wakeword event is triggered whenever it is determined by a wakeword detector that wakeword likelihood (probability that a wakeword has been detected) exceeds a predefined threshold. For example, the threshold may be a predetermined threshold which is tuned to give a good compromise between rates of false acceptance and false rejection. Following a wakeword event, a device may enter a state (i.e., an "awakened" state or a state of "attentiveness") in which it listens for a command and passes on a received command to a larger, more computationally-intensive device (e.g., recognizer) or system.

An orchestrated system including multiple smart audio devices requires some understanding of the location of a user in order to at least: (a) select a best microphone for voice pickup; and (b) emit audio from sensible locations. Existing techniques include selecting a single microphone (which captures audio indicative of high wakeword confidence) and acoustic source localization algorithms using multiple synchronous microphones to estimate the coordinates of the user relative to the devices.

More generally, when training audio machine learning systems (e.g., wakeword detectors, voice activity detectors, speech recognition systems, speaker recognizers, or other speech analytics systems, and/or noise suppressors), especially those based on deep learning, it is often essential to augment the clean training dataset by adding reverberation, noise and other conditions that will be encountered by the system when running in the real world.

Speech analytics systems (for example, noise suppression systems, wakeword detectors, speech recognizers, and speaker (talker) recognizers) are often trained from a corpus of training examples. For example, a speech recognizer may be training from a large number of recordings of people uttering individual words or phrases along with a transcription or label of what was said.

In such training systems, it is often desirable to record clean speech (for example in a low-noise and low reverberation environment such as a recording studio or sound booth using a microphone situated close to the talker's mouth) because such clean speech corpora can be efficiently collected at scale. However, once trained, such speech analytics systems rarely perform well in real-world conditions that do not closely match the conditions under which the training set was collected. For example, the speech from a person speaking in a room in a typical home or office to a microphone located several metres away will typically be polluted by noise and reverberation.

In such scenarios it is also common that one or more devices (e.g., smart speakers) are playing music (or other sound, e.g., podcast, talkback radio, or phone call content) as the person speaks. Such music (or other sound) may be considered echo and may be cancelled, suppressed or managed by an echo management system that runs ahead of the speech analytics system. However, such echo management systems are not perfectly able to remove echo from the recorded microphone signal and echo residuals may be present in the signal presented to the speech analytics system.

Furthermore, speech analytics systems often need to run without complete knowledge of the frequency response and sensitivity parameters of the microphones. These parameters may also change over time as microphones age and as talkers move their location within the acoustic environment.

This can lead to a scenario where there is substantial mismatch between the examples shown to the speech analytics system during training and the actual audio shown to the system in the real world. These mismatches in noise, reverberation, echo, level, equalization and other aspects of the audio signal often reduce the performance of a speech analytics system trained on clean speech. It is often desirable, therefore to augment the clean speech training data during the training process by adding noise, reverberation and/or echo and by varying the level and/or equalisation of the training data. This is commonly known in speech technology as "multi-style training.

The conventional approach to multi-style training often involves augmenting PCM data to create new PCM data in a data preparation stage prior to the training process proper. Since the augmented data must be saved to disc, memory, etc., ahead of training, the diversity of the augmentation that can be applied is limited. For example, a <NUM> GB training set augmented with <NUM> different sets of augmentation parameters (e.g., <NUM> different room acoustics) will occupy <NUM> GB. This limits the number of distinct augmentation parameters that can be chosen and often leads to overfitting of the acoustic model to the particular set of chosen augmentation parameters leading to suboptimal performance in the real world.

Conventional multi-style training is usually done by augmenting the data in the time domain (for example by convolving with an impulse response) prior to the main training loop and often suffers from severe overfitting due to the limited number of augmented versions of each training vector that can be practically created.

<CIT> discloses a method comprising blending near field speech training data with far field speech training data to generate blended speech training data, wherein the far field speech training data is obtained by performing data augmentation processing for the near field speech training data; using the blended speech training data to train a deep neural network to generate a far field recognition acoustic model.

Throughout this disclosure, including in the claims, "speaker" and "loudspeaker" are used synonymously to denote any sound-emitting transducer (or set of transducers) driven by a single speaker feed. A typical set of headphones includes two speakers. A speaker may be implemented to include multiple transducers (e.g., a woofer and a tweeter), all driven by a single, common speaker feed (the speaker feed may undergo different processing in different circuitry branches coupled to the different transducers).

Throughout this disclosure including in the claims, the expression "system" is used in a broad sense to denote a device, system, or subsystem. For example, a subsystem that implements a decoder may be referred to as a decoder system, and a system including such a subsystem (e.g., a system that generates X output signals in response to multiple inputs, in which the subsystem generates M of the inputs and the other X - M inputs are received from an external source) may also be referred to as a decoder system.

Throughout this disclosure including in the claims, the term "processor" is used in a broad sense to denote a system or device programmable or otherwise configurable (e.g., with software or firmware) to perform operations on data (e.g., audio data). Examples of processors include a field-programmable gate array (or other configurable integrated circuit or chip set), a digital signal processor programmed and/or otherwise configured to perform pipelined processing on audio data, a graphics processing unit (GPU) configured to perform processing on audio data, a programmable general purpose processor or computer, and a programmable microprocessor chip or chip set.

Throughout this disclosure including in the claims, the term "couples" or "coupled" is used to mean either a direct or indirect connection. Thus, if a first device is said to be coupled to a second device, that connection may be through a direct connection, or through an indirect connection via other devices and connections.

Throughout this disclosure including in the claims, "audio data" denotes data indicative of sound (e.g., speech) captured by at least one microphone, or data generated (e.g., synthesized) so that said data are renderable for playback (by at least one speaker) as sound (e.g., speech) or are useful in training a speech analytics system (e.g., a speech analytics system which operates only in the band energy domain). For example, audio data may be generated so as to be useful as a substitute for data indicative of sound (e.g., speech) captured by at least one microphone. Herein, the expression "training data" denotes audio data which is useful (or intended for use) for training an acoustic model.

Throughout this disclosure including in the claims, the term "adding" (e.g., a step of "adding" augmentation to training data) is used in a broad sense which denotes adding (e.g., mixing or otherwise combining) and approximate implementations of adding.

Many embodiments of the present invention are technologically possible. It will be apparent to those of ordinary skill in the art from the present disclosure how to implement them. With reference to the Figures, we next describe examples of embodiments of the inventive system and method.

<FIG> is a diagram of an environment (a living space) which includes a system including a set of smart audio devices (devices <NUM>) for audio interaction, speakers (<NUM>) for audio output, and controllable lights (<NUM>). In an example, each of the devices <NUM> contains (and/or is coupled to) at least one microphone, so that the environment also includes the microphones, and the microphones provide devices <NUM> a sense of where (e.g., in which zone of the living space) is a user (<NUM>) who issues a wakeword command (a sound which the devices <NUM> are configured to recognize, under specific circumstances, as a wakeword). The system (e.g., one or more of devices <NUM> thereof) may be configured to implement an embodiment of the present invention. Using various methods, information may be obtained collectively from the devices of <FIG> and used to provide a positional estimate of the user who issues (e.g., speaks) the wakeword.

In a living space (e.g., that of <FIG>), there are a set of natural activity zones where a person would be performing a task or activity, or crossing a threshold. These action areas (zones) are where there may be an effort to estimate the location (e.g., to determine an uncertain location) or context of a user. In the <FIG> example, the key zones are.

In accordance with some embodiments of the invention, a system that estimates (e.g., determines an uncertain estimate of) where a signal (e.g., a wakeword or other signal for attention) arises or originates, may have some determined confidence in (or multiple hypotheses for) the estimate. , if a user happens to be near a boundary between zones of the system's environment, an uncertain estimate of location of the user may include a determined confidence that the user is in each of the zones. In some conventional implementations of voice interface (e.g., Alexa) it is required that the voice assistant's voice is only issued from one location at a time, this forcing a single choice for the single location (e.g., one of the eight speaker locations, <NUM> and <NUM>, in <FIG>). However, based on simple imaginary role play, it is apparent that (in such conventional implementations) the likelihood of the selected location of the source of the assistant's voice (i.e., the location of a speaker included in or coupled to the assistant) being the focus point or natural return response for expressing attention may be low.

<FIG> is a diagram of another environment (<NUM>) which is an acoustic space including a user (<NUM>) who utters direct speech <NUM>. The environment also includes a system including a set of smart audio devices (<NUM> and <NUM>), speakers for audio output, and microphones. The system may be configured in accordance with an embodiment of the invention. The speech uttered by user <NUM> (sometimes referred to herein as a talker) may be recognized by element(s) of the system as a wakeword.

More specifically, elements of the <FIG> system include:.

The <FIG> system may also include at least one speech analytics subsystem (e.g., the below-described system of <FIG> including classifier <NUM>) configured to perform speech analytics on (e.g., including by classifying features derived from) microphone outputs of the system (e.g., to indicate a probability that the user is in each zone, of a number of zones of environment <NUM>). For example, device <NUM> (or device <NUM>) may include a speech analytics subsystem, or the speech analytics subsystem may be implemented apart from (but coupled to) devices <NUM> and <NUM>.

<FIG> is a block diagram of elements of a system which may be implemented in accordance with embodiment of the invention (e.g., by implementing wakeword detection, or other speech analytics processing, with training in accordance with an embodiment of the invention). The <FIG> system (which includes a zone classifier) is implemented in an environment having zones, and includes:.

We next describe example implementations of zone classifier <NUM> of <FIG>.

Let xi(n) be the ith microphone signal, i = {<NUM>. N}, at discrete time n (i.e., the microphone signals xi(n) are the outputs of the N microphones <NUM>). Processing of the N signals xi(n) in echo management subsystem <NUM> generates 'clean' microphone signals ei(n), where i = {<NUM>. N}, each at a discrete time n. Clean signals ei(n), referred to as 203A in <FIG>, are fed to wakeword detectors <NUM>. Each wakeword detector <NUM> produces a vector of features wi(j), referred to as 206A in <FIG>, where j = {<NUM>. J} is an index corresponding to the jth wakeword utterance. Classifier <NUM> takes as input an aggregate feature set W(j) = <MAT>.

A set of zone labels Ck, for k = {<NUM>. K}, is prescribed to correspond to zones (a number, K, of different zones) in the environment (e.g., a room). For example, the zones may include a couch zone, a kitchen zone, a reading chair zone, etc..

In some implementations, classifier <NUM> estimates (and outputs signals indicative of) posterior probabilities p(Ck|W(j)) of the feature set W(j), for example by using a Bayesian classifier. Probabilities p(Ck|W(j)) indicate a probability (for the "j"th utterance and the "k"th zone, for each of the zones Ck, and each of the utterances) that the user is in each of the zones Ck, and are an example of output <NUM> of classifier <NUM>.

Typically, training data are gathered (e.g., for each zone) by having the user utter the wakeword in the vicinity of the intended zone, for example at the center and extreme edges of a couch. Utterances may be repeated several times. The user then moves to the next zone and continues until all zones have been covered.

An automated prompting system may be used to collect these training data. For example, the user may see the following prompts on a screen or hear them announced during the process:.

In the unlabeled case, training of the model implemented by classifier <NUM> (or training of another model in accordance with an embodiment of the invention) includes automatically splitting data into K clusters, where K may also be unknown. The unlabeled automatic splitting can be performed, for example, by using a classical clustering technique, e.g., the k-means algorithm or Gaussian Mixture Modelling.

In order to improve robustness, regularization may be applied to the model training (which may be performed in accordance with an embodiment of the inventive method) and model parameters may be updated over time as new utterances are made.

We next describe further aspects of examples in which an embodiment of the inventive method is implemented to train a model (e.g., a model implemented by element <NUM> of the <FIG> system).

An example feature set (e.g., features 206A of <FIG>, derived from outputs of microphones in zones of an environment) includes features indicative of the likelihood of wakeword confidence, mean received level over the estimated duration of the most confident wakeword, and maximum received level over the duration of the most confident wakeword. Features may be normalized relative to their maximum values for each wakeword utterance. Training data may be labeled and a full covariance Gaussian Mixture Model (GMM) trained to maximize expectation of the training labels. The estimated zone is the class that maximizes posterior probability.

The above description pertains to learning an acoustic zone model from a set of training data collected during a collection process (e.g., a prompted collection process). In that model, training time (operation in a configuration mode) and run time (operation in a regular mode) can be considered two distinct modes in which the microphones of the system may operate. An extension to this scheme is online learning, in which some or all of the acoustic zone model is learnt or adapted online (i.e., during operation in the regular mode).

An online learning mode may include steps of:.

Explicit techniques for obtaining feedback include:.

The goal of predicting the acoustic zone (in which the user is located) may be to inform a microphone selection or adaptive beamforming scheme that attempts to pick up sound from the acoustic zone of the user more effectively, for example, in order to better recognize a command that follows the wakeword. In such scenarios, implicit techniques for obtaining feedback on the quality of zone prediction may include:.

Techniques for the aposteriori updating of the zone mapping model after one or more wakewords have been spoken include:.

<FIG> is a block diagram that shows examples of components of an apparatus (<NUM>) that may be configured to perform at least some of the methods disclosed herein. In some examples, apparatus <NUM> may be or may include a personal computer, a desktop computer, a graphics processing unit (GPU), or another local device that is configured to provide audio processing. In some examples, apparatus <NUM> may be or may include a server. According to some examples, apparatus <NUM> may be a client device that is configured for communication with a server, via a network interface. The components of apparatus <NUM> may be implemented via hardware, via software stored on non-transitory media, via firmware and/or by combinations thereof. The types and numbers of components shown in <FIG>, as well as other figures disclosed herein, are merely shown by way of example. Alternative implementations may include more, fewer and/or different components.

Apparatus <NUM> of <FIG> includes an interface system <NUM> and a control system <NUM>. Apparatus <NUM> may be referred to as a system, and elements <NUM> and <NUM> thereof may be referred to as subsystems of such system. Interface system <NUM> may include one or more network interfaces, one or more interfaces between control system <NUM> and a memory system and/or one or more external device interfaces (e.g., one or more universal serial bus (USB) interfaces). In some implementations, interface system <NUM> may include a user interface system. The user interface system may be configured for receiving input from a user. In some implementations, user interface system may be configured for providing feedback to a user. For example, the user interface system may include one or more displays with corresponding touch and/or gesture detection systems. In some examples, the user interface system may include one or more microphones and/or speakers. According to some examples, the user interface system may include apparatus for providing haptic feedback, such as a motor, a vibrator, etc. Control system <NUM> may, for example, include a general purpose single- or multi-chip processor, a digital signal processor (DSP), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, and/or discrete hardware components. Control system <NUM> may also include one or more devices implementing non-transitory memory.

In some examples, apparatus <NUM> may be implemented in a single device. However, in some implementations, the apparatus <NUM> may be implemented in more than one device. In some such implementations, functionality of control system <NUM> may be included in more than one device. In some examples, apparatus <NUM> may be a component of another device.

In some embodiments, apparatus <NUM> is or implements a system for training an acoustic model, wherein the training includes a data preparation phase and a training loop which follows the data preparation phase, and wherein the training loop includes at least one epoch. In some such embodiments,.

According to some examples, elements <NUM>, <NUM>, <NUM>, and <NUM> of the <FIG> system, implemented in accordance with an embodiment of the invention, may be implemented via one or more systems (e.g., control system <NUM> of <FIG>). Similarly, elements of other embodiments of the invention (e.g., elements configured to implement the method described herein with reference to <FIG>) may be implemented via one or more systems (e.g., control system <NUM> of <FIG>).

With reference to <FIG>, we next describe an example of a conventional multi-style training method. <FIG> is a flowchart (100A) of a conventional multi-style training method for training acoustic model 114A. The method may be implemented by a programmed processor or other system (e.g., control system <NUM> of <FIG>), and steps (or phases) of the method may be implemented by one or more subsystems of the system. Herein, such a subsystem is sometimes referred to as a "unit" and the step (or phase) implemented thereby is sometimes referred to as a function.

The <FIG> method includes a data preparation phase 130A, and a training loop (training phase) 131A which is performed after the data preparation phase 130A. In the method, augmentation function (unit) 103A augments audio training data (<NUM>) during data preparation phase 130A.

Elements of <FIG> include the following:.

Next, with reference to <FIG>, we describe an example of a multi-style training method according to an embodiment of the present invention. The method may be implemented by a programmed processor or other system (e.g., control system <NUM> of <FIG>), and steps (or phases) of the method may be implemented by one or more subsystems of the system. Herein, such a subsystem is sometimes referred to as a "unit" and the step (or phase) implemented thereby is sometimes referred to as a function.

<FIG> is a flowchart (100B) of the multi-style training method for training acoustic model 114B. The <FIG> method includes data preparation phase 130B, and training loop (training phase) 131B which is performed after the data preparation phase 130B. In the method of <FIG>, augmentation function 103B augments audio training data (features 111B generated from training data <NUM>) during training loop 131B to generate augmented features 113B.

In contrast to the conventional approach of <FIG>, the augmentation (by function/unit 103B of <FIG>) is performed in the feature domain and during the training loop (phase 131B) rather than directly on input audio training data (<NUM>) in the data preparation phase (phase 130A of <FIG>). Elements of <FIG> include the following:.

Examples of types of augmentations that may be applied (e.g., by augmentation function 103B of <FIG> on training data features) in accordance with embodiments of the invention include (but are not limited to) the following:.

An embodiment of the invention, which includes fixed spectrum stationary noise augmentation, will be described with reference to <FIG>.

In the <FIG> example, training data (e.g., features 111B of <FIG>) are augmented (e.g., by function/unit 103B of <FIG>) by addition of fixed spectrum stationary noise addition thereto in accordance with the embodiment. Elements of <FIG> include the following:.

Another embodiment of the invention, which includes microphone equalization augmentation, will be described with reference to <FIG>. In the <FIG> example, training data (e.g., features 111B of <FIG>) are augmented (e.g., by function/unit 103B of <FIG>) by applying thereto, during each epoch of a training loop, a filter (e.g., a different filter for each different epoch) having a randomly chosen linear magnitude response. The characteristics of the filter (for each epoch) are determined from a randomly chosen microphone tilt (e.g., a tilt, in dB/octave, chosen from a normal distribution of microphone tilts). Elements of <FIG> include the following:.

With reference to <FIG>, we next describe another embodiment of augmentation applied to training data (e.g., by unit 103B of <FIG>) in accordance with the invention, in which the augmentation includes variable spectrum semi-stationary noise addition. In this embodiment, for each epoch (pass) of the training loop (or once, for use for plurality of successive epochs of the training loop), a Signal to Noise Ratio (SNR) is drawn from an SNR distribution (as in an embodiment employing fixed spectrum stationary noise addition). Also, for each epoch of the training loop (or once, for use for plurality of successive epochs of the training loop), a random stationary noise spectrum is selected from a distribution of noise spectrum shapes (for example, a distribution of linear slope values in dB/octave, or a distribution over DCT values of the log mel spectrum (cepstral)). For each epoch, an augmentation (i.e., noise, whose power as a function of frequency is determined from the chosen SNR and the chosen shape) is applied to each set of training data (e.g., each training vector). In some implementations, the shape of the noise is varied slowly over time (e.g., during one epoch) by, for example, choosing a rate of change for each cepstral value per second and using that to modulate the noise shape.

With reference to <FIG>, we next describe another embodiment of augmentation applied to training data (e.g., by unit 103B of <FIG>) in accordance with the invention, in which the augmentation includes non-stationary noise addition. Elements of <FIG> include the following:.

We next describe another embodiment of augmentation applied to training data (e.g., by unit 103B of <FIG>) in accordance with the invention. In this embodiment, the augmentation implements and applies a simplified reverberation model. The model is an improved (simplified) version of a band-energy domain reverberation algorithm described in above-referenced <CIT>. The simplified reverberation model has only two parameters: RT60, and Direct To Reverb Ratio (DRR). Mean free path and source distance are summarized into the DRR parameter.

With reference to <FIG>, we next describe time frequency tile classifier training pipeline <NUM> which is implemented in some embodiments of the invention. Training pipeline <NUM> (e.g., implemented in some embodiments of training loop 131B of <FIG>) augments training data (input features 128B) in a training loop of a multi-style training method and also generates class label data (<NUM>, <NUM>, <NUM>, and <NUM>) in the training loop. The augmented training data (<NUM>) and class labels (<NUM>-<NUM>) can be used (e.g., in the training loop) to train a model (e.g., model 114B of <FIG>) so that the trained model (e.g., implemented by classifier <NUM> of <FIG> or another classifier) is useful to classify time-frequency tiles of input features as speech, stationary noise, non-stationary noise, or reverberation. Such a trained model is useful for noise suppression (e.g., including by classifying time-frequency tiles of input features as speech, stationary noise, non-stationary noise, or reverberation, and suppressing unwanted non-speech sounds). <FIG> includes steps <NUM>, <NUM>, and <NUM> which are performed to augment the incoming training data at training time (for example in the "logmelspec" band energy domain on a GPU).

The class labels <NUM>-<NUM> can be compared with the model output (the output of the model being trained) in order to compute a loss gradient to backpropagate during training. A classifier (e.g., a classifier implementing model 114B of <FIG>) which has been (or is being) trained using the <FIG> scheme could, for example, include an element-wise softmax in its output (e.g., the output of prediction step 105B of the training loop of <FIG>) which indicates speech, stationary noise, nonstationary noise, and reverb probabilities in each time-frequency tile. These predicted probabilities could be compared with the class labels <NUM>-<NUM> using, for example, cross entropy loss and gradients backpropagated to update (e.g., in step 106B of the training loop of <FIG>) the model parameters.

<FIG> shows examples of four augmented training vectors (<NUM>, <NUM>, <NUM>, and <NUM>), each generated by applying a different augmentation to the same training vector (e.g., input features 128B of <FIG>) for use during a different training epoch of a training loop. Each of the augmented training vectors (<NUM>-<NUM>) is an example of augmented features <NUM> (of <FIG>), which has been generated for use during a different training epoch of a training loop which implements the <FIG> method.

Each of vectors <NUM>-<NUM> includes banded frequency components (in frequency bands) for each of a sequence of frames, with frequency indicated on the vertical axis and time indicated (in frames) on the horizontal axis. In <FIG>, scale <NUM> indicates how shades of (i.e., different degrees of brightness of different areas in) vectors <NUM>-<NUM> correspond to powers in dB.

We next describe an example of simulated echo residuals augmentation with reference to the following Julia <NUM> code listing ("Listing <NUM>"). When executed by a processor (e.g., a processor programmed to implement function 103B of <FIG>), the Julia <NUM> code of Listing <NUM> generates simulated echo residuals (music-like noise, determined using data values indicative of melody, tempo, and pitchiness, as indicated in the code) to be added to training data (e.g., features 111B of <FIG>) to be augmented. The residuals may then be added to frames of features (the training data to be augmented) to generate augmented features for use in one epoch of a training loop to train an acoustic model (e.g., an echo cancellation or echo suppression model). More generally, simulated music (or other simulated sound) residuals may be combined with (e.g., added to) training data to generate augmented training data for use in an epoch of a training loop (e.g., an epoch of training loop 131B of <FIG>) to train an acoustic model.

Generate a batch of synthesized music residuals to be combined with a batch of input speech by taking the element-wise maximum, where.

The following function generates a 3D array of residual band energies in dB of dimensions (nband, nframe, nvector).

We next describe another example of simulated echo residuals augmentation with reference to the following Julia <NUM> code listing ("Listing 1B"). When executed by a processor (e.g., a processor programmed to implement function 103B of <FIG>), the code of Listing 1B generates simulated echo residuals (synthesized music-like noise) to be added to training data (e.g., features 111B of <FIG>) to be augmented. The amount (magnitude) of simulated echo residuals is varied according to the position of an utterance in the training data (a training vector).

Generate a batch of synthesized music residuals to be combined with a batch of input speech by taking the element-wise maximum.

An example implementation of augmentation of training data by adding variable spectrum stationary noise thereto (e.g., as described above with reference to <FIG>) will be described with reference to the following Julia <NUM> code listing ("Listing <NUM>"). When executed by a processor (e.g., a processor programmed to implement function 103B of <FIG>), the code of Listing <NUM> generates stationary noise (having a variable spectrum), to be combined with (e.g., in step <NUM> of <FIG>) the unaugmented training data to generate augmented training data for use in training an acoustic model.

An example implementation of augmentation of training data by adding non-stationary noise thereto (e.g., as described above with reference to <FIG>) will be described with reference to the following Julia <NUM> code listing ("Listing <NUM>"). When executed by a processor (e.g., a GPU or other processor programmed to implement function 103B of <FIG>), the code of Listing <NUM> generates non-stationary noise, to be combined with (e.g., in step <NUM> of <FIG>) the unaugmented training data to generate augmented training data for use in training an acoustic model.

while t < size(x,<NUM>)
# draw a random event length
attack_time = min(t, <NUM>)
release_time = min(size(x,<NUM>)-t, <NUM>)
# choose different random cepstra for the event across the vectors in the batch
peak_cepstrum = xrandn(X, length(coef. cepstrum_dB_stddev), <NUM>, size(c,<NUM>)). cepstrum_dB_stddev. cepstrum_dB_mean. + cnep
attack_dcepstrum = (xrandn(X, length(coef. attack_cepstrum_dB_per_frame_stddev), <NUM>,
size(c,<NUM>)). attack_cepstrum_dB_per_frame_stddev. attack_cepstrum_dB_per_frame_mean)
release_dcepstrum = (xrandn(X, length(coef. release_cepstrum_dB_per_frame_stddev),
<NUM>, size(c,<NUM>)). release_cepstrum_dB_per_frame_stddev. release_cepstrum_dB_per_frame_mean)
# write the event into the cepstral buffer
batch_write_nonstationary_event!(c, peak_cepstrum, attack_dcepstrum,
release_dcepstrum, t, attack_time, release_time)
# draw a random time until next event
dt = max(round(Int, randn(X) * coef. dt_between_events_frames_stddev +
coef. dt_between_events_frames_mean), <NUM>)
t += dt
end
for v = <NUM>:size(x,<NUM>)
# transform cepstrum to spectrum
s[:,:,v] = coef. basis' * c[:,:,v]
end
s
end.

An example implementation of augmentation of training data (input features) to generate reverberant training data (as described above with reference to <FIG>) will be described with reference to the following Julia <NUM> code listing ("Listing <NUM>"). When executed by a processor (e.g., a GPU or other processor programmed to implement function 103B of <FIG>), the code of Listing <NUM> generates reverberant energy values to be combined with the unaugmented training data and combines (i.e., implements step <NUM> of <FIG>) the values with the training data to generate augmented training data for use in training an acoustic model.

Global parameters affecting all simulated reverb.

@kwdef struct ReverbDomain{P<:Real}
c_m_per_s::P
fsplit::P
end
Sensible defaults for simulating reverb in air. reverb_in_air(::Type{P}) where {P<:Real} = ReverbDomain(P(<NUM>), P(<NUM>)).

@kwdef struct BatchReverbParams{P<:Real} <MAT> <MAT> <MAT> <MAT> <MAT> <MAT>
end
struct BatchReverbCoef{X<:Real}
rt60_ms_derate::AbstractVector{X} # How many ms to derate RT60 at each
frequency band
dt ms::X
params::BatchReverbParams{X}
end
"'""
"'""
function BatchReverbCoef(params::BatchReverbParams{X},
fband::AbstractVector{X}, dt_ms::X) where {X<:Real}
rt60_ms_derate = [(f <= params. fsplit) ? X(0f0) : X(-<NUM>) for f in fband]
BatchReverbCoef(rt60_ms_derate, dt_ms, params)
end
"'"".

Draw random reverb parameters from distributions, return a reverberated version of 'x'. x is a 3D array describing the band energies in dB of a batch of training vectors in which:.

This function returns a tuple (y,mask) where y is a 3D array of reverberated band energies of the same size as x. Mask is a 3D array of the same size as y which is:.

function batch_reverb_mask(coef::BatchReverbCoef{X}, x::AbstractArray{X,<NUM>},
rng::AbstractRNG) where {X<:Real}
rt60_ms = max. (X(<NUM>), coef. rt60_ms_mean. + randn(rng, X,
size(x,<NUM>))*coef. rt60_ms_stddev)
drr_dB = max. (X(<NUM>), coef. drr_dB_mean. + randn(rng, X,
size(x,<NUM>))*coef. drr_dB_stddev)
batch_rt60_ms = rt60_ms'. rt60_ms_derate
feedback = X(-<NUM>). (batch_rt60_ms, X(<NUM>)))
noise_dB = randn(rng, X, size(x)). noise_dB_stddev
y = similar(x)
<IMG>.

Aspects of the invention include a system or device configured (e.g., programmed) to perform any embodiment of the inventive method, and a tangible computer readable medium (e.g., a disc) which stores code for implementing any embodiment of the inventive method or steps thereof. For example, the inventive system can be or include a programmable general purpose processor, digital signal processor, or microprocessor, programmed with software or firmware and/or otherwise configured to perform any of a variety of operations on data, including an embodiment of the inventive method or steps thereof. Such a general purpose processor may be or include a computer system including an input device, a memory, and a processing subsystem that is programmed (and/or otherwise configured) to perform an embodiment of the inventive method (or steps thereof) in response to data asserted thereto.

Some embodiments of the inventive system are implemented as a configurable (e.g., programmable) digital signal processor (DSP) or graphics processing unit (GPU) that is configured (e.g., programmed and otherwise configured) to perform required processing on audio signal(s), including performance of an embodiment of the inventive method or steps thereof. Alternatively, embodiments of the inventive system (or elements thereof) are implemented as a general purpose processor (e.g., a personal computer (PC) or other computer system or microprocessor, which may include an input device and a memory) which is programmed with software or firmware and/or otherwise configured to perform any of a variety of operations including an embodiment of the inventive method. Alternatively, elements of some embodiments of the inventive system are implemented as a general purpose processor, or GPU, or DSP configured (e.g., programmed) to perform an embodiment of the inventive method, and the system also includes other elements (e.g., one or more loudspeakers and/or one or more microphones). A general purpose processor configured to perform an embodiment of the inventive method would typically be coupled to an input device (e.g., a mouse and/or a keyboard), a memory, and a display device.

Another aspect of the invention is a computer readable medium (for example, a disc or other tangible storage medium) which stores code for performing (e.g., coder executable to perform) any embodiment of the inventive method or steps thereof.

Claim 1:
A method of training an acoustic model (114B), wherein the training includes a data preparation phase (130B) implemented on a data preparation subsystem, and a training loop (131B) implemented on a training subsystem, wherein the training loop (131B) follows the data preparation phase (130B) and includes a plurality of successive epochs, said method including:
in the data preparation phase (130B):
providing time domain training data (<NUM>), wherein the training data (<NUM>) are or include at least one example of audio data,
extracting at least one feature (111B) of the time domain training data (<NUM>), and
providing the at least one feature (111B) to the training subsystem;
during each epoch of the training loop:
augmenting the at least one feature (111B), wherein the augmentation occurs in at least one feature domain, thereby generating augmented training data (113B) which is temporary and only used during training in one epoch; and
using at least some of the augmented training data (113B) to train the model (114B),
wherein the training data (<NUM>) are indicative of features comprising frequency bands, and the augmentation occurs in the frequency domain, and
wherein augmenting the at least one feature comprises drawing a random Signal to Noise Ratio, SNR, from an SNR distribution (<NUM>) and drawing a random stationary noise spectrum shape (<NUM>) from a distribution of noise spectrum shapes, and determining the augmentation as noise whose power as a function of frequency is determined from the drawn random SNR and the drawn random stationary noise spectrum.