Patent Description:
Conventional speech enhancement introduces artifacts into the enhanced audio and thus degrade the performance of live captioning (i.e., automatic speech recognition (ASR) for audio transcriptions). Thus, when noise-suppression is used on audio, the performance of online real-time captioning is degraded. Conventional approaches also use parallel simulated data (noisy and corresponding clean speech signals) to train the model. Simulated data limits the ability of the speech enhancement model to adapt to real-world situations.

In view of the foregoing, there is an ongoing need for improved systems and methods for generating training data and for training models, including the deployment of such models, for improved speech enhancement.

Xiao Xiong et al: "On time-frequency mask estimation for MVDR beamforming with application in robust speech recognition", <NUM> IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), IEEE, <NUM> March <NUM> describes acoustic beamforming has played a key role in the robust automatic speech recognition (ASR) applications. Accurate estimates of the speech and noise spatial covariance matrices (SCM) are crucial for successfully applying the minimum variance distortionless response (MVDR) beamforming. Reliable estimation of time-frequency (TF) masks can improve the estimation of the SCMs and significantly improve the performance of the MVDR beamforming in ASR tasks. The authors focus on the TF mask estimation using recurrent neural networks (RNN). Specifically, the methods include training the RNN to estimate the speech and noise masks independently, training the RNN to minimize the ASR cost function
directly, and performing multiple passes to iteratively improve the mask estimation. The proposed methods are evaluated individually and overall on the CHiME-<NUM> challenge. The results show that the proposed methods improve the ASR performance individually and also work complementarily. The overall performance achieves a word error rate of <NUM>% with <NUM>-microphone configuration, which is much better than <NUM>% achieved with the state-of-the-art MVDR implementation.

<NPL> describes deep learning based speech separation approaches have received great interest, among which the recent speaker-aware speech enhancement methods are promising for solving difficulties such as arbitrary source permutation and unknown number of sources. In this paper, the authors propose a novel training framework which jointly learns the speaker-conditioned target speaker extraction model and its associated speaker embedding model. The resulting unified model directly learns the appropriate speaker embedding for improved target speech enhancement. The authors demonstrate, on a large simulated noisy and far-field evaluation sets of overlapped speech signals, that the proposed approach significantly improves the speech enhancement performance compared to the baseline speaker-aware speech enhancement models.

<NPL> describes speech enhancement has benefited from the success of deep learning in terms of intelligibility and perceptual quality. Conventional time-frequency (TF) domain methods focus on predicting TF-masks or speech spectrum, via a naive convolution neural network (CNN) or recurrent neural network (RNN). Some recent studies use complex-valued spectrogram as a training target but train in a real-valued network, predicting the magnitude and phase component or real and imaginary part, respectively. Particularly, convolution recurrent network (CRN) integrates a convolutional encoder-decoder (CED) structure and long short-term memory (LSTM), which has been proven to be helpful for complex targets. In order to train the complex target more effectively, in this paper, the authors design a new network structure simulating the complex-valued operation, called Deep Complex Convolution Recurrent Network (DCCRN), where both CNN and RNN structures can handle complex-valued operation. The proposed DCCRN models are competitive over other previous networks, either on objective or subjective metric. With only <NUM> parameters, the DCCRN models submitted to the Interspeech <NUM> Deep Noise Suppression (DNS) challenge ranked first for the real-time-track and second for the non-real-time track in terms of Mean Opinion Score (MOS).

<NPL> describes a novel multichannel end-to-end speech recognition architecture that integrates the components of multichannel speech enhancement and speech recognition into a single neural-network-based architecture and demonstrated its fundamental utility for automatic speech recognition (ASR). However, the behavior of the proposed integrated system remains insufficiently clarified. An open question is whether the speech enhancement component really gains speech enhancement (noise suppression) ability, because it is optimized based on end-to-end ASR objectives instead of speech enhancement objectives. In this paper, the authors solve this question by conducting systematic evaluation experiments using the CHiME-<NUM> corpus. First show that the integrated end-to-end architecture successfully obtains adequate speech enhancement ability that is superior to that of a conventional alternative (a delay-and-sum beamformer) by observing two signal-level measures: the signal-to distortion ratio and the perceptual evaluation of speech quality. The findings suggest that to further increase the performances of an integrated system, boost the power of the latter-stage speech recognition component. However, an insufficient amount of multichannel noisy speech data is available. Based on these situations, they next investigate the effect of using a large amount of single-channel clean speech data, e.g., the WSJ corpus, for additional training of the speech recognition component. They also show that the approach with clean speech significantly improves the total performance of multichannel end-to-end architecture in the multichannel noisy ASR tasks.

Disclosed embodiments include systems, methods and devices used for training machine learning models to generate noise-suppressed speech outputs that have been optimized for retaining signal quality and for producing corresponding speech transcriptions.

Some disclosed systems are configured to obtain both a speech enhancement model trained on a first training dataset to generate noise-suppressed speech outputs and an automatic speech recognition model trained on a second training dataset to generate transcription labels for spoken language utterances. The systems also apply a third training dataset that comprises a set of spoken language utterances to the speech enhancement model to obtain a first noise-suppressed speech output. Subsequently, the systems also apply the first noise-suppressed speech output from the speech enhancement model to the automatic speech recognition model to generate a noise-suppressed transcription output for the set of spoken language utterances.

Disclosed systems are also configured to obtain ground truth transcription labels for the set of spoken language utterances included in the third training dataset and to update one or more speech enhancement model parameters to optimize the speech enhancement model(s) to generate optimized noise-suppressed speech outputs based on a comparison of the noise-suppressed transcription output and the ground truth transcription labels.

Disclosed systems are configured to utilize multi-task training framework for facilitating speech enhancement. In such configurations, for example, systems are configured to first obtain or generate a speech enhancement model trained with a first training dataset to generate noise-suppressed speech outputs, as well as to obtain or generate an automatic speech recognition model trained with a second training dataset to generate transcription labels for spoken language utterances.

The systems also obtain or utilize a third training dataset comprising (i) a first data subset comprising a set of spoken language utterances and ground truth transcription labels corresponding to the set of spoken language utterances and (ii) a second data subset comprising noisy audio data and clean reference audio data corresponding to the noisy audio data. Then, the set of spoken language utterances included in the third training dataset is applied to the speech enhancement model to generate or obtain a first noise-suppressed speech output. Subsequently, the first noise-suppressed speech output from the speech enhancement model is applied to the automatic speech recognition model to generate a noise-suppressed transcription output for the set of spoken language utterances. The noisy audio data included in the third training dataset is also applied to the speech enhancement model to obtain a second noise-suppressed speech output.

Finally, the systems are further configured to update one or more speech enhancement model parameters to optimize the speech enhancement model to generate optimized noise-suppressed speech outputs based on (i) a first comparison of the noise-suppressed transcription output and the ground truth transcription labels to facilitate a first improvement in generating noise-suppressed transcription outputs and (ii) a second comparison of the second noise-suppressed speech output and the clean reference audio data to facilitate a second improvement in generating noise-suppressed speech outputs.

Disclosed systems are also configured to use an optimized speech enhancement model to generate noise-suppressed speech outputs and corresponding noise-suppressed speech transcriptions. In such configurations, the systems obtain electronic content comprising an audio stream comprising spoken language utterances. The systems also access a speech enhancement model that is trained on a first training dataset and optimized to generate optimized noise-suppressed speech outputs by updating one or more speech enhancement model parameters based on a first comparison of noise-suppressed transcription outputs obtained from applying noise-suppressed speech outputs to an automatic speech recognition model and ground truth transcription labels corresponding to the noise-suppressed speech outputs. Finally, the systems generate an optimized noise-suppressed speech output based on providing and/or applying a portion of the audio stream (as input) to the speech enhancement model.

Disclosed embodiments are directed towards improved systems, methods and frameworks for training and utilizing machine learning models to perform optimized speech enhancement. The disclosed embodiments include systems and methods that are specifically configured to implement and perform joint or multi-task training of speech enhancement models to facilitate simultaneous human listening and live captioning.

More particularly, some of the disclosed embodiments are directed to improved training frameworks for optimizing deep learning based speech enhancement models for both improved audio communication and transcription quality by leveraging pre-trained ASR models. The disclosed frameworks are configured to build and/or utilize SE models that achieve superior ASR performance while retaining the same or better speech qualities as SE models trained only for SE objectives (i.e., retaining signal quality during audio enhancement).

The disclosed embodiments provide many technical advantages over existing systems, including the ability to leverage in-domain real-world noisy recordings which do not need the corresponding clean speech signals to optimize the SE network. This ability is very beneficial and can significantly improve the quality of live captioning, for example. The training data used by the disclosed systems comprises a mix of simulated and non-simulated audio data that can be further used to adapt SE models to particular applications, e.g., Microsoft's Teams video interfacing application.

Attention will now be directed to <FIG>, which illustrates components of a computing system <NUM> which may include and/or be used to implement aspects of the disclosed invention. As shown, the computing system includes a plurality of machine learning (ML) engines, models, neural networks and data types associated with inputs and outputs of the machine learning engines and models.

Attention will be first directed to <FIG>, which illustrates the computing system <NUM> as part of a computing environment <NUM> that also includes third-party system(s) <NUM> in communication (via a network <NUM>) with the computing system <NUM>. The computing system <NUM> is configured to train and optimize a speech enhancement model, along with an automatic speech recognition model to generate noise-suppressed speech outputs that are optimized for human listening and live captioning. The computing system <NUM> is also configured to operate machine learning models, including an optimized speech enhancement model.

The computing system <NUM>, for example, includes one or more processor(s) (such as one or more hardware processor(s)) <NUM> and a storage (i.e., hardware storage device(s) <NUM>) storing computer-readable instructions <NUM> wherein one or more of the hardware storage device(s) <NUM> is able to house any number of data types and any number of computer-readable instructions <NUM> by which the computing system <NUM> is configured to implement one or more aspects of the disclosed embodiments when the computer-readable instructions <NUM> are executed by the one or more processor(s) <NUM>. The computing system <NUM> is also shown including user interface(s) <NUM> and input/output (I/O) device(s) <NUM>.

As shown in <FIG>, hardware storage device(s) <NUM> is shown as a single storage unit. However, it will be appreciated that the hardware storage device(s) <NUM> is, a distributed storage that is distributed to several separate and sometimes remote systems and/or third-party system(s) <NUM>. The computing system <NUM> can also comprise a distributed system with one or more of the components of computing system <NUM> being maintained/run by different discrete systems that are remote from each other and that each perform different tasks. In some instances, a plurality of distributed systems performs similar and/or shared tasks for implementing the disclosed functionality, such as in a distributed cloud environment.

The hardware storage device(s) <NUM> are configured to store the different data types including simulated audio data <NUM>, natural audio data <NUM>, ground truth transcription labels <NUM>, clean reference audio data <NUM>, noise-suppressed speech output <NUM>, and noise-suppressed transcription outputs <NUM>, described herein.

The storage (e.g., hardware storage device(s) <NUM>) includes computer-readable instructions <NUM> for instantiating or executing one or more of the models and/or engines shown in computing system <NUM> (e.g., speech enhancement model <NUM> and/or automatic speech recognition model <NUM>). The models are configured as machine learning models or machine learned models, such as deep learning models and/or algorithms and/or neural networks. In some instances, the one or more models are configured as engines or processing systems (e.g., computing systems integrated within computing system <NUM>), wherein each engine (i.e., model) comprises one or more processors (e.g., hardware processor(s) <NUM>) and computer-readable instructions <NUM> corresponding to the computing system <NUM>.

Simulated audio data <NUM> comprises a mixture of simulated clean speech (e.g., clean reference audio data <NUM>) and one or more of: room impulse responses, isotropic noise, or ambient or transient noise for any particular actual or simulated environment. Thus, parallel clean audio data and noisy audio data is generated using the clean reference audio data <NUM> on the one hand, and a mixture of the clean reference audio data <NUM> and background noise data. Simulated noisy speech data is also generated by distorting the clean reference audio data <NUM>.

Natural audio data <NUM> comprises spoken language utterances without a corresponding clean speech reference signal. Natural audio data <NUM> is recorded from a plurality of sources, including applications, meetings comprising one or more speakers, ambient environments including background noise and human speakers, etc..

Ground truth transcription labels <NUM> comprise phoneme labeling for audio data, in particular the natural audio data <NUM> and/or the simulated audio data <NUM>. The ground truth transcription labels <NUM> are text transcriptions that correspond to spoken language utterances included in the natural audio data <NUM> or other audio data. The ground truth transcription labels <NUM> are obtained via human transcription or machine transcription (e.g., speech-to-text machine learning models).

Noise-suppressed speech outputs <NUM> comprise audio data that is obtained from a speech enhancement model. For example, when simulated audio data <NUM> and/or natural audio data <NUM> are applied to a speech enhancement model, any non-human speech is suppressed or removed to provide audio data or speech output that is enhanced (i.e., noise-suppressed). The present invention is directed to generating noise-suppressed speech outputs <NUM> that are optimized for generating noise-suppressed transcription outputs <NUM> that correspond to the noise-suppressed speech outputs <NUM>. The noise-suppressed transcription outputs <NUM> comprise phoneme labeling or speech-to-text data for spoken language utterances included in the noise-suppressed speech outputs <NUM>. The noise-suppressed transcription outputs <NUM> are used for live-captioning during automatic speech recognition and speech enhancement tasks.

An additional storage unit for storing machine learning (ML) Engine(s) <NUM> is presently shown in <FIG> as storing a plurality of machine learning models and/or engines. For example, computing system <NUM> comprises one or more of the following: a data retrieval engine <NUM>, a training engine <NUM>, a backpropagation engine <NUM>, an updating engine <NUM>, and an implementation engine <NUM>, which are individually and/or collectively configured to implement the different functionality described herein.

For example, the data retrieval engine <NUM> is configured to locate and access data sources, databases, and/or storage devices comprising one or more data types from which the data retrieval engine <NUM> can extract sets or subsets of data to be used as training data. The data retrieval engine <NUM> receives data from the databases and/or hardware storage devices, wherein the data retrieval engine <NUM> is configured to reformat or otherwise augment the received data to be used as training data. Additionally, or alternatively, the data retrieval engine <NUM> is in communication with one or more remote systems (e.g., third-party system(s) <NUM>) comprising third-party datasets and/or data sources. In some instances, these data sources comprise visual services that record or stream text, images, and/or video.

The data retrieval engine <NUM> accesses electronic content comprising simulated audio data <NUM>, natural audio data <NUM>, ground truth transcription labels <NUM>, clean reference audio data <NUM>, noise-suppressed speech output <NUM>, and noise-suppressed transcription outputs <NUM> and/or other types of audio-visual data including video data, image data, holographic data, <NUM>-D image data, etc. The data retrieval engine <NUM> is a smart engine that is able to learn optimal dataset extraction processes to provide a sufficient amount of data in a timely manner as well as retrieve data that is most applicable to the desired applications for which the machine learning models/ engines will be trained. For example, the data retrieval engine <NUM> can learn which databases and/or datasets will generate training data that will train a model (e.g., for a specific query or specific task) to increase accuracy, efficiency, and efficacy of that model in the desired audio data processing techniques.

The data retrieval engine <NUM> locates, selects, and/or stores raw recorded source data wherein the data retrieval engine <NUM> is in communication with one or more other ML engine(s) and/or models included in computing system <NUM>. In such instances, the other engines in communication with the data retrieval engine <NUM> are able to receive data that has been retrieved (i.e., extracted, pulled, etc.) from one or more data sources such that the received data is further augmented and/or applied to downstream processes. For example, the data retrieval engine <NUM> is in communication with the training engine <NUM> and/or implementation engine <NUM>.

The data retrieval engine <NUM> is configured to retrieve training datasets comprising simulated audio data (i.e., a mixture of clean speech and room impulse responses, isotropic noise, or transient noise), non-simulated audio data including spoken language utterances without corresponding clean speech reference signals, speech data for a target domain corresponding to a target enterprise, a target speaking context, or a particular target user.

The training engine <NUM> is in communication with one or more of the data retrieval engine <NUM>, the backpropagation engine <NUM>, updating engine <NUM>, or the implementation engine <NUM>. In such embodiments, the training engine <NUM> is configured to receive one or more sets of training data from the data retrieval engine <NUM>. After receiving training data relevant to a particular application or task, the training engine <NUM> trains one or more models on the training data. The training engine <NUM> is configured to train a model via unsupervised training and/or supervised training. The training engine <NUM> is configured to train one or more machine learning models various datasets, including simulated audio data <NUM> and natural audio data <NUM>.

The training engine <NUM> is configured to train a speech enhancement model on a training dataset comprising simulated parallel noisy and clean audio data to generate noise-suppressed speech outputs and train an automatic speech recognition model on a training dataset comprising simulated and/or natural audio data and corresponding ground truth transcription labels to generate transcription labels for spoken language utterances included in the simulated and/or natural audio data. The training engine <NUM> is also configured to apply a training dataset comprising a set of spoken language utterances to the speech enhancement model to obtain a noise-suppressed speech output. The training engine <NUM> also trains the speech enhancement model on a training dataset comprising noisy audio data and clean reference audio data to minimize the signal quality loss during generation of the optimized noise-suppressed speech outputs.

The backpropagation engine <NUM> is configured to perform backpropagation of the cross-entropy loss to the speech enhancement model <NUM> based on a comparison of the noise-suppressed transcription outputs <NUM> and the ground truth transcription labels <NUM>. The cross-entropy loss is also calculated based on one or more labeling errors identified in the noise-suppressed transcription outputs <NUM>. The backpropagation engine <NUM> is also configured to perform backpropagation of the phase-aware loss based on a comparison of the noise-suppressed speech outputs <NUM> and the clean reference audio data <NUM> corresponding to the noisy audio data used to obtain the noise-suppressed speech outputs <NUM>.

The updating engine <NUM> is configured to update one or more speech enhancement model parameters to optimize the speech enhancement model to generate optimized noise-suppressed speech outputs that retain signal quality and produce optimized noise-suppressed transcription outputs when applied to an automatic speech recognition model <NUM>. The updating engine <NUM> is configured to update one or more speech enhancement model parameters based on a comparison of the noise-suppressed transcription outputs <NUM> and the ground truth transcription labels <NUM>. The updating engine <NUM> minimizes the signal quality loss during generation of the optimized noise-suppressed speech outputs based on a comparison of the noise-suppressed speech outputs <NUM> and clean reference audio data <NUM>.

The updating engine <NUM> is configured to update one or more speech enhancement model parameters by adjusting a probability parameter corresponding to a frequency at which the speech enhancement model is updated. The probability parameter also determines whether the phase-aware loss or the cross-entropy loss is backpropagated to facilitate the updating of the speech enhancement model parameters. The one or more speech enhancement model parameters are also updated to optimize (i) a word error rate of the noise-suppressed transcription outputs and (ii) a mean opinion score of the noise-suppressed speech outputs.

As part of the joint-training framework, the one or more speech enhancement model parameters are updated by identifying one or more labeling errors in the noise-suppressed transcription outputs, calculating a cross-entropy loss based on the one or more labeling errors identified in the noise-suppressed transcription outputs, performing a backpropagation of the cross-entropy loss to the speech enhancement model, the one or more speech enhancement model parameters being updated to optimize the noise-suppressed speech outputs to minimize an occurrence of labeling errors in the noise-suppressed transcription outputs, calculating a phase-aware loss based on a second comparison of noise-suppressed speech outputs and clean reference audio data corresponding to noisy audio data used to obtain the noise-suppressed speech outputs, and performing a backpropagation of the phase-aware loss to the speech enhancement model, the one or more speech enhancement model parameters being updated to optimize the noise-suppressed speech outputs to minimize a signal quality loss in the noise-suppressed speech outputs.

The updating engine <NUM> is also configured to freeze a set of internal layers of the automatic speech recognition model prior to updating the speech enhancement model. After obtaining the speech enhancement model and the automatic speech recognition model, but prior to updating the speech enhancement model parameters, the updating engine <NUM> is configured to concatenate the trained speech enhancement model and the trained automatic speech recognition model.

The computing system <NUM> includes an implementation engine <NUM> in communication with any one of the models and/or ML engine(s) <NUM> (or all of the models/engines) included in the computing system <NUM> such that the implementation engine <NUM> is configured to implement, initiate or run one or more functions of the plurality of ML engine(s) <NUM>. In one example, the implementation engine <NUM> is configured to operate the data retrieval engines <NUM> so that the data retrieval engine <NUM> retrieves data at the appropriate time to be able to generate training data for the training engine <NUM>. The implementation engine <NUM> facilitates the process communication and timing of communication between one or more of the ML engine(s) <NUM> and is configured to implement and operate a machine learning model (or one or more of the ML engine(s) <NUM>) which is configured as a speech enhancement model <NUM>.

The implementation engine <NUM> is configured to operate a speech enhancement model to generate optimized noise-suppressed speech output based on an input audio stream and operate an automatic speech recognition model to generate an optimized noise-suppressed transcription output based on the optimized noise-suppressed speech output from the speech enhancement model.

In some instances, the speech enhancement model <NUM> is configured as a personalized speech enhancement model. While speech enhancement models are trained to remove background noise and keep only the human speech from a noisy audio signal, when there is more than one speaker, unconditional models keep all the speaker's voices. To prevent leaking of sensitive material transmitted over the audio and/or to keep only one or a select few of the speakers, the speech enhancement model is configurable to keep speech from one or more target speakers.

A speaker embedding vector is extracted from a user's enrollment data and used to lock onto the user's voice and filter all other audio sources. In a first stage, a speaker identification/ verification model is used to extract the embedding vector of the target speaker and used as a condition to the SE systems. For example, the embedding vector is appended to an internal layer input of the SE model by mapping the tensor shape of the embedding vector to the particular internal layer. The seed SE model is trained to be speaker independent initially, wherein only <NUM> to <NUM> seconds of audio from the target speaker is required to achieve the personalization. Multiple embedding vectors for multiple target speakers can be appended to the SE model such that the SE model retains human speech from the multiple target speakers. With regard to the foregoing, it will be appreciated that different types of speech enhancement models can also be used, as well as customized models, to provide the desired speech enhancement results. In other words, the described multi-task training can be applied to all kinds of neural network-based speech enhancement models.

The computing system is in communication with third-party system(s) <NUM> comprising one or more processor(s) <NUM>, one or more of the computer-readable instructions <NUM>, and one or more hardware storage device(s) <NUM>. It is anticipated that, in some instances, the third-party system(s) <NUM> further comprise databases housing data that could be used as training data, for example, audio data not included in local storage. Additionally, or alternatively, the third-party system(s) <NUM> include machine learning systems external to the computing system <NUM>. The third-party system(s) <NUM> are software programs or applications.

Attention will now be directed to <FIG>, in some reference to <FIG>, which illustrates an example embodiment for training machine learning models for improved speech enhancement and speech recognition. <FIG> shows the probability parameter <NUM> which determines the type and frequency of each training iteration using either ASR training data <NUM> or NS training data <NUM>. The ASR training data <NUM> comprises natural audio data <NUM> and ground truth transcription labels <NUM>, without clean reference audio data <NUM>. Input audio data <NUM> is applied to the speech enhancement or noise-suppression model (e.g., NS model <NUM>) which produces "S" output <NUM>. A feature extraction <NUM> is applied to "S" output <NUM> prior to being used as input to the ASR model (e.g., Seq2Seq model <NUM>) which produces "Y" output <NUM>. "Y" output is used to generate labels <NUM> for spoken language utterances includes in the input audio data <NUM>.

Based on a cross-entropy loss calculated from a comparison of the ASR training data <NUM> and the labels <NUM>, the system performs ASR loss backpropagation <NUM> to update one or more parameters corresponding to the NS model <NUM> to improve the word error rate of the labels <NUM>. During the ASR backpropagation <NUM>, a set of internal layers of the Seq2Seq model <NUM> are frozen, such that only the NS model <NUM> is updated. The Seq2Seq model <NUM> is previously trained on ASR training data <NUM> to perform speech recognition and generate transcription labels for input audio data.

The NS training data <NUM> comprises simulated parallel noisy and clean speech audio data without ground truth transcription labels. Input audio <NUM> is applied to the NS model <NUM> (which is shared between the ASR and NS training iterations) to produce "S" output <NUM> which is configured as clean speech <NUM> (i.e., enhanced and/or noise-suppressed speech output). Based on a comparison of the clean speech <NUM> and the NS training data <NUM>, the system performs NS loss backpropagation <NUM> to update one or more parameters of the NS model <NUM> to minimize signal quality loss. In either the ASR or NS training iterations, the NS model <NUM> is previously trained on NS training data <NUM> to generate noise-suppressed speech outputs.

Alternating between the ASR training and NS training iteration steps, the NS model <NUM> is simultaneously optimized to generate optimized noise-suppressed speech outputs that retain signal quality during noise-suppression for improved human listening and live-captioning (e.g., automatic speech recognition tasks). For the NS training iteration step, clean speech and noisy speech samples are mixed and paired to evaluate a signal difference-based loss function. The SE model parameter gradients with respect to this loss function are computed to update the speech enhancement model parameters (e.g., layers of the NS model <NUM>).

During the ASR training iteration step, noisy training samples in a mini batch are fed to the SE network (e.g., NS model <NUM>). The generated enhanced signals are input to the ASR network (e.g., Seq2Seq model <NUM>). A loss function is evaluated by comparing the ASR model output and the reference transcriptions. The loss is back-propagated all the way down to the SE network, and only the SE model parameters are updated. The objective is to find SE model parameter values that work for existing well-trained ASR systems; thus, the ASR network should not be adapted to the characteristics of the SE model.

This multi-task approach beneficially provides the advantage of using real noisy speech samples that only need reference transcriptions for the SE model training. At each training iteration step, the update step to be used is chosen randomly from a Bernoulli distribution. The probability of choosing the SE-step is the "SE-step probability. " The probability parameter <NUM> is the parameter that determines the frequency and timing of each iteration step. Attention will now be directed to <FIG>, which illustrates a novel embodiment of an exemplary speech enhancement model framework. It will be appreciated that the speech enhancement model trained and operated in methods and systems described herein is configured as a DCCRN model (as shown in <FIG>), a PNS, DCUNET, or any other seed or customized models.

<FIG> shows an exemplary speech enhancement model (e.g., NS model <NUM>) configured as a DCCRN model <NUM> including a real short time Fourier transform layer (STFT <NUM>), a complex encoder layer <NUM>, a complex unified long short term memory layer (Complex Uni-LSTM layer <NUM>), an FC layer <NUM>, a complex decoder layer <NUM>, a concatenation layer <NUM>, and an imaginary short time Fourier transform layer (ISTFT <NUM>).

The DCCRN model <NUM> is configured to take noisy input audio <NUM> and generate noise-suppressed output audio <NUM>. As shown in <FIG>, the DCCRN model <NUM> takes the real and imaginary parts of a noisy spectrum as input and estimates a complex ratio mask and applies it to the noisy speech. The masked signal is converted back to the time domain with the ISTFT <NUM>.

Attention will now be directed to <FIG>, with some reference to <FIG> and <FIG>, which illustrates a novel embodiment of a process flow diagram for generating an optimized speech enhancement model.

A first training dataset <NUM> (e.g., NS training data <NUM>) is used to train (see step 1A) the speech enhancement model <NUM> having a plurality of speech enhancement model parameters <NUM> to generate noise-suppressed speech outputs. The first training dataset <NUM> comprises a large-scale and high-quality simulated dataset, which includes around <NUM>,<NUM> hours of paired noisy and clean speech samples. The mixtures are created from using non-stationary noise recordings, internal noise recordings, and colored stationary noise as noise sources. In addition, the clean speech in each mixture is convolved with acoustic room impulse response (RIR) sampled from measured and simulated responses.

A second training dataset <NUM> (e.g., ASR training data <NUM>) is used to train (see step 1B) an automatic speech recognition model <NUM> to generate transcription outputs for spoken language utterances included in input audio. The second training dataset <NUM> comprises anonymized and transcribed spoken language utterances, totally <NUM>,<NUM> hours. It should be appreciated that any amount sufficient to effectively train the ASR model may be used.

A third training dataset <NUM> is used to train the speech enhancement model <NUM> to generate an optimized speech enhancement model <NUM> by performing alternating training iterations using the third training data subset 412A and the third training data subset 412B. Third training data subset 412A comprises noisy natural (and/or simulated) audio data and corresponding ground truth transcription labels without corresponding clean reference audio data. The third training data subset 412A comprises one or more subsets (e.g., subset 408A and/or subset 408B) of the second training dataset <NUM>. The particular subsets to be included in the third training dataset <NUM> are selected, in part, based on criteria such as in-domain vs. out-of-domain and including vs. excluding simulated data. The simulated data included in the first training dataset for the SE training is different from the simulated/augmented data included in the second training dataset for the ASR training.

Third training data subset 412B comprises parallel noisy and clean audio data. This is simulated audio data because strict alignment is required between the parallel noisy and clean audio data for effective training of the speech enhancement model to minimize signal quality loss. The third training data subset 412B comprises one or more subsets (e.g., subset 402A and/or subset 402B) of the first training dataset <NUM>. The third training data subset 412B is used to update one or more of the speech enhancement model parameters <NUM> (e.g., see updated speech enhancement model parameters <NUM>) such that the optimized speech enhancement model <NUM> is configured to generate optimized noise-suppressed speech outputs that are optimized for signal quality retention, as well as optimized downstream ASR tasks which are described below.

The third training dataset <NUM> (or more specifically, third training data subset 412A) comprises a set of spoken language utterances which are applied (see step <NUM>) to the speech enhancement model <NUM> to obtain (see step <NUM>) a noise-suppressed speech output <NUM>. The noise-suppressed speech output <NUM> is applied (see step <NUM>) to the automatic speech recognition model <NUM> to generate (see step <NUM>) a noise-suppressed transcription output <NUM>. Ground truth transcription labels <NUM> are obtained for the set of spoken language utterances included in the third training data subset 412A. The noise-suppressed transcription output <NUM> is compared (see step <NUM>) against the ground truth transcription labels <NUM>. From this comparison, a cross-entropy loss is calculated and backpropagated to the speech enhancement model <NUM>. The automatic speech recognition model is frozen to prevent the cross-entropy loss backpropagation from updating the automatic speech recognition model <NUM>.

Based on this comparison, one or more of the speech enhancement model parameters <NUM> are updated to optimize the speech enhancement model <NUM> in order to generate (see step <NUM>) an optimized speech enhancement model <NUM>. The optimized speech enhancement model <NUM> is configured to generate (see step <NUM>) optimized noise-suppressed speech outputs <NUM> based on an input audio stream <NUM> which can subsequently be applied (see step <NUM>) to the automatic speech recognition model <NUM> to generate optimized noise-suppressed transcription outputs <NUM> (see step <NUM>).

Attention will now be directed to <FIG>, which illustrates a graph showing improvements in a speech enhancement model after employing disclosed training methods. The optimized speech enhancement model is evaluated using both simulated and real test data and using various evaluation techniques including PESQ, STOI, SDR, and pMOS metrics. Specifically, as referenced in <FIG>, pMOS <NUM> is a neural network based non-intrusive MOS (mean opinion score) estimator that shows high correlations with human MOS ratings without requiring reference signals.

<FIG> shows various evaluation ratings for seed SE models and their multi-task trained versions for simulated and real recordings graphed based on the pMOS score and the ASR performance (measured by word error rate percentage, i.e., WER% score <NUM>). During an initial passthrough <NUM> of the audio data, the ASR performance yielded a low WER% which is desirable, but a low pMOS score which is undesirable. To improve the pMOS score, the audio was processed by the SE seed model <NUM>. However, while the pMOS score was raised significantly with this SE seed model <NUM>, the SE seed model <NUM> was degraded in that the ASR performance also resulted in a relatively higher WER%.

After training with methods such as those disclosed herein (e.g., the multi-task training), the audio processed by an optimized SE model (e.g., SE Model <NUM> and/or SE model <NUM>) was found to have a significantly improved WER% as compared to the SE seed model <NUM>, while also having a relatively higher pMOS score. In some instances, in the audio processed by SE model <NUM>, the pMOS score was degraded slightly, while also lowering the WER%, thus showing a trade-off between the ASR and SE quality.

Accordingly, it will be appreciated that it is possible to implement different modifications and multi-task training optimizations, as desired, by adjusting the SE-step probability based on different application needs. For instance, performing the ASR-step more frequently will result in ASR performance improvement at the expense of pMOS scores compared to the SE Seed model, whereas a moderate SE-step probability is optimal for serving both human listening and live captioning tasks.

The foregoing results also show that the stronger ASR back-end models used for generating the optimized SE models are more effective in closing the WER gap while preserving the SE improvement. It should be noted that a strong ASR model is trained on natural audio data training datasets described herein, along with training datasets including clean signals. The mixture of simulated and natural audio data provides beneficial acoustic diversity in terms of noise and reverberation conditions. A mixture of in-domain and out-of-domain data also led to improved WER% scores.

Attention will now be directed to <FIG> which illustrates a flow diagram <NUM> that includes various acts (act <NUM>, act <NUM>, act <NUM>, act <NUM>, act <NUM>, and act <NUM>) associated with exemplary methods that can be implemented by computing system <NUM> for obtaining training data, training, and updating a machine learning model for optimizing a speech enhancement model.

The first illustrated act includes an act of obtaining a speech enhancement model trained on a first training dataset to generate noise-suppressed speech outputs (act <NUM>). The computing system also obtains an automatic speech recognition model trained on a second training dataset to generate transcription labels for spoken language utterances (act <NUM>). A third training dataset comprising a set of spoken language utterances is applied to the speech enhancement model to obtain a first noise-suppressed speech output (act <NUM>). Subsequently, the first noise-suppressed speech output from the speech enhancement model to the automatic speech recognition model to generate a noise-suppressed transcription output for the set of spoken language utterances (act <NUM>).

The computing system obtains ground truth transcription labels for the set of spoken language utterances included in the third training dataset (act <NUM>). Finally, one or more speech enhancement model parameters are updated to optimize the speech enhancement model to generate optimized noise-suppressed speech outputs based on a first comparison of the noise-suppressed transcription output and the ground truth transcription labels (act <NUM>).

Prior to updating the one or more speech enhancement model parameters, freezing a set of internal layers of the automatic speech recognition model.

After obtaining the speech enhancement model and the automatic speech recognition model but prior to updating the one or more speech enhancement model parameters, concatenating the speech enhancement model and the automatic speech recognition model.

The computing system also obtains a fourth training dataset comprising noisy audio data and clean reference audio data corresponding to the noisy audio data, applies the noisy audio data to the speech enhancement model to obtain a second noise-suppressed speech output, and updates the one or more speech enhancement model parameters to minimize signal quality loss during generation of the optimized noise-suppressed speech outputs based on a second comparison of the second noise-suppressed speech output and the clean reference audio data. The fourth training dataset comprises a subset of the first training dataset.

The computing system is also configured to obtain user enrollment data comprising a speaker embedding vector corresponding to a target speaker, extract the speaker embedding vector corresponding to the target speaker, and personalize the speech enhancement model to the target speaker by appending the speaker embedding vector to an internal layer of the speech enhancement model to configure the speech enhancement model to remove background noise and non-target speaker speech in order to generate personalized noise-suppressed speech outputs.

The speech enhancement model configured as a deep complex convolution recurrent network for phase-aware speech enhancement comprising one or more short time Fourier transform layers, a complex. The automatic speech recognition model configured as a sequence-to-sequence model using an attention-based encoder-decoder structure.

The first training dataset comprising simulated data comprising a mixture of clean speech and one or more of: room impulse responses, isotropic noise, or transient noise. the second training dataset comprising non-simulated audio data comprising spoken language utterances without a corresponding clean speech reference signal. Alternatively, the second training dataset comprising non-simulated audio data and simulated audio data. The third training dataset comprising a subset of the second training dataset. The third training dataset comprising speech data for a target domain corresponding to one or more of: a target enterprise or a target speaking context, or a particular target user.

The computing system is configured to update the one or more speech enhancement model parameters by adjusting a probability parameter corresponding to a frequency at which the speech enhancement model is updated. Additionally, the computing system is configured to update the one or more speech enhancement model parameters to optimize (i) a word error rate of the noise-suppressed transcription output and (ii) a mean opinion score of the first noise-suppressed speech output.

Attention will now be directed to <FIG> which illustrates a flow diagram <NUM> that includes various acts (act <NUM>, act <NUM>, act <NUM>, act <NUM>, act <NUM>, act <NUM>, and act <NUM>) associated with exemplary methods that can be implemented by computing system <NUM> for obtaining training data and training a machine learning model for updating a speech enhancement model.

The first illustrated act includes an act of obtaining a speech enhancement model trained on a first training dataset to generate noise-suppressed speech outputs (act <NUM>). The computing system also obtains an automatic speech recognition model trained on a second training dataset to generate transcription labels for spoken language utterances (act <NUM>) and obtains a third training dataset comprising (i) a first data subset comprising a set of spoken language utterances and ground truth transcription labels corresponding to the set of spoken language utterances and (ii) a second data subset comprising noisy audio data and clean reference audio data corresponding to the noisy audio data (act <NUM>).

The set of spoken language utterances included in the third training dataset is applied to the speech enhancement model to obtain a first noise-suppressed speech output (act <NUM>) which is then applied to the automatic speech recognition model to generate a noise-suppressed transcription output for the set of spoken language utterances (act <NUM>). The computing system also applies the noisy audio data included in the third training dataset to the speech enhancement model to obtain a second noise-suppressed speech output (act <NUM>).

Finally, the computing system updates one or more speech enhancement model parameters to optimize the speech enhancement model to generate optimized noise-suppressed speech outputs based on alternating between (i) a first comparison of the noise-suppressed transcription output and the ground truth transcription labels to facilitate a first improvement in generating noise-suppressed transcription outputs and (ii) a second comparison of the second noise-suppressed speech output and the clean reference audio data to facilitate a second improvement in generating noise-suppressed speech outputs (act <NUM>).

Attention will now be directed to <FIG> which illustrates a flow diagram <NUM> that includes various acts (act <NUM>, act <NUM>, and act <NUM>) associated with exemplary methods that can be implemented by computing system <NUM> for obtaining training data and training a machine learning model for updating a speech enhancement model.

The first illustrated act includes an act of obtaining electronic content comprising an audio stream comprising spoken language utterances (act <NUM>). The computing system then accesses a speech enhancement model that is trained on a first training dataset and optimized to generate optimized noise-suppressed speech outputs by updating one or more speech enhancement model parameters based on a first comparison of noise-suppressed transcription outputs obtained from applying noise-suppressed speech outputs to an automatic speech recognition model and ground truth transcription labels corresponding to the noise-suppressed speech outputs (act <NUM>). Subsequently, the computing system operates the speech enhancement model to generate an optimized noise-suppressed speech output based on a portion of the audio stream (act <NUM>).

The computing system also accesses the automatic speech recognition model that is trained on a second training dataset to generate transcription labels for spoken language utterances and operates the automatic speech recognition model to generate an optimized noise-suppressed transcription output based on the optimized noise-suppressed speech output from the speech enhancement model.

The one or more speech enhancement model parameters are updated by identifying one or more labeling errors in the noise-suppressed transcription outputs, calculating a cross-entropy loss based on the one or more labeling errors identified in the noise-suppressed transcription outputs, and performing a backpropagation of the cross-entropy loss to the speech enhancement model, the one or more speech enhancement model parameters being updated to optimize the noise-suppressed speech outputs to minimize an occurrence of labeling errors in the noise-suppressed transcription outputs.

The parameters are also updated by calculating a phase-aware loss based on a second comparison of noise-suppressed speech outputs and clean reference audio data corresponding to noisy audio data used to obtain the noise-suppressed speech outputs and performing a backpropagation of the phase-aware loss to the speech enhancement model, the one or more speech enhancement model parameters being updated to optimize the noise-suppressed speech outputs to minimize a signal quality loss in the noise-suppressed speech outputs
In view of the foregoing, it will be appreciated that the disclosed embodiments provide many technical benefits over conventional systems and methods for generating machine learning training data configured to train a machine learning model to learn speech representations. The disclosed embodiments beneficially improve conventional techniques for learning and generating speech representations.

Embodiments of the present invention may comprise or utilize a special purpose or general-purpose computer (e.g., computing system <NUM>) including computer hardware, as discussed in greater detail below. Computer-readable media (e.g., hardware storage device(s) <NUM> of <FIG>) that store computer-executable instructions (e.g., computer-readable instructions <NUM> of <FIG>) are physical hardware storage media/devices that exclude transmission media. Computer-readable media that carry computer-executable instructions or computer-readable instructions (e.g., computer-readable instructions <NUM>) in one or more carrier waves or signals are transmission media. Thus, by way of example, and not limitation, embodiments of the invention can comprise at least two distinctly different kinds of computer-readable media: physical computer-readable storage media/devices and transmission computer-readable media.

Physical computer-readable storage media/devices are hardware and include RAM, ROM, EEPROM, CD-ROM or other optical disk storage (such as CDs, DVDs, etc.), magnetic disk storage or other magnetic storage devices, or any other hardware which can be used to store desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

A "network" (e.g., network <NUM> of <FIG>) is defined as one or more data links that enable the transport of electronic data between computer systems and/or modules and/or other electronic devices. When information is transferred or provided over a network or another communications connection (either hardwired, wireless, or a combination of hardwired or wireless) to a computer, the computer properly views the connection as a transmission medium. Transmission media can include a network and/or data links which can be used to carry, or desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general purpose or special purpose computer.

Computer-executable instructions comprise, for example, instructions and data which cause a general-purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions.

Claim 1:
A computing system (<NUM>) comprising:
One or more processors (<NUM>); and
one or more storage devices storing computer-readable instructions (<NUM>) that are executable by the one or more processors (<NUM>) to configure the computing system (<NUM>) to at least:
obtain a speech enhancement model trained on a first training dataset to generate noise-suppressed speech outputs (<NUM>);
obtain an automatic speech recognition model trained on a second training dataset to generate transcription labels for spoken language utterances (<NUM>);
optimize the speech enhancement model by alternating between automated speech recognition ASR training and noise suppression NS training iteration steps;
wherein an ASR training iteration step comprises:
apply a third training dataset comprising a set of spoken language utterances to the speech enhancement model to obtain a first noise-suppressed speech output (<NUM>);
apply the first noise-suppressed speech output from the speech enhancement model to the automatic speech recognition model to generate a noise-suppressed transcription output for the set of spoken language utterances (<NUM>);
obtain ground truth transcription labels for the set of spoken language utterances included in the third training dataset (<NUM>); and
update one or more speech enhancement model parameters to optimize the speech enhancement model to generate optimized noise-suppressed speech outputs based on a first comparison of the noise-suppressed transcription output and the ground truth transcription labels (<NUM>); and
wherein an NS training iteration step comprises
obtain a fourth training dataset comprising noisy audio data and clean reference audio data corresponding to the noisy audio data;
apply the noisy audio data to the speech enhancement model to obtain a second noise-suppressed speech output; and
update the one or more speech enhancement model parameters to minimize signal quality loss during generation of the optimized noise-suppressed speech outputs based on a second comparison of the second noise-suppressed speech output and the clean reference audio data.