Patent Description:
Acoustic model adaptation may be used to address such mismatches. For example, a speaker-independent (SI) DNN acoustic model may perform well with respect to the speech of most speakers. Model adaptation attempts to adapt the SI DNN acoustic model into a personalized speaker-dependent (SD) DNN acoustic model associated with a given target speaker that achieves improved speech recognition performance when applied to speech of the given target speaker. Such speaker-based model adaptation is more challenging than other types of domain adaptation because the amount of available adaptation data (i.e., speech of the target speaker) is typically limited. Moreover, adaptation of an SI DNN model, which usually includes a large number of parameters, may cause the SD DNN to be overfitted to the limited adaptation data.

Improved speaker-based adaptation of SI DNN acoustic models used for speech recognition is desired. Such improvements may adapt a SI DNN acoustic model based on limited adaptation data while substantially maintaining the distribution of the SI DNN acoustic model, in order to provide improved performance with respect to a target speaker.

<NPL> describes that we propose a novel regularized adaptation technique for context dependent deep neural network hidden Markov models (CD-DNN-HMMs). The CD-DNN-HMM has a large output layer and many large hidden layers, each with thousands of neurons. The huge number of parameters in the CD-DNN-HMM makes adaptation a challenging task, esp. when the adaptation set is small. The technique developed in this paper adapts the model conservatively by forcing the senone distribution estimated from the adapted model to be close to that from the unadapted model. This constraint is realized by adding Kullback-Leibler divergence (KLD) regularization to the adaptation criterion. We show that applying this regularization is equivalent to changing the target distribution in the conventional backpropagation algorithm. Experiments on Xbox voice search, short message dictation, and Switchboard and lecture speech transcription tasks demonstrate that the proposed adaptation technique can provide <NUM>%-<NUM>% relative error reduction against the already very strong speaker independent CD-DNN-HMM systems using different adaptation sets under both supervised and unsupervised adaptation setups.

The following description is provided to enable any person in the art to make and use the described embodiments. Various modifications, however, will remain readily apparent to those of ordinary skill in the art.

Some embodiments provide an adversarial speaker adaptation scheme in which adversarial learning is applied to regularize the distribution of deep hidden features in a SD DNN acoustic model to be similar to that of a pre-trained SI DNN acoustic model. A discriminator network is used during the adversarial learning to distinguish deep features generated by the SD model from deep features generated by the SI model.

With the pre-trained SI model as the reference, and based on adaptation data (i.e., a target speaker's speech frames), the SD model is jointly optimized with the discriminator network to minimize the senone classification loss, and simultaneously to mini-maximize the SI/SD discrimination loss. Consequently, a senone-discriminative deep feature is learned in the SD model which exhibits a similar distribution to that of the pre-trained SI model. Using such a regularized and adapted deep feature, the SD model can provide improved automatic speech recognition on the target speaker's speech.

<FIG> illustrates system <NUM> according to some embodiments. System <NUM> may be used to train SD feature extractor <NUM> to learn senone-discriminative features based on adaptation frames <NUM> of a target speaker and exhibiting a distribution similar to the deep features of pre-trained SI feature extractor <NUM>. SD senone classifier <NUM> predicts senone posteriors based on features received from SD feature extractor <NUM>. Discriminator network <NUM> receives features generated based on adaptation frames <NUM> from SD feature extractor <NUM> and features generated based on adaptation frames <NUM> from SI feature extractor <NUM>, and predicts whether the features were generated by SD feature extractor <NUM> or by SI feature extractor <NUM>.

During training, SD feature extractor <NUM> receives adaptation frames <NUM> associated with a target speaker and maps the frames to intermediate deep hidden features. SD senone classifier <NUM> receives the intermediate deep hidden features and maps the features to senone posteriors. The parameters of SD feature extractor <NUM> and SD senone classifier <NUM> are optimized in order to minimize senone loss <NUM>.

Also during training, pre-trained SI feature extractor <NUM> receives adaptation frames <NUM> and maps the frames to intermediate deep hidden features. Discriminator network <NUM> receives features from SD feature extractor <NUM> and from SI feature extractor <NUM> and predicts SD or SI posteriors based on the received deep hidden features, and the parameters of discriminator network <NUM> are optimized to minimize discrimination loss <NUM>. Moreover, the parameters of SD feature extractor <NUM> are jointly trained with an adversarial objective to maximize discrimination loss <NUM>. Such optimization is based on an understanding that the ability of discriminator network <NUM> to accurately discriminate between features generated by SD feature extractor <NUM> and SI feature extractor <NUM> domain is inversely related to the similarity of the distributions of the intermediate deep hidden features generated by each extractor.

In one example of adapting a SI DNN to a target speaker according to some embodiments, adaptation frames <NUM> include adaptation speech frames X = {x<NUM>,. , xT }, xt ∈ Rrx, t = <NUM>,. , T of the target speaker and a sequence of senone labels Y = {y<NUM>,. , yT }, yt ∈ R aligned with X. For supervised adaptation, Y is generated by aligning the adaptation data against a transcription generated using a previously well-trained SI DNN acoustic model. For unsupervised adaptation, the adaptation data is first decoded using the SI DNN acoustic model and the one-best path of the decoding lattice is used as Y.

The first few layers of the previously well-trained SI DNN acoustic model are viewed as a SI feature extractor network MfSI (i.e., SI feature extractor <NUM>) with parameters θfSI and the upper layers of the SI DNN acoustic model are viewed as a SI senone classifier MySI with parameters θySI. MfSI maps input adaptation speech frames X to intermediate SI deep hidden features FSI = {f<NUM>SI,. , fTSI }, ftSI ∈ Rrf, i.e., ftSI = MfSI(xt). MySI maps the deep hidden features FSI to the senone posteriors p(s|ftSI; θySI), s ∈ S as follows: <MAT>.

An SD DNN acoustic model to be trained using speech from a target speaker is initialized from the SI DNN acoustic model. According to some embodiments, MfSI (i.e., SI feature extractor <NUM>) is used to initialize SD feature extractor MfSD (i.e., SD feature extractor <NUM>) with parameters θfSD and MySI is used to initialize SD senone classifier MySD (i.e., SD senone classifier <NUM>) with parameters θySD. Accordingly, in the SD acoustic model, MfSD maps xt to SD deep features ftSD and MySD further transforms ftSD to a same set of senone posteriors p(s|ftSD ; θySD) s ∈ S as follows: <MAT>.

To adapt the SI model to the speech X of the target speaker, the initialized SD model is trained by minimizing the cross-entropy senone classification loss between the predicted senone posteriors and the senone labels Y as follows: <MAT> where <MAT> is the indicator function which equals <NUM> if the condition in the squared bracket is true and <NUM> otherwise.

As described above, due to the typically limited amount of adaptation data X (e.g., adaptation frames <NUM>), a thusly-adapted SD model may become overfitted to the adaptation data X. Some embodiments address this issue by converging the distribution of deep hidden features FSD of the SD model to that of the deep hidden features FSI of the SI model, while minimizing the senone loss <IMG> as follows: <MAT> <MAT>.

The above convergence is promoted using a discriminator network Md (i.e., discriminator network <NUM>) with parameters θd which takes FSD and FSI as input and outputs the posterior probability that an input deep feature is generated by the SD model, i.e., <MAT> <MAT> where DSD and DSI denote the sets of SD and SI deep features, respectively.

The discrimination loss (e.g., discrimination loss <NUM>) <IMG>(θf, θd) for Md may be formulated below using cross-entropy: <MAT>.

To make the distribution of FSD similar to that of FSI, adversarial training of MfSD and Md is performed to minimize <IMG> with respect to θd and maximize<IMG> with respect to θfSD. This minimax competition will first increase the capability of MfSD to generate FSD with a distribution similar to that of FSI and increase the discrimination capability of Md. It will eventually converge to the point where MfSD generates FSD such that Md is unable to determine whether FSD is generated by MfSD or MfSI. At such a point, the SD model has been regularized such that its distribution is substantially similar to the SI model.

Moreover, because the SD feature extractor MfSD is also to produce senone-discriminative features, the acoustic model network consisting of the SD feature extractor and the SD senone classifier, and the discriminator network are trained to jointly optimize the primary task of senone classification and the secondary task of SD/SI classification with an adversarial objective function as follows: <MAT> <MAT> where λ controls the trade-off between the senone classification loss <IMG> and the discriminator loss <IMG>, and θ̂fSD , θ̂ySD and θ̂d are the optimized network parameters. The SI model only serves as a reference during training and its parameters θfSI, θySI are fixed throughout training.

These parameters may be updated during training via back propagation with stochastic gradient descent as follows: <MAT> <MAT> <MAT> where µ is the learning rate. The negative coefficient -λ induces a reversed gradient that maximizes <IMG>(θfSD,θd) to result in speaker-invariant deep features. Gradient reversal layer <NUM> may provide an identity transform in the forward propagation and multiply the gradient by -λ during the backward propagation.

The optimized DNN acoustic model consisting of MfSD (e.g., SD feature extractor <NUM> and MySD (e.g., SD senone classifier <NUM>) are used for automatic speech recognition of target speaker speech, while MfSI (e.g., SI feature extractor <NUM>) and Md (e.g., discriminator network <NUM>) are discarded after parameter training. <FIG> shows system <NUM> including SD feature extractor <NUM> and SD senone classifier <NUM> including trained parameters θ̂fSD and θ̂ySD , respectively, according to some embodiments.

SD feature extractor <NUM> receives input utterance of a target speaker (i.e., the target speaker whose utterances were used as adaptation data to train parameters θ̂fSD and θ̂ySD) and operates as trained to generate substantially speaker-invariant and senone-discriminative frame-level deep hidden features. SD senone classifier <NUM> receives the features and also operates according to its trained parameters to produce posterior features for each frame f, which provide statistical likelihoods that the frame f is generated by various senones. The posteriors may be used in various embodiments to identify words represented by the utterance, for example, to determine whether a key phrase is present, to identify the content of a command or query, to perform transcription, etc..

<FIG> is a flow diagram of a process to train a SD DNN acoustic model to generate senone-discriminative features according to some embodiments. The <FIG> process and the other processes described herein may be performed using any suitable combination of hardware and software. Software program code embodying these processes may be stored by any non-transitory tangible medium, including a fixed disk, a volatile or non-volatile random access memory, a DVD, a Flash drive, or a magnetic tape, and executed by any number of processing units, including but not limited to processors, processor cores, and processor threads. Embodiments are not limited to the examples described below.

Initially, at S310, a SI DNN including a feature extractor and a senone classifier is trained to classify senones based on input utterances. <FIG> illustrates training of a system according to some embodiments. Model training platform <NUM> may comprise any suitable system to instantiate and train one or more artificial neural networks of any type. Generally, model training platform <NUM> operates to input training data to a system of one or more DNNs, evaluate the resulting output of the system with respect to training objectives, modify parameters of the DNNs accordingly, and repeat the process until the training objectives are sufficiently met.

<FIG> illustrates system <NUM> implemented by platform <NUM> at S310 of process <NUM>. System <NUM> includes SI DNN <NUM> consisting of SI feature extractor <NUM> and SI senone classifier <NUM>. The training data comprises a plurality of pre-captured utterances of a plurality of speakers and senones (e.g., in text format) represented by each utterance. Platform <NUM> controls training of SI DNN <NUM> (i.e., of the parameters of SI feature extractor <NUM> and SI senone classifier <NUM>) to minimize senone loss <NUM>.

In one non-exhaustive example of S410, SI feature extractor <NUM> and SI senone classifier <NUM> are implemented as a long short-term memory (LSTM) hidden Markov model (HMM) acoustic model. A neural network (e.g., deep learning, deep convolutional, or recurrent) according to some embodiments comprises a series of "neurons," such as LSTM nodes, arranged into a network. A neuron is an architecture used in data processing and artificial intelligence, particularly machine learning, that includes memory that may determine when to "remember" and when to "forget" values held in that memory based on the weights of inputs provided to the given neuron. Each of the neurons used herein are configured to accept a predefined number of inputs from other neurons in the network to provide relational and sub-relational outputs for the content of the frames being analyzed. Individual neurons may be chained together and/or organized into tree structures in various configurations of neural networks to provide interactions and relationship learning modeling for how each of the frames in an utterance are related to one another.

For example, an LSTM serving as a neuron includes several gates to handle input vectors, a memory cell, and an output vector. The input gate and output gate control the information flowing into and out of the memory cell, respectively, whereas forget gates optionally remove information from the memory cell based on the inputs from linked cells earlier in the neural network. Weights and bias vectors for the various gates are adjusted over the course of a training phase, and once the training phase is complete, those weights and biases are finalized for normal operation. Neurons and neural networks may be constructed programmatically (e.g., via software instructions) or via specialized hardware linking each neuron to form the neural network.

SI DNN <NUM> may include <NUM> hidden layers and <NUM> hidden units within each layer. A <NUM>-dimensional projection layer is inserted on top of each hidden layer to reduce the number of parameters. The output layer of the LSTM includes <NUM> output units corresponding to <NUM> senone labels. There is no frame stacking, and the output HMM senone label is delayed by <NUM> frames. <NUM>-dimensional log Mel filterbank features are extracted from the SI training frames and are fed as the input to the LSTM after global mean and variance normalization.

Next, at S320, a SD DNN is initialized with the feature extractor and the senone classifier of the trained SI DNN. For example, SI feature extractor network <NUM> (MfSI) is identified as the first Nh layers of the trained SI DNN <NUM> and SI senone classifier <NUM> (MySI) is formed as the remaining (e.g., <NUM> - Nh) trained layers and the trained output layer. Nh indicates the position of the deep hidden features in the SI and SD acoustic models. A SD feature extractor network MfSD and a SD senone classifier MySD are then cloned from MfSI and MySI.

At S330, a discriminator (Md) is initialized to receive features generated by the SD feature extractor MfSD and features generated by the SI feature extractor MfSI and to predict whether a received feature extractor was generated by the SD feature extractor MfSD. Md may comprise a feedforward DNN including a <NUM>-dimensional input layer, <NUM> hidden layers and <NUM> hidden units for each layer. The output layer of Md may include <NUM> unit predicting the posteriors of input deep features generated by the SD feature extractor MfSD.

MfSD, MySD, and Md are jointly trained at S340 with an adversarial multi-task objective. Platform <NUM> of <FIG> illustrates such joint training of system <NUM> according to some embodiments.

In particular, at S342, FSD is made senone-discriminative by training the SD feature extractor MfSD and the SD senone classifier MySD based on adaptation frames <NUM> of a target speaker to minimize the cross-entropy senone classification loss between the predicted senone posteriors and the senone labels as described above. Simultaneously, at S344, to make the distribution of FSD similar to that of FSI, the discriminator Md is trained at S344 to minimize <IMG> with respect to θd and the SD feature extractor MfSD is trained at S346 to maximize <IMG> with respect to θfSD.

<FIG> illustrates system <NUM> to train a SD DNN acoustic model according to some embodiments. System <NUM> includes SI senone classifier <NUM>, which consists of the upper layers of a previously-trained SI DNN acoustic model from which SI feature extractor <NUM> was formed. Accordingly, SI feature extractor <NUM> and SI senone classifier <NUM> represent the previously-trained SI DNN acoustic model.

Moreover, discriminator network <NUM> receives senone posteriors from SD senone classifier <NUM> and from SI senone classifier <NUM>. Discriminator network <NUM> may therefore output posteriors predicting whether received senone posteriors were generated by SD senone classifier <NUM> or from SI senone classifier <NUM>.

According to the <FIG> embodiment, SD feature extractor <NUM> (MfSD) and SD senone classifier <NUM> (MySD) are trained at S342 based on adaptation frames <NUM> of a target speaker to minimize the cross-entropy senone classification loss between the predicted senone posteriors and the senone labels as described above. Simultaneously, to make the distribution of FSD similar to that of FSI, discriminator <NUM> (Md) is trained at S344 to minimize discriminator loss <NUM> (<IMG>) with respect to θd and SD feature extractor <NUM> (MfSD) and SD senone classifier <NUM> (MySD) are both trained at S346 to maximize discriminator loss <NUM> (<IMG>) with respect to θfSD and θySD.

The thusly-trained acoustic model consisting of MfSD and MySD may then be used at S350 to recognize speech of the recognition. The trained acoustic model can be used as a component of an automatic speech recognition unit in any number of different types of devices and systems. For example, automatic speech recognition using the trained acoustic model can be implemented in digital assistants, chatbots, voice control applications, and other related devices and systems including in associated voice services such as software development kit (SDK) offerings. Automatic speech recognition services using the trained acoustic model can be implemented in cloud architectures. A chatbot is a program that can conduct conversations via auditory or textual methods. A bot is a program that can access web sites and gather content based on a provided search index. The web sites can be coupled to the Internet, an intranet, or the web sites may be databases, each database accessible by its own addresses according to a protocol for the respective database.

<FIG> illustrates architecture <NUM> providing speech recognition services according to some embodiments. System <NUM> may be cloud-based and components thereof may be implemented using on-demand virtual machines, virtual servers and cloud storage instances. Such cloud-based components may be connected to the Internet and/or to any network or combinations of networks. A cloud can include a wide area network (WAN) like the public Internet or a private, national or global network, and may include a local area network (LAN) within an organization providing the services of the data center.

As shown, automatic speech recognition service <NUM> may be implemented as a cloud service providing transcription of speech audio signals received over cloud <NUM>. Automatic speech recognition service <NUM> may include a SI acoustic model trained using any suitable training system and one or more SD acoustic models trained as described above. Each of the one or more SD acoustic models may be usable to recognize the speech of a respective target speaker.

Each of client devices <NUM> and <NUM> may be operated to request services such as search service <NUM> and voice assistant service <NUM>. Services <NUM> and <NUM> may, in turn, request automatic speech recognition functionality from automatic speech recognition service <NUM>. Such a request may include an identifier of a target speaker and/or adaptation data (e.g., utterances) associated with the target speaker. Using such an identifier and/or data, service <NUM> may identify a SD acoustic model associated with the target speaker or adapt the SI acoustic model to generate a SD acoustic model associated with the target speaker. Service <NUM> may then use the identified and/or generated SD acoustic model to perform automatic speech recognition on speech of the target speaker.

<FIG> is a block diagram of system <NUM> according to some embodiments. System <NUM> may comprise a general-purpose server computer and may execute program code to provide an automatic speech recognition service as described herein. System <NUM> may be implemented by a cloud-based virtual server according to some embodiments.

System <NUM> includes processing unit <NUM> operatively coupled to communication device <NUM>, persistent data storage system <NUM>, one or more input devices <NUM>, one or more output devices <NUM> and volatile memory <NUM>. Processing unit <NUM> may comprise one or more processors, processing cores, etc. for executing program code. Communication interface <NUM> may facilitate communication with external devices, such as client devices, and data providers as described herein. Input device(s) <NUM> may comprise, for example, a keyboard, a keypad, a mouse or other pointing device, a microphone, a touch screen, and/or an eye-tracking device. Output device(s) <NUM> may comprise, for example, a display (e.g., a display screen), a speaker, and/or a printer.

Data storage system <NUM> may comprise any number of appropriate persistent storage devices, including combinations of magnetic storage devices (e.g., magnetic tape, hard disk drives and flash memory), optical storage devices, Read Only Memory (ROM) devices, etc. Memory <NUM> may comprise Random Access Memory (RAM), Storage Class Memory (SCM) or any other fast-access memory.

Trained SI acoustic model <NUM> may comprise program code executed by processing unit <NUM> to cause system <NUM> to recognize senones based on input speech signals using senone-discriminative deep features as described herein. Trained SD acoustic models <NUM> may be associated with respective target speakers and may be adapted from SI acoustic model <NUM> based on adaptation data of the target speakers as described herein. Accordingly, SD acoustic models <NUM> may comprise program code executed by processing unit <NUM> to cause system <NUM> to recognize senones based on input speech signals of the target speakers.

Adversarial speaker adaptation training <NUM> may comprise program code executed by processing unit <NUM> to cause system <NUM> to train a SD acoustic model based on a SI acoustic model as described herein. Node operator libraries <NUM> may comprise program code to execute functions of neural network nodes based on parameter values. Data storage device <NUM> may also store data and other program code for providing additional functionality and/or which are necessary for operation of system <NUM>, such as device drivers, operating system files, etc..

Each functional component and process described herein may be implemented at least in part in computer hardware, in program code and/or in one or more computing systems executing such program code as is known in the art. Such a computing system may include one or more processing units which execute processor-executable program code stored in a memory system.

Processor-executable program code embodying the described processes may be stored by any non-transitory tangible medium, including a fixed disk, a volatile or non-volatile random access memory, a DVD, a Flash drive, or a magnetic tape, and executed by any number of processing units, including but not limited to processors, processor cores, and processor threads. Embodiments are not limited to the examples described below.

The foregoing diagrams represent logical architectures for describing systems according to some embodiments, and actual implementations may include more or different components arranged in other manners. Other topologies may be used in conjunction with other embodiments. Moreover, each component or device described herein may be implemented by any number of devices in communication via any number of other public and/or private networks. Two or more of such computing devices may be located remote from one another and may communicate with one another via any known manner of network(s) and/or a dedicated connection. Each component or device may comprise any number of hardware and/or software elements suitable to provide the functions described herein as well as any other functions. For example, any computing device used in an implementation of a system according to some embodiments may include a processor to execute program code such that the computing device operates as described herein.

The diagrams described herein do not imply a fixed order to the illustrated methods, and embodiments may be practiced in any order that is practicable. Moreover, any of the methods described herein may be performed by hardware, software, or any combination of these approaches. For example, a computer-readable storage medium may store thereon instructions which when executed by a machine result in performance of methods according to any of the embodiments described herein.

Claim 1:
A system comprising:
a processing unit (<NUM>); and
a memory storage device (<NUM>) including program code that when executed by the processing unit causes to the system to:
initialize a speaker-dependent deep neural network, DNN, with a feature extractor (<NUM>) and senone classifier of a trained speaker-independent DNN, wherein the trained speaker-independent DNN is trained on speech;
initialize a discriminator (<NUM>) to receive features generated by a speaker-dependent feature extractor (<NUM>) and features generated by the speaker-independent feature extractor (<NUM>) and to output a prediction of whether received data was generated by the speaker-dependent feature extractor;
train the parameters of the speaker-dependent feature extractor (<NUM>) and a speaker-dependent senone classifier (<NUM>) based on adaptation frames (<NUM>) of a target speaker to minimize a cross-entropy senone classification loss; and
simultaneously train the speaker-dependent feature extractor (<NUM>) to maximize a discrimination loss and the discriminator to minimize the discrimination loss.