Conservatively adapting a deep neural network in a recognition system

Various technologies described herein pertain to conservatively adapting a deep neural network (DNN) in a recognition system for a particular user or context. A DNN is employed to output a probability distribution over models of context-dependent units responsive to receipt of captured user input. The DNN is adapted for a particular user based upon the captured user input, wherein the adaption is undertaken conservatively such that a deviation between outputs of the adapted DNN and the unadapted DNN is constrained.

BACKGROUND

Many different types of computer-implemented recognition systems exist, wherein such recognition systems are configured to perform some form of classification with respect to input data set forth by a user. For example, computer-implemented speech recognition systems are configured to receive spoken utterances of a user and recognize words in the spoken utterances. In another example, handwriting recognition systems have been developed to receive a handwriting sample and identify, for instance, an author of the handwriting sample, individual letters in the handwriting sample, words in the handwriting sample, etc. In still yet another example, computer-implemented recognition systems have been developed to perform facial recognition, fingerprint recognition, and the like.

With more particularity with respect to speech recognition, such type of recognition has been the subject of a significant amount of research and commercial development. For example, automatic speech recognition (ASR) systems have been incorporated into mobile telephones, desktop computers, automobiles, gaming consoles, customer service centers, etc., in order to recognize commands/questions and provide an appropriate response to such commands/questions. For instance, in a mobile telephone equipped with an ASR system, a user can utter a name of a contact retained in a contacts list on the mobile telephone, and the mobile telephone can initiate a call to the contact.

Even after decades of research, however, the performance of ASR in real-world usage scenarios remains far from satisfactory. Conventionally, hidden Markov models (HMMs) have been the dominant technique for larger vocabulary continuous speech recognition (LVCSR). In conventional HMMs used for ASR, observation probabilities for output states are modeled using Gaussian mixture models (GMMs). These GMM-HMM systems are typically trained to maximize the likelihood of generating observed features in training data. Recently, various discriminate strategies and large margin techniques have been explored. The potential of such techniques, however, is restricted by limitations of the GMM emission distribution model.

More recent research in ASR has explored layered architectures to perform speech recognition, motivated partly by the desire to capitalize on some analogous properties in the human speech generation and perception systems. In these studies, learning of model parameters (weights and weight biases corresponding to synapses in such layered architectures) has been one of the most prominent and difficult problems. In parallel with the development in ASR research, recent progresses made in learning methods from neural network research have ignited interest in exploration of deep neural networks (DNNs). A DNN is a densely connected directed belief network with many hidden layers. In general, DNNs can be considered as a highly complex, nonlinear feature extractor with a plurality of layers of hidden units and at least one layer of visible units, where each layer of hidden units is learned to represent features that capture higher-order correlations in original input data.

Conventionally, ASR systems that utilize DNNs are trained to be speaker/channel independent. In other words, parameters (e.g., weights and weight biases) of the DNN are not learned with respect to a particular speaker and/or channel. This is for at least two reasons: first, it is often difficult to obtain a sufficient amount of training data to robustly learn the parameters for a speaker and/or channel, as most users do not desire to spend a significant amount of time providing labeled utterances to train an ASR system. Furthermore, DNNs typically have many more parameters due to wider and deeper hidden layers, and also have a much larger output layer that is designed to model senones directly. This makes adapting a DNN utilized in connection with speech recognition a relatively difficult task.

SUMMARY

Described herein are various technologies pertaining to adapting at least one parameter of a deep neural network (DNN) that is employed in a recognition system, wherein the adaption is undertaken for a particular user or context. In an exemplary embodiment, the DNN can be employed in an automatic speech recognition (ASR) system as a portion of a context-dependent deep neural network hidden Markov model (CD-DNN-HMM) system. A computing device, such as a mobile telephone, a computing device in an automobile, a computing device in a call center, a gaming console, a server, etc., can include an ASR system that comprises a speaker independent (SI) CD-DNN-HMM system that has been trained utilizing training data from a plurality of different users. To improve recognition capabilities of the CD-DNN-HMM system for a particular user or context (e.g., a particular mobile phone), it may be desirable to adapt the DNN to the particular user or context. Technologies described herein can be employed to perform such adaption without requiring the user to set forth a large amount of training data.

In the context of ASR, DNNs are configured to directly model context dependent units, which are referred to herein as senones. A senone may be a triphone, a quinphone, or the like. A DNN in a SI ASR system, as noted above, can be trained utilizing training data corresponding to a plurality of different users. When a particular user sets forth a speech utterance that is desirably decoded through utilization of an ASR system that includes a DNN, the speech utterance is partitioned into a plurality of frames, and the output of the DNN for an individual frame is a probability distribution over the senones modeled by the DNN.

To avoid over-fitting to the training data for the particular user (e.g., a captured utterance of the user, often called adaptation data) when adapting the DNN for such user, such adapting can be undertaken in a conservative manner, such that a deviation in output between an unadapted DNN and an adapted DNN (a resultant DNN with parameters adapted using the adaptation data) is constrained. In an exemplary embodiment, this can be accomplished through regularizing a cost function when performing the adaption, although other approaches can also be employed to cause the adaption to be undertaken conservatively. For instance, in an exemplary embodiment, a Kullback-Leibler Divergence (KLD) regularization constraint can be added to the adaption criterion. Therefore, the senone distribution over multiple frames estimated by the adapted DNN is forced to be relatively close to the senone distribution estimated by the unadapted DNN.

DETAILED DESCRIPTION

With reference now toFIG. 1, an exemplary computing device100that comprises a recognition system102that can recognize input set forth by a user104of the computing device100is illustrated. The computing device100can be a client-side computing device, such as a mobile telephone, a gaming console, a desktop computer, a laptop computer, a tablet computing device (slate computing device), a computing device in an automobile, etc. In another example, the computing device100may be a computing device that is accessed remotely by the user104through utilization of a computing device or telephone, such as a computing device included in a customer service center that is remotely located from the user104.

The recognition system102can be any suitable recognition system that can recognize user input. For example, the recognition system102may be a system for recognizing gestures set forth by the user104, a system for recognizing characters in handwriting set forth by the user104, an automatic speech recognition (ASR) system that can decode words in utterances set forth by the user104, etc. For purposes of explanation, the recognition system102will be described herein as being an ASR system. It is to be understood, however, that the recognition system102can be any of the aforementioned types of recognition systems, that the adaption techniques for adapting model parameters described herein with respect to ASR systems can be employed in these other types of recognition systems.

In an exemplary embodiment, the recognition system102can include context-dependent-deep neural network-hidden Markov model (CD-DNN-HMM). A CD-DNN-HMM comprises a context-dependent deep neural network (which will be referred to herein as a deep neural network (DNN)106) and a hidden Markov model108(HMM). The DNN106is a multi-layer perceptron with a plurality of layers110. The plurality of layers110include an input layer (sometimes referred to as an observation layer) that receives an input observation, wherein the observation is provided to the DNN106over several frames (e.g. 9 to 13) of acoustic features. The plurality of layers110further comprises a plurality of hidden layers. For example, a number of hidden layers in the plurality of hidden layers can be at least three hidden layers. In some embodiments, a number of hidden layers in the plurality of hidden layers can be between three and ten hidden layers, between three and twenty hidden layer, between three and one hundred hidden layers, or between three and one thousand hidden layers.

Each hidden layer in the plurality of hidden layers comprises a respective plurality of nodes that are configured to perform nonlinear transformations on outputs of nodes from adjacent layers in the DNN106. Thus, hidden layers in the DNN106are stacked one on top of another, such that output of a node in a first hidden layer is an input node in a second, immediately adjacent hidden layer that is above the first hidden layer in the DNN106. Generally, in robust ASR systems that include a DNN, each hidden layer includes a relatively large number of nodes (e.g., at least one thousand nodes).

The plurality of layers110additionally includes an output layer, wherein the output layer includes models of context-dependent units (e.g., senones). In an exemplary embodiment, each senone modeled in the DNN106can be modeled as a multi-state HMM. The output of the DNN106is a probability distribution over modeled senones. The HMM108is generally employed to compute transition probabilities between such senones.

The recognition system102further comprises a decoder112, which can decode at least one word in a spoken utterance of the user104based upon the output of the HMM108. Generally, ASR systems that include CD-DNN-HMM systems have exhibited relatively high recognition accuracy rates when compared to conventional GMM-HMM ASR systems.

Oftentimes, it is desirable to train the DNN106in the recognition system102to be speaker independent (SI). For instance, generally, users do not wish to spend a significant amount of time setting forth training data to customize an ASR system for the voice and cadence of such users. Accordingly, the training data used to initially learn parameters of the DNN106may correspond to multiple different users, thereby causing the ASR system to be relatively robust across users. As will be described in greater detail herein, during training of the SI-DNN, both frame-level training and sequence-level training can be employed to learn parameters of the SI-DNN. Frame-level training refers to the learning of parameters of the DNN based upon individual frames, while sequence-level training refers to the learning of parameters of the DNN based upon a sequence of frames and other information such as language models. During training of the SI-DNN using the sequence-level criterion, it may be beneficial to regularize the sequence-level criterion by interpolating it with the frame-level criterion.

As the user104employs the recognition system102, however, it may be desirable to adapt the DNN106to the user104, since speech of such user104will have relatively unique characteristics (e.g., tone, pitch, cadence, . . . ).

To facilitate adaption of the DNN106to be customized for the user104of the computing device100, the computing device100can include an adapter component114that adapts parameters of the DNN106based upon spoken utterances set forth by the user104. As will be described in greater detail below, the adapter component114adapts the parameters of the DNN106conservatively, such that the output of the resultant adapted DNN is forced to be relatively similar to the output of the unadapted DNN (the DNN106prior to the adapter component114adapting parameters thereof). The adapted DNN can thereafter be employed to perform recognition tasks.

An example is set forth herein for purposes of explanation. Such example is not intended to be limiting as to the scope of the claims. The computing device100can be a mobile telephone that is being employed by the user104for a first time. The user104can initiate an application thereon that utilizes the recognition system102to recognize a spoken utterance of the user104. A sensor116of the computing device100(a microphone) captures the spoken utterance of the user104. The spoken utterance is received by the recognition system102, which acts as described above to decode at least one word in the spoken utterance set forth by the user104. The user may then set forth input that indicates whether or not the word recognized by the ASR system is correct.

Upon the DNN106receiving frames of acoustic features, nodes in hidden layers of the DNN106perform nonlinear transformations to produce higher ordered correlations between features, and outputs of nodes are passed via weighted synapses to other nodes. The output of the DNN106is a probability distribution over the senones modeled by the DNN106. Using an expressed or inferred label (e.g., in unsupervised adaptation, the label can be inferred from a transcription output by the decoder112), a respective target probability distribution over the senones for each frame can be identified for the spoken utterance of the user104. Conventionally, parameters (weights and weight biases) of the DNN106are learned to maximize the negative cross entropy (averaged over the frames) between the probability distribution output by the DNN106and the target probability distribution. The adapter component114can adapt the parameters of the DNN106such that the probability distribution output by the adapted DNN for the spoken utterance set forth by the user104does not greatly deviate from the probability distribution output by the DNN106of the ASR that was included in the mobile telephone. Accordingly, for instance, the adapter component114can regularize the conventional learning process to cause the adaption of the DNN106based upon the spoken utterance set forth by the user104to be undertaken conservatively.

The regularization of the conventional learning process can include the utilization of a regularization weight. As the value of the regularization weight grows, the adaption of the DNN106undertaken by the adapter component114becomes more conservative. For example, as the value of the regularization weight becomes larger, less and less deviation between outputs of the adapted and unadapted DNNs occurs. Conversely, as the value of the regularization weight becomes smaller, more and more deviation between outputs of the adapted and unadapted DNNs may be observed.

In an exemplary embodiment, the adapter component114can select a value of the regularization parameter as a function of an amount of speech data received from the user104. As more speech data is received from the user104, a value of the regularization weight selected by the adapter component114can decrease. If adaption is to be undertaken based upon a smaller amount of speech data, the adapter component114can cause the value for the regularization weight to be larger (e.g., to avoid overfitting based upon the speech data from the user104).

In another exemplary embodiment, the adapter component114can select the value of the regularization weight based upon whether supervised learning or unsupervised learning is employed to perform adaption. If supervised learning is employed to perform adaption (e.g., the speech data from the user is labeled—such as when the user104reads from a known text), the adapter component114can select a lower value for the regularization weight, thereby allowing for greater deviation in outputs of the unadapted and adapted DNNs. Conversely, if unsupervised learning is employed to perform adaption, then the adapter component114can select a larger value for the regularization parameter, thereby reducing an amount of deviation between outputs of the unadapted and adapted DNNs.

Furthermore, as noted above, the DNN106can include multiple hidden layers, each with hundreds or more (e.g., even thousands) nodes therein, and wherein each connection between nodes may have a respective weight and/or weight bias. Accordingly, a relatively large number of parameters for the DNN106can be learned by the adapter component114when adapting the DNN106. In an exemplary embodiment, the adapter component114can adapt all parameters of the DNN106. In other embodiments, the adapter component114can adapt a subset of parameters of the DNN106. For instance, the adapter component114can cause parameters of a single hidden layer to be adapted, can cause parameters corresponding to certain nodes to be adapted, etc. Selectively updating a subset of parameters of the DNN106may be beneficial in situations where the computing device102has received a relatively large amount of speech data from the user104, and there is a time constraint on the adapting of the DNN106.

In some embodiments, the adapter component114can adapt parameters of the DNN106after each use of the recognition system102. In another example, each time the computing device100is powered on, the adapter component114can ascertain if new speech data has been received from the user102, and can adapt the DNN106in the speech recognition system102using any newly received speech data (e.g., where newly received speech data is speech data received since the last time the computing device100was powered on). Thus, the adapter component114can incrementally adapt a DNN over time. In still other embodiments, the adapter component114can continuously adapt a DNN as more and more speech data is received from the user104.

In still yet another exemplary embodiment, the adapter component114can adapt the DNN106as the recognition system102is being utilized to perform a recognition task. Thus, the recognition system performance will be enhanced as the user104uses such recognition system102. In such an exemplary embodiment, the user104can set forth an utterance which is decoded by the recognition system102as described above. This can, for example, result in a transcription of the spoken utterance, which is employed to identify a target probability distribution over senones for the spoken utterance. The adapter component114can adapt the DNN106based upon such target probability distribution (to form an adapted DNN), and the same spoken utterance can be provided to the recognition system102, where the adapted DNN is employed in connection with decoding the utterance.

Additional detail pertaining to DNNs and the adaption process described above are now set forth. The information set forth with respect toFIG. 1pertains to adaption (customized for a particular user and/or context); as indicated above, adaption can be can be carried out using frame-level criterion, sequence-level criterion, or an interpolated sequence-level and frame-level criterion. As indicated, the DNN106accepts an input observation x, which typically includes 9 to 13 frames of acoustic features, and processes it through many layers of nonlinear transformation as follows:
hil=σ(zil(vl))=σ((wil)Tvl+ail),  (1)
where wland alare the weight matrix and bias, respectively, at hidden layer l, hilis the output of the ith node (neuron),
zl(vl)=(wl)Tvl+al(2)
is the excitation vector given input vl, vl=hl−1when l>0 and v0=x, and σ(x)=1/(1+exp(−x)) is the sigmoid function applied element-wise. At the top layer L, the softmax function

p⁡(y=s❘vL)=exp⁡((wsL)T⁢vL+asL)∑y′⁢exp⁡((wy′L)T⁢vL+ay′L)(3)
is used to estimate the state posterior probability p(y=s|x), which is converted to the HMM state emission probability as follows:

p⁡(x❘y=s)=p⁡(y=s❘x)p⁡(y=s)·p⁡(x),(4)
where sε{1, 2, . . . , S} is a senone id, S is the total number of senones, p(y=s) is the prior probability of senone s, and p(x) is independent of state s.

Further, as noted above, the parameters of DNNs are conventionally trained to maximize a negative cross-entropy as follows:

D_=⁢1N⁢∑t=1N⁢D⁡(xt)=⁢1N⁢∑t=1N⁢∑y=1S⁢p~⁡(y❘xt)⁢log⁢⁢p⁡(y❘xt),(5)
where N is the number of samples in the training set and {tilde over (p)}(y|xt) is the target probability. In some cases, a hard alignment from an existing system can be used as a training label under which condition {tilde over (p)}(y|xt)=δ(y=st), where δ is a Kronecker delta and stis the label of the t-th sample (e.g., the t-th observation frame in a training corpus). Pursuant to an example, training is often carried out using a back propagation algorithm and can be quickened using GPU and mini-batch updates.

The adapter component114adapts the DNN106such that the posterior senone distribution estimated from the adapted model does not deviate too far from that estimated using the unadapted model, particularly when the adaption set (data provided by the user104) is relatively small.

Since outputs of the DNN106are probability distributions, deviation can be measured by the adapter component114using Kullback-Leibler Divergence (KLD). By adding such divergence as a regularization term to Eq. (5) and removing terms unrelated to the model parameters, the regularized optimization criterion employed by the adapter component114may be as follows:

D^=⁢(1-ρ)⁢D_+ρ⁢1N⁢∑t=1N⁢∑y=1S⁢pSI⁡(y❘xt)⁢log⁢⁢p⁡(y❘xt),(6)
where pSI(y|xt) is the posterior probability estimated from the unadapted model and computed with a forward pass using the unadapted model, and ρ is the regularization weight. Eq. (6) can be re-organized as follows:

Through comparison of Eqs. (5) and (7), it can be ascertained that applying KLD regularization to the original training (optimization) criterion is equivalent to changing the target probability distribution from {tilde over (p)}(y|xt) to {circumflex over (p)}(y|xt), which is a linear interpolation of the distribution estimated from the unadapted model and the ground truth alignment of the adaption data. Such interpolation prevents overtraining by ensuring that output of the adapted DNN does not stray far from the unadapted DNN. It can be noted that this differs from L2 regularization, which constrains the model parameters themselves, rather than the output probabilities. This also indicates that the normal back propagation algorithm can be directly used to adapt the DNN106, as all that changes is the error signal at the output layer of the DNN106, which can be defined using {circumflex over (p)}(y|xt).

The interpolation weight, which can be directly derived from the regularization weight ρ, can be adjusted, typically using a development set, based on the size of the adaption set, the learning rate used, and whether the adaption is supervised or unsupervised. For instance, when ρ=1, the unadapted model is entirely trusted and all new information from the adaption data is ignored. When ρ=0, the model is adapted based solely on the adaption set, ignoring information from the unadapted model, except using it as a starting point.

While KLD has been set forth above as being employable by the adapter component114when adapting weights of the DNN106, it is to be understood that the adapter component114can utilize other approaches when performing such adaption (or may use a combination of approaches to adapt parameters of the DNN106). Pursuant to an example, certain parameters of the DNN106can be restricted from being adapted by the adapter component114, which may effectively constrain deviation in outputs of the adapted DNN and unadapted DNN. For instance, the adapter component114may only adapt weights of synapses in the DNN106with an absolute value above a predefined threshold. In other examples, the adapter component114may be configured to only adapt weights of synapses with absolute values below a predefined threshold. In still yet another example, the adapter component114can be configured to adapt some percentage of parameters in the DNN106, wherein parameters included in such percentage can be selected in any suitable manner. For instance, weights having absolute values in the top 10% of all weight values, can be subject to adaption, while the other 90% of weights are unadapted. Likewise, a certain percentage of parameters can be randomly selected for adaption, or parameters in a subset of layers can be adapted.

Now turning toFIG. 2, a graphical representation of an exemplary DNN200is illustrated. The DNN200comprises an observed data layer202, which captures an input sample in the form of a vector V0. The input is denoted inFIG. 2by X, which is an I×1 vector. The DNN further comprises a plurality of hidden layers204-208. Each of the hidden layers204-208comprises a respective plurality of hidden units, and wherein each hidden unit comprises a respective activation function. Hidden units in adjacent layers are potentially connected by way of weighted synapses, which can be collectively represented by weight matrices210and212between hidden layers. As shown, the weight matrix210represents weighted synapses between hidden units in the hidden layer204(hidden layer HL-2) and hidden units in the hidden layer206(hidden layer HL-1). Similarly, the weight matrix212represents weighted synapses between hidden units in the hidden layer206and hidden units in the hidden layer208(hidden layer HL). A layer214in the DNN200is the output, which is determined based upon the weighted synapses and activation functions of hidden units in the DNN200. The output is denoted inFIG. 2as Y. The adapter component114can adapt weights of the weight matrix210and/or the weight matrix212, as well as weight biases when performing adaption.

With reference now toFIG. 3, the adapter component114is shown as being able to contemplate both frame level adaption criterion302and sequence level adaption criterion304when adapting parameters of the DNN106. In sequence level training, cross entropy between outputs is calculated based upon a sequence of senones, rather than based upon probability of existence of senones in a certain frame. Accordingly, rather than adapting parameters of the DNN106solely using frame level criterion302, the adapter component114can alternatively or additionally adapt parameters of the DNN106utilizing sequence level adaption criterion. If both frame level and sequence level adaption criterion are used, the adapter component114can interpolate between adaptions made based upon the frame level adaption criterion302and the sequence level adaption criterion304. That is, the adapter component114can compute first adaption values, as described above, based upon frames, and the adapter component114can also determine adaption values based upon the sequence level adaption criterion304. The adapter component114may subsequently interpolate between corresponding adaption values. Further, it is to be understood that both frame level and sequence level adaption criterion can be employed when initially learning parameters of the unadapted DNN (the SI-DNN). When sequence level criterion is used to train the SI-DNN, it may be beneficial to interpolate the sequence-level criterion with the frame-level criterion. During adaption, however, in an example, KLD can be added to the frame level criterion and/or the sequence level criterion, effectively restricting deviation in output between the adapted and unadapted model.

As noted above, a CD-DNN-HMM can model the posterior probability Ps|o(s|o) of a senone s given an observation o. The CD-DNN-HMM includes a stack of (L+1) layers of log-linear models of the form P(hl|vl)=1/Zlexp((Wl)Tvl+a) with layer-type specific partition functions Zl), weight matrices Wland bias vectors al(the model parameters to train), and vland hldenoting the input and output of each layer.

For hidden layers, the components of hlare assumed binary and conditionally independent, such that P(hl|vl) has the form of a component-wise sigmoid. With the “mean-field approximation”, the expected value of hlis used as the input to the next layer: vl+1Eh|vl{hl|vl}. For the output layer, hLis a unit vector with the position of the 1 denoting the senone s: Ps|o(s|o)=P(hsL=1|vL). Such constraint gives rise to the form of softmax.

For decoding and lattice generation, the senone posteriors are converted in the HMM's emission likelihoods by dividing the senone priors Ps(s):
logpo|s(o|s)=logPs|o(s|o)−logPs(s)+logpo(o),  (9)
where the observation vectors o are acoustic feature vectors augmented with neighbor frames. po(o) is unknown but can be ignored, as it cancels out in best-path decisions and word-posterior computation. Likewise, ZLcan be ignored in Ps|o(s|o).

As referenced above, CD-DNN-HMMs can be trained with a stochastic-gradient error back-propagation method, typically after initialization through a pre-training step. Of relevance is an error signal of the top layer:

An exemplary approach to train CD-DNN-HMMs is to maximize the total log posterior probability over training frames or(t) with ground truth labels ŝr(t). This is known as the cross entropy (CE) criterion (with Kronecker delta δ):
CE=ΣrΣtlogPs|o(ŝr(t)|or(t))  (11)
esCE(r,t)=δs,ŝr(t)−Ps|o(s|or(t)).  (12)

Sequence training incorporates HMM, lexical, and language model constraints of the actual mean average precision (MAP) decision rule. Popular sequence objectives, known from GMM systems, are maximum mutual information (MMI), boosted MMI (BMMI), and minimum Bayes risk (MBR), as set forth below:

MMI=∑r⁢log⁢⁢P⁡(S^r❘Or)(13)BMMI=∑r⁢log⁢P⁡(S^r❘Or)∑S′⁢P⁡(S′❘Or)·ⅇ-b·Ar⁡(S′)(14)MBR=∑r⁢∑S⁢P⁡(S❘Or)·Ar⁡(S).(15)
The P(S|O) are path posteriors given the current model:

The acoustic likelihoods p(O|S) are computed using Eq. (9). The P(S) are path priors that consider HMM transitions, lexicon, and LM, and κ is the acoustic weight. Ar(S) is the accuracy function. Its sMBR variant can be used, which counts correct frames in path S against ground truth Ŝr. With s(t) denoting the senone on S at frame t, the error signals are:
eSMMI(r,t)=δS,Ŝ(t)−γSr(t)  (17)
eSMBR(r,t)=κγSr(t)[E{Ar(S)|s(t)=s}−E{Ar(S)}]  (18)
with

eSBMMI(r, t) is the same as eSMMI(r, t), except for a γSr(t) modified analogously toBMMI. Such error signals can be computed efficiently using forward-backward procedures. Accordingly, sequence-training BP can reuse the existing CE BP machinery, augmented with an additional, more complex computation of the error signal.

Any of Eqs. (13)-(15) can be employed to initially learn weights and/or weight biases of the SI-DNN. As indicated above, during adaption based upon user-specific or context-specific adaption data, the aforementioned criterion can be augmented to include regularization criterion, which restricts deviation in output between the SI-DNN and the resultant adapted DNN.

Now referring to solely toFIG. 4, an exemplary methodology400that facilitates adapting a DNN used in a recognition system is illustrated. The methodology400starts at402, and at404features for observed input data are received. For example, such features can be based upon a spoken utterance of a particular user.

At406, parameters of a DNN are conservatively adapted based upon the input data observed at404. As indicated above, conservative adaption refers to the constraint in deviation between probability distributions over context dependent phones between an unadapted model and the model that has been conservatively adapted. An amount of conservativeness can be based upon a regularization weight, a value for which can be selected based upon the length and/or amount of training data (the length of the spoken utterance). The resultant adapted DNN is customized for the user who set forth the input data. The methodology400completes at408.

With reference now toFIG. 5, an exemplary methodology500that facilitates adapting parameters of a DNN using a regularization weight is illustrated. The methodology500starts at502, and at504features for a spoken utterance of a particular user are received. At506, a value for a regularization weight to employ when adapting parameters of a DNN is computed. Again, this can be computed as a function of amount of training data to be used when adapting parameters of the DNN, whether supervised learning or unsupervised learning is employed when performing the adaption, etc. At508, the parameters of the DNN are adapted using the value for the regularization weight computed at506. The methodology500completes at510.

Referring now toFIG. 6, a high-level illustration of an exemplary computing device600that can be used in accordance with the systems and methodologies disclosed herein is illustrated. For instance, the computing device600may be used in a system that supports conservatively adapting a DNN of a recognition system for a particular user or context. The computing device600includes at least one processor602that executes instructions that are stored in a memory604. The instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more components discussed above or instructions for implementing one or more of the methods described above. The processor602may access the memory604by way of a system bus606. In addition to storing executable instructions, the memory604may also store matrix weights, weight of a regularization parameter, a weight bias, training data, etc.

The computing device600additionally includes a data store608that is accessible by the processor602by way of the system bus606. The data store608may include executable instructions, learned parameters of a DNN, etc. The computing device2600also includes an input interface610that allows external devices to communicate with the computing device600. For instance, the input interface610may be used to receive instructions from an external computer device, from a user, etc. The computing device600also includes an output interface612that interfaces the computing device600with one or more external devices. For example, the computing device600may display text, images, etc. by way of the output interface612.