Patent Description:
Direct acoustics-to-word (A2W) automatic speech recognitions (ASR) systems use a neural network to directly recognize words from an input speech utterance without using any external decoder or language model. However, A2W ASR systems are trained with a fixed vocabulary, referred to as in-vocabulary (IV) words, and cannot easily recognize out-of-vocabulary (OOV) words. The present invention permits a user to enter a new OOV word as a character sequence into an A2W ASR system to permit the OOV word to be added into the listing of A2W word embeddings so that the ASR can directly recognize OOV words from speech acoustic features easily at test time without any further training.

According to an exemplary embodiment, the present invention describes a method for learning Out-of-Vocabulary (OOV) words in an Automatic Speech Recognition (ASR) system, the method including: using an Acoustic Word Embedding Recurrent Neural Network (AWE RNN) to receive a character sequence for a new OOV word for the ASR system, the RNN providing an Acoustic Word Embedding (AWE) vector as an output thereof; providing the AWE vector output from the AWE RNN as an input into an Acoustic Word Embedding-to-Acoustic-to-Word Neural Network (AWE→A2W NN) trained to provide an OOV word weight value from the AWE vector; and inserting the OOV word weight into a listing of Acoustic-to-Word (A2W) word embeddings used by the ASR system to output recognized words from an input of speech acoustic features, wherein the OOV word weight is inserted into the A2W word embeddings list relative to existing weights in the A2W word embeddings list.

Preferably, the present invention provides a method wherein the AWE RNN is initially trained as an overall subnetwork using character sequences of In-Vocabulary (IV) words, wherein the initial training further involves an Acoustic Embedding Recurrent Neural Network (AE RNN) that receives an acoustic sequence correspondingly to each character sequence of an IV word used during training, wherein outputs of the AWE RNN and AE RNN are passed into a contrastive loss function, and wherein the AWE RNN and AWE→A2W NN are trained using a backpropagation algorithm to train weights of the AWE RNN, weights of the AE RNN, and weights of the AWE→A2W NN to minimize the contrastive loss function.

Preferably, the present invention provides a method, wherein, subsequent to the initial training of the overall subnetwork, the AE RNN is not used for normal operation of the ASR system and only the AWE RNN is used for a subsequent introduction of OOV words into the ASR system.

Preferably, the present invention provides a method wherein the ASR system further comprises an Acoustic-to-Word Recurrent Neural Network (A2W RNN) that receives speech acoustic features as an input therein and an output of the A2W RNN is compared to embeddings of the A2W word embeddings listing using a dot product, and wherein, during a normal operation mode of the ASR system in which recognized words are output by the ASR system in response to speech acoustic features from an acoustic input into the ASR system, a word from the A2W word embeddings listing having a highest comparison result is provided as an output of the ASR system as a recognized word for the input speech acoustic features.

Preferably, the present invention provides a method wherein an overall subnetwork including the A2W RNN is trained using In-Vocabulary (IV) words, wherein speech acoustic features of an IV word and a word sequence corresponding to that IV word are provided into a loss function, and wherein a backpropagation algorithm updates weights of the A2W RNN in order to minimize this loss function and to provide the A2W word embeddings listing.

In another exemplary aspect, also described herein is a method for Automatic Speech Recognition (ASR), including: receiving a character sequence for an Out-of-Vocabulary (OOV) word into an Acoustic Word Embedding Recurrent Neural Network (AWE RNN) of an ASR system, as a mechanism to receive a character sequence for a new OOV word for the ASR system, the AWE RNN providing an Acoustic Word Embedding (AWE) vector as an output thereof; providing the AWE vector output from the AWE RNN as an input into an Acoustic Word Embedding-to-Acoustic-to-Word Neural Network (AWE→A2W NN) trained to provide an OOV word weight value from the AWE vector; and inserting the OOV word weight into a listing of Acoustic-to-Word (A2W) word embeddings used by the ASR system to output recognized words from an input of speech acoustic features, wherein the OOV word weight is inserted into the A2W word embeddings list relative to existing weights in the A2W word embeddings list.

Preferably, the present invention provides a method, wherein the AWE RNN is initially trained as an overall subnetwork using character sequences of In-Vocabulary (IV) words, wherein the initial training further involves an Acoustic Embedding Recurrent Neural Network (AE RNN) that receives an acoustic sequence correspondingly to each character sequence of an IV word used during training, wherein outputs of the AWE RNN and AE RNN are passed into a contrastive loss function, and wherein the AWE RNN and AWE→A2W NN are trained using a backpropagation algorithm to train weights of the AWE RNN, weights of the AE RNN, and weights of the AWE→A2W NN to minimize the contrastive loss function.

Preferably, the present invention provides a method wherein, subsequent to the initial training of the overall subnetwork, the AE RNN is not used for normal operation of the ASR system and only the AWE RNN is used for a subsequent introduction of OOV words into the ASR system.

In another exemplary aspect, also described herein is a method for Automatic Speech Recognition (ASR), including: initially training an overall subnetwork comprising an Acoustic-to-Word Recurrent Neural Network (A2W RNN), the A2W RNN receiving In-Vocabulary (IV) words for the initial training, the initial training using IV words resulting in a listing of Acoustic-to-Word (A2W) Word Embeddings stored in a memory of an ASR system performing the ASR processing; receiving an Out-of-Vocabulary (OOV) word as a character sequence into an Acoustic Word Embedding Recurrent Neural Network (AWE RNN), as a mechanism to receive a character sequence for a new OOV word for the ASR system, the AWE RNN providing an Acoustic Word Embedding (AWE) vector as an output thereof; providing the AWE vector output from the AWE RNN as an input into an Acoustic Word Embedding-to-Acoustic-to-Word Neural Network (AWE→A2W NN) trained to provide an OOV word weight value from the AWE vector; and inserting the OOV word weight into a listing of Acoustic-to-Word (A2W) word embeddings used by the ASR system to output recognized words from an input of speech acoustic features, wherein the OOV word weight is inserted into the A2W word embeddings list relative to existing weights in the A2W word embeddings list.

In another exemplary aspect, also described herein is a method for training an Automatic Speech Recognition (ASR) system, including: receiving an acoustic sequence for each In-Vocabulary (IV) character sequence used to initially train the ASR system; concurrently receiving a word sequence corresponding to each IV character sequence; and preparing a listing of Acoustic-to-Word (A2W) Word Embeddings of the IV character sequence, wherein the initial training uses a discriminative loss function, the discriminative loss function forcing, for each IV word in the listing of A2W word embeddings, the acoustic embedding for that IV word to be close to its text embedding in the listing.

An Automatic Speech Recognition (ASR) system, comprising: a processor in a computer system; and one or more memory devices accessible to the processor, wherein at least one memory of the at least one memory device of the one or more memory devices stores a set of machine-readable instructions to configure the computer system to function as the ASR system, the ASR system comprising an Acoustics-to-Word Recurrent Neural Network (A2W RNN), as implemented by a processor on a computer system, the A2W RNN configured to receive speech acoustic features of character sequences to be automatically recognized, the A2W RNN providing an acoustic embedding of the speech acoustic features of input words; and an Acoustic-to-Word (A2W) Word Embeddings list storing a listing of recognized character sequences, wherein the ASR is configured to select a character sequence from the A2W Word Embeddings list for output of the ASR as a recognized character sequence by selecting a character sequence from the A2W Word Embedding list that most closely matches an output acoustic embedding of the A2W RNN for input speech acoustic features, and wherein the ASR is initially trained by: receiving an acoustic sequence for each In-Vocabulary (IV) character sequence used to initially train the ASR system; concurrently receiving a character sequence corresponding to each IV character sequence; and preparing a listing of Acoustic-to-Word (A2W) character sequence embeddings of the IV character sequence, wherein the initial training uses a discriminative loss function, the discriminative loss function forcing, for each IV word in the listing of A2W word embeddings, the acoustic embedding for that IV character sequence to be close to its text embedding in the listing.

In-Vocabulary (IV) words are words used as input data to train a neural network (NN) in an Automatic Speech Recognition (ASR) system, the training providing settings for parameters of one or more layers of the neural network, using a backpropagation mechanism. Out-of-Vocabulary (OOV) words are therefore words that are received by the ASR system subsequent to the training of the NN, so that OOV words are not necessarily readily recognized by the previous training of the ASR NN. Several current approaches related to handling OOV words in direct acoustics-to-word (A2W) ASR systems are described as follows.

For example, several conventional systems train an acoustics-to-subword (A2S) model instead of an A2W model. These subwords include characters and word fragments/pieces. Predicting such subword units makes the ASR system open-vocabulary and hence enables it to recognize OOV words. However, such subword ASR systems still use a decoder and externally-trained language model to perform well, which is much more complex than recognizing speech with an A2W system.

The spell-and-recognize (SAR) model, another approach to ASR, trains an A2W system to first spell a word and then recognize it. For example, given the word sequence "THE CAT IS BLACK", the SAR system is trained to predict "THE" as _THE_, "CAT" as _CAT_, "IS" as _IS_, and "BLACK" as _BLACK_, where "_" denotes the space character. At test time, wherever the system predicts an <OOV> word token, the system backs off to the previously predicted character sequence. An example implementation is provided e.g. in the scientific paper "<NPL>. The main limitation of this approach is that OOV words are often incorrectly recognized due to spelling errors.

Another method proposes an extension of the SAR model by sharing neural network hidden layers for a word and character ASR model. Whenever an <OOV> word token is emitted, the system backs off to character predictions. Yet another proposed method decomposes OOV words into sub-word units and known words during training. This model also suffers from the limitations of the subword-based model first mentioned above.

These conventional methods in recognizing OOV words in an A2W ASR system thus have drawbacks, including a key drawback of dependence on an external language model and decoder to perform well. This is because greedy "decoder-less" speech recognition using a subword-based model is not guaranteed to produce correct words. A different approach is disclosed in the scientific paper "<NPL>, disclosing a joint learning of acoustic and orthographic sequences corresponding to single words in a single embedding space.

<FIG> shows an exemplary embodiment of the present invention, in which there is an upper subnetwork <NUM> and lower subnetwork <NUM>. In summary and in contrast to these conventional methods described above, the present invention discloses to incorporate a novel, lower subnetwork <NUM> that permits a user-entered OOV word to be entered into the A2W Word Embeddings list <NUM> as a new OOV without the additional training or language model required in conventional ASRs.

This subnetwork <NUM> is initially trained using IV words, similar to initial training of the top subnetwork <NUM> using IV words. After initial training the lower subnetwork <NUM> permits a user to enter an OOV word character sequence <NUM> as an input into AWE RNN <NUM>, which provides an AWE vector <NUM> as an output.

The AWE vector <NUM> then transformed into a weight vector by the AWE → A2W Neural Network (AWE → A2W NN) <NUM>, which weight vector <NUM> is similar in nature to weights derived for IV words during initial training of the upper subnetwork <NUM> to develop the A2W Word Embeddings list <NUM> in the initial training of the upper subnetwork <NUM>, so that the new OOV word weight can then be used to appropriately place a newly-entered OOV word into the A2W Word Embeddings list <NUM>. The AWE → A2W NN <NUM> is also initially trained using IV words, and permits the weight vector connecting the OOV word unit to be appropriately set into the output layer of the A2W ASR model to the previous layer (dotted line <NUM> in <FIG>) as corresponding to the OOV word CAT <NUM> rather than an <OOV> token <NUM>. This mechanism enables the A2W ASR system to then directly recognize an OOV word without needing any external decoder or language model, by a user entering the character sequence of an OOV word into the lower subnetwork <NUM> along with an acoustic utterance containing the OOV word in the upper subnetwork <NUM>.

A key advantage of this invention over prior methods dealing with OOV word recognition is that it does not rely on subword units and instead directly predicts both IV words and OOV words. The prior methods using subword units require the use of a decoder and external language model, and such requirement takes away from the simplicity and decoding speed of an A2W model.

Once the ASR system has been trained using IV words, as described below, along with any OOV words subsequently entered by a user, the upper subnetwork <NUM> can then receive speech acoustic features <NUM> during normal ASR operation, in which recognized words <NUM> are provided as an output via decoder <NUM>, as based on matching the input speech acoustic features <NUM>, as the output <NUM> from the A2W RNN <NUM>, with the N words in the A2W Word Embeddings listing <NUM>, using, for example, a comparison technique such as dot product <NUM>.

<FIG> shows the upper A2W subnetwork <NUM> as a baseline acoustics-to-word (A2W) automatic speech recognition (ASR) model, which is initially trained by presenting A2W RNN and the final A2W embeddings (i.e., the entire network) with speech acoustic features <NUM> and corresponding word sequences <NUM>. It is noted that, in the context of explaining the present invention, that "word sequences" are not the same as "character sequences". For example, a word sequence is "THE CAT IS BLACK" and the corresponding character sequence is "T H E C A T I S B L A C K". These word sequences are not input into any separate RNN, but are only used during a "connectionist temporal classification (CTC)" loss function <NUM> shown in <FIG>, use for comparing/correlating the predictions of the A2W network with the correct word sequence. Backpropagation is used to update the entire network's weights in order to minimize this loss function, thereby providing the weights to the AWE RNN <NUM> achieved during initial training using IV words. At this stage, all out-of-vocabulary (OOV) words in the training word sequences in the A2W Word Embedding listing <NUM> are replaced by the <OOV> token.

Thus, this initial training uses a discriminative loss function, such as the constructive loss function <NUM>, to force the acoustic <NUM> and text <NUM> embeddings to be close in the A2W Word Embeddings list <NUM> when they correspond to the same word and far apart when they correspond to different words. At the convergence of this training, the text <NUM> embedding is highly correlated to the acoustic <NUM> embedding of the same IV word in the A2W Word Embeddings <NUM>. Because of this mechanism of providing a high correlation between acoustic and text embeddings in the A2W Word Embeddings <NUM> for the same word, the present invention will be able to insert OOV words into the A2W Word Embeddings <NUM> using an input character sequence <NUM> and the lower subnetwork <NUM> which is initially trained to provide a weight associated with character sequences of OOV words entered by a user.

It is further noted that, because sequences are used as the input formats into both the upper subnetwork <NUM> and the lower subnetwork <NUM>, the initial neural networks of both subnetworks <NUM>, <NUM> are recurrent neural networks (RRNs), since RNNs are a type of deep neural networks that apply a same set of weights recursively over a structure to provide a structured prediction over a variable-length input by a given structure in topological order.

<FIG> shows how training proceeds for the lower subnetwork <NUM> of <FIG>. During this training, two RNNs - the AWE RNN <NUM> (also shown in <FIG>) and an additional RNN, the Acoustic Embedding (AE) RNN <NUM> are being trained. The AWE RNN <NUM> receives the character sequence <NUM> of an IV word as input while the AE RNN <NUM> receives the corresponding acoustic sequence <NUM> of the IV word. The two RNNs <NUM>, <NUM> compute one embedding vector each, that are passed into the contrastive loss (error) function <NUM>. The backpropagation algorithm is used to train the entire network's weights to minimize this loss function. Once initially trained, the AE RNN <NUM> can be discarded, with only the AWE RNN <NUM> being then used for introducing new OOV words, as shown in <FIG>.

Thus, the lower subnetwork <NUM> in <FIG> defines a key aspect of the present invention in that this subnetwork <NUM> permits a user to type a character sequence <NUM>, such as C A T, along with entering its speech acoustic features <NUM> in the upper subnetwork <NUM>, as a new OOV word not currently in the A2W Word Embeddings listing <NUM> as a text embedding. The lower subnetwork permits the new OOV word to be appropriately placed into the Acoustic Word Embeddings <NUM> using the weight factor calculated in the lower network.

The present invention therefore also trains a deep neural network (e.g., the AWE → A2W NN) <NUM> that, once trained, takes a vector <NUM> in the AWE space <NUM> and produces an output vector in the space of weight vectors of the output linear layer <NUM> of the A2W model <NUM> (see dotted line <NUM> in <FIG>). The AWE → A2W NN <NUM> is trained on A2W and AWE embeddings of IV words that are known during training.

Therefore, given an OOV word, entered as a character sequence <NUM> and an acoustic utterance containing the OOV word by a user, the present invention uses the trained AWE network <NUM> to produce an acoustic embedding <NUM> from the character sequence <NUM> of the OOV word. This OOV AWE vector <NUM> is then mapped to the A2W space (i.e., subnetwork <NUM>) using the AWE → A2W NN <NUM>. The output vector of this neural network <NUM> is used as the weight vector connecting the new output unit for the OOV word to the previous layer in the A2W model, so that the OOV word CAT will now have its own A2W Word Embedding <NUM>.

In order to recognize the newly-entered OOV word (e.g., CAT) in an input speech utterance <NUM>, the invention first picks the highest scoring word among the IV word list <NUM> and the <OOV> symbol <NUM>. If the <OOV> symbol <NUM> is predicted, the invention picks the highest scoring word from OOV units, so that the new OOV "CAT" is entered into the Word Embeddings list <NUM> in replacement of the <OOV> token with the highest score.

The present invention can be implemented in a number of various computer implementations, including implementations involving a cloud service. Therefore, although this disclosure includes a detailed description on cloud computing, as follows, one having ordinary skill would understand that implementation of the teachings recited herein are not limited to a cloud computing environment.

Examples of hardware components include: mainframes <NUM>; RISC (Reduced Instruction Set Computer) architecture-based servers <NUM>; servers <NUM>; blade servers <NUM>; storage devices <NUM>; and networks and networking components <NUM>.

Workloads layer <NUM> provides examples of functionality for which the cloud computing environment may be utilized. Examples of workloads and functions which may be provided from this layer include tasks related to the implementation of the present invention in which FOPE is incorporated, for example, into a DBaaS-based cloud service.

The descriptions of the various embodiments of the present invention have been presented for purposes of illustration, but are not intended to be exhaustive or limited to the embodiments disclosed. The terminology used herein was chosen to best explain the principles of the embodiments, the practical application or technical improvement over technologies found in the marketplace, or to enable others of ordinary skill in the art to understand the embodiments disclosed herein.

While the invention has been described in terms of several exemplary embodiments, those skilled in the art will recognize that the invention can be practiced with modification.

Claim 1:
A method for learning Out-of-Vocabulary (OOV) words in an Automatic Speech Recognition (ASR) system, the method comprising:
using an Acoustic Word Embedding Recurrent Neural Network (AWE RNN) to receive a character sequence for a new OOV word for the ASR system, the RNN providing an Acoustic Word Embedding (AWE) vector as an output thereof;
providing the AWE vector output from the AWE RNN as an input into an Acoustic Word Embedding-to-Acoustic-to-Word Neural Network (AWE→A2W NN) trained to provide an OOV word weight value from the AWE vector; and
inserting the OOV word weight into a listing of Acoustic-to-Word (A2W) word embeddings used by the ASR system to output recognized words from an input of speech acoustic features, wherein the OOV word weight is inserted into the A2W word embeddings list relative to existing weights in the A2W word embeddings list.