Patent Description:
Modern automated speech recognition (ASR) systems focus on providing not only high quality (e.g., a low word error rate (WER)), but also low latency (e.g., a short delay between the user speaking and a transcription appearing). Moreover, when using an ASR system today there is a demand that the ASR system decode utterances in a streaming fashion that corresponds to real-time or even faster than real-time. To illustrate, when an ASR system is deployed on a mobile phone that experiences direct user interactivity, an application on the mobile phone using the ASR system may require the speech recognition to be streaming such that words appear on the screen as soon as they are spoken. Here, it is also likely that the user of the mobile phone has a low tolerance for latency. Due to this low tolerance, the speech recognition strives to run on the mobile device in a manner that minimizes an impact from latency and inaccuracy that may detrimentally affect the user's experience.

Document <NPL>, discloses an examplary recurrent neural network transducer (RNN-T) model for streaming end-to-end speech recognition.

The invention is defined by the appended independent claims, with the dependent claims providing further preferred embodiments.

In some examples, weighting the embedding proportional to the similarity between the embedding and the respective position vector includes weighting the embedding proportional to a cosine similarity between the embedding and the respective position vector. The sequence of non-blank symbols output by the final Softmax layer include wordpieces. Optionally, the sequence of non-blank symbols output by the final Softmax layer may include graphemes. Each of the embeddings may include a same dimension size as each of the position vectors. In some implementations, the sequence of non-blank symbols received as input is limited to the N previous non-blank symbols output by the final Softmax layer. In these implementations, N may be equal to two. Alternatively, N may be equal to five.

In some examples, the prediction network includes a multi-headed attention mechanism that shares the shared embedding matrix across each head of the multi-headed attention mechanism. In these examples, at each of the plurality of time steps subsequent to the initial time step the prediction network is configured to, at each head of the multi-headed attention mechanism, and for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step: generate, using the shared embedding matrix, the same embedding of the corresponding non-blank symbol as the embedding generated at each other head of the multi-headed attention mechanism; assign a different respective position vector to the corresponding non-blank symbol than the respective position vector assigned to the corresponding non-blank symbol at each other head of the multi-headed attention mechanism; and weight the embedding proportional to the similarity between the embedding and the respective position vector. Here, the prediction network is also configured to generate, as output from the corresponding head of the multi-headed attention mechanism, a respective weighted average of the weighted embeddings of the sequence of non-blank symbols and generate, as output, the single embedding vector at the corresponding time step by averaging the respective weighted averages output from the corresponding heads of the multi-headed attention mechanisms. In these examples, the multi-headed attention mechanism may include four heads. Optionally, the prediction network may tie a dimensionality of the shared embedding matrix to a dimensionality of an output layer of the joint network.

In some examples, weighting the embedding proportional to the similarity between the embedding and the respective position vector includes weighting the embedding proportional to a cosine similarity between the embedding and the respective position vector.

The sequence of non-blank symbols output by the final Softmax layer may include wordpieces. Optionally, the sequence of non-blank symbols output by the final Softmax layer may include graphemes. Each of the embeddings may include a same dimension size as each of the position vectors. In some implementations, the sequence of non-blank symbols received as input is limited to the N previous non-blank symbols output by the final Softmax layer. In these implementations, N may be equal to two. Alternatively, N may be equal to five. Optionally, the prediction network may tie a dimensionality of the shared embedding matrix to a dimensionality of an output layer of the joint network.

In some examples, the prediction network includes a multi-headed attention mechanism that shares the shared embedding matrix across each head of the multi-headed attention mechanism. The multi-headed attention mechanism may include four heads. In these examples, at each of the plurality of time steps subsequent to the initial time step, the operations may further include, at each head of the multi-headed attention mechanism, and for each non-blank symbol in the sequence of non-blank symbols received as input at the corresponding time step: generating, by the prediction network and using the shared embedding matrix, the same embedding of the corresponding non-blank symbol as the embedding generated at each other head of the multi-headed attention mechanism; assigning, by the prediction network, a different respective position vector to the corresponding non-blank symbol than the respective position vector assigned to the corresponding non-blank symbol at each other head of the multi-headed attention mechanism; and weighting, by the prediction network, the embedding proportional to the similarity between the embedding and the respective position vector. Here, at each of the plurality of time steps subsequent to the initial time step and at each head of the multi-headed attention mechanism, the operations may also include generating, by the prediction network as output from the corresponding head of the multi-headed attention mechanism, a respective weighted average of the weighted embeddings of the sequence of non-blank symbols. Thereafter, at each of the plurality of time steps subsequent to the initial time step, the operations may further include generating, as output from the prediction network, the single embedding vector at the corresponding time step by averaging the respective weighted averages output from the corresponding heads of the multi-headed attention mechanisms.

<FIG> is an example of a speech environment <NUM>. In the speech environment <NUM>, a user's <NUM> manner of interacting with a computing device, such as a user device <NUM>, may be through voice input. The user device <NUM> (also referred to generally as a device <NUM>) is configured to capture sounds (e.g., streaming audio data) from one or more users <NUM> within the speech environment <NUM>. Here, the streaming audio data may refer to a spoken utterance <NUM> by the user <NUM> that functions as an audible query, a command for the device <NUM>, or an audible communication captured by the device <NUM>. Speech-enabled systems of the device <NUM> may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device <NUM> may correspond to any computing device associated with a user <NUM> and capable of receiving audio data. Some examples of user devices <NUM> include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device <NUM> includes data processing hardware <NUM> and memory hardware <NUM> in communication with the data processing hardware <NUM> and stores instructions, that when executed by the data processing hardware <NUM>, cause the data processing hardware <NUM> to perform one or more operations. The user device <NUM> further includes an audio system <NUM> with an audio capture device (e.g., microphone) <NUM>, 16a for capturing and converting spoken utterances <NUM> within the speech environment <NUM> into electrical signals and a speech output device (e.g., a speaker) <NUM>, 16b for communicating an audible audio signal (e.g., as output audio data from the device <NUM>). While the user device <NUM> implements a single audio capture device 16a in the example shown, the user device <NUM> may implement an array of audio capture devices 16a without departing from the scope of the present disclosure, whereby one or more capture devices 16a in the array may not physically reside on the user device <NUM>, but be in communication with the audio system <NUM>.

In the speech environment <NUM>, an automated speech recognition (ASR) system <NUM> implementing a recurrent neural network-transducer (RNN-T) model <NUM> and an optional rescorer <NUM> resides on the user device <NUM> of the user <NUM> and/or on a remote computing device <NUM> (e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device <NUM> via a network <NUM>. The user device <NUM> and/or the remote computing device <NUM> also includes an audio subsystem <NUM> configured to receive the utterance <NUM> spoken by the user <NUM> and captured by the audio capture device 16a, and convert the utterance <NUM> into a corresponding digital format associated with input acoustic frames <NUM> capable of being processed by the ASR system <NUM>. In the example shown, the user speaks a respective utterance <NUM> and the audio subsystem <NUM> converts the utterance <NUM> into corresponding audio data (e.g., acoustic frames) <NUM> for input to the ASR system <NUM>. Thereafter, the RNN-T model <NUM> receives, as input, the audio data <NUM> corresponding to the utterance <NUM>, and generates/predicts, as output, a corresponding transcription <NUM> (e.g., recognition result/hypothesis) of the utterance <NUM>. In the example shown, the RNN-T model <NUM> may perform streaming speech recognition to produce an initial speech recognition result <NUM>, 120a and the rescorer <NUM> may update (i.e., rescore) the initial speech recognition result 120a to produce a final speech recognition result <NUM>, 120b.

The user device <NUM> and/or the remote computing device <NUM> also executes a user interface generator <NUM> configured to present a representation of the transcription <NUM> of the utterance <NUM> to the user <NUM> of the user device <NUM>. As described in greater detail below, the user interface generator <NUM> may display the initial speech recognition results 120a in a streaming fashion during time <NUM> and subsequently display the final speech recognition result 120b during time <NUM>. In some configurations, the transcription <NUM> output from the ASR system <NUM> is processed, e.g., by a natural language understanding (NLU) module executing on the user device <NUM> or the remote computing device <NUM>, to execute a user command/query specified by the utterance <NUM>. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device <NUM> or the remote computing device <NUM>) may convert the transcription into synthesized speech for audible output by the user device <NUM> and/or another device.

In the example shown, the user <NUM> interacts with a program or application <NUM> (e.g., the digital assistant application <NUM>) of the user device <NUM> that uses the ASR system <NUM>. For instance, <FIG> depicts the user <NUM> communicating with the digital assistant application <NUM> and the digital assistant application <NUM> displaying a digital assistant interface <NUM> on a screen of the user device <NUM> to depict a conversation between the user <NUM> and the digital assistant application <NUM>. In this example, the user <NUM> asks the digital assistant application <NUM>, "What time is the concert tonight?" This question from the user <NUM> is a spoken utterance <NUM> captured by the audio capture device 16a and processed by audio systems <NUM> of the user device <NUM>. In this example, the audio system <NUM> receives the spoken utterance <NUM> and converts it into acoustic frames <NUM> for input to the ASR system <NUM>.

Continuing with the example, the RNN-T model <NUM>, while receiving the acoustic frames <NUM> corresponding to the utterance <NUM> as the user <NUM> speaks, encodes the acoustic frames <NUM> and then decodes the encoded acoustic frames <NUM> into the initial speech recognition results 120a. During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the initial speech recognition results 120a of the utterance <NUM> to the user <NUM> of the user device <NUM> in a streaming fashion such that words, word pieces, and/or individual characters appear on the screen as soon as they are spoken. In some examples, the first look ahead audio context is equal to zero.

During time <NUM>, the user interface generator <NUM> presents, via the digital assistant interface <NUM>, a representation of the final speech recognition result 120b of the utterance <NUM> to the user <NUM> of the user device <NUM> rescored by the rescorer <NUM>. In some implementations, the user interface generator <NUM> replaces the representation of the initial speech recognition results 120a presented at time <NUM> with the representation of the final speech recognition result 120b presented at time <NUM>. Here, time <NUM> and time <NUM> may include timestamps corresponding to when the user interface generator <NUM> presents the respective speech recognition result <NUM>. In this example, the timestamp of time <NUM> indicates that the user interface generator <NUM> presents the initial speech recognition result 120a at an earlier time than the final speech recognition result 120b. For instance, as the final speech recognition result 120b is presumed to be more accurate than the initial speech recognition results 120a, the final speech recognition result 120b ultimately displayed as the transcription <NUM> may fix any terms that may have been misrecognized in the initial speech recognition results 120a. In this example, the streaming initial speech recognition results 120a output by the RNN-T model <NUM> are displayed on the screen of the user device <NUM> at time <NUM> are associated with low latency and provide responsiveness to the user <NUM> that his/her query is being processed, while the final speech recognition result 120b output by the rescorer <NUM> and displayed on the screen at time <NUM> leverages an additional speech recognition model and/or a language model to improve the speech recognition quality in terms of accuracy, but at increased latency. However, since the initial speech recognition results 120a are displayed as the user speaks the utterance <NUM>, the higher latency associated with producing, and ultimately displaying the final recognition result is not noticeable to the user <NUM>.

In the example shown in <FIG>, the digital assistant application <NUM> may respond to the question posed by the user <NUM> using natural language processing. Natural language processing generally refers to a process of interpreting written language (e.g., the initial speech recognition results 120a and/or the final speech recognition result 120b) and determining whether the written language prompts any action. In this example, the digital assistant application <NUM> uses natural language processing to recognize that the question from the user <NUM> regards the user's schedule and more particularly a concert on the user's schedule. By recognizing these details with natural language processing, the automated assistant returns a response <NUM> to the user's query where the response <NUM> states, "Venue doors open at <NUM>:<NUM> PM and concert starts at 8pm. " In some configurations, natural language processing occurs on a remote server <NUM> in communication with the data processing hardware <NUM> of the user device <NUM>.

Referring to <FIG>, an example frame alignment-based transducer model <NUM> includes a Recurrent Neural Network-Transducer (RNN-T) model architecture which adheres to latency constraints associated with interactive applications. The RNN-T model <NUM> provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device <NUM> (e.g., no communication with a remote server is required). The RNN-T model <NUM> includes an encoder network <NUM>, a prediction network <NUM>, and a joint network <NUM>. The prediction and joint networks <NUM>, <NUM> may collectively provide an RNN-T decoder. The encoder network <NUM>, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a recurrent network of stacked Long Short-Term Memory (LSTM) layers. For instance, the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames <NUM> (<FIG>)) x = (x<NUM>, x<NUM>, ···, xT), where <MAT>, and produces at each time step a higher-order feature representation. This higher-order feature representation is denoted as <MAT>.

Similarly, the prediction network <NUM> is also an LSTM network, which, like a language model (LM), processes the sequence of non-blank symbols <NUM> output by a final Softmax layer <NUM> so far, y<NUM>,. , yui-<NUM>, into a representation pui. Described in greater detail below, the representation pui <NUM> includes a single embedding vector. Notably, the sequence of non-blank symbols <NUM> received at the prediction network <NUM> capture linguistic dependencies between non-blank symbols <NUM> predicted during the previous time steps so far to assist the joint network <NUM> in predicting the probability of a next output symbol or blank symbol during the current time step. As described in greater detail below, to contribute to techniques for reducing the size of the prediction network <NUM> without sacrificing accuracy/performance of the RNN-T model <NUM>, the prediction network <NUM> may receive a limited-history sequence of non-blank symbols yui-n,. ,yui-<NUM> that is limited to the N previous non-blank symbols <NUM> output by the final Softmax layer <NUM>.

Finally, with the RNN-T model architecture, the representations produced by the encoder and prediction networks <NUM>, <NUM> are combined by the joint network <NUM>. The joint network then predicts Zi = P(yi|xti, y<NUM>,. , yui-<NUM>), which is a distribution over the next output symbol. Stated differently, the joint network <NUM> generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the "possible speech recognition hypotheses" correspond to a set of output labels each representing a symbol/character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (<NUM>) symbols, e.g., one label for each of the <NUM>-letters in the English alphabet and one label designating a space. Accordingly, the joint network <NUM> may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network <NUM> can include a posterior probability value for each of the different output labels. Thus, if there are <NUM> different output labels representing different graphemes or other symbols, the output yi of the joint network <NUM> can include <NUM> different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer <NUM>) for determining the transcription <NUM>.

The Softmax layer <NUM> may employ any technique to select the output label/symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model <NUM> at the corresponding output step. In this manner, the RNN-T model <NUM> does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model <NUM> does assume an output symbol is independent of future acoustic frames <NUM>, which allows the RNN-T model to be employed in a streaming fashion.

In some examples, the encoder network <NUM> of the RNN-T model <NUM> is made up of eight <NUM>,<NUM>-dimensional LSTM layers, each followed by a <NUM>-dimensional projection layer. In other implementations, the encoder network <NUM> includes a network of conformer or transformer layers. The prediction network <NUM> may have two <NUM>,<NUM>-dimensional LSTM layers, each of which is also followed by <NUM>-dimensional projection layer as well as an embedding layer of <NUM> units. Finally, the joint network <NUM> may also have <NUM> hidden units. The Softmax layer <NUM> may be composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in training data. When the output symbols/labels include wordpieces, the set of output symbols/labels may include <NUM>,<NUM> different word pieces. When the output symbols/labels include graphemes, the set of output symbols/labels may include less than <NUM> different graphemes.

<FIG> shows the prediction network <NUM> of the RNN-T model <NUM> receiving, as input, a sequence of non-blank symbols yui-n,. , yui-<NUM> that is limited to the N previous non-blank symbols 301a-n output by the final Softmax layer <NUM>. In some examples, N is equal to two. In other examples, N is equal to five, however, the disclosure is nonlimiting and N may equal any integer. The sequence of non-blank symbols 301a-n indicates an initial speech recognition result 120a (<FIG>). In some implementations, the prediction network <NUM> includes a multi-headed attention mechanism <NUM> that shares a shared embedding matrix <NUM> across each head 302A-<NUM> of the multi-headed attention mechanism. In one example, the multi-headed attention mechanism <NUM> includes four heads. However, any number of heads may be employed by the multi-headed attention mechanism <NUM>. Notably, the multi-headed attention mechanism improves performance significantly with minimal increase to model size. As described in greater detail below, each head 302A-H includes its own row of position vectors <NUM>, and rather than incurring an increase in model size by concatenating outputs 318A-H from all the heads, the outputs 318A-H are instead averaged by a head average module <NUM>.

Referring to the first head 302A of the multi-headed attention mechanism <NUM>, the head 302A generates, using the shared embedding matrix <NUM>, a corresponding embedding <NUM>, 306a-n (e.g., <MAT>) for each non-blank symbol <NUM> among the sequence of non-blank symbols yui-n,. , yui-<NUM> received as input at the corresponding time step from the plurality of time steps. Notably, since the shared embedding matrix <NUM> is shared across all heads of the multi-headed attention mechanism <NUM>, the other heads 302B-H all generate the same corresponding embeddings <NUM> for each non-blank symbol. The head 302A also assigns a respective position vector PVAa-An <NUM>, 308Aa-An (e.g., <MAT>) to each corresponding non-blank symbol in the sequence of non-blank symbols yui-n,. , yui-<NUM>. The respective position vector PV <NUM> assigned to each non-blank symbol indicates a position in the history of the sequence of non-blank symbols (e.g., the N previous non-blank symbols output by the final Softmax layer <NUM>). For instance, the first position vector PVAa is assigned to a most recent position in the history, while the last position vector PVAn is assigned to a last position in the history of the N previous non-blank symbols output by the final Softmax layer <NUM>. Notably, each of the embeddings <NUM> may include a same dimensionality (i.e., dimension size) as each of the position vectors PV <NUM>.

While the corresponding embedding generated by shared embedding matrix <NUM> for each for each non-blank symbol <NUM> among the sequence of non-blank symbols 301a-n, yui-n,. , yui-<NUM>, is the same at all of the heads 302A-H of the multi-headed attention mechanism <NUM>, each head 302A-H defines a different set/row of position vectors <NUM>. For instance, the first head 302A defines the row of position vectors PVAa-An 308Aa-An, the second head 302B defines a different row of position vectors PVBa-Bn <NUM>Ba-Bn,. , and the Hth head <NUM> defines another different row of position vectors PVHa-Hn<NUM>Ha-Hn.

For each non-blank symbol in the sequence of non-blank symbols 301a-n received, the first head 302A also weights, via a weight layer <NUM>, the corresponding embedding <NUM> proportional to a similarity between the corresponding embedding and the respective position vector PV <NUM> assigned thereto. In some examples, the similarity may include a cosine similarity (e.g., cosine distance). In the example shown, the weight layer <NUM> outputs a sequence of weighted embeddings <NUM>, 312Aa-An each associated the corresponding embedding <NUM> weighted proportional to the respective position vector PV <NUM> assigned thereto. Stated differently, the weighted embeddings <NUM> output by the weight layer <NUM> for each embedding <NUM> may correspond to a dot product between the embedding <NUM> and the respective position vector PV <NUM>. The weighted embeddings <NUM> may be interpreted as attending over the embeddings in proportion to how similar they are to the positioned associated with their respective position vectors PV <NUM>. To increase computational speed, the prediction network <NUM> includes non-recurrent layers, and therefore, the sequence of weighted embeddings 312Aa-An are not concatenated, but instead, averaged by a weighted average module <NUM> to generate, as output from the first head 302A, a weighted average 318A of the weighted embeddings 312Aa-An represented by: <MAT> In Equation <NUM>, h represents the index of the heads <NUM>, n represents position in context, and e represents the embedding dimension. Additionally, in Equation <NUM>, H, N, and de include the sizes of the corresponding dimensions. The position vector PV <NUM> does not have to be trainable and may include random values. Notably, even though the weighted embeddings <NUM> are averaged, the position vectors PV <NUM> can potentially save position history information, alleviating the need to provide recurrent connections at each layer of the prediction network <NUM>.

The operations described above with respect to the first head 302A, are similarly performed by each other head 302B-H of the multi-headed attention mechanism <NUM>. Due to the different set of positioned vectors PV <NUM> defined by each head <NUM>, the weight layer <NUM> outputs a sequence of weighted embeddings 312Ba-Bn, 312Ha-Hn at each other head 302B-H that is different than the sequence of weighted embeddings 312Aa-Aa at the first head 302A. Thereafter, the weighted average module <NUM> generates, as output from each other corresponding head 302B-H, a respective weighted average 318B-H of the corresponding weighted embeddings <NUM> of the sequence of non-blank symbols.

In the example shown, the prediction network <NUM> includes a head average module <NUM> that averages the weighted averages 318A-H output from the corresponding heads 302A-H. A projection layer <NUM> with SWISH may receive, as input, an output <NUM> from the head average module <NUM> that corresponds to the average of the weighted averages 318A-H, and generate, as output, a projected output <NUM>. A final layer normalization <NUM> may normalize the projected output <NUM> to provide the single embedding vector Pui <NUM> at the corresponding time step from the plurality of time steps. The prediction network <NUM> generates only a single embedding vector Pui <NUM> at each of the plurality of time steps subsequent to an initial time step.

In some configurations, the prediction network <NUM> does not implement the multi-headed attention mechanism <NUM> and only performs the operations described above with respect to the first head 302A. In these configurations, the weighted average 318A of the weighted embeddings 312Aa-An is simply passed through the projection layer <NUM> and layer normalization <NUM> to provide the single embedding vector Pui <NUM>.

Referring back to <FIG>, the joint network <NUM> receives the single embedding vector Pui <NUM> from the prediction network <NUM> and the higher-order feature representation <MAT> from the encoder <NUM>. The joint network <NUM> generates a probability distribution P(yi |xti, y<NUM>,. , yui-<NUM>) over possible speech recognition hypotheses at the corresponding time step. Here, the possible speech recognition hypotheses correspond to a set of output label that each represent a symbol character in a specified natural language. The probability distribution P(yi|xti,y<NUM>,. , yui-<NUM>) over the possible speech recognition hypotheses indicates a probability for the final speech recognition result 120b (<FIG>). That is, the joint network <NUM> determines the probability distribution for the final speech recognition result 120b using the single embedding vector <NUM> that is based on the sequence of non-blank symbols (e.g., initial speech recognition result 120a). The final Softmax layer <NUM> receives the probability distribution for the final speech recognition result 120b and selects the output label/symbol with the highest probability to produce the transcription.

The final speech recognition result 120b is presumed to be more accurate than the initial speech recognition result 120a because the RNN-T model <NUM> determines the initial speech recognition results 120a in a streaming fashion and the final speech recognition results 120b using the prior non-blank symbols from the initial speech recognition result 120a. That is, the final speech recognition results 120b take into account the prior non-blank symbols and thus are presumed more accurate because the initial speech recognition results 120a do not take into account any prior non-blank symbols. Moreover, the rescorer <NUM> (<FIG>) may update the initial speech recognition result 120a with the final speech recognition result 120b to provide the transcription via the user interface generator <NUM> to the user <NUM>.

In some implementations, to further reduce the size of the RNN-T decoder, i.e., the prediction network <NUM> and the joint network <NUM>, parameter tying between the prediction network <NUM> and the joint network <NUM> is applied. Specifically, for a vocabulary size |V| and an embedding dimension de, the shared embedding matrix <NUM> at the prediction network is <MAT>. Meanwhile, a last hidden layer includes a dimension size dh at the joint network <NUM>, feed-forward projection weights from the hidden layer to the output logits will be <MAT>, with an extra blank token in the vocabulary. Accordingly, the feed-forward layer corresponding to the last layer of the joint network <NUM> includes a weight matrix [dh, |V]|. By having the prediction network <NUM> to tie the size of the embedding dimension de to the dimensionality dh of the last hidden layer of the joint network <NUM>, the feed-forward projection weights of the joint network <NUM> and the shared embedding matrix <NUM> of the prediction network <NUM> can share their weights for all non-blank symbols via a simple transpose transformation. Since the two matrices share all their values, the RNN-T decoder only needs to store the values once on memory, instead of storing two individual matrices. By setting the size of the embedding dimension de equal to the size of the hidden layer dimension dh, the RNN-T decoder reduces a number of parameters equal to the product of the embedding dimension de and the vocabulary size |V|. This weight tying corresponds to a regularization technique.

<FIG> is a plot <NUM> depicting word error rate (WER) versus the number of parameters of the RNN-T decoders. Here, <FIG> the plot <NUM> illustrates WER versus the number of parameters for a tied RNN-T decoder <NUM> (illustrated with the solid line), a non-tied RNN-T decoder <NUM> (illustrated with the dotted line), and a long-short term memory (LSTM) network <NUM> (illustrated with the dashed line). Specifically, the plot <NUM> depicts the size of the prediction and joint networks <NUM>, <NUM> with and without tied output and embeddings. The plot <NUM> shows varying embedding dimensions de to perform a sweep over the model size. As shown in <FIG>, the non-tied RNN-T decoder <NUM> includes four measurements including the embedding dimensions de of <NUM>, <NUM>, <NUM>, <NUM>, and <NUM>. Here, the tied RNN-T decoder <NUM> includes three measurements including the embedding dimensions of <NUM>, <NUM>, and <NUM>. In the non-tied RNN-T decoder <NUM> case, the last hidden layer of the joint network <NUM> always includes the dimensionality dh of size <NUM> (dimensionality dh not illustrated in the plot <NUM>). In the tied RNN-T decoder <NUM> case, the plot <NUM> also shows the dimensionality dh (dimensionality dh not illustrated in the plot <NUM>) of the last hidden layer of the joint network <NUM> equal to the size of the embedding dimension de of the prediction network <NUM> such that the size and performance of the RNN-T decoder is more sensitive to changes in that dimension. Accordingly, the results depicted by plot <NUM> indicate that weight-tying is more parameter efficient, thereby achieving better performance with fewer parameters. Additionally, for large-enough models using weight-tying, the same word error rate is reached as a conventional RNN-T decoder using the LSTM network <NUM>.

<FIG> is a flowchart of an exemplary arrangement of operations for a computer-implemented method <NUM> for executing a tied and reduced RNN-T model <NUM>. At each of a plurality of time steps subsequent to an initial time step, the method <NUM> performs operations <NUM>-<NUM>. At operation <NUM>, the method <NUM> includes receiving, as input to a prediction network <NUM> of a recurrent neural network-transducer (RNN-T) model <NUM>, a sequence of non-blank symbols <NUM>, 301a-n yui-n,. , yui-<NUM> output by a final Softmax layer <NUM>. For each non-blank symbol in the sequence of non-blank symbols received as input during the corresponding time step, the method <NUM> performs operations <NUM>-<NUM>. At operation <NUM>, the method <NUM> includes generating, by the prediction network <NUM>, using a shared embedding matrix <NUM>, an embedding <NUM> of the corresponding non-blank symbol. At operation <NUM>, the method <NUM> includes assigning, by the prediction network <NUM>, a respective position vector PVAa-An <NUM>, 308Aa-An to the corresponding non-blank symbol. At operation <NUM>, the method <NUM> includes weighting, by the prediction network <NUM>, the embedding <NUM> proportional to a similarity between the embedding <NUM> and the respective position vector <NUM>.

At operation <NUM>, the method <NUM> includes generating, as output from the prediction network <NUM>, a single embedding vector <NUM> at the corresponding time step. Here, the single embedding vector <NUM> is based on a weighted average 318A-H of the weighted embeddings 312Aa-An. At operation <NUM>, the method <NUM> includes generating, by a joint network <NUM> of the RNN-T model <NUM>, using the single embedding vector <NUM> generated as output from the prediction network <NUM> at the corresponding time step, a probability distribution P(yi|xti, y<NUM>,. , yui-<NUM>) over possible speech recognition hypotheses at the corresponding time step.

Claim 1:
A recurrent neural network-transducer (RNN-T) model (<NUM>) comprising: an audio encoder (<NUM>), a prediction network (<NUM>), a joint network (<NUM>) and a final Softmax layer (<NUM>);
the prediction network (<NUM>) configured to, at each of a plurality of time steps subsequent to an initial time step:
receive, as input, a sequence of non-blank symbols (<NUM>) output by the final Softmax layer (<NUM>);
for each non-blank symbol (<NUM>) in the sequence of non-blank symbols (<NUM>) received as input at the corresponding time step:
generate, using a shared embedding matrix (<NUM>), an embedding (<NUM>) of the corresponding non-blank symbol (<NUM>);
assign a respective position vector (<NUM>) to the corresponding non-blank symbol (<NUM>); and
weight the embedding (<NUM>) proportional to a similarity between the embedding (<NUM>) and the respective position vector (<NUM>); and
generate, as output, a single embedding vector (<NUM>) at the corresponding time step, the single embedding vector (<NUM>) based on a weighted average (<NUM>) of the weighted embeddings (<NUM>);
the audio encoder (<NUM>) configured to:
receive, as input, a sequence of acoustic frames (<NUM>); and
generate, at each of the plurality of time steps, a higher order feature representation for a corresponding acoustic frame (<NUM>) in the sequence of acoustic frames (<NUM>);
the joint network (<NUM>) configured to, at each of the plurality of time steps subsequent to the initial time step:
receive, as input, the single embedding vector (<NUM>) generated as output from the prediction network (<NUM>) at the corresponding time step;
receive the higher order feature representation generated by the audio encoder (<NUM>) at the corresponding time step as input;
generate a probability distribution over possible speech recognition hypotheses at the corresponding time step; and
the final Softmax layer (<NUM>) configured to select, based on the probability distribution over possible speech recognition hypotheses at the corresponding time step, a non-blank symbol with the highest probability in the distribution, to output by the final Softmax layer, as a next non-blank symbol in the sequence of non-blank symbols.