Patent ID: 12211509

Like reference symbols in the various drawings indicate like elements.

DETAILED DESCRIPTION

A recurrent neural network-transducer (RNN-T) architecture is an end-to-end solution (e.g., a single neural network model) that can be used for streaming automatic speech recognition (ASR) of streaming audio, among other uses. An RNN-T may be part of a speech recognition model or system. To estimate output distributions over word or subword units, an RNN-T includes a joint network to fuse (i) higher order feature representations (also referred to generally as acoustic representations) generated by an audio encoder with (ii) dense representations (also referred to generally as text representations) generated by a prediction network based on previously decoded text using a recurrent structure between the previous and current text in an output text sequence. The audio encoder receives, as input, a sequence of acoustic frames characterizing an input utterance, and generates, at each of a plurality of output steps, the higher order feature representation for a corresponding acoustic frame in the sequence of acoustic frames. The prediction network receives, as input, a sequence of non-blank symbols output by a final Softmax layer of the RNN-T, and generates, at each of the plurality of output steps, a dense representation. The joint network receives, as input, the dense representation generated by the prediction network at each of a plurality of output steps and the higher order feature representation generated by the audio encoder at each of the plurality of output steps, and generates, at each of the plurality of output steps, a probability distribution over possible speech recognition hypotheses. An output layer (e.g., a final Softmax layer) selects, based on the probability distribution, as an output transcription, the candidate transcription or hypothesis having the highest likelihood score of accurately representing the sequence of acoustic frames characterizing the input utterance.

In more particularity, the RNN-T performs ASR by finding the most probable text sequence y given a sequence of acoustic frames x1:T. According to Baye's rule, decoding may follow a maximum a posteriori rule to search over each possible hypothesized text sequence y using, for example, the following mathematical expression:
P(y|x1:T)∝p(x1:T|y)P(y)  (1)
where P(x1:T|y) is estimated by the audio encoder and represents the likelihood that x1:Twas spoken given y, and P(y) is estimated by the prediction network using a language model (LM) that represents an underlying probabilistic distribution of the text. RNN-T models P(y|x1:T) use a single end-to-end model (e.g., a single neural network). Assuming y=y1:U, where U is the number of subword units in y, then for streaming audio data without any look ahead frames or time reduction, a Dencdimensional higher order feature representation htencgenerated by the audio encoder at time t, a Dpreddimensional dense representation hupredof the uthsubword unit generated by the prediction network, and a Djointdimensional fused representation ht,ujointgenerated by the joint network can be expressed as follows:
htenc=AcousticEncoder(x1:t)  (2)
hupred=PredictionNetwork(y1:u-1)  (3)
ht,ujoint=JointNetwork(htenc,huPred)  (4)
P(ŷi=k|y0:u-1,x1:t)=Softmax(Woutht,ujoint)|k(5)
where y0refers to a special start of sentence symbol, and k and Woutare, respectively, the kthnode and weights of the output layer.

In some examples, the AcousticEncoder in EQN (2) includes a conformer encoder with a fixed number of look ahead frames and a fixed time reduction rate, the PredictionNetwork in EQN (3) includes a multi-layer long short-term memory (LSTM) model, and the JointNetwork in EQN (4) includes a fully connected (FC) layer, where
ht,ujoint=tan h(W1jointhtenc+W2jointhupred)  (6)
and W1jointand W2jointare weight matrices. When htencis ignored in EQN (6), the prediction network, joint network, and output layer jointly form an LSTM language model (LM) that may be referred to as an internal LM. However, studies show fusing the acoustic representations and text representations can improve ASR accuracy.

Gating has been used as a technique in recurrent structures to fuse information. For example, gating has been used in RNN-Ts to fuse acoustic and text representations by allowing each element in a representation vector to be scaled with a different weight before being integrated via, for example, vector addition. This allows, for example, the relative fusing of acoustic and text representations to be adjusted. In more particularity, with gating, the Djointdimensional fused representation ht,ujointgenerated by the joint network can be expressed mathematically as follows:
ht,ugate=gt,ugate⊙ tan h(W1jointhtenc)+1−gt,ugate)⊙ tan h(W2jointhupred)  (7)
where gt,ugateis a gating vector that can be expressed mathematically as follows:
gt,ugate=σ(W1gatehtenc+W2gatehupred)  (8)
where σ( ) is a sigmoid function, and W1gateand W2gateare weight matrices of the gating layer.

An even more powerful, albeit more computationally expensive, technique of fusing information, such as the acoustic and text representations, is bilinear pooling. Bilinear pooling combines the representation vectors using a bilinear form, which can be expressed as follows:
ht,u,djoint=(htenc)TWdbihupred(9)
where Wdbiis a Denc×Dpreddimensional matrix, and ht,u,djointis the dthelement of ht,ujoint. Considering all elements in ht,ujoint, [W1bi, . . . , WDjointbi] is a Denc×Dpred×Djointdimensional weight tensor, and fusing of htencand htpredusing bilinear pooling can be mathematically expressed as follows:
ht,u,djoint=([Vector(W1bi), . . . ,Vector(WDjointbi)])TVector(htenc⊗hupred)  (10)
where Vector and ⊗ refer, respectively, to vectorization and outer product. Compared to gating, bilinear pooling first computes the outer product of the two representation vectors to capture the multiplicative interactions between all possible element pairs in a more expressive Denc×Dpreddimensional space, and then projects that into a Djointdimensional vector space.

Implementations herein are directed toward combining the use of gating and bilinear pooling in the joint network of an RNN-T to balance and improve the fusion of higher order feature representations (also referred to generally as acoustic representations) encoded by the audio encoder from input acoustic frames and dense representations (also referred to generally as text representations). Disclosed herein are novel structures for the joint network of an RNN-T that include gating and bilinear pooling to improve the fusion of acoustic and text representations. By combining gating with bilinear pooling, resultant joint networks leverage the respective strengths and complimentary features of gating and bilinear pooling while fusing the text representation (i.e., dense representation) generated by the prediction network and the acoustic representation (i.e., first higher order feature representation) generated by the audio encoder.

It has been observed that, because text priors are often easier to learn than acoustic features, the prediction network of an RNN-T may converge faster than the audio encoder of the RNN-T. This may result in the joint network of the RNN-T becoming overly reliant on the text representations generated by the prediction network over the acoustic representations generated by the audio encoder when performing ASR on training utterances. For example, the joint network of the RNN-T may overly depend on hupredoutput by the prediction network when computing ht,ujoint. In such situations, the audio encoder may be less well trained to encode the audio samples that are associated with higher prediction network scores. In order to reduce these training imbalances, prediction network regularization routines may be applied, for example, at the start of training the RNN-T model. Implementations herein are further directed toward using prediction network regularization routines with the joint network having the novel combination structure (e.g., see EQN (11) below) that stacks gating and bilinear pooling to fuse the dense representation generated by the prediction network and the higher order feature representation generated by the encoder network, or with joint networks configured with other structures capable fusing acoustic and text representations (e.g., see EQN (6), EQN (7), EQN (9), or EQN (10)). Example prediction network regularization routines disclosed herein reduce the gradients back-propagated into the prediction network during training in order to optimally balance the fusing of htencand hupredby the joint network. For example, during training, the prediction network regularization routines re-compute the dense representation hupredusing a scaling factor and a stop gradient function having an input tensor with zero gradients.

FIG.1is an example of a speech environment100. In the speech environment100, a user's104manner of interacting with a computing device, such as a user device10, may be through voice input. The user device10(also referred to generally as a device10) is configured to capture sounds (e.g., streaming audio data) from one or more users104within the speech environment100. Here, the streaming audio data may refer to a spoken utterance106by the user104that functions as an audible query, a command for the device10, or an audible communication captured by the device10. Speech-enabled systems of the device10may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device10may correspond to any computing device associated with a user104and capable of receiving audio data. Some examples of user devices10include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, vehicle infotainment systems, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device10includes data processing hardware12and memory hardware14in communication with the data processing hardware12. The memory hardware14stores instructions that, when executed by the data processing hardware12, cause the data processing hardware12to perform one or more operations. The user device10further includes an audio system16with an audio capture device (e.g., microphone)16,16afor capturing and converting spoken utterances106within the speech environment100into electrical signals and a speech output device (e.g., a speaker)16,16bfor communicating an audible audio signal (e.g., as output audio data from the device10). While the user device10implements a single audio capture device16ain the example shown, the user device10may implement an array of audio capture devices16awithout departing from the scope of the present disclosure, whereby one or more capture devices16ain the array may not physically reside on the user device10, but be in communication with the audio system16.

In the speech environment100, an ASR system118implementing an ASR model, such as an RNN-T model200, and an optional rescorer180resides on the user device10of the user104and/or on a remote computing device60(e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device10via a network40. The user device10and/or the remote computing device60also includes an audio subsystem108configured to receive the utterance106spoken by the user104and captured by the audio capture device16a, and convert the utterance106into a corresponding digital format associated with input acoustic frames110capable of being processed by the ASR system118. In the example shown, the user speaks a respective utterance106and the audio subsystem108converts the utterance106into corresponding audio data (e.g., acoustic frames)110for input to the ASR system118. Thereafter, the RNN-T model200receives, as input, the audio data110corresponding to the utterance106, and generates/predicts, as output, a corresponding transcription120(e.g., recognition result/hypothesis) of the utterance106. In the example shown, the RNN-T model200may perform streaming speech recognition to produce an initial speech recognition result120,120aand the rescorer180may update (e.g., rescore) the initial speech recognition result120ato produce a final speech recognition result120,120b. The server60includes data processing hardware62, and memory hardware64in communication with the data processing hardware62. The memory hardware64stores instructions that, when executed by the data processing hardware62, cause the data processing hardware62to perform one or more operations, such as those disclosed herein.

The user device10and/or the remote computing device60also executes a user interface generator107configured to present a representation of the transcription120of the utterance106to the user104of the user device10. As described in greater detail below, the user interface generator107may display the initial speech recognition results120ain a streaming fashion during time1and subsequently display the final speech recognition result120bduring time2. In some configurations, the transcription120output from the ASR system118is processed, e.g., by a natural language understanding/processing (NLU/NLP) module executing on the user device10or the remote computing device60, to execute a user command or respond to a query specified by the utterance106. Additionally or alternatively, a text-to-speech system (TTS) (not shown) (e.g., executing on any combination of the user device10or the remote computing device60) may convert the transcription120into synthesized speech for audible output by the user device10and/or another device.

In the example shown, the user104interacts with a program or application50(e.g., the digital assistant application50) of the user device10that uses the ASR system118. For instance,FIG.1depicts the user104communicating with the digital assistant application50and the digital assistant application50displaying a digital assistant interface18on a screen of the user device10to depict a conversation between the user104and the digital assistant application50. In this example, the user104asks the digital assistant application50, “What time is the concert tonight?” This question from the user104is a spoken utterance106captured by the audio capture device16aand processed by audio systems16of the user device10. In this example, the audio system16receives the spoken utterance106and converts it into acoustic frames110for input to the ASR system118.

Continuing with the example, the RNN-T model200, while receiving the acoustic frames110corresponding to the utterance106as the user104speaks, encodes the acoustic frames110and then decodes the encoded acoustic frames110into the initial speech recognition results120a. During time1, the user interface generator107presents, via the digital assistant interface18, a representation of the initial speech recognition results120aof the utterance106to the user104of the user device10in a streaming fashion such that words, word pieces, and/or individual characters appear on the screen of the user device10as soon as they are spoken. In some examples, first look ahead audio context is set equal to zero.

During time2, the user interface generator107presents, via the digital assistant interface18, a representation of the final speech recognition result120bof the utterance106to the user104of the user device10rescored by the rescorer180. In some implementations, the user interface generator107replaces the representation of the initial speech recognition results120apresented at time1with the representation of the final speech recognition result120bpresented at time2. Here, time1and time2may include timestamps corresponding to when the user interface generator107presents the respective speech recognition result120. In this example, the timestamp of time1indicates that the user interface generator107presents the initial speech recognition result120aat an earlier time than the final speech recognition result120b. For instance, as the final speech recognition result120bis presumed to be more accurate than the initial speech recognition results120a, the final speech recognition result120bthat is ultimately displayed as the transcription120may fix any terms that may have been misrecognized in the initial speech recognition results120a. In this example, the streaming initial speech recognition results120aoutput by the RNN-T model200displayed on the screen of the user device10at time1are associated with low latency and provide responsiveness to the user104that his/her query is being processed, while the final speech recognition result120boutput by the rescorer180and displayed on the screen at time2leverages an additional speech recognition model and/or a language model to improve the speech recognition quality in terms of accuracy, but at increased latency. However, because the initial speech recognition results120aare displayed as the user speaks the utterance106, the higher latency associated with producing, and ultimately displaying the final recognition result120bis not noticeable to the user104.

In the example shown inFIG.1, the digital assistant application50may respond to the question posed by the user104using natural language processing (NLP). NLP generally refers to a process of interpreting written language (e.g., the initial speech recognition results120aand/or the final speech recognition result120b) and determining whether the written language prompts any response or action. In this example, the digital assistant application50uses NLP to recognize that the question from the user104regards the user's schedule and, more particularly, a concert on the user's schedule. By recognizing these details with NLP, the automated assistant returns a response19to the user's question, where the response19states, “Venue doors open at 6:30 PM and concert starts at 8 pm.” In some configurations, NLP occurs on a remote server60in communication with the data processing hardware12of the user device10.

FIG.2is a schematic view of an example RNN-T model200that fuses higher order feature representations (also referred to generally as acoustic representations)224output by the audio encoder network220and dense representations (also referred to generally as text representations)232output by the prediction network230. In particular, the RNN-T model200includes a novel joint network210that combines gating with bilinear pooling to improve the fusion of the acoustic and text representations224,232. By combining gating with bilinear pooling, the joint network210leverages the respective strengths and complimentary features of gating and bilinear pooling.

As shown, the RNN-T model200includes an encoder network220, a prediction/decoder network230, the joint network210, and a final Softmax output layer240. The encoder network220(e.g., an audio encoder), which is roughly analogous to an acoustic model (AM) in a traditional ASR system, receives a sequence of feature vectors (e.g., the acoustic frames110ofFIG.1) x=(x1, x2, . . . , xt)222, where xi∈d, and produces at each output step a higher-order feature representation (e.g., acoustic representation)224denoted as htenc=(h1enc, . . . , htenc).

In the example shown, the prediction/decoder network230includes an LSTM-based prediction network that, like a language model (LM), processes a sequence of non-blank symbols y0, . . . , yu-1242output so far by the Softmax layer240into a dense representation hupred232, where y0represents a special start of sequence symbol.

The joint network210fuses the representations htenc224and hupred232produced, respectively, by the encoder network220and the prediction network230. The joint network210generates P(ŷi|xt, y0, . . . , yu-1), which is a distribution over the next candidate output symbol. Stated differently, the joint network210generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses212. Here, the “possible speech recognition hypotheses” correspond to a set of output labels each representing a word/wordpiece/symbol/character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, e.g., one label for each of the 26-letters in the English alphabet and one label designating a space. Accordingly, the joint network210may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network210can include a posterior probability value for each of the different output labels. Thus, when there are 100 different output labels representing different graphemes or other symbols, the output yiof the joint network210can include 100 different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the final Softmax output layer240) for determining the transcription120.

The joint network210includes a novel structure that combines gating with bilinear pooling to improve the fusion of the higher order feature representation htenc224and the dense representation hupred232received by the joint network210at each of the plurality of output steps when performing speech recognition on an utterance106(FIG.1). In the example shown, the joint network210includes a bilinear pooling layer250and a gating layer260. In some examples, the bilinear pooling layer250is stacked on top of the gating layer260. In these examples, the stacking of the bilinear pooling layer250and the gating layer260can be expressed mathematically as follows:
ht,ujoint=Wproj(tan h((W1gate)Thtenc))⊙(tan h((W2gate)Tht,ugate))  (11)
where ht,ugaterefers to the joint representation of EQN (7), and W1gateand W2gateare weight matrices of the gating layer260.

The Softmax layer240may employ any technique to select the output label/symbol with the highest probability in the distribution ĥt,ujoint232as the next output symbol predicted by the RNN-T model200at the corresponding output step. In this manner, the RNN-T model200does not make a conditional independence assumption. Instead, the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model200does assume an output symbol is independent of future acoustic frames110, which allows the RNN-T model200to be employed in a streaming fashion. In some examples, the Softmax layer240is composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in a plurality of training data sets.

In some examples, the feature vectors x222include 80-dimensional log-Mel filter bank features formed by stacking three 32 millisecond (ms) acoustic frames with a 10 ms shift to form a 240-dimensional input representation with a 30 ms frame rate, which are then transformed using a first linear projection to a 512-dimensional representation with added positional embeddings. Continuing with this example, the encoder network220may include twelve conformer encoder blocks with 8-head self-attention and a convolution kernel size of 15 to further transform the stacked features. Here, the encoder network220performs a concatenation operation after the third conformer block to achieve a time reduction rate of two. A fourth conformer block transforms the resulting 1024-dimensional vectors, and then the encoder network220projects them back to 512-dimensional using a second linear transform. The remaining eight conformer blocks follow the second linear transformation, followed by a final linear normalization layer to make the dimension Denc=512 for the higher order feature representation htenc224. While the encoder network220described has a stack of multi-head attention layers/blocks that include conformer layers/blocks (e.g., twelve conformer blocks), the present disclosure is not so limited. For instance, the encoder network220may include a stack of transformer layers/blocks or other type of multi-head attention layers/bocks. The encoder network220may include a series of multi-headed self-attention, depth-wise convolutional and feed-forward layers. Alternatively, the encoder network220may include a plurality of long-short term memory (LSTM) layers in lieu of multi-head attention layers/blocks.

Continuing with the example, the prediction network230is an LTSM-based network including two layers of 2,048-dimensional LSTM with a 640-dimensional linear projection to make Dpred=640 for the dense representation hupred232. The dimension Djointof the fused representation ĥt,ujoint232is also set to 640. In some examples, the joint network210includes hidden units. Additionally or alternatively, the joint network210does not include a fully connected (FC) layer.

Alternatively, the encoder network220includes a stack of self-attention layers/blocks. Here, the stack of self-attention blocks may include a stack of transformer blocks, or a different stack of conformer blocks.

Alternatively, the prediction network230may include a stack of transformer or conformer blocks (or other type of multi-head attention blocks). The prediction network230may also be replaced with an embedding look-up table (e.g., a V2 embedding look-up table) to improve latency by outputting looked-up sparse embeddings in lieu of generating dense representations. In some implementations, the prediction network230is a stateless prediction network.

As discussed above, the prediction network230may converge faster than the encoder network220during training, which may result in the joint network210becoming overly reliant on the dense representations hupred232generated by the prediction network230over the higher order feature representations htenc224generated by the encoder network220when performing ASR on training utterances. In order to reduce such training imbalances, prediction network regularization routines may be applied, for example, at the beginning of training the RNN-T model200. More specifically, training of the RNN-T model may include using the prediction network regularization routines together with the joint network having the novel combination structure (e.g., see EQN (11)) that stacks gating and bilinear pooling in order to balance the fusing of the dense representations232generated by the prediction network230and the higher order feature representations224generated by the encoder network220, or with joint networks configured with other structures capable fusing acoustic and text representations (e.g., see EQN (6), EQN (7), EQN (9), or EQN (10)). In some examples, the prediction network regularization routines reduce the gradients back-propagated into the prediction network230during training in order to optimally balance the fusing of htenc224and hupred232by the joint network210. For example, applying the prediction network regularization routines during training may re-compute the dense representation hupred232using a scaling factor and a stop gradient function having an input tensor with zero gradients. For instance, re-computing the dense representation hupred232can be expressed as follows:
hupred=αmhupred−sg((αm−1)hupred)  (12)
where m is the index in the current training step, αmis a scaling factor, and sg( ) is the stop gradient functions whose input tensor will have zero gradients. In this example, when 0≤αm≤1, the value of hupredis not changed, but the corresponding gradients that are back-propagated into the prediction network230will be reduced by a factor of αm. This slows down the convergence of the prediction network230, and allows for balancing the fusing of the htenc224and hupred232by the joint network210during training. In some examples, the prediction network regularization routine selects the value of αmusing a piece-wise linear schedule as follows:

αm={0if⁢m<m1m/(m2-m1)else⁢if⁢m1≤m≤m2,1otherwise(13)
where m1and m2are two pre-defined parameters. Notably, applying the prediction network regularization routine is different from initializing the RNN-T model200with a pre-trained connectionist temporal classification (CTC) model, even when m=0, because the prediction network230provides a random but fix-valued projection through which the RNN-T model200is still able to obtain yu-1. Compared to other conventional training techniques, training the joint network210with the prediction network regularization routine improves the integration of the internal LM during both training and test-time by initially discounting the internal LM during training. Notably, the joint networks210and/or prediction network regularization routines are applicable to stateless RNN-T models in which the LM history embedded in the prediction network230is limited and/or reset for each utterance.

FIG.3is a schematic view of an example conformer block300that can be used to implement one of the conformer blocks in the stack of conformer blocks of the encoder network220ofFIG.2. The conformer block300includes a first half feed-forward layer310, a second half feed-forward layer340, with a multi-head self-attention block320and a convolution layer330disposed between the first and second half feed-forward layers310,340, and concatenation operators305. The first half feed-forward layer310processes the input audio data102including an input Mel-spectrogram sequence. Subsequently, the multi-head self-attention block320receives the input audio data102concatenated with the output of the first half-feed forward layer310. Intuitively, the role of the multi-head self-attention block320is to summarize noise context separately for each input frame that is to be enhanced. The convolution layer330subsamples the output of the multi-head self-attention block320concatenated with the output of the first half feed forward layer310. Thereafter, the second half-feed forward layer340receives a concatenation of the convolution layer330output and the multi-head self-attention block320. A layernorm module350processes the output from the second half feed-forward layer340. The conformer block300transforms input features x, using modulation features m, to produce output features y360, which can, for example, be mathematically expressed as:

x^=x+r⁡(m)⊙x+h⁡(m)⁢x~=x^+12⁢FFN⁡(x^),n~=n+12⁢FFN⁡(n)⁢x′=x~+Conv⁡(x~),n′=n~+Conv⁡(n~)⁢x″=x′+MHCA⁡(x′,n′)⁢x′′′=x′⊙r⁡(x″)+h⁡(x″)⁢x′′′′=x′+MHCA⁡(x′,x′′′)⁢y=LayerNorm⁡(x′′′′+12⁢FFN⁡(x′′′′)).

FIG.4is a flowchart of an exemplary arrangement of operations for a computer-implemented method400of improving the fusion of acoustic and text representations in an RNN-T model, such as the RNN-T model200. Data processing hardware510(e.g., the data processing hardware12of the device10and/or the data processing hardware62of the computing system60ofFIG.1) may execute the operations for the method400by executing instructions stored on memory hardware520(e.g., the memory hardware14,64).

At operation402, the method400includes receiving a sequence of acoustic frames x=(x1, x2, . . . , xt) 222 characterizing an input utterance106. The method400performs operations404,406,408at each of a plurality of output steps. At operation404, the method400includes generating, by an encoder network220of the RNN-T model200, a higher order feature representation htenc224for a corresponding acoustic frame222in the sequence of acoustic frames222.

At operation406, the method400includes generating, by a prediction network230of the RNN-T model200, a dense representation hupred232for a corresponding sequence of non-blank symbols (y0, . . . , yu-1)242output by a final Softmax output layer (e.g., the Softmax layer240). Here, y0may represent a special start of sequence symbol.

At operation408, the method400includes generating, by a joint network210of the RNN-T model200that receives the higher order feature representation htenc224and the dense representation hupred232a probability distribution ĥt,ujoint212over possible speech recognition hypotheses ŷu. For example, the joint network210may generate the probability distribution ĥt,ujoint212using a bilinear pooling layer250stacked on a gating layer260, as discussed above in connection with the joint network210ofFIG.2. For example, at operation408, the method400can use EQN (11) to compute the probability distribution ĥt,ujoint212as output from the joint network210.

FIG.5is schematic view of an example computing device500that can be used to implement the systems and methods described in this document. The computing device500is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computer devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and/or claimed in this document.

The computing device500includes a processor510(i.e., data processing hardware) that can be used to implement the data processing hardware12and/or62, memory520(i.e., memory hardware) that can be used to implement the memory hardware14and/or64, a storage device530(i.e., memory hardware) that can be used to implement the memory hardware14and/or64, a high-speed interface/controller540connecting to the memory520and high-speed expansion ports550, and a low speed interface/controller560connecting to a low speed bus570and a storage device530. Each of the components510,520,530,540,550, and560, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor510can process instructions for execution within the computing device500, including instructions stored in the memory520or on the storage device530to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display580coupled to high speed interface540. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices500may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory520stores information non-transitorily within the computing device500. The memory520may be a computer-readable medium, a volatile memory unit(s), or non-volatile memory unit(s). The non-transitory memory520may be physical devices used to store programs (e.g., sequences of instructions) or data (e.g., program state information) on a temporary or permanent basis for use by the computing device500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM)/programmable read-only memory (PROM)/erasable programmable read-only memory (EPROM)/electronically erasable programmable read-only memory (EEPROM) (e.g., typically used for firmware, such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM) as well as disks or tapes.

The storage device530is capable of providing mass storage for the computing device500. In some implementations, the storage device530is a computer-readable medium. In various different implementations, the storage device530may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory520, the storage device530, or memory on processor510.

The high speed controller540manages bandwidth-intensive operations for the computing device500, while the low speed controller560manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller540is coupled to the memory520, the display580(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports550, which may accept various expansion cards (not shown). In some implementations, the low-speed controller560is coupled to the storage device530and a low-speed expansion port590. The low-speed expansion port590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device500may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server500aor multiple times in a group of such servers500a, as a laptop computer500b, or as part of a rack server system500c.

Various implementations of the systems and techniques described herein can be realized in digital electronic and/or optical circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

A software application (i.e., a software resource) may refer to computer software that causes a computing device to perform a task. In some examples, a software application may be referred to as an “application,” an “app,” or a “program.” Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, non-transitory computer readable medium, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and/or data to a programmable processor.

The processes and logic flows described in this specification can be performed by one or more programmable processors, also referred to as data processing hardware, executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuitry, e.g., an FPGA (field programmable gate array) or an ASIC (application specific integrated circuit). Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor will receive instructions and data from a read only memory or a random access memory or both. The essential elements of a computer are a processor for performing instructions and one or more memory devices for storing instructions and data. Generally, a computer will also include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto optical disks, or optical disks. However, a computer need not have such devices. Computer readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto optical disks, and CD ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, one or more aspects of the disclosure can be implemented on a computer having a display device, e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen for displaying information to the user and optionally a keyboard and a pointing device, e.g., a mouse or a trackball, by which the user can provide input to the computer. Other kinds of devices can be used to provide interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In addition, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's client device in response to requests received from the web browser.

Unless expressly stated to the contrary, “or” refers to an inclusive or and not to an exclusive or. For example, “A, B, or C” refers to any combination or subset of A, B, C such as: (1) A alone; (2) B alone; (3) C alone; (4) A with B; (5) A with C; (6) B with C; and (7) A with B and with C. Similarly, the phrase “at least one of A or B” is intended to refer to any combination or subset of A and B such as: (1) at least one A; (2) at least one B; and (3) at least one A and at least one B. Moreover, the phrase “at least one of A and B” is intended to refer to any combination or subset of A and B such as: (1) at least one A; (2) at least one B; and (3) at least one A and at least one B.

A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.