Robustness Aware Norm Decay for Quantization Aware Training and Generalization

A method includes obtaining a plurality of training samples, determining a minimum integer fixed-bit width representing a maximum quantization of an automatic speech recognition (ASR) model, and training the ASR model on the plurality of training samples using a quantity of random noise. The ASR model includes a plurality of weights that each include a respective float value. The quantity of random noise is based on the minimum integer fixed-bit value. After training the ASR model, the method also includes selecting a target integer fixed-bit width greater than or equal to the minimum integer fixed-bit width, and for each respective weight of the plurality of weights, quantizing the respective weight from the respective float value to a respective integer associated with a value of the selected target integer fixed-bit width. The operations also include providing the quantized trained ASR model to a user device.

TECHNICAL FIELD

This disclosure relates to robustness aware norm decay for quantization aware training and generalization.

BACKGROUND

Modern automated speech recognition (ASR) systems focus on providing not only high quality (e.g., a low word error rate (WER)), but also low latency (e.g., a short delay between the user speaking and a transcription appearing). Moreover, when using an ASR system today there is a demand that the ASR system decode utterances in a streaming fashion that corresponds to real-time or even faster than real-time. To illustrate, when an ASR system is deployed on a mobile phone that experiences direct user interactivity, an application on the mobile phone using the ASR system may require the speech recognition to be streaming such that words appear on the screen as soon as they are spoken. Here, it is also likely that the user of the mobile phone has a low tolerance for latency. Due to this low tolerance, the speech recognition strives to run on the mobile device in a manner that minimizes an impact from latency and inaccuracy that may detrimentally affect the user's experience. However, mobile phones often have limited resources, which limit the size of the ASR model.

SUMMARY

One aspect of the disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations that include obtaining a plurality of training samples, determining a minimum integer fixed-bit width representing a maximum quantization of an automatic speech recognition (ASR) model, and training the ASR model on the plurality of training samples using a quantity of random noise. Each respective training sample of the plurality of training samples includes a respective speech utterance and a respective textual utterance representing a transcription of the respective speech utterance. The ASR model includes a plurality of weights, wherein each respective weight of the plurality of weights includes a respective float value. The quantity of random noise used for training the ASR model is based on the minimum integer fixed-bit value. After training the ASR model, the operations also include selecting a target integer fixed-bit width greater than or equal to the minimum integer fixed-bit width, and for each respective weight of the plurality of weights, quantizing the respective weight from the respective float value to a respective integer with the selected target integer fixed-bit width. The operations also include providing the quantized trained ASR model to a user device.

Implementations of the disclosure may include one or more of the following optional features. In some implementations, training the ASR model using the quantity of random noise includes, for each respective channel of each respective tensor of the ASR model, determining a respective maximum value for the respective channel of the respective tensor, and adding, to the respective channel of the respective tensor, a uniform distribution of noise based on the respective maximum value and the minimum integer fixed-bit width. In these implementations, the uniform distribution of noise may represent the entire range of noise the ASR model receives due to quantization up to the minimum integer fixed-bit width. Additionally, adding the uniform distribution of noise may include scaling the uniform distribution of noise based on the respective maximum value, while scaling the uniform distribution of noise may be further based on a sensitivity of the respective channel to scaling.

In some examples, the maximum quantization level includes a 4-bit quantization. In other examples, the maximum quantization level includes a 2-bit quantization. The random noise may be drawn from a uniform distribution of noise.

In some additional implementations, training the ASR model using the quantity of random noise includes adding, during forward propagation of the ASR model, the quantity of random noise. The ASR model may further include a plurality of activations each associated with a respective float value such that for each respective action of the plurality of activations, the operations further include quantizing the respective activation from the respective float value to the respective integer with the selected target fixed-bit width.

Another aspect of the disclosure provides a system that includes data processing hardware and memory hardware storing instructions that when executed on the data processing hardware causes the data processing hardware to perform operations that include obtaining a plurality of training samples, determining a minimum integer fixed-bit width representing a maximum quantization of an automatic speech recognition (ASR) model, and training the ASR model on the plurality of training samples using a quantity of random noise. Each respective training sample of the plurality of training samples includes a respective speech utterance and a respective textual utterance representing a transcription of the respective speech utterance. The ASR model includes a plurality of weights, wherein each respective weight of the plurality of weights includes a respective float value. The quantity of random noise used for training the ASR model is based on the minimum integer fixed-bit value. After training the ASR model, the operations also include selecting a target integer fixed-bit width greater than or equal to the minimum integer fixed-bit width, and for each respective weight of the plurality of weights, quantizing the respective weight from the respective float value to a respective integer with the selected target integer fixed-bit width. The operations also include providing the quantized trained ASR model to a user device

This aspect of the disclosure may include one or more of the following optional features. In some implementations, training the ASR model using the quantity of random noise includes, for each respective channel of each respective tensor of the ASR model, determining a respective maximum value for the respective channel of the respective tensor, and adding, to the respective channel of the respective tensor, a uniform distribution of noise based on the respective maximum value and the minimum integer fixed-bit width. In these implementations, the uniform distribution of noise may represent the entire range of noise the ASR model receives due to quantization up to the minimum integer fixed-bit width. Additionally, adding the uniform distribution of noise may include scaling the uniform distribution of noise based on the respective maximum value, while scaling the uniform distribution of noise may be further based on a sensitivity of the respective channel to scaling.

In some examples, the maximum quantization level includes a 4-bit quantization. In other examples, the maximum quantization level includes a 2-bit quantization. The random noise may be drawn from a uniform distribution of noise.

In some additional implementations, training the ASR model using the quantity of random noise includes adding, during forward propagation of the ASR model, the quantity of random noise. The ASR model may further include a plurality of activations each associated with a respective float value such that for each respective action of the plurality of activations, the operations further include quantizing the respective activation from the respective float value to the respective integer with the selected target fixed-bit width

Another aspect of the present disclosure provides a computer-implemented method that when executed on data processing hardware causes the data processing hardware to perform operations that include obtaining a plurality of training samples, determining a minimum integer fixed-bit width representing a maximum quantization of a model, and training the model on the plurality of training samples using a quantity of random noise. The model includes a plurality of weights, wherein each respective weight of the plurality of weights includes a respective float value. The quantity of random noise used for training the model is based on the minimum integer fixed-bit value. After training the model, the operations also include selecting a target integer fixed-bit width greater than or equal to the minimum integer fixed-bit width, and for each respective weight of the plurality of weights, quantizing the respective weight from the respective float value to a respective integer with the selected target integer fixed-bit width. The operations also include providing the quantized trained ASR model to a user device.

This aspect of the disclosure may include one or more of the following optional features. In some implementations, the model includes an automated speech recognition (ASR) model. In other implementations, the model includes a large language model. In yet other implementations, the model includes an image processing model.

DETAILED DESCRIPTION

With the fast growth of voice search and speech-interactive features, automatic speech recognition (ASR) has become an essential component for user-interactive services and devices (e.g., search by voice functions in search engines and smartphones). Modern ASR applications are often developed based on an end-to-end model, which has been shown to achieve significant recognition performance improvements compared to conventional hybrid systems with a much smaller model size. Improving latency and model size without compromising recognition quality has been an active pursuit to benefit live ASR applications with both server-side and on-device models.

Quantization is a technique to reduce the computational and memory costs of ASR models by representing the weights and/or activations with lower precision data types (e.g., an 8-bit integer) instead of a conventional 32-bit floating point value. Among modern model quantization methods, post training quantization (PTQ) with 8-bit integers (int8) is a popular and easy to use technique that has been successfully applied in many applications. However, one of the drawbacks of such a technique is the potential performance degradation due to the loss of precision. Moreover, PTQ, during training, generally does not expose the model to noise the model experiences after training, which causes accuracy to further suffer.

Quantization aware training (QAT) is an alternative technique to PTQ that emulates quantization during the forward pass in training, which allows the network to learn from the quantization. That is, the quantization emulation introduces error to the model which the model attempts to compensate for by adjusting parameters (i.e., weights). Thus, QAT tends to reduce loss versus conventional PTQ methods. However, conventional QAT relies on the user committing to the amount of quantization during training. That is, conventional QAT requires the user to select what weights and/or activations will be quantized and to what extent (e.g., 8-bit, 4-bit, etc.) during training. The resulting trained model will generally lose accuracy whenever quantized in a manner different from what was emulated during training. For example, a model trained using QAT for 8-bit quantization will be less accurate at 4-bit quantization than a model trained using 4-bit quantization. The reverse is also true in that a model trained using QAT for 4-bit quantization will be less accurate at 8-bit quantization than a model trained using 8-bit quantization. Put another way, the model tends to be tuned to the particular use case dictated during training. This requires training separate models for each potential quantization use case. Because very large and complex models (such as large language models (LLMs)) take considerable resources to train and store, the training of models individually is not ideal.

Implementations herein are directed toward a model trainer that trains an ASR model (or a large language model (LLM) or image processing model) using robustness aware norm decay for QAT. The model trainer trains the ASR model to be robust to quantization with a per channel scale by adding an expected noise distribution during training. For example, the model trainer determines a worst-case amount of noise the ASR model is subject to from quantization. The model trainer adds a uniform distribution of noise to the ASR model during training based on the worst-case amount of noise. The model trainer improves the robustness of learned weights to deployment time perturbation by pushing the converged set of weights toward a flat minimum, which lowers the loss increase after rounding. The model trainer also makes the converged set of weights closer to the quantization centroid to decrease rounding noise. Additionally, the model trainer trains the ASR model to be both robust at lower precisions and higher precisions compared to conventional models. In some implementations, the model trainer lowers the quantization scale parameter (i.e., the matrix norm of the weight matrix) in order to ensure the rounding noise is less scaled. While specific examples depict the training and quantization of an ASR model, the techniques disclosed herein are equally applicable to training and quantizing other types of models, such as, without limitation, large language models (LLMs) and image processing models.

FIG.1Ais an example of a system100operating in a speech environment101. In the speech environment101, a user's104manner of interacting with a computing device, such as a user device10, may be through voice input. The user device10(also referred to generally as a device10) is configured to capture sounds (e.g., streaming audio data) from one or more users104within the speech environment101. Here, the streaming audio data may refer to a spoken utterance106by the user104that functions as an audible query, a command for the device10, or an audible communication captured by the device10. Speech-enabled systems of the device10may field the query or the command by answering the query and/or causing the command to be performed/fulfilled by one or more downstream applications.

The user device10may correspond to any computing device associated with a user104and capable of receiving audio data. Some examples of user devices10include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, internet of things (IoT) devices, vehicle infotainment systems, smart displays, smart speakers, etc. The user device10includes data processing hardware12and memory hardware14in communication with the data processing hardware12and stores instructions, that when executed by the data processing hardware12, cause the data processing hardware12to perform one or more operations. The user device10further includes an audio system16with an audio capture device (e.g., microphone)16,16afor capturing and converting spoken utterances106within the speech environment101into electrical signals and a speech output device (e.g., a speaker)16,16bfor communicating an audible audio signal (e.g., as output audio data from the device10). While the user device10implements a single audio capture device16ain the example shown, the user device10may implement an array of audio capture devices16awithout departing from the scope of the present disclosure, whereby one or more capture devices16ain the array may not physically reside on the user device10, but be in communication with the audio system16.

In the speech environment101, an automated speech recognition (ASR) system118includes a model200(such as a recurrent neural network-transducer (RNN-T) model or other conformer transducer model/multi-pass model) that resides on the user device10of the user104and/or on a remote computing device60(e.g., one or more remote servers of a distributed system executing in a cloud-computing environment) in communication with the user device10via a network40. The remote computing device is equipped with data processing hardware62and memory hardware64. The user device10and/or the remote computing device60also includes an audio subsystem108configured to receive the utterance106spoken by the user104and captured by the audio capture device16a, and convert the utterance106into a corresponding digital format associated with input acoustic frames110capable of being processed by the ASR system118. In the example shown, the user speaks a respective utterance106and the audio subsystem108converts the utterance106into corresponding audio data (e.g., acoustic frames)110for input to the ASR system118. Thereafter, the model200receives, as input, the audio frames110(i.e., audio data) corresponding to the utterance106, and generates/predicts, as output, a corresponding transcription120(e.g., speech recognition result/hypothesis) of the utterance106.

The user device10and/or the remote computing device60also executes a user interface generator107configured to present a representation of the transcription120of the utterance106to the user104of the user device10. As described in greater detail below, the user interface generator107may display the speech recognition results120in a streaming fashion. In some configurations, the transcription120output from the ASR system118is processed, e.g., by a natural language understanding (NLU) module executing on the user device10or the remote computing device60, to execute a user command/query specified by the utterance106. Additionally or alternatively, a text-to-speech system (not shown) (e.g., executing on any combination of the user device10or the remote computing device60) may convert the transcription into synthesized speech for audible output by the user device10and/or another device.

In the example shown, the user104interacts with a program or application50(e.g., the digital assistant application50) of the user device10that uses the ASR system118. For instance,FIG.1Adepicts the user104communicating with the digital assistant application50and the digital assistant application50displaying a digital assistant interface18on a screen of the user device10to depict a conversation between the user104and the digital assistant application50. In this example, the user104asks the digital assistant application50, “What time is the concert tonight?” This question from the user104is a spoken utterance106captured by the audio capture device16aand processed by audio systems16of the user device10. In this example, the audio system16receives the spoken utterance106and converts it into acoustic frames110for input to the ASR system118.

Referring now toFIG.1B, the remote computing device60executes a model trainer150to train the model200ofFIG.1A. The model trainer150obtains a plurality of training samples152,152a-n(e.g., from the memory hardware64). Each training sample152includes a spoken training utterance154(i.e., a sequence of input audio features) paired with a corresponding textual utterance156representing a transcription156of the utterance154. The model trainer150determines a minimum integer fixed-bit width160. The minimum integer fixed-bit width160represents a maximum quantization of an ASR model200. That is, the minimum integer fixed-bit width160represents the smallest integer bit-width that the model200may be quantized to. In some examples, the minimum integer fixed-bit width160is 4-bit. In other examples, the minimum integer fixed-bit width160is 2-bit. Any other bit width is possible.

The model trainer150trains the ASR model200on the plurality of training samples152. The ASR model200includes weights202and activations204. In some examples, the weights202and/or activations204are 32-bit float values. In other examples, the weights202and/or activations204are other float values. As discussed in more detail below, the model trainer150uses quantization aware training to adjust the weights202and/or activations204of the ASR model200by adding a quantity of random noise162to the weights202and/or the activations204during training.

After training the ASR model200, a model quantizer170selects a target integer fixed-bit width172for the ASR model200. The target integer fixed-bit width172may be any value greater than or equal to the minimum integer fixed-bit width160. For example, when the minimum integer fixed-bit width160is 4-bit, the target integer fixed-bit width172may be 4-bit, 8-bit, 16-bit, 32-bit, etc. The target integer fixed-bit width172may be pre-determined for the use case of the ASR model200or based on the user device10(or whatever computing the device the ASR model200is to execute on). For example, the target integer fixed-bit width172may be 4-bit when the computing device (e.g., the user device10) has minimal computing resources. In another example, the target integer fixed-bit width172may be 8-bit when the computing device has more significant computing resources. In some examples, the target integer fixed-bit width172is configurable (e.g., by the user104). The model quantizer170, for some or all of the weights202of the ASR model200, quantizes the respective weight202from the respective float value (e.g., 32-bit float value) to a respective integer with a value of the selected target integer fixed-bit width172. For example, when the selected target integer fixed-bit width172is 4-bit, the model quantizer170quantizes the float values of the weights202and/or the activations204to 4-bit integer values. In some implementations, the remote computing device60provides the quantized model200,200Q with the quantized weights202,202Q and/or the quantized activations204,204Q to the user device10.

Neural networks generally involve tensors that include matrix multiplications. The matrix multiplication of the tensors may be modeled as y=Wx where x is an input column vector, W is a weight matrix, and y is an output column vector. Accordingly, each row may be modeled by yi=Wix where i is the row. Each row may be referred to as a channel of the respective tensor. Conventionally, the values of the weight matrix (i.e., W) are 32-bit float values. During quantization, these float values are converted to a corresponding integer value (i.e., a 4-bit integer, an 8-bit integer, etc.). This necessarily involves a loss of precision via rounding. This rounding introduces noise to the weights, and this noise may introduce loss of accuracy during inference. Quantization aware training attempts to make the network aware of this noise by introducing noise comparable to the noise introduced during quantization. For example, during conventional QAT, a model that is to be quantized to 8-bit integers has noise introduced that emulates the noise generated by rounding the float values to 8-bit integers. However, this noise is specific to the specific quantization the model is trained for, making the model sub-optimal for other quantization values.

In contrast, the model trainer150introduces the ASR model200to noise that allows the model200to adapt to noise present for different quantization values simultaneously. In some implementations, the model trainer150, while training the ASR model200, randomly draws the noise162from a uniform distribution of noise. In some of these implementations, the model trainer150determines a respective maximum value for each channel of each tensor of the ASR model200. For example, the maximum value represents the absolute value of the maximum weight value for the channel. The model trainer150adds noise to each respective channel of each respective tensor of the ASR model200(e.g., during forward propagation) separately and independently based on the maximum value for the respective channel and the minimum integer fixed-bit width160. In some implementations, the uniform distribution of noise represents the entire range of noise the ASR model200may receive due to quantization up to the minimum integer fixed-bit width160. Generally speaking, the greater the quantization (i.e., the smaller the integer width the weights202and/or activations204are rounded to), the greater the noise introduced to the ASR model200. The uniform distribution of noise may include up to the maximum amount of noise the ASR model200may be introduced to based on the minimum integer fixed-bit width160. For example, when the minimum integer fixed-bit width160is 4-bit, the uniform distribution includes noise up to the maximum the ASR model200may experience when quantized to 4-bit integers. In some implementations, the model trainer150adds noise per the following equation, where y is the output, W is the weight, bit is the minimum integer fixed-bit width160(i.e., four for 4-bit quantization, eight for 8-bit quantization, etc.), and N is the added noise.

yi=(Wi+max⁡(❘"\[LeftBracketingBar]"Wi❘"\[RightBracketingBar]")2b⁢i⁢t-1-1⁢Ni)⁢x,
where every entry of Niis drawn from Unif(−0.5,0.5)  (1)

By sampling the added noise from such a uniform distribution, the model trainer150exposes the ASR model200to noise that emulates a range of quantization from the maximum quantization represented by the minimum integer fixed-bit width160to no quantization. This allows the ASR model200to learn to adapt to each of these different noise scenarios. The model trainer150adds this noise on a per channel basis because adding the noise globally (e.g., at the tensor level) would introduce incorrect per noise power, which would degrade performance of the ASR model200.

In some implementations, the model trainer150scales the added noise to attenuate outliers (i.e., general robustness aware norm decay). That is, the model trainer150may make the ASR model200“scale-aware” so that information from the noise gradient is not discarded during training. For example, the model trainer may instead model max(|Wi|) from the above equation as maybe_stop_grad(|Wi|) to simultaneously induce robustness to noise and provide a gradient to selectively attenuate outliers. That is, in some implementations, the model trainer150back propagates to the norm term ∥Wi∥, which allows the network to learn to lower the scale in a principled way. For example, channels that are more sensitive to scaling may be attenuated more than channels that are relatively less sensitive to scaling. The sensitivity of the channel to scaling represents an amount that scaling affects the output of the channel.

Thus, the model trainer150trains the ASR model200to be robust to quantization with a per channel scale by adding the expected noise distribution during training. Because the model trainer150adds uniform noise, the weights202converge to a point where the average loss of the hyper-rectangle around it is minimized. The length of each side of the rectangle depends on the scale for the respective weight202. The exposure to noise causes the weights202to converge toward a flatter minima, while the norm (scale) decay gradients make the hyperrectangle smaller in a principled way. That is, when the network loss value is plotted against the values of the network weights, minimizing average loss over a hyper rectangle causes the weights to converge to a region with a flat loss surface. Because the scale parameter used to scale the noise is based on the worst case noise that the ASR model200is designed for (e.g., 4-bit quantization), the noise also covers the ranges of noise encountered for higher precision, allowing the ASR model200to operate effectively at both lower precision and higher precision.

The model trainer150may apply the techniques described herein to some or all of the weights202of the ASR model200, some or all of the activations204of the ASR model200, or any combination thereof. The model trainer150enables training of a single ASR model200that can be effectively quantized to a number of different precisions, which greatly improves efficiency of research and evaluation. While examples herein describe an ASR model200, the model trainer150may train any other type of model, such as a large language model (LLM), a vision model, etc.

Additionally, techniques herein may be applied to improve generalization of the ASR model200. That is, ∥Wi∥ is a general norm, and during quantization, the model trainer150uses an infinity norm. This may be changed to an L1 or an L2 norm to allow the ASR mode1200to learn to reduce the L1 or L2 norm of the matrix depending on how that set of weights are affected by perturbations. This is an improvement over conventional L1 or L2 regularization, where instead every weight is shifted down depending on magnitude only. Because the norm of the weight matrix is related to how much amplification any input or weight noise will encounter, the model trainer150may be used to improve generalization of extremely large neural network models. The model trainer150may decay the matrix norm depending on how sensitive the norm is to noise perturbations as opposed to robustness unaware technique such as L2 regularization.

Referring now toFIG.2, an example model200includes a Recurrent Neural Network-Transducer (RNN-T) model architecture which adheres to latency constrains associated with interactive applications. The use of the RNN-T model architecture is exemplary, and the model200may include other architectures such as transformer-transducer and conformer-transducer model architectures among others. The RNN-T model200provides a small computational footprint and utilizes less memory requirements than conventional ASR architectures, making the RNN-T model architecture suitable for performing speech recognition entirely on the user device10(e.g., no communication with a remote server is required). In this example, the RNN-T model200aincludes an encoder network210, a prediction network300, and a joint network230. The encoder network210, which is roughly analogous to an acoustic model (AM) in a traditional ASR system, includes a stack of self-attention layers (e.g., Conformer or Transformer layers) or a recurrent network of stacked Long Short-Term Memory (LSTM) layers. For instance, the encoder reads a sequence of d-dimensional feature vectors (e.g., acoustic frames) x=(x1, x2, . . . , xT), where x1∈Rd, and produces at each output step a higher-order feature representation. This higher-order feature representation is denoted as h1enc, . . . ,hTenc.

Similarly, the prediction network300may also be an LSTM network, which, like a language model (LM), processes the sequence of non-blank symbols output by a final Softmax layer240so far, y0, . . . , yui-1, into a dense representation put. Finally, with the RNN-T model architecture, the representations produced by the encoder and prediction/decoder networks210,300are combined by the joint network230. The prediction network300may be replaced by an embedding look-up table to improve latency by outputting looked-up sparse embeddings in lieu of processing dense representations. The joint network then predicts P(yi|xti, y0, . . . , yui-1), which is a distribution over the next output symbol. Stated differently, the joint network230generates, at each output step (e.g., time step), a probability distribution over possible speech recognition hypotheses. Here, the “possible speech recognition hypotheses” correspond to a set of output labels each representing a symbol/character in a specified natural language. For example, when the natural language is English, the set of output labels may include twenty-seven (27) symbols, e.g., one label for each of the 26-letters in the English alphabet and one label designating a space. Accordingly, the joint network230may output a set of values indicative of the likelihood of occurrence of each of a predetermined set of output labels. This set of values can be a vector and can indicate a probability distribution over the set of output labels. In some cases, the output labels are graphemes (e.g., individual characters, and potentially punctuation and other symbols), but the set of output labels is not so limited. For example, the set of output labels can include wordpieces and/or entire words, in addition to or instead of graphemes. The output distribution of the joint network230can include a posterior probability value for each of the different output labels. Thus, if there are 100 different output labels representing different graphemes or other symbols, the output yiof the joint network230can include 100 different probability values, one for each output label. The probability distribution can then be used to select and assign scores to candidate orthographic elements (e.g., graphemes, wordpieces, and/or words) in a beam search process (e.g., by the Softmax layer240) for determining the transcription120.

The Softmax layer240may employ any technique to select the output label/symbol with the highest probability in the distribution as the next output symbol predicted by the RNN-T model200at the corresponding output step. In this manner, the RNN-T model200does not make a conditional independence assumption, rather the prediction of each symbol is conditioned not only on the acoustics but also on the sequence of labels output so far. The RNN-T model200does assume an output symbol is independent of future acoustic frames110, which allows the RNN-T model to be employed in a streaming fashion.

In some examples, the encoder network (i.e., audio encoder)210of the RNN-T model200includes a stack of multi-head attention layers or self-attention layers/blocks, such as one or more conformer blocks/layers or one or more transformer blocks/layers. Optionally, the encoder210(i.e., the audio encoder) includes a first pass causal encoder and a second pass non-causal encoder for a multi-pass architecture. This multi-pass model unifies the streaming and non-streaming ASRs, where the causal encoder uses only left context and produces partial results with minimal latency, and the non-causal encoder can provide more accurate hypothesis by using both left and right context. In this example, each conformer block includes a series of multi-headed self attention, depth wise convolution, and feed-forward layers. The prediction network300may have two 2,048-dimensional LSTM layers, each of which is also followed by 640-dimensional projection layer. Alternatively, the prediction network300may include a stack of transformer or conformer blocks, or an embedding look-up table in lieu of LSTM layers. Finally, the joint network230may also have 640 hidden units. The Softmax layer240may be composed of a unified word piece or grapheme set that is generated using all unique word pieces or graphemes in a plurality of training data sets.

FIG.3illustrates an exemplary prediction network300of the RNN-T model200receiving, as input, a sequence of non-blank symbols yui-n, . . . , yui-1that is limited to the N previous non-blank symbols301a-noutput by the final Softmax layer240. In some examples, N is equal to two. In other examples, N is equal to five, however, the disclosure is non-limiting and N may equal any integer. The sequence of non-blank symbols301a-nindicates an initial speech recognition result120a(FIG.1). In some implementations, the prediction network300includes a multi-headed attention mechanism302that shares a shared embedding matrix304across each head302A-302H of the multi-headed attention mechanism. In one example, the multi-headed attention mechanism302includes four heads. However, any number of heads may be employed by the multi-headed attention mechanism302. Notably, the multi-headed attention mechanism improves performance significantly with minimal increase to model size. As described in greater detail below, each head302A-H includes its own row of position vectors308, and rather than incurring an increase in model size by concatenating outputs318A-H from all the heads, the outputs318A-H are instead averaged by a head average module322.

Referring to the first head302A of the multi-headed attention mechanism302, the head302A generates, using the shared embedding matrix304, a corresponding embedding306,306a-n(e.g., X∈N×de) for each non-blank symbol301among the sequence of non-blank symbols yui-n, . . . , yui-1received as input at the corresponding time step from the plurality of time steps. Notably, since the shared embedding matrix304is shared across all heads of the multi-headed attention mechanism302, the other heads302B-H all generate the same corresponding embeddings306for each non-blank symbol. The head302A also assigns a respective position vector PVAa-An308,308Aa-An (e.g., P∈H×N×de) to each corresponding non-blank symbol in the sequence of non-blank symbols yui-n, . . . , yui-1. The respective position vector PV308assigned to each non-blank symbol indicates a position in the history of the sequence of non-blank symbols (e.g., the N previous non-blank symbols output by the final Softmax layer240). For instance, the first position vector PVAais assigned to a most recent position in the history, while the last position vector PVAnis assigned to a last position in the history of the N previous non-blank symbols output by the final Softmax layer240. Notably, each of the embeddings306may include a same dimensionality (i.e., dimension size) as each of the position vectors PV308.

While the corresponding embedding generated by shared embedding matrix304for each for each non-blank symbol301among the sequence of non-blank symbols301a-n, yui-n, . . . , yui-1, is the same at all of the heads302A-H of the multi-headed attention mechanism302, each head302A-H defines a different set/row of position vectors308. For instance, the first head302A defines the row of position vectors PVAa-An308Aa-An, the second head302B defines a different row of position vectors PVBa-Bn308Ba-Bn, . . . , and the Hthhead302H defines another different row of position vectors PVHa-Hn308Ha-Hn.

For each non-blank symbol in the sequence of non-blank symbols301a-nreceived, the first head302A also weights, via a weight layer310, the corresponding embedding306proportional to a similarity between the corresponding embedding and the respective position vector PV308assigned thereto. In some examples, the similarity may include a cosine similarity (e.g., cosine distance). In the example shown, the weight layer310outputs a sequence of weighted embeddings312,312Aa-An each associated the corresponding embedding306weighted proportional to the respective position vector PV308assigned thereto. Stated differently, the weighted embeddings312output by the weight layer310for each embedding306may correspond to a dot product between the embedding306and the respective position vector PV308. The weighted embeddings312may be interpreted as attending over the embeddings in proportion to how similar they are to the positioned associated with their respective position vectors PV308. To increase computational speed, the prediction network300includes non-recurrent layers, and therefore, the sequence of weighted embeddings312Aa-An are not concatenated, but instead, averaged by a weighted average module316to generate, as output from the first head302A, a weighted average318A of the weighted embeddings312Aa-An represented by:

In the above equation, h represents the index of the heads302, n represents position in context, and e represents the embedding dimension. Additionally, H, N, and deinclude the sizes of the corresponding dimensions. The position vector PV308does not have to be trainable and may include random values. Notably, even though the weighted embeddings312are averaged, the position vectors PV308can potentially save position history information, alleviating the need to provide recurrent connections at each layer of the prediction network300.

The operations described above with respect to the first head302A are similarly performed by each other head302B-H of the multi-headed attention mechanism302. Due to the different set of positioned vectors PV308defined by each head302, the weight layer310outputs a sequence of weighted embeddings312Ba-Bn,312Ha-Hn at each other head302B-H that is different than the sequence of weighted embeddings312Aa-Aa at the first head302A. Thereafter, the weighted average module316generates, as output from each other corresponding head302B-H, a respective weighted average318B-H of the corresponding weighted embeddings312of the sequence of non-blank symbols.

In the example shown, the prediction network300includes a head average module322that averages the weighted averages318A-H output from the corresponding heads302A-H. A projection layer326with SWISH may receive, as input, an output324from the head average module322that corresponds to the average of the weighted averages318A-H, and generate, as output, a projected output328. A final layer normalization330may normalize the projected output328to provide the single embedding vector Pui350at the corresponding time step from the plurality of time steps. The prediction network300generates only a single embedding vector Pui350at each of the plurality of time steps subsequent to an initial time step.

In some configurations, the prediction network300does not implement the multi-headed attention mechanism302and only performs the operations described above with respect to the first head302A. In these configurations, the weighted average318A of the weighted embeddings312Aa-An is simply passed through the projection layer326and layer normalization330to provide the single embedding vector Pui350.

In some implementations, to further reduce the size of the RNN-T decoder, i.e., the prediction network300and the joint network230, parameter tying between the prediction network300and the joint network230is applied. Specifically, for a vocabulary size |V| and an embedding dimension de, the shared embedding matrix304at the prediction network is E∈|V|×de. Meanwhile, a last hidden layer includes a dimension size dhat the joint network230, feed-forward projection weights from the hidden layer to the output logits will be W∈dh×|V+1|, with an extra blank token in the vocabulary. Accordingly, the feed-forward layer corresponding to the last layer of the joint network230includes a weight matrix [dh, |V]|. By having the prediction network300to tie the size of the embedding dimension deto the dimensionality dhof the last hidden layer of the joint network230, the feed-forward projection weights of the joint network230and the shared embedding matrix304of the prediction network300can share their weights for all non-blank symbols via a simple transpose transformation. Since the two matrices share all their values, the RNN-T decoder only needs to store the values once on memory, instead of storing two individual matrices. By setting the size of the embedding dimension deequal to the size of the hidden layer dimension dh, the RNN-T decoder reduces a number of parameters equal to the product of the embedding dimension deand the vocabulary size |V|. This weight tying corresponds to a regularization technique.

FIG.4is a flowchart of an exemplary arrangement of operations for a computer-implemented method400of quantizing a model. The computer-implemented method400may execute on the data processing hardware62of the remote computing device60using instructions stored on the memory hardware64of the remote computing device60. The computer-implemented method400includes, at operation402, includes obtaining a plurality of training samples152. Each respective training sample152of the plurality of training samples152includes a respective speech utterance154and a respective textual utterance156representing a transcription of the respective speech utterance154. At operation404, the method400includes determining a minimum integer fixed-bit width160representing a maximum quantization of an automatic speech (ASR) recognition model200. The ASR model200includes a plurality of weights202. Each respective weight202of the plurality of weights202includes a respective float value. The method400, at operation406, includes training the ASR model200on the plurality of training samples152using a quantity of random noise162. The quantity of random noise162is based on the minimum integer fixed-bit width160. At operation408, the method400includes, after training the ASR model200, selecting a target integer fixed-bit width172that is greater than or equal to the minimum integer fixed-bit width610. At operation410, the method400includes, for each respective weight202of the plurality of weights202, quantizing the respective weight202from the respective float value to a respective integer associated with a value of the selected target integer fixed-bit width172. The method400, at operation412, includes providing the quantized trained ASR model200Q to a user device10.

The computing device500includes a processor510, memory520, a storage device530, a high-speed interface/controller540connecting to the memory520and high-speed expansion ports550, and a low speed interface/controller560connecting to a low speed bus570and a storage device530. Each of the components510,520,530,540,550, and560, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor510(i.e., data processing hardware62of the computing device60ofFIG.1A) can process instructions for execution within the computing device500, including instructions stored in the memory520(i.e., memory hardware64of the computing device60ofFIG.1) or on the storage device530to display graphical information for a graphical user interface (GUI) on an external input/output device, such as display580coupled to high speed interface540. In other implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices500may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The storage device530is capable of providing mass storage for the computing device500. In some implementations, the storage device530is a computer-readable medium. In various different implementations, the storage device530may be a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. In additional implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer- or machine-readable medium, such as the memory520, the storage device530, or memory on processor510.

The high speed controller540manages bandwidth-intensive operations for the computing device500, while the low speed controller560manages lower bandwidth-intensive operations. Such allocation of duties is exemplary only. In some implementations, the high-speed controller540is coupled to the memory520, the display580(e.g., through a graphics processor or accelerator), and to the high-speed expansion ports550, which may accept various expansion cards (not shown). In some implementations, the low-speed controller560is coupled to the storage device530and a low-speed expansion port590. The low-speed expansion port590, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet), may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device500may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server500aor multiple times in a group of such servers500a, as a laptop computer500b, or as part of a rack server system500c.