Patent Description:
There are many different types of ASR systems that provide different services. For example, some ASR systems are speech-to-speech systems configured to translate a speech between different languages in substantially real-time to remove the need for a human interpreter. As another example, some ASR systems are human-to-machine communication systems configured to perform various actions in response to voice commands, such as voice search systems, personal digital assistant systems, gaming systems, living room interaction systems, and in-vehicle infotainment systems.

In many cases, the computation cost of an ASR model increases when the input audio becomes longer. Such an ASR model may require the computing system to be sufficiently powerful to execute the model. Additionally, in many cases, a tradeoff between latency and accuracy is inevitable. Existing ASR systems may train a different model with a different latency for a different scenario, and deploy a particular model onto a particular type of device.

<NPL> describes that Transformer based end-to-end models have achieved great success in many areas including speech recognition. However, compared to LSTM models, the heavy computational cost of the Transformer during inference is a key issue to prevent their applications. The potential of Transformer Transducer (T-T) models for the first pass decoding with low latency and fast speed on a large-scale dataset is considered. The idea of Transformer-XL and chunk-wise streaming processing to design a streamable Transformer Transducer model is proposed. T-T models demonstrably outperform the hybrid model, RNN Transducer (RNN-T), and streamable Transformer attention-based encoder-decoder model in a streaming scenario. Furthermore, the runtime cost and latency can be optimized with a relatively small look-ahead.

<NPL> describes a Transformer-Transducer model architecture and a training technique to unify streaming and non-streaming speech recognition models into one model. The model is composed of a stack of transformer layers for audio encoding with no lookahead or right context and an additional stack of transformer layers on top trained with variable right context. In inference time, the context length for the variable context layers can be changed to trade off the latency and the accuracy of the model. It is shown that this model can be run in a Y-model architecture with the top layers running in parallel in low latency and high latency modes. This allows streaming speech recognition results with limited latency and delayed speech recognition results with large improvements in accuracy (<NUM>% relative improvement for voice-search task). With limited right context (<NUM>-<NUM> seconds of audio) and small additional latency (<NUM>-<NUM> milliseconds) at the end of decoding, similar accuracy can be achieved to models using unlimited audio right context.

<NPL> describes a dynamic encoder transducer (DET) for on-device speech recognition. One DET model scales to multiple devices with different computation capacities without retraining or finetuning. To trade off accuracy and latency, DET assigns different encoders to decode different parts of an utterance. The layer dropout and the collaborative learning for DET training are compared. The layer dropout method that randomly drops out encoder layers in the training phase, can do on-demand layer dropout in decoding. Collaborative learning jointly trains multiple encoders with different depths in one single model. Experiment results on Librispeech and in-house data show that DET provides a flexible accuracy and latency trade-off. Results on Librispeech show that the full-size encoder in DET relatively reduces the word error rate of the same size baseline by over <NUM>%. The lightweight encoder in DET trained with collaborative learning reduces the model size by <NUM>% but still gets similar WER as the full-size baseline. DET gets similar accuracy as a baseline model with better latency on a large in-house data set by assigning a lightweight encoder for the beginning part of one utterance and a full-size encoder for the rest. <NPL> describes options to use Transformer networks in a neural transducer for end-to-end speech recognition. Transformer networks use self-attention for sequence modeling which comes with advantages in parallel computation and capturing contexts. Two methods are proposed: <NUM>) using VGGNet with causal convolution to incorporate positional information and reduce frame rate for efficient inference <NUM>) using truncated self-attention to enable streaming for Transformer and reduce computational complexity. All experiments are conducted on the public LibriSpeech corpus. The proposed Transformer-Transducer outperforms neural transducer with LSTM/BLSTM networks and achieved word error rates of <NUM> % on the test-clean set and <NUM> % on the test-other set, while remaining streamable, compact with <NUM> parameters for the entire system, and computationally efficient with complexity of O(T), where T is input sequence length.

<CIT> describes a system, method and computer-readable storage device that provides an improved speech processing approach in which hyper parameters used for speech recognition are modified dynamically or in batch mode rather than fixed statically. The method includes estimating, via a model trained on audio data and/or metadata, a set of parameters useful for performing automatic speech recognition, receiving speech at an automatic speech recognition system, applying, by the automatic speech recognition system, the set of parameters to processing the speech to yield text and outputting the text from the automatic speech recognition system.

<CIT>describes techniques related to implementing neural networks for speech recognition systems. Such techniques may include implementing frame skipping with approximated skip frames and/or distances on demand such that only those outputs needed by a speech decoder are provided via the neural network or approximation techniques.

<NPL> describes that, while the community keeps promoting end-to-end models over conventional hybrid models, which usually are long short-term memory (LSTM) models trained with a cross entropy criterion followed by a sequence discriminative training criterion, such conventional hybrid models can still be significantly improved. In this paper, recent efforts to improve conventional hybrid LSTM acoustic models for high-accuracy and low-latency automatic speech recognition are detailed. To achieve high accuracy, a contextual layer trajectory LSTM (cltLSTM) is used, which decouples the temporal modeling and target classification tasks, and incorporates future context frames to get more information for accurate acoustic modeling. The training strategy is further improved with sequence-level teacher-student learning. To obtain low latency, a two-head cltLSTM is designed, in which one head has zero latency and the other head has a small latency, compared to an LSTM. When trained with Microsoft's <NUM> thousand hours of anonymized training data and evaluated with test sets with <NUM> million words, the proposed two-head cltLSTM model with the proposed training strategy yields a <NUM>% relative WER reduction over the conventional LSTM acoustic model, with a similar perceived latency.

<NPL> describes that the attention-based Transformer model has achieved promising results for speech recognition (SR) in offline mode. However, in streaming mode, the Transformer model usually incurs significant latency to maintain its recognition accuracy when applying a fixed-length look-ahead window in each encoder layer. In this paper, a novel low-latency streaming approach for Transformer models is described, which consists of a scout network and a recognition network. The scout network detects the whole word boundary without seeing any future frames, while the recognition network predicts the next subword by utilizing the information from all the frames before the predicted boundary. The model achieves the best performance (<NUM>/<NUM> WER) with only <NUM> latency on the test-clean and test-other data sets of Librispeech.

<NPL> describes transformer-based acoustic models (AMs) for hybrid speech recognition. Several modeling choices are discussed in this work, including various positional embedding methods and an iterated loss to enable training deep transformers. A preliminary study is further presented of using limited right context in transformer models, which makes it possible for streaming applications. It is demonstrated that on the widely used Librispeech benchmark, the transformer-based AM outperforms the best published hybrid result by <NUM>% to <NUM>% relative when the standard n-gram language model (LM) is used. Combined with neural network LM for rescoring, the proposed approach achieves state-of-the-art results on Librispeech. The findings are also confirmed on a much larger internal dataset.

This Summary is provided to introduce a selection of concepts in a simplified form that is further described below in the Detailed Description.

The principles described herein are related to (<NUM>) a computing system configured to use a transformer-transducer-based deep neural network to train an end-to-end (E2E) automatic speech recognition (ASR) model, (<NUM>) a device configured to execute the E2E ASR model, and/or (<NUM>) a method for dynamically adjusting one or more adjustable hyperparameters of an E2E ASR model based on the computational power of the device.

The computing system is configured to generate a transformer-transducer-based deep neural network. The transformer-transducer-based deep neural network comprises a transformer encoder network and a transducer predictor network. The transformer encoder network has a plurality of transformer layers. Each of the plurality of transformer layers includes a multi-head attention network sublayer and a feed-forward network sublayer. The computing system then trains an E2E ASR model, using the transformer-transducer-based deep neural network. The E2E ASR model is trained to have one or more adjustable hyperparameters that are configured to dynamically adjust an efficiency or a performance of the E2E ASR model when the E2E ASR model is deployed onto a particular device or executed by the particular device.

In some embodiments, the one or more adjustable hyperparameters includes at least one of (<NUM>) a number of layers that are to be implemented at the transformer encoder network, (<NUM>) a history window size indicating a number of history frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) a look-ahead window size indicating a number of look-ahead frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) a chunk size indicating a total number of frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) an attention mask indicating particular items in a frame index matrix that are to be set as "<NUM>", the frame index representing a particular configuration of the transformer encoder network, and/or (<NUM>) a transducer path that is to be executed by the transducer predictor network.

The E2E ASR model is trained in a particular manner based on the transformer-transducer-based deep neural network, such that when the E2E ASR model is deployed onto the device, the E2E ASR model is configured to identify one or more conditions of the device associated with the computational power of the device, and dynamically set at least one of the one or more adjustable hyperparameters of the E2E ASR model based on the identified one or more conditions of the device.

In order to describe the manner in which the above-recited and other advantages and features can be obtained, a more particular description of the subject matter briefly described above will be rendered by reference to specific embodiments, which are illustrated in the appended drawings. Understanding that these drawings depict only typical embodiments and are not, therefore, to be considered to be limiting in scope, embodiments will be described and explained with additional specificity and details through the use of the accompanying drawings in which:.

An end-to-end (E2E) automatic speech recognition (ASR) system is an ASR system configured to translate an input speech sequence into an output token sequence (such as sub-words or words) using a single neural network model without the requirement of a separate language model. Hence, the size of the E2E ASR system is small, and the inference thereof is fast. Further, it is much easier to deploy an E2E ASR system onto a portable device (compared to a traditional hybrid ASR system).

One type of E2E ASR neural network architecture is transformer-based. Unlike recurrent-based E2E ASR architectures, a transformer-based E2E ASR architecture encodes features in parallel and implements a self-attention mechanism that is capable of capturing long dependency between input features. As such, models built by a neural network having a transformer-based E2E ASR architecture generally perform better than models built by a neural network having a recurrent-based E2E ASR architecture.

The principles described herein further include a transducer in a transformer-based E2E ASR system. Such a transformer-transducer-based E2E ASR system shows even better results in streaming scenarios, and is also easy to extend with additional reranking strategies.

However, the computation cost of the transformer-transducer-based E2E ASR system linearly increases as the input audio becomes longer. Moreover, a tradeoff between latency and accuracy is often inevitable in a speech recognition system. One solution for this problem is to build a separate model for each type of device having particular computational power. For example, some of the models are built for allowing long input audios, and these models will only be deployed onto devices with sufficient computational power to achieve a real-time factor (RTF) of less than <NUM>. On the other hand, some of the models are built for only allowing short input audios, and these models will only be deployed onto devices with low computational power. The RTF is the ratio of the speech recognition response time to the utterance duration. Such a solution would use not only exhaustive human efforts but also valuable GPU devices.

The principles described herein solve the above-described problem by training an efficiency and latency adjustable transformer-transducer-based model that is capable of dealing with different applications, latencies, and efficiencies by dynamically changing the inference chunk size and/or encoder layers within a single model.

The computing system is configured to generate a transformer-transducer-based deep neural network. The transformer-transducer-based deep neural network comprises a transformer encoder network and a transducer predictor network. The transformer encoder network has a plurality of transformer layers. Each of the plurality of transformer layers includes a multi-head attention network sublayer and a feed-forward network sublayer. In some embodiments, the plurality of transformer layers are identical layers. In some embodiments, each of the multi-head attention network sublayer and the feed-forward network sublayer further includes an "add & norm" component configured to perform a residual connection and a layer normalization. In some embodiments, the transducer predictor network includes a plurality of long-short-term memory (LSTM) networks.

The computing system then trains an E2E ASR model, using the transformer-transducer-based deep neural network. The E2E ASR model is trained to have one or more adjustable hyperparameters that are configured to dynamically adjust an efficiency or a performance of the E2E ASR model when the E2E ASR model is deployed onto a particular device or executed by the particular device.

In some embodiments, the one or more adjustable hyperparameters include at least one of (<NUM>) a number of layers that are to be implemented at the transformer encoder network, (<NUM>) a history window size indicating a number of history frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) a look-ahead window size indicating a number of look-ahead frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) a chunk size indicating a total number of frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) an attention mask indicating particular items in a frame index matrix that are to be set as "<NUM>", the frame index representing a particular configuration of the transformer encoder network, and/or (<NUM>) a transducer path that is to be executed by the transducer predictor network.

The E2E ASR model is trained in a particular manner based on the transformer-transducer-based deep neural network, such that when the E2E ASR model is deployed onto the device, the E2E ASR model is configured to identify one or more conditions of the device associated with the computational power of the device, and set at least one of the one or more adjustable hyperparameters of the E2E ASR model based on the identified one or more conditions of the device.

In some embodiments, when setting the one or more adjustable hyperparameters includes enumerating a plurality of paths in the transducer predictor network. Next, a performance of each of the plurality of paths is determined, and a particular path among the plurality of paths that has a best performance is selected, and the particular path is set as the transducer path that is to be executed by the transducer predictor network.

In some embodiments, setting the one or more adjustable hyperparameters includes setting a maximum chunk size, a maximum history window size, or a maximum look-ahead window size. The attention mask is generated based on the maximum chunk size, the maximum history window size, and/or the maximum look-ahead window size.

According to the invention, the one or more conditions of the device include at least one of the following hardware conditions: (<NUM>) a type of processor that is installed on the device, (<NUM>) a number of processors that is installed on the device, (<NUM>) a type of memory that is installed on the device, and/or (<NUM>) a total amount of memory that is installed on the device. In some embodiments, the one or more conditions of the device include one or more runtime conditions, such as (but not limited to) (<NUM>) a function of a particular application that employs the E2E ASR model on the device, and/or (<NUM>) a current status of the device. In some embodiments, the function of the particular application that employs the E2E ASR model is (<NUM>) a streaming application configured to process a stream of speech in substantially real time, or (<NUM>) a post-processing application configured to process a file of a recorded speech. The current status of the device includes at least one of (<NUM>) a thermal status of the device, (<NUM>) a throttling status of the device, (<NUM>) other applications that are currently executing at the device, (<NUM>) a battery level of the device, and/or (<NUM>) a battery-saving status of the device.

Since the principles described herein are related to a transformer-transducer-based deep neural network, a short introduction to a transducer or a transformer is provided below. A transformer or a transducer is a particular scheme of deep neural network architecture. <FIG> illustrates an example of a transducer-based deep neural network architecture 100A. The transducer-based deep neural network architecture 100A includes an encoder network 110A and a predictor network 120A. The output of the encoder network110A and the output of the predictor network120A are then joined at a joint network 130A. Finally, the output of the joint network 130A is then sent to a softmax regression unit 140A to generate a prediction and/or a classification result.

As illustrated in <FIG>, each of the predictor network 120A and the encoder network 110A is a network that has one or more layers. In some embodiments, the encoder network 110A is a transformer-based encoder network. Such a transducer-based neural network having a transformer-based encoder network is also referred to as a "transformer-transducer-based deep neural network.

<FIG> illustrates an example of a transformer-transducer-based deep neural network architecture 100B for training an E2E ASR model. The transformer-transducer-based deep neural network architecture 100B includes a transformer encoder network 110B, a transducer predictor network 120B, and a feed-forward network (FFN) joint network 130B. The transformer encoder network 110B corresponds to the encoder network 110A of <FIG>, the predictor 120B corresponds to the predictor network 120A of <FIG>, and the FFN joint network 130B corresponds to the joint network 130A of <FIG>.

The transformer encoder network 110B includes a stack of multiple (N) transformer layers, each of which has two sublayers, namely (<NUM>) a multi-head self-attention network sublayer 116B and (<NUM>) a feed-forward network (FFN) sublayer 112B. Further, an "add & norm" component 114B, 118B is employed in both sublayers. The "add & norm" component 114B, 118B is configured to perform a residual connection followed by a layer normalization. Further, the multi-head self-attention network 116B is a type of recurrent neural network. The FFN 112B is configured to transform the representation of all the sequences using a same multi-layer perceptron (MLP). In the feed-forward network, queries, keys, and values are from the output of the previous encoder layer. As a result, the transformer encoder network 110B outputs a multi-dimensional vector representation for each position of the input sequence.

For example, given an input X, including a plurality of frames, the input X first goes through residual connection and layer normalization via the "add & norm" component 118B to generate another plurality of frames. The resulting plurality of frames (generated by the "add & norm" component 118B) is then sent to the multi-head self-attention network sublayer 116B. The multi-head self-attention network sublayer 116B is configured to perform a linear transformation to the result of the "add & norm" component 118B. Based on the result of the linear transformation, the sublayer 112B then computes an attention weight for each frame. Based on the attention weights, the sublayer 116Bthen generates a linear combination value for each frame, applying the attention weights to the values of the respective frames. The linear combination values of the plurality of frames then go through residual connection and layer normalization via the "add & norm" component 114B. The resulting plurality of frames (generated by the "add & norm" component 114B) is then sent to the FFN 112B, which also generates a result of a plurality of frames.

The resulting plurality of frames (generated by the FFN 112b) is then processed by a second transformer network 110B. For example, the resulting plurality of frames (generated by the FFN 112B) goes through the "add & norm" component 118B, the result of which is then processed by the multi-head self-attention network sublayer 116B, the result of which is then processed by the "Add & Norm" component 114B, and the result of which is then processed by the FFN 112B. This process repeats for N times(e.g., N = <NUM>) during the training process.

The transducer predictor network 120B also includes a stack of one or more layers, each of which is a Long-Short-Term Memory (LSTM) network. An LSTM network is a type of recurrent neural network configured to address long-term information preservation and short-term input skipping in latent variable models. In some embodiment, an LSTM network includes two dedicated gates. One gate is configured to govern how much new data is taken into account, and the other gate is configured to govern how much of the old memory cell content is retained.

As illustrated in <FIG>, in some embodiments, two LSTM network layers 122B and 124B are implemented. In the first LSTM network 122B, the queries, keys, and values are from the previous tokens. In the second LSTM network 124B, the queries, keys, and values are from the output of the first LSTM network 122B. The output of the transformer encoder network 110B and the output of the transducer predictor network 120B are then sent to the FFN joint network layer 130B. In the FFN joint network 130B, queries are from the outputs of the transducer predictor network 120B, and the keys and values are from the outputs of the transformer encoder network 110B.

Existing technologies only consider a number of acoustic frames T in determining a batch size. However, during the transformer-transducer training, the consumed memory relies on a length of acoustic frames T and sentence pieces U. In particular, the memory consumed on the transformer encoder network 110B is based on a batch size B and a number of acoustic frames T; the memory consumed on the predictor 110A is based on the batch size B and a number of sentence pieces U; and the memory consumed on the joint network 130B is based on a matrix multiplication in the output layer: (B, T, U, V).

In some embodiments, an improved batching is implemented. The improved batching is based on T*U or α*T + β*T*U, where α and β are estimated by solving pseudo inverse, given the consumed memory, and B, T, and U. Based on the experiments conducted by the inventors, batching based on T*U speeds up the training by about <NUM> times, and batching based on α*T + β*T*U speeds up the training by about <NUM> times, compared to the traditional batching based on T.

In some embodiments, half-precision floating-point format (FP16) with multiple nodes parallel training is performed. In some embodiments, an O2 optimization level is used. Based on the experiments conducted by the inventors, the FP16 training using an O2 optimization level improves both memory and training speed, while achieving a same convergence without or with little performance loss (compared to FP32 training). In particular, batching based on α*T + β*T*U and FP16 speeds up training by about <NUM> times (compared to batching based on T and FP32 training).

Once the ASR model is trained, the number of layers of the transformer encoder network 110B in the trained ASR model that is to be used to encode a received input is adjustable depending on the circumstances, such as the computational power of the device. In other words, the number of layers of the transformer encoder network 110B of the trained ASR model is an adjustable hyperparameter. In some embodiments, other than the number of layers in the transformer encoder network 110B of a trained ASR model, other hyperparameters are also adjustable based on the computational power of the device and/or the current physical conditions of the device. Such adjustable hyperparameters include (but are not limited to) (<NUM>) whether the transformer is with a history and/or a size of a history window, (<NUM>) whether the transformer is with a look ahead and/or a size of a look-ahead window, (<NUM>) a chunk size of the transformer encoder network, (<NUM>) a shape and size of an attention mask that is to be applied onto a frame index matrix corresponding to the transformer encoder network, and/or (<NUM>) a transducer path of the transducer predictor network.

<FIG> illustrate the concepts of history and look ahead in a transformer encoder network of an E2E ASR model. For clarity, in each of <FIG>, only the input and output connections associated with the <NUM>th frame in each layer are shown. Referring to <FIG>, a transformer encoder network configuration 200A is with a full history, because the <NUM>th frame x10' in the second layer receives input from each of the historical inputs x1 through x9, and the same pattern repeats in a next layer, and so on and so forth. Notably, in <FIG>, the transformer encoder network configuration 200A does not include a look ahead. Referring to <FIG>, a transform encoder network configuration 200B not only includes a full history, but also includes a look ahead with a window size of <NUM>, because frame x10' not only receives input from each of the historical frame or input x1 through x10, but also receives a following frame or input (i.e., a look ahead input) from x11, and this pattern also repeats in a next layer, and so on and so forth.

A total number of frames that are directly connected to a higher frame in the next level in the deep neural network is called a "chunk size. " For example, the chunk size of the transformer encoder network configuration 200A is <NUM>, and the chunk size of the transformer encoder 200B is <NUM>. As shown in <FIG>, when the chunk size, the history window size, and/or the look-ahead window size increases, the memory and runtime cost increase significantly.

The E2E ASR model described herein is built in such a way, the chunk size, the history window size, and/or the look-head window size of the transformer encoder network are adjustable depending on the computational power of the device, the current status of the device and/or the application. <FIG>, <FIG>, <FIG>, <FIG>, and <FIG> illustrate examples of configurations of the E2E ASR model having different chunk sizes, history window sizes, and/or look-ahead window sizes.

<FIG> further illustrate an example of a transformer encoder network configuration 300A and a corresponding frame index matrix 300B. The transformer encoder configuration 300A is with a full history, and a full look ahead, which means every frame of x1 through x14 is considered by each frame in a next layer, and similarly, every frame in the next layer is considered by its next layer, and so on and so forth. Such a full history and full look ahead configuration may also be presented in the frame index matrix 300B in <FIG>. As shown in <FIG>, the whole matrix is filled with binary ones ("<NUM>"), with no binary zeros ("<NUM>") at all. As such, the transformer encoder configuration 300A requires the whole utterance, and such a configuration 300A is more desirable for a post-processing application, but not a real-time-processing application, although the configuration 300A is likely to provide accurate performance. When a user inputs a previously recorded speech into a post-processing application, and the user's device has sufficient computational power, such a configuration is likely to be implemented. However, when an application requires processing streams of speech data in substantially real time or the device's computational power is insufficient, such a configuration would not be proper.

<FIG> illustrate another example of a transformer encoder configuration 400A and a corresponding frame index matrix 400B. The transformer encoder configuration 400A has a chunk size <NUM>, a maximum history window size <NUM>, and a look ahead window size <NUM>. Similar to the transformer encoder configuration 300A, the transformer encoder configuration 400A can also be represented by a frame index matrix 400B. As illustrated in <FIG>, along a diagonal line, a lower-left half of the matrix 400B is filled with <NUM>'s, and an upper-right half of the matrix 400B is filled with <NUM>'s. The portion of the matrix that is filled with "<NUM>" is like placing a triangle-shaped mask over the matrix 400B (having all the "<NUM>"). Such a mask is also referred to as an "attention mask. " In some embodiments, when configuring the transformer encoder, the system may simply configure an attention mask (having a particular shape and/or size) for the frame index matrix, which takes into account the chunk size, the history window size, and/or the look-ahead window size. Since this configuration has no look ahead, it can be implemented in a real-time processing application. However, since this configuration of <FIG> is with the full history, it would require fairly high computational power. Only when the device has sufficient computational power, such a configuration should be implemented.

<FIG> illustrate another example of a transformer encoder configuration 500A and a corresponding frame index matrix 500B. The transformer encoder configuration 500A has a chunk size <NUM>, a history window size <NUM>, and a look ahead window size <NUM>. Again, the transformer encoder configuration 500A can also be presented by a frame index matrix 500B having an attention mask 510B, 520B. As illustrated in <FIG>, here, the attention mask has two triangular pieces 510B and 520B. Further, compared to the configuration of <FIG>, the configuration of <FIG> requires much less computational power, because much fewer frames and nodes are interconnected with each other.

In some embodiments, the chunk size, the history window size, and/or the look-ahead window size may not be the same size for each frame as those in <FIG>. <FIG> illustrate another example of a transformer encoder configuration 600A and a corresponding frame index matrix 600B. The transformer encoder configuration 600A has a maximum chunk size <NUM>, a maximum history window size <NUM>, and a maximum look-ahead window size <NUM>. However, not every frame is with the maximum chunk size <NUM>, the maximum history window <NUM>, and/or the maximum look-ahead window size <NUM>. Again, the transformer encoder configuration 500A can also be represented by a frame index matrix 600B having an attention mask. As illustrated in <FIG>, here, the attention mask has multiple rectangular pieces 510B, 520B, 530B, 540B, 550B, 560B, and 570B.

In some embodiments, the different pieces of the masks are not necessarily interconnected with each other. <FIG> illustrate another example of a transformer encoder configuration 700A and a corresponding frame index matrix 700B. The transformer encoder configuration 700A has a maximum chunk size <NUM>, a maximum history window size <NUM>, and a maximum look-ahead window size <NUM>. As illustrated, the frame index matrix 700B and its attention mask here are even more complicated. The attention mask includes multiple pieces 710B, 712B, 714B, 716B, 718B, 720B, 722B, 724B, 728B, and 730B. Some of these pieces (e.g., 724B, 728B, 730B) of the attention mask are isolated pieces. In other words, they are not necessarily connected to the other piece of the attention mask.

<FIG>, <FIG>, <FIG>, <FIG>, and 7A-8B are merely a few examples of transformer encoder network configurations that may be implemented based on the computational power of a device, the current status of the device, and/or a particular application (e.g., a post-processing application, or a streaming application). There are a large number of configurations and/or attention masks that may be selected based on the device and the circumstances. In some embodiments, based on the circumstances, a maximum chunk size, a maximum history window size, and/or a maximum look-ahead window size are determined. Based on the determined maximum chunk size, history window size, and/or look-ahead window size, an attention mask is then determined. In some embodiments, a plurality of attention masks having the maximum chunk size, history window size, and/or look-ahead window size are enumerated. For each of the plurality of attention masks, a performance is estimated. Based on the estimated performance, a particular attention mask among the plurality of attention masks is then selected.

Further, as briefly discussed above, the transducer predictor network includes one or more LSTM networks, which has a latency depending on which path it takes to process the acoustic frames T. Various metrics may be used to evaluate a latency of a transducer predictor network. Commonly used metrics include (<NUM>) a partial recognition (PR) latency, and (<NUM>) an end-pointer (EP) latency. Referring to <FIG>, the PR latency is an offset between a time when the ground truth last word was received and a time when the last word is decoded. The EP latency is an offset between ground truth last word and an <eos> token at the end of a sequence. In some embodiments, the PR latency and/or the EP latency are measured or estimated. Based on the measured or estimated PR latency and/or the EP latency, the transformer encoder is configured in a particular manner or reconfigured to increase or decrease the PR latency and/or the EP latency.

In some embodiments, multiple paths in the transducer predictor network 120B are enumerated. For each of the multiple paths, a performance (e.g., an PR, and/or an EP) is estimated. A particular path among the multiple paths that have a best performance is selected to be applied by the transducer predictor network 120B. <FIG> and <FIG> illustrate two examples of enumerated paths 900A and 900B. As illustrated, path 900B may be faster than path 900A. Thus, path 900B will be selected to be applied by the transducer predictor network.

<FIG> illustrates a flowchart of an example method <NUM> for dynamically adjusting an ARS system based on one or more conditions of a device. In some embodiments, the method <NUM> includes receiving an E2E ASR model trained using a transformer-transducer-based deep neural network (act <NUM>). The method <NUM> also includes determining one or more conditions of the device associated with computational power of the device (act <NUM>). According to the invention, the one or more conditions include at least one of the following hardware conditions: (<NUM>) a type of processor installed on the device (<NUM>), (<NUM>) a number of processors installed on the device (<NUM>), (<NUM>) a type of memory installed on the device (<NUM>), and/or (<NUM>) a total amount of memory installed on the device (<NUM>). In some embodiments, one or more conditions include one or more runtime conditions, such as (<NUM>) a type of application that employs the E2E ASR model (<NUM>) and/or (<NUM>) a current status of the device (<NUM>).

For example, the type of application that employs the E2E ASR model may be (but are not limited to) a post-processing application or a real-time streaming application. As another example, the current status of the device may be (but are not limited to) (<NUM>) a thermal status of the device, (<NUM>) a throttling status of the device, (<NUM>) other applications that are currently executing at the device, (<NUM>) a battery level of the device, or (<NUM>) a battery-saving status of the device.

The method <NUM> further includes setting at least one of one or more adjustable hyperparameters based on the determination (act <NUM>). The one or more hyperparameters may include (but are not limited to): (<NUM>) a number of layers that are to be implemented at the transformer encoder network, (<NUM>) a history window size indicating a number of history frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) a look-ahead window size indicating a number of look-ahead frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) a chunk size indicating a total number of frames in a previous layer of the transformer encoder network that is to be considered by a frame of a current layer, (<NUM>) an attention mask indicating particular items in a frame index matrix that are to be set as "<NUM>", the frame index representing a particular configuration of the transformer encoder network, and/or (<NUM>) a transducer path that is to be executed by the transducer predictor network.

<FIG> illustrates a flowchart of an example of method <NUM> for selecting a path for a transducer predictor network 120B of an E2E ASR model. The method <NUM> includes enumerating a plurality of paths in the transducer predictor network (act <NUM>). The method <NUM> also includes determining performance of each of the plurality of paths (act <NUM>). The method <NUM> also includes selecting a particular path among the plurality of paths that has a best performance (act <NUM>) and applying the particular path in the transducer predictor network (act <NUM>).

Finally, because the principles described herein may be performed in the context of a computing system (for example, the training of the E2E ASR model is performed by one or more computing systems, and the E2E ASR model is deployed onto one or more computing systems) some introductory discussion of a computing system will be described with respect to <FIG>.

In this description and in the claims, the term "computing system" is defined broadly as including any device or system (or a combination thereof) that includes at least one physical and tangible processor and a physical and tangible memory capable of having thereon computer-executable instructions that may be executed by a processor.

As illustrated in <FIG>, in its most basic configuration, a computing system <NUM> typically includes at least one hardware processing unit <NUM> and memory <NUM>. The processing unit <NUM> may include a general-purpose processor and may also include a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit. The memory <NUM> may be physical system memory, which may be volatile, non-volatile, or some combination of the two. The term "memory" may also be used herein to refer to non-volatile mass storage such as physical storage media. If the computing system is distributed, the processing, memory, and/or storage capability may be distributed as well.

The computing system <NUM> also has thereon multiple structures often referred to as an "executable component". For instance, memory <NUM> of the computing system <NUM> is illustrated as including executable component <NUM>. The term "executable component" is the name for a structure that is well understood to one of ordinary skill in the art in the field of computing as being a structure that can be software, hardware, or a combination thereof. For instance, when implemented in software, one of ordinary skill in the art would understand that the structure of an executable component may include software objects, routines, methods, and so forth, that may be executed on the computing system, whether such an executable component exists in the heap of a computing system, or whether the executable component exists on computer-readable storage media.

In such a case, one of ordinary skill in the art will recognize that the structure of the executable component exists on a computer-readable medium such that, when interpreted by one or more processors of a computing system (e.g., by a processor thread), the computing system is caused to perform a function. Such a structure may be computer-readable directly by the processors (as is the case if the executable component were binary). Alternatively, the structure may be structured to be interpretable and/or compiled (whether in a single stage or in multiple stages) so as to generate such binary that is directly interpretable by the processors. Such an understanding of example structures of an executable component is well within the understanding of one of ordinary skill in the art of computing when using the term "executable component".

The term "executable component" is also well understood by one of ordinary skill as including structures, such as hardcoded or hardwired logic gates, that are implemented exclusively or near-exclusively in hardware, such as within a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or any other specialized circuit.

In the description above, embodiments are described with reference to acts that are performed by one or more computing systems. For example, such computer-executable instructions may be embodied in one or more computer-readable media that form a computer program product. If such acts are implemented exclusively or near-exclusively in hardware, such as within an FPGA or an ASIC, the computer-executable instructions may be hardcoded or hardwired logic gates.

While not all computing systems require a user interface, in some embodiments, the computing system <NUM> includes a user interface system <NUM> for use in interfacing with a user. The user interface system <NUM> may include output mechanisms 1212A as well as input mechanisms 1212B. The principles described herein are not limited to the precise output mechanisms 1212A or input mechanisms 1212B as such will depend on the nature of the device. However, output mechanisms 1212A might include, for instance, speakers, displays, tactile output, holograms, and so forth. Examples of input mechanisms 1212B might include, for instance, microphones, touchscreens, holograms, cameras, keyboards, mouse or other pointer input, sensors of any type, and so forth.

Embodiments described herein may comprise or utilize a special purpose or general-purpose computing system including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below. Such computer-readable media can be any available media that can be accessed by a general-purpose or special purpose computing system.

Transmissions media can include a network and/or data links that can be used to carry desired program code means in the form of computer-executable instructions or data structures and which can be accessed by a general-purpose or special-purpose computing system.

Computer-executable instructions comprise, for example, instructions and data which, when executed at a processor, cause a general-purpose computing system, special purpose computing system, or special purpose processing device to perform a certain function or group of functions. Alternatively or in addition, the computer-executable instructions may configure the computing system to perform a certain function or group of functions. The computer-executable instructions may be, for example, binaries or even instructions that undergo some translation (such as compilation) before direct execution by the processors, such as intermediate format instructions such as assembly language, or even source code.

Those skilled in the art will appreciate that the invention may be practiced in network computing environments with many types of computing system configurations, including personal computers, desktop computers, laptop computers, message processors, handheld devices, multi-processor systems, microprocessor-based or programmable consumer electronics, network PCs, minicomputers, mainframe computers, mobile telephones, PDAs, pagers, routers, switches, data centers, wearables (such as glasses) and the like. The invention may also be practiced in distributed system environments where local and remote computing systems, which are linked (either by hardwired data links, wireless data links, or by a combination of hardwired and wireless data links) through a network, both perform tasks.

The remaining figures may discuss various computing systems which may correspond to the computing system <NUM> previously described. The computing systems of the remaining figures include various components or functional blocks that may implement the various embodiments disclosed herein. The various components or functional blocks may be implemented on a local computing system or may be implemented on a distributed computing system that includes elements resident in the cloud or that implement aspect of cloud computing. The various components or functional blocks may be implemented as software, hardware, or a combination of software and hardware. The computing systems of the remaining figures may include more or less than the components illustrated in the figures, and some of the components may be combined as circumstances warrant. Although not necessarily illustrated, the various components of the computing systems may access and/or utilize a processor and memory, such as processor <NUM> and memory <NUM>, as needed to perform their various functions.

For the processes and methods disclosed herein, the operations performed in the processes and methods may be implemented in differing order. Furthermore, the outlined operations are only provided as examples, and some of the operations may be optional, combined into fewer steps and operations, supplemented with further operations, or expanded into additional operations without detracting from the essence of the disclosed embodiments.

Claim 1:
A computing system (<NUM>) comprising:
one or more processors (<NUM>); and
one or more computer-readable media (<NUM>) having stored thereon computer-executable instructions (<NUM>) that are structured such that, when executed by the one or more processors, cause the computing system to:
generate a transformer-transducer-based deep neural network (100B), the transformer-transducer-based deep neural network (100B) comprising a transformer encoder network (110B) and a transducer predictor network (120B), and the transformer encoder network having a plurality of layers, each of the plurality of layers including a multi-head attention network sublayer (116B) and a feed-forward network sublayer (112B);
train an end-to-end, E2E, automatic speech recognition, ASR, model, using the transformer-transducer-based deep neural network (100B), the E2E ASR model being trained to have one or more adjustable hyperparameters that are configured to dynamically adjust an efficiency or a performance of the E2E ASR model when the E2E ASR model is deployed onto a device or executed by the device, the E2E ASR model configured to perform at least the following at the device:
identify one or more conditions of the device associated with computational power of the device, wherein the one or more conditions of the device comprises at least one of following hardware conditions of the device (<NUM>) a type of processor that is installed on the device, (<NUM>) a number of processors that is installed on the device, (<NUM>) a type of memory that is installed on the device, or (<NUM>) a total amount of memory installed on the device, and
set at least one of the one or more adjustable hyperparameters based on the one or more conditions of the device; and
provide the E2E ASR model to the device to be used to perform ASR in response to receiving a stream of speech.