Patent Description:
<CIT> describes a method which may include obtaining first audio data of a communication session between a first device and a second device, obtaining a text string that is a transcription of the first audio data, and selecting a contiguous sequence of words from the text string as a first word sequence. The method may further include comparing the first word sequence to multiple word sequences obtained before the communication session and in response to the first word sequence corresponding to one of the multiple word sequences, incrementing a counter of multiple counters associated with the one of the multiple word sequences. The method may also include deleting the text string and the first word sequence and training and after deleting the text string and the first word sequence, training a language model of an automatic transcription system using the multiple word sequences and the multiple counters.

<NPL>" describes the acoustic-to-word model based on the Connectionist Temporal Classification (CTC) criterion is a natural end-to-end (E2E) system directly targeting word as output unit. Two issues exist in the system: first the current output of the CTC model relies on the current input and does not account for context weighted inputs. This is the hard alignment issue. Second, the word-based CTC model suffers from the out-of-vocabulary (OOV) issue. This means it can model only frequently occurring words while tagging the remaining words as OOV. Hence, such a model is limited in its capacity in recognizing only a fixed set of frequent words. In this study, these problems are addressed using a combination of attention mechanism and mixed-units. In particular introduced are Attention CTC, Self-Attention CTC, Hybrid CTC, and Mixed-unit CTC. First, blended are attention modeling capabilities directly into the CTC network using Attention CTC and Self-Attention CTC. Second, to alleviate the OOV issue, presented is Hybrid CTC which uses a word and letter CTC with shared hidden layers. The Hybrid CTC consults the letter CTC when the word CTC emits an OOV. Then, proposed is a much better solution by training a Mixed-unit CTC which decomposes all the OOV words into sequences of frequent words and multi-letter units. Evaluated on a <NUM> hours Microsoft Cortana voice assistant task, a final acoustic-to-word solution using attention and mixed-units achieves a relative reduction in word error rate (WER) over the vanilla word CTC by <NUM>%. Such an E2Emodel without using any language model (LM) or complex decoder also outperforms a traditional context-dependent (CD) phoneme CTC with strong LM and decoder by <NUM>% relative.

<NPL>, describes a Fast & Slow (F&S) acoustic model (AM) in an encoder-decoder architecture for streaming automatic speech recognition (ASR). The Slow model represents a baseline ASR model; it's significantly larger than Fast model and provides stronger accuracy. The Fast model is generally developed for related speech applications. It has weaker ASR accuracy but is faster to evaluate and consequently leads to better user-perceived latency. Proposed is a joint F&S model that encodes output state information from Fast model, feeds that to Slow model to improve overall model accuracy from F&S AM. Demonstrated are scenarios where individual Fast and Slow models are already available to build the joint F&S model. The work is applied on a large vocabulary ASR task. Compared to Slow AM, our Fast AM is <NUM>-4x smaller and <NUM>% relatively weaker in ASR accuracy. The proposed F&S AM achieves <NUM>% relative gain over the Slow AM. A progression of techniques and improve the relative gain to <NUM>% by encoding additional Fast AM outputs is reported. The proposed framework has generic attributes - demonstrated is a specific extension by encoding two Slow models to achieve <NUM>% relative gain.

<CIT> describes a method for automated speech recognition which includes generating first and second pluralities of candidate speech recognition results corresponding to audio input data using a first general-purpose speech recognition engine and a second domain-specific speech recognition engine, respectively. The method further includes generating a third plurality of candidate speech recognition result including a plurality of words included in one of the first plurality of speech recognition results and at least one word included in another one of the second plurality of speech recognition results, ranking the third plurality of candidate speech recognition results using a pairwise ranker to identify a highest ranked candidate speech recognition result, and operating the automated system using the highest ranked speech recognition result as an input from the user.

Disclosed speech recognition techniques improve user-perceived latency while maintaining accuracy at least by performing the following operations. An audio stream is received, in parallel, by a primary (e.g., accurate) speech recognition engine (SRE) and a secondary (e.g., fast) SRE. A primary result is generated with the primary SRE. A secondary result is generated with the secondary SRE. The secondary result is appended to a word list. The primary result is merged into the secondary result in the word list. Merging includes, for example, synchronizing the primary result with the secondary result; determining, within the primary result or the secondary result, whether at least some words belong to a class model; determining a word in the primary result that corresponds with a corresponding word in the secondary result; and if the corresponding word in the secondary result does not belong to a class model, replacing the corresponding word in the secondary result with the word in the primary result.

The various examples will be described in detail with reference to the accompanying drawings. References made throughout this disclosure relating to specific examples and implementations are provided solely for illustrative purposes but, unless indicated to the contrary, are not meant to limit all examples.

Aspects of the disclosure advantageously improve the speed and accuracy of speech recognition, thus improving user-perceived latency in speech recognition while maintaining high accuracy by merging a primary (e.g., high accuracy) speech recognition engine (SRE) result into a secondary (e.g., low latency) SRE result in a word list. This high accuracy, low-latency combination provides enhanced performance over using only a high accuracy, long latency SRE or a short latency, low accuracy SRE alone. This approach eliminates the traditional trade-off between sacrificing accuracy for rapid transcription results or suffering from sluggish response in order to achieve high accuracy. Aspects of the disclosure also operate in an unconventional manner at least by replacing a corresponding word in the secondary result with a word in the primary result, based on at least determining that the corresponding word in the secondary result does not belong to a class model. As explained below, this special treatment of words found to belong to class models may further improve accuracy. The use of multiple SREs as described herein thus reduces user-perceived latency. Further, in a hybrid model example, processing speed on at least one device is increased, and model size on the device is reduced, thereby using less storage.

Multiple SREs may be operated in parallel, with the audio signal input into the SREs in a synchronized manner. A primary SRE uses a high accuracy model that may have high latency, while a secondary SRE uses a lower latency model that may have lower accuracy. During speech recognition processing, a user sees an incrementally increasing intermediate result while speaking, and then a final result when a spoken phrase is complete. The intermediate result from the secondary SRE is generated with low latency to reduce user-perceived latency, while the final result is generated from the primary SRE with higher accuracy. The final result persists for the user and, in some examples, is used for voice commands, dictation, storage, transcription, further processing, or other uses.

A merging process permits the seamless use of results from both the primary SRE and the secondary SRE, which may typically be ahead of the primary SRE in reporting results. The primary SRE results are stitched into a word list, replacing earlier-emitted words as necessary to improve accuracy, while the word list is able to grow rapidly, with low latency. Stitching is paused for words that belong to class models (e.g., words that may be checked against an external reference, such as contact list names and other common words, phrases, or names). Also, in some examples, stitching is paused across grammar models (statistical models for sequences of words). In some examples of operation, the secondary SRE emits a partial result. The partial result from the primary SRE is stored but not emitted, and the merge algorithm copies the stored primary partial result in its entirety. The merge algorithm finds the word boundary of the final word in that partial result, and appends the words from the secondary result for words whose beginning word boundary are greater or equal to the word boundary found previously. The resulting partial result is displayed to the user. Partial results are those shown in real-time, whereas final results appear when speech ceases. In some examples, merging is performed on partial results, but not final results.

Tuning for the primary and secondary SREs may be accomplished independently, providing flexibility in the trade-off between accuracy and latency. The secondary (e.g., generally faster) SRE may be application-specific, for example, with processing that favors outputting certain vocabularies (e.g., medical, financial, legal, gaming, and other specialized terminology). The secondary SRE uses a fast acoustic model (AM) and encodes sequences that include future output states, thereby generating probability distributions over senones. A senone is a cluster of shared hidden Markov model states, each of which shares an output probability distribution. For example, a single hidden Markov model (HMM) models the relationship between input audio frames and a triphone (a phoneme in context of other phonemes). An HMM has multiple states, a transition probability matrix, a start state, and an end state. Different states across the set of Markov models may share output probability distributions. Sharing an output probability distribution increases the amount of training data assigned to it, and therefore improves the robustness of its parameter estimates. Such clusters are referred to as senones. The output of a neural network (NN), for example a deep NN (DNN) in a DNN-HMM hybrid decoder includes a probability distribution for supported senones (a triplet of phones happening in a sequence). For a given input audio frame, the neural network outputs the posterior probability of the senones in a set of senones for a chosen acoustic model. The encoded sequences are sent to a decoder that decodes the sequences to output words.

The primary SRE uses an AM that may be slower, and also encodes sequences to generating probability distributions over senones. In some examples, encoded sequences from the fast AM (including the future output states) are combined with the encoded sequences from the primary SRE's AM. The jointly encoded sequences are then provided to the primary SRE decoder to output words with a higher accuracy. In some examples, the primary SRE is on a remote node across a network to take advantage of higher power processing and larger storage (e.g., in a cloud services provider network), while the secondary SRE remains local.

<FIG> illustrates an arrangement <NUM> for speech recognition that advantageously improves user-perceived latency while maintaining accuracy. An audio stream <NUM> is captured by a microphone <NUM> from a user <NUM>. The current time in audio stream <NUM> is indicated by time marker <NUM>. Arrangement <NUM> generates a word list <NUM> that may be a transcription of a live conversation (e.g., captioning for a video conference or video streaming, or for voice commands). The latest results (words or letters) in word list <NUM> are displayed at a time indicated by a time marker <NUM>, which lags behind time marker <NUM>. This results in a user-perceived latency <NUM>. More accurate results are stitched into word list <NUM> at a time indicated by a time marker <NUM>, which lags behind time marker <NUM> by a further time delay <NUM>. If not for the advantageous teachings herein, the user perceived latency would be a potential latency <NUM>, which is the combination of user-perceived latency <NUM> plus further time delay <NUM>. Because user-perceived latency <NUM> lag is shorter than potential latency <NUM>, the disclosure enables improving user-perceived latency while maintaining accuracy.

Audio stream <NUM> is input into an automatic speech recognition (ASR) feature extraction (FE) stage <NUM> sent in parallel to a primary SRE <NUM> and a secondary SRE <NUM>. Primary SRE <NUM> emphasizes high accuracy at the expense of potentially higher latency (slower performance), whereas secondary SRE <NUM> emphasizes low latency at the expense of potentially lower accuracy. Primary SRE <NUM> has a look-ahead buffer <NUM>, and secondary SRE <NUM> similarly has a look-ahead buffer <NUM>, although in some examples, look-ahead buffer <NUM> is longer to provide higher accuracy, whereas look-ahead buffer <NUM> is shorter to provide higher speed.

Primary SRE <NUM> and secondary SRE <NUM> may include machine learning (ML) models, for example NNs. In some examples, primary SRE <NUM> and/or secondary SRE <NUM> use attention-based models that use an encoder network to map input acoustics into a higher-level representation, and an attention-based decoder that predicts the next output symbol (e.g., word, phrase, or letter) conditioned on a sequence of previous predictions. As illustrated, primary SRE <NUM> includes a primary decoder <NUM> and a primary encoder <NUM> as part of a primary AM <NUM>, and secondary SRE <NUM> includes a secondary decoder <NUM> and a secondary encoder <NUM> as part of a secondary AM <NUM>. Further detail for primary SRE <NUM> and secondary SRE <NUM>, and their respective components are provided in <FIG> and <FIG>.

Primary SRE <NUM> generates a primary result <NUM> which indicates a word <NUM>, and in some examples indicates multiple words <NUM>, for example phrases. In some examples, primary result <NUM> further includes a start sync marker <NUM>, a stop sync marker <NUM>, a class tag <NUM>, and a grammar tag <NUM>. Similarly, secondary SRE <NUM> generates a secondary result <NUM> which indicates a word <NUM>, and in some examples indicates multiple words <NUM>, for example phrases, and/or individual letters. In some examples, secondary result <NUM> further includes a start sync marker <NUM>, a stop sync marker <NUM>, a class tag <NUM>, and a grammar tag <NUM>. Start sync marker <NUM>, start sync marker <NUM>, stop sync marker <NUM>, and stop sync marker <NUM> may be based on a timer <NUM> that may also ensure that primary SRE <NUM> and secondary SRE <NUM> are synchronized (or separate timers are used, in some examples). In some examples, the sync markers comprise timestamps.

Class tag <NUM> and class tag <NUM> may be class start tags, class stop tags, or another type of tag indicating a class. Class tags are used, in some examples, to pause stitching, so that recognized words identified as belonging to a class model <NUM> are not stitched (e.g., are not changed or replaced). Class models <NUM> include words that may have to be checked against a reference of words that the user may be likely to use, such as names in a contact list <NUM>, application names <NUM>, and other words <NUM>. Other words may be times, dates, locations (place names) and common names, such as names of celebrities or sports teams. Class models <NUM> may be grouped as sets of similarly-themed words. Grammar tag <NUM> and grammar tag <NUM> are used to prevent stitching across grammar models, for example stitching across different phrases. Further detail on pausing stitching within a class model or across grammar models is provided in the description of <FIG>.

Primary result <NUM> and secondary result <NUM> are provided to merging logic <NUM> which merges the results to produce word list <NUM>. Merging logic <NUM> contains stitching logic <NUM>, which stitches primary result <NUM> into secondary result <NUM>. In some examples, stitching occurs at the word level, rather than as partial words. In some examples, stitching also occurs at the phrase level (multiple words), and stitching across grammar models is undesirable. Further detail on merging logic <NUM> and stitching logic <NUM> is provided in relation to <FIG>. As indicated in <FIG>, merging logic <NUM> has already appended secondary result <NUM> to word list <NUM>, and is stitching word <NUM> into word list <NUM> to replace word <NUM>. This occurs because words <NUM> and <NUM> correspond, according to start sync marker <NUM>, start sync marker <NUM>, stop sync marker <NUM>, and stop sync marker <NUM>. That is, word <NUM> and word <NUM> occur at the same time.

In some examples, secondary SRE <NUM> is sufficiently ahead of primary SRE <NUM> that by the time word <NUM> from primary result <NUM> is stitched into secondary result <NUM> (e.g., at time marker <NUM>), secondary SRE <NUM> has already generated a later secondary result 150a (with at least a word 152a), and merging logic <NUM> had already appended later secondary result 150a to word list <NUM> (e.g., at time marker <NUM>). In some examples, primary result <NUM> and secondary result <NUM> represent different time segments within audio stream <NUM> and therefore different portions of word list <NUM>. In some examples, primary result <NUM> and secondary result <NUM> nearly coincide in time. In some examples, secondary result <NUM> is ahead of primary result <NUM> by three or four words.

The tags described herein may take any form, such as markup language tags (e.g., extensible markup language tags).

<FIG> illustrates further details of word list <NUM>. As indicated, secondary result <NUM> is reflected within word list <NUM>, although, in some examples, not all of the elements indicated as being within secondary result <NUM> are actually present within word list <NUM>. In some examples, some of the tags are retained by merging logic <NUM> to identify whether to stitch in portions of primary result <NUM>, and word list <NUM> contains words and a limited set of metadata (e.g., timestamps). That is, in some examples, tags are removed prior to displaying word list <NUM>.

As an example, audio stream <NUM> contains "Hey, Assistant. What's the weather?" This has some notable features. "Hey Assistant" and "What's the weather?" are different phrases. These correspond to different grammar models, and so stitching may occur within "Hey Assistant" or within "What's the weather?", but not across "Assistant. " Additionally, Assistant is the name of an application that responds to voice commands, and thus belongs to a class model <NUM>, specifically application names <NUM>. For this example, both secondary SRE <NUM> and primary SRE <NUM> correctly identified two grammar models and both correctly identified "Assistant" as belonging to a grammar model. However, secondary SRE <NUM> improperly recognized "What's" as "Where's.

In secondary result <NUM>, word W1 is "Hey," word W2 is "Assistant," word W3 is "Where's," word W4 is "the," and word W5 is "weather. " In primary result <NUM>, word W1' is "Hey," word W2' is "Assistant," word W3' is "What's," word W4 is "the," and word W5 is "weather. " In this example, word W3' represents word <NUM> of <FIG>, and word W3 represents corresponding word <NUM>. Word W3 (word <NUM>) "Where's" will be replaced by word W3' (word <NUM>) "What's.

Secondary result <NUM> has a grammar start tag GS1 that starts the grammar model "Hey, Assistant" and a grammar stop tag GT1 that stops this grammar model. Secondary result <NUM> also has a grammar start tag GS2 that starts the grammar model "Where's the weather?" and a grammar stop tag GT2 that stops this grammar model. Grammar tags GT1 and GS2 will cause a pause in the stitching, so that stitching is limited to occurring between grammar tags GS1 and GT1 and also between grammar tags GS2 and GT2. Grammar tags GS1, GT1, GS2, and GT2 are examples of grammar tag <NUM> of <FIG>. Primary result <NUM> has a grammar start tag GS1' that starts the grammar model "Hey, Assistant" and a grammar stop tag GT1' that stops this grammar model. Primary result <NUM> also has a grammar start tag GS2' that starts the grammar model "What's the weather?" and a grammar stop tag GT2' that stops this grammar model. Grammar tags GT1' and GS2' will cause a pause in the stitching, so that stitching is limited to occurring between grammar tags GS1' and GT1' and also between grammar tags GS2' and GT2'. Grammar tags GS1', GT1', GS2', and GT2' are examples of grammar tag <NUM> of <FIG>. In some examples, stitching is paused by grammar tags in either primary result <NUM> or secondary result <NUM>.

Secondary result <NUM> has a start sync marker SS1 and a stop sync marker ST1 for word W1, a start sync marker SS2 and a stop sync marker ST2 for word W2, a start sync marker SS3 and a stop sync marker ST3 for word W3, a start sync marker SS4 and a stop sync marker ST4 for word W4, and a start sync marker SS5 and a stop sync marker ST5 for word W5. Similarly, primary result <NUM> has a start sync marker SS1' and a stop sync marker ST1' for word W1', a start sync marker SS2' and a stop sync marker ST2' for word W2', a start sync marker SS3' and a stop sync marker ST3' for word W3', a start sync marker SS4' and a stop sync marker ST4' for word W4', and a start sync marker SS5' and a stop sync marker ST5' for word W5'. Start and stop sync markers mark the beginnings and ends of words, so that merging logic <NUM> is able to ascertain which words in primary result <NUM> and secondary result <NUM> correspond (e.g., occur in the same time windows). Start sync markers SS1-SS5 are examples of start sync marker <NUM>, stop sync markers ST1-STS are examples of stop sync marker <NUM>, start sync markers SS1'-SS5' are examples of start sync marker <NUM>, and stop sync markers ST1'-ST5' are examples of stop sync marker <NUM>.

In some examples, the sync markers vary slightly, so merging logic <NUM> will allow for some timing precision tolerance. In some scenarios, secondary SRE <NUM> and primary SRE <NUM> may interpret audio stream <NUM> differently, such that one recognizes two short words, whereas the other recognizes a single longer word. Deferring to the higher accuracy of primary SRE <NUM>, with the limitation of not stitching within class models, single-versus-multiple words may be resolved using the earliest and latest sync markers, and duplicate words may also be detected and removed. In some scenarios, the (generally) more accurate primary SRE <NUM> recognizes an additional word. For example, secondary SRE <NUM> recognizes "John" and primary SRE <NUM> recognizes "John Smith. " If both of these recognized words are outside of a class model (e.g., neither word belongs to a class model), then "Smith" will be stitched into word list <NUM> after "John.

Because "Assistant" belongs to class model <NUM>, a class start tag CS1 precedes word W2, and a class stop tag CT1 follows word W2 in secondary result <NUM>. In class models containing multiple words, class start tag CS1 will precede the initial word and class stop tag CT1 will follow the final word. When class start tag CS1 is encountered by merging logic <NUM>, stitching is paused until after class stop tag CT1. Similarly, primary result <NUM> has a class start tag CS1' preceding word W2', and a class stop tag CT1' following word W2'. In some examples, a class model is detected by either primary SRE <NUM> or secondary SRE <NUM>, and stitching will pause based on either class start tag CS1 or class start tag CS1'. Class tags CS1 and CT1 are examples of class tag <NUM>, and class tags CS1' and CT1' are examples of class tag <NUM>.

<FIG> illustrates further details for primary SRE <NUM> and secondary SRE <NUM>, including a joint encoder 326b, as used in some examples. In some examples, primary SRE <NUM> uses a large primary AM <NUM>, whereas secondary AM <NUM> in secondary SRE <NUM> is smaller and structurally simpler. In some examples, to save memory space (or other computational burdens) primary SRE <NUM> and secondary SRE share some resources (e.g., shared resources <NUM>). Shared resources <NUM> may include memory space and even NN components. As illustrated, primary SRE includes primary decoder <NUM>, although primary encoder <NUM> is shown as now having two portions: early stages encoder 326a and joint encoder 326b. Secondary encoder <NUM> feeds into joint encoder 326b. For primary AM <NUM> using encoding with six hidden layers, early stages encoder 326a may have four layers. Early stages encoder 326a produces encoded sequences (senones), and may be structurally richer than secondary encoder <NUM>. In some examples, primary AM <NUM> uses a unidirectional long short-term memory (LSTM) network, a bi-directional LSTM network, or a different network architecture.

LSTM networks are a form of a recurrent neural network (RNN) architecture that has feedback connections, and are used for classifying, processing and making predictions based on time series data, since there can be lags of unknown duration between important events in a time series. A bidirectional LSTM network uses LSTM networks going in opposite directions, one taking the input in a forward direction, and the other in a backward direction. Primary AM <NUM> and secondary AM <NUM> may have different architectures, for example, primary AM <NUM> uses a hybrid model and secondary AM <NUM> uses a recurrent neural network transducer (RNN-T) model, as shown in <FIG>. In some examples, secondary AM <NUM> is one-third to one-fourth the size of primary AM <NUM>.

In some examples, encoded output of secondary encoder <NUM> is provided to joint encoder <NUM> at one of the hidden layers. Secondary encoder <NUM> outputs may be ahead of early stages encoder 326a by a few frames, so secondary encoder <NUM> may be able to encode both its current state as well as some future output states. The secondary encoder <NUM> outputs may be concatenated to the early stages encoder 326a output and provided to joint encoder 326b. In some examples, joint encoder 326b performs a softmax operation, which transforms a set of inputs to values between <NUM> and <NUM> so that they can be interpreted as probabilities. In some scenarios, this joint encoding has the potential to yield more accurate speech recognition results.

In some examples, merging logic <NUM> and/or stitching logic <NUM> use rule-tree metadata to identify a sub-grammar from which each word comes, to apply additional merging restrictions and further improve user experience. Further detail on the operations of merging logic <NUM> and/or stitching logic <NUM> is provided in relation to <FIG>.

<FIG> illustrates a version <NUM> of arrangement <NUM>, in which user <NUM> is using microphone <NUM> to capture audio stream <NUM> on a local node <NUM> (e.g., a smartphone, tablet, personal computer (PC) or other computing apparatus) and primary SRE <NUM> executes on a remote node <NUM> across a network <NUM>. Secondary SRE <NUM> remains on local node <NUM>. In some examples, local node <NUM> and remote node <NUM> each comprises a computing device <NUM> of <FIG>. Local node <NUM> transmits audio stream <NUM> to remote node <NUM>, along with timing information for audio stream <NUM> (so that sync markers may be accurate) for processing by primary SRE <NUM>. Remote node <NUM> returns primary result <NUM> for use by merging logic <NUM> to produce word list <NUM> for display in a display element <NUM> (e.g., a portion of presentation components <NUM> of <FIG>, such as a video screen). By watching display element <NUM>, user <NUM> perceives only latency <NUM>.

Version <NUM> may be used when, for example, primary SRE <NUM> is too large or computationally burdensome to execute on local node <NUM>. In some examples, primary SRE <NUM> is four or five times the size of secondary SRE <NUM>, and demand significantly more computational power. If local node <NUM> is a small, portable battery-operated device, then performance of arrangement <NUM> may suffer if primary SRE <NUM> executes locally. However, in some examples of arrangement <NUM>, primary SRE <NUM> and secondary SRE both reside on a single computing device (e.g., computing device <NUM> or local node <NUM>), which may be a smartphone, a tablet, or a PC, such as a desktop PC or a notebook PC.

<FIG> illustrates an example recurrent neural network transducer (RNN-T) <NUM> that may be used within arrangement <NUM>, for example within secondary SRE <NUM>, as noted above. That is, in some examples, secondary SRE comprises RNN-T <NUM>. RNN-T <NUM> processes input samples and streams output symbols, which is useful for speech dictation. In some examples, output symbols are words or alphabet characters. RNN-T <NUM> recognizes outputs one-by-one, as speech is received, with white spaces between words.

In operation, an audio segment <NUM>, which is a portion of audio stream <NUM>, is input into an encoder <NUM>, which may be used as secondary encoder <NUM>. A joint network <NUM> receives the output of encoder <NUM> and a prediction network <NUM>, which makes predictions based on prior output <NUM>. That is, prediction network <NUM> predicts future outputs based on current outputs, in the time reference that prior output <NUM> is "current" and the output of joint network <NUM> is the "future" output. The output of joint network <NUM> is subject to a softmax operation <NUM> which transforms a set of inputs to values between <NUM> and <NUM> so that they can be interpreted as probabilities. RNN-T <NUM> emits an output <NUM> that is copied, via a feedback loop to prior output <NUM> for use in the next iteration, with the next audio segment <NUM>.

<FIG> is a flowchart <NUM> illustrating exemplary operations involved in performing speech recognition that advantageously improves user-perceived latency while maintaining accuracy. In some examples, operations described for flowchart <NUM> are performed by computing device <NUM> of <FIG>. <FIG> should be viewed with <FIG>.

Flowchart <NUM> commences with operation <NUM>, which includes receiving audio stream <NUM>, in parallel, by primary SRE <NUM> and secondary SRE <NUM>. In some examples, operation <NUM> includes receiving audio stream <NUM> by secondary SRE <NUM> on local node <NUM> (a computing device) and transmitting audio stream <NUM> to remote node <NUM> for processing by primary SRE <NUM>. In some examples, timing information for audio stream <NUM> is also transmitted, so that sync markers may be accurate. Operation <NUM> includes generating, with secondary SRE <NUM>, secondary result <NUM>. In some examples, secondary SRE <NUM> comprises secondary encoder <NUM> and secondary decoder <NUM>. In some examples, secondary SRE <NUM> comprises an ML model. In some examples, secondary SRE <NUM> comprises an NN. In some examples, secondary encoder <NUM> outputs senones with probabilities. Secondary decoder <NUM> outputs words (e.g., word <NUM>). In some examples, secondary result <NUM> comprises an identified word <NUM>, start sync marker <NUM>, stop sync marker <NUM>, and class tag <NUM>. In some examples, the sync markers (e.g., start sync marker <NUM> and stop sync marker <NUM>) comprise timestamps. In some examples, class tag <NUM> comprises a class start tag or a class stop tag.

Operation <NUM> includes determining, within secondary result <NUM>, whether at least some words belong to class model <NUM>, as part of operation <NUM> (in some examples). If so, class tags are applied before and after the words that belong to class model <NUM>, in some examples (see <FIG>). In some examples, class model <NUM> is selected from a list comprising: a contact name, a date, a time, an application name, a filename, a location, a commonly-recognized name. In some examples, a commonly-recognized name comprises a sports team name or a name of a well-known person.

Operation <NUM> includes generating, with primary SRE <NUM>, primary result <NUM>. In some examples, primary SRE <NUM> has a higher accuracy than secondary SRE <NUM>, at a cost of potentially higher latency. In some examples, secondary SRE <NUM> has a lower latency than primary SRE <NUM>, at a cost of potentially lower accuracy. In some examples, primary SRE <NUM> comprises primary encoder <NUM> and primary decoder <NUM>. In some examples, primary SRE <NUM> comprises an ML model. In some examples, primary SRE <NUM> comprises an NN. In some examples, primary encoder <NUM> outputs senones with probability distributions. Primary decoder <NUM> outputs words (e.g., word <NUM>). In some examples, primary result <NUM> comprises an identified word <NUM>, start sync marker <NUM>, stop sync marker <NUM>, and class tag <NUM>. In some examples, the sync markers (e.g., start sync marker <NUM> and stop sync marker <NUM>) comprise timestamps. In some examples, class tag <NUM> comprises a class start tag or a class stop tag. According to the claimed invention, operation <NUM> further includes operation <NUM>, providing encoded sequences from secondary SRE <NUM> into an intermediate stage of primary SRE <NUM> (as shown in <FIG>).

Operation <NUM> includes determining, within primary result <NUM> whether at least some words belong to class model <NUM>, also as part of operation <NUM> (in some examples). Operation <NUM> includes receiving primary result <NUM> from remote node <NUM> across network <NUM> (e.g., receiving, from remote node <NUM>, primary result <NUM>), when version <NUM> of arrangement <NUM> is used (e.g., primary SRE <NUM> resides on remote node <NUM> and secondary SRE <NUM> resides on local node <NUM>).

Some examples of arrangement <NUM> append word list <NUM> with secondary results <NUM>, whereas some examples of arrangement <NUM> append word list <NUM> with whichever of secondary results <NUM> and primary results <NUM> is ahead (e.g. in time). Although secondary SRE <NUM> may be generally faster than primary SRE <NUM>, in some scenarios, primary SRE <NUM> may actually be ahead of secondary SRE <NUM>. Thus, operation <NUM> includes determining whether secondary SRE <NUM> is ahead of primary SRE <NUM> or whether primary SRE <NUM> is ahead of secondary SRE <NUM>. This may be accomplished using sync markers <NUM>, <NUM>, <NUM>, and <NUM>. Operation <NUM> includes appending results to word list <NUM>. In some examples, the default is appending secondary result <NUM> to word list <NUM>. In some examples, operation <NUM> appends secondary result <NUM> to word list <NUM> when secondary result <NUM> is ahead of primary result <NUM>. In such examples, appending secondary result <NUM> to word list <NUM> comprises, based on at least secondary SRE <NUM> being ahead of primary SRE <NUM>, appending secondary result <NUM> to word list <NUM>. In some examples, however, operation <NUM> includes, based on at least primary SRE <NUM> being ahead of secondary SRE <NUM>, appending primary result <NUM> to word list <NUM>.

Operation <NUM> includes merging primary result <NUM> into secondary result <NUM> in word list <NUM>. Operation <NUM> comprises operations <NUM>-<NUM>, and is an ongoing loop operation, inside of looping operations <NUM>-<NUM>. In some examples, merging operation <NUM> does not occur across different grammar models. Operation <NUM> includes synchronizing primary result <NUM> with secondary result <NUM>. In some examples, synchronizing primary result <NUM> with secondary result <NUM> comprises comparing a sync marker of primary result <NUM> with a sync marker of secondary result <NUM> (e.g., comparing start sync marker <NUM> with start sync marker <NUM>, and/or comparing stop sync marker <NUM> with stop sync marker <NUM>).

Operation <NUM> includes, based on at least the synchronizing, determining word <NUM> in primary result <NUM> that corresponds with corresponding word <NUM> in secondary result <NUM>. Decision operation <NUM> includes determining whether word <NUM> in primary result <NUM> differs from corresponding word <NUM> in secondary result <NUM>. If decision operation <NUM> determines that there is no difference, some examples forego stitching and proceed to the next word or grammar model. Decision operation <NUM> includes determining whether a class model or grammar model pauses stitching. For example, decision operation <NUM> includes determining, within primary result <NUM> or secondary result <NUM>, whether word <NUM> or corresponding word <NUM> belongs to class model <NUM>. Additionally, decision operation <NUM> may include determining whether stitching will occur within a grammar model (allowable) or across grammar classes (prohibited, in some examples). If stitching is paused, operation <NUM> returns to <NUM> or <NUM>.

If stitching is permitted (e.g., not paused), operation <NUM> includes, based on at least determining that corresponding word <NUM> in secondary result <NUM> does not belong to class model <NUM>, replacing corresponding word <NUM> in secondary result <NUM> with word <NUM> in primary result <NUM>. In some examples, replacing corresponding word <NUM> in secondary result <NUM> with word <NUM> in primary result <NUM> comprises, based on at least determining that word <NUM> in primary result <NUM> differs from corresponding word <NUM> in secondary result <NUM> and determining that corresponding word <NUM> in secondary result <NUM> does not belong to class model <NUM>, replacing corresponding word <NUM> in secondary result <NUM> with word <NUM> in primary result <NUM>.

Operation <NUM> includes displaying word list <NUM>, for example displaying word list <NUM> to user <NUM> on display element <NUM>, for example as captioning for video streaming or a video conference. In some examples, word list <NUM> comprises at least a portion of a real-time transcription of a live conversation. Operation <NUM> includes using recognized words in word list <NUM> as voice commands. Operations <NUM>-<NUM> form an ongoing loop, with operation <NUM> looping internally.

<FIG> is a flowchart <NUM> illustrating exemplary operations that may be used in conjunction with flowchart <NUM> of <FIG>. In some examples, operations described for flowchart <NUM> are performed by computing device <NUM> of <FIG>. Flowchart <NUM> commences with operation <NUM>, in which secondary SRE <NUM> emits secondary result <NUM>, which may be a partial result. In operation <NUM>, primary SRE <NUM> produces primary result <NUM>, which may also be a partial result and primary result <NUM> is stored. Merging logic <NUM> copies stored primary result <NUM> in operation <NUM>, and finds the word boundary of the final word in primary result <NUM> (e.g., using the final stop sync marker <NUM>). In operation <NUM>, merging logic <NUM> appends the words from secondary result <NUM> whose beginning word boundary (e.g., the earliest start sync marker <NUM>) is greater than or equal to the word boundary found for primary result <NUM> (e.g., using the final stop sync marker <NUM>). This corresponds to operation <NUM> of flowchart <NUM>. The result is displayed as word list <NUM> to user <NUM>, in operation <NUM> (corresponding to operation <NUM> of flowchart <NUM>). In some examples, merging is accomplished using partial results, rather than waiting for final results, in order to reduce latency. Merging uses sync markers (e.g., word timestamps) and may use rule-tree metadata to identify a sub-grammar from which each word comes, to apply additional merging restrictions and further improve user experience.

<FIG> is a flowchart <NUM> that illustrates exemplary operations involved in performing speech recognition that advantageously improves user-perceived latency while maintaining accuracy. In some examples, operations described for flowchart <NUM> are performed by computing device <NUM> of <FIG>. Flowchart <NUM> commences with operation <NUM>, which includes receiving an audio stream, in parallel, by a primary SRE and a secondary SRE. Operation <NUM> includes generating, with the primary SRE, a primary result. Operation <NUM> includes generating, with the secondary SRE, a secondary result. Operation <NUM> includes appending the secondary result to a word list. Operation <NUM> includes merging the primary result into the secondary result in the word list, which comprises operations <NUM>-<NUM>.

Operation <NUM> includes synchronizing the primary result with the secondary result. Operation <NUM> includes determining, within the primary result or the secondary result, whether at least some words belong to a class model. Operation <NUM> includes, based on at least the synchronizing, determining a word in the primary result that corresponds with a corresponding word in the secondary result. Operation <NUM> includes, based on at least determining that the corresponding word in the secondary result does not belong to a class model, replacing the corresponding word in the secondary result with the word in the primary result.

An example method of speech recognition comprises: receiving an audio stream, in parallel, by a primary SRE and a secondary SRE; generating, with the primary SRE, a primary result; generating, with the secondary SRE, a secondary result; appending the secondary result to a word list; and merging the primary result into the secondary result in the word list, wherein the merging comprises: synchronizing the primary result with the secondary result; determining, within the primary result or the secondary result, whether at least some words belong to a class model; based on at least the synchronizing, determining a word in the primary result that corresponds with a corresponding word in the secondary result; and based on at least determining that the corresponding word in the secondary result does not belong to a class model, replacing the corresponding word in the secondary result with the word in the primary result.

An example system for speech recognition comprises: a processor; and a computer-readable medium storing instructions that are operative upon execution by the processor to: receive an audio stream, in parallel, by a primary SRE and a secondary SRE; generate, with the primary SRE, a primary result; generate, with the secondary SRE, a secondary result; append the secondary result to a word list; and merge the primary result into the secondary result in the word list, wherein the merging comprises: synchronizing the primary result with the secondary result; determining, within the primary result or the secondary result, whether at least some words belong to a class model; based on at least the synchronizing, determining a word in the primary result that corresponds with a corresponding word in the secondary result; and based on at least determining that the corresponding word in the secondary result does not belong to a class model, replacing the corresponding word in the secondary result with the word in the primary result.

One or more example computer storage devices (e.g., a computing device) have computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising: receiving an audio stream, in parallel, by a primary speech SRE and a secondary SRE; generating, with the primary SRE, a primary result; generating, with the secondary SRE, a secondary result; appending the secondary result to a word list; and merging the primary result into the secondary result in the word list, wherein the merging comprises: synchronizing the primary result with the secondary result; determining, within the primary result or the secondary result, whether at least some words belong to a class model; based on at least the synchronizing, determining a word in the primary result that corresponds with a corresponding word in the secondary result; and based on at least determining that the corresponding word in the secondary result does not belong to a class model, replacing the corresponding word in the secondary result with the word in the primary result.

A, example computing device has computer-executable instructions stored thereon, which, on execution by a computer, cause the computer to perform operations comprising: receiving an audio stream by a secondary speech recognition engine (SRE) on the computing device; transmitting the audio stream to a remote node for processing by a primary SRE; receiving, from the remote node, a primary result; generating, with the secondary SRE, a secondary result; appending the secondary result to a word list; and merging the primary result into the secondary result in the word list, wherein the merging comprises: synchronizing the primary result with the secondary result; determining, within the primary result or the secondary result, whether at least some words belong to a class model; based on at least the synchronizing, determining a word in the primary result that corresponds with a corresponding word in the secondary result; and based on at least determining that the corresponding word in the secondary result does not belong to a class model, replacing the corresponding word in the secondary result with the word in the primary result.

Alternatively, or in addition to the other examples described herein, examples may include any combination of the following:.

<FIG> is a block diagram of an example computing device <NUM> for implementing aspects disclosed herein, and is designated generally as computing device <NUM>. Computing device <NUM> is but one example of a suitable computing environment and is not intended to suggest any limitation as to the scope of use or functionality of the examples disclosed herein. Neither should computing device <NUM> be interpreted as having any dependency or requirement relating to any one or combination of components/modules illustrated. The examples disclosed herein may be described in the general context of computer code or machine-useable instructions, including computer-executable instructions such as program components, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program components including routines, programs, objects, components, data structures, and the like, refer to code that performs particular tasks, or implement particular abstract data types. The disclosed examples may be practiced in a variety of system configurations, including personal computers, laptops, smart phones, mobile tablets, hand-held devices, consumer electronics, specialty computing devices, etc. The disclosed examples may also be practiced in distributed computing environments when tasks are performed by remote-processing devices that are linked through a communications network.

Computing device <NUM> includes a bus <NUM> that directly or indirectly couples the following devices: computer-storage memory <NUM>, one or more processors <NUM>, one or more presentation components <NUM>, I/O ports <NUM>, I/O components <NUM>, a power supply <NUM>, and a network component <NUM>. While computing device <NUM> is depicted as a seemingly single device, multiple computing devices <NUM> may work together and share the depicted device resources. For example, memory <NUM> is distributed across multiple devices, and processor(s) <NUM> is housed with different devices.

Bus <NUM> represents what may be one or more busses (such as an address bus, data bus, or a combination thereof). Although the various blocks of <FIG> are shown with lines for the sake of clarity, delineating various components may be accomplished with alternative representations. For example, a presentation component such as a display device is an I/O component in some examples, and some examples of processors have their own memory. Distinction is not made between such categories as "workstation," "server," "laptop," "hand-held device," etc., as all are contemplated within the scope of <FIG> and the references herein to a "computing device. " Memory <NUM> may take the form of the computer-storage media references below and operatively provide storage of computer-readable instructions, data structures, program modules and other data for the computing device <NUM>. In some examples, memory <NUM> stores one or more of an operating system, a universal application platform, or other program modules and program data. Memory <NUM> is thus able to store and access data 912a and instructions 912b that are executable by processor <NUM> and configured to carry out the various operations disclosed herein.

In some examples, memory <NUM> includes computer-storage media in the form of volatile and/or nonvolatile memory, removable or non-removable memory, data disks in virtual environments, or a combination thereof. Memory <NUM> may include any quantity of memory associated with or accessible by the computing device <NUM>. Memory <NUM> may be internal to the computing device <NUM> (as shown in <FIG>), external to the computing device <NUM> (not shown), or both (not shown). Examples of memory <NUM> in include, without limitation, random access memory (RAM); read only memory (ROM); electronically erasable programmable read only memory (EEPROM); flash memory or other memory technologies; CD-ROM, digital versatile disks (DVDs) or other optical or holographic media; magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices; memory wired into an analog computing device; or any other medium for encoding desired information and for access by the computing device <NUM>. Additionally, or alternatively, the memory <NUM> may be distributed across multiple computing devices <NUM>, for example, in a virtualized environment in which instruction processing is carried out on multiple devices <NUM>. For the purposes of this disclosure, "computer storage media," "computer-storage memory," "memory," and "memory devices" are synonymous terms for the computer-storage memory <NUM>, and none of these terms include carrier waves or propagating signaling.

Processor(s) <NUM> may include any quantity of processing units that read data from various entities, such as memory <NUM> or I/O components <NUM>. Specifically, processor(s) <NUM> are programmed to execute computer-executable instructions for implementing aspects of the disclosure. The instructions may be performed by the processor, by multiple processors within the computing device <NUM>, or by a processor external to the client computing device <NUM>. In some examples, the processor(s) <NUM> are programmed to execute instructions such as those illustrated in the flow charts discussed below and depicted in the accompanying drawings. Moreover, in some examples, the processor(s) <NUM> represent an implementation of analog techniques to perform the operations described herein. For example, the operations are performed by an analog client computing device <NUM> and/or a digital client computing device <NUM>. Presentation component(s) <NUM> present data indications to a user or other device. Exemplary presentation components include a display device, speaker, printing component, vibrating component, etc. One skilled in the art will understand and appreciate that computer data may be presented in a number of ways, such as visually in a graphical user interface (GUI), audibly through speakers, wirelessly between computing devices <NUM>, across a wired connection, or in other ways. I/O ports <NUM> allow computing device <NUM> to be logically coupled to other devices including I/O components <NUM>, some of which may be built in. Example I/O components <NUM> include, for example but without limitation, a microphone, joystick, game pad, satellite dish, scanner, printer, wireless device, etc..

The computing device <NUM> may operate in a networked environment via the network component <NUM> using logical connections to one or more remote computers. In some examples, the network component <NUM> includes a network interface card and/or computer-executable instructions (e.g., a driver) for operating the network interface card. Communication between the computing device <NUM> and other devices may occur using any protocol or mechanism over any wired or wireless connection. In some examples, network component <NUM> is operable to communicate data over public, private, or hybrid (public and private) using a transfer protocol, between devices wirelessly using short range communication technologies (e.g., near-field communication (NFC), Bluetooth™ branded communications, or the like), or a combination thereof. Network component <NUM> communicates over wireless communication link <NUM> and/or a wired communication link 926a to a cloud resource <NUM> across network <NUM>. Various different examples of communication links <NUM> and 926a include a wireless connection, a wired connection, and/or a dedicated link, and in some examples, at least a portion is routed through the internet.

Although described in connection with an example computing device <NUM>, examples of the disclosure are capable of implementation with numerous other general-purpose or special-purpose computing system environments, configurations, or devices. Examples of well-known computing systems, environments, and/or configurations that may be suitable for use with aspects of the disclosure include, but are not limited to, smart phones, mobile tablets, mobile computing devices, personal computers, server computers, hand-held or laptop devices, multiprocessor systems, gaming consoles, microprocessor-based systems, set top boxes, programmable consumer electronics, mobile telephones, mobile computing and/or communication devices in wearable or accessory form factors (e.g., watches, glasses, headsets, or earphones), network PCs, minicomputers, mainframe computers, distributed computing environments that include any of the above systems or devices, virtual reality (VR) devices, augmented reality (AR) devices, mixed reality (MR) devices, holographic device, and the like. Such systems or devices may accept input from the user in any way, including from input devices such as a keyboard or pointing device, via gesture input, proximity input (such as by hovering), and/or via voice input.

By way of example and not limitation, computer readable media comprise computer storage media and communication media. Computer storage media include volatile and nonvolatile, removable and non-removable memory implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, or the like. Computer storage media are tangible and mutually exclusive to communication media. Computer storage media are implemented in hardware and exclude carrier waves and propagated signals. Computer storage media for purposes of this disclosure are not signals per se. Exemplary computer storage media include hard disks, flash drives, solid-state memory, phase change random-access memory (PRAM), static random-access memory (SRAM), dynamic random-access memory (DRAM), other types of random-access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that may be used to store information for access by a computing device. In contrast, communication media typically embody computer readable instructions, data structures, program modules, or the like in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media.

The order of execution or performance of the operations in examples of the disclosure illustrated and described herein is not essential, and may be performed in different sequential manners in various examples.

Claim 1:
A method of speech recognition, the method comprising:
receiving (<NUM>) an audio stream (<NUM>), in parallel, by a primary speech recognition engine, SRE, (<NUM>) and a secondary SRE (<NUM>);
providing (<NUM>) encoded sequences from the secondary SRE into an intermediate stage of the primary SRE;
generating (<NUM>), with the primary SRE, a primary result (<NUM>);
generating (<NUM>), with the secondary SRE, a secondary result (<NUM>);
appending (<NUM>) the secondary result to a word list (<NUM>); and
merging (<NUM>) the primary result into the secondary result in the word list, wherein the merging comprises:
synchronizing (<NUM>) the primary result with the secondary result;
determining (<NUM>), within the primary result or the secondary result, whether at least some words belong to a class model (<NUM>);
based on at least the synchronizing, determining (<NUM>) a word (<NUM>) in the primary result that corresponds with a corresponding word (<NUM>) in the secondary result; and
based on at least determining that the corresponding word in the secondary result does not belong to the class model, replacing (<NUM>) the corresponding word in the secondary result with the word in the primary result.