Frame skipping with extrapolation and outputs on demand neural network for automatic speech recognition

Techniques related to implementing neural networks for speech recognition systems are discussed. Such techniques may include implementing frame skipping with approximated skip frames and/or distances on demand such that only those outputs needed by a speech decoder are provided via the neural network or approximation techniques.

BACKGROUND

Neural networks including deep neural networks may be used for machine learning and perceptual and cognitive systems. Such neural networks may be used in a variety of implementations such as speech recognition systems. For example, neural networks may include interconnected layers of neurons or nodes. Input values for each layer include inputs to the system (e.g., at the input layer) or outputs from a previous layer in the neural network. Output values from the output layer of the neural network may include output values, distance values, or classification values, or the like such that the input to the neural network may be classified via the neural network and/or additional processing. Such neural network processing and classifications may be used to perform classifications or other tasks that may be difficult or impossible to perform with more standard rule-based processing systems.

As discussed, such neural networks may be implemented in automatic speech recognition (ASR) systems and, in some implementations, they may be the most important component of such systems. A problem with current neural networks in real-time applications is the large computational effort needed to evaluate the neural network. To address this problem, some current implementations offload neural network computations from a central processing unit (CPU) of a device to a graphics processing unit (GPU) of the device. However, such offloading may cause conflicts with other GPU-intensive applications such as games being run on the device or the device's camera or the like. Furthermore, intensive use of the device's GPU may increase power usage and thereby limit battery life for mobile devices. In other implementations, single instruction, multiple data (SIMD) platforms and/or optimizations such as batched lazy evaluation models (which may delay calculations until needed) may be used. However, such implementations may have reduced classification accuracy.

As such, existing techniques do not provide real-time, efficient, and accurate neural network implementations. Such problems may become critical as the desire to utilize classifications via neural networks such as in speech recognition becomes more widespread.

DETAILED DESCRIPTION

Methods, devices, apparatuses, computing platforms, and articles are described herein related to neural networks implemented for speech recognition and, in some examples, to frame skipping techniques and output values on demand techniques implemented via the neural network.

As described above, implementing neural networks in real time may be advantageous to users but difficult due to limited computational resources and intensive use of battery resources. Furthermore, attempts to reduce such resource usage may provide inaccurate classification results. Optimizing neural networks may have a direct correlation to total cost of ownership in services hosted via a data center and to battery life in applications implemented via a mobile device.

In some embodiments discussed herein, frame skipping techniques may be implemented via a neural network. For example, when using frame skipping, neural network outputs (e.g., distance values) may be calculated or determined for every Nthtime instance or frame. For time instances where neural network distance values are not determined, such distance values may be approximated based on neural network determined distance values from one or more prior time instances or frames. For example, evaluating the neural network may be computationally complex as, in some examples, the entire neural network is evaluated at that time instance or for that frame. Distance values determined by approximation methods may be computed with much less computational complexity (and without evaluation of the neural network). For example, approximated distance values may be determined by extrapolation techniques using one or more prior frames of distance values. In some examples, the extrapolation may include a linear extrapolation based on distance values from two previous frames. As used herein, evaluation frames or non-skip frames refer to distance values determined by evaluation of the neural network and skip frames refer to distance values determined by approximation techniques. Such frame skipping combined with approximation of distance values for skip frames may provide substantial computational savings with no or minimal cost of speech recognition accuracy as is discussed further herein.

Furthermore, in some embodiments discussed herein, neural network outputs (e.g., distance values) on demand (e.g., distances on demand or DOD) techniques may be implemented via a neural network. For example, when using distances on demand techniques, a downstream decoder such as a speech decoder may provide, for a time instance or frame, requested distance values that are a subset of all available distance values. For example, a speech decoder such as a Viterbi beam searching decoder may, for a particular time instance, only require a subset of the distance values available from the neural network. In such examples, the speech decoder may provide output indices (e.g., indicators of which outputs or distance values are needed for a particular time instance) to the neural network. The neural network, as is discussed further herein, may include an input layer, one or more hidden layers, and an output layer. For example, outputs or distance values from the output layer may be provided to the speech decoder. Since, in some examples, each node of the output layer is connected to every node of the final hidden layer (e.g., the hidden layer connected to the output layer), the final hidden layer must be fully evaluated to evaluate even one node of the output layer. Therefore, in some examples, the neural network may be fully evaluated through the final hidden layer, but only the subset of nodes associated with the requested output indices may be evaluated. By not evaluating non-requested output nodes, substantial computational savings may be made, particularly when the output layer is a substantial portion of the entire neural network as is the case in many implementations.

In some embodiments, such frame skipping techniques and such distances on demand techniques may be combined. For example, the speech decoder may provide output indices as discussed for every time instance. For evaluation or non-skip frames, the subset of distance values associated with the output indices may be determined as discussed (e.g., the neural network may be fully evaluated through the final hidden layer and only those output nodes associated with the output indices may be evaluated with the resultant distance values provided to the speech decoder). Furthermore, the subset of distance values and resultant values from the final hidden layer may be saved in memory for subsequent usage as is discussed below. For skip frames, the subset of distance values associated with the output indices may be approximated using extrapolation techniques, for example. Such techniques may require, for a particular distance value to be approximated, associated (e.g., from the same output layer node) distance values from prior frame(s). In some examples, such distance values from prior frame(s) may have been previously requested by the speech decoder, determined via the neural network, and saved to memory as discussed.

In other examples, such distance values from prior frame(s) may not have been previously determined via the neural network. In such examples, the saved final hidden layer values may be used via the neural network to re-evaluate the nodes of the output layer of the neural network for the needed prior frame(s) distance values. For example, the needed distance values may be from a previous frame or time instance but necessary for approximating a current distance value for a current frame or time instance. The current distance value or values may then be approximated as discussed using, for example, linear extrapolation techniques or the like. Such combination of frame skipping and distances on demand techniques may eliminate evaluation of the neural network for skip frames or skip time instances and may substantially reduce the evaluation of the output layer of the neural network for evaluation frames (and subsequent re-evaluation as discussed).

The discussed distance values may be used by the speech decoder to determine a sequence of textual elements such as words or phrases or n-grams or the like. The techniques discussed herein may save computational resources, battery life for mobile device implementations, cost of ownership for cloud or remote server implementations, or the like. Furthermore, such techniques may provide real-time implementations for speech recognition as is discussed further herein.

FIG. 1is an illustrative diagram of an example setting100for providing speech recognition, arranged in accordance with at least some implementations of the present disclosure. As shown inFIG. 1, setting100may include a user101providing speech103for evaluation by device102. For example, device102may provide speech recognition such that speech103may be translated into text or textual elements such as words, sentences, n-grams, or the like. As shown, in some examples, a speech recognition system may be implemented via a device such as device102. As illustrated, in some examples, device102may be a smartphone. However, device102may be any suitable device such as a computer, a laptop, an ultrabook, a tablet, or the like. In some examples, device102may be a wearable device such as a smart watch or smart glasses or the like. In other examples, speech recognition may be provided via a system remote to device102such as a server or servers in a cloud speech recognition system. In some examples, speech103may be received via a microphone104of device102(illustrated on a bottom of device102). In other examples, speech103may be received as a pre-recording of speech or a speech signal or the like. Furthermore, in some examples, the textual elements may be provided to user101via a display105of device102. In other examples, the textual elements may be saved to a memory of device102or to a remote cloud memory or the like. In some examples, device102may be described as a computing device as used herein.

FIG. 2is an illustrative diagram of an example system200for providing speech recognition, arranged in accordance with at least some implementations of the present disclosure. As shown inFIG. 2, system200may include microphone104, a feature extraction module202, a distance values computation module204, and a speech decoder module206. In some examples, speech decoder module206may be coupled to statistical models (not shown) implemented via memory, for example, which may be compared to distance values205to determine recognized word sequence207. As shown, microphone104may receive speech103from user101. Speech103may be issued by user101and microphone104may receive speech103(e.g., as sound waves in the air) and convert speech103to an electrical signal such as a digital signal to generate speech recording201. For example, speech recording201may be stored in memory (not shown inFIG. 2). In other examples, speech recording201may be pre-recorded and speech recording201may be received by system200via another device.

Feature extraction module202may receive speech recording201from microphone104or from memory of system200and feature extraction module202may generate features203associated with speech103. Features203may include any suitable features representing speech103and features203may be represented in any suitable format such as a feature vector format or the like. For example, features203may be coefficients representing a power spectrum of the received speech or other spectral analysis coefficients or parameters. In some examples, features203may be Mel frequency cepstrum coefficients (MFCCs). In some examples, feature extraction module202may process a speech wave signal of speech recording201to generate a feature vector. In examples where features203are represented via feature vectors, each feature vector of features203may be based on a time window of speech103(and/or speech recording201). For example, the time window may be a certain time instance or recording duration (e.g., 10 milliseconds or the like) of speech recording201that slides across speech recording201. For example, each feature vector of features203may thereby be determined based on an evaluation (e.g., a power spectrum analysis or the like) of the associated time window. Furthermore, in some examples, features203may include a stack of feature vectors (e.g., feature vectors from multiple time instances). Features203may include any number of features. For example, features203may include 200 to 260 features, 250 to 300 features, or 300 to 400 features or the like. In an embodiment, features203include 253 features. In another embodiment, features203include 256 features. As is discussed further herein, features203may be provided to an input layer of a neural network. Feature extraction module202may transfer features203to distance values computation module204and/or a memory of system200.

Distance values computation module204may receive features203from feature extraction module202or from memory. Distance values computation module204may take as input to a neural network (e.g., either via a pre-processor, not shown, or via the neural network itself) features203. Furthermore, in some examples, distance values computation module204may receive output indices via speech decoder module206. For example, features203may include a stack of feature vectors that may include a current feature vector and a predetermined number of feature vectors preceding and/or succeeding the current feature vector. In an embodiment, features203includes a current feature vector, 5 preceding feature vectors, and 5 succeeding feature vectors for a stack of 11 feature vectors. In some examples, each feature vector includes 23 features. In examples where a stack includes 11 feature vectors each having 23 features, the number of inputs to neural network301may be 253 inputs (e.g., 23×11 inputs). In such examples, a neural network implemented via distance values computation module204may have an input layer including 253 nodes or neurons (e.g., a number of input layer nodes equal to the number of inputs to the neural network) as is discussed further herein. In some examples, such features may be provided to distance values computation module204at each time instance (e.g., for each time window as discussed above).

As is discussed further below, distance values computation module204may implement a neural network and/or a distance value approximation module to generate distance values205. As discussed, in examples implementing distances on demand techniques, distance values computation module204may receive output indices208from speech decoder module206and distance values computation module204may provide distance values only for those associated with output indices208. In some examples, no distances on demand is utilized and, in such examples, output indices208may not be implemented. Furthermore, distance values computation module204may implement frame skipping techniques such that at some time instances (e.g., for evaluation frames) distance values205are provided via a neural network and at other time instances (e.g., for skip frames) distance205are provided via approximation techniques based on distance values from one or more prior frame(s) evaluated via the neural network. As shown, distance values computation module204may transfer distance values205to speech decoder module206and/or to a memory of system200.

Speech decoder module206may receive distance values205from distance values computation module204or from memory. Speech decoder module206may decode distance values205and search for a most likely textual elements and/or recognized word sequence match. For example, speech decoder module206may receive distance values205for every time instance (e.g., 10 milliseconds or the like) and deliver recognized word sequence207after an end of speech is detected. Speech decoder module206may include any suitable speech decoder. In an example, speech decoder module206is a Viterbi beam search decoder. As shown, in some examples, speech decoder module206may provide a recognized word sequence207as an output. Recognized word sequence207may be stored to a memory of system200and/or displayed to user101via display105or the like. In some examples, recognized word sequence207may be provided to another module or software application or the like for use by the module or software application. Recognized word sequence207or textual elements as used herein may include any suitable sequence of words, sub-word units, n-grams, syllables, letters, or the like. As shown, speech decoder module206may generate recognized word sequence207based on distance values20). Furthermore, in distances on demand implementations, speech decoder module206may generate output indices208and provide such output indices208to distance values computation module204. For example, output indices208may indicate a subset (e.g., one or more) of available distance values (e.g., available via distance values computation module204) needed at a particular time instance. For example, speech decoder module206may not need all available distance values at each time instance in various embodiments.

As shown inFIG. 2, in some examples, distance values computation module204may be implemented as part of a speech recognition system. However, distance values computation module204may be implemented in any suitable system such as perceptual computing systems, machine learning systems, cognitive computing systems, image processing systems, or optical character recognition systems or the like. Furthermore, a neural network of distance values computation module204may be pre-trained based on training sets or the like prior to implementation via system200to determine weights and/or biases of the neural network. In some examples, pre-training may be implemented via system200itself. In other examples, such pre-training or other pre-implementation steps may performed by a separate system.

FIG. 3is an illustrative diagram of example distance values computation module204, arranged in accordance with at least some implementations of the present disclosure. As shown, distance values computation module204may include a neural network301, a controller302, and a distance values approximation module303. In the embodiment ofFIG. 3, distance values computation module204may implement frame skipping with distance value approximation but not distances on demand techniques and therefore, no output indices may be received via distance values computation module204. As shown, distance values computation module204may receive features203via neural network301. Neural network301may include any suitable neural network such as a deep neural network or the like.

FIG. 4is an illustrative diagram of example neural network301, arranged in accordance with at least some implementations of the present disclosure. As shown, neural network301may include an input layer401, hidden layers402,403,404,405, and output layer406. Furthermore, hidden layer405may be characterized as a final hidden layer as it is adjacent to output layer406. Also as shown, input layer401may include input layer nodes407. As discussed, input layer401may include any number of input layer nodes407. For example, input layer401may include a number of nodes equal to the number of elements features203. For example, input layer401may have 253 or 256 or the like input layer nodes407.

Furthermore, as in the illustrated example, neural network301includes four hidden layers402-405. However, in other examples, neural network may include three, five, six, or more hidden layers. Hidden layers402-405may include any number of hidden layer nodes408,409,410,411. For example, hidden layers402-405may each include 100 to 200 nodes, 200 to 300 nodes, or the like. In an embodiment, neural network301includes four hidden layers402-405each having 192 nodes. In some examples, hidden layers402-405each have the same number of nodes and, in other examples, one or more of hidden layers402-405may have different numbers of nodes.

Output layer406may include any suitable number of output layer nodes412such that distance values (DVs)205include values for comparison and/or search to determine textual elements or recognized word sequences or the like. For example, output layer406may include 400 to 800 nodes, 800 to 1,500 nodes, or 1,500 to 2,500 nodes or more. In an embodiment, output layer406includes 512 output layer nodes412. In an embodiment, output layer406includes 1,015 output layer nodes412. In the illustrated example, data flows from the left to the right from input layer401, through hidden layers402-405, and through output layer406as shown such that the output of input layer401is the input to hidden layer402, the output of hidden layer402is the input to hidden layer403and so on, and such that the output of output layer405is the output of neural network301(e.g., distance values205). In some examples, every node in a layer may be connected to every node in the adjacent layer (e.g., the layers may be fully connected). In an example, a layer with h nodes may be connected to its neighbor layer with hh nodes through h×hh weights. In an example, input layer401has 253 input layer nodes407, hidden layers402-405each have 192 hidden layer nodes408-411, output layer406has 1,105 output layer nodes412, and neural network301has about 354,000 weights. For example, every input layer node407of input layer401may be connected to every hidden layer node408of hidden layer402, every hidden layer node408of hidden layer402may be connected to every hidden layer node409of hidden layer403, and so on. In other examples, some connections between nodes may not be made.

Evaluation (e.g., computation) of neural network301may include any suitable technique or techniques. For example, input layer nodes407of input layer401may be calculated based on features203, weights associated with each feature of features203, and/or activation functions for each of input layer nodes407. In an example, each of input layer nodes407may be determined by generating a weighted sum of products of features203and their associated weights (e.g., weights for different features may be different) and applying an activation function to the weighted sum. Hidden layer nodes408may be determined based on input layer nodes407, weights associated with each of input layer nodes407(e.g., weights between different connections of input layer nodes407and hidden layer nodes408may be different), biases for each of hidden layer nodes408, and/or activation functions for each of hidden layer nodes408. In an example, each of hidden layer nodes408are determined by generating a weighted sum of products of input layer nodes407and associated weights, applying a bias to the weighted sum, and applying an activation function to the biased weighted sum. Hidden layer nodes409,410,411may be determined similarly to hidden layer nodes but using the preceding layer as inputs to the respective hidden layer. Furthermore, output layer nodes may be determined based on final hidden layer nodes411, weights associated with each of final hidden layer nodes411(e.g., the weights may be different), and/or biases for each of output layer nodes412. In an example, each of output layer nodes412are determined by generating a weighted sum of products of final hidden layer nodes411and associated weights and applying a bias to the weighted sum. As discussed, other techniques may be used to evaluate nodes of neural network301and the techniques discussed herein are not limited to any neural network evaluation technique or techniques.

In some examples, neural network301may be implemented for speech recognition in a test or implementation phase after neural network301has been trained in a training phase. Such a training phase may determine weights for nodes of neural network301, biases for nodes of neural network301, and the like. In some examples, during cross-entropy training (e.g., during the training phase) of neural network301, output layer406may have a softmax activation function that may be omitted during the implementation or test phase. In some examples, during implementation, outputs from output layer406may be scaled based on class probabilities prior to being provided as distance values304.

Also as shown inFIG. 4, neural network301may be operated under control of controller302via neural network (NN) control signal305. For example, in frame skipping implementations as discussed, neural network control signal305may control whether or not neural network301is evaluated at a particular time instance. For example, for evaluation frames, neural network control signal305may signal for evaluation of neural network301and, for skip frames, neural network control signal305may signal for no evaluation of neural network301.

Returning toFIG. 3, as discussed, for evaluation frames or time instances, controller302may provide neural network301neural network control signal305for evaluation of neural network301. At such time instances or for such frames, neural network301may generate distance values304based on features203and other characteristics (e.g., weights, biases, activation functions, and the like) of neural network301to generate distance values304, which, as shown, may be provided for such evaluation frames or time instances as distance values205. During such evaluation frames or time instances, controller302may also signal to distance value approximation module303for no approximations to be made. As shown, distance values304may also be provided to distance value approximation303for use in subsequent time instances.

For skip frames, controller302may provide neural network control signal305indicating no evaluation of neural network301. Furthermore, controller302may provide distance values approximation module303an indication that distance values307are to be generated via distance values approximation control signal306for such a skip frame or time instance associated with a skip frame. For example, at such time instances, neural network301may not be evaluated (e.g. it may be off) saving substantial computing resources. Furthermore, at such time instances, distance values205may be provided as approximated distance values307.

For example, features203may be highly correlated over time and, as a result, consecutive feature vectors of features203may be substantially similar. Furthermore, if fully evaluated for such features, neural network301would provide substantially similar or correlated distance values over time. Such similarities over time may be utilized to avoid evaluating neural network301at each time instance as discussed herein (e.g., such that frame skipping may be implemented). For example, neural network301may be evaluated for every Nthframe instead of every frame where N=2, 3, 4, or more and distance values from previously evaluated time instances (e.g., evaluation frames) may be used to approximate distance values for such skip frames.

FIG. 5illustrates an example timeline500of evaluation and skip frames, arranged in accordance with at least some implementations of the present disclosure. InFIG. 5, the x-axis of timeline500illustrates increasing time over example time instances 0, 1, 2, 3, 4, and 5. As shown, at time instances 0, 2, 4, evaluation frames501,502,503may be generated via a neural network such as neural network301. Furthermore,FIG. 5illustrates example neural network determined distance values504,505. For example, neural network determined distance values504,505may both be associated with an output node of output layer nodes412. For example, distance value504may be the output of the node at time t=0 and distance value505may be the output of the node at time t=2.FIG. 5also illustrates approximated (e.g., skip) frames506,507,508. For example, at time instances 1, 3, 5, approximated frames506,507,508may be generated via distance values approximation module303based on neural network determined distance values for previous evaluation frames501,502,503. For example,FIG. 5illustrates approximated distance value509determined based on neural network determined distance values504,505.

Approximated distance value509may be determined using any suitable approximation technique or techniques. Approximated distance value509may be associated with the same output node of output layer nodes412(e.g., although approximated distance value509is approximated and not an output of neural network301). In the illustrated example, approximated distance value509is determined based on an extrapolation using two previous neural network determined distance values504,505. For example, approximated distance value509may be extrapolated based on previous neural network determined distance values504,505and, as shown, the time instance for previous neural network determined distance values505may be before the time instance for approximated distance value509and the time instance for previous neural network determined distance values504may be before the time instance for previous neural network determined distance values505. In other examples, approximated distance value509may be determined using only one previous neural network determined distance value (e.g., previous neural network determined distance value505). In yet other examples, approximated distance value509may be determined using only three or more previous neural network determined distance values. Although illustrated and discussed with respect to extrapolation techniques, approximated distance value509may be determined using interpolation techniques (e.g., based on previous neural network determined distance value505and an analogous subsequent neural network determined distance value from subsequent evaluation frame503).

In some examples, approximated distance value509may be determined using a linear extrapolation technique. For example, approximated distance value509may be determined based on adding previous neural network determined distance value505to half of a difference between previous neural network determined distance value505and previous neural network determined distance value504. In some examples, approximated distance value509may be determined as shown in Equation (1):
o(3)=o(2)+0.5×(o(2)−o(0))  (1)
where o(3) may be the approximated distance value509, o(2) may be previous neural network determined distance value505, and o(0) may be previous neural network determined distance value504.

As shown inFIG. 5, in some examples, every other frame may be an approximated or skip frame (e.g., every Nthframe such that N=2). For example, such frame skipping may be implemented via a modulo operation where N=2 and if the time instance modulo 2 is zero, the frame is an evaluation frame. In other examples, every third frame (e.g., N=3), every fourth frame (e.g., N=4), or every fifth frame (e.g., N=5), or the like may be an approximated or skip frame. In yet other examples, a skipping pattern may be heuristically determined such that the number of skipped frames is increased or decreased based on the accuracy needs of speech decoder system206or the like. For example, controller302may determine a skipping pattern or frame skipping rate based on one or more of accuracy needs of speech decoder system206, available computing resources of speech decoder system206, or a current real time factor. For example, the real time factor may measure the speed of speech decoder system206(e.g., if it takes time P to process an input of duration I, the real time factor, RTF, may be defined as P/I). In some examples, the frame skipping rate may be dynamically determined or adjusted based on accuracy needs of speech decoder system206, available computing resources of speech decoder system206, or a current real time factor, or the like. For example, if greater accuracy is needed the frame skipping rate may be reduced, if computing resources are not available the frame skipping rate may be increased, or if the current real time factor is too high or increasing the frame skipping rate may be increased. In some examples, all three factors and/or additional factors may be used to determine the frame skipping rate via controller302or another module of system200. The frame skipping rate may be implemented via controller302to control neural network301and distances value approximation module303as discussed herein. For example, increasing the frame skipping rate may include providing one or more additional skip frames between an evaluation frame and a skip frame and reducing the frame skipping rate may include removing one or more additional skip frames from between an evaluation frame and a skip frame.

Furthermore, as discussed, a linear extrapolation may be used to determine approximated distance value509. In other examples, an extrapolation may be performed based on a non-linear function or a variance function or the like. In some examples, all distance values of an approximation frame may be determined using the same approximation technique (e.g., linear extrapolation) and, in other examples, distance values of an approximation frame may be determined using different approximation techniques (e.g., some linear, some non-linear). Furthermore,FIG. 5illustrates t=1 frame506as an approximated frame. For example, frame506may have for reference only previous evaluation frame501and as such, frame506may be approximated as equal to previous evaluation frames501. In other examples, t=1 frame506may be determined via an evaluation of neural network301. As discussed elsewhere herein, evaluation frames501,502,503may be saved in memory for retrieval and generation of approximated frames506,507,508. Furthermore, such extrapolation techniques including linear extrapolation techniques may be processed in runtime for real-time speech recognition results.

FIG. 6is a flow diagram illustrating an example process600for determining distance values using frame skipping and distance value approximation, arranged in accordance with at least some implementations of the present disclosure. Process600may include one or more operations601-608as illustrated inFIG. 6. Process600may be performed by a device such as device102or a remote server or the like. Process600may be used to determine and provide distance values for use by speech decoder as discussed herein. For example, process600may be implemented by controller302, neural network301, and distance values approximation module303.

As shown, process600may begin at starting operation601and process600may continue at decision operation602, “Evaluation or Skip Frame”, where a determination may be made as to whether the current frame is an evaluation frame or a skip frame. For example, controller302may determine whether the current frame is an evaluation frame or a skip frame. As shown, if the current frame is an evaluation frame, process600may continue at operation603, “Evaluate Neural Network to Determine Distance Values”, where the distance values may be determined for the current frame based on an evaluation of a neural network. For example, distance values304may be determined by neural network301as implemented via distance values computation module204at operation603.

If the current frame is a skip frame, process600may continue at operation604, “Approximate Distance Values based on Distance Values of Prior Frame(s)”, where distance values may be approximated based on distance values of prior neural network calculated frames. For example, distance values may be approximated using linear extrapolation as discussed herein. In an example, distance values307may be determined by distance value approximation module303as implemented via distance values computation module204at operation604.

As shown, in either the case of an evaluation frame or a skip frame, process600may continue at operation605, “Provide Distance Values”, where distance values may be provided to a speech decoder for the determination of a sequence of textual elements as discussed herein. For example, distance values computation module204may provide distance values205(e.g., including distance values304or distance values307depending on the frame type) to speech decoder module206for the generation of recognized word sequence207.

Process600may continue at decision operation606, “Last Time Instance/Frame?”, where a determination may be made as to whether the current frame is a last frame. If the current frame is not the last frame, process600may continue at operation607, “Go to Next Time Instance/Frame”, where process600may continue at a next time instance for next frame at decision operation602as shown. If the current frame is the last frame, process600may end at ending operation608.

As discussed, distance values computation module204including neural network301, controller302, and distance values approximation module303may implement frame skipping to substantially reduce computational loads in automatic speech recognition implementations. Such implementations may offer fast and accurate speech recognition results in various computing environments. For example, such frame skipping techniques may provide for a 50% or more reduction in computation cost with no loss of accuracy. Furthermore, tradeoffs between speed and accuracy may be made either by setting the number of skip frames prior to runtime or during runtime by adjusting the number of skip frames during processing. Furthermore, in contrast to prior frame skipping or multi-frame neural network implementations, the described frame skipping techniques do not require adjustment or optimization of the neural network prior to implementation or additional specifically trained knowledge sources (e.g., to train the neural network). Table 1 illustrates example results of the described frame skipping techniques.

Table 1 provides results using the same speech recognition engine for all testing. The reported results represent the mean of six recognition experiments based on about 25,000 total spoken utterances in total. In the results of Table 1, frame skipping with distance value approximation is based on linear extrapolation based on two prior neural network distances as discussed with respect to Equation (1). As shown, for prior frame skipping techniques, the word error rate steadily increase from a baseline (e.g., with no skipping at N=1) to 8% with frame skipping at N=4. With the discussed techniques, there is no increase in error rate with frame skipping at N=2 and N=3. For N=4, the increase in word error rate using distance value approximation using linear extrapolation is 4%, which is half of the increase based on prior frame skipping techniques. For example, at N=3, an automatic speech recognition system using the discussed frame skipping with distance value approximation techniques may reduce computational costs by more than 50% without a loss in accuracy. For example, the neural network may contribute as much as 70% to the overall footprint of the automatic speech recognition system and such reduction of the use of the neural network may reduce the computation footprint of the automatic speech recognition system significantly.

Furthermore, Pseudo-code (1) provides an example process for providing frame skipping with linear approximation.

As shown in Pseudo-code (1), for non-skipped frames (e.g., evaluation frames), the modulo of the time frame (e.g., time instance or frame) and a skip rate (e.g., N value) may be 0 and for such time frames, the distance values (e.g., outputs) may be determined by evaluating the neural network (e.g., compute_DNN). For skip frames, the module of the time frame and skip rate may be non-zero and the indices of the previous evaluation frames may be determined (e.g., via s and p in Pseduo-code (1)). The distance values for the skip frame may then be determined based on the previously determined neural network outputs by applying a factor (e.g., fac) to a delta between the previously determined neural network outputs and adding the delta to the most recent previously determined neural network output. For example, the factor may be based on the skip frame location relative to the prior neural network computed frames. In the example of Equation (1) with N=1, the factor may be 0.5 for example as the modulo (e.g., m) is one and the skip rate (e.g. N) is two. In examples with more skip frames between evaluation frames, the factor may vary depending on how recent the evaluation frame is to the skip frame. For example, if N=3 and the skip frame is immediately following an evaluation frame the factor may be ⅓ and if the skip frame is a second skip frame after an evaluation frame, the factor may be ⅔ for example.

As discussed, distance values computation module204including neural network301, controller302, and distance values approximation module303may implement frame skipping to substantially reduce computational loads in automatic speech recognition implementations. In other embodiments, such frame skipping techniques may be combined with distances on demand techniques.

FIG. 7is an illustrative diagram of example distance values computation module204, arranged in accordance with at least some implementations of the present disclosure. As discussed, distance values computation module204may include neural network301, controller302, and distance values approximation module303. In the embodiment ofFIG. 7, distance values computation module204may implement frame skipping with distance value approximation and distances on demand techniques. For example, distance values computation module204may receive output indices208from speech decoder module206(please refer toFIG. 2). In such examples, distance values computation module204and speech decoder module206may be bi-directionally coupled. Such output indices208may include indicators of which distance values (e.g., outputs) speech decoder module206is requesting at a particular time instance (or for a particular frame or the like). Output indices208may include any suitable indicators such as indicators associated with output layer nodes412and/or a time stamp indicating the time instance for the request. As shown, distance values computation module204may receive features203via neural network301and output indices208via controller302. Neural network301may include any suitable neural network such as a deep neural network or the like. For example, neural network301may include any neural network as discussed herein.

FIG. 8is an illustrative diagram of example neural network301, arranged in accordance with at least some implementations of the present disclosure. As shown, neural network301may include input layer401, hidden layers402,403,404,405, and output layer406. Furthermore, as discussed, hidden layer405may be characterized as a final hidden layer as it is adjacent to output layer406. Also as shown, input layer401may include input layer nodes407, hidden layers402,403,404may include hidden layer nodes408,409,410, respectively, and final hidden layer405may include final hidden layer nodes411. Also, output layer406may include output layer nodes412. The characteristics of neural network301were discussed with respect toFIG. 4and will not be repeated for the sake of brevity.

Returning toFIG. 7, controller302may receive output indices208. Controller302may also determine whether a current time instance is associated with an evaluation frame or a skip frame. If the current time instance is associated with an evaluation frame (e.g., such that distance values are to be determined based on an evaluation of neural network301), controller302may provide neural network (NN) control signal305to neural network301. Neural network control signal305may indicate the neural network is to be evaluated at the current time instance and the output nodes for which distance values are requested. For example, output indices208may indicate a subset of all available distance values and the neural network may only be requested to provide such distance values.

In such evaluation frame examples, neural network301may evaluate all layers of the network through final hidden layer405. For example, to evaluate even one output node of output layer nodes412, all layers through final hidden layer405may need to be evaluated. Returning toFIG. 8, as shown, in such examples, neural network301may evaluate all of input layer nodes407, hidden layer nodes,408,409,410, and final hidden layer nodes411. The final hidden layer values determined via final hidden layer nodes411may be saved in memory for future use as is discussed further herein with respect to skip frame examples. Furthermore, neural network301may evaluate only those output layer nodes412that are requested (e.g., via output indices208). Output layer406may also include non-evaluated output layer nodes801-805such that non-evaluated output layer nodes801-805(e.g., those that are blacked out inFIG. 8) are not calculated and only requested output layer nodes412(e.g., those that are in white) are calculated.

Returning toFIG. 7, neural network301may provide the distance values requested via controller302as requested distance values (RDVs)701, which may be provided via distance values computation module204as a portion of distance values205to speech decoder module206(please refer toFIG. 2).

As discussed, if the current frame is an evaluation frame, neural network301may be implemented to determine requested distance values701. If instead the current frame (or a subsequent frame) is a skip frame, controller302may provide distance values approximation control signal306to distance values approximation module303requesting approximation of the distance values requested via output indices208. However, as discussed herein, distance values approximation module303may generate approximated distance values based on previous distance values calculated via neural network301. Furthermore, as discussed with respect to the evaluation frame example, only a subset of neural network output layer nodes may be evaluated and only the corresponding subset of neural network determined distance values may be available. If the skip frame requested output indices208correspond to distance values determined at a previous evaluation frame, distance values approximation module303may use such previously determined distance values to generate requested distance values702via approximation techniques as discussed herein.

However, if such previously determined distance values (e.g., via neural network301) are not available (e.g., they were not previously calculated and saved via memory), controller302may provide, via neural network control signal305, a request to neural network301to determine the needed distance values for the previous frame. For example, neural network301may load saved final hidden layer values for the previous evaluation frame and evaluate the newly requested nodes of output layer406. For example, referring toFIG. 8, a distance value associated with previously non-requested output layer node804may now be needed to approximate a distance value for a (current) skip frame. Neural network301may evaluate the requested node and provide the requested distance value701to distance values approximation module303as shown inFIG. 7. Such a process may be repeated for any number of needed neural network determined distance values and for any number of previous evaluation frames (e.g., typically two previous evaluation frames).

Returning toFIG. 7, distance values approximation module303, now with the needed neural network determined distance values, may generate requested distance values702for the current skip frame. For example, distance values approximation module303may determine such requested distance values702using extrapolation or interpolation techniques based on linear, non-linear, or variance functions as described herein. Distance values approximation module303may determine such requested distance values702using any techniques or characteristics discussed herein and such techniques or characteristics will not be repeated for the sake of brevity.

Referring toFIG. 2, speech decoder module206may receive such (requested) distance values205and may continue to decode and/or search for recognized word sequences. Furthermore, speech decoder module206may generate, for a next time frame, output indices208, which may indicate distance values205that speech decoder needs for the next frame to continue the described decoding/searching. For example, speech decoder module206may be a Viterbi beam searching or pruning speech decoder that may limit the number or inventory of hypotheses being evaluated such that a subset of available distance values may be used to effectively search for recognized word sequence207or a portion thereof.

FIG. 9is a flow diagram illustrating an example process900for determining distance values using frame skipping, distances on demand, and distance value approximation, arranged in accordance with at least some implementations of the present disclosure. Process900may include one or more operations901-913as illustrated inFIG. 9. Process900may be performed by a device such as device102or a remote server or the like. Process900may be used to determine and provide distance values for use by speech decoder as discussed herein. For example, process900may be implemented by controller302, neural network301, and distance values approximation module303.

As shown, process900may begin at starting operation901and process900may continue at operation902, “Receive Output Indices for Time Instance/Frame”, where output indices may be received for a current time instance or frame. For example, speech decoder module206may generate output indices208, which may be provided to and received by distance values computation module204. Process900may continue at decision operation903, “Evaluation or Skip Frame”, where a determination may be made as to whether the current frame is an evaluation frame or a skip frame. For example, controller302may determine whether the current frame is an evaluation frame or a skip frame based on a modulo calculation using a frame skip rate.

As shown, if the current frame is an evaluation frame, process900may continue at operation904, “Evaluate Neural Network through Final Hidden Layer”, where a neural network may be evaluated through a final hidden layer. For example, neural network301may be fully evaluated from input layer401through final hidden layer405. Process900may continue at operation905, “Determine Distance Values via Output Layer Nodes Associated with Output Indices”, where the output layer nodes of a neural network may be evaluated to determine distance values associated with the output indices (e.g., the requested distance values). For example, using neural network301, a subset of output layer nodes412of output layer406may be evaluated to determine the requested distance values. For example, requested distance values701may be determined by neural network301as implemented via distance values computation module204at operation905. Process900may continue at operation906, “Save Final Hidden Layer Values and Distance Values”, where final hidden layer values associated with a final hidden layer of a neural network and the neural network determined distance values may be saved for future use. For example, final hidden layer values determined via final hidden layer nodes411of neural network301may be saved via a memory for use in the approximation of subsequent distance values as is discussed further herein. Similarly, distance values determined via the activated subset of output layer406may be saved for use in the approximation of subsequent distance values. Such saved final hidden layer values and neural network determined distance values may be discarded when no longer needed (e.g., when a current frame may no longer call back to such an evaluation frame for the approximation of distance values).

If the current frame is a skip frame, process900may continue at operation907, “Retrieve Final Hidden Layer Values and/or Distance Values for Prior Frame(s)”, where final hidden layer values and/or prior neural network determined distance values (as saved at operation906) may be retrieved. For example, such values may be retrieved from memory via controller302or distance values approximation module303. For example, if an output index of the output indices for the current frame is associated with a previously neural network determined distance value, such previously neural network determined distance value may be retrieved. If an output index value of the output indices for the current fame is associated with a neural network node that was not previously determined, the final hidden layer values may be retrieved.

Process900may continue at operation908, “Determine Prior Distance Values, as Needed, via Associated Output Layer Nodes”, where prior distance values may be determined for output indices via associated output layer nodes. For example, for distance values that are currently needed for approximation, but were not previously determined via a neural network, such distance values may be determined via output layer nodes of the neural network. For example, final hidden layer values retrieved at operation907may be used to evaluate a subset of (e.g., one or more of) output layer nodes412of output layer406. In some examples, such distance values may have already been saved at operation906and operation908may be skipped.

Process900may continue at operation909, “Approximate Distance Values Associated with Output Indices based on Prior Distance Values”, where distance values may be approximated based on prior neural network calculated distance values. The approximated distance values may be those associated with the received output indices and the prior distance values may be saved via operation906or determined via operation908as discussed. For example, distance values may be approximated using linear extrapolation as discussed herein. In an example, requested distance values702may be determined by distance value approximation module303as implemented via distance values computation module204at operation909.

As shown, in either the case of an evaluation frame or a skip frame, process900may continue at operation910, “Provide Distance Values”, where distance values may be provided to a speech decoder for the determination of a sequence of textual elements as discussed herein. For example, distance values computation module204may provide distance values205(e.g., including requested distance values701or requested distance values702depending on the frame type) to speech decoder module206for the generation of recognized word sequence207.

Process900may continue at decision operation911, “Last Time Instance/Frame?”, where a determination may be made as to whether the current frame is a last frame. If the current frame is not the last frame, process900may continue at operation912, “Go to Next Time Instance/Frame”, where process900may continue at a next time instance for next frame at operation902as shown. If the current frame is the last frame, process900may end at ending operation913.

As discussed, distance values computation module204including neural network301, controller302, and distance values approximation module303may implement frame skipping and distances on demand to substantially reduce computational loads in automatic speech recognition implementations. In some examples, such distances on demand techniques may be implemented without the implementation of such frame skipping techniques. For example, with reference toFIG. 7, distance values computation module204may be implemented without distance values approximation module303and, for each time instance or frame, controller302may control neural network to provide only those distance values (e.g., requested distance values701) associated with output indices208. With reference toFIG. 9, such a distances on demand process may include operations902,904,905,910,911,912, and913such that output indices may be received, a neural network may be fully evaluated through a final hidden layer, only a subset of output layer nodes corresponding to the output indices may be evaluated, and the determined distance values (e.g., the subset) may be provided to a speech decoder. Such a process may not necessitate saving the final hidden layer values or distance values for future use for example nor the implementation of the branch of operations (e.g., operations903,907,908) providing for the integration of skip frame techniques.

Such frame skipping techniques may be considered approximation techniques as distance values are approximated via extrapolation or interpolation techniques as discussed. Such distances on demand techniques may be considered non-approximation techniques since the speech decoder is receiving only those distance values it needs for a current frame and (in the case of neural network determined distance values) such distance values are not approximated. Therefore, such distances on demand techniques may reduce computational costs without reducing accuracy. For example, in comparison to frame skipping only, the addition of distances on demand techniques may decrease computation by 22%. In some examples, the output layer may be about 50% of the neural network and evaluating only a requested subset of the output layer may save about 45% of the computational cost of the neural network. In various examples, evaluating only a requested subset of the output layer may save 0 to 50% of the computational cost of the neural network. Table 2 illustrates example results of the described frame skipping and distances on demand techniques.

Table 2 provides example results for a neural network having about 354,000 weights as discussed in the example having 253 input layer nodes, 4 hidden layers with 192 hidden layer nodes each, and 1,015 output layer nodes. Furthermore, the results were attained with the same speech recognition engine for all testing The number of words in the applied statistical language model was 1,000. The frame skipping rate was set to N=3 (e.g., module 3). The speech decoder was a beam width speech decoder provided with conservative settings for the best possible speech recognition performance. As shown, the overall (e.g., feature extraction, neural network and/or value approximation module, and speech decoder) system performance provided a real time factor (RTF) for brute force neural network computation (e.g., no frame skipping nor distances on demand) of 2.49 seconds to process 1 second of speech with a compute cost (e.g., the required number of central processing unit (CPU) cycles per second for the system to run in real time) of about 800 MPCS (mega cycles per second). Distances on demand alone reduced the real time factor by about 22% and frame skipping alone reduced the real time factor by about 64%. Applying distances on demand on top of frame skipping provided an additional reduction of about 22% and brought the overall system to a real time factor or 1.00. In the given example, the combination of frame skipping and distances on demand with linear extrapolation allows an automated speech recognition system to run in real time without loss of accuracy.

Furthermore, Pseudo-code (2) provides an example process for providing frame skipping with linear approximation combined with distances on demand.

As shown in Pseudo-code (1), for non-skipped frames (e.g., evaluation frames), the modulo may be 0 and requested distance values may be computed via the neural network (e.g., DNN). For skipped frames, prior needed distance values may be computed via the neural network and used for approximation via extrapolation (e.g., linear_extrapolate) of the current distance value.

As discussed, distance values computation module204including neural network301, controller302, and distance values approximation module303may implement frame skipping and/or distances on demand to substantially reduce computational loads in automatic speech recognition implementations with no or little reduction in speech recognition accuracy. Furthermore, such distance values computation module204including neural network301, controller302, and distance values approximation module303may be implemented via the same processing device (e.g., a central processing unit, graphics processing unit, signal processor, or the like) or various portions of the neural network may be implemented via different processing devices.

FIG. 10is an illustrative diagram of an example system1000for implementing frame skipping and/or distances on demand for generating distance values205from features203and output indices208, arranged in accordance with at least some implementations of the present disclosure. As shown inFIG. 10, system1000may include distance values computation module204having neural network301, controller302, and distance values approximation module303and memory stores1001configured to store neural network data, distance values (e.g. prior distance values relevant to a current frame), and hidden layer values (e.g., prior hidden layer values relevant to a current frame), or the like. In some examples, distance values computation module204may be implemented via a central processing unit or other processor as is discussed further herein. As discussed, in some examples, some or all of the modules distance values computation module204may be implemented via different processors.

As shown and as discussed elsewhere herein, distance values computation module204may receive features203(e.g., via feature extraction module202) and output indices208(e.g., via speech decoder module206). Distance values computation module204may also receive neural network weights, biases, and corrections (e.g., neural network data) via memory stores1001. Furthermore, distance values computation module204may receive prior distance values and/or prior hidden layer values via memory stores1001. For example, features203may provide inputs to an input layer of neural network301. Neural network301may be implemented via a node scoring module that may determine node scores for layers of the neural network, a score bias module that may bias such node scores to generated biased scores, and an output/activation function module that may generate outputs for the nodes based on the biased scores. For example, for hidden layer nodes and input layer nodes, the output/activation function module may implement an activation function to generate an output and for output layer nodes, the output/activation function module may provide the corrected biased scores as the node outputs. As discussed, in some examples, the output layer may be controlled in distances on demand implementations to only provide distance values associated with output indices208.

Furthermore, distance values approximation module303may receive neural network determined distance values either from memory stores1001or from neural network301and distance values approximation module303may, for example, extrapolate distance values based on the received neural network determined distance values. The distance values determined via neural network301and/or distance values approximation module303may be provided as distance values205as discussed herein.

FIG. 11is a flow diagram illustrating an example process1100for providing automatic speech recognition, arranged in accordance with at least some implementations of the present disclosure. Process1100may include one or more operations1101-1103as illustrated inFIG. 11. Process1100may form at least part of a computer-implemented method for providing automatic speech recognition. By way of non-limiting example, process1100may form at least part of an automatic speech recognition process for an attained speech recording such as speech recording201as undertaken by systems200or1000as discussed herein. Further, process1100will be described herein in reference to system1200ofFIG. 12.

FIG. 12is an illustrative diagram of an example system1200for providing speech recognition, arranged in accordance with at least some implementations of the present disclosure. As shown inFIG. 12, system1200may include one or more central processing units (CPU)1201, a graphics processing unit (GPU)1202, system memory1203, and microphone104. Also as shown, CPU1201may include feature extraction module202, distance values computation module204, and speech decoder module206. Furthermore, distance values computation module204may include neural network301, controller302, and distance values approximation module303. As shown, in the example of system1200, system memory1203may include memory stores1001, which may store neural network data, distance values, and/or hidden layer values. Furthermore, system memory1203may store any other data as discussed herein such as speech recordings, features, feature vectors, distance values, recognized word sequences, or the like. Microphone104may include any suitable device or devices that may receive speech103(e.g., as sound waves in the air, please refer toFIG. 1) and convert speech103to an electrical signal such as a digital signal. In an embodiment, microphone converts speech103to speech recording201. In an embodiment, speech recording201may be stored in system memory1203for access by CPU1201.

CPU1201and graphics processing unit1202may include any number and type of processing units that may provide the operations as discussed herein. Such operations may be implemented via software or hardware or a combination thereof. For example, graphics processing unit1202may include circuitry dedicated to manipulate data obtained from system memory1203or dedicated graphics memory (not shown). Furthermore, central processing units1201may include any number and type of processing units or modules that may provide control and other high level functions for system1200as well as the operations as discussed herein. System memory1203may be any type of memory such as volatile memory (e.g., Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), etc.) or non-volatile memory (e.g., flash memory, etc.), and so forth. In a non-limiting example, system memory1203may be implemented by cache memory. As shown, in an embodiment, feature extraction module202, distance values computation module204, and speech decoder module206may be implemented via CPU1201. In some examples, feature extraction module202, distance values computation module204, and speech decoder module206may be provided by software as implemented via CPU1201. In other examples, one or more of feature extraction module202, distance values computation module204, and speech decoder module206may be implemented via a digital signal processor or the like. In another embodiment, one or more of feature extraction module202, distance values computation module204, and speech decoder module206may be implemented via an execution unit (EU) of graphics processing unit1202. The EU may include, for example, programmable logic or circuitry such as a logic core or cores that may provide a wide array of programmable logic functions.

Returning to discussion ofFIG. 11, process1100may begin at operation1101, “Evaluate a Neural Network to Determine a First Distance Value as an Output of the Neural Network”, where a neural network may be evaluated to determine a first distance value as an output of the neural network. For example, the first distance value may be associated with a first time instance. For example, neural network301of distance values computation module204as implemented via CPU1201may determine a distance value as an output of neural network301.

In some examples, process1100may implement a frame skipping technique but not a distances on demand technique as discussed herein. In such examples, the neural network may comprise an output layer having multiple output layer nodes and evaluating the neural network at operation1101may include evaluating all of the output nodes of the neural network. In other examples, process1100may implement distances on demand techniques with or without frame skipping. In such examples, prior to operation1101, output indices may be generated for the first time instance (e.g., via a speech decoder). For example, the first distance value may be associated with an output index of the output indices. In such examples, the neural network may include an output layer having multiple output layer nodes such that evaluating the neural network for the first time instance includes evaluating a subset of the multiple output layer nodes such that the subset is associated with the output indices. As discussed, in such examples, a final hidden layer of the neural network having final hidden layer nodes may be fully evaluated for the first time instance and the final hidden layer node values may be saved.

Process1100may continue at operation1102, “Approximate a Second Distance Value for a Subsequent Time Instance based on the Neural Network Determined Distance Value”, where, for a second time instance subsequent to the first time instance, a second distance value may be approximated based at least in part on the first distance value and such that the neural network is not evaluated for the second time instance. For example, at the second time instance, no distance values may be direct outputs of the neural network. For example, distance values approximation module303of distance values computation module204as implemented via CPU1201may approximate the second distance value. The second distance value may approximated via extrapolation based on a linear function, a non-linear function, a variance function, or the like. In some examples, the approximation of the second distance value may be based on an extrapolation using the first distance value and a third distance value from a time instance preceding the time instance associated with the first distance value. In some examples, such an extrapolation based on the first and third distance value may be based on a linear extrapolation and the third distance value may have been previous determined via the neural network. In some examples, such an extrapolation may be provided as shown with respect to Equation (1) such that the linear extrapolation includes the first distance value added to half of a difference between the first distance value and the third distance value.

As discussed, in some examples, frame skipping and distances on demand may be implemented together. In such examples, prior to approximating the second distance value, output indices may be generated (e.g., via a speech decoder) for the second time instance. For example, the second distance value may be associated with an output index of the output indices. In such examples, evaluating the neural network at the first time instance may include evaluating all final hidden layer nodes of a final hidden layer to generate final hidden layer values, which may be saved. At the second time instance the neural network for the first time instance may be re-evaluated by evaluating the output layer nodes associated with the output indices for the second time instance based on saved hidden layer values for the first time instance to determine a neural network determined distance value that may be used to approximate the second distance value.

Process1100may continue at operation1103, “Determine a Sequence of Textual Elements based on the First and Second Distance Values”, where a sequence of textual elements may be determined based on the first and second distance values. For example, speech decoder module206as implemented via CPU1201may determine recognized word sequence207as discussed herein. In some examples, the speech decoder includes a Viterbi beam searching decoder.

As discussed, process1100may implement a frame skipping technique. Such a technique may skip any number of frames. For example, frames determined using a neural network may be described as neural network evaluation frames and frames determined using approximation techniques may be described as skip frames. In some examples, the first time instance may be associated with a neural network evaluation frame, the second time instance may be associated with a skip frame, and the evaluation frame and skip frame may be adjacent frames. In other examples, one, two, three, or more skip frames may be between the evaluation frame and the skip frame. Furthermore, in some examples, a frame skipping rate (e.g., based on accuracy needs, available computing resources, or a current real time factor) may be determined and implemented via controller302of distance values computation module204as implemented via CPU1201to provide additional or remove skip frames from between evaluation frames.

Process1100may be repeated any number of times either in series or in parallel for any number of time instances and/or speech recordings. Process1100may provide for determining distance values and generating a sequence of textual elements via a device such as device102as discussed herein or via a server as part of a cloud computing platform or the like. Also as discussed herein, prior to such processing in real-time, various components of the neural network may be pre-trained, biases and/or weights may be determined, or the like via, in some examples, a separate system. As discussed, in some examples, process1100may be implemented via CPU1201. In other examples, process1100(and the associated modules) may be implemented via a dedicated processor such as a co-processor or the like.

Furthermore, prior to operation1101, in some examples, received speech may be converted to a speech recording. For example, speech103may be converted to speech recording201via microphone104of system1100and/or related circuitry. Furthermore, features203(e.g., feature vectors) may be determined or extracted based on speech recording201by feature extraction module202as implemented via CPU1201and such features may be provided to neural network301of distance values computation module204as implemented via CPU1201. In some examples, feature extraction module202may be implemented via a digital signal processor (not shown) of system1200. In some examples, speech decoder module206as implemented via CPU1201may determine recognized word sequence207by comparing distance values205to statistical models (not shown) as attained via system memory1203.

FIG. 13is an illustrative diagram of an example system1300, arranged in accordance with at least some implementations of the present disclosure. In various implementations, system1300may be a mobile system although system1300is not limited to this context. For example, system1300may be incorporated into a personal computer (PC), laptop computer, ultra-laptop computer, tablet, touch pad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cellular telephone, combination cellular telephone/PDA, television, smart device (e.g., smart phone, smart tablet, smart watch, smart glasses or smart television), mobile internet device (MID), messaging device, data communication device, cameras (e.g. point-and-shoot cameras, super-zoom cameras, digital single-lens reflex (DSLR) cameras), and so forth.

In various implementations, system1300includes a platform1302coupled to a display1320. Platform1302may receive content from a content device such as content services device(s)1330or content delivery device(s)1340or other similar content sources. As shown, in some examples, system1300may include microphone104implemented via platform1302. Platform1302may receive speech such as speech103via microphone104as discussed herein. A navigation controller1350including one or more navigation features may be used to interact with, for example, platform1302and/or display1320. Each of these components is described in greater detail below.

Processor1310may be implemented as a Complex Instruction Set Computer (CISC) or Reduced Instruction Set Computer (RISC) processors, x86 instruction set compatible processors, multi-core, or any other microprocessor or central processing unit (CPU). In various implementations, processor1310may be dual-core processor(s), dual-core mobile processor(s), and so forth.

Graphics subsystem1315may perform processing of images such as still or video for display. Graphics subsystem1315may be a graphics processing unit (GPU) or a visual processing unit (VPU), for example. An analog or digital interface may be used to communicatively couple graphics subsystem1315and display1320. For example, the interface may be any of a High-Definition Multimedia Interface, DisplayPort, wireless HDMI, and/or wireless HD compliant techniques. Graphics subsystem1315may be integrated into processor1310or chipset1305. In some implementations, graphics subsystem1315may be a stand-alone device communicatively coupled to chipset1305.

In various implementations, display1320may include any television type monitor or display. Display1320may include, for example, a computer display screen, touch screen display, video monitor, television-like device, and/or a television. Display1320may be digital and/or analog. In various implementations, display1320may be a holographic display. Also, display1320may be a transparent surface that may receive a visual projection. Such projections may convey various forms of information, images, and/or objects. For example, such projections may be a visual overlay for a mobile augmented reality (MAR) application. Under the control of one or more software applications1316, platform1302may display user interface1322on display1320.

In various implementations, content services device(s)1330may be hosted by any national, international and/or independent service and thus accessible to platform1302via the Internet, for example. Content services device(s)1330may be coupled to platform1302and/or to display1320. Platform1302and/or content services device(s)1330may be coupled to a network1360to communicate (e.g., send and/or receive) media information to and from network1360. Content delivery device(s)1340also may be coupled to platform1302and/or to display1320.

In various implementations, platform1302may receive control signals from navigation controller1350having one or more navigation features. The navigation features of controller1350may be used to interact with user interface1322, for example. In various embodiments, navigation controller1350may be a pointing device that may be a computer hardware component (specifically, a human interface device) that allows a user to input spatial (e.g., continuous and multi-dimensional) data into a computer. Many systems such as graphical user interfaces (GUI), and televisions and monitors allow the user to control and provide data to the computer or television using physical gestures.

Movements of the navigation features of controller1350may be replicated on a display (e.g., display1320) by movements of a pointer, cursor, focus ring, or other visual indicators displayed on the display. For example, under the control of software applications1316, the navigation features located on navigation controller1350may be mapped to virtual navigation features displayed on user interface1322, for example. In various embodiments, controller1350may not be a separate component but may be integrated into platform1302and/or display1320. The present disclosure, however, is not limited to the elements or in the context shown or described herein.

In various implementations, drivers (not shown) may include technology to enable users to instantly turn on and off platform1302like a television with the touch of a button after initial boot-up, when enabled, for example. Program logic may allow platform1302to stream content to media adaptors or other content services device(s)1330or content delivery device(s)1340even when the platform is turned “off” In addition, chipset1305may include hardware and/or software support for 5.1 surround sound audio and/or high definition 7.1 surround sound audio, for example. Drivers may include a graphics driver for integrated graphics platforms. In various embodiments, the graphics driver may comprise a peripheral component interconnect (PCI) Express graphics card.

In various implementations, any one or more of the components shown in system1300may be integrated. For example, platform1302and content services device(s)1330may be integrated, or platform1302and content delivery device(s)1340may be integrated, or platform1302, content services device(s)1330, and content delivery device(s)1340may be integrated, for example. In various embodiments, platform1302and display1320may be an integrated unit. Display1320and content service device(s)1330may be integrated, or display1320and content delivery device(s)1340may be integrated, for example. These examples are not meant to limit the present disclosure.

As described above, system1300may be embodied in varying physical styles or form factors.FIG. 13illustrates implementations of a small form factor device1300in which system1300may be embodied. In various embodiments, for example, device1300may be implemented as a mobile computing device a having wireless capabilities. A mobile computing device may refer to any device having a processing system and a mobile power source or supply, such as one or more batteries, for example. In some examples, device1300may include a microphone (e.g., microphone104) and/or receive speech (e.g., speech103) for real time speech recognition via implementation of neural network as discussed herein.

As described above, examples of a mobile computing device may include a personal computer (PC), laptop computer, ultra-laptop computer, tablet, touch pad, portable computer, handheld computer, palmtop computer, personal digital assistant (PDA), cellular telephone, combination cellular telephone/PDA, television, smart device (e.g., smart phone, smart tablet or smart television), mobile internes device (MID), messaging device, data communication device, cameras (e.g. point-and-shoot cameras, super-zoom cameras, digital single-lens reflex (DSLR) cameras), and so forth.

As shown inFIG. 14, device1400may include a housing1402, a display1404, an input/output (I/O) device1406, and an antenna1408. Device1400also may include navigation features1412. Display1404may include any suitable display unit for displaying information appropriate for a mobile computing device. Display1404may include a touchscreen region1410for receiving I/O commands. In some examples, touchscreen region1410may be substantially the same size as display1404. I/O device1406may include any suitable I/O device for entering information into a mobile computing device. Examples for I/O device1406may include an alphanumeric keyboard, a numeric keypad, a touch pad, input keys, buttons, switches, rocker switches, microphones, speakers, voice recognition device and software, and so forth. Information also may be entered into device1400by way of microphone (not shown). Such information may be digitized by a voice recognition device (not shown). The embodiments are not limited in this context.

In one or more first embodiments, a computer-implemented method for providing automatic speech recognition comprises evaluating, for a first time instance, a neural network to determine at least one first distance value associated with the first time instance, wherein the at least one first distance value comprises an output from the neural network, approximating, for a second time instance, at least one second distance value based at least in part on the first distance value, wherein the neural network is not evaluated for the second time instance, and determining a sequence of textual elements based at least in part on the first distance value and the second distance value.

Further to the first embodiments, the method further comprises generating one or more output indices for the first time instance, wherein the first distance value is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes, and wherein evaluating the neural network for the first time instance comprises evaluating only a subset of the plurality of output layer nodes associated with the output indices.

Further to the first embodiments, the method further comprises generating one or more output indices for the first time instance, wherein the first distance value is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes, and wherein evaluating the neural network for the first time instance comprises evaluating only a subset of the plurality of output layer nodes associated with the output indices, wherein the neural network further comprises a final hidden layer having final hidden layer nodes, and wherein evaluating the neural network for the first time instance comprises evaluating all of the final hidden layer nodes.

Further to the first embodiments, the method further comprises generating one or more output indices for the first time instance, wherein the first distance value is associated with a first output index of the output indices, the neural network comprises an output layer having a plurality of output layer nodes, evaluating the neural network for the first time instance comprises evaluating only a subset of the plurality of output layer nodes associated with the output indices and/or wherein the neural network further comprises a final hidden layer having final hidden layer nodes and evaluating the neural network for the first time instance comprises evaluating all of the final hidden layer nodes.

Further to the first embodiments, approximating the second distance value comprises extrapolating the second distance value based at least in part on the first distance value based on at least one of a linear function, a non-linear function, or a variance function.

Further to the first embodiments, approximating the second distance value comprises extrapolating the second distance value based on the first distance value and a third distance value associated with a third time instance, wherein the first time instance is prior to the second time instance and the third time instance is prior to the first time instance.

Further to the first embodiments, approximating the second distance value comprises extrapolating the second distance value based on the first distance value and a third distance value associated with a third time instance, wherein the first time instance is prior to the second time instance and the third time instance is prior to the first time instance, wherein extrapolating the second distance value comprises extrapolating the second distance value via a linear function based on the first distance value and the third distance value, wherein the third distance value is determined based on an evaluation of the neural network.

Further to the first embodiments, approximating the second distance value comprises extrapolating the second distance value based on the first distance value and a third distance value associated with a third time instance, wherein the first time instance is prior to the second time instance and the third time instance is prior to the first time instance, wherein extrapolating the second distance value comprises extrapolating the second distance value via a linear function based on the first distance value and the third distance value, wherein the third distance value is determined based on an evaluation of the neural network, wherein the linear function comprises the first distance value added to half of a difference between the first distance value and the third distance value.

Further to the first embodiments, approximating the second distance value comprises extrapolating the second distance value based on the first distance value and a third distance value associated with a third time instance, the first time instance being prior to the second time instance and the third time instance being prior to the first time instance, and/or extrapolating the second distance value comprises extrapolating the second distance value via a linear function based on the first distance value and the third distance value, and/or the third distance value is determined based on an evaluation of the neural network.

Further to the first embodiments, the method further comprises generating one or more output indices for the second time instance, wherein a third distance value for the second time instance is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes and a final hidden layer having a plurality of final hidden layer nodes, and wherein said evaluating the neural network for the first time instance comprises evaluating all final hidden layer nodes of the final hidden layer to generate a plurality of final hidden layer values, determining, at the second time instance, a fourth distance value for the first time instance by evaluating an output layer node of the plurality of output layer nodes associated with the fourth distance value based on the final hidden layer values, and approximating, for the second time instance, the third distance value based at least in part on the fourth distance value.

Further to the first embodiments, the neural network comprises an output layer having a plurality of output layer nodes, and wherein evaluating the neural network for the first time instance comprises evaluating all of the plurality of output layer nodes.

Further to the first embodiments, the first time instance is associated with a neural network evaluation frame, the second time instance is associated with a skip frame, and wherein one, two, or three additional skip frames are between the evaluation frame and the skip frame.

Further to the first embodiments, the neural network comprises an output layer having a plurality of output layer nodes and evaluating the neural network for the first time instance comprises evaluating all of the plurality of output layer nodes, and/or the first time instance is associated with a neural network evaluation frame, the second time instance is associated with a skip frame, and wherein one, two, or three additional skip frames are between the evaluation frame and the skip frame, and/or determining the sequence of textual elements comprises determining the sequence of textual elements via a Viterbi beam searching decoder.

Further to the first embodiments, the first time instance is associated with a neural network evaluation frame, the second time instance is associated with a skip frame, and the method further comprises determining a frame skipping rate based on at least one of available computing resources or a current real time factor and providing an additional skip frame between the evaluation frame and the skip frame based on the frame skipping rate.

Further to the first embodiments, determining the sequence of textual elements comprises determining the sequence of textual elements via a Viterbi beam searching decoder.

Further to the first embodiments, the method further comprises converting received speech to a speech recording, extracting feature vectors associated with time windows of the speech recording, and providing the feature vectors as input to the neural network.

In one or more second embodiments, a system for providing a providing automatic speech recognition comprises a memory configured to store speech recognition data and a central processing unit coupled to the memory, wherein the central processing unit comprises neural network circuitry configured to implement, for a first time instance, a neural network to determine at least one first distance value associated with the first time instance, distance value approximation circuitry configured to approximate, for a second time instance, at least one second distance value based at least in part on the first distance value, and speech decoder circuitry configured to determine a sequence of textual elements based at least in part on the first distance value and the second distance value.

Further to the second embodiments, the speech decoder circuitry is further configured to generate one or more output indices for the first time instance, wherein the first distance value is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes, and wherein the neural network circuitry is configured to evaluate only a subset of the plurality of output layer nodes associated with the output indices for the first time instance.

Further to the second embodiments, the speech decoder circuitry is further configured to generate one or more output indices for the first time instance, wherein the first distance value is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes, and wherein the neural network circuitry is configured to evaluate only a subset of the plurality of output layer nodes associated with the output indices for the first time instance, wherein the neural network further comprises a final hidden layer having final hidden layer nodes, and wherein the neural network circuitry is configured to evaluate all of the final hidden layer nodes for the first time instance.

Further to the second embodiments, the speech decoder circuitry is further configured to generate one or more output indices for the first time instance, the first distance value being associated with a first output index of the output indices, the neural network comprising an output layer having a plurality of output layer nodes, and the neural network circuitry being configured to evaluate only a subset of the plurality of output layer nodes associated with the output indices for the first time instance, and/or the neural network further comprises a final hidden layer having final hidden layer nodes, the neural network circuitry being configured to evaluate all of the final hidden layer nodes for the first time instance.

Further to the second embodiments, the distance value approximation circuitry being configured to approximate the second distance value comprises the distance value approximation circuitry being configured to extrapolate the second distance value based on the first distance value and a third distance value associated with a third time instance, wherein the first time instance is prior to the second time instance and the third time instance is prior to the first time instance.

Further to the second embodiments, the distance value approximation circuitry is configured to extrapolate the second distance via a linear function based on the first distance value and the third distance value, wherein the neural network circuitry is configured to implement the neural network to determine the third distance value.

Further to the second embodiments, the distance value approximation circuitry being configured to approximate the second distance value comprises the distance value approximation circuitry being configured to extrapolate the second distance value via a linear function based on the first distance value and a third distance value associated with a third time instance, the first time instance being prior to the second time instance and the third time instance being prior to the first time instance, and/or the third distance value is determined based on an evaluation of the neural network.

Further to the second embodiments, the distance value approximation circuitry being configured to approximate the second distance value comprises the distance value approximation circuitry being configured to extrapolate the second distance value via a linear function based on the first distance value and a third distance value associated with a third time instance, wherein the first time instance is prior to the second time instance and the third time instance is prior to the first time instance, wherein the third distance value is determined based on an evaluation of the neural network.

Further to the second embodiments, the speech decoder circuitry is further configured to generate one or more output indices for the second time instance, wherein a third distance value for the second time instance is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes and a final hidden layer having a plurality of final hidden layer nodes, and wherein the neural network circuitry is configured to evaluate all final hidden layer nodes for the first time instance to generate a plurality of final hidden layer values, wherein the neural network circuitry is further configured to determine, at the second time instance, a fourth distance value for the first time instance by evaluating an output layer node of the plurality of output layer nodes associated with the fourth distance value based on the final hidden layer values, and wherein the distance value approximation circuitry is configured to approximate, for the second time instance, the third distance value based at least in part on the fourth distance value.

Further to the second embodiments, the neural network comprises an output layer having a plurality of output layer nodes, and wherein the neural network circuitry being configured to evaluate the neural network for the first time instance comprises the neural network circuitry being configured to evaluate all of the plurality of output layer nodes.

Further to the second embodiments, the first time instance is associated with a neural network evaluation frame, the second time instance is associated with a skip frame, and wherein one, two, or three additional skip frames are between the evaluation frame and the skip frame.

Further to the second embodiments, the speech decoder circuitry comprises a Viterbi beam searching decoder.

Further to the second embodiments, the system further comprises feature extraction circuitry configured to extract feature vectors associated with time windows of a speech recording and controller circuitry configured to determine a frame skipping rate based on at least one of available computing resources of the system or a current real time factor.

In one or more third embodiments, a system for providing a providing automatic speech recognition comprises means for evaluating, for a first time instance, a neural network to determine at least one first distance value associated with the first time instance, wherein the at least one first distance value comprises an output from the neural network, means for approximating, for a second time instance, at least one second distance value based at least in part on the first distance value, wherein the neural network is not evaluated for the second time instance, and means for determining a sequence of textual elements based at least in part on the first distance value and the second distance value.

Further to the third embodiments, the system further comprises means for generating one or more output indices for the first time instance, wherein the first distance value is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes and a final hidden layer having final hidden layer nodes, and wherein evaluating the neural network for the first time instance comprises evaluating only a subset of the plurality of output layer nodes associated with the output indices and evaluating all of the final hidden layer nodes.

Further to the third embodiments, approximating the second distance value comprises extrapolating the second distance value via a linear function based on the first distance value and a third distance value associated with a third time instance, wherein the first time instance is prior to the second time instance and the third time instance is prior to the first time instance, and wherein the third distance value is determined based on an evaluation of the neural network.

In one or more fourth embodiments, at least one machine readable medium comprises a plurality of instructions that, in response to being executed on a computing device, cause the computing device to provide automatic speech recognition by evaluating, for a first time instance, a neural network to determine at least one first distance value associated with the first time instance, wherein the at least one first distance value comprises an output from the neural network, approximating, for a second time instance, at least one second distance value based at least in part on the first distance value, wherein the neural network is not evaluated for the second time instance, and determining a sequence of textual elements based at least in part on the first distance value and the second distance value.

Further to the fourth embodiments, the machine readable medium further comprises instructions that, in response to being executed on the computing device, cause the computing device to perform speech recognition by generating one or more output indices for the first time instance, wherein the first distance value is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes, and wherein evaluating the neural network for the first time instance comprises evaluating only a subset of the plurality of output layer nodes associated with the output indices.

Further to the fourth embodiments, approximating the second distance value comprises extrapolating the second distance value via a linear function based on the first distance value and a third distance value associated with a third time instance, wherein the first time instance is prior to the second time instance and the third time instance is prior to the first time instance, wherein the third distance value is determined based on an evaluation of the neural network.

Further to the fourth embodiments, the machine readable medium further comprises instructions that, in response to being executed on the computing device, cause the computing device to perform speech recognition by generating one or more output indices for the second time instance, wherein a third distance value for the second time instance is associated with a first output index of the output indices, wherein the neural network comprises an output layer having a plurality of output layer nodes and a final hidden layer having a plurality of final hidden layer nodes, and wherein said evaluating the neural network for the first time instance comprises evaluating all final hidden layer nodes of the final hidden layer to generate a plurality of final hidden layer values, determining, at the second time instance, a fourth distance value for the first time instance by evaluating an output layer node of the plurality of output layer nodes associated with the fourth distance value based on the final hidden layer values, and approximating, for the second time instance, the third distance value based at least in part on the fourth distance value.

Further to the fourth embodiments, the machine readable medium further comprises instructions that, in response to being executed on the computing device, cause the computing device to perform speech recognition by converting received speech to a speech recording, extracting feature vectors associated with time windows of the speech recording, and providing the feature vectors as input to the neural network.

In one or more fifth embodiments, at least one machine readable medium may include a plurality of instructions that in response to being executed on a computing device, causes the computing device to perform a method according to any one of the above embodiments.

In one or more sixth embodiments, an apparatus may include means for performing a method according to any one of the above embodiments.