Patent Description:
In the following description, reference is made to the accompanying drawings that form a part hereof, and in which is shown by way of illustration specific embodiments which may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that structural, logical and electrical changes may be made without departing from the scope of the present invention. The following description of example embodiments is, therefore, not to be taken in a limited sense, and the scope of the present invention is defined by the appended claims.

The functions or algorithms described herein may be implemented in software in one embodiment. The software may consist of computer executable instructions stored on computer readable media or computer readable storage device such as one or more non-transitory memories or other type of hardware-based storage devices, either local or networked. Further, such functions correspond to modules, which may be software, hardware, firmware or any combination thereof. Multiple functions may be performed in one or more modules as desired, and the embodiments described are merely examples. The software may be executed on a digital signal processor, ASIC, microprocessor, or other type of processor operating on a computer system, such as a personal computer, server or other computer system, turning such computer system into a specifically programmed machine.

The functionality can be configured to perform an operation using, for instance, software, hardware, firmware, or the like. For example, the phrase "configured to" can refer to a logic circuit structure of a hardware element that is to implement the associated functionality. The phrase "configured to" can also refer to a logic circuit structure of a hardware element that is to implement the coding design of associated functionality of firmware or software. The term "module" refers to a structural element that can be implemented using any suitable hardware (e.g., a processor, among others), software (e.g., an application, among others), firmware, or any combination of hardware, software, and firmware. The term, "logic" encompasses any functionality for performing a task. For instance, each operation illustrated in the flowcharts corresponds to logic for performing that operation. An operation can be performed using, software, hardware, firmware, or the like. The terms, "component," "system," and the like may refer to computer-related entities, hardware, and software in execution, firmware, or combination thereof. A component may be a process running on a processor, an object, an executable, a program, a function, a subroutine, a computer, or a combination of software and hardware. The term, "processor," may refer to a hardware component, such as a processing unit of a computer system.

Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computing device to implement the disclosed subject matter. The term, "article of manufacture," as used herein is intended to encompass a computer program accessible from any computer-readable storage device or media. Computer-readable storage media can include, but are not limited to, magnetic storage devices, e.g., hard disk, floppy disk, magnetic strips, optical disk, compact disk (CD), digital versatile disk (DVD), smart cards, flash memory devices, among others. In contrast, computer-readable media, i.e., not storage media, may additionally include communication media such as transmission media for wireless signals and the like.

The goal of a speech application is to produce the highest possible accuracy given reasonable constraints in computing power and latency. Over the past years, speech researchers have developed a variety of algorithms and architectures to learn speech models, as well as, speech features robust to acoustic scenarios. Recently, deep long-short term memory (LSTM) approaches have been used to improve the performance of deep neural network (DNN) speech models. LSTM models explicitly control the memory of the network in terms of input and forget gate modules to provide a control over the information flow in the network and alleviate a gradient vanishing problem associated with deep networks. These and other newer advances in deep learning have provided many improvements in end-to-end speech processing systems.

Besides speech features and model structures, speech recognition (SR) systems also leverage techniques in model or speaker adaptation that personalize models for a specific scenario or speaker. These adaptation techniques provide significant value on top of speaker-independent (SI) models. In addition, acoustic model combination (AMOC) has also been used to improve various aspects of speech recognition performance.

The present inventive subject matter uses static and dynamic combination of acoustic model states to provide an improvement to many types and applications of ASR. A state-dependent set of weights are learned in a data-driven framework that aligns with ASR training criterion. These weights are dynamic, where the weights are obtained from a prediction model.

The accuracy of speech recognition is improved through combination of different acoustic models. State predictions from two different models are combined, element by element, with a different weight for each element. The weights are determined dynamically. This allows a lower confidence level for an element to receive less weight in the overall confidence level. This leads to better accuracy in speech recognition.

<FIG> is block diagram of a system <NUM> that combines two acoustic models, model <NUM> at <NUM> and model <NUM> at <NUM>. Note that many more acoustic models may be combined in further examples. Model <NUM> may be an acoustic model that has been trained to work very well for a broad range of acoustic scenarios. Model <NUM> may be referred to as a generic acoustic model. Model <NUM> may be a dedicated model that is specifically trained for a far-field speaker scenario, or another scenario. Features <NUM>, such as speech are provided to the two models. The features may include different kinds of input, such as normal speech, far field speech, or other kinds of speech. recognition example, which is not in the claim scope, the feature may include input corresponding to daylight images or images taken at night, or in the rain. Note that the generic model may be trained on a training data set that includes many if not all the different kinds of speech, while dedicated models may be trained on one or a few of the different kinds of speech.

The two models <NUM> and <NUM> are combined at <NUM> such that the single combined model shows strong gains for each of the scenarios over the previous best results in that scenario. In some embodiments, more than two models, such as N models may be combined. The combination is performed based on state dependent weights <NUM>, which have been derived based on the different kinds of input expected. The weights for each state are independently derived, such as by a deep learning model. In speech recognition, the states may comprise senones, speech segments, or other sounds, while in image recognition, outside the claim scope the states may comprise sets of pixels representing lines or other combinations of pixels. The ability to combine models at the state level can be applied to many other practical applications by identifying different conditions under which input is collected and training different models for such different conditions.

In further examples a diverse number of dedicated models for different speech and noise scenarios may be used. The ability to combine such dedicated models can be helpful for server applications, where robust ASR performance may be obtained despite highly varied acoustic application environments, audio processing pipelines, and speaker bases, etc. With a single combined model, the diverse acoustic scenarios and applications may be served without the need for additional modifications.

State combination of the acoustic models is an effective technique in the broad scope of model combination. LSTM-RNN models consist of a few layers of LSTM cells along with a top layer that includes a function to receive an input vector of K real numbers and normalize the input function into a probability distribution consisting of K probabilities. The top layer may be referred to as a SoftMax layer. Context-dependent tied tri-phones, a class of sounds in context, constitute the acoustic states in one example. The output of the combined model provides a predication distribution over the states for a given frame of speech features.

Combining the models at the state level is equivalent to combining the predicted distributions for respective models, and has been applied before. However, most previous work uses a fixed state-independent weight for model combination, where the weight is tuned on the task of interest. In contrast, the present inventive subject matter utilizes state dependent weights by analyzing the combination weights, ingesting new capabilities in the state combination framework. In particular, a data-driven framework learns the state combination weights, and ASR criterion is used to learn the model weights to be best aligned with ASR performance. State-dependent capabilities are utilized in the combination weights. Static as well as dynamic prediction frameworks are also used for the weights.

In a further example, the combination weights in the framework of the ASR itself may be used. A data-driven framework is used for learning the combination weights for both static state prediction (SSP) and dynamic state prediction (DSP) approaches.

<FIG> is a flowchart illustrating a computer implemented method <NUM> for combining model states utilizing state dependent combination weights. Multiple different kinds of related input is classified by method <NUM>. The input is related in that it is of the same general type, such as speech, or images, or other types of input, however, the kind of speech or images are different as previously described. Multiple operations are described at a general level, with algorithm details described further below.

Method <NUM> starts by obtaining a set of features from an input at operation <NUM>. The input in one example is speech, such as the phrase: "set an alarm. " The features may be obtained in a known manner and comprise digital representations of speech. At operation <NUM>, the features are provided to multiple speech recognition models that have been trained on different kinds of speech and/or each of the multiple models may have different structures or modeling technologies. The different kinds of speech may include two or more of speech in a noisy environment, native speech, non-native speech, child speech, whispered speech, natural conversation speech, and distant speech, as well as other kinds of speech, which may vary for different applications. The models may be deep learning LSTM acoustic models in one embodiment. Some types of models may be used in further embodiments, such as Bidirectional LSTM, generic Recurrent Neural Networks (RNN), Convolutional Neural Networks (CNN) or other Feed-forward acoustic models.

Two different models with different structure or modeling technologies may be trained on the same kind of speech, or on different kinds of speech. The models generate state predictions based on the features in accordance with their specific training.

At operation <NUM>, state dependent predicted weights are obtained. The predicted weights may be generated by a trained deep learning network, which is trained on a training set of state labeled features. Weights may be time dependent static weights or may be provided by a prediction cell based on the current features.

The state predictions from the models are combined based on the state dependent predicted weights for classification of the set of features at operation <NUM>. An utterance from a user desiring to set an alarm may take the form of "set an alarm. " A prediction of the input - "set an alarm" - is then provided at operation <NUM>. Note that the prediction may take any form, such as the actual spoken words, or even a code corresponding to the spoken words, such as a command for input into an application for setting an alarm.

Referring again to system <NUM>, the extracted features <NUM> are provided to the different models <NUM> and <NUM> that have been trained for generating states based on different kinds of input. State dependent weights <NUM> may utilize a small prediction model, referred to as a combination cell that dynamically evaluates the model combination weight αt[k] for time instant t and acoustic state k. A time-independent α[k] may also be trained for static state prediction such that a prediction model need not be used.

The SSP approach uses time-independent combination weights α[k]. For SSP, the combined model states are represented in Equation <NUM>: <MAT>.

In Equation <NUM>, the state combination weights are α[k] with k indicating an acoustic state. Equation <NUM> is state dependent with dimension as the number of states in the acoustic model, and Si and S<NUM> are the state predictions from the two acoustic models. If N acoustic models are to be combined, the weight for the states of each of the separate models will add up to <NUM>.

All weights α[k] are between <NUM> and <NUM>, and may be initialized to a fixed value, such as i. Standard ASR training criterion may be used to train state-dependent combination weights. In a further example, α[k] may be restrained to be identical for all states but is sub-optimal compared state-dependent α[k]. The choice of the value of initialization parameter i likely depends on the application scenario, and the nature of the models. In one application, the initialization weight, i = <NUM> is used and works well as an initialized value for the weights. Equation <NUM>, referred to as, combination, may be applied to the Softmax S<NUM> and S<NUM> state predictions by models <NUM> and <NUM> respectively.

The state-independent combination weights, i.e. with identical α[k] for all states, is a special case of the state-dependent weights. Beginning from the general case of state-dependent model combination in Equation <NUM>, a special case exists where for a particular speech frame, only one of the states, say k, is dominant for both the models, and rest of states, i.e., S<NUM>[j] and S<NUM>[j] are either <NUM> or significantly small for j ≠ k. In that restricted special case, Equation <NUM> is equivalent to: <MAT> Where <IMG> indicates element-wise product. The predicted weights α is identically α[k] for all states k in above state-independent combination approach.

In comparison to the baseline combination with identical α for all acoustic states, SSP offers additional advantages. SSP training aligns with the ASR training objectives to learn state-dependent combination weights. This allows SSP to best leverage the state classification boundaries from individual models. The training criterion converges to a state-dependent α. The predicted α for states like "sil", "noise" strongly favor the generic model.

The generic model, model <NUM> at <NUM> may be trained using a large corpus including mobile and close-talking data. Thus, the generic model learns the classification for silence and noise. In comparison, the dedicated model training in one acoustic based example may predominantly consist of far-field and noisy data, where the classification boundaries for silence and noise are fuzzy. Overall, SSP learns a way to best leverage the classification boundaries from the individual models.

In one embodiment, dynamic state prediction (DSP) is used to generate dynamic state dependent combination weights. In SSP, SR training criterion is leveraged to train state-dependent combination weights. Audio from different acoustic conditions exhibit different characteristics, therefore, static combination weights are likely sub-optimal.

<FIG> is a block diagram of a framework <NUM> for dynamic acoustic model combination using SSP. Framework <NUM> is described in terms of a speech recognition application, but is also applicable to other applications, such as image recognition outside the claim scope. Audio <NUM> is provided to a feature extraction mechanism <NUM> to extract features from speech. The speech may be received from a microphone, or received in digital form from a recording.

The extracted features from mechanism <NUM> is provided to two different models <NUM> and <NUM> that have been trained for generating states based on different kinds of input. The features are also provided to a weight prediction model <NUM> that operates to predict weights for combining the states as indicated at a combining mechanism <NUM>. The combined states are used to provide predictions using a small prediction model based on the combination of states. The predictions are provided to a speech decoder <NUM>, which outputs the predicted speech at <NUM>.

Scenario-dependent combination may be leveraged by dynamically predicting time and state-dependent combination weights αt[k] in: <MAT>.

A prediction model is used to predict αt[k] at time instant t and acoustic state k. A variety of prediction models may be used in the DSP framework. In one embodiment, the acoustic model consists of LSTM cells, making it logical to use a one-layer LSTM cell to model and predict αt[k]. The prediction model aligns well with the core ASR models; SR features and ASR training criterion may be reused to predict αt[k]. <FIG> illustrates a framework <NUM> that is an extension of the DSP approach. Features <NUM> are provided to model <NUM><NUM> and model <NUM><NUM>. Hidden layer outputs H1t and H2t respectively from the models are provided to a concatenation function <NUM>. The concatenated hidden layer outputs are provided to the weight prediction model <NUM>, with the weights provided to a combiner <NUM> to produce the prediction that is provided to speech decoder <NUM>, which outputs the predicted speech at <NUM>. Framework <NUM> is based on an understanding that in a deep network, the initial layers normalize the features and make it robust across speakers and acoustic environments. Whereas, the upper layers gradually learn decision boundaries. Framework <NUM> allows incorporation of some information from the individual ASR models.

<FIG> is a block schematic diagram of a computer system <NUM> to implement one or more methods of combining states of multiple models for providing predictions corresponding to different types of input according to example embodiments. All components need not be used in various embodiments.

One example computing device in the form of a computer <NUM> may include a processing unit <NUM>, memory <NUM>, removable storage <NUM>, and non-removable storage <NUM>. Although the example computing device is illustrated and described as computer <NUM>, the computing device may be in different forms in different embodiments. For example, the computing device may instead be a smartphone, a tablet, smartwatch, smart storage device (SSD), or other computing device including the same or similar elements as illustrated and described with regard to <FIG>. Devices, such as smartphones, tablets, and smartwatches, are generally collectively referred to as mobile devices or user equipment.

Although the various data storage elements are illustrated as part of the computer <NUM>, the storage may also or alternatively include cloud-based storage accessible via a network, such as the Internet or server based storage. Note also that an SSD may include a processor on which the parser may be run, allowing transfer of parsed, filtered data through I/O channels between the SSD and main memory.

Memory <NUM> may include volatile memory <NUM> and non-volatile memory <NUM>. Computer <NUM> may include - or have access to a computing environment that includes - a variety of computer-readable media, such as volatile memory <NUM> and non-volatile memory <NUM>, removable storage <NUM> and non-removable storage <NUM>. Computer storage includes random access memory (RAM), read only memory (ROM), erasable programmable read-only memory (EPROM) or electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions.

Computer <NUM> may include or have access to a computing environment that includes input interface <NUM>, output interface <NUM>, and a communication interface <NUM>. Output interface <NUM> may include a display device, such as a touchscreen, that also may serve as an input device. The input interface <NUM> may include one or more of a touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated within or coupled via wired or wireless data connections to the computer <NUM>, and other input devices. The computer may operate in a networked environment using a communication connection to connect to one or more remote computers, such as database servers. The remote computer may include a personal computer (PC), server, router, network PC, a peer device or other common data flow network switch, or the like. The communication connection may include a Local Area Network (LAN), a Wide Area Network (WAN), cellular, Wi-Fi, Bluetooth, or other networks. According to one embodiment, the various components of computer <NUM> are connected with a system bus <NUM>.

Claim 1:
A computer implemented method (<NUM>) for classification of input corresponding to multiple different kinds of input, the method comprising:
obtaining (<NUM>, <NUM>) a set of features from the input, wherein the input comprises speech (<NUM>) and the set of features comprise digital representations of speech;
providing (<NUM>) the set of features to multiple different speech recognition models (<NUM>, <NUM>), that have been trained on different kinds of input, or have different structures or modeling technologies, to generate state predictions, wherein the speech recognition models comprise hidden layer outputs;
generating (<NUM>, <NUM>), using a weight prediction model (<NUM>, <NUM>) a set of state-dependent predicted weights from the hidden layer outputs; and
combining (<NUM>, <NUM>) the state predictions from the multiple speech recognition models, based on the state-dependent predicted weights for classification of the set of features;
providing the combined state predictions to a speech decoder (<NUM>) to classify the input as one or more spoken words;
wherein the set of state dependent predicted weights are dynamic weights;
wherein the hidden layer outputs are concatenated hidden layer outputs that are generated by providing the hidden layer outputs from the different speech recognition models to a concatenation function.