Patent Publication Number: US-10762417-B2

Title: Efficient connectionist temporal classification for binary classification

Description:
CROSS-REFERENCE TO RELATED APPLICATIONS 
     This application claims the benefit of and priority to U.S. Provisional Patent Application No. 62/457,678 filed Feb. 10, 2017 and entitled “EFFICIENT CONNECTIONIST TEMPORAL CLASSIFICATION (CTC) ALGORITHM FOR BINARY CLASSIFICATION” which is incorporated herein by reference in its entirety. 
    
    
     TECHNICAL FIELD 
     The present disclosure generally relates to information classification, and more particularly, for example, to binary classification systems and methods for use in speech recognition systems. 
     BACKGROUND 
     Binary classification is the task of classifying the elements of a given set into two groups on the basis of a classification rule. There are several applications for binary classification, such as medical testing indicating whether or not a patient has a certain disease, quality control in factories where the product is tested to determine whether or not it meets a product specification, and a keyword spotting system where a system is designed to detect a voice command. 
     In many binary classification applications, the overall accuracy of the classification result may not be as important as the relative proportion of different error types. For example, in medical application a false positive (detecting a disease when it is not present) is considered differently from a false negative (not detecting a disease when it is present). There are several binary classification methods that may be used to solve this type of problem including decision trees, Bayesian networks, support vector machines, neural networks, prohibit regression, and logistic regression. 
     Among different methods, neural networks have been widely used because performance is sufficient to solve a wide variety of tasks that are hard to solve using the other methods. However, such approaches are often highly complex and may not be practical for use in many low power devices such as mobile phones. Efforts to reduce complexity often come at the cost of less flexibility, memory inefficiencies, and other undesirable performance measures. Thus, there is a need in the art for solutions to optimize binary classification systems that are both fast and resource efficient. 
     SUMMARY 
     The present disclosure provides systems and methods that address a need in the art for improved binary classification. In various embodiments, a computer-implemented method for training a neural network includes receiving, at a computing device, a stream of segmented, labeled training data having a sequence of frames, computing, by the computing device, a stream of input features data for the sequence of frames, generating, by the computing device, neural network outputs for the sequence of frames in a forward pass through the training data and in accordance weights and biases, and updating, by the computer device, weights and biases in a backward pass through the training data. The backward pass includes determining Region of Target (ROT) information from the stream of segmented, labeled training data, computing modified forward and backward variables based on the neural network outputs and the ROT information, deriving a signal error for each frame within the sequence of frames based on the modified forward and backward variables, and updating the weights and biases based on the derived signal error. The method may further include adaptively learning to improve a convergence rate of the neural network by determining a momentum for the weights and biases of an output layer, and computing the root mean square of gradients for the weights and biases not in the output layer. 
     In some embodiments, the neural network includes a recurrent neural network, such as a Long Short-Term Memory (LSTM) network, and the method includes storing the neural network outputs for the sequence of frames in a buffer. The backward pass may further include starting from a last frame in the sequence of frames and going backward in time to compute the signal error for each frame at the input of a softmax node. In some embodiments the method further includes propagating the signal error backwards through the neural network using back propagation through time. 
     In some embodiments, each sequence of frames includes a ROT, determining the ROT information further comprises determining a beginning of the ROT and determining an end of the ROT, and ROT information is used to compute the signal error. The signal error may be computed based on network outputs and an estimated soft target. 
     In various embodiments, a classification training system includes a neural network operable to classify input data, a training dataset providing segmented labeled training data comprising a sequence of frames, and a classification training module operable to train the neural network using the segmented labeled training data, the classification training module comprising a forward pass processing module and a backward pass processing module. The forward pass processing module may be operable to train the neural network by generating neural network outputs for the training data using weights and biases for the neural network. The neural network may include a recurrent neural network, such as an LSTM network. The forward pass processing module may be further operable to store the neural network outputs for the frames in the sequence of frames in a buffer. An adaptive learning module may be provided to improve a convergence rate of the neural network by determining a momentum for the weights and biases of an output layer, and computing the root mean square of gradients for the weights and biases not in the output layer. 
     In one or more embodiments, the backward pass processing module is operable to train the neural network by updating the weights and biases by passing backward through generated neural network outputs, the backward pass processing module operable to determine Region of Target (ROT) information from the segmented, labeled training data, compute modified forward and backward variables based on the neural network outputs and the ROT information, derive a signal error for each frame within the sequence of frames based on the modified forward and backward variables, and update the weights and biases based on the derived signal error. The backward pass processing module may further be operable to start from a last frame in the sequence of frames and going backward in time to compute the signal error for each frame at an input of a softmax node, and propagate the signal error backwards through the neural network using back propagation through time. 
     In some embodiments, each sequence of frames includes a ROT, and the backward pass processing module is further operable to determine a beginning of the ROT and an end of the ROT, and use the ROT information to compute the signal error. The signal error may be computed based on network outputs and an estimated soft target. 
     The scope of the disclosure is defined by the claims, which are incorporated into this section by reference. A more complete understanding of embodiments of the present disclosure will be afforded to those skilled in the art, as well as a realization of additional advantages thereof, by a consideration of the following detailed description of one or more embodiments. Reference will be made to the appended sheets of drawings that will first be described briefly. 
    
    
     
       BRIEF DESCRIPTION OF THE DRAWINGS 
       Aspects of the disclosure and their advantages can be better understood with reference to the following drawings and the detailed description that follows. The included drawings are for illustrative purposes and serve only to provide examples of possible systems and methods for the disclosed methods and systems. These drawings in no way limit any changes in form and detail that may be made to that which is disclosed by one skilled in the art without departing from the spirit and scope of this disclosure. 
         FIG. 1  illustrates an exemplary binary classification system for performing a forward pass on training data to train an artificial neural network, in accordance with an embodiment. 
         FIG. 2  illustrates an exemplary binary classification system for performing a backward pass on training data to train an artificial neural network, in accordance with an embodiment. 
         FIG. 3  is a flow diagram illustrating an exemplary operation of a binary classification training process, in accordance with an embodiment. 
         FIG. 4  illustrates an exemplary sequence of input data separated into multiple frames, in accordance with an embodiment. 
         FIG. 5  illustrates an exemplary application of a training process on a sequence of input data, in accordance with an embodiment. 
         FIG. 6  is a block diagram illustrating a binary classification training system, in accordance with an embodiment. 
         FIG. 7  is a block diagram illustrating a binary classification system, in accordance with an embodiment. 
     
    
    
     DETAILED DESCRIPTION 
     In accordance with various embodiments of the present disclosure, systems and methods for binary classification are disclosed. 
     A common solution for the task of binary classification such as keyword spotting is to use Hidden Markov Models (HMM) (e.g., see the publication titled “An application of recurrent neural networks to discriminative keyword spotting” by Santiago Fernandez, Alex Graves and Jurgen Schmidhuber, in Proceedings of ICANN. Porto. pp 220-229 (2007), which is incorporated by reference herein in its entirety). A major difficulty with HMM based systems is that they are forced to model the garbage (i.e. non-keyword) parts of the signal as well as the keywords themselves. Further disadvantages of HMM modeling are the suboptimal convergence of the Expectation Maximization (EM) algorithm to local maxima, the assumption of conditional independence of the observations, and the fact that HMMs do not directly maximize the keyword detection rate. 
     Among the recurrent neural networks, Long Short-Term Memory (LSTM) or bidirectional LSTM (BLSTM) networks are a promising technique for improving context modeling to build a better keyword spotting system. The methods using LSTM networks can be divided in two categories. The first category uses the LSTM network to provide higher level features for a classifier like Dynamic Bayesian Network (DBN) to perform keyword spotting (see, e.g., the publication titled “Improving Keyword Spotting with a Tandem BLSTM-DBN Architecture,” by M. Wollmer, F. Eyben, A. Graves, B. Schuller and G. Rigoll, in Non-Linear Speech Processing, J. Sole-Casals and V. Zaiats (Eds.), LNAI 5933, pp. 68-75, Springer Heidelberg, 2010; the publication titled “Bidirectional LSTM networks for context-sensitive keyword detection in a cognitive virtual agent framework” by M. Woellmer, F. Eyben, A. Graves, B. Schuller, and G. Rigoll, in Cognitive Computation, Special Issue on Non-Linear and Non-Conventional Speech Processing, 2010; and the publication titled “Spoken term detection with connectionist temporal classification—a novel hybrid CTC-DBN approach” by M. Woellmer, F. Eyben, B. Schuller, and G. Rigoll, in Proc. of ICASSP, Dallas, Tex., 2010, pp. 5274-5277, which are incorporated by reference herein in their entirety). These approaches combine the flexibility of the classifier like DBN architecture with the ability of a BLSTM network to capture long-range time dependencies. This category of methods has the flexibility of defining new keywords without retraining the network and thereby does not require excessive amounts of training data for large vocabularies. However, such methods have high complexity due to the large network (e.g., a BLSTM network) which provides higher level features such as phoneme recognitions for a keyword spotting task. Therefore, these methods are not practical for use in low power devices such as mobile phones. 
     The second category of methods using LSTM networks exploit LSTM in combination with Connectionist Temporal Classification (CTC) which is a discriminative system based on recurrent neural networks that use information from long time spans to estimate keyword probabilities. In a keyword spotting task, for example, the method attempts to model full keywords in the sequence data stream, while the first category systems are based on sub-word units (phonemes) (see, e.g., the publication titled “An application of recurrent neural networks to discriminative keyword spotting” by Santiago Fernandez, Alex Graves and Jurgen Schmidhuber, in: Proceedings of ICANN. Porto. pp 220-229 (2007) and the publication titled “Keyword spotting exploiting Long Short-Term Memory” by M. Wollmer, B. Schuller, and G. Rigoll, Speech Communication, 2012, which are incorporated by reference herein in their entirety). Unlike the methods in the first category, these methods are not flexible to new keyword selection in a keyword spotting task and require sufficient keyword samples in the training data. The conventional solution using LSTM based on CTC has some limitations such as unnecessary high computational complexity for binary classification and the method is not memory efficient. 
     For many binary classifications such as keyword spotting, the CTC training is more appropriate than the frame-wise training. First, the trained system is ideally expected to have only one or more spikes anywhere during the keyword duration indicating the detection of the keyword. So the system does not need to produce spikes at all the frames of the keyword. This leads to better system training that provides better performance as it is shown in the publication “An application of recurrent neural networks to discriminative keyword spotting” by Santiago Fernandez, Alex Graves and Jurgen Schmidhuber, which is incorporated by reference herein in its entirety. A CTC network may be designed to fulfill this by maximizing the probability of having correct labeling over all possible labeling. On the other hand, frame-wise training may be designed to force the network to produce spikes at all the frames of the keyword. 
     Second, frame-wise training is often very sensitive to the alignment of labeled training data. This means that the training data should be correctly labeled consistently across all data. However this requirement is not essential in CTC training which doesn&#39;t require pre-segmented training data. Finally, a network using a frame-wise algorithm is typically trained to make a series of independent label classifications. This implies that the network output is post processed to give the final label sequence. 
     Although CTC training has several advantages over frame-wise training especially for binary classification, the conventional CTC training algorithm is not computationally efficient and it has some problems and limitations such as an underflow problem for the computation and it can have undesirable latency. It is contemplated that the embodiments described herein present a better solution to train an artificial neural network, such as an LSTM network, for binary classification that overcomes the above identified limitations and that provides better performance. One difference between the embodiments presented herein and the conventional solution for training a CTC network is that pre-segmented labeled data is used. This information is used to propose a more computational efficient algorithm for training and it can also lead to better performance. 
     The embodiments described herein have several advantages over conventional methods utilizing LSTM for binary classification. First, the CTC network such as the one proposed in the publication titled “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets” by Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J, computes two variables, namely the forward and backward variables, recursively for the entire input sequence. This increases the computational complexity of the method because of the high number of multiplications with small valued numbers ranging between zero and one. The embodiments described herein simplify the computation of these variables and are well designed for the application of binary classification to train an LSTM network efficiently. 
     Second, in a binary classification such as a keyword spotting task, it is desirable to build a system to predict that an occurrence of a keyword exists towards the ending of the occurrence of the keyword. Otherwise, the system can have high false-detections for the case of non-keywords containing part of the keyword (e.g. the beginning of a non-keyword is the same as the beginning of the keyword). Conventionally trained CTC networks, such as the one proposed in the publication “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets” by Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J. cannot guarantee that the trained system will predict the keyword towards the ending of its occurrence since the network is forced to maximize the correct output labeling regardless of the alignment. In contrast, the embodiments described herein do not have this limitation; the trained network using the methods described herein can predict the true keyword towards the ending of the keyword. 
     Third, existing methods utilizing a CTC network have high memory requirements during training since they operate by obtaining the network&#39;s output for the entire input sequence as well as recursively computing the forward and backward variables of the same length of input sequence. For the task of keyword spotting, for example, when the input sequence is very long, the conventional methods, such as the methods proposed in the publication titled “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets” by Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J., are not memory efficient. In contrast, the training methods of some embodiments described herein are memory efficient; in one embodiment, the training method updates the network frame-wise for non-keyword parts of the input sequence and it concentrates on a particular region, which is called a Region of Target (ROT), that can include the tail of the keyword. 
     Referring to  FIGS. 1 and 2 , an embodiment of a general structure of the proposed system and process to train a recurrent neural network  102 , such as an LSTM network, for binary classification will now be described.  FIG. 1  illustrates the general structure of the binary classification system of some embodiments during an initial forward pass process while  FIG. 2  illustrates the general structure of the binary classification system of some embodiments during a subsequent backward pass process.  FIG. 3  illustrates the training process  300  performed by the binary classification system of some embodiments using an artificial neural network, such as an LTSM network. 
     The training procedure will be first briefly discussed below by reference to  FIGS. 1-3 . The procedure will then be described in more detail in the subsections below. The training process  300  includes a forward pass process  100  and a backward pass process  200 . 
     In the forward pass process, a stream of input features  108  having a sequence of T frames is received (at step  305  of  FIG. 3 ). The stream of input can include audio input received from an audio inputting device (e.g., a microphone) over a period of time. Optionally, the process  300  may slice the input into a sequence of frames (e.g., a sequence of frames  108  of  FIG. 1  having T number of frames in the order of receipt), with each frame having the same duration. Next, the input features for each frame of input are calculated at step  305  of  FIG. 3 . The process  300  then uses a neural network to compute (at step  310  of  FIG. 3 ) the network outputs for all the two nodes (y t   g ,y t   k =1−y t   g ) at t-th frame of a sequence of training data (e.g., the process  300  computes a network output utilizing an existing artificial neural network for each one of the T frames). In one embodiment, the binary classification system is configured to classify 2 classes, and the neural network includes 2 nodes at the output layer. The process may compute the network output in an order that is consistent with the sequence of the frames. In some embodiments, the network outputs for all the frames within the sequence are stored in a buffer  105 . 
     The backward pass process of some embodiments starts from the last frame and propagates backwards in time to compute a signal error for each frame at the input of a softmax node. Then, the backward pass process propagates this error backwards through the unfolded net, for example, by using the standard Back Propagation Through Time (BPTT) equations (see, e.g., the publication titled “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets” by Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J., which is incorporated by reference in its entirety as if fully set forth herein). 
     In the illustrated embodiment, the backward pass process has several steps as follows. Initially, for each sequence of data (e.g., a sequence of frames), the process  300  identifies (at step  315  of  FIG. 3 ) a Region Of Target (ROT) within the sequence of input frames and derives associated information of the ROT (see also, reference  210 ,  FIG. 2 ). The beginning and the end of this ROT is used to compute the signal error. Using the ROT information, the process  300  computes (at step  320  of  FIG. 3 ) forward and backward variables for the sequence of input data. 
     Then, the process  300  (at step  325  of  FIG. 3 ) modifies the forward and backward variables to generate modified forward and backward variables. Next, the process  300  (at step  330  of  FIG. 3 ) computes a signal error based on the network outputs and the estimated soft target (Y t   g  in Formula (5) as described below). At step  335  of  FIG. 3 , an adaptive learning algorithm may be used to update the weights and biases of the neural network. 
     A conventional CTC network proposed in the publication titled “Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural nets” by Graves, A., Fernandez, S., Gomez, F., Schmidhuber, J. includes a softmax output layer with one more output node than there are labels. If the number of labels is M, then the activations of the first M output nodes are interpreted as the probabilities of observing the corresponding labels at particular times. The activation of the extra node is the probability of observing a ‘blank’, or no label. The network outputs define the probabilities of all possible ways of aligning all possible label sequences with the input sequence. The total probability of any one label sequence can then be found by summing the probabilities of its different alignments. The goal of the CTC algorithm is to maximize the probabilities of the correct labeling. However, for applications such as keyword spotting where there are only two classes (keyword and garbage), such a CTC algorithm is not efficient because the CTC network would be required to have three nodes including the extra blank node and the forward and backward variables are required to be computed for the entire training sequence. As a result, the aforementioned method is not memory and computationally efficient. 
     In some embodiments disclosed herein, a binary classification system using an improved CTC algorithm is contemplated. An example for a keyword spotting application is illustrated in  FIG. 4  and is used below for the purpose of illustrating this approach, and does not limit the application of this binary classification system.  FIG. 4  illustrates an example input in the form of an audio waveform  405 . The audio waveform  405  is a representation of an audio input across a period of time frame, which can be obtained via an audio input device such as one or more microphone. Upon receiving the audio waveform  405 , the binary classification system of some embodiments divides the audio waveform  405  into a sequence of audio frames  410  (e.g., T number of frames in order). Preferably, each frame in the sequence of audio frames includes substantially equal time duration. In this example, we have a keyword from Frame t=r to Frame t=r+N−1 and the other part (the other frames) of this sequence is not the keyword (it is garbage). The keyword region is known as the Region Of Target (ROT), as it is shown in  FIG. 4 . The binary system of some embodiments produces at least one spike anywhere during this region using an existing artificial neural network (e.g., an existing LSTM network stored within the memory of the binary classification system). It can be a single spike or multiple spikes. In practice, the ROT can be considered the tail of the keyword since the spike should be produced almost at the end of the keyword. A goal is to maximize the probability of having one or multiple spikes in the ROT and to get zero all non-ROT. Next, the proposed CTC algorithm using the ROT information is presented. 
     1) Compute the Modified Forward and Backward Variables (See, Reference  212 ,  FIG. 2 ) 
     As mentioned before, for binary classification such as the example of  FIG. 4 , there are only two classes, e.g. keyword and garbage. An illustration of the forward and backward algorithms applied to the labeling in the ROT of  FIG. 4  is shown in  FIG. 5 . 
       FIG. 5  illustrates a binary system of some embodiments labeling each frame in the ROT region of the sequence of inputs. In  FIG. 5 , the black (solid) circles represent indications of the keyword and the white (hollow) circles represent indication of the garbage in the sequence of inputs. The arrows represent different transitions that the binary system can traverse to form different paths across the sequence. As mentioned above, the goal is to maximize the probability of having at least one spike in the ROT. The paths shown in  FIG. 5  that have at least one black circle are valid paths for this region. As there are only two classes, the forward and backward variables are two-dimensional matrixes. Also in the illustrated embodiment the forward and backward variables are computed only for the ROT. In some embodiments, these variables can be computed in logarithm domain using the proposed algorithm as given below. 
     a. Forward Variable Calculation in Logarithm Domain for ROT 
     
       
         
           
             
               
                 
                   
                     initialization 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     formula 
                   
                   ⁢ 
                   
                      
                   
                   ⁢ 
                   
                     
                       
                         
                           α 
                           ^ 
                         
                         r 
                       
                       ⁡ 
                       
                         ( 
                         0 
                         ) 
                       
                     
                     = 
                     
                       log 
                       ⁡ 
                       
                         ( 
                         
                           y 
                           g 
                           r 
                         
                         ) 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       
                         
                           α 
                           ^ 
                         
                         r 
                       
                       ⁡ 
                       
                         ( 
                         1 
                         ) 
                       
                     
                     = 
                     
                       log 
                       ⁡ 
                       
                         ( 
                         
                           1 
                           - 
                           
                             y 
                             g 
                             r 
                           
                         
                         ) 
                       
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     recursion 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     formula 
                     ⁢ 
                     
                         
                     
                     ⁢ 
                     
                       ( 
                       
                         r 
                         &lt; 
                         t 
                         ≤ 
                         
                           r 
                           + 
                           N 
                           - 
                           1 
                         
                       
                       ) 
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     
                       
                         
                           α 
                           ^ 
                         
                         t 
                       
                       ⁡ 
                       
                         ( 
                         s 
                         ) 
                       
                     
                     = 
                     
                       { 
                       
                         
                           
                             
                               A 
                               + 
                               
                                 log 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     y 
                                     g 
                                     t 
                                   
                                   ) 
                                 
                               
                             
                           
                           
                             
                               
                                 if 
                                 ⁢ 
                                 
                                     
                                 
                                 ⁢ 
                                 
                                   mod 
                                   ⁡ 
                                   
                                     ( 
                                     
                                       s 
                                       , 
                                       2 
                                     
                                     ) 
                                   
                                 
                               
                               == 
                               0 
                             
                           
                         
                         
                           
                             
                               A 
                               + 
                               
                                 log 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     1 
                                     - 
                                     
                                       y 
                                       g 
                                       t 
                                     
                                   
                                   ) 
                                 
                               
                             
                           
                           
                             otherwise 
                           
                         
                       
                       } 
                     
                   
                   ⁢ 
                   
                     
 
                   
                   ⁢ 
                   
                     A 
                     = 
                     
                       log 
                       ⁡ 
                       
                         ( 
                         
                           
                             exp 
                             ⁡ 
                             
                               ( 
                               
                                 
                                   
                                     α 
                                     ^ 
                                   
                                   
                                     t 
                                     - 
                                     1 
                                   
                                 
                                 ⁡ 
                                 
                                   ( 
                                   s 
                                   ) 
                                 
                               
                               ) 
                             
                           
                           + 
                           
                             exp 
                             ⁡ 
                             
                               ( 
                               
                                 
                                   
                                     α 
                                     ^ 
                                   
                                   
                                     t 
                                     - 
                                     1 
                                   
                                 
                                 ⁡ 
                                 
                                   ( 
                                   
                                     s 
                                     - 
                                     1 
                                   
                                   ) 
                                 
                               
                               ) 
                             
                           
                         
                         ) 
                       
                     
                   
                 
               
               
                 
                   ( 
                   1 
                   ) 
                 
               
             
           
         
       
     
     b. Backward Variable Calculation in Logarithm Domain for ROT 
                     initialization   ⁢           ⁢   formula     ⁢     
     ⁢           β   ^       r   +   N   -   1       ⁡     (   0   )       =     log   ⁡     (     y   g     r   +   N   -   1       )         ⁢     
     ⁢           β   ^       r   +   N   -   1       ⁡     (   1   )       =     log   ⁡     (     1   -     y   g     r   +   N   -   1         )         ⁢     
     ⁢     recursion   ⁢           ⁢   formula   ⁢           ⁢     (     r   ≤   t   &lt;     r   +   N   -   1       )       ⁢     
     ⁢           β   ^     t     ⁡     (   s   )       =     {           A   +     log   ⁡     (     y   g   t     )                 if   ⁢           ⁢     mod   ⁡     (     s   ,   2     )         ==   0               A   +     log   ⁡     (     1   -     y   g   t       )             otherwise         }       ⁢     
     ⁢     A   =     log   ⁡     (       exp   ⁡     (         β   ^       t   +   1       ⁡     (   s   )       )       +     exp   ⁡     (         β   ^       t   +   1       ⁡     (     s   +   1     )       )         )                 (   2   )               
where {circumflex over (α)} t (s) and {circumflex over (β)} t (s) are the modified of forward and backward variables in logarithm domain, respectively, and y t   d  is the output of the network for garbage node at t-th frame.
 
2) Compute Signal Error for the Two Output Nodes (See, Reference  214 ,  FIG. 2 )
 
     In this section, an embodiment of how to efficiently compute the signal error at each of the two nodes of the network output is discussed. In various embodiments, the signal error at each of the two nodes of the network output is computed before applying a softmax nonlinear function. It is assumed that the network output for garbage (keyword) at frame t before and after the softmax nonlinear function are denoted by z t   g  (z t   k ) and y t   g  (y t   k =1−y t   g ), respectively. The relationship between z t   g  and y t   g  is given below. 
     
       
         
           
             
               
                 
                   
                     y 
                     g 
                     t 
                   
                   = 
                   
                     
                       e 
                       
                         z 
                         g 
                         t 
                       
                     
                     
                       
                         e 
                         
                           z 
                           k 
                           t 
                         
                       
                       + 
                       
                         e 
                         
                           z 
                           g 
                           t 
                         
                       
                     
                   
                 
               
               
                 
                   ( 
                   3 
                   ) 
                 
               
             
           
         
       
     
     To compute the signal error or the objective function derivatives with respect to the unnormalized outputs, consider the example in  FIG. 4 . It is assumed that the signal error at frame t for garbage (keyword) is denoted by δ t   g  (δ t   k ). In this example, different cost function is used for different parts of the training sequence. In particular, the cross entropy cost function is used for non-ROT (0≤t&lt;r or r+N−1&lt;t≤T−1). This is because in this region, the probability of having no spike should be maximized. But the proposed CTC algorithm is used for ROT. So the signal error for each region can be computed as follows. 
     a) if frame t is in non-ROT (0≤t&lt;r or r+N−1&lt;t≤T−1):
 
δ t   g =1− y   t   g  
 
δ t   k =−δ t   g   (4)
 
     b) if frame t is in ROT (r≤t≤r+N−1): 
                       δ   g   t     =       Y   g   t     -     y   g   t         ⁢     
     ⁢       δ   k   t     =     -     δ   g   t         ⁢     
     ⁢   where   ⁢     
     ⁢       Y   g   t     =         Z   0     -   G         Z   0     +     Z   1     -   G         ⁢     
     ⁢       Z   0     =     exp   ⁡     (           α   ^     t     ⁡     (   0   )       +         β   ^     t     ⁡     (   0   )       -     log   ⁡     (     y   g   t     )         )         ⁢     
     ⁢       Z   1     =     exp   ⁡     (           α   ^     t     ⁡     (   1   )       +         β   ^     t     ⁡     (   1   )       -     log   ⁡     (     1   -     y   g   t       )         )         ⁢     
     ⁢     G   =     exp   ⁡     (       ∑     t   =   r       R   +   N   -   1       ⁢     log   ⁡     (     y   g   t     )         )                 (   5   )               
where Z 0  (Z 1 ) is the probability of all the paths that go through the symbol s=0 (s=1) at time t. G is the probability of the path that includes no keyword. As the goal is to have at least one spike in the ROT, this probability is subtracted as it can be seen in Formula (5).
 
3) Adaptive Learning Rate (See, Reference  216 ,  FIG. 2 )
 
     In order to improve the performance and increase the training convergence rate, an adaptive learning rate algorithm can be utilized in some embodiments. Two embodiments for updating the weights and biases of the network are described below. The weights and biases connected to the output layer are updated according to the following rule for i-th epoch. 
                           ⁢           X   weight     ⁡     (   i   )       =       m   ×       Δ   weight     ⁡     (     i   -   1     )         +     μ   ×       δ   weight     ⁡     (   i   )             ⁢     
     ⁢         Δ   weight     ⁡     (   i   )       =     {             X   weight     ⁡     (   i   )               if   ⁢           ⁢     θ   low   weight       ≤       X   weight     ⁡     (   i   )       ≤     θ   up   weight                   Δ   weight     ⁡     (     i   -   1     )           otherwise         }       ⁢     
     ⁢           ⁢     update   =     {             X   weight     ⁡     (   i   )               if   ⁢           ⁢     θ   low   weight       ≤       X   weight     ⁡     (   i   )       ≤     θ   up   weight               0       otherwise         }       ⁢     
     ⁢           ⁢       weight   ⁡     (   i   )       =       weight   ⁡     (     i   -   1     )       +   update                 (   6   )               
Where m is the momentum (e.g., m=0.9) and μ is the learning rate (e.g., μ=1e−3). θ low   weight  and θ up   weight  are the lower bound and upper bound for the update of the weights (or the biases). δ weight (i) is the error signal which is received at the corresponding weights (or biases) using the standard BPTT. For example, for bias of the garbage node of the output layer, δ weight (i) equals to δ t   g  for t-th frame. As shown in Formula (6), the weights (or the biases) at each epoch are updated using the “update” value. For other weights and biases of the neural network, the following rule may be used to update them.
 
                           initilization   ⁢           ⁢   for   ⁢           ⁢   the   ⁢           ⁢   first   ⁢           ⁢   epoch     →     E   ⁡     (   0   )         =   0     ⁢     
     ⁢     E   ⁡     (   i   )       =       ρ   ×     E   ⁡     (     i   -   1     )         +       (     1   -   ρ     )     ×       (       δ   weight     ⁡     (   i   )       )     2           ⁢     
     ⁢     RMS   =         E   ⁡     (   i   )       +   ɛ         ⁢     
     ⁢         X   weight     ⁡     (   i   )       =       μ   RMS     ⁢       δ   weight     ⁡     (   i   )           ⁢     
     ⁢     update   =     {             X   weight     ⁡     (   i   )               if   ⁢           ⁢     θ   low   weight       ≤       X   weight     ⁡     (   i   )       ≤     θ   up   weight               0       otherwise         }       ⁢     
     ⁢       weight   ⁡     (   i   )       =       weight   ⁡     (     i   +   1     )       +   update               (   7   )               
where ρ is a smoothing factor (e.g, ρ=0.95) is a smoothing factor and c can be a fixed constant to avoid infinity (e.g., ε=1e−6) can be a fixed constant to avoid infinity. The other parameters may be similar to what is discussed above.
 
       FIG. 5  illustrates an exemplary binary classification training system  500  of some embodiments that performs at least part of the binary classification training process described above. The binary classification training system  500  is programmed to perform the training processes described above. The binary classification training system includes the training dataset  537  which contain the pre-segmented labeled training data. Training dataset  537  is connected to the binary classification training module  540  which includes the ROT table  544 . The ROT table is built using the information about the segmented training data provided by training dataset  537 . The binary classification training module  540  also includes a forward pass processing module  541  programmed to perform the forward pass process described above and a backward pass processing module  542  programmed to perform the backward pass processes described above. The adaptive learning module  543  includes one or more adaptive learning algorithms that can be used by the backward pass processing module to update the weights and biases of a neural network, such as the LSTM network, as described above. 
     In some of these embodiments, the binary classification training module  540  can iteratively perform the training processes using different training data to continuously improve and update the neural network. The binary classification training module  540  can store the updated neural network in the memory  528 . The processor  520 , can be a micro-controller, a digital signal processor (DSP), or other processing components, for controlling and facilitating the operations of the binary classification training system  500 , including controlling communications with internal and external devices. 
     The binary classification training system  500  includes one or more communication channels such as a bus for facilitating communication of data between various components of the binary classification system  500 . Components may include device modules  525  for providing device operation and functionality, which may include input/output components  526  such as a touch screen, keyboard, mouse, etc., a display  527  such as a monitor, a memory  528  such as RAM, solid state drive, disk drive, database, etc., and a communications interface  529 . 
     Training the neural network is performed by binary classification training system  500  offline and the trained model including the weights and biases of the neural network, such as an LSTM network, is stored in Binary classification system  600 . 
       FIG. 6  illustrates an exemplary binary classification system  600  of some embodiments that performs at least part of the binary classification training process described above. The binary classification system  600  may be implemented as a mobile device, such as a smart phone or a laptop computer, a television or display monitor, a display computer, a computer server, an automobile, a speech recognition system, or any other device that provides audio keywords (e.g., commands) recognition capability. The binary classification system  600  is communicatively coupled with one or more audio inputting devices  605  such as a microphone and optionally also with one or more audio outputting devices  610  such as a loudspeaker. 
     In some embodiments, the binary classification system  600  can include an analog-to-digital converter  615  that converts the analog audio signals received from the audio inputting devices  605  into digital audio signals and sends the digital audio signals to processor  620 , which can be a micro-controller, a digital signal processor (DSP), or other processing components, for controlling and facilitating the operations of the binary classification system  600 , including controlling communications with internal and external devices. The binary classification system  600  may also include a digital-to-analog converter  650  that converts digital audio signals generated by the different modules and components of the binary classification system  600  to analog signals before transmitting the analog signals to the one or more audio outputting devices  610 . 
     The binary classification system  600  includes one or more communication channels such as a bus for facilitating communication of data between various components of the binary classification system  600 . Components may include device modules  625  for providing device operation and functionality, which may include input/output components  626  such as a touch screen, keyboard, mouse, etc., a display  627  such as a monitor, a memory  628  such as RAM, solid state drive, disk drive, database, etc., and a communications interface  629 . In some embodiments, the communications interface  629  may include a network interface (e.g., Ethernet interface) or a wireless transceiver for enabling the binary classification system to communicate with remote devices over a network  632 . Remote devices may include user devices  630  (e.g., household appliances, other user electronics, etc.), or a web server  635  that is communicatively coupled with a media storage  637 . 
     The binary classification system  600  also includes a binary classification module  645  programmed to use the neural network that was trained and updated by the binary classification training system  500 . The binary classification module includes the forward pass processing module  644 . In one embodiment, forward pass processing module  644  is programmed to perform binary classification and prediction on audio input data received from audio inputting devices  605 . In various embodiments, binary classification module  645  may include an automatic speech recognition (ASR) module providing voice command processing, an image search and classification system, an object detection system, medical diagnostic module, or other application configured for use with the neural network as described herein. As mentioned, for a given set of audio input data, the forward pass processing module  644  can produce either one of only two possible responses (e.g., a positive response or a negative response). Preferably, the forward pass processing module  644  can generate a response based on a neural network in real-time or close to real-time of the audio input. In some embodiments, the binary classification module  645  can be programmed to output the response as an audio sound via the digital-to-analog converter  650  and the audio outputting devices  610 . Instead of or in addition to producing an audio signal, the binary classification module  645  can be programmed to send a signal to an external device (e.g., to initiate an action or a transaction through the external device) based on the generated response. For example, the binary classification system  600  can be part of or communicatively coupled with a smart home system, and send a signal to a user device (e.g., a household appliance) via a network  632  (e.g., a local area network within the user&#39;s residence) based on the generated response (e.g., sending a signal to turn on the television based on the response generated by an audio input from the user). In another example, the binary classification system  600  can initiate a transaction with a web server  635  over the Internet based on the generated response (e.g., sending a signal to the web server  635  to purchase a movie stored in media storage  637  based on the response generated by an audio input from the user). The binary classification system disclosed herein is not limited to processing audio signals, but can be used to train a neural network to process different input (e.g., image data, video data, etc.) as well. 
     It should be noted that any language directed to a computer should be read to include any suitable combination of computing devices, including servers, interfaces, systems, databases, agents, peers, engines, modules, controllers, or other types of computing devices operating individually or collectively. One should appreciate the computing devices comprise a processor configured to execute software instructions stored on a tangible, non-transitory computer readable storage medium (e.g., hard drive, solid state drive, RAM, flash, ROM, etc.). The software instructions preferably program the computing device to provide the roles, responsibilities, or other functionality as discussed above with respect to the disclosed apparatus. In especially preferred embodiments, the various servers, systems, databases, or interfaces exchange data using standardized protocols or algorithms, possibly based on HTTP, HTTPS, AES, public-private key exchanges, web service APIs, known financial transaction protocols, or other electronic information exchanging methods. Data exchanges preferably are conducted over a packet-switched network, the Internet, LAN, WAN, VPN, or other type of packet switched network. 
     Where applicable, various embodiments provided by the present disclosure may be implemented using hardware, software, or combinations of hardware and software. Also, where applicable, the various hardware components and/or software components set forth herein may be combined into composite components comprising software, hardware, and/or both without departing from the spirit of the present disclosure. Where applicable, the various hardware components and/or software components set forth herein may be separated into sub-components comprising software, hardware, or both without departing from the scope of the present disclosure. In addition, where applicable, it is contemplated that software components may be implemented as hardware components and vice-versa. 
     Software, in accordance with the present disclosure, such as program code and/or data, may be stored on one or more computer readable mediums. It is also contemplated that software identified herein may be implemented using one or more general purpose or specific purpose computers and/or computer systems, networked and/or otherwise. Where applicable, the ordering of various steps described herein may be changed, combined into composite steps, and/or separated into sub-steps to provide features described herein. 
     The foregoing disclosure is not intended to limit the present disclosure to the precise forms or particular fields of use disclosed. As such, it is contemplated that various alternate embodiments and/or modifications to the present disclosure, whether explicitly described or implied herein, are possible in light of the disclosure. Having thus described embodiments of the present disclosure, persons of ordinary skill in the art will recognize that changes may be made in form and detail without departing from the scope of the present disclosure. Thus, the present disclosure is limited only by the claims.