# Approximate LSTM Computing for Energy-Efficient Speech Recognition
## 2 Background
### DeepSpeech Network and LSTM Operations  
For the DNN-based recognition system, in this work, we adopt the recent DeepSpeech network from [15] as a baseline network to develop our energy-efficient optimization schemes. Accepting the input MFCC features, as illustrated in Figure 2, the DeepSpeech network utilizes several processing layers; three fully-connected layers (FC1\(\sim\)FC3) for tailoring the input MFCC features, a bi-directional LSTM layer over \(n\) feature sets, and two fully-connected layers (FC4\(\sim\)FC5) for making the reliability values of 29 character candidates. Here, \(n\) denotes the number of windows for the input speech data. Note that the character candidates include 26 alphabets, an apostrophe symbol, a blank symbol, and a punctuation mark. With the proper pre-/post-processing steps, including feature extractions and CTC operations, as described in [14], the DeepSpeech network achieved an attractive word-error-rate (WER) of 9.33% when it applied to recognize Librispeech dataset [19].  
The LSTM cell operation in the DeepSpeech network is a key computation for providing such a high recognition quality. More precisely, as described in Figure 3, each LSTM operation receives the current input data, which is a \(d\times 1\) vector \(\mathbf{x}_{t}\), and the hidden state and cell state of the previous LSTM unit, which are two \(k\times 1\) vectors denoted as \(\mathbf{h}_{t-1}\) and \(\mathbf{c}_{t-1}\), respectively. Note that the subscript \(t\) here is the index of the window sample, representing the timing stamp during the recognition process. The LSTM cell then computes the output cell and hidden states, i.e., \(\mathbf{c}_{t}\) and \(\mathbf{h}_{l}\). More precisely, three gate operations called input, output, and forget gates, first generate internal vectors denoted as \(\mathbf{f}_{t}\), \(\mathbf{i}_{t}\), and \(\mathbf{o}_{t}\), respectively, which are formulated as follows.  
\[\mathbf{f}_{t}=\sigma(\mathbf{W}_{xf}\mathbf{x}_{t}+\mathbf{W}_{ hf}\mathbf{h}_{t-1}+\mathbf{b}_{f}) \tag{1}\] \[\mathbf{i}_{t}=\sigma(\mathbf{W}_{xi}\mathbf{x}_{t}+\mathbf{W}_{ hi}\mathbf{h}_{t-1}+\mathbf{b}_{i})\] (2) \[\mathbf{o}_{t}=\sigma(\mathbf{W}_{xo}\mathbf{x}_{t}+\mathbf{W}_{ ho}\mathbf{h}_{t-1}+\mathbf{b}_{o}) \tag{3}\]  
Figure 1: Processing overview of end-to-end speech recognition.  
To reflect the contributions of input data and the hidden state, as shown above, we introduce two types of weight matrices \(\mathbf{W}_{x}\) and \(\mathbf{W}_{h}\) at each gate operation. As elements in each matrix are trained individually for the gate operations, it is required to keep a total of eight matrices for realizing the LSTM cell. In addition, a vector \(\mathbf{b}\) having the same size as the input data is used as bias values. As depicted in Figure 3, note that non-linear operators like a sigmoid function and a hyperbolic tangent function are applied at the end of gate operations. The final outputs of the LSTM cell are then calculated as follows, where \(\odot\) stands for the element-wise product between two vectors.  
\[\mathbf{c}_{t}=\mathbf{f}_{t}\odot\mathbf{c}_{t-1}+\mathbf{i}_{t}\odot\tanh( \mathbf{W}_{xg}\mathbf{x}_{t}+\mathbf{W}_{hg}\mathbf{h}_{t-1}+\mathbf{b}_{g}) \tag{4}\]  
\[\mathbf{h}_{t}=\mathbf{o}_{t}\odot\tanh(\mathbf{c}_{t}) \tag{5}\]