# Sequential Characteristics Based Operators Disassembly Quantization Method for LSTM Layers
## 3 Proposed Methodology
### LSTM Operators Disassembly Design  
The LSTM is a specific network layer in the neural network model, but the computational logic of this layer is more complex than the typical layer. The input of the LSTM layer is \(x\) and \(Cont\), and the output is \(h\). \(x\) is a tensor of (T, I), where T is the time step and I is the input feature dimension. \(Cont\) is a continuity identifier, consisting of 0, 1, where 0 means start and 1 means continuous, and the dimension is T. \(h\) is a tensor of (T, O), where O is the output feature dimension.  
LSTM first divides the calculation into T small computing units and then divides \(x\) into T (1, I) tensors equally according to the size of the time step T, named \(\{x_{0},x_{1}\cdots x_{t},\cdots x_{T-1}\}\), respectively. These T divided inputs will be used as the input \(x_{t}\) of each computational unit in the LSTM. It is worth noting here that \(h_{0}\) and \(c_{0}\) are all-zero tensor sequences (1,O), and \(h_{t}\) and \(c_{t}\) are the result of the previous \(h_{t-1}\) and \(c_{t-1}\). Finally, the sequence of T tensors (I, O) from \(h_{0}\) to \(h_{T-1}\) are combined into one output \(h\) (T, O) as the output of the LSTM layer.  
In the specific Caffe implementation process, the input of LSTM can be expressed as \(W_{h}*h+W_{x}*x+b\), so we use two fully connected layers to replace the matrix multiplication operations in Equations (1)-(4). The first is \(W_{x}\times x+b\), and the second is \(W_{h}\times h\), where \(W_{x}=[W_{xi},W_{xf},W_{xo},W_{xg}]\), \(W_{h}=[W_{hi},W_{hf},W_{ho},W_{hg}]\), \(b=[b_{xi},b_{xf},b_{xo},b_{xg}]\).  
Figure 1: Overall method design process.  
In terms of layer type, Equations (1)-(6) are divided into the LSTM layer operation processes calculated in the following 9-layer order:  
Layer 1--FC layer:  
\[data\_fc1=W_{x}*x_{t}+b \tag{9}\]  
Layer 2--FC layer:  
\[data\_fc2=W_{h}*h_{t} \tag{10}\]  
Layer 3--Eltwise Add layer:  
\[data\_sum=data\_fc1+data\_fc2 \tag{11}\]  
Layer 4--Non-linear layer:  
\[\begin{array}{c}data\_it=sigmoid(data\_sum[0:O])\\ data\_ft=sigmoid(data\_sum[O:2*O])\\ data\_ot=sigmoid(data\_sum[2*O:3*O])\\ data\_gt=tanh(data\_sum[3*O:4*O])\end{array} \tag{12}\]  
Layer 5--Eltwise Prod layer:  
\[eltwise1=data\_it*data\_gt \tag{13}\]  
Layer 6--Eltwise Prod layer:  
\[eltwise2=data\_ft*data\_ct \tag{14}\]  
Layer 7--Eltwise Add layer:  
\[data\_ct=eltwise1+eltwise2 \tag{15}\]  
Layer 8--Non-linear layer:  
\[data\_ct\_tanh=tanh(data\_ct) \tag{16}\]  
Layer 9--Eltwise Prod layer:  
\[data\_ht=data\_ot*data\_ct\_tanh \tag{17}\]  
Because hardware accelerators can only process integer data, when we perform nonlinear function operations, there are two steps in total. The first step is to build a nonlinear lookup table in the quantization software. If there are two nonlinear functions, two lookup tables are created. The specific operation is to first dequantize all integer values in the bit width range to a floating point according to the inverse of Equation (8). The dequantized floating-point values are then mapped according to the nonlinear functions sigmoid and tanh, respectively. Finally, the mapped value is quantized to an integer again according to Equation (8). The sequence of input integers and output integers form a lookup table. The second step is to feed the lookup table into the neural network accelerator. The corresponding rules between the source register and the destination register are used to map one-by-one to complete the calculation of the nonlinear layer.