# Automatic Lip Reading System Based on a Fusion Lightweight Neural Network with Raspberry Pi
## 2 Proposed Model
### LSTM (Long Short-Term Memory)  
In order to solve the problem of gradient disappearance and gradient explosion when RNN processes long-sequence data, [PERSON] [26] proposed an improved form of RNN, called long short-term memory (LSTM), which is specially used to deal with information missing in long-term dependent sequences [27]. LSTM stores historical information by introducing memory units. By introducing three control gate structures, including the input gate, forget gate, and output gate, LSTM controls the increase and removal of information flow in the network. To better discover and utilize long-term dependencies from sequence data (such as video, audio, and text), memory cell remembers the associated information that needs to be remembered in a long sequence and forgets some of the useless information. Figure 6 shows the operations performed within a single LSTM cell. Among them, \(x_{t}\) represents the input vector of the network node at t time, \(h_{t}\) represents the output vector of the network node at t time, \(i_{t}\), \(f_{t}\), \(o_{t}\), and \(c_{t}\) represent the input gate, forget gate, output gate and memory unit at t time respectively. The calculation steps of the input gate, forget gate, memory unit, and output gate in the LSTM unit are as follows:  
1. Input gate: This gate is used to control the input node information. The mathematical expressions of the input gate output and candidate information are as follows: \[i_{t}=\sigma(U_{i}x_{t}+W_{i}h_{t-1}+b_{i})\] (3)  
Figure 5: Depthwise separable convolutions basic structure. Figure 6: Long Short-Term Memory basic unit diagram. \[g_{t}=\tan h\Big{(}U_{g}x_{t}+W_{g}h_{t-1}+b_{g}\Big{)}\] (4) Among them, \(U_{i}\), \(W_{i}\), and \(b_{i}\) represent the weights and biases of input gates, \(U_{g}\), \(W_{g}\), and \(b_{g}\) represent the weights and biases of candidate states, \(\sigma\) represents the sigmoid activation function, and \(\tan h\) is the activation function. 2. Forget gate: This gate is used to control which information is discarded by the current LSTM unit. The mathematical expression of the forget gate is as follows: \[f_{t}=\sigma\Big{(}U_{f}x_{t}+W_{f}h_{t-1}+b_{f}\Big{)}\] (5) Among them, \(U_{f}\), \(W_{f}\), and \(b_{f}\) denote the weights and biases of the forget gates respectively, and \(\sigma\) represents the sigmoid activation function. 3. The memory unit (memory cell): is used to save the state information and update the state. The mathematical expression of the memory unit c is as follows: \[c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot g_{t}\] (6) Among them, \(\odot\) represents the Hadamar product. 4. Output gate: The gate is used to output the control of node information. The mathematical expression of the initial output value and the output of the LSTM unit is: \[o_{t}=\sigma(U_{o}x_{t}+W_{o}h_{t-1}+b_{o})\] (7) \[h_{t}=o_{t}\odot\tan h(c_{t})\] (8) Among them, \(U_{o}\)\(W_{o}\), and \(b_{o}\) represent the weight and bias of the output gate, respectively. We input the pre-processed images into Mobilenets, extract the high-dimensional features of the images in fully connected layers, and input the features into LSTM model for learning the past and future information of the sequence features. In the memory unit of LSTM, putting in all the data passes through only one cell unit in different timing states. Also, it can reduce the number of parameters by keeping updating the weights. (See Figure 7) Among them, W(f), W(i), W(j), W(o) are weight parameters in the cell unit of LSTM. We aim to train these four weight parameters to optimize the LSTM network and reduce the input parameters. The Dropout technique is used to mitigate the over-fit problems that have occurred during the training process. The Dropout technique reduces the complexity of the model by randomly dropping part of the neurons during each training process, thus improving the generalization ability of the model.