## 2 Proposed Lip-Reading Model
### Attention-Based LSTM  
In general, RNN can obtain sequence output with time information through sequence input [20]. For example, LSTM network which is a special type of RNN can learn long-term dependency information. LSTM was proposed by [PERSON] and [PERSON] [21] and it was improved and promoted by [PERSON] in 2012 [22]. In many practical applications, LSTM has achieved considerable success and it has been widely used. The first step in LSTM is making a decision to discard useless information from the cell state, which is accomplished by a decision called \"the forget gate\". This gate reads \(h_{t-1}\) and \(x_{t}\), then it outputs a value in the [0,1] interval for each number in cell state C\({}_{t-1}\). The calculation process is as follows:  
\[f_{t}\ =\sigma\big{(}W_{f}\cdot[h_{t-1},x_{t}]\ +\ b_{f}\big{)}, \tag{3}\]  
where \(\sigma\) is the hidden activation function, \(h_{t-1}\) is the hidden state at time \(t-1\), \(x_{t}\) is the input at time \(t\), and \(b\) is the bias. Then, it is determined that new useful information is stored in the cell state. It consists of two parts: First, a sigmoid layer is called the \"input gate layer\" and it determines which value will be updated. Then a new candidate value vector is created by tanh, the activation function that processes the data on the state and output is tanh in LSTM. \(\overset{\rightarrow}{\text{C}}_{t}\) is added to the state, and the old cell state C\({}_{t-1}\) is updated to C\({}_{t}\). Second, the cell updates useful information into cell status and multiply the old cell state C\({}_{t-1}\) and the output of \"forget gate\" \(f_{t}\) as the part input of cell, then summing it with the product of \"input gate\" output \(i_{t}\) and candidate information \(\overset{\rightarrow}{\text{C}}_{t}\). The result of the calculation is the updated C\({}_{t}\). This is the new candidate and it changes based on how much we decide to update each state. The calculation processes are as follows:  
\[i_{t}\ =\ \sigma(W_{i}\cdot[h_{t-1},x_{t}]\ +\ b_{i}), \tag{4}\]  
\[\overset{\rightarrow}{\text{C}}_{t}\ =\ \text{tanh}(W_{\text{C}}\cdot[h_{t-1},x_{t}]\ +\ b_{\text{C}}), \tag{5}\]  
Figure 2: Preprocessing steps with example frames. \[C_{t}\ =f_{t}*C_{t-1}+i_{t}*\overset{\rightarrow}{C}_{t}, \tag{6}\]  
Finally, the output value is determined based on the filtered cell state. Firstly, the sigmoid layer determines the output portion of the cell state. Then, the cell state is passed through tanh and multiplied by the output of the sigmoid layer to obtain the result of the cell. The value range of the sigmoid function is [0,1], which is most suitable for controlling the opening and closing of various doors. This part of the calculation processes is shown as follows:  
\[\upalpha_{t}\ =\sigma(W_{o}\cdot[h_{t-1},x_{t}]\ +\ b_{o}), \tag{7}\]  
\[h_{t}\ =\upalpha_{t}*\tanh(C_{t}), \tag{8}\]  
The key to the LSTM network is the cell state. As shown in Figure 3, the calculation process runs through the horizontal line. It runs directly across the chain with only a small amount of linear interaction and it will be easy to keep the information flowing on it. Therefore, LSTM network can successfully discover sequence relationships and we have added an attention mechanism based on it. CNN network extracts the spatial features to obtain fixed-length feature vectors, and the LSTM network identifies the video contents based on the input feature vectors. We use the framework described in Figure 4, which combines attention-based LSTM with a deep CNN to train spatial-temporal features on video sequences. Figure 4: Attention-based LSTM model. Figure 3: LSTM unit structure. The CNN is used as the encoder and the LSTM network is used as the decoder. In the decoding process, we introduce the attention mechanism and learn the attention weights (\(\alpha\)), thus the model pays more attention to the effective area of the whole video [23].