# Single-Channel Blind Source Separation of Spatial Aliasing Signal Based on Stacked-LSTM
## 3 Background
### Lstm  
As is shown in Figure 1, LSTM has three control gates: input gate, output gate, and forget gate, and they jointly control the unit state A. As is shown in Formula (3), the function of the forget gate is to determine how much the state of the unit at time \(t-1\)\(A_{t-1}\) is retained to \(A_{t}\). \(W_{f}\) and \(U_{f}\) are the weights of the forget gate, and \(\sigma\) is the sigmoid function [41].  
\[\mathbf{f}_{t}=\sigma\Big{(}\mathbf{W}_{f}\mathbf{x}_{t}+\mathbf{U}_{f}\mathbf{h}_{t-1}\Big{)} \tag{3}\]  
As is shown in Formula (4), the input gate controls how much the input \(\mathbf{x}_{t}\) of the network at time t retained to \(\mathbf{A}_{t}\).  
\[\mathbf{i}_{t}=\sigma(\mathbf{W}_{i}\mathbf{x}_{t}+\mathbf{U}_{i}\mathbf{h}_{t-1}) \tag{4}\]  
As is shown in Formula (5), \(\mathbf{z}_{t}\) is used to describe the current input unit state.  
\[\mathbf{z}_{t}=\tanh(\mathbf{W}_{z}\mathbf{x}_{t}+\mathbf{U}_{z}\mathbf{h}_{t-1}) \tag{5}\]  
As is shown in Formula (6), The current unit state \(\mathbf{A}_{t}\) is calculated as follows: the last unit state \(\mathbf{A}_{t-1}\) is element-wise multiplied by the forgetting gate \(\mathbf{f}_{t}\), and then the current input unit state \(\mathbf{z}_{t}\) is element-wise multiplied by the input gate \(\mathbf{i}_{t}\), and the two products are added together. In this way, the current memory \(\mathbf{z}_{t}\) and the long-term memory \(\mathbf{A}_{t-1}\) can be combined to form a new unit state \(\mathbf{A}_{t}\). Because of the control of the forget gate, it can save information from a long time ago, and because of the control of the input gate, it can avoid the current unimportant content from entering the memory.  
\[\mathbf{A}_{t}=\mathbf{f}_{t}\cdot\mathbf{A}_{t-1}+\mathbf{i}_{t}\cdot\mathbf{z}_{t} \tag{6}\]  
As is shown in Formula (7), the function of the output gate is to control how much the unit state at time t \(\mathbf{A}_{t}\) retains to the final output value \(\mathbf{h}_{t}\) at time t.  
\[\mathbf{o}_{t}=\sigma(\mathbf{W}_{o}\mathbf{x}_{t}+\mathbf{U}_{o}\mathbf{h}_{t-1}) \tag{7}\]  
Figure 1: Structure of LSTM.  
Finally, the current unit state \(\mathbf{A}_{t}\) passes the tanh activation function, and then the result is multiplied by the output gate to get the final current moment output, as shown in Formula (8):  
\[\mathbf{h}_{t}=\mathbf{o}_{t}\cdot\tanh(\mathbf{A}_{t}) \tag{8}\]  
LSTM can solve the problem of gradient disappearance and explosion of RNN very well. The reason why the RNN gradient disappears and explodes is that when the number of network layers is very deep, the weight matrix W between the state and the hidden layer may be reused for the product at the same time. Suppose W has the following characteristic decomposition:  
\[\mathbf{W}=\mathbf{Q}\,\text{diag}(\lambda)\mathbf{Q}^{-1} \tag{9}\]  
Then there is Formula (10):  
\[\mathbf{W}_{t}=\left(\mathbf{Q}\,\text{diag}(\lambda)\mathbf{Q}^{-1}\right)^{t}=\mathbf{Q}\, \text{diag}(\lambda)^{t}\mathbf{Q}^{-1} \tag{10}\]  
The problem of gradient disappearance or explosion is due to the size of the eigenvalue. When the eigenvalue is greater than 1, the gradient explodes, and when it is less than 1, the gradient disappears. The above problems will lead to the inability to know the next adjustment direction of the loss function and make the learning process extremely unstable. Another problem of RNN is that when the number of network layers is too deep, the network memory function will be weakened, and there is a problem of long-term dependence. There are three solutions to the disappearance of the gradient of the recurrent neural network: first, ReLU can be selected as the activation function; second, Batch Normalization (BN) can be used, and third, the network structure can be improved, such as LSTM Network structure. LSTM can solve the problem of vanishing gradient and long-term dependence.