# Joint Learning of Temporal Models to Handle Imbalanced Data for Human Activity Recognition
## 3 Methods
### Model Selection and Architecture
#### 3.1.1 Lstm  
LSTM extends the memory of the Recurrent Neural Network (RNN) to learn patterns from temporal sequential data [34]. It has been used to process the sensor data collected in a smart home for human activity recognition [17; 18]. Different from RNN, LSTM solves the vanishing gradient problem, which learns long-term sequences and the effect of initial dependencies in the sequence. LSTM has been widely used for the applications with temporal dependence between observations [35], such as natural language processing [36], stock market prediction [37], and speech recognition [38]. Each LSTM has three gates, which are forget, input, and output gates, which remove or add information to the cell state. The cell state is the key component to LSTMs which store and pass information between LSTMs. Figure 1 shows how the three gates are connected to the cell state and to each other. Each LSTM cell operates as a memory to erase, read, and write information based on the outcomes rendered by forget, output, and input gates, respectively. Forget gate receives both a new time step \(X_{t}\) and the previous output \(h_{t-1}\) as input and renders the output using sigmoid activation function to decides what information will be kept or deleted. The information will be kept if the output of the sigmoid function is 1, while the information will be completely removed if the output of the sigmoid function is 0. Equation (1) shows how the forget gates is computed. The next step comprises two parts to specifies what new information should be stored in the cell state. Input gate is the first part and specify what new information from the current input \((X_{t},h_{t-1})\) is added to the cell state. The second part is the tanh activation function that generates \(\hat{C}_{t}\) a vector of new candidate values and could be appended to the cell state. The multiplication of these two parts will be added to the multiplication of forget gate with previous cell state to create a new cell state \(C_{t}\). When forget gate multiplied by the previous cell state, part of the information which was specified to be deleted earlier will be forgotten. Then the new candidate values is scaled by how much the cell state is updated using \(i_{t}\times\tilde{C}_{t}\). Equations (2)-(4) show how the input gate, new candidate values, and cell state are computed, respectively. Finally, the output gate is computed based on filtered information using two different activation function and also specify the next hidden state. First, previous hidden state \(h_{t-1}\) and the current input time step \(x_{t}\) are passed into the sigmoid activation function. Next, the new updated cell state is fed to the tanh activation function. The output of sigmoid function mulitplies by the output of tanh functions to generate the next hidden state. The updated cell state and the new generated hidden state pass information to the next time step. Equations (5) and (6) show the calculation of output gate and hidden state. \[f_{t}=\sigma(W_{f}.[h_{t-1},x_{t}]+b_{f})\hskip 14.226378 ptf_{t}\]  
represents forget gate (1)  
\[i_{t}=\sigma(W_{i}.[h_{t-1},x_{t}]+b_{i})\hskip 14.226378 pti_{t}\]  
represents input gate (2)  
\[\tilde{C}_{t}=\tanh(W_{C}.[h_{t-1},x_{t}]+b_{c})\hskip 14.226378 pt\tilde{C}_{t}\]  
represents candidate values (3)  
\[C_{t}=f_{t}\times C_{t-1}+i_{t}\times\tilde{C}_{t}\hskip 14.226378 ptC_{t}\]  
represents Cell state (4)  
\[o_{t}=\sigma(W_{o}.[h_{t-1},x_{t}]+b_{o})\hskip 14.226378 pto_{t}\]  
represents output gate (5)  
\[h_{t}=o_{t}\times\tanh C_{t}\hskip 14.226378 pth_{t}\]  
represents hidden state (6)  
where x is the input data, \(\sigma\) is the sigmoid activation function, tanh is the hyperbolic tangent activation function, W is the weight matrix. LSTM has been adopted in activity recognition applications and obtained promising results [18; 20; 39; 40; 41]. Therefore, LSTM as a temporal model is used in this study.