# Hybrid Deep Learning Approach for Automated Sleep Cycle Analysis
## 3 Methodology
### Model Training
#### 3.4.8 LSTM Layers  
Long short-term memory (LSTM) networks are variants of traditional recurrent neural networks (RNNs) that addresses vanishing and exploding gradient issues [62]. Unlike standard RNNs, LSTM models include internal gates, namely input, forget, and output gates, designed to control which data are stored, discarded, or passed on. This architecture makes LSTM particularly suitable for complex sequence tasks such as text analysis, machine translation, and time-series forecasting [63]. In an LSTM layer, several parallel cells manage different elements of the sequential input. At each time step \(t\), the memory state \(C_{t}\) updates according to [64]  
\[C_{t}=f_{t}C_{t-1}+i_{t}\bar{C}_{t}, \tag{23}\]  
where \(f_{t}\), \(i_{t}\), and \(\bar{C}_{t}\) correspond to the forget gate, input gate, and candidate memory content, respectively. These components are computed internally within each LSTM cell through a specific operation [8]. The forget gate regulates the amount of historical data to retain:  
\[f_{t}=\sigma(W_{f}[h_{t-1},x_{t}]+b_{f}), \tag{24}\]  
Figure 7: A 1D CNN structure with input, convolution, and pooling layers. where \(W_{f}\) represents the weight parameters linked to the forget gate, \(h_{t-1}\) is the hidden state from the last time step, \(x_{t}\) indicates the current input, \(b_{f}\) is the bias for the forget gate, and \(\sigma\) is the sigmoid function that produces output values between 0 and 1. Similarly, the input gate determines the amount of new information to be stored in the cell state:  
\[i_{t}=\sigma(W_{i}[h_{t-1},x_{t}]+b_{i}), \tag{25}\]  
where \(W_{i}\) denotes the weights and \(b_{i}\) is the bias associated with the input gate. The candidate memory content, which proposes possible modifications to the cell state, is computed as  
\[\tilde{C}_{t}=\tanh(W_{C}[h_{t-1},x_{t}]+b_{C}), \tag{26}\]  
where \(W_{C}\) and \(b_{C}\) are the respective weights and bias for the candidate memory. The use of \(\tanh\) allows the regulation of values in the range \([-1,1]\), introduces variability, and captures positive and negative influences on the state of the memory. The output gate regulates the impact of the stored memory on the hidden state:  
\[o_{t}=\sigma(W_{o}[h_{t-1},x_{t}]+b_{o}), \tag{27}\]  
with \(W_{o}\) and \(b_{o}\) representing the weights and bias linked to the output gate. Finally, the hidden state \(h_{t}\) is updated during each time step according to the output gate and the current memory state:  
\[h_{t}=o_{t}\tanh(C_{t}). \tag{28}\]  
The overall architecture of an LSTM cell is shown in Figure 8.  
Each gate operation involves matrix multiplications of size \(d\times(d+n)\), which leads to a computational cost per step of \(\mathcal{O}(d\cdot n)\). Consequently, processing a sequence of length \(T\) across \(L\) stacked layers leads to an overall computational cost of \(\mathcal{O}(T\sum_{l=1}^{L}d_{l}n)\)[65], highlighting the importance of optimizing LSTM architectures to meet hardware constraints. An extension of LSTM is BiLSTM, which processes sequences in both forward and backward directions [9]. In a BiLSTM model, each time step \(t\) has two associated hidden states:  
\[h_{t}^{\text{BiLSTM}}= \begin{bmatrix}h_{t}^{-\ast}\\ h_{t}^{+\ast}\end{bmatrix} , \tag{29}\]  
where \(h_{t}^{-\ast}\) represents the hidden state moving forward through time, and \(h_{t}^{+\ast}\) represents the hidden state moving backward [66]. The structure of BiLSTM is shown in Figure 9, where the resulting output \(h_{t}\) of the primary LSTM layer is connected to the input of the secondary LSTM layer moving forward in time. This architecture allows the model to leverage contextual information from both past and future inputs, enhancing its sequence understanding [65]. Figure 8: Structure and connections of an LSTM cell. Although BiLSTM significantly improves the performance by capturing information flowing in both temporal directions, it inherently doubles the memory usage and computational requirements compared to standard unidirectional LSTM.