# Enhancing Kazakh Sign Language Recognition with BiLSTM Using YOLO Keypoints and Optical Flow
## 3 Importance of Components
### BiLSTM-Based Temporal Modeling  
Sign language recognition requires a model that can effectively process sequential data since gestures are dynamic and change over time. Traditional recurrent neural networks often struggle with long-term dependencies due to the vanishing gradient problem. To address this, long short-term memory networks were introduced, which are capable of learning longer sequences by using gated mechanisms to retain important information while forgetting irrelevant details [14].  
However, standard LSTMs process sequences in a single direction (forward), meaning they may not leverage future context while interpreting a sequence. This limitation is particularly critical for sign language recognition, where understanding a gesture often depends on both previous and upcoming movements [15]. To improve temporal modeling, we employ a Bidirectional LSTM, which processes sequences in both forward and backward directions (Figure 1). This architecture provides a richer representation by considering the entire temporal context.  
The LSTM updates its states at each time step \(t\) using the following equations:  
\[f_{t} =\sigma(W_{t}x_{t}+U_{t}h_{t-1}+b_{f})\] (Forget gate) (5) \[i_{t} =\sigma(W_{i}x_{t}+U_{t}h_{t-1}+b_{i})\] (Input gate) (6) \[\bar{c_{t}} =\tanh(W_{c}x_{t}+U_{c}h_{t-1}+b_{c})\] (Candidate cell state) (7) \[c_{t} =f_{t}\odot c_{t-1}+i_{t}\odot\bar{c_{t}}\] (Cell state update) (8) \[o_{t} =\sigma(W_{o}x_{t}+U_{o}h_{t-1}+b_{o})\] (Output gate) (9) \[h_{t} =o_{t}\odot\tanh(c_{t})\] (Hidden state update) (10)  
These equations define the standard architecture of the long short-term memory (LSTM) network, originally proposed by [PERSON] and [PERSON] [16]. The model's ability to control the forgetting and retention of information over time makes it particularly well-suited for learning temporal dependencies in sequential data. Further studies such as [PERSON] et al. [17] have analyzed and validated the effectiveness of LSTM variants across various tasks.  
The BiLSTM network in our project is trained on NumPy arrays containing three types of input features: keypoints extracted by YOLOv8, representing the spatial locations of hands in each frame; bounding boxes, which define the regions surrounding the detected hands and assist in gesture localization; and optical flow features, capturing motion dynamics around the keypoints to distinguish between static and dynamic gestures.  
**Input Layer.**  
The input sequence is a time-series representation of the hand gestures. At each timestep \(t\), the input vector consists of the concatenated values of keypoints, bounding box coordinates, and optical flow features.  
**Bidirectional Processing.**  
Figure 1: The structure of bidirectional long short-term memory (BiLSTM).  
The sequence is simultaneously passed through two separate LSTM layers. The forward LSTM processes the sequence from \(t=1\) to \(t=N\), while the backward LSTM processes it from \(t=N\) to \(t=1\).  
The hidden states are computed as follows:  
\[\overrightarrow{h}_{t} =\text{LSTM}_{\text{forward}}(x_{t},\overrightarrow{h}_{t-1}) \tag{11}\] \[\overleftarrow{h}_{t} =\text{LSTM}_{\text{backward}}(x_{t},\overleftarrow{h}_{t+1})\] (12) \[h_{t} =\text{concat}(\overrightarrow{h}_{t},\overleftarrow{h}_{t}) \tag{13}\]  
**Fully Connected Layer and Classification:**  
The concatenated hidden state is passed through a fully connected layer that maps it to class probabilities.  
A softmax activation function is applied to generate the final classification of the gesture [18]:  
\[P(y=k|h_{t})=\frac{\exp(W_{k}h_{t}+b_{k})}{\sum_{j}\exp(W_{j}h_{t}+b_{j})} \tag{14}\]  
**Training and Loss Function:**  
The model is trained using cross-entropy loss, which is optimized via backpropagation through time (BPTT) [19].  
The Adam optimizer is used to adjust weights for better performance.  
By combining YOLO-based keypoint detection, optical flow computation, and BiLSTM-based sequence modeling, our method achieves accurate and robust recognition of dynamic gestures in Kazakh Sign Language.