# Multimodal Fall Detection Using Spatial-Temporal Attention and Bi-LSTM-Based Feature Fusion
## 4 Proposed Methodology
### Skeleton Feature Using GSTCAN  
The GSTCAN model executes action recognition grounded in a Graph Convolutional Network (GCN) framework, which autonomously identifies spatial and temporal patterns through data procured via AlphaPose as its input. The joints of the human skeleton are used as nodes of the graph, and the human body structures and their connections in time are used as edges of the graph. GSTCAN is mainly composed of a GSTCAN unit and a pooling layer [46]. The GCTCAN unit is constructed by the convolution of the spatial and temporal graph and is constructed by the CA module. The inputs are of the form \((N,C,T,V)\), where \(N\) is the batch size, \(C\) is the number of channels, \(T\) is the sequence length, and \(V\) is the number of nodes. Spatial convolution extracts each joint's features based on the relationship between nodes. The distance connections between nodes are classified into three categories: connections between nodes with the same distance from the center of gravity, connections between nodes in the direction closer to the center of gravity, and connections between nodes in the direction away from the center of gravity. Temporal convolution captures changes in the same joint over time. For example, it learns how the elbow position moves in the far frame. This GSTCAN unit is applied 6 times to the input data to generate higher-order feature maps on the graph. The number of channels is 64 for the first two layers, 128 for the next two layers, and 256 for the last two layers. A channel-attention mechanism is added at the end of each unit [59; 60; 61; 62; 63; 64]. The addition of the CA mechanism distinguishes between \"important\" and \"unimportant\" learned features, allowing us to focus on more important features. It enables more accurate behavior recognition. Figure 6 shows the CA approach's working architecture.  
The CA mechanism is constructed utilizing the following components: (1) GlobalAveragePooling, (2) Dense (N/4), (3) BatchNorm, (4) ReLU, (5) Dense (N), and (6) Sigmoid. Through the final Sigmoid function, a scalar value ranging from 0 to 1 is produced for each respective channel. This scalar value signifies the relative importance of each channel, whereby significant features are allocated a value approaching 1 or 1, while features of lesser significance are assigned a value nearing 0 or 0. After that, to preserve important features, these values are multiplied by the previously learned feature graph to produce a stronger feature graph. Finally, the pooling layer takes this feature map as input and calculates the average value of all elements, which is converted to a single scalar value.  
The final skeleton feature \(f_{\text{skeleton}}\) is computed as:  
\[f_{\text{skeleton}}=\mathcal{P}\left(\sum_{i=1}^{N}\mathcal{A}(\mathcal{T}( \mathcal{G}(\mathcal{B}(X))))\oplus X\right) \tag{5}\]  
Figure 6: Visualization of channel attention mechanism [57; 60; 61].  
where, \(X\) denotes input skeleton information \((N,C,T,V)\), \(N\) is the batch size, \(C\) is the number of channels, \(T\) is the length of the sequence, \(V\) is the number of nodes (joints in the skeleton). \(\mathcal{G}(X)\) denotes GCN, which extracts spatial features based on the skeleton's graph structure. \(\mathcal{T}(\cdot)\) denotes TCN, which captures temporal dependencies in joint movements. \(\mathcal{A}(\cdot)\) denotes the Channel Attention (CA) module, which assigns importance to features by weighting different channels. \(\oplus\) denotes Skip connection, which concatenates or adds the refined features to the original input. \(\mathcal{P}(\cdot)\) denotes Global Average Pooling (GAP), which computes the final compact representation of the skeleton features.