# A Deep Attention Model for Action Recognition from Skeleton Data
## 3 Proposed Method  
### IndRNN-Based Deep Attention Model  
In this paper, the independently recurrent neural network (IndRNN) [14] is used as a basic RNN component to construct a deep RNN for classification to leverage IndRNN's capability of learning deeper and longer features than LSTM. It follows  
\[\mathbf{h}_{t}=\sigma(\mathbf{Wx}_{t}+\mathbf{u}\odot\mathbf{h}_{t-1}+\mathbf{ b}) \tag{1}\]where \(\mathbf{x}_{t}\in\mathbb{R}^{M}\) and \(\mathbf{h}_{t}\in\mathbb{R}^{N}\) are the input and hidden state at time step \(t\), respectively. \(\mathbf{W}\in\mathbb{R}^{N\times M}\), \(\mathbf{u}\in\mathbb{R}^{N}\), and \(\mathbf{b}\in\mathbb{R}^{N}\) are the weight for the current input, the weight for the recurrent input, and the bias of the neurons. \(\odot\) represents the Hadamard product (element-wise multiplication). \(\sigma\) is an element-wise activation function of the neurons, which is the ReLU (rectified linear unit) in this paper, and \(N\) is the number of neurons in this IndRNN layer. Each neuron in one layer is independent from the others, and connection between neurons is achieved by stacking two or more layers of IndRNNs as presented later. Figure 1a shows the framework of the IndRNN-based deep attention model for skeleton-based action recognition. It consists of a main classification network and an attention network. The main classification network is composed of several IndRNN layers, and batch normalization layers are inserted after each IndRNN layer (ignored in Figure 1a for simplicity). Residual connections are also used to further facilitate the gradient propagation across layers, and each residual block consists of two IndRNN layers. A fully connected layer (FC layer) is added at the final time step for classification (also ignored in Figure 1a for simplicity). Due to the use of hidden states at each time step to obtain the attention model, the statistics for the batch normalization is estimated for each time step, while the parameters for the affine mapping is shared over time. The attention for each time step (\(s_{t}\)) is obtained using an attention network based on the current input and the hidden state of the last IndRNN layer at the previous time step. Unlike the conventional attention network that consists of only one FC layer, a multiple-layer network is proposed as the attention network. It avoids the issue that the conventional one-layer attention network cannot robustly estimate attention, leading to the attention being mainly placed on joints with larger movements. Specifically, since the hidden state of the last IndRNN layer captures the high-level information after going through several IndRNN layers, a few FC layers are first used to extract efficient features of the input skeleton before concatenating it with the hidden state of IndRNN. Figure 1b illustrates the attention network, where residual networks are used with residual connections across every two FC layers. The residual network used to extract the features of the input skeleton, referred to as feature extractor, consists of 5 FC layers, while the residual network used to process the concatenated features, referred to as attention estimator, consists of 4 FC layers. Figure 1: The framework of the spatial IndRNN-based attention models for skeleton-based action recognition. (**a**) The attention network obtains the attention based on the last hidded state and the current input. The main classification network is a deep residual IndRNN network where each residual block contains two IndRNN layers. (**b**) Illustration of the attention network used to process the input and the last hidden state from IndRNN. Details on the architecture will be further explained in Section 4. A softmax function over all the joints is added at the end to produce the attention weights. The input to the IndRNN net is \(x^{\prime}_{t}=(x^{\prime}_{t,1},x^{\prime}_{t,2},\ldots,x^{\prime}_{t,k})\) with \(x^{\prime}_{t,j}=s_{t,i}\cdot x_{t,i}\) where \(x_{t,i}\) and \(s_{t,i}\) represent the feature and attention weight of the \(i\)-th joint.