# Multi-Term Attention Networks for Skeleton-Based Action Recognition
## 4 Methods
### Multi-Term Attention Recurrent Neural Network
#### 4.1.1 Multi-Term Temporal Sliding LSTM  
Inspired by [PERSON] et al. [8], we adopt multiple parallel Temporal Sliding LSTM networks (TS-LSTM) in an MT-TS-LSTM. Straightforward LSTM captures only long-term dependency, while the MT-TS-LSTM can effectively obtain the diversity of temporal dependencies. Comparison of different LSTM structures is shown in Figure 3. We can adapt TS-LSTM into various dependencies through controlling of temporal stride and internal LSTM time-step size. To minimize the complexity of the network and reduce the GPU occupancy, we apply three parallel TS-LSTM networks, including long-term TS-LSTM, medium-term TS-LSTM, and short-term TS-LSTM networks, to obtain the temporal features simultaneously from skeleton sequences. Compared with straightforward LSTM, MT-TS-LSTM can capture various temporal dependencies, including long-term, medium-term, and short-term dependencies, which is effective for classifying various actions.  
Figure 3: Comparison of the structures between straightforward LSTM and TS-LSTM. (**a**) shows the structure of the straightforward LSTM. (**b**) shows the conceptual diagram to illustrate the structure of the \(l\)th TS-LSTM when \(N_{l}=3\), \(W_{l}=5\), and \(TS_{l}=2\).  
We can set various parameters for the \(l\)th TS-LSTM, including the LSTM network numbers (\(N_{l}\)), the size of LSTM window (\(W_{l}\)), and temporal strides (\(TS_{l}\)). In the process of the variable sequence of action, more temporal features can be obtained by adjusting the size of the window and the size of the temporal strides of the TS-LSTM. For example, when \(N_{l}=3\), \(W_{l}=5\) and \(TS_{l}=2\), the structure conceptual diagram of the TS-LSTM is shown in Figure 3b.  
The input \(X_{i}\) of TS-LSTM is expressed as \(X_{t}^{l}\) in the MT-TS-LSTM, where \(l\) is TS-LSTM module number, \(t\) represents the \(t\)th frame. [PERSON] et al. (2018) showed the specific calculation formula of this method, including the memory cell, input gate, output gate, forgetting gate and output vector of \(t\)th frame of \(n\)th LSTM of \(l\)th TS-LSTM, and so forth.  
In the MTA-RNN, the size and shape of \(X\in\mathbb{R}^{(T\times N)}\) will be updated through an FC layer as \(X_{i}\gets FC(X)\in\mathbb{R}^{(T\times K)}\), where \(K\) represents the number of neuron units in a TS-LSTM, and \(n\) denotes the number of TS-LSTM in the MT-TS-LSTM. \(X_{i}\in\mathbb{R}^{(T\times K)}\) can be expressed as:  
\[X_{i}=\begin{bmatrix}X(1)\\ \vdots\\ X(t)\\ \vdots\\ X(T)\end{bmatrix}=\begin{bmatrix}x_{1}(1)&\cdots&x_{k}(1)&\cdots&x_{K}(1)\\ \vdots&\ddots&\vdots&&\vdots\\ x_{1}(t)&\cdots&x_{k}(t)&\cdots&x_{K}(t)\\ \vdots&&\vdots&\ddots&\vdots\\ x_{1}(T)&\cdots&x_{k}(T)&\cdots&x_{K}(T)\end{bmatrix} \tag{4}\]  
\(X_{i}\in\mathbb{R}^{(T\times K)}\) is used for convenience, where \(x(t)\) is the row vector of \(X_{i}\), representing the sequence of \(t\)th frame \([x_{1}(t),\cdots,x_{K}(t)]\), and \(t\in(1,\cdots,T)\).  
Different from other models, we extract a variety of action features with different time scales through multiple LSTM networks with different parameters for enriching the types and quantity of features. There are three parallel TS-LSTM networks applied in one MT-TS-LSTM: long-term TS-LSTM networks, medium-term TS-LSTM networks, and short-term TS-LSTM networks. For example, the output of the long-term TS-LSTM networks is \(F_{L}(X_{i})\in\mathbb{R}^{(T\times K)}\). The temporal features information for the skeleton joint is expressed as:  
\[F_{L}(X_{i})=[TS_{L}(1),\cdots,TS_{L}(t)]^{\text{T}}\in\mathbb{R}^{(T\times K )}, \tag{5}\]  
where \(F_{L}(X_{i})\) summarizes memory information of skeleton and joint in the whole sequence of TS-LSTM.  
Similarly, the final output of medium-term and short-term TS-LSTM network is:  
\[F_{M}(X_{i})=[TS_{M}(1),\cdots,TS_{M}(t)]^{\text{T}}\in\mathbb{R}^{(T\times K )}. \tag{6}\]  
\[F_{S}(X_{i})=[TS_{S}(1),\cdots,TS_{S}(t)]^{\text{T}}\in\mathbb{R}^{(T\times K )}. \tag{7}\]