# Activity Recognition with Combination of Deeply Learned Visual Attention and Pose Estimation
## 3 Proposed Algorithm
### Pose Estimation
#### 3.2.2 Pose Sequence Modelling  
In general, 3D pose estimation from RGB data can be noisy depending on the estimation model and the available training dataset. In this subsection, we propose an LSTM-based temporal model that is suitable for estimating the temporal dependency between noisy skeletal pose estimates. Our approach has two main components: (1) a feed-forward network for expanding the data to a high-dimensional space, and (2) multi-layer LSTM units for modelling the temporal dependency. (see Figure 5) First, the description of data expansion. An estimated 3D skeleton with \(J\) number of joints is a vector in \(\mathbb{R}^{3J}\). Hence, a noisy joint estimate is directly reflected on some of the dimensions of the observed vector. One typical solution for removing noise and redundancy is to contract the data to a lower dimensional space [44]. On the contrary, in this paper, we expand the data to a higher dimensional space. The main motivation for expanding the data is to disentangle explanatory factors that are obscured by noisy joint estimates. Consequently, the parameters of the expansion function are learned directly from the training dataset. Expansion of an observed skeleton is defined as follows:  
\[\tilde{x}=tanh(Wx+b), \tag{15}\]  
where \(W\) is a \(k\times 3J\) matrix with \(k\gg 3J\), \(b\) is a bias vector in a \(k\)-dimensional space, and the \(\tilde{x}\) denotes the expanded pose estimate. Second, the description of temporal model and activity labeling. The temporal dependency between the sequential data points modelled using layers of LSTM units [6]. An LSTM is a gated recurrent neural network that models temporal dependency as a stationary process. While it has several components, we herein will refer to the integrated computational unit as LSTM. Subsequently, given an expanded input data \(\tilde{x}\), we estimate hierarchical latent variables by layering LSTM units one on top of another, see Figure 5. Consequently, the inferred latent space from the \(i\) pose estimate is given as  
\[h_{i}^{L}=LSTM(\tilde{x}_{i}), \tag{16}\]  
where \(L\) denotes the index of the last LSTM layer. Finally an activity label from a set \(\Psi\) is assigned to a sequence as  
\[\tilde{\psi}=\underset{\psi\in\Psi}{\operatorname{argmax}}(tanh(Wh_{n}^{L}+b)), \tag{17}\]  
where \(n\) is the index of the last pose estimate. \(\Psi\) is the set of activity labels, \(\psi\) is the activity label that can be output as a result, and \(\tilde{\psi}\) is the final determined activity label. The connection weights and biases of the overall network (temporal model and data expansion) are trained together by minimizing the cross-entropy between the predicted and the given probability of an activity label via back-propagation and back-propagationthrough time [42]. In Figure 6, you can see that each joint represents a human skeleton. This uses VNect, and if the pose estimation result and the RGB-based result are combined as a result, the accuracy of similar motions that could not be properly classified is improved.