# Activity Recognition with Combination of Deeply Learned Visual Attention and Pose Estimation
## 3 Proposed Algorithm
### Visual Attention
#### 3.1.2 LSTM and Attention Mechanism  
In this paper, we implemented using LSTM proposed by [PERSON] et al. [17]. \[\left( \begin{array}{c}i_{t}\\ f_{t}\\ o_{t}\\ g_{t}\end{array} \right)=\left( \begin{array}{c}\sigma(W_{i}x_{t}+W_{i}h_{t-1}+ b_{i})\\ \sigma(W_{f}x_{t}+W_{f}h_{t-1}+b_{f})\\ \sigma(W_{o}x_{t}+W_{o}h_{t-1}+b_{0})\\ tanh(W_{g}x_{t}+W_{g}h_{t-1}+b_{g}\end{array} \right)M\left( \begin{array}{c}h _{t-1}\\ x_{t}\end{array} \right), \tag{3}\]  
\[c_{t}=f_{t}\odot c_{t-1}+i_{t}\odot g_{t}, \tag{4}\]  
\[h_{t}=o_{t}\odot tanh(c_{t}), \tag{5}\]  
where \(i_{t}\) is the \(t\) input gate, \(f_{t}\) is the \(t\) forget gate, \(o_{t}\) is the \(t\) output gate, and \(g_{t}\) is \(t\) memory gate, and is calculated as shown in Equation (3). \(\sigma(\cdot)\) is the sigmoid function, and \(\sigma(s)\) is defined as follows. \[\sigma(s)=\frac{1}{1+e^{-s}}, \tag{6}\]  
\[W_{i}x_{t},W_{f}x_{t},W_{o}x_{t},W_{g}x_{t},\] is 4 weights used in each gate of the \(t\) time with \(x_{t}\), and \(W_{i}h_{t-1}\), \(W_{f}h_{t-1}\), \(W_{g}h_{t-1}\) is the 4 weights used in each gate of the \(t-1\) time with \(h_{t-1}\). \(b_{i},b_{f},b_{o},b_{g}\) is 4 weights bias used in each gate. \(c_{t}\) is the \(t\) cell state, \(h_{t-1}\) is the \(t-1\) hidden state, \(x_{t}\) represents the input to the LSTM at time step \(t\). and \(f_{t}\odot c_{t-1}\) denotes the Hadamard product of the \(t\) forgetting gate \(f_{t}\) and the \(t-1\) cell state \(c_{t}\). \(\tanh(\cdot)\) is the sigmoid function, and \(\tanh(s)\) is defined as follows. \[\tanh(s)=\frac{e^{s}-e^{-s}}{e^{s}+e^{-s}}, \tag{7}\]  
\(M\) is an affine transformation consisting of trainable parameters from dimension \(a\) to dimension \(b\) (\(\mathbb{R}^{a}\rightarrow\mathbb{R}^{b}\)). At each time step \(t\), the model predicts \(l_{t+1}\), which is used as an input to the next time step \(t+1\), through the LSTM model. Then, the model predicts the label class \(y_{t}\) corresponding to the \(t\) through \(tanh\) activation and Softmax (see Figure 3). The location softmax is defined as follows:  
\[l_{t,i}=p(L_{t}=i\mid h_{t-1})=\frac{exp(W_{i}^{T}h_{t-1})}{\sum_{j=1}^{K\times K }exp(W_{j}^{T}h_{t-1})}\qquad i\in 1,\dots,K^{2}, \tag{8}\]  
where \(W_{i}\) are the weights mapping to the \(i^{th}\) element of the location softmax and \(L_{t}\) is a random variable which can take 1-of-\(K^{2}\) values. And \(W_{j}\) is a weight mapped to the \(j\) element calculated through a Softmax operation, and \(l_{t,i}\) is the \(i\) element among \(K^{2}\) of \(l_{t}\) used as inputs. This softmax can be thought of as the probability with which our model believe the corresponding region in the input frame is important. After calculating theseprobabilities, the soft attention mechanism computes the expected value of the input at the next time-step \(x_{t}\) by taking expectation over the feature slices at different region (see Figure 4):  
\[x_{t}=\mathbb{E}_{p(L_{t}|h_{t-1})}[X_{t}]=\Sigma_{i=1}^{K^{2}}l_{t,i}X_{t,i}, \tag{9}\]  
where \(X_{t}\) is the feature cube and \(X_{t,i}\) is the \(i\) slice of the feature cube at time-step \(t\). Note that in the hard attention based models, we would sample \(L_{t}\) from a softmax distribution of Equation (8). The input \(x_{t}\) would then be the feature slice at the sampled location instead of taking expectation over all the slices. Thus, hard attention based models are not differentiable and have to resort to some form of sampling. We use the following initialization strategy (see [PERSON] et al. [17]) for the cell state and the hidden state of the LSTM for faster convergence:  
\[c_{0}=f_{init,c}(\frac{1}{T}\Sigma_{t=1}^{t}(\frac{1}{K^{2}}\Sigma_{i=1}^{K^{2 }}X_{t,i})) \tag{10}\]  
\[h_{0}=f_{init,h}(\frac{1}{T}\Sigma_{t=1}^{t}(\frac{1}{K^{2}}\Sigma_{i=1}^{K^{2 }}X_{t,i})) \tag{11}\]  
where \(f_{init,c}\) and \(f_{init,h}\) are two multilayer perceptrons and \(T\) is the number of time-steps in the model. These values are used to calculate the first location softmax \(l_{1}\) which determines the initial input \(x_{1}\).