# Temporal Modeling on Multi-Temporal-Scale Spatiotemporal Atoms for Action Recognition
## 3 Related Works
### Temporal Evolution Modeling on Atoms  
An action is considered as a temporal evolution of the spatiotemporal atoms. Thus, unordered modeling is insufficient to represent and recognize an action. The RNN can be used to model the temporal evolution of the input sequences and has been applied to many sequence modeling tasks, including speech recognition [42], machine translation [43] and action recognition [44]. Therefore, we propose to employ LSTM, a RNN model, to model the temporal evolution of atoms for action recognition. Different from the CNN-RNN works [21; 22; 23; 24] which applied an LSTM to the deep features of 2D CNN, we impose LSTM on the spatiotemporal atoms that are the key deep features of 3D CNN.  
As a typical RNN architecture, LSTM has the capability to discover the action dynamics from extracted information by using memory cells to store and modify the internal state and has become  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Features Selection** & **Video Accuracy (Pool5)** \\ \hline Sparse split & 87.1\% \\ Dense split & 87.3\% \\
16/32 atoms mining & 86.7\%/86.9\% \\
16/32 random selection & 86.3\%/86.4\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Comparison of the atoms mining with other methods in the RGB stream of 8f temporal scale on UCF101 split-1.  
Figure 5: The mined atoms (S = 4) sorted by timestamps and their corresponding clips in the 8f temporal scale.  
increasingly popular. Figure 6 illustrates a LSTM cell at time step \(t\). Given an input sequence \(X=(x_{1},x_{2},...,x_{t})\), the hidden state of the LSTM is formulated as follows:  
\[i_{t}=\sigma(W_{ix}x_{t}+W_{ih}h_{t-1}+b_{i}) \tag{6}\]  
\[f_{t}=\sigma(W_{fx}x_{t}+W_{fh}h_{t-1}+b_{f}) \tag{7}\]  
\[o_{t}=\sigma(W_{ox}x_{t}+W_{oh}h_{t-1}+b+o) \tag{8}\]  
\[g_{t}=tanh(W_{gx}x_{t}+W_{gh}h_{t-1}+b_{g}) \tag{9}\]  
\[c_{t}=f_{t}*c_{t-1}+i_{t}*g_{t} \tag{10}\]  
\[h_{t}=o_{t}*tanh(c_{t}) \tag{11}\]  
where \(x_{t}\) is the input at time step \(t\), \(i_{t}\) is the input gate, \(f_{t}\) is the forget gate, \(o_{t}\) is the output gate, \(c_{t}\) is the memory cell state and \(h_{t}\) is the hidden state. The input and forget gates determine whether to consider new information or forget old information. In addition, the output gate controls the amount of memory cell state that is passed to the hidden state to influence the computation in the next time step.  
In each stream, we sort the atoms \(Y*\) by timestamps and denote the sorted atoms as \(X=\{x_{1},x_{2},...,x_{num}\}\), where \(num\leq max\_num\), which are fed into LSTM to model the temporal evolution of action. We use a fully-connected layer on the top of the LSTM'S hidden state, followed by a softmax layer, to produce a prediction for each time step. As described in Section 3.3, the number of atoms (\(num\)) associated with each action is variable and the maximum number of atoms (\(max\_num\)) is \(S\), therefore, we pad the atoms to reach the maximum number and perform dynamic LSTM. We tested three types of LSTM inferences: (a) the prediction at the \(max\_num\) time step, (b) the prediction at the \(num\) time step and (c) summing the predictions of the first \(num\) time steps. These three inferences are illustrated in Figure 7 and recognition results are presented in Table 3. The actual number of atoms is recorded by the \(num\) time step and for the videos with atoms less than \(max\_num\), the input of the time steps from \(num+1\) to \(max\_num\) is padded with zero. Therefore, the prediction at the time steps from \(num+1\) to \(max\_num\) is not accuracy. While, the summing method considers the outputs of all the actual atoms and performs best, thus, it is used in our method.  
Figure 6: The structure of long short-term memory (LSTM) unit [45].