# Temporal Modeling on Multi-Temporal-Scale Spatiotemporal Atoms for Action Recognition
## 3 Related Works
### Unsupervised Action Atom Mining  
As basic action units, atoms capture the key spatiotemporal information in video sequences. In this paper, we mine atoms from the deep spatiotemporal information in each temporal scale. Obviously, atoms mining follows the same idea of video summarization, which compactly depicts the original video well using key elements, such as frames [36,37] or clips [38,39]. We propose to extend the maximization of mutual information (MMI) video summarization [40] to action atoms mining in the deep spatiotemporal space and model the temporal evolution of the mined atoms for action recognition. Let \(Y\) be the set of spatiotemporal features for a video action \(A\),\(Y=[y_{1},\ y_{2},\ldots,y_{N}]\), where \(y_{i}\in\mathbb{R}^{n}\) is the Pool5 feature extracted by Res3D, \(N\) is the clip number of action \(A\) and \(n=512\) is the dimension of the Pool5 feature. The summarization technology aims to use a subset \(Y*\) of size \(S\) to represent the original feature sets \(Y\). In the MMI algorithm, the \(Y*\) that most reduces the entropy of the rest of the sets \(Y\backslash Y*\) is selected, which is formulated by  
\[\underset{Y^{*}}{argmax}I(Y^{*};Y\backslash Y*) \tag{1}\]  
This problem is resolved by a simple greedy algorithm that starts with \(Y*=\varnothing\) and iteratively selects a feature \(y*\) from the remaining of the set \(Y\backslash Y*\) until the size of \(Y*\) is \(S\). In each iteration, \(y*\) is selected to provide the maximum increase in mutual information. The objective function is formulated by  
\[\underset{y*\in Y\backslash Y*}{argmax}(H(y*|Y*)-H(y*|\overline{Y}*)) \tag{2}\]  
where \(\overline{Y}*\) denotes \(Y\backslash(Y*\cup y*)\). \(H(y*|Y*)\) forces \(y*\) to be most different from the already selected \(Y*\), while \(-H(y*|\overline{Y}*)\) forces \(y*\) to be most representative among the remaining features. Using the GP model [41], \(H(y*|Y*)\) is evaluated as a closed-form Gaussian conditional entropy:  
\[H(y*|Y*)=\frac{1}{2}log(2\pi eV(y*|Y*)) \tag{3}\]  
and \(V(y*|Y*)\) is evaluated as a closed-form conditional variance:  
\[V(y*|Y*)=K(y*,y*)-K^{T}(y*,Y*)K^{-1}(Y*,Y*)K(y*,Y*) \tag{4}\]  
where \(K\) is the covariance estimation. The objective function Equation (5) can be written in a closed form using Equations (3) and (4):  
\[\underset{y*\in Y\backslash Y*}{argmax}\frac{K(y*,y*)-K^{T}(y*,Y*)K^{-1}(Y*,Y* )K(y*,Y*)}{K(y*,y*)-K^{T}(y*,\overline{Y}*)K^{-1}(\overline{Y}*,\overline{Y}* )K(y*,\overline{Y}*)} \tag{5}\]  
In each temporal scale of our method, to preserve considerably more of the key spatiotemporal information, we split the videos into 50% overlapped clips. We name the non-overlapped split clip in the original Res3D work \"sparse clip\" and name the overlapped split clip in this paper \"dense clip.\" For each video action, we use the trained Res3D to extract the Pool5 feature of the dense clips and use MMI summarization to mine the set of atoms with size \(S\). For the video actions which have the dense clips less than \(S\), the features of all the clips are considered as the atoms. Therefore, the maximum number of atoms associated with an action is \(S\). Figure 5 illustrates the mined spatiotemporal atoms (\(S=4\)) sorted by timestamps and their corresponding clips in the RGB stream of 8f temporal scale. We also use the aforementioned metric video accuracy (Pool5) to measure the performance of unordered modeling on atoms (\(S=16\) and \(S=32\)) and compare it with random selection, sparse split and dense split in the RGB stream of 8f temporal scale. The results presented in Table 2 show that the performance of atoms mining is better than that of random selection and is close to that of the sparse split and densesplit. However, atoms mining can be considered as a kind of simple 'hard weight assignment,' the features mined as atoms are 'assigned' to weight 1 and involved in the recognition.