# Temporal Modeling on Multi-Temporal-Scale Spatiotemporal Atoms for Action Recognition
## 3 Related Works
### Fusion Operations  
Two fusion operations are used in our method. One is the RGB and OF streams fusion, the other is the multiple temporal scales fusion. First, we discuss the two streams fusion. As illustrated in Figure 8, after the deep spatiotemporal information is extracted from RGB and optical flow fields, there are three possible types of fusions.  
1. Spatiotemporal information fusion (early fusion): The deep spatiotemporal information of the RGB and OF streams are fused. Then action atoms are mined from the fused spatiotemporal information.
2. Atom fusion (middle fusion): The action atoms mined from the deep spatiotemporal information of the RGB and OF streams are fused. Then the fused atoms are fed into LSTM.
3. Prediction fusion (late fusion): In each of the RGB and OF streams, LSTM are performed on the mined atoms. Then the softmax predictions of these two streams are fused.  
For a video action, the information in the RGB and OF streams captures respective characteristic and in each stream, the clips of video action have different temporal dynamic. Therefore, the early fusion and middle fusion, in which the information of RGB and OF streams is fused and the temporal evolution of action in two steams is model using a same LSTM, would not perform as well as the  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Methods** & **Video Accuracy** \\ \hline the max\_num step & 20.4\% \\ the num step & 87.2\% \\ Summation of the first num steps & 87.9\% \\ \hline \hline \end{tabular}
\end{table}
Table 3: Comparison of different LSTM inferences in the RGB stream of 8f temporal scale on UCF101 split-1.  
Figure 7: Three types of LSTM inferences.  
late fusion. Moreover, in the middle fusion, the atoms of RGB and OF streams are mined in different feature spaces. Thus, the atoms in these two streams may correspond to different video clips and the middle fusion would perform worst among these three methods.  
Early and middle fusions are feature-level fusions. We adopt summation, an effective feature-level fusion method for them. The late fusion is prediction-level fusion. We also use summation to fuse the softmax scores of the two streams. We compare the performance of single stream with two streams using different types of fusions in 8f temporal scale. The results are presented in Table 4. It is noticed that the recognition accuracy is improved from 87.9% to 92.2%, 91.5% and 92.4%, respectively, by the early, middle and late fusion, which demonstrates our design that the recognition performance benefits a lot from optical flow. On the other hand, the results also support our assumption that the late fusion outperforms the early and middle fusions and the middle fusion results in the worst performance, although the difference in accuracy among all three types of fusions is less than 1%.  
For the multi-temporal-scale fusion, we sum the prediction of each temporal scale to form the final recognition result. We compare the recognition accuracy of the multi-temporal-scale method with single-temporal-scale methods. Table 5 reports the results, which show that the recognition accuracy can be significantly improved by fusing the prediction of the multiple temporal scales.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Methods** & **Video Accuracy** \\ \hline \multicolumn{2}{c}{**Single Stream**} \\ \hline RGB stream & 87.9\% \\ OF stream & 84.7\% \\ \hline \multicolumn{2}{c}{**Two Streams**} \\ \hline Early fusion & 92.2\% \\ Middle fusion & 91.5\% \\ Late fusion & 92.4\% \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison of single stream with two streams in the 8f temporal scale on UCF101 split-1.  
Figure 8: Fusions of the RGB and OF streams.