Title: WHERE AND WHEN TO LOOK? SPATIO-TEMPORAL ATTENTION FOR ACTION RECOGNITION IN VIDEOS

Abstract: Inspired by the observation that humans are able to process videos efficiently by only paying attention where and when it is needed, we propose a novel spatialtemporal attention mechanism for video-based action recognition. For spatial attention, we learn a saliency mask only using convolutional layers to allow the model to focus on the most salient parts of the feature maps. For temporal attention, we employ a convolutional LSTM based attention mechanism to identify the most relevant frames from an input video. Further, we propose a set of regularizers that ensure that our attention mechanism attends to coherent regions in space and time. Our model can not only effectively improve video action recognition accuracy, but also can localize discriminative regions both spatially and temporally, despite only trained in a weakly-supervised manner with only classification labels (no bounding box spatial labels and time frame temporal labels). We evaluate our proposed approach on several public video action recognition datasets with ablation studies. Furthermore, we quantitatively and qualitatively evaluate our model's ability to localize discriminative regions spatially and critical frames temporally. Experimental results demonstrate the efficacy of our approach, showing superior or comparable accuracy with the state-of-the-art methods with the same input. frame 1 frame 2 frame n CNN CNN CNN

1. INTRODUCTION:

An important property of human perception is that one does not need to process a whole scene in its entirety at once. Instead, humans focus attention selectively on parts of the visual space to acquire information where and when it is needed, and combine information from different fixations over time to build up an internal representation of the scene (Rensink, 2000) , which can then be used for interpretation or decision making.

In computer vision and natural language processing, over the last couple of years, attention models have proved similarly important. Particularly for the tasks where interpretation or explanation requires only a small portion of the image or video. Examples include visual question answering (Lu et al., 2016; Xu & Saenko, 2016; Xiong et al., 2016) , activity recognition (Sharma et al., 2015; Girdhar & Ramanan, 2017; Li et al., 2018b) , and natural machine translation (Bahdanau et al., 2015) . These models have also provided a level of interpretability, by visualizing regions selected or attended over for a particular task or decision. In particular, for video action classification, a proper attention model can help answer the question of where and when it needs to look at the image evidence to draw a classification decision. It intuitively explains which part the model attends to when making a particular decision, which is very helpful in real applications, e.g., medical AI systems or self-driving cars.

In this paper, we propose a novel spatio-temporal attention mechanism that is designed to address these challenges. Our attention mechanism is efficient, due to its space-and time-separability, and yet flexible enough to enable encoding of effective regularizers (or priors). As such, our attention mechanism consists of spatial and temporal components shown in Fig. 1 . The spatial attention component, that attenuates frame-wise CNN image features, consists of the saliency mask; regularized to be discriminative and spatially smooth. The temporal component consists of a uni-modal soft attention mechanism that aggregates information over the near-by attenuated frame features before passing it into Convolutional LSTM for class prediction.

Contributions: In summary, the main contributions of this work are: (1) We introduce a simple yet effective spatial-temporal attention for video action recognition, which consists of the saliency mask 4) Finally, we qualitatively and quantitatively show that our spatio-temporal attention is able to localize discriminative regions and important frames, despite being trained in a purely weakly-supervised manner with only classification labels.

2. RELATED WORK:



2.1. NETWORK INTERPRETATION:

Various methods have been proposed to try to explain neural networks (Zeiler & Fergus, 2014; Springenberg et al., 2014; Mahendran & Vedaldi, 2016; Zhou et al., 2016; Zhang et al., 2016; Simonyan et al., 2013; Ramprasaath et al., 2016; Ribeiro et al., 2016; 2018; Chang et al., 2018) in various of ways, including visualizing the gradients, perturbing the inputs, and bridging relations with other well-studied systems. Visual attention is also one way that tries to explain which part of the image is responsible for the network's decision (Li et al., 2018a; Jetley et al., 2018) . Besides the explanation, Li et al. (2018a) build up an end-to-end model to provide supervision directly on these explanations, specifically network's attention.

2.2. VISUAL ATTENTION FOR VIDEO ACTION RECOGNITION:

For video action recognition, visualizing which part of the frame and which frame of the video sequence that the model was attending to provides valuable insight into the model's behavior. Sharma et al. (2015) develop an attention-driven LSTM by highlighting important spatial locations for action recognition. Girdhar & Ramanan (2017) introduce an attention mechanism based on a derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods. However, these work only focus on the crucial spatial locations of each image, without considering temporal relations among different frames in a video sequence. To alleviate this shortcoming, visual attention is incorporated in the motion stream (Wang et al., 2016b; Li et al., 2018b; Du et al., 2018) . However, the motion stream only employs the optical flow frames generated from consequent frames, cannot consider the long-term temporal relations among different frames in a video sequence. Moreover, motion stream needs additional optical flow frames as input, which imposes burden due to additional optical flow extraction, storage and computation and is especially severe for large datasets. Torabi & Sigal (2017) propose an attention based LSTM model to hightlight frames in videos, but spatial information is not used for temporal attention. An end-to-end spatial

3.4. LOSS FUNCTION:

Considering the spatial and temporal nature of our video action recognition; we would like to learn (1) a sensible attention mask for spatial attention, (2) reasonable importance weighting scores for different frames, and (3) improve the action recognition accuracy at the same time. Therefore, our loss function L:

EQUATION

where L CE is the cross entropy loss for classification, L TV represents the total variation regularization (Rudin et al., 1992) ; L contrast represents the mask and background contrast regularizer; and L unimodal represents unimodality regularizer. λ TV , λ contrast and λ unimodal are the weights for corresponding regularizers.

The total variation regularization L TV of the learnable attention mask encourages spatial smoothness of the mask and is defined as:

L TV = n i=1   j,k |M j+1,k i -M j,k i | + j,k |M j,k+1 i -M j,k i |   (8)

where M i is the mask for the i-th frame, and M j,k i is entry at the (j, k)-th spatial location of the mask. Different from the total variation of the mask of using L 2 loss in Dabkowski & Gal (2017) , we use L 1 loss instead. The contrast regularization L contrast of learnable attention mask is to suppress the irrelevant information and highlight important information:

EQUATION

where B i = I{M i > 0.5} represents the binarized mask, I is the indicator function applied elementwise.

The unimodality regularizer L unimodal encourages the temporal attention weights to be unimodal, biasing against spurious temporal weights. This stems from our observation that in most cases only one activity would be present in the considered frame window, with possible irrelevant information on either or both sides. Here we use the log concave distribution to encourage the unimodal pattern of temporal attention weights:

EQUATION

where T represents the ConvLSTM time sequence length and n is the number of frames for each video. More details on this log concave sequence please refer to Appendix A.

4. EXPERIMENTS:

In this section, we first conduct experiments to evaluate our proposed method on video action recognition task on three public available datasets. Then we evaluate our spatial attention mechanism on the spatial localization task and our temporal attention mechanism on the temporal localization task respectively.

4.1. VIDEO ACTION RECOGNITION:

We first conduct extensive studies on the widely used HMDB51 and UCF101 datasets. The purpose of these experiments is mainly for ablation study to examine the effects of different sub-components.

Then we show that our method can be applied to the challenging large-scale Moments in Time dataset.

Datasets. HMDB51 dataset (Kuehne et al., 2011) contains 51 distinct action categories, each containing at least 101 clips for a total of 6,766 video clips extracted from a wide range of sources. These videos include general facial actions, general body movements, body movements with object interaction, body movements for human interaction.

Model HMDB51 UCF101 Visual attention (Sharma et al., 2015) 41.31 84.96 VideoLSTM (Li et al., 2018b) 43.30 79.60 Attentional Pooling (Girdhar & Ramanan, 2017) UCF101 dataset (Soomro et al., 2012 ) is an action recognition dataset of realistic action videos, collected from YouTube, with 101 action categories.

Moments in Time Dataset (Monfort et al., 2018) is a collection of one million short videos with one action label per video and 339 different action classes. As there could be more than one action taking place in a video, action recognition models may predict an action correctly yet be penalized because the ground truth does not include that action. Therefore, it is believed that top 5 accuracy measure will be more meaningful for this dataset.

Experimental setup. We use the same parameters for HMDB51 and UCF101: single Convolutional LSTM layer with hidden-state dimension 512, sequence length T = 25, λ T V = 10 -5 , λ contrast = 10 -4 , λ unimodal = 1. For the Moments in Time dataset, we use time sequence length T = 15. For more details on the experimental setup please refer to Appendix B.1.

Quantitative results. We show the top-1 video action classification accuracy for HMDB51 and UCF101 dataset in Table 1 . Our proposed model outperforms previous attention based model (Sharma et al., 2015; Li et al., 2018b; Girdhar & Ramanan, 2017) and conventional ResNet101-ImageNet. From the ablation experiments, it demonstrates that all the sub-components of the proposed method contribute to improving the final performance.

The results on the Moments in Time dataset are reported in Table 2 . Our method achieves the best accuracy comparing to other single-modality-based methods, and obtains better or comparative results comparing to the methods which uses more than one modality. TRN-Multiscale (Zhou et al., 2018) , which uses both RGB and optical flow images, has better performance than ours, however, extracting optical flow images for such large datasets is very time-consuming and needs the same order of magnitude of storage as RGB images.

Model Modality:

Top-1 (%) Top-5 (%) ResNet50-ImageNet (Monfort et al., 2018) Qualitative results. We visualize the spatial attention and temporal attention results in Fig. 4 . We can see that the spatial attention can correctly focus on important spatial area of the image, and the temporal attention shows a unimodal distribution for the entire action from starting the action to completing the action. More results are shown in Appendix C.1. 

4.2. WEAKLY SUPERVISED LOCALIZATION:

Due to the existence of spatial and temporal attention mechanisms, our model can not only classify the action of the video, but also give a better interpretability of the results, i.e. telling which region and frames contribute more to the prediction. In other words, our proposed model can also localize the most discriminant region and frames at the same time. To verify this, we conduct the spatial localization and temporal localization experiments.

4.2.1. SPATIAL ACTION LOCALIZATION:

Dataset. UCF101-24 is a subset of 24 classes out of 101 classes of UCF101 that comes with spatio-temporal localization annotation, released as bounding box annotations of humans with THU-MOS2013 and THUMOS2014 challenge (Jiang et al., 2014) .

Experimental setup. For training, we only use the classification labels without spatial bounding box labels. For evaluation, we threshold the produced saliency mask at 0.5 and the tightest bounding box that contains the thresholded saliency map is set as the predicted localization box for each frame. Then these predicted localization boxes are compared with the ground truth bounding boxes at different Intersection Over Union (IOU) levels.

Qualitative results. We show some qualitative results in Fig. 5 . Our spatial attention can attend to important action areas. The ground truth bounding boxes include all the entire human actions, while our attention could attend to crucial parts of an action such as in Fig. 5 (d ) and (e). Furthermore, our attention mechanism is able to attend to areas with multiple human actions. For instance, in Fig. 5 (f) the ground truth only includes one person bicycling, but our attention can include both people bicycling. More qualitative results including failure cases are included in Appendix C.2.

Quantitative results. Table 3 shows the quantitative results for UCF101-24 spatial localization results. Our attention mechanism works better compared with the baseline methods when the IoU threshold is lower mainly because our model only focuses on important spatial areas rather than the entire human action annotated by bounding boxes. Compared with the baseline methods train-Method α = 0.1 α = 0.2 α = 0.3 α = 0.4 α = 0.5 (Yeung et al., 2016) 

5. CONCLUSION:

In this work, we develop a novel spatial-temporal attention mechanism for the task of video action recognition, demonstrating the efficacy across three publicly available datasets. Also, we introduce a set of regularizers that ensure our attention mechanism attends to coherent regions in space and time, further improving the performance and increasing the model interpretability. Moreover, we qualitatively and quantitatively show that our spatio-temporal attention is able to localize discriminative regions and important frames, despite being trained in a purely weakly-supervised manner with only classification labels.

A LOG-CONCAVE SEQUENCE:

In probability and statistics, a unimodal distribution is a probability distribution that has a single peak or mode. If a distribution has more modes it is called multimodal. The temporal attention weights are a univariate discrete distribution over the frames, indicating the importance of the frames for the task of classification. In the context of activity recognition, it is reasonable to assume that the frames that contain salient information should be consecutive, instead of scattered around. Therefore, we would like to design a regularizer that encourages unimodality. To this end, we introduce a mathematical concept called the log-concave sequence and define the regularizer based on it.

We first give a formal definition of the unimodal sequence. Definition 1. A sequence {a i } n i=1 is unimodal if for some integer m,

a i-1 ≤ a i if i ≤ m, a i ≥ a i+1 if i ≥ m.

A univariate discrete distribution is unimodal, if its probability mass function forms a unimodal sequence. The log-concave sequence is defined as follows.

Definition 2. A non-negative sequence

{a i } n i=1 is log-concave if a 2 i ≥ a i-1 a i+1 .

This property gets its name from the fact that if {a i } n i=1 is log-concave, then the sequence {log a k } n i=1 is concave. The connection between unimodality and log-concavity is given by the following proposition. Proposition 1. A log-concave sequence is unimodal.

Proof. Rearranging the defining inequality for log-concavity, we see that a i a i-1 ≥ a i+1 a i , so the ratio of consecutive terms is decreasing. Until the ratios decrease below 1, the sequence is increasing, and after this point, the sequence is decreasing, so it is unimodal. Fig. 10 shows more results on temporal localization with our temporal attention.

:



Figures and Tables:

Figure1: Spatio-temporal attention for video action recognition. The convolutional features are attended over both spatially, in each frame, and subsequently temporally. Both attentions are soft, meaning that the effective final representation at time t of an RNN, used to make the prediction, is a spatio-temporally weighted aggregation of convolutional features across the video along with the past hidden state from t -1. For details please refer to Sec. 3.

Figure 4: Examples of spatial temporal attention. (Best viewed in color.) A frame sequence from a video of Drink action in HMDB51. The original images are shown at the top row, spatial attention is shown as heatmap (red means important) in the middle row, and temporal attention score is shown as the gray image (the brighter the frame is, the more crucial the frame is) at the bottom row. It shows that spatial attention can focus on important areas while temporal attention can attend to crucial frames. The temporal attention also shows a unimodal distribution for the entire action from starting to drink to completing the action.

Figure 7: Multiple actions in one image for video action recognition The Sit action from HMDB51.In the first two frames, there is no sitting action while the spatial attention capture the important area, but the temporal attention can effectively ignore them as the background information. It is interesting that in the last few frames, there is another person trying to sit down, but the visual attention can only capture one sitting person.

Figure 10: Failure example for temporal attention localization. A sequence of Tennis Swing action from one video of THUMOS14. All temporal localization is correctly localized except the frame (b). The labeled action starts from frame (b) but our temporal attention module still assigns a low importance score. Base network ResNet50 ResNet101 ResNet152 w/o spatial-temporal attention 47.5 49.7 50.1 w our spatial-temporal attention 49.8 53.1 54.4

Top-1 accuracy (%) on HMDB51 and UCF101 dataset.

Table content: <table><tr><td>50.80</td><td>-</td></tr></table>

Results on Moments in Time dataset. ResNet50-ImageNet and TRN-Multiscale spatial results reported here are based on authors'(Monfort et al., 2018) released trained model.

Table content: <table><tr><td>RGB</td><td>26.98</td><td>51.74</td></tr></table>

Temporal action localization results on THUMOS'14 dataset measured by mAP at different IoU thresholds α.

Table content: <table><tr><td/><td>48.9%</td><td>44.0%</td><td>36.0%</td><td>26.4%</td><td>17.1%</td></tr><tr><td colspan="2">(Wang et al., 2017) 44.4%</td><td>37.7%</td><td>28.2%</td><td>21.1%</td><td>13.7%</td></tr><tr><td>Ours</td><td>70.0%</td><td>61.4%</td><td>48.6%</td><td>32.6%</td><td>17.9%</td></tr></table>

Top-1 accuracy (%) on HMDB51 with different base networks C.3 MORE ACTION RECOGNITION RESULTS

Table content: <table/>

shows results of our spatial-temporal attention model with different base networks. Our spatial-temporal attention mechanism is a easy plug-in model which could be based on different network architectures, and can boost performance.

Table content: <table/>

Peer review (iclr2019):
 # 1. Summary This paper presents a novel spatio-temporal attention mechanism. The spatial attention is decomposed from the temporal attention and acts on each frame independently, while the temporal attention is applied on top of it on the temporal domain. The main contribution of the paper is the introduction of regularisers that improve performance and interpretability of the model. Strengths: * Quality of the paper, although some points need to clarified and expanded a bit more (see #2) * Nice diversity of experiments, datasets and tasks that the method is tested on (see #4) Weaknesses: * The paper do not present substantial novelty compared to previous work (see #3) # 2. Clarity and Motivation The paper is in general clear and well motivated, however there are few points that need to be improved: * How is the importance mask (Eq. 1) is defined? The authors said “we simply use three convolutional layers to learn the importance mask”, however the convolutional output should be somehow processed to get out the importance map, in order to match the same sizes of X_i. The details of this network are missing to be able to reproduce the model. * The authors introduced .phi(H) and .phi(X) which are feedforward networks, but their definition and specifics are not mentioned in the paper. * It is not clear how Eq. 9 performs regularization of the mask. Can the authors give an intuition about the definition of L_{contrast}? What does it encourages? In which cases might it be useful? * Why does L_{unimodal} need to encourage the temporal attention weights to be unimodal? It seems that the assumption is valid because of the nature of the dataset, i.e., the video clips contain only a single action with some “background” frames in the beginning and the end. This is not valid in general. Can the authors discuss about this maybe with an example? # 3. Novelty The main concern of the proposal in this paper is its novelty. Temporal attention pooling have been explored in other papers; just to cite a popular one among others: * Long, Xiang, et al. "Attention clusters: Purely attention based local feature integration for video classification." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018. * Other paper from the youtube8m workshops explore the same ideas: https://research.google.com/youtube8m/workshop2017/ Sec. 2.2 should be expanded by including papers and discuss how the presented temporal attention differs from that. Moreover spatio-temporal attention has been previously explored. For example, the following paper also decouple the spatial and temporal component as the proposal: * Song, Sijie, et al. "An End-to-End Spatio-Temporal Attention Model for Human Action Recognition from Skeleton Data." AAAI. Vol. 1. No. 2. 2017. This is just an example, but there are there are other papers that model the spatio-temporal extent of videos without attention for action recognition. The authors should expand Sec. 2 by including such relevant literature. # 4. Experimentation The experiments are carried on video action recognition task on three public available datasets, including HMDB51, UCF101 and Moments in Time. The authors show a nice ablation study by removing the main components of the proposed method and show nice improvements with respect to some baseline (Table 1). Although the results are not too close to the state of the art for video action recognition on HMDB51 and UCF101, the authors first show nice accuracy on Moments in Time (Table 2). Moreover the authors show that the model can be useful on the more challenging task of weakly supervised action localization (UCF101-24, THUMOS). Specifically, spatial attention is used to localize the action in each frame by thresholding, showing competitive results (Table 3). Although some more recent references are missing, see the following paper for example: * G. Singh, S Saha, M. Sapienza, P. H. S. Torr and F Cuzzolin. "Online Real time Multiple Spatiotemporal Action Localisation and Prediction." ICCV, 2017. Then the authors tested also for temporal action localization (Table 4). In general, the paper is not showing state-of-the-art results, however the diversity of experiments, datasets and tasks that are presented makes it pretty solid and interesting.