Year: 2018

Abstract:
Inspired by the observation that humans are able to process videos efficiently by only paying attention where and when it is needed, we propose a novel spatialtemporal attention mechanism for video-based action recognition. For spatial attention, we learn a saliency mask only using convolutional layers to allow the model to focus on the most salient parts of the feature maps. For temporal attention, we employ a convolutional LSTM based attention mechanism to identify the most relevant frames from an input video. Further, we propose a set of regularizers that ensure that our attention mechanism attends to coherent regions in space and time. Our model can not only effectively improve video action recognition accuracy, but also can localize discriminative regions both spatially and temporally, despite only trained in a weakly-supervised manner with only classification labels (no bounding box spatial labels and time frame temporal labels). We evaluate our proposed approach on several public video action recognition datasets with ablation studies. Furthermore, we quantitatively and qualitatively evaluate our model’s ability to localize discriminative regions spatially and critical frames temporally. Experimental results demonstrate the efficacy of our approach, showing superior or comparable accuracy with the state-of-the-art methods with the same input.

heading: 1 INTRODUCTION
text: An important property of human perception is that one does not need to process a whole scene in its entirety at once. Instead, humans focus attention selectively on parts of the visual space to acquire information where and when it is needed, and combine information from different fixations over time to build up an internal representation of the scene (Rensink, 2000), which can then be used for interpretation or decision making.
In computer vision and natural language processing, over the last couple of years, attention models have proved similarly important. Particularly for the tasks where interpretation or explanation requires only a small portion of the image or video. Examples include visual question answering (Lu et al., 2016; Xu & Saenko, 2016; Xiong et al., 2016), activity recognition (Sharma et al., 2015; Girdhar & Ramanan, 2017; Li et al., 2018b), and natural machine translation (Bahdanau et al., 2015). These models have also provided a level of interpretability, by visualizing regions selected or attended over for a particular task or decision. In particular, for video action classification, a proper attention model can help answer the question of where and when it needs to look at the image evidence to draw a classification decision. It intuitively explains which part the model attends to when making a particular decision, which is very helpful in real applications, e.g., medical AI systems or self-driving cars.
In this paper, we propose a novel spatio-temporal attention mechanism that is designed to address these challenges. Our attention mechanism is efficient, due to its space- and time- separability, and yet flexible enough to enable encoding of effective regularizers (or priors). As such, our attention mechanism consists of spatial and temporal components shown in Fig. 1. The spatial attention component, that attenuates frame-wise CNN image features, consists of the saliency mask; regularized to be discriminative and spatially smooth. The temporal component consists of a uni-modal soft attention mechanism that aggregates information over the near-by attenuated frame features before passing it into Convolutional LSTM for class prediction.
Contributions: In summary, the main contributions of this work are: (1) We introduce a simple yet effective spatial-temporal attention for video action recognition, which consists of the saliency mask
for spatial attention learned by ConvNets and temporal attention learned by convolutional LSTM. (2) We introduce three different regularizers, two for spatial and one for temporal attention components, to improve performance and interpretability of our model; (3) We demonstrate the efficacy of our model for video action recognition in three public datasets and explore the importance of our modeling choices through ablation experiments; (4) Finally, we qualitatively and quantitatively show that our spatio-temporal attention is able to localize discriminative regions and important frames, despite being trained in a purely weakly-supervised manner with only classification labels.

heading: 2 RELATED WORK

heading: 2.1 NETWORK INTERPRETATION
text: Various methods have been proposed to try to explain neural networks (Zeiler & Fergus, 2014; Springenberg et al., 2014; Mahendran & Vedaldi, 2016; Zhou et al., 2016; Zhang et al., 2016; Simonyan et al., 2013; Ramprasaath et al., 2016; Ribeiro et al., 2016; 2018; Chang et al., 2018) in various of ways, including visualizing the gradients, perturbing the inputs, and bridging relations with other well-studied systems. Visual attention is also one way that tries to explain which part of the image is responsible for the network’s decision (Li et al., 2018a; Jetley et al., 2018). Besides the explanation, Li et al. (2018a) build up an end-to-end model to provide supervision directly on these explanations, specifically network’s attention.

heading: 2.2 VISUAL ATTENTION FOR VIDEO ACTION RECOGNITION
text: For video action recognition, visualizing which part of the frame and which frame of the video sequence that the model was attending to provides valuable insight into the model’s behavior. Sharma et al. (2015) develop an attention-driven LSTM by highlighting important spatial locations for action recognition. Girdhar & Ramanan (2017) introduce an attention mechanism based on a derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods. However, these work only focus on the crucial spatial locations of each image, without considering temporal relations among different frames in a video sequence. To alleviate this shortcoming, visual attention is incorporated in the motion stream (Wang et al., 2016b; Li et al., 2018b; Du et al., 2018). However, the motion stream only employs the optical flow frames generated from consequent frames, cannot consider the long-term temporal relations among different frames in a video sequence. Moreover, motion stream needs additional optical flow frames as input, which imposes burden due to additional optical flow extraction, storage and computation and is especially severe for large datasets. Torabi & Sigal (2017) propose an attention based LSTM model to hightlight frames in videos, but spatial information is not used for temporal attention. An end-to-end spatial
and temporal attention model is proposed in (Song et al., 2017) for human action recognition, but additional skeleton data is needed.

heading: 3 SPATIAL-TEMPORAL ATTENTION MECHANISM
text: Our overall model is an Recurrent Neural Network (RNN) that aggregates frame-based convolutional features across the video to make action predictions as shown in Fig. 1. The convolutional features are attended over both spatially, in each frame, and subsequently temporally. Both attentions are soft, meaning that the effective final representation at time t of an RNN, used to make the prediction, is a spatio-temporally weighted aggregation of convolutional features across the video along with the past hidden state from t − 1. The core novelty is the overall form of our attention mechanism and the additional terms of the loss function that induce sensible spatial and temporal attention priors.

heading: 3.1 CONVOLUTIONAL FRAME FEATURES
text: We use the last convolutional layer output extracted by ResNet50 or ResNet101 (He et al., 2016), pretrained on the ImageNet (Deng et al., 2009) dataset and fine-tuned for the target dataset, as our frame feature representation. We acknowledge that more accurate feature extractors (for instance, network with more parameters such as ResNet-152 or higher performance networks such as DenseNet (Huang et al., 2017) or SENet (Hu et al., 2018)) and optical flow features will likely lead to better overall performance. Our primary purpose in this paper is to prove the efficacy of our spatial-temporal attention mechanism. Hence we kept the features relatively simple.

heading: 3.2 SPATIAL ATTENTION WITH IMPORTANCE MASK
text: We apply an importance mask Mi to the i-th image features Xi to obtain attended image features by element-wise multiplication:
X̃i = Xi Mi, (1)
for 1 ≤ i ≤ n. This operation attenuates certain regions of the feature map based on their estimated importance. Here we simply use three convolutional layers to learn the importance mask (please refer to Appendix B.2 for network architecture details). Fig. 2 illustrates our spatial attention mechanism. However, if left uncontrolled, an arbitrarily structured mask could be learned, leading to possible overfitting. We posit that, in practice, it is often useful to attend to a few important larger regions (e.g., objects, elements of the scene). To induce this behavior, we encourage smoothness of the mask by introducing total variation loss on the spatial attention, as will be described in Sec. 3.4.

heading: 3.3 TEMPORAL ATTENTION
text: Inspired by attention for neural machine translation (Bahdanau et al., 2015), we introduce the temporal attention mechanism which generates energy for each attended frame X̃i at each time step t,
eti = Φ(Ht−1, X̃i), (2)
where Ht−1 represents the ConvLSTM hidden state at time t−1 that implicitly contains all previous information up to time step t− 1, X̃i represents the i-th frame masked features. Φ = ΦH(Ht−1) + ΦX(X̃i), where ΦH and ΦX are feed-forward neural networks which are jointly trained with all other components of the proposed system.
This temporal attention model directly computes soft attention weight for each frame at each time t as shown in Fig. 3. It allows the gradient of the cost function to be backpropagated through. This gradient can be used to train the entire spatial-temporal attention model jointly.
The importance weight wti for each frame is:
wti = exp(eti)∑n
i=1(exp(eti)) (3)
for 1 ≤ i ≤ n, 1 ≤ t ≤ T . This importance weighting mechanism decides which frame of the video to pay attention to. The final feature map Yt to the ConvLSTM is a weighted sum of the feature from all of the frames as the ConvLSTM cell inputs:
Yt = 1
n n∑ i=1 wtiX̃i (4)
where X̃i denotes the i-th masked frame of each video, n represents the total number of frames for each video.
For RNN, instead of using conventional LSTM (Graves, 2013), we use Convolutional LSTM (ConvLSTM) (Shi et al., 2015) instead. The drawback of conventional LSTM is its use of full connections in the input-to-state and state-to-state transitions in which no spatial information is encoded. In contrast, each input, cell output, hidden state, gate are 3D tensors whose last two dimensions are spatial dimensions which can preserve spatial information, which is more suitable for image inputs.
We use the following initialization strategy for the ConvLSTM cell state and hidden state for faster convergence:
C0 = gc( 1
n n∑ i=1 X̃i), H0 = gh( 1 n n∑ i=1 X̃i) (5)
where gc and gh are two layer convolutional networks with batch normalization (Ioffe & Szegedy, 2015).
We calculate the average hidden states of ConvLSTM over time length T ,
H = 1
T T∑ i=1 Hi (6)
and send it to a fully connected classification layer for the final video action classification.

heading: 3.4 LOSS FUNCTION
text: Considering the spatial and temporal nature of our video action recognition; we would like to learn (1) a sensible attention mask for spatial attention, (2) reasonable importance weighting scores for different frames, and (3) improve the action recognition accuracy at the same time. Therefore, our loss function L:
L = LCE + λTVLTV + λcontrastLcontrast + λunimodalLunimodal, (7)
where LCE is the cross entropy loss for classification, LTV represents the total variation regularization (Rudin et al., 1992); Lcontrast represents the mask and background contrast regularizer; and Lunimodal represents unimodality regularizer. λTV, λcontrast and λunimodal are the weights for corresponding regularizers.
The total variation regularizationLTV of the learnable attention mask encourages spatial smoothness of the mask and is defined as:
LTV = n∑ i=1 ∑ j,k |M j+1,ki −M j,k i |+ ∑ j,k |M j,k+1i −M j,k i |  (8) where Mi is the mask for the i-th frame, and M j,k i is entry at the (j, k)-th spatial location of the mask. Different from the total variation of the mask of using L2 loss in Dabkowski & Gal (2017), we use L1 loss instead. The contrast regularization Lcontrast of learnable attention mask is to suppress the irrelevant information and highlight important information:
Lcontrast = n∑ i=1 ( −1 2 Mi Bi + 1 2 Mi (1−Bi) ) (9)
where Bi = I{Mi > 0.5} represents the binarized mask, I is the indicator function applied elementwise.
The unimodality regularizer Lunimodal encourages the temporal attention weights to be unimodal, biasing against spurious temporal weights. This stems from our observation that in most cases only one activity would be present in the considered frame window, with possible irrelevant information on either or both sides. Here we use the log concave distribution to encourage the unimodal pattern of temporal attention weights:
Lunimodal = T∑ t=1 n−1∑ i=2 max{0, wt,i−1wt,i+1 − w2t,i} (10)
where T represents the ConvLSTM time sequence length and n is the number of frames for each video. More details on this log concave sequence please refer to Appendix A.

heading: 4 EXPERIMENTS
text: In this section, we first conduct experiments to evaluate our proposed method on video action recognition task on three public available datasets. Then we evaluate our spatial attention mechanism on the spatial localization task and our temporal attention mechanism on the temporal localization task respectively.

heading: 4.1 VIDEO ACTION RECOGNITION
text: We first conduct extensive studies on the widely used HMDB51 and UCF101 datasets. The purpose of these experiments is mainly for ablation study to examine the effects of different sub-components. Then we show that our method can be applied to the challenging large-scale Moments in Time dataset.
Datasets. HMDB51 dataset (Kuehne et al., 2011) contains 51 distinct action categories, each containing at least 101 clips for a total of 6,766 video clips extracted from a wide range of sources. These videos include general facial actions, general body movements, body movements with object interaction, body movements for human interaction.

heading: Model HMDB51 UCF101

heading: Ablation Experiments
text: UCF101 dataset (Soomro et al., 2012) is an action recognition dataset of realistic action videos, collected from YouTube, with 101 action categories.
Moments in Time Dataset (Monfort et al., 2018) is a collection of one million short videos with one action label per video and 339 different action classes. As there could be more than one action taking place in a video, action recognition models may predict an action correctly yet be penalized because the ground truth does not include that action. Therefore, it is believed that top 5 accuracy measure will be more meaningful for this dataset.
Experimental setup. We use the same parameters for HMDB51 and UCF101: single Convolutional LSTM layer with hidden-state dimension 512, sequence length T = 25, λTV = 10−5, λcontrast = 10−4, λunimodal = 1. For the Moments in Time dataset, we use time sequence length T = 15. For more details on the experimental setup please refer to Appendix B.1.
Quantitative results. We show the top-1 video action classification accuracy for HMDB51 and UCF101 dataset in Table 1. Our proposed model outperforms previous attention based model (Sharma et al., 2015; Li et al., 2018b; Girdhar & Ramanan, 2017) and conventional ResNet101ImageNet. From the ablation experiments, it demonstrates that all the sub-components of the proposed method contribute to improving the final performance.
The results on the Moments in Time dataset are reported in Table 2. Our method achieves the best accuracy comparing to other single-modality-based methods, and obtains better or comparative results comparing to the methods which uses more than one modality. TRN-Multiscale (Zhou et al., 2018), which uses both RGB and optical flow images, has better performance than ours, however, extracting optical flow images for such large datasets is very time-consuming and needs the same order of magnitude of storage as RGB images.
Qualitative results. We visualize the spatial attention and temporal attention results in Fig. 4. We can see that the spatial attention can correctly focus on important spatial area of the image, and the temporal attention shows a unimodal distribution for the entire action from starting the action to completing the action. More results are shown in Appendix C.1.

heading: 4.2 WEAKLY SUPERVISED LOCALIZATION
text: Due to the existence of spatial and temporal attention mechanisms, our model can not only classify the action of the video, but also give a better interpretability of the results, i.e. telling which region and frames contribute more to the prediction. In other words, our proposed model can also localize the most discriminant region and frames at the same time. To verify this, we conduct the spatial localization and temporal localization experiments.

heading: 4.2.1 SPATIAL ACTION LOCALIZATION
text: Dataset. UCF101-24 is a subset of 24 classes out of 101 classes of UCF101 that comes with spatio-temporal localization annotation, released as bounding box annotations of humans with THUMOS2013 and THUMOS2014 challenge (Jiang et al., 2014).
Experimental setup. For training, we only use the classification labels without spatial bounding box labels. For evaluation, we threshold the produced saliency mask at 0.5 and the tightest bounding box that contains the thresholded saliency map is set as the predicted localization box for each frame. Then these predicted localization boxes are compared with the ground truth bounding boxes at different Intersection Over Union (IOU) levels.
Qualitative results. We show some qualitative results in Fig. 5. Our spatial attention can attend to important action areas. The ground truth bounding boxes include all the entire human actions, while our attention could attend to crucial parts of an action such as in Fig.5 (d) and (e). Furthermore, our attention mechanism is able to attend to areas with multiple human actions. For instance, in Fig.5 (f) the ground truth only includes one person bicycling, but our attention can include both people bicycling. More qualitative results including failure cases are included in Appendix C.2.
Quantitative results. Table 3 shows the quantitative results for UCF101-24 spatial localization results. Our attention mechanism works better compared with the baseline methods when the IoU threshold is lower mainly because our model only focuses on important spatial areas rather than the entire human action annotated by bounding boxes. Compared with the baseline methods train-
ing with ground truth bounding boxes, we only use the action classification label, no ground truth bounding boxes are used.

heading: 4.2.2 TEMPORAL ACTION LOCALIZATION
text: Dataset. The action detection task of THUMOS14 (Jiang et al., 2014) consists of 20 classes of sports activities, and contains 2765 trimmed videos for training, while 200 and 213 untrimmed videos for validation and test respectively. More details on this dataset and pre-processing are included in Appendix B.1.
Experimental setup. We use the same hyperparameters for THUMOS14 as HMDB51, UCF101 and UCF101-24. For training, we only use the classification labels without temporal annotation labels. For evaluation, we threshold the normalized temporal attention importance weight at 0.5. Then these predicted temporal localization frames are compared with the ground truth annotation at different IoU thresholds.
Qualitative results. We first visualize some examples of learned attention weights on the test data of THUMOS14 in Fig. 6. We see that our temporal attention module is able to automatically highlight important frames and to avoid irrelevant frames corresponding to background or non-action human poses. More qualitative results are included in Appendix C.4.
Quantitative results. With our spatial temporal attention mechanism, the video action classification accuracy for the THUMOS’14 20 classes improved from 74.45% to 78.33%: a 3.88% increase. Besides improving the classification accuracy, we show our temporal attention mechanism is able to highlight discriminative frames quantitatively in Table 4. Compared with reinforcement learning based method (Yeung et al., 2016) and weakly supervised method (Wang et al., 2017), our method achieves the best accuracy in terms of different levels of IoU thresholds.

heading: 5 CONCLUSION
text: In this work, we develop a novel spatial-temporal attention mechanism for the task of video action recognition, demonstrating the efficacy across three publicly available datasets. Also, we introduce a set of regularizers that ensure our attention mechanism attends to coherent regions in space and time, further improving the performance and increasing the model interpretability. Moreover, we qualitatively and quantitatively show that our spatio-temporal attention is able to localize discriminative regions and important frames, despite being trained in a purely weakly-supervised manner with only classification labels.

heading: A LOG-CONCAVE SEQUENCE
text: In probability and statistics, a unimodal distribution is a probability distribution that has a single peak or mode. If a distribution has more modes it is called multimodal. The temporal attention weights are a univariate discrete distribution over the frames, indicating the importance of the frames for the task of classification. In the context of activity recognition, it is reasonable to assume that the frames that contain salient information should be consecutive, instead of scattered around. Therefore, we would like to design a regularizer that encourages unimodality. To this end, we introduce a mathematical concept called the log-concave sequence and define the regularizer based on it.
We first give a formal definition of the unimodal sequence. Definition 1. A sequence {ai}ni=1 is unimodal if for some integer m,{
ai−1 ≤ ai if i ≤ m, ai ≥ ai+1 if i ≥ m.
A univariate discrete distribution is unimodal, if its probability mass function forms a unimodal sequence. The log-concave sequence is defined as follows. Definition 2. A non-negative sequence {ai}ni=1 is log-concave if a2i ≥ ai−1ai+1.
This property gets its name from the fact that if {ai}ni=1 is log-concave, then the sequence {log ak}ni=1 is concave. The connection between unimodality and log-concavity is given by the following proposition. Proposition 1. A log-concave sequence is unimodal.
Proof. Rearranging the defining inequality for log-concavity, we see that ai ai−1 ≥ ai+1 ai ,
so the ratio of consecutive terms is decreasing. Until the ratios decrease below 1, the sequence is increasing, and after this point, the sequence is decreasing, so it is unimodal.
Given the definition of log-concavity, it is straightforward to design a regularization term that encourages log-concavity:
R = n−1∑ i=2 max{0, ai−1ai+1 − a2i }. (11)
By Proposition 1, this regularizer also encourages unimodality.

heading: B MORE DATASETS AND IMPLEMENTATION DETAILS

heading: B.1 MORE DETAILS ON THE DATASET AND EXPERIMENTAL SETUP
text: HMDB51 and UCF101 The dataset pre-processing and data augumentation are the same as the ResNet ImageNet experiment (He et al., 2016). All the videos are resized to 224 × 224 resolution and fed into a ResNet-50 pretrained on ImageNet. The last convolutional layer feature map size is 2048×7×7. The experimental setup for the Moments in Time dataset is the same as HMDB51 and UCF101 except time sequence and image resolution.
Moments in Time For the Moments in Time dataset (Monfort et al., 2018), the videos only have 3 seconds, much shorter than HMDB51 and UCF101. We extract RGB frames from the raw videos at 5 fps. Therefore, the sequence length is T = 15. Following the practice in (Monfort et al., 2018) to make all the videos uniform resolution, we resize the RGB frames to 340 × 256 pixels. When extracting features, we use the ResNet-50 pretrained on ImageNet model using resized images with resolution of 256 × 256 pixels. The data augmentation is the same as the ResNet ImageNet experiment (He et al., 2016). The feature map size of the last convolutional layer is 2048× 8× 8. THUMOS14 The action detection task of THUMOS’14 (Jiang et al., 2014) consists of 20 classes of sports activities, and contains 2765 trimmed videos for training, while 200 and 213 untrimmed videos for validation and test respectively. Following the standard practice (Yeung et al., 2016; Zhao et al., 2017), we use the validation set as training and evaluate on the testing set. Following the standard practice (Xu et al., 2017) to avoid the training ambiguity, we remove the videos with multiple labels. We extract RGB frames from the raw videos at 10 fps. The last convolutional layer feature map size is 2048× 7× 7.

heading: B.2 SPATIAL ATTENTION NETWORK ARCHITECTURE
text: The detailed architecture of spatial attention network described in Section 3.2 is listed in the Table 5.

heading: B.3 MORE IMPLEMENTATION DETAILS
text: All the experiments are evaluated on machines with a single Nvidia GeForce GTX 1080Ti GPU. The networks are implemented using the Pytorch library and our code will be publicly available with the paper.

heading: C MORE RESULTS

heading: C.1 MORE SPATIAL TEMPORAL ATTENTION RESULTS
text: Fig. 7 shows more results on spatial temporal attention.

heading: C.2 MORE SPATIAL LOCALIZATION RESULTS
text: Fig. 8 shows more spatial localization results. Fig. 9 shows some failure cases.

heading: C.3 MORE ACTION RECOGNITION RESULTS
text: Table 6 shows results of our spatial-temporal attention model with different base networks. Our spatial-temporal attention mechanism is a easy plug-in model which could be based on different network architectures, and can boost performance.

heading: C.4 MORE TEMPORAL LOCALIZATION RESULTS
text: Fig. 10 shows more results on temporal localization with our temporal attention.

Review summaries:
The paper propose an end-to-end technique that applies both spatial and temporal attention. The spatial attention is done by training a mask-filter, while the temporal-attention use a soft-attention mechanism.  In addition the authors propose several regularization terms  to directly improve attention. The evaluated datasets are action recognition datasets, such as HMDB51, UCF10, Moments in Time, THUMOS’14. The paper reports SOTA on all three datasets. 

This paper presents a novel spatio-temporal attention mechanism. The spatial attention is decomposed from the temporal attention and acts on each frame independently, while the temporal attention is applied on top of it on the temporal domain. The main contribution of the paper is the introduction of regularisers that improve performance and interpretability of the model.