Title: Recurrent Mixture Density Network for Spatiotemporal Visual Attention
Authors: Loris Bazzani
Hugo Larochelle
Lorenzo Torresani
Venue: ICLR 2017
Keywords: Computer vision
Deep learning
Applications

Abstract: In many computer vision tasks, the relevant information to solve the problem at hand is mixed to irrelevant, distracting information. This has motivated researchers to design attentional models that can dynamically focus on parts of images or videos that are salient, e.g., by down-weighting irrelevant pixels. In this work, we propose a spatiotemporal attentional model that learns where to look in a video directly from human fixation data. We model visual attention with a mixture of Gaussians at each frame. This distribution is used to express the probability of saliency for each pixel. Time consistency in videos is modeled hierarchically by: 1) deep 3D convolutional features to represent spatial and short-term time relations and 2) a long short-term memory network on top that aggregates the clip-level representation of sequential clips and therefore expands the temporal domain from few frames to seconds. The parameters of the proposed model are optimized via maximum likelihood estimation using human fixations as training data, without knowledge of the action in each video. Our experiments on Hollywood2 show state-of-the-art performance on saliency prediction for video. We also show that  our attentional model trained on Hollywood2 generalizes well to UCF101 and it can be leveraged to improve action classification accuracy on both datasets.


Review discussion:
Most attention-based models in deep learning are in some sense "unsupervised" attentions, where the attentions are some hidden variables jointly trained with the rest of the network. What is the advantage of the "supervised" attention in this paper, if the goal is to solve some high-level task (e.g. classification)?
The main motivation of the paper is to create a model that is able to mimic human attention. Since human fixations are not specialized for a single task but instead are functional to all forms of our visual processing, we believe that our model may serve as a general-purpose saliency estimator that can be useful for different applications.
Conversely, “unsupervised” attention models learned as part of a supervised task are more specific by design and they may be harder to transfer to other tasks.
We decided to evaluate our model on the task-agnostic task of saliency estimation.
Moreover, we chose to test the model on the high-level task of action recognition because 1) it is a typical problem in the context of video, 2) we wanted to compare with Mathe & Sminchisescu, 2015 and 3) we aimed to show that we can train attention on one dataset (Hollywood2) and use it on another (UCF101).
However, we believe that our model may be useful in other application scenarios, precisely because it has been trained to mimic human attention.
In Table 1 you evaluate among others the AUC metric. Could you clarify which kind of AUC metric you use? I guess it is the AUC metric with a uniform nonfixation distribution (e.g. using more or less all pixels of the saliency map as nonfixated pixels).
Additionally to get a better understanding for how good the model is I think it would be very helpful to see not only the centerbias baseline, but also the performance of the empirical distribution that you show in the video you refer to.
Besides this, as you train the model for log-likelihood, why don't report this metric too? I think it might be very informative, especially for getting and intuition of how predictable the data is.
We used the area under Receiver Operating Characteristic curve (AUC) commonly applied in the context of saliency estimation (Judd et al., 2012; Borji et al., 2013). (Borji et al., 2013) explain it as follows: “human fixations are then used as ground truth. By varying the threshold [on the saliency map], the ROC curve is drawn as the false positive rate vs. true positive rate, and the area under this curve indicates how well the saliency map predicts actual human eye fixations.” Moreover, Fig. 7 in [A] shows a nice visualization that aids comprehension.
We are not 100% sure what you mean by "empirical distribution".
But we guess that the reviewer is effectively after an experiment that would estimate what is the best achievable performance. Is that correct? In this case, we'd argue that the Human performance (reported in Table 2) essentially achieves this objective.
About the log-likelihood metric, it is a good point since it is another way to compare the models with respect to the training objective. We did not report it because we were interested in the final accuracy, in order to look at the standard metrics for saliency estimation and eventually compare with other methods.
[A] X. Hou, J. Harel, and Christof Koch, ”Image Signature: Highlighting sparse salient regions,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012.
Thank you for your answer. You are right, Human performance is exactly what I was refering by emperical distribution. I missed that it is given in Table 2 -- I still think it might make sense to have it in Table 1, too.
How is the model for human performance formulated?
Log-likelihoods would make a good performance metric because they measure how good the model actually is at prediction fixation locations (as opposed to the standard metrics like AUC) and give you a feeling for how peaked the true distribution and the model predictions are.
In Table 1 you report a Central Bias AUC performance of 87.25%, in Table 2 the Central Bias AUC performance for the same dataset is only 84%. What am I missing?
The trained central bias of Table 1 has been trained using all the fixations of the training set (see third paragraph of sec. 4.1 for details).
Instead the central bias of Table 2 is manually created by Mathe & Sminchisescu, 2015 and there is no training at all.
The human performance reported in Table 2 has been taken from Mathe & Sminchisescu, 2015 for comparisons. We can easily put the AUC in Table 1, however we do not have the NSS, the CC and the Sim measurements for the human performance.
Human performance (Mathe & Sminchisescu, 2015) is computed by deriving a saliency map from half of our human subjects and is evaluated with respect to fixations of the remaining ones. From their paper, it seems that the saliency map is computed in the same way as the ground truth saliency map, but considering half of the subjects.
Qualitative results that show how peaked are the predicted distributions can be found in the video available at https://youtu.be/aXOwc17nx_s. Can the reviewer clarify why the log-likelihood would be a good performance metric rather than the well-established measures (AUC, NSS, ...) for saliency prediction?
# Section 3.3
What is the effect of normalising the probability map to sum to one at each time-step? How do you represent the saliency in a frame where nothing is salient? Is that an issue?
# Section 3.4
The rational for concatenating the features extracted from the original clip,
and the features extracted from the saliency weighted clip seems to contradict the initial hypothesis that `eliminating or down-weighting pixels that are not important' will improve performance.
Can you elaborate on this point?
# Section 4.1
Can you send an email to the authors to get their computational timings for a comparison?
# Section 4.2
Concatenating features from two sources will double the SVM model complexity, making the comparison between [(1),(2)] and (3) in Table 3 unfair.
The dimensionality of the features can have a large effect on the ease of classification.
What are your thoughts on that?
Missing references in the introduction.
Paragraphs 1-2, an example of using attention modelling to improve action recognition:
http://www.eleonoravig.com/papers/eccv12_action_recognition.pdf
Paragraph 3, an example of identifying spatio-temporal volumes which are salient to improve action recognition, and an example of joint saliency prediction and action recognition:
http://www.robots.ox.ac.uk/~tvg/publications/2013/sapienza_IJCV13_accepted.pdf
Normalizing the saliency map to sum up to one does not have any major implications. We did it to facilitate evaluation and to have a normalized reweighting of the video for the action recognition task.
In practice, one can use directly the learned Gaussian mixture as probability distribution, to sample fixations for example.
When there is nothing salient in the image (e.g., when the ground truth human fixations are spread and do not show a clear pattern), the model still outputs a saliency map.
But in such cases we empirically observed that the variances of the Gaussian components are quite large, and they collectively approximate the uniform distribution.
The concatenation of the two descriptors produces a representation that 1) includes the general context of the action in the videos and 2) down-weights pixels deemed less important, thus encouraging the recognition model to focus on parts that may be more salient. Since the RMDN is computing anyway the context features for each clip in order to estimate the saliency map, reusing them for action recognition seems straightforward and does not add computations.
We sent an email to the authors requesting their runtimes for comparison. Unfortunately, we cannot promise that they will respond in time.
The SVM model complexity for experiment (3) in Table 3 is indeed twice as large as the complexity for [(1),(2)]. The same applies when comparing (5) against [(1),(4)].
In fact, we added PCA dimensionality reduction to experiment (5) in order to match the same feature dimensionality as (1) (and (4)).
Although the validation accuracies are similar, the testing mAP drops from 54.85% of experiment (5) to 51.82% of the PCA experiment. This is not surprising, since the extra dimensions provided by the use of the saliency map are not redundant with respect to the context representation (1).
Therefore, concatenation seems to be the best way to make use of the saliency map.
Thank you for providing additional references that we missed. We will include them in newer versions of the paper.
The authors of the method we compared in Sec. 4.1 replied to our question about the runtimes. Unfortunately, they did not store this information, since measuring computational cost was not the focus of their work.
Also, we added the suggested references to the new version of the manuscript.
This paper proposes a new method for estimating visual attention in videos. The input clip is first processed by a convnet (in particular, C3D) to extract visual features. The visual features are then passed to LSTM. The hidden state at each time step in LSTM is used to generate the parameters in a Gaussian mixture model.
Finally, the visual attention map is generated from the Gaussian mixture model.
Overall, the idea in this paper is reasonable and the paper is well written. RNN/LSTM has been used in lots of vision problem where the outputs are discrete sequences, there has not been much work on using RNN/LSTM for problems where the output is continuous like in this paper.
The experimental results have demonstrated the effectiveness of the proposed approach.
In particular, it outperforms other state-of-the-art on the saliency prediction task on the Hollywood2 datasets. It also shows improvement over baselines (e.g. C3D + SVM) on the action recognition task.
My only "gripe" of this paper is that this paper is missing some important baseline comparisons.
In particular, it does not seem to show how the "recurrent" part help the overall performance.
Although Table 2 shows RMDN outperforms other state-of-the-art, it might be due to the fact that it uses strong C3D features (while other methods in Table 2 use traditional handcrafted features). Since saliency prediction is essentially a dense image labeling problem (similar to semantic segmentation). For dense image labeling, there has been lots of methods proposed in the past two years, e.g. fully convolution neural network (FCN) or deconvnet. A straightforward baseline is to simply take FCN and apply it on each frame. If the proposed method still outperforms this baseline, we can know that the "recurrent" part really helps.
We agree with the reviewer that it is instructive to separately assess the importance of the recurrent part of our model.
However, we believe that comparing our model with a fully convolutional network or a deconvolutional network would not provide an answer to this question, since these are methods that differ considerably from our proposed model, not only in their lack of recurrence. If the reviewer agrees, we propose instead to run a more informative ablation study, involving the removal of the recurrent link between time t-1 and time t from our model. This will provide a more direct answer concerning the usefulness of recurrency.
We run the ablation study where the recurrent link between time t-1 and time t is removed from our model. The results in terms of AUC are 1.2% and 2.4% lower with respect to the RMDN which uses RNN and LSTM, respectively. We reported this result in the paper at page 7, second paragraph.
This work proposes to a spatiotemporal saliency network that is able to mimic human fixation patterns,
thus helping to prune irrelevant information from the video and improve action recognition.
The work is interesting and has shown state-of-the-art results on predicting human attention on action videos.
It has also shown promise for helping action clip classification.
The paper would benefit from a discussion on the role of context in attention.
For instance, if context is important, and people give attention to context, why is it not incorporated automatically in your model?
One weak point is the action recognition section, where the comparison between the two (1)(2) and (3) seems unfair.
The attention weighted feature maps in fact reduce the classification performance, and only improve performance when doubling the feature and associated model complexity by concatenating the weighted maps with the original features.
Is there a way to combine the context and attention without concatenation?
The rational for concatenating the features extracted from the original clip,
and the features extracted from the saliency weighted clip seems to contradict the initial hypothesis that `eliminating or down-weighting pixels that are not important' will improve performance.
The authors should also mention the current state-of-the-art results in Table 4, for comparison.
# Other comments:
# Abstract
- Typo: `mixed with irrelevant ...'
``Time consistency in videos ... expands the temporal domain from few frames to seconds'' - These two points are not clear, probably need a re-write.
# Contributions
- 1) `The model can be trained without having to engineer spatiotemporal features' - you would need to collect training data from humans though..
# Section 3.1
The number of fixation points is controlled to be fixed for each frame - how is this done?
In practice we freeze the layers of the C3D network to values pretrained by Tran etal.
What happens when you allow gradients to flow back to the C3D layers?
Is it not better to allow the features to be best tuned for the final task?
The precise way in which the features are concatenated needs to be clarified in section 3.4.
Minor typo:
`we added them trained central bias'
The context and attention parts are capturing slightly different information (as confirmed empirically in our study). The context features represent the video in its entirety.
Therefore they may contain elements from the background, which may be useful for action categorization but at times also potentially distracting.
Instead, the attention part focuses only on the most salient spatiotemporal volumes in the video. We agree that there are many other ways to use the saliency maps produced by our model.
For example, the saliency map could be used to sample higher-resolution crops of the frames. We opted for the simplest model in order to show that what is contributing to the improvement in action categorization is truly coming from the saliency map and not from a more complex model.
“Comparison between [(1), (2)] and (3) seems unfair.” As we mentioned in the pre-review answer “Answer to few questions, comments”, we ran an experiment with PCA to yield the same feature dimensionality as (1) in order to have a more fair comparison. The resulting testing mAP of this compressed descriptor (using both context and attention) is 51.82%, which is quite a bit better than method (1) (based on context only), despite having the same dimensionality. We added this relevant result to the paper (last paragraph in page 8).
The number of fixations are randomly subsampled (first paragraph, Sec. 3.1) if greater than A.
Conversely, they are  randomly duplicated if less than A.
In our experiments, we froze the layers of the C3D. We ran several fine-tuning experiments for action recognition using the Hollywood2 dataset, however we have never obtained any significant improvement. We think that this might be due to the small size of the dataset and to the fact that the C3D features are already representative and general since they were trained on a huge dataset. This discouraged us to run fine-tuning experiments for saliency prediction.
About feature concatenation: for each clip, we stacked the d-dimensional context feature vector on top of the d-dimensional attention-based feature vector to form a (2*d)-dimensional descriptor.
The authors formulate a recurrent deep neural network to predict human fixation locations in videos as a mixture of Gaussians. They train the model using maximum likelihood with actual fixation data.
Apart from evaluating how good the model performs at predicting fixations, they combine the saliency predictions with the C3D features for action recognition.
quality: I am missing a more thorough evaluation of the fixation prediction performance. The center bias performance in Table 1 differs significantly from the on in Table 2. All the state-of-the-art models reported in Table 2 have a performance worse than the center bias performance reported in Table 1. Is there really no other model better than the center bias?
Additionally I am missing details on how central bias and human performance are modelled. Is human performance cross-validated?
You claim that your "results are very close to human performance (the difference is only 3.2%). This difference is actually larger than the difference between Central Bias and your model reported in Table 1.
Apart from this, it is dangerous to compare AUC performance differences due to e.g. saturation issues.
clarity: the explanation for Table 3 is a bit confusing, also it is not clear why the CONV5 and the FC6 models differ in how the saliency map is used.
At least one should also evaluate the CONV5 model when multiplying the input with the saliency map to see how much of the difference comes from the different ways to use the saliency map and how much from the different features.
Other issues:
You cite Kümmerer et. al 2015 as a model which "learns ... indirectly rather than from explicit information of where humans look", however the their model has been trained on fixation data using maximum-likelihood.
Apart from these issues, I think the paper make a very interesting contribution to spatio-temporal fixation prediction. If the evaluation issues given above are sorted out, I will happily improve my rating.
As mentioned in the pre-review answer “Answer to Central Bias”, the trained central bias is reported in Table 1 while Table 2 reports the manually-created central bias from Mathe & Sminchisescu, 2015. The trained central bias is used as a comparative simple baseline that does not consider the content of the frames.
Human performance is computed as described below in the pre-review answer “Answer to empirical distribution”. Since the human performance uses fixations to create the saliency maps, we believe that Mathe & Sminchisescu, 2015 tuned the parameters \sigma and p (Sec. 4.1).
However, this information is missing from their paper.
We clarified these points in the new version of the manuscript (last paragraph of Sec. 4.1).
Our results on Table 2 are in between the human performance and the trained central bias. We agree that AUC is not the best metric for evaluation because of the saturation problem.
In fact we used also NSS, CC and Sim to evaluate the variants of our method.
However, we needed to use the AUC to compare with the methods of Mathe & Sminchisescu, 2015.
About the evaluation in Table 3, we wanted to evaluate two different ways to merge the saliency map. We thank the reviewer for pointing out that one relevant baseline is missing: CONV5 when multiplying the input with the saliency map. We are in the process of running this experiment. We will report the result as soon as we obtain it.
Kümmerer et. al 2015 reference was misplaced by mistake.
In fact, their model is trained on fixations for saliency prediction in images. We removed it from the second paragraph of Sec. 2 and correctly placed it in the third paragraph of Sec. 2.
In the new version of the manuscript, we added the experiments that use CONV5 when multiplying the input with the saliency map (second row of Table 3). We also modified the text to discuss the new results (second and fourth paragraph of Sec. 4.2).
We called "feature mode" when the convolutional representation is multiplied by the saliency, and "clip mode" when the saliency maps are used to weight the input video pixels.
In summary: 1) the clip mode preforms better than the feature mode and 2) CONV5 and FC6 give similar results when considering the clip mode, but FC6 is preferred since it provides a more compact representation.

Meta-review:
role: PC
data: comment: The paper describes a model for video saliency prediction using a combination of spatio-temporal ConvNet features and LSTM. The proposed method outperforms the state of the art on the saliency prediction task and is shown to improve the performance of a baseline action classification model.

Decision: Accept (Poster)