Title: Adaptive Feature Abstraction for Translating Video to Language
Authors: Yunchen Pu
Martin Renqiang Min
Zhe Gan
Lawrence Carin
Venue: ICLR 2017
Keywords: Computer vision
Deep learning

Abstract: Previous models for video captioning often use the output from a specific layer of a Convolutional Neural Network (CNN) as video representations, preventing them from modeling rich, varying context-dependent semantics in video descriptions. In this paper, we propose a new approach to generating adaptive spatiotemporal representations  of videos for a captioning task.  For this purpose, novel attention mechanisms with spatiotemporal alignment is employed to adaptively and sequentially focus on different layers of CNN features (levels of feature ``abstraction''), as well as local spatiotemporal regions of the feature maps at each layer. Our approach is evaluated on three benchmark datasets: YouTube2Text, M-VAD and MSR-VTT.  Along with visualizing the results and how the model works, these experiments quantitatively demonstrate the effectiveness of the proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantics.


Review discussion:
Nice paper and convincing results! Here are some pre-review questions:
1) As a significant part of the problems you have to solve are due to different dimensions of the feature layers, have you tried a simple baseline consisting of simply reshaping video tensors to the same size? You are already doing it for space, and subsampling frames, so this is not too far from what you are already doing, and it would make these particular issues disappear. Note that this might seem naive, but it often works in practice for action recognition.
Another more complex way to achieve the same effect without reshaping the input tensor would be using dilated convolutions (as is often done for semantic segmentation, I am not aware of works that tried 3D dilated convolutions though, but I have not checked thoroughly).
2) Have you considered comparing to a baseline using hyper-columns (https://arxiv.org/abs/1411.5752)? In a similar spirit to the dichotomy between (spatio-temporal) scale selection vs. multi-scale processing in traditional computer vision (e.g., for interest point detection), hyper-columns is the alternative "feature abstraction combination" to your selection / attention mechanism (which is your main contribution in the context of C3D). It is also straightforwardly applicable to different sizes of feature maps (cf. the original paper).
3) Could you please enrich figure 1 to support a bit better the (lengthy) explanation in section 3.2, especially concerning the projections to \hat{a}_l ?
4) Is the very technical description of hard attention really necessary, as (i) you mostly follow prior works, (ii) this form of attention is not significantly better than the (much simpler) soft attention in almost all your results?
Also, before reaching the middle of the paper, I felt that your description was implying the simultaneous use of both attention mechanisms (which made little sense to me, and this is indeed not what you are doing in the end). I think the hard attention is not bringing much added value to the paper, unless I am missing something.
Thanks for your comments
1. The dimension problem is mainly due to the dimensions of features vary across layers (and also they are not spatiotemporal aligned).
Thus, we cannot utilize standard attention model to select/weight features from different layer directly. If we reshape video tensors to the same size, this problem will still exist.
In addition, the length of each video varies from a few seconds to several minutes. Reshaping video tensors to the same size might result in incapable of capturing rich video contents (reshaping to a small size) or increase in computational cost (reshaping to a large size).
2. We will consider using hyper-columns as suggested.
However, in this paper, we focus on attention mechanism.
3. The attention model (section 3.2) is illustrated in figure 2.
4. The mechanisms behind hard attention and soft attention are different although the quantitative results of them are close. We do not claim hard attention is better than soft attention or vice versa. We believe that they can be a good complement to each other. We will clarify our attention mechanism in the revision.
Thank you for your reply.
1) You are correct. It would be possible, as I suggest, to avoid the work-arounds you introduce to allow for the use of attention, for instance by reshaping, designing an architecture with the same number of channels per layer combined with dilated convolutions (instead of repurposing C3D), and / or pooling.
But I agree this is more work than just an additional baseline. The reason why I was suggesting this direction is because I thought your contribution was more on the "how to allow for the use of attention" rather than the attention mechanisms themselves.
Regarding the time dimension, this is why I said it is naive, but it often works as well as heavy frame subsampling (which you do already) in action recognition.
However, I could see why for captioning it could be different (if the caption length depends on the video length somehow, and if the videos have widely different durations as you mention). Thanks for the clarification.
2) Great! I think the comparison with hyper-columns will be really interesting to show the benefit (or lack thereof) of using attention (which is your main claim).
3) I know :-) I was simply suggesting to make figure 1 a bit more informative, as it is not showing any detail about the attention, and figure 2 is not very explicit either (only showing the sequence from \hat{a} to s to z). A potential way to improve figure 1 (and maybe 2) would be to add attention visualization as done in appendix.
4) How are they complementary? They serve the same purpose, no? Do you use them simultaneously? Sorry, your last comment actually just reinforced my initial confusion...
Thanks for the suggestion of using dilated convolutions. We will consider it as an interesting future work.
However, it cannot be applied directly to our model since it cannot introduce spatiotemporal alignment (Please see Appendix C for details). Such spatiotemporal alignment allow us to quantify the value of attention at a specific spatiotemporal location based on information from all CNN layers. We agree with your comments on our contribution. Actually, our main contribution is “how to leverage features extracted all layers”. Attention is the tool we use to achieve this task.
We do not use hard attention and soft attention simultaneously. We mean that hard attention and soft attention may have their own advantages.
For instance, in the experiments of Xu et. al., hard attention is consistently better than soft attention on the task of image captioning, while we find soft attention sometimes provides better performance for video captioning.
Thanks for your comment 3). We will modify figure 1 highlighting the use of adaptive spatiotemporal feature abstraction to distinguish our work from previous work.
Thanks for suggesting the nice work of hypercolumn representation.
Although hypercolumn representation for pixel classification has similar motivation to our work, the details significantly differ regardless of using attention. Our approach employs an elegant 3D convolution to achieve spatiotemporal alignment across different abstraction levels, but the simple resizing/upsampling technique in hypercolumns might not produce good spatiotemporal representations for video captioning. We will investigate and discuss this new direction in the revision.
1) It seems that this paper is highly related to the approach of Xu et al., except attention is extended to multiple layers, and there are some temporal components due to the video setting.
Therefore, it seems that the problem of attending over vectors of different sizes is the only new technically novel component. Have the authors explored an approach other than filter and max pool for this component? Would it be worth including in the results?
2) Hard attention working a bit worse than Soft attention is at odds with Xu et al, who found hard attention to work slightly better. Any opinions as to what could account for the discrepancy?
Thanks for your comments!
1)	Our model is inspired by Xu et al.
However, the main difference between our model and Xu et al. is that we leverage features from all CNN layers while Xu et al. is done with features from a manually selected CNN layer. Our baseline experiments “Spatiotemporal attention + LSTM” are analogous to Xu et al. Our model quantitatively and qualitatively outperforms these baselines on all three datasets, demonstrating the effectiveness of our abstraction layer attention. An alternative transformation approach is to employ a multi-layer perceptron.
However, we found this approach performs much worse, partially due to the drastic number of parameters (several hundred times larger than 3D convolutional transformation).
2)	The reasons of this discrepancy could be the difference of tasks (Video captioning vs Image captioning) and the characteristics of datasets and extracted features.
In addition, the difference between hard attention and soft attention is usually very small in both our experiments and Xu et al.’s, e.g. the differences of BLEU-4 score is all smaller than 1% on all datasets except Flickr8K.
Yao et al. https://arxiv.org/abs/1502.08029 also proposes a related approach that should be cited.
It appears many of the baselines as well as the proposed approach have very similar encoder - decoder architectures with relatively minor differences. Can you provide further comment on the main insights of the paper and broader impact / value to the community?
Thanks for pointing out this paper. We will discuss it in the revision. Yao et al.
also employed an encoder-decoder network but implemented the encoder with a 3D convolutional neural network instead of mean-pooling or LSTM used in other previous work.
To our knowledge, we are the first to adaptively and sequentially select different CNN layers of features for generating captions. This is our main contribution and distinct from almost all other models. As investigated in Mahendran & Vedaldi and Zeiler & Fergus, the features from layers at or near the top of a CNN tend to focus on global semantic visual percepts, low-layer features provide more local, detailed information. Leveraging features from all layers rather than a manually selected certain layer could generate richer and more descriptive captions. We believe this model also has potential to benefit other visual analysis tasks, e.g., image captioning, video action recognition and visual question answering.
In addition, rather than compressing the rich video content into a single feature vector in most encoder-decoder models, we extend Xu et al. to attend local spatiotemporal regions in feature maps for video caption generation.
1) Summary
This paper proposes a video captioning model based on a 3D (space+time) convnet (C3D) encoder and a LSTM decoder. The authors investigate the benefits of using attention mechanisms operating both at the spatio-temporal and layer (feature abstraction) levels.
2) Contributions
+ Well motivated and implemented attention mechanism to handle the different shapes of C3D feature maps (along space, time, and feature dimensions).
+ Convincing quantitative and qualitative experiments on three challenging datasets (Youtube2Text, M-VAD, MSR-VTT) showing clearly the benefit of the proposed attention mechanisms.
+ Interesting comparison of soft vs hard attention showing a slight performance advantage for the (simpler) soft attention mechanism in this case.
3) Suggestions for improvement
Hypercolumns comparison:
As mentioned during pre-review questions, it would be interesting to compare to the hypercolumns of https://arxiv.org/abs/1411.5752, as they are an alternative to the proposed attention mechanisms, with the same purpose of leveraging different feature abstraction levels.
Minor clarifications in the text and figures as agreed with the authors in our pre-review discussions.
4) Conclusion
Although the novelty with existing video captioning approaches is limited, the paper is relevant to ICLR, as the proposed simple but efficient implementation and benefits of spatio-temporal + feature abstraction attention are clearly validated in this work.
Thanks for your review. We will investigate and compare with the simple hypercolumn representation in the revision.
Following your suggestions,  we updated the introduction, adding three technical challenges to overcome to use adaptive spatiotemporal feature representations with dynamic feature abstraction for the captioning task. We also explained why naïve approaches such as MLP/max(average)-pooling did not meet our requirements.
We updated the related work including the hypercolumn representation.
We put additional experimental results comparing our approach ASTAR to hypercolumns, MLP, and max/average-pooling in Table 1 on Page 8. Our results clearly demonstrate the significance of selecting a specific level. The new results in the paper show that ASTAR works much better than hypercolumns. One possible reason is that hypercolumns result in a lot of parameters for the decoder LSTM and make training harder.
Another reason might be that attending to levels is more effective than concatenating information from all levels without selection/filtering.
We updated Figure 1 and moved Figure 2 below it to emphasize our contributions.
The authors apply the image captioning architecture of Xu et al. 2015 to video captioning. The model is extended to have attention over multiple layers of the ConvNet instead of just a single layer. Experiments on YouTube2Text, M-VAD and MSR-VTT show that this works better than only using one of the layers at a time.
I think this is solid work on the level of a well-executed course project or a workshop paper. The model makes sense, it is adequately described, and the experiments show that attending over multiple layers works better than attending over any one layer in isolation. Unfortunately, I don't think there is enough to get excited about here from a technical perspective and it's not clear what value the paper brings to the community. Other aspects of the paper, such as including the hard attention component, don't seem to add to the paper but take up space.
If the authors want to contribute a detailed, focused exploration of multi-level features this could become a more valuable paper, but in that case I would expect a much more thorough exploration of the choices and tradeoffs of different schemes without too many spurious aspects such as video features, hard attention, etc.
Thanks a lot for your review.
Employing an efficient 3D convolution to achieve spatiotemporal alignment for the attention mechanism is our major technical innovation and the key to achieve impressive performance, which involves precisely calibrating the receptive fields of different convnet layers. This efficient approach informs the representation learning community of a new way of considering multiple-level CNN representations other than hypercolumn representations or our failed MLP attempt. Before applying hard attention even with enhanced multi-sample Monte Carlo objective, we do not know how well it performs on video captioning, a completely different task from image captioning. Reporting a different performance observation from previous work about hard attention vs. soft attention is valuable. All these findings are important to know by the community.
Simply extending Xu et al.'s architecture to video captioning without considering the nature of video data has already been explored by a Master thesis (http://shikharsharma.com/publications/pdfs/msc-thesis.pdf), but their performance is much worse than previously established methods, and far from the state-of-the-art result that we achieved. We will cite this work in the revision for comparison to show the significance of our attention even with a similarly simple encoder-decoder framework. Considering that the three datasets are very challenging, our performance with a simple model without complexities such as hierarchical RNN or a 3D/2D combination is very impressive compared to previous results.
Exploring feature representation across layers on a standard CNN for classification independent of applications is a good suggestion, but it leads to a completely different paper. The current paper aims for video captioning and achieved the state-of-the-art performance.
Following your suggestion, we explored different ways of utilizing multi-level features. We put additional experimental results comparing our approach ASTAR to hypercolumns, MLP, and max/average-pooling in Table 1 on Page 8. Our results clearly demonstrate the significance of having adaptive feature abstraction across levels and the advantages of our specific way to achieve it. Naive approaches have much worse performance than our approach.
We updated the abstract, emphasizing that the goal of our paper is not only about presenting another architecture for video captioning, but also about new video representations that are suitable for the video captioning task. Attention mechanisms are just technical means to achieve this goal.
We updated the introduction, adding three technical challenges to overcome to use adaptive spatiotemporal feature representations with dynamic feature abstraction for the captioning task. We also explained why naïve approaches such as MLP/max(average)-pooling did not meet our requirements.
This paper presents a model for video captioning with both soft and hard attention, using a C3D network for the encoder and a RNN for the decoder. Experiments are presented on YouTube2Text, M-VAD, and MSR-VTT.
While the ideas of image captioning with soft and hard attention, and video captioning with soft attention, have already been demonstrated in previous work, the main contribution here is the specific architecture and attention over different layers of the CNN.
The work is well presented and the experiments clearly show the benefit of attention over multiple layers.
However, in light of previous work in captioning, the contribution and resulting insights is too incremental for a conference paper at ICLR.
Further experiments and analysis of the main contribution would strengthen the paper, but I would recommend resubmission to a more suitable venue.
"Your result is super, but sorry, I just don't like your paper. Q.E.D." :-)
We updated the abstract, emphasizing that the goal of our paper is not only about presenting another architecture for video captioning, but also about new video representations that are suitable for the video captioning task. Attention mechanisms are just technical means to achieve this goal.
We updated the introduction, adding three technical challenges to overcome to use adaptive spatiotemporal feature representation with dynamic feature abstraction for the captioning task. We also explained why naïve approaches such as MLP/max(average)-pooling did not meet our requirements. Our updated experimental results support our claims.
We believe that our contributions are clear and important to know by the representation learning community. This line of thinking might influence other researchers to perform additional research on classification and other tasks. It also inspires us to design new deep architectures to efficiently learn and effectively utilize different levels of feature representations in a dynamic fashion.
We thank all the reviewers for the critical comments.
All the reviewers agree that our experimental results convincingly support our hypothesis. The disagreement is that whether our paper is suitable for ICLR and whether our contribution is important to know by the representation learning community.
We made the following changes in the pdf file revision addressing the reviewers’ concerns:
1. We updated the abstract, emphasizing that the goal of our paper is not only about presenting another architecture for video captioning, but also about new video representations that are suitable for the video captioning task. Attention mechanisms are just technical means to achieve this goal.
2. We updated the introduction, adding three technical challenges to overcome to use adaptive spatiotemporal feature representations with dynamic feature abstraction for the captioning task. We also explained why naïve approaches such as MLP/max(average)-pooling did not meet our requirements.
3. We updated the related work including the hypercolumn representation.
4. We put additional experimental results comparing our approach ASTAR to hypercolumns, MLP, and max/average-pooling in Table 1 on Page 8. Our results clearly demonstrate the significance of dynamically selecting a specific level. The hypercolumn representation without level selection has much worse performance than our method.
5. We updated Figure 1 and moved Figure 2 below it to emphasize our contributions following the review comments.
6. We added the reference of Yao et al., ICCV 2015.
In summary, we believe that our contributions are clear and important to know by the representation learning community. This line of thinking might influence other researchers to perform additional research on classification and other tasks. It also inspires us to design new deep architectures to efficiently learn and effectively utilize different levels of feature representations in a dynamic fashion.
Thank you for your revision and comments. It addresses most of my concerns.
Even though I agree with the other reviewers that novelty is limited, the experimental conclusions are clear and should be beneficial to the community.
At least one person (me) learned something (e.g., the potential importance of dynamic attention on a specific layer).
So I will maintain my rating.
Good luck!

Meta-review:
role: PC
data: comment: Reviewers feel the work is well executed and that the model makes sense, but two of the reviewers were not convinced that the proposed method contains enough novelty in light of prior work. The comparison of the soft vs hard attention model variations is perhaps one of the more novel aspects of the work; however, the degree of novelty within these formulations and the insights obtained from their comparison were not perceived as being enough to warrant higher ratings. We would like to invite the authors to submit this paper to the workshop track.

Decision: Invite to Workshop Track