Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: Nice paper and convincing results! Here are some pre-review questions:

1) As a significant part of the problems you have to solve are due to different dimensions of the feature layers, have you tried a simple baseline consisting of simply reshaping video tensors to the same size? You are already doing it for space, and subsampling frames, so this is not too far from what you are already doing, and it would make these particular issues disappear. Note that this might seem naive, but it often works in practice for action recognition. Another more complex way to achieve the same effect without reshaping the input tensor would be using dilated convolutions (as is often done for semantic segmentation, I am not aware of works that tried 3D dilated convolutions though, but I have not checked thoroughly).

2) Have you considered comparing to a baseline using hyper-columns (https://arxiv.org/abs/1411.5752)? In a similar spirit to the dichotomy between (spatio-temporal) scale selection vs. multi-scale processing in traditional computer vision (e.g., for interest point detection), hyper-columns is the alternative "feature abstraction combination" to your selection / attention mechanism (which is your main contribution in the context of C3D). It is also straightforwardly applicable to different sizes of feature maps (cf. the original paper).

3) Could you please enrich figure 1 to support a bit better the (lengthy) explanation in section 3.2, especially concerning the projections to \hat{a}_l ?

4) Is the very technical description of hard attention really necessary, as (i) you mostly follow prior works, (ii) this form of attention is not significantly better than the (much simpler) soft attention in almost all your results? Also, before reaching the middle of the paper, I felt that your description was implying the simultaneous use of both attention mechanisms (which made little sense to me, and this is indeed not what you are doing in the end). I think the hard attention is not bringing much added value to the paper, unless I am missing something.
review_title: Pre-review questions

final_score: rating: null
review_content: Yao et al. https://arxiv.org/abs/1502.08029 also proposes a related approach that should be cited.

It appears many of the baselines as well as the proposed approach have very similar encoder - decoder architectures with relatively minor differences. Can you provide further comment on the main insights of the paper and broader impact / value to the community?
review_title: related work and broader impact

final_score: rating: 7: Good paper, accept
review_content: 1) Summary

This paper proposes a video captioning model based on a 3D (space+time) convnet (C3D) encoder and a LSTM decoder. The authors investigate the benefits of using attention mechanisms operating both at the spatio-temporal and layer (feature abstraction) levels.

2) Contributions

+ Well motivated and implemented attention mechanism to handle the different shapes of C3D feature maps (along space, time, and feature dimensions).
+ Convincing quantitative and qualitative experiments on three challenging datasets (Youtube2Text, M-VAD, MSR-VTT) showing clearly the benefit of the proposed attention mechanisms.
+ Interesting comparison of soft vs hard attention showing a slight performance advantage for the (simpler) soft attention mechanism in this case.

3) Suggestions for improvement

Hypercolumns comparison:
As mentioned during pre-review questions, it would be interesting to compare to the hypercolumns of https://arxiv.org/abs/1411.5752, as they are an alternative to the proposed attention mechanisms, with the same purpose of leveraging different feature abstraction levels.

Minor clarifications in the text and figures as agreed with the authors in our pre-review discussions.

4) Conclusion

Although the novelty with existing video captioning approaches is limited, the paper is relevant to ICLR, as the proposed simple but efficient implementation and benefits of spatio-temporal + feature abstraction attention are clearly validated in this work.
review_title: Clear and efficient space+time+feature attention mechanisms for video captioning

final_score: rating: null
review_content: 1) It seems that this paper is highly related to the approach of Xu et al., except attention is extended to multiple layers, and there are some temporal components due to the video setting. Therefore, it seems that the problem of attending over vectors of different sizes is the only new technically novel component. Have the authors explored an approach other than filter and max pool for this component? Would it be worth including in the results?

2) Hard attention working a bit worse than Soft attention is at odds with Xu et al, who found hard attention to work slightly better. Any opinions as to what could account for the discrepancy?
review_title: pre-review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors apply the image captioning architecture of Xu et al. 2015 to video captioning. The model is extended to have attention over multiple layers of the ConvNet instead of just a single layer. Experiments on YouTube2Text, M-VAD and MSR-VTT show that this works better than only using one of the layers at a time.

I think this is solid work on the level of a well-executed course project or a workshop paper. The model makes sense, it is adequately described, and the experiments show that attending over multiple layers works better than attending over any one layer in isolation. Unfortunately, I don't think there is enough to get excited about here from a technical perspective and it's not clear what value the paper brings to the community. Other aspects of the paper, such as including the hard attention component, don't seem to add to the paper but take up space. 

If the authors want to contribute a detailed, focused exploration of multi-level features this could become a more valuable paper, but in that case I would expect a much more thorough exploration of the choices and tradeoffs of different schemes without too many spurious aspects such as video features, hard attention, etc.

review_title: solid work but lack of novelty

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper presents a model for video captioning with both soft and hard attention, using a C3D network for the encoder and a RNN for the decoder. Experiments are presented on YouTube2Text, M-VAD, and MSR-VTT. While the ideas of image captioning with soft and hard attention, and video captioning with soft attention, have already been demonstrated in previous work, the main contribution here is the specific architecture and attention over different layers of the CNN.

The work is well presented and the experiments clearly show the benefit of attention over multiple layers. However, in light of previous work in captioning, the contribution and resulting insights is too incremental for a conference paper at ICLR. Further experiments and analysis of the main contribution would strengthen the paper, but I would recommend resubmission to a more suitable venue.
review_title: state-of-the-art results but too incremental

Meta-review: Reviewers feel the work is well executed and that the model makes sense, but two of the reviewers were not convinced that the proposed method contains enough novelty in light of prior work. The comparison of the soft vs hard attention model variations is perhaps one of the more novel aspects of the work; however, the degree of novelty within these formulations and the insights obtained from their comparison were not perceived as being enough to warrant higher ratings. We would like to invite the authors to submit this paper to the workshop track.
Decision: Invite to Workshop Track