Venue: ICLR 2017 conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: The authors propose a "hierarchical" attention model for video captioning.  They introduce a model composed of three parts: the temporal modeler (TEM) that takes as input the video sequence and outputs a sequential representation of the video to the HAM; the hierarchical attention/memory mechanism (HAM) implements a soft-attention mechanism over the sequential video representation; and finally a decoder that generates a caption. 

Related to the second series of questions above, it seems as though the authors have chosen to refer to their use of an LSTM (or equivalent RNN) as the output of the Bahdanau et al (2015) attention mechanism as a hierarchical memory mechanism. I am actually sympathetic to this terminology in the sense that the recent popularity of memory-based models seems to neglect the memory implicit in the LSTM state vector, but that said, this seems to seriously misrepresent the significance fo the contribution of this paper. 

I appreciate the ablation study presented in Table 1. Not enough researchers bother with this kind of analysis. But it does show that the value of the contributions is not actually clear. In particular the case for the TEM is quite weak.

Regarding the quantitative evaluation presented in Table 2, the authors are carving out a fairly specific set of features to describe the set of "fair" comparators from the literature. Given the variability of the models and alternate training datasets that are in use, I would find it more compelling if the authors just set about trying to achieve the best results they can, if that includes the fine-tuning of the frame model, so be it. The value of this work is as an application paper, so the discovery and incorporation of elements that can significantly improve performance would seems warranted. 

Overall, at this point, I do not see a sufficient contribution to warrant publication in ICLR.

review_title: Review

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper proposes an attention-based approach for video description. The approach uses three LSTMs and two attention mechanisms to sequentially predict words from a sequence of frames.
In the LSTM-encoder of the frames (TEM), the first attention approach predicts a spatial attention per frame, and computes the weighted average. The second LSTM (HAM) predicts an attention over the hidden states of the encoder LSTM.
The third LSTM which run temporally in parallel to the second LSTM generates the sentence, one word at a time.


Strength:
===============

-	The paper works on a relevant and interesting problem.
-	Using 2 layers of attention in the proposed way have to my knowledge not been used before for video description. The exact architecture is thus novel (but the work claims much more without sufficient attribution, see blow)
-	The experiments are evaluated on two datasets, MSVD and Charades, showing performance on the level of related work for MSVD and improvements for Charades.

Weaknesses:
===============

1.	Claims about the contribution/novelty of the model seem not to hold: 
1.1.	One of the main contributions is the Hierarchical Attention/Memory (HAM):
1.1.1.	It is not clear to me how the presented model (Eq 6-8), are significantly different from the presented model in Xu et al / Yao et al. While Xu et al. attends over spatial image locations and Yao et al. attend over frames, this model attends over encoded video representations h_v^i. A slight difference might be that Xu et al. use the same LSTM to generate, while this model uses an additional LSTM for the decoding.
1.1.2.	The paper states in section 3.2 “we propose f_m to memorize the previous attention”, however H_m^{t’-1} only consist of the last hidden state. Furthermore, the model f_m does not have access to the “attention” \alpha. This was also discussed in comments by others, but remains unclear.
1.1.3.	In the discussion of comments the authors claim that “attention not only is a function a current time step but also a function of all previous attentions and network states.”: While it is true that there is a dependency but that is true also for any LSTM, however the model does not have access to the previous network states as H_g^{t’-1} only consist of the last hidden state, as well as H_m^{t’-1} [at least that is what the formulas say and what Figure 1 suggests]. 
1.1.4.	The authors claim to have multi-layer attention in HAM, however it remains unclear where the multi-layer comes from.
1.2.	The paper states that in section 3.1. “[CNN] features tend to discard the low level information useful in modeling the motion in the video (Ballas et al., 2016).” This suggests that the approach which follows attacks this problem. However, it cannot model motion as attention \rho between frames is not available when predicting the next frame. Also, it is not clear how the model can capture anything “low level” as it operates on rather high level VGG conv 5 features.

2.	Related work: The difference of HAM to Yao et al. and Xu et al. should be made more clear / or these papers should be cited in the HAM section.

3.	Conceptual Limitation of the model: The model has two independent attention mechanisms, a spatial one, and a temporal one. The spatial (within a frame) is independent of the sentence generation. It thus cannot attend to different aspects of the frames for different words which would make sense. E.g. if the sentence is “the dog jumps on the trampoline”, the model should focus on the dog when saying “dog” and on the trampoline when saying “trampoline”, however, as the spatial attention is fixed this is difficult. Also, the encoder model does not have an explicitly way to look at different aspects in the frame during the encoding so might likely get stuck and always predict the same spatial attention for all frames (or it might e.g. always attend to the dog which moves around, but never on the scene).

4.	Eq 11 contradicts Fig 1: How is the model exactly receiving the previous word as input. Eq. 11 suggests it is the softmax. If this is the case, the authors should emphasize this in the text as this is unusual. More common would be to use the ground truth previous word during training (which Fig 11 suggests) and the “hardmax”, i.e. the highest predicted previous word encoded as one-hot vector at test time.

5.	Clarity:
5.1.	It would be helpful if the same notation would be used in Eq 2-5 and 6-9. Why is a different notation required?
5.2.	It would be helpful if Fig 1 could contain more details or additional figures for the corresponding parts would be added. If space is a problem, e.g. the well-known equations for LSTM, softmax (Eq 2), and log likelihood loss (Eq 12) could be omitted or inlined.

6.	Evaluation:
6.1.	The paper claims that the “the proposed architecture outperforms all previously proposed methods and leads to a new state of the art results”.
6.1.1.	For the MSVD dataset this clearly is wrong, even given the same feature representation. Pan et al. (2016 a) in Table 2 achieve higher METEOR (33.10).
6.1.2.	For this strong claim, I would also expect that it outperforms all previous results independent of the features used, which is also wrong again, Yu et al achieve higher performance in all compared metrics.
6.1.3.	For Charades dataset, this claim is also too bold as hardly any methods have been evaluated on this dataset, so at least all the ablations reported in Table 1 should also be reported for the Charades dataset, to make for this dataset any stronger claims.
6.2.	Missing qualitative results of attention: The authors should show qualitative results of the attention, for both attention mechanisms to understand if anything sensible is happening there. How diverse is the spatial and the temporal attention? Is it peaky or rather uniform?
6.3.	Performance improvement is not significant over model ablations: The improvements over Att+No TEM is only 0.5 Meteor, 0.7 Blue@4 and the performance drops for CIDEr by 1.7.
6.4.	Missing human evaluation: I disagree with the authors that a human evaluation is not feasible. 1. An evaluation on a subset of the test data is not so difficult. 2. Even if other authors do not provide their code/model [and some do], they are typically happy to share the predicted sentences which is sufficient and even better for human evaluation [if not I would explicitly mention that some authors did not share sentences, as this seems clearly wrong]. 3. For model ablations the sentences are available to the authors.

7.	Several of the comments raised by reviewers/others have not yet been incorporated in a revised version of the paper and/or are still not clear from the explanations given. E.g. including SPICE evaluation and making fixes seems trivial.	

8.	Hyperparameters are inconsistent: Why are the hyperparemters inconsistent between the ablation analysis (40 frames are sampled) and the performance comparison (8 frames)? Should this not be selected on the validation set? What is the performance of all the ablations with 8 frames?

Other (minor/discussion points)
-	Equation 10: what happens with h_m, and h_g, the LSTM formulas provided only handle two inputs. Are h_m and h_g concatenated.
-	There is a section 4.1 but no 4.2.
-	The paper states in section 4.1 “our proposed architecture can alone not only learn a representation for video that can model the temporal structure of a video sequence, but also a representation that can effectively map visual space to the language space.” However, this seems to be true also for many/most other approaches, e.g. [Venugopalan et al. 2015 ICCV]

Summary:
===============

While the paper makes strong claims w.r.t. to the approach and results, the approach lacks novelty and the results are not convincing over related work and ablations. Furthermore, improved clarity and visualizations of the model and attention results would benefit the paper.

review_title: Review: Novelty and performance claims do not hold, missing clarity, no significant improvements over ablations.

final_score: rating: null
review_content: It the abstract, the authors claim: "While these types of models [attention-based neural networks] have produced impressive results, they are not able to model the higher-order interactions involved in problems such as video description/captioning, where the relationship between parts of the video and the concepts being depicted is complex." Do you have any evidence to support this claim?

Regarding the hierarchical attention/memory (HAM) component of the model, I see the attention mechanism of Bahdanau et al (2015), but where is the memory mechanism? Also what is hierarchical about this structure? Also, what specifically are the contributions beyond the soft attention model presented by Yao et al (2015)?

review_title: Question

final_score: rating: null
review_content: 
1) can the authors clarify why Table 1 shows better results for both HAM and TEM? It seems to me that the results are not conclusive as the takeaways from different metrics are in conflict.

2) can the authors clarify, in english, what the HAM module is doing? It seems that this is a core contribution of the paper and it is very poorly explained. There are a lot of vectors, matrices, subscripts and superscripts here. is f_m and f_{\textbf{m}} different? Where does U come from? I also assume that Figure 1 is misleading and that there isn't a small RBM inside the HAM module.
review_title: clarifications

final_score: rating: null
review_content: You mention in Section 3 that one of the main contributions of this work is to use "a global state to generate any new word". I am wondering how this is different from other RNN/LSTM based approaches which typically use the "global" state of the LSTM to predict the next word.

Which metric do you use for hyper parameters optimization? Or did you use word softmax accuracy?

The metrics you report are known to not necessarily align very well with human judgments.
It would be great if you can report SPCIE (Anderson et al ECCV 2016) which showed much better correlation with human judgments.
Would it be feasible to do a human evaluation to see if the changes in metrics are actually show improvements?

Thanks for your clarifications.
review_title: Clarifications

final_score: rating: 4: Ok but not good enough - rejection
review_content: 
This paper addresses video captioning with a TEM-HAM architecture, where a HAM module attends over attended outputs of the TEM module when generating the description. This gives a kind of 2-level attention. The model is evaluated on the Charades and MSVD datasets.

1. Quality/Clarify: I found this paper to be poorly written and relatively hard to understand. As far as I can tell the TEM module of Section 3.1 is a straight-forward attention frame encoder of Bahdanau et al. 2015 or Xu et al. 2015. The decoder of Section 3.3 is a standard LSTM with log likelihood. The HAM module of Section 3.2 is the novel module but is not very well described. It looks to be an attention LSTM where the attention is over the TEM LSTM outputs, but the attention weights are additionally conditioned on the decoder state. There are a lot of small problems with the description, such as notational discrepancy in using \textbf in equations and then not using it in the text. Also, I spent a long time trying to understand what f_m is. The authors say: 

"In order to let the network remember what has been attended before and the temporal
structure of a video, we propose f_m to memorize the previous attention and encoded version of an
input video with language model. Using f_m not only enables the network to memorize previous
attention and frames, but also to learn multi-layer attention over an input video and corresponding
language."

Where one f_m is bold and the other f_m is not. Due to words such as "we propose f_m" assumed this was some kind of a novel technical contribution I couldn't find any details about but it is specified later in Section 3.3 at the end that f_m is in fact just an LSTM. It's not clear why this piece of information is in Section 3.3, which discusses the decoder. The paper is sloppy in other parts. For example in Table 1 some numbers have 1 significant digit and some have 2. The semantics of the horizontal line in Table 2 are not explained in text. 

2. Experimental results: The ablation study shows mixed results when adding TEM and HAM to the model. Looking at METEOR which was shown to have the highest correlation to humans in the COCO paper compared to the other evaluation criteria, adding TEM+HAM improves the model from 31.20 to 31.70. It is not clear how significant this improvement is, especially given that the test set is only 670 videos. I have doubts over this result. In Table 2, the METEOR score of Pan et al. 2016a is higher [33.10 vs. 31.80], but this discrepancy is not addressed in text. This is surprising because the authors explicitly claim "state of the art results".

3. Originality/Significance: The paper introduces an additional layer of attention over a more standard sequence to sequence setup, which is argued to alleviate the burden on the LSTM's memory. This is moderately novel but I don't believe that the experimental results make it sufficiently clear that it is also worth doing. If the paper made the standard model somehow simpler instead of more complex I would be more inclined to judge it favorably.


Minor:
In response to the author's comment "not sure what causes to think of RBM. We don't model any part of our architecture using RBM. We'd be appreciated if you please elaborate more about your confusion about figure 1 so we can address it accordingly.", I created a diagram to hopefully make this more clear:  https://imgur.com/a/4MJaG . It is very common to use 2 rows of circles with lines between them pairwise to denote an RBM. The lines indicate undirected edges of the graphical model. A typical RBM diagram can have, for example, 3 circles on top and 4 circles on the bottom joined with edges. Your diagram has 4 circles on the top and 5 circles on the bottom joined with edges. However, not all of your circles from the top and bottom are connected, which further adds to the difficulty. In particular, your last circle on the bottom is only connected to 2 circles on the top. If the authors are trying to denote a neural network I would advise using arrows instead of lines. However, since the HAM module uses an LSTM just like the encoder and the decoder it is unclear why this module has a visually distinct appearance at all.
review_title: lacks polish and clarity, moderately original but weak results

Meta-review: There appears to be consensus among the reviewers that the paper appears the overstate its contributions: the originality of the proposed temporal modeler (TEM) is limited, and the experimental evaluation (which itself is of good quality!) does not demonstrate clear merits of the TEM architecture. As a result, the impact of this paper is expected to be limited.
Decision: Reject