Title: Memory-augmented Attention Modelling for Videos
Authors: Rasool Fakoor
Abdel-rahman Mohamed
Margaret Mitchell
Sing Bing Kang
Pushmeet Kohli
Venue: ICLR 2017
Keywords: Deep learning
Multi-modal learning
Computer vision

Abstract: Recent works on neural architectures have demonstrated the utility of attention mechanisms for a wide variety of tasks. Attention models used for problems such as image captioning typically depend on the image under consideration, as well as the previous sequence of words that come before the word currently being generated. While these types of models have produced impressive results, they are not able to model the higher-order interactions involved in problems such as video description/captioning, where the relationship between parts of the video and the concepts being depicted is complex. Motivated by these observations, we propose a novel memory-based attention model for video description. Our model utilizes memories of past attention when reasoning about where to attend to in the current time step, similar to the central executive system proposed in human cognition. This allows the model to not only reason about local attention more effectively, it allows it to consider the entire sequence of video frames while generating each word. Evaluation on the challenging and popular MSVD and Charades datasets show that the proposed architecture outperforms all previously proposed methods and leads to a new state of the art results in the video description.

TL;DR: We propose a novel memory-based attention model for video description

Review discussion:
1) can the authors clarify why Table 1 shows better results for both HAM and TEM? It seems to me that the results are not conclusive as the takeaways from different metrics are in conflict.
2) can the authors clarify, in english, what the HAM module is doing? It seems that this is a core contribution of the paper and it is very poorly explained. There are a lot of vectors, matrices, subscripts and superscripts here. is f_m and f_{\textbf{m}} different? Where does U come from? I also assume that Figure 1 is misleading and that there isn't a small RBM inside the HAM module.
Thanks for your feedback and questions.
1) can the authors clarify why Table 1 shows better results for both HAM and TEM? It seems to me that the results are not conclusive as the takeaways from different metrics are in conflict.
Table 1 provides an ablative analysis of our approach. Its goal is to show the significance of different components of our proposed model.
As there is no perfect metric for this task, we show the results on a set of metrics that are popular in the NLP community. The last two rows show the significant of having HAM and TEM in our model. As the results indicate, HAM and TEM lead to the higher score in METEOR and BLEU-1, while  HAM + No TEM shows higher score in BLEU-2-4. This deviation in the result is based on the fact that METEOR and BLEU consider different properties of the result.
For instance, arguably METEOR prefers fluency in the generated captions.
In addition, as we mentioned in section 4.1, dropping TEM has little effect on the YouTube dataset as per our observation that its videos tend to have a single action, furthermore, motion in many videos in the dataset doesn't explicitly get reflected in the captions.
However, for the problem of video captioning in general TEM is an important component as the results show in Tables 2 and 3.
It is worth noting this experiment shows that adding HAM component to the model causes the improvement in overall performance (please refer to third and fourth rows).
2) can the authors clarify, in english, what the HAM module is doing? It seems that this is a core contribution of the paper and it is very poorly explained. There are a lot of vectors, matrices, subscripts and superscripts here. is f_m and f_{\textbf{m}} different? Where does U come from? I also assume that Figure 1 is misleading and that there isn't a small RBM inside the HAM module.
- The standard attention model (Bahdanau et al. (2015)) lacks global attention modeling abilities which may result in the model, at the end of the sentence production phase, forgetting what was in the first few frames. The HAM utilizes memories of past attention when reasoning about where to attend to in a current time step. It uses past memories and network states thus allowing it to consider the entire video as it generates each word. As the memory is defined over past attention, the resulting (global) attention can be interpreted as hierarchical.
In addition, it learns to memorize an encoding of the entire video at the time step given previously generated words and network states.
-U is the softmax parameter: \propto \exp(U^T Q_A). We use 'softmax' instead of actual formula to simplify the equation.
-about f_m and f_{\textbf{m}}: If you are referring page 5, line 12 and 13, yes, they are the same. \textbf{} shouldn't be here. We will update paper to reflect your comment. Thanks.
-About figure 1: not sure what causes to think of RBM. We don't model any part of our architecture using RBM. We'd be appreciated if you please elaborate more about your confusion about figure 1 so we can address it accordingly. It is worth mentioning that all equations for HAM component are mentioned in equation 6-9 and HAM is explained in section 3.2.
Please let us know if you have further questions.  Thanks again.
You mention in Section 3 that one of the main contributions of this work is to use "a global state to generate any new word". I am wondering how this is different from other RNN/LSTM based approaches which typically use the "global" state of the LSTM to predict the next word.
Which metric do you use for hyper parameters optimization? Or did you use word softmax accuracy?
The metrics you report are known to not necessarily align very well with human judgments.
It would be great if you can report SPCIE (Anderson et al ECCV 2016) which showed much better correlation with human judgments.
Would it be feasible to do a human evaluation to see if the changes in metrics are actually show improvements?
Thanks for your clarifications.
Thanks for reading our paper and your questions.
1) You mention in Section 3 that one of the main contributions of this work is to use "a global state to generate any new word". I am wondering how this is different from other RNN/LSTM based approaches which typically use the "global" state of the LSTM to predict the next word.
Given the sequential structural in the RNN/LSTM, it puts higher weight on the local state ( i.e. neighbors) than global ones. Consider the state update equation in RNN: h^(t)= \sigma( W*h^(t-1) + U*X^(t) ). In order to update the state(h) at time step (t), it considers the state in (t-1) and current input.
Even though h^(t-1) can hold an encoding of previous time steps [1..t-1], information about local neighbor is much stronger. This happens, because during training of RNN with BPTT, the contribution of gradient values gradually become weaker (i.e. lot so multiplication hence smaller number) as they propagate to earlier time-steps.
In addition in a scenario like when one sequence is video and another one is language, capturing global dependency is much harder.
Specifically, when the model gets to the end of a sentence, it already forgets what was in the first few frames.
On the other hand, our model learns how to memorize/model long dependency better that just using RNN/LSTM even with a single step attention. As our experiments confirm, our model can capture more global structure than just RNN/LSTM.
It is interesting to note that LSTM can "theoretically" capture arbitrarily complex temporal relationships, however, in reality, we need to augment the structure to be able to achieve the task in hand, e.g. LSTM vs NTM (Graves et. al. (2014)) or LSTM  vs Memory Networks (Weston et. al. (2015)).
2) Which metric do you use for hyperparameters optimization? Or did you use word softmax accuracy?
We optimize the hyperparameters on the validation set using random search. The best model is selected during hyperparameters optimization is based on the METEOR score on the validation set.
When the model shows no improvement on METEOR score (i.e. produces the same METEOR as the best one), it looks at BLEU4 to select the best model.
It is worth noting that all of the hyperparameter optimization is done on the validation set. We are going to add more details related to the hyperparameters tuning to the paper.
3) The metrics you report are known to not necessarily align very well with human judgments. It would be great if you can report SPCIE (Anderson et al ECCV 2016) which showed much better correlation with human judgments.
It is worth noting that SPICE measures how well "caption generators recover objects, attributes and the relations between them." As such, it neglects fluency, arguably more than the METEOR metric we utilize here. We agree as the authors state (Anderson et al ECCV 2016), they "have not included any fluency adjustments....To model human judgment in a particular task as closely as possible, a carefully tuned ensemble of metrics" is useful. We will be happy to add the results corresponding to SPICE metric in revised version. Please note that SPICE was concurrent with our work.
In addition, none of the previous and current works on video caption generation report any number for SPICE metric and most of the previous works didn't release any code so we can't report this score on their model;
hence, this metric was out of scope for the initial submission.
4) Would it be feasible to do a human evaluation to see if the changes in metrics are actually show improvements?
Our objective was to show the significance of the proposed memory model by comparing the results of prior models. Human experiments are the best indication of the performance of a system but there are a number of issues. Since some of the prior work has not released their code or models, it is difficult to compare results in a human study.
Also, experimental setup might introduce demographic (subjective) and contextual biases. This is why we used the objective metrics (BLEU and METEOR) that have been used in prior work.
Please let us know if you have further questions. Thanks again.
This paper addresses video captioning with a TEM-HAM architecture, where a HAM module attends over attended outputs of the TEM module when generating the description. This gives a kind of 2-level attention. The model is evaluated on the Charades and MSVD datasets.
1. Quality/Clarify: I found this paper to be poorly written and relatively hard to understand. As far as I can tell the TEM module of Section 3.1 is a straight-forward attention frame encoder of Bahdanau et al. 2015 or Xu et al. 2015. The decoder of Section 3.3 is a standard LSTM with log likelihood. The HAM module of Section 3.2 is the novel module but is not very well described. It looks to be an attention LSTM where the attention is over the TEM LSTM outputs, but the attention weights are additionally conditioned on the decoder state. There are a lot of small problems with the description, such as notational discrepancy in using \textbf in equations and then not using it in the text.
Also, I spent a long time trying to understand what f_m is. The authors say:
"In order to let the network remember what has been attended before and the temporal
structure of a video, we propose f_m to memorize the previous attention and encoded version of an
input video with language model. Using f_m not only enables the network to memorize previous
attention and frames, but also to learn multi-layer attention over an input video and corresponding
language."
Where one f_m is bold and the other f_m is not. Due to words such as "we propose f_m" assumed this was some kind of a novel technical contribution I couldn't find any details about but it is specified later in Section 3.3 at the end that f_m is in fact just an LSTM. It's not clear why this piece of information is in Section 3.3, which discusses the decoder. The paper is sloppy in other parts.
For example in Table 1 some numbers have 1 significant digit and some have 2. The semantics of the horizontal line in Table 2 are not explained in text.
2. Experimental results: The ablation study shows mixed results when adding TEM and HAM to the model. Looking at METEOR which was shown to have the highest correlation to humans in the COCO paper compared to the other evaluation criteria, adding TEM+HAM improves the model from 31.20 to 31.70. It is not clear how significant this improvement is, especially given that the test set is only 670 videos. I have doubts over this result. In Table 2, the METEOR score of Pan et al. 2016a is higher [33.10 vs. 31.80], but this discrepancy is not addressed in text. This is surprising because the authors explicitly claim "state of the art results".
3. Originality/Significance: The paper introduces an additional layer of attention over a more standard sequence to sequence setup, which is argued to alleviate the burden on the LSTM's memory. This is moderately novel but I don't believe that the experimental results make it sufficiently clear that it is also worth doing. If the paper made the standard model somehow simpler instead of more complex I would be more inclined to judge it favorably.
Minor:
In response to the author's comment "not sure what causes to think of RBM. We don't model any part of our architecture using RBM. We'd be appreciated if you please elaborate more about your confusion about figure 1 so we can address it accordingly.", I created a diagram to hopefully make this more clear:  https://imgur.com/a/4MJaG . It is very common to use 2 rows of circles with lines between them pairwise to denote an RBM. The lines indicate undirected edges of the graphical model. A typical RBM diagram can have, for example, 3 circles on top and 4 circles on the bottom joined with edges. Your diagram has 4 circles on the top and 5 circles on the bottom joined with edges.
However, not all of your circles from the top and bottom are connected, which further adds to the difficulty.
In particular, your last circle on the bottom is only connected to 2 circles on the top. If the authors are trying to denote a neural network I would advise using arrows instead of lines.
However, since the HAM module uses an LSTM just like the encoder and the decoder it is unclear why this module has a visually distinct appearance at all.
Thanks for your comment and feedbacks.
1.
About f_m and other notes regarding table-1
- We clarify further the f_m in the HAM section.
- We rounded outputs of evaluation script (MSCOCO evaluation server) to 2 decimal places. We copied exactly those numbers in the table.
However, we'll make the table numbers presentation consistent in next version.
2.
About  Experimental results:
- In regards to Pan et al. we agree that it has better METEOR score, but our scores in BLUE@N are significantly better, especially BLUE@4. In our view, this shows the difference between our approach and Pan et al. to model the video which doesn't discount our claim. That being said, we agree with the reviewer that in order to be fair to this paper we should acknowledge this in the paper more clearly.
3. One major problem with video caption generation methods that model the problem as sequence-to-sequence learning [Sutskever et al., 2014] by including/excluding attention model [Bahdanau et al (2015)] is that they followed similar architecture for mapping from visual space to the language space as it for language to language but it doesn't usually work as good as machine translation. One reason that works in the machine translation not video caption generation is that because of mapping from visual space to language space is harder than mapping from a language to a language given the complexity structure of a video. By looking at the results especially comparing with standard sequence-to-sequence model [Table 2, Venugopalan et al. (2015a) Venugopalan et al. (2015a) + Flow], it confirms our argument that the standard seq-to-seq doesn't work well with video caption generation because of the mentioned reasons.
-about figure 1
Thanks for your explaining what causes the confusion. We will add upward pointing arrows in the figure to distinguish it from the undirected RBM model.
Thanks again for the input, we hopefully upload a new version in a couple of days incorporating your comments. We hope that our responses shed more light on the ambiguity that you may have and help to consider re-evaluate our paper.
The paper proposes an attention-based approach for video description. The approach uses three LSTMs and two attention mechanisms to sequentially predict words from a sequence of frames.
In the LSTM-encoder of the frames (TEM), the first attention approach predicts a spatial attention per frame, and computes the weighted average. The second LSTM (HAM) predicts an attention over the hidden states of the encoder LSTM.
The third LSTM which run temporally in parallel to the second LSTM generates the sentence, one word at a time.
Strength:
-	The paper works on a relevant and interesting problem.
-	Using 2 layers of attention in the proposed way have to my knowledge not been used before for video description. The exact architecture is thus novel (but the work claims much more without sufficient attribution, see blow)
-	The experiments are evaluated on two datasets, MSVD and Charades, showing performance on the level of related work for MSVD and improvements for Charades.
Weaknesses:
1.	Claims about the contribution/novelty of the model seem not to hold:
1.1.	One of the main contributions is the Hierarchical Attention/Memory (HAM):
1.1.1.	It is not clear to me how the presented model (Eq 6-8), are significantly different from the presented model in Xu et al / Yao et al.
While Xu et al. attends over spatial image locations and Yao et al. attend over frames, this model attends over encoded video representations h_v^i. A slight difference might be that Xu et al. use the same LSTM to generate, while this model uses an additional LSTM for the decoding.
1.1.2.	The paper states in section 3.2 “we propose f_m to memorize the previous attention”, however H_m^{t’-1} only consist of the last hidden state.
Furthermore, the model f_m does not have access to the “attention” \alpha. This was also discussed in comments by others, but remains unclear.
1.1.3.	In the discussion of comments the authors claim that “attention not only is a function a current time step but also a function of all previous attentions and network states.”: While it is true that there is a dependency but that is true also for any LSTM, however the model does not have access to the previous network states as H_g^{t’-1} only consist of the last hidden state, as well as H_m^{t’-1} [at least that is what the formulas say and what Figure 1 suggests].
1.1.4.	The authors claim to have multi-layer attention in HAM, however it remains unclear where the multi-layer comes from.
1.2.	The paper states that in section 3.1. “[CNN] features tend to discard the low level information useful in modeling the motion in the video (Ballas et al., 2016).” This suggests that the approach which follows attacks this problem.
However, it cannot model motion as attention \rho between frames is not available when predicting the next frame.
Also, it is not clear how the model can capture anything “low level” as it operates on rather high level VGG conv 5 features.
2.	Related work: The difference of HAM to Yao et al. and Xu et al. should be made more clear / or these papers should be cited in the HAM section.
3.	Conceptual Limitation of the model: The model has two independent attention mechanisms, a spatial one, and a temporal one. The spatial (within a frame) is independent of the sentence generation. It thus cannot attend to different aspects of the frames for different words which would make sense. E.g. if the sentence is “the dog jumps on the trampoline”, the model should focus on the dog when saying “dog” and on the trampoline when saying “trampoline”, however, as the spatial attention is fixed this is difficult.
Also, the encoder model does not have an explicitly way to look at different aspects in the frame during the encoding so might likely get stuck and always predict the same spatial attention for all frames (or it might e.g. always attend to the dog which moves around, but never on the scene).
4.	Eq 11 contradicts Fig 1: How is the model exactly receiving the previous word as input. Eq. 11 suggests it is the softmax. If this is the case, the authors should emphasize this in the text as this is unusual. More common would be to use the ground truth previous word during training (which Fig 11 suggests) and the “hardmax”, i.e. the highest predicted previous word encoded as one-hot vector at test time.
5.	Clarity:
5.1.	It would be helpful if the same notation would be used in Eq 2-5 and 6-9. Why is a different notation required?
5.2.	It would be helpful if Fig 1 could contain more details or additional figures for the corresponding parts would be added. If space is a problem, e.g. the well-known equations for LSTM, softmax (Eq 2), and log likelihood loss (Eq 12) could be omitted or inlined.
6.	Evaluation:
6.1.	The paper claims that the “the proposed architecture outperforms all previously proposed methods and leads to a new state of the art results”.
6.1.1.	For the MSVD dataset this clearly is wrong, even given the same feature representation. Pan et al. (2016 a) in Table 2 achieve higher METEOR (33.10).
6.1.2.	For this strong claim, I would also expect that it outperforms all previous results independent of the features used, which is also wrong again, Yu et al achieve higher performance in all compared metrics.
6.1.3.	For Charades dataset, this claim is also too bold as hardly any methods have been evaluated on this dataset, so at least all the ablations reported in Table 1 should also be reported for the Charades dataset, to make for this dataset any stronger claims.
6.2.	Missing qualitative results of attention: The authors should show qualitative results of the attention, for both attention mechanisms to understand if anything sensible is happening there. How diverse is the spatial and the temporal attention? Is it peaky or rather uniform?
6.3.	Performance improvement is not significant over model ablations: The improvements over Att+No TEM is only 0.5 Meteor, 0.7 Blue@4 and the performance drops for CIDEr by 1.7.
6.4.	Missing human evaluation: I disagree with the authors that a human evaluation is not feasible. 1. An evaluation on a subset of the test data is not so difficult. 2.
Even if other authors do not provide their code/model [and some do], they are typically happy to share the predicted sentences which is sufficient and even better for human evaluation [if not I would explicitly mention that some authors did not share sentences, as this seems clearly wrong]. 3. For model ablations the sentences are available to the authors.
7.	Several of the comments raised by reviewers/others have not yet been incorporated in a revised version of the paper and/or are still not clear from the explanations given. E.g.
including SPICE evaluation and making fixes seems trivial.
8.	Hyperparameters are inconsistent: Why are the hyperparemters inconsistent between the ablation analysis (40 frames are sampled) and the performance comparison (8 frames)? Should this not be selected on the validation set? What is the performance of all the ablations with 8 frames?
Other (minor/discussion points)
-	Equation 10: what happens with h_m, and h_g, the LSTM formulas provided only handle two inputs. Are h_m and h_g concatenated.
-	There is a section 4.1 but no 4.2.
-	The paper states in section 4.1 “our proposed architecture can alone not only learn a representation for video that can model the temporal structure of a video sequence, but also a representation that can effectively map visual space to the language space.” However, this seems to be true also for many/most other approaches, e.g. [Venugopalan et al. 2015 ICCV]
Summary:
While the paper makes strong claims w.r.t. to the approach and results, the approach lacks novelty and the results are not convincing over related work and ablations.
Furthermore, improved clarity and visualizations of the model and attention results would benefit the paper.
Thanks for your comment and feedbacks.
1.1.1. and 2.
About differences with  Xu et al. / Yao et al.:
- Xu model is used to model the image caption generation;
however, in our model, we tackle the problem of video caption generation. In our model, TEM component learns how to model the temporal structure in the video while in Xu the goal is to learn an encoding for an image that can be used during the generation phase.
In addition, the attention (given the decoder states) in Xu model directly applies to the spatial features;
however, this is not the case in our model.
Moreover, there are significant differences between our model and Yao et al. (2015): 1. We model the video caption generation problem as a variant of a sequence to sequence problem which is not the case in Yao et al. (2015). 2. In Yao et al. (2015) uses attention to calculating the video features that "directly" input to the decoder which in our case we have TEM to model a video then using HAM to calculate some encoding given all previously generate words, memory states, and network states which will be used as input to the decoder. 3. As the results show our model significantly outperforms Yao et al (2015) even when that model uses C3D. This by itself shows that our model is not simply an iterative contribution based on attention model in that paper.
1.1.2, 1.1.3, 1.1.4 About f_m and attention update and memory component:
- In the attention update step (6-8), attention is updated based on current memory state at time t-1, decoder state at time t-1, and the TEM states. As we stated in equation 6-8, the attention update takes into account in each time steps what was attended before in addition to other network states. The reason that alpha is not directly fed to the HAM and F_hat instead because the goal of the network is to learn how to not forget the frame sequence for an input video and learn a global attention over the video.
In summary, the attention in fact updated in each time steps based all previously attentions which are memorized in the HAM state(s).
- We agree with the reviewer about dependency in LSTM.
That said, in our model, even though attention only has access (at each time step) to only states in t-1 from memory and decoder, these states already encoded information about the dependency to the past attentions and network states which by itself leads to building a global attention based on each of these local attentions at each time step.
- As we mentioned in the paper, we can interpret our model in two ways: as hierarchical attention or memory: at each time steps the current attention is built upon previous attentions as well as current local one which we consider this a global or hierarchical attention or simply attenton_t  = f( f( f( f ( ...), a_t-2), a_t-1), a_t )
1.2.
About [CNN] features tend to discard the low level ...
- CNN Features which are extracted from last layers of CNN (i.e. fully connected layers) tend to be class specific and very good for the problems like image recognition given their abstract representations.
However, when we want to attend to the important part of an image/frame, convolution maps show more promising results because there are not as abstract as fully connected features (Xu et al. 2015). We agree that calling them low-level features can be confusing;
however, we just follow the same naming convention as other papers (Xu et al. 2015; Ballas et al. 2016).
- To model the motion, our TEM at each time steps learn to attend to some part of the next frames that is more important (refer to equation 2-5). Learning to attend in each frame can help to model a rough motion model that can be important for the caption generation.
3. Conceptual Limitation of the model:
- The spatial attention is not independent of sentence generation given how error backpropagates in the network.
That said, we acknowledge that it is possible to make TEM part of the loss function, by including the score on action recognition for example.
However, some change like this requires that dataset has action labels as well as descriptions.
4. clarification about Eq.11 and softmax:
We follow the exact procedure as the reviewer suggested.
However, Eq.11, 12 shows how the scores aggregate into loss function during training to calculate the errors. We clarify this in the text. Thanks.
5. Clarity
Thanks, we will address this as well.
6. Evaluation:
6.1, 6.1.1. One claim that we make is that our model can not only model the temporal structure of the video but also can generate a sentence in an end-to-end network. Our method significantly outperforms Yu et al.
when it doesn't use C3D features. Please note that C3D features are trained on sport-1M ( one million samples) and fine-tuned on UCF101 datastet in compare with our model that train on only 1200 examples for MSVD dataset. To clarify we don't suggest a method/model should not use external features, our argument is that if we compare them in a similar setup, our method performs much better.
6.1, 6.1.2 In regards to Pan et al. we agree that it has better METEOR score, but our score in BLUE@N is significantly better, especially BLUE@4. In our view, this shows the difference between our approach and Pan et al. to model the video which doesn't discount our claim. That being said, we agree with the reviewer that in order to be fair to this paper we should acknowledge this in the paper more clearly.
6.1.3. The Charades dataset is proposed in parallel with our work and this is the reason other papers didn't use it. The reason we consider this data set because it shows more real-life videos than MSVD videos which are downloaded from youtube.
6.3, 8: The number frame samples is indeed a hyperparameter which is selected among 4. 8, 16, 40 on the validation set.
When we did ablation study,  Att + No TEM and No HAM + TEM got the best result on the validation set with 40 videos.
Even though HAM + No TEM and HAM + TEM got best results with 8 frames [The last two models (HAM + No TEM and HAM + TEM)  have more parameters so they are in favor of shorter videos]. We though it is better to fix on 40 videos to be fair to the first two rows on that table.
Otherwise, the HAM+TAM results from table2 should be in Table 1.
6. We will share our codes but not sure we will be able to set up human evaluation for the time being.
7. We intentionally waited to upload a revision because we thought it'd be better to first see reviews and incorporate everything in a new version. We will upload a revised paper in next couple of days containing SPICE score for our model.
Equation 10:
- No, h_m and h_g are not concatenated. The LSTM in 10 handles three inputs. We will clarify this, tnx.
-about Venugopalan et al. 2015a
We agree that we share the same goal with this paper, but our results are stronger than Venugopalan and it shows the effectiveness of our proposed method.
However, it may not be completely true about other papers (e.g.
when using C3D features) since most of other works in this area focus more on the decoder rather than encoder part given at the end of the day, video caption generation are evaluated based on languages scores which don't necessarily reflect what happen with other parts of a model.
Thanks again for the input, we hopefully upload a new version in a couple of days incorporating your comments. We hope that our responses shed more light on the ambiguity that you may have and help to consider re-evaluate our paper.
While the authors comments were partially helpful, it would have been important to see the clarifications, arguments, and revised claims in a updated pdf revision to understand if this makes more sense and if the made claims are more precise and correct.
Unfortunately this did not happen till today and I thus will not increase my score.
It the abstract, the authors claim: "While these types of models [attention-based neural networks] have produced impressive results, they are not able to model the higher-order interactions involved in problems such as video description/captioning, where the relationship between parts of the video and the concepts being depicted is complex." Do you have any evidence to support this claim?
Regarding the hierarchical attention/memory (HAM) component of the model, I see the attention mechanism of Bahdanau et al (2015), but where is the memory mechanism?
Also what is hierarchical about this structure?
Also, what specifically are the contributions beyond the soft attention model presented by Yao et al (2015)?
The authors propose a "hierarchical" attention model for video captioning.  They introduce a model composed of three parts: the temporal modeler (TEM) that takes as input the video sequence and outputs a sequential representation of the video to the HAM; the hierarchical attention/memory mechanism (HAM) implements a soft-attention mechanism over the sequential video representation; and finally a decoder that generates a caption.
Related to the second series of questions above, it seems as though the authors have chosen to refer to their use of an LSTM (or equivalent RNN) as the output of the Bahdanau et al (2015) attention mechanism as a hierarchical memory mechanism. I am actually sympathetic to this terminology in the sense that the recent popularity of memory-based models seems to neglect the memory implicit in the LSTM state vector, but that said, this seems to seriously misrepresent the significance fo the contribution of this paper.
I appreciate the ablation study presented in Table 1. Not enough researchers bother with this kind of analysis.
But it does show that the value of the contributions is not actually clear.
In particular the case for the TEM is quite weak.
Regarding the quantitative evaluation presented in Table 2, the authors are carving out a fairly specific set of features to describe the set of "fair" comparators from the literature. Given the variability of the models and alternate training datasets that are in use, I would find it more compelling if the authors just set about trying to achieve the best results they can, if that includes the fine-tuning of the frame model, so be it. The value of this work is as an application paper, so the discovery and incorporation of elements that can significantly improve performance would seems warranted.
Overall, at this point, I do not see a sufficient contribution to warrant publication in ICLR.
Thanks for your comment and feedbacks. We agree with the reviewer that we should have described our model as a hierarchical attention one without using the term “memory” which is implicit in the model.
About the questions and your comments:
-Although we present the model for the video captioning task, it should be effective for any sequence transduction task where source and/or target sequence lengths are long.
-One property of video captioning is that one word in the target could be related to many different input frames that are scattered over the source. The use of a softmax in the basic soft attention encourages competition between different source locations. The proposed model soften this effect by going deeper in attention using a higher order recurrent network (i.e. the HAM component) to account for arbitrary interaction/alignment of source and target. In our model, the attention vector at each time step is not only a function of the current hidden state but rather as a function of all previously generated attention vectors through HAM which goes beyond a simple sum, as a result it is hierarchical. Our experiments in Table 1 and 2 confirmed the ineffectiveness the baseline seq-to-seq model with attention compared to the proposed approach.
Our model goes further than basic soft attention because the attention vector at each time step is not only a function of the current hidden state (we call it local attention) but rather as a function of all previously generated attention vectors thought the HAM component (i.e. a higher order LSTM which does a sophisticated aggregation of past attention vectors that goes beyond a simple sum), as a result, it is a hierarchical one (or global one).
-We would appreciate it if the reviewer could make the comments of the results in Table 2 clearer. We didn't cherry-pick any competing results or metrics but rather we reported all previous work that we are aware of.

Meta-review:
role: PC
data: comment: There appears to be consensus among the reviewers that the paper appears the overstate its contributions: the originality of the proposed temporal modeler (TEM) is limited, and the experimental evaluation (which itself is of good quality!) does not demonstrate clear merits of the TEM architecture. As a result, the impact of this paper is expected to be limited.

Decision: Reject