Title: MEMORY-AUGMENTED ATTENTION MODELLING FOR VIDEOS

Abstract: Recent works on neural architectures have demonstrated the utility of attention mechanisms for a wide variety of tasks. Attention models used for problems such as image captioning typically depend on the image under consideration, as well as the previous sequence of words that come before the word currently being generated. While these types of models have produced impressive results, they are not able to model the higher-order interactions involved in problems such as video description/captioning, where the relationship between parts of the video and the concepts being depicted is complex. Motivated by these observations, we propose a novel memory-based attention model for video description. Our model utilizes memories of past attention when reasoning about where to attend to in the current time step, similar to the central executive system proposed in human cognition (Baddeley & Hitch, 1974) . This allows the model to not only reason about local attention more effectively, it allows it to consider the entire sequence of video frames while generating each word. Evaluation on the challenging and popular MSVD and Charades datasets show that the proposed architecture outperforms all previously proposed methods and leads to a new state of the art results in the video description.

1. INTRODUCTION:

Deep neural architectures have led to remarkable progress in computer vision and natural language understanding problems. Image captioning is one such application that has seen the combination of convolutional structures (Krizhevsky et al., 2012; LeCun et al., 1998) , which have been shown to be very effective for problems like object detection, with sequential recurrent structures, which have been shown to be very effective for problems like machine translation (Sutskever et al., 2014) . One of the key modelling paradigms shared by most models for image captioning is the notion of an attention mechanism that guide the model to attend to certain parts of the image while generating.

The attention models used for problems such as image captioning typically depend on the single image under consideration and the partial output generated so far, jointly capturing one region of an image and the words being generated. However, such models cannot capture the temporal reasoning necessary to effectively produce words that refer to actions and events taking place over multiple frames in a video. For example, in a video depicting "someone waving a hand", the "waving" action can start from any frame and can continue on for a variable number of following frames. More importantly, it is likely in a given video quite few frames do not contain any useful information or motion in regard to a given task. Given this, it is not surprising that even with recent advancements in image captioning Xu et al. (2015a) ; Johnson et al. (2016) ; Vinyals et al. (2015) , video captioning has remained challenging.

Motivated by these observations, we introduce a memory-based attention mechanism for video captioning and description. Our model utilizes memories of past attention in the video when reasoning about where to attend to in a current time step. This allows the model to not only effectively leverage local attention, but also to consider the entire video as it generates each word. This mechanism is similar to the proposed central executive system in human cognition, which is thought to permit human performance on two simultaneous tasks (e.g., seeing and saying) using two separate percep-tual domains (e.g., visual and linguistic) by binding information from both sources into coherent structure that enables coordination, selective attention, and inhibition.

Our work shares the same goals as recent work on attention mechanisms for sequence-to-sequence architectures, such as Rocktäschel et al. (2016) and Yang et al. (2016) . However, there are major differences between this work and our current work. Rocktäschel et al. (2016) considers the domain of entailment relations, where the goal is to determine entailment given two input sentences. They propose a soft attention model that is not only focused on the current state, but the previous as well. In our model, we explicitly store all previous attention into memory. In addition, our memory memorizes the encoded version of the input videos conditioned on previously seen words. Even though Yang et al. (2016) and our work both try to solve the problem of locality of attention, our work is different from them in how the memory architecture is modelled. More specifically, they incorporate discriminative supervision into their "reviewer" mechanism, which is not the case in our model. Further, their model is applied to image caption generation, which is to some extent simpler than video caption generation because there is no temporal structure to model. We apply our model on the video captioning problem and evaluate it on the MSVD (Chen & Dolan, 2011) and the Charades (Sigurdsson et al., 2016) datasets. Experimental results show that our proposed architecture outperforms all previous methods and leads to new state of the art results. While we have chosen the video captioning problem for our experiment, the model is general enough that it can be applied to other problems where attention models are used.

2. RELATED WORK:

One of the primary challenges in learning a mapping from a visual space (i.e., video or image) to a language space is learning a representation that not only effectively represents each of these modalities, but is also able to translate a representation from one space to the other. Rohrbach et al. (2013) developed a model that generates a semantic representation of visual content that can be used as the source language for the language generation module. Venugopalan et al. (2015b) proposed a deep method to translate a video into a sentence where an entire video is represented with a single vector based on the mean pool of frame features. However, representing a video by an average of its frames misses the temporal structure of the video. To address this problem, recent work (Yao et al., 2015; Pan et al., 2016a; Venugopalan et al., 2015a; Andrew Shin, 2016; Pan et al., 2016b; Xu et al., 2015b; Ballas et al., 2016; Yu et al., 2016) proposed methods to model temporal structure of video as well as language.

The majority of these methods are inspired by sequence to sequence (Sutskever et al., 2014) and attention (Bahdanau et al., 2015) models. Sequence learning (Sutskever et al., 2014) was originally proposed to map the input sequence of a source language to a target language. Even though applying this method with a combination of attention to the problem of translating a video to a description shows promising results, there are some shortcomings. First of all, modelling the video content with a fixed-length vector in order to map it to a language space is a much harder problem than mapping from a language to a language given the complexity of visual content. Since not all frames in a video are equally salient for a short description, and an event can happen in multiple frames, it is important for a model to identify which frames are most salient. Further, the model should be able to focus on points of interest within these frames to select what to talk about. Even using a variable-length vector to represent a video using attention (Yao et al., 2015) can have some problems. More specifically, current attention methods are local Yang et al. (2016) , since the attention mechanism works in a sequential structure, and lacks the ability to capture global structure. Moreover, combining a video and a language description as a sequence to sequence is usually done by some variant of a recurrent neural network (RNN) (Hochreiter & Schmidhuber, 1997) . Given the limited capacity of a recurrent network to model very long sequences, memory networks (Weston et al., 2014; Sukhbaatar et al., 2015) have been introduced to help the RNN memorize sequences. However, one problem these memory networks suffer from is the difficulty in training the model. The model in Weston et al. (2014) requires supervision at each layer which makes training with backpropagation a challenging task. Even though Sukhbaatar et al. (2015) proposed a memory network that can be trained endto-end, working with memory is still a challenging problem in deep learning especially with write operation (Graves et al., 2014) . To address these problems, we propose a memory-based attention sequence-to-sequence model that not only can learn hierarchical attention relationships, but provides a simple and effective memory structure. In the next section, we explain our model in more detail.

3. LEARNING TO ATTEND AND MEMORIZE:

Our goal is to design an architecture that learns where to look and what to look for in a video, in order to talk about it in the description. To achieve this goal, we formulate the problem as sequence learning to maximize the probability of generating a correct description given a video:

EQUATION

where S is the description, f 1 , f 2 , ..., f N are the input video frames, and Θ is the model parameter vector. The main modelling challenges in video description are to develop a system that can model the temporal structure of the video, learn to attend to the important parts of a video, how to memorize the video that was described given all the generated words so far, then generate a new word by looking at the entire video. To address these issues, we propose an end-to-end network that has three components (Figure 1 ): Temporal Model (TEM), Hierarchical Attention/Memory (HAM), and the Decoder. The goal in TEM is to capture the temporal structure and track motion in a video. The HAM component acts as a hierarchical attention or memory between an input video and the description. More specifically, the HAM learns a hierarchical attention structure that learns where to attend in video given all previously generated words and previous states. The HAM can be interpreted as a memory structure as well, where it learns to memorize an encoded version of a video with language. HAM provides the decoder with the ability to look at an entire video plus all the previously generated words before generating any new words. This is important because a single action normally exhibits multiple frames in the input video. By employing the HAM, the model can effectively model the action over these frames. One of the main contributions of this work is to use a global state to generate any new word. This global state aggregates information from previously generated words and all input frames. We will first describe each component of our model, then explain details of training and inference.

3.1. TEMPORAL MODELER (TEM):

One important question is how to encode the temporal structure of the input video for caption generation. Recently, it has been shown that Recurrent Neural Networks (RNN) has the ability to model the temporal structure in sequential data such as video (Ballas et al., 2016; Sharma et al., 2015; Venugopalan et al., 2015a) and speech (Graves & Jaitly, 2014) . Moreover, since frame-to-frame temporal variation tend to be local (Brox & Malik, 2011) and critical in the motion modeling (Ballas et al., 2016) , it is important to consider a frame representation that can preserve frame-to-frame temporal variation. Even though using features extracted from the fully connected layers of Convolutional Neural Networks (CNNs) have shown state of the art results in image classification and recognition (Simonyan & Zisserman, 2014; He et al., 2016) , these features tend to discard the low level information useful in modeling the motion in the video (Ballas et al., 2016) .

To address the temporal modeling and video representation problems, we use an RNN to model the temporal structure of the video where at each time step, a frame encoding with size of R D is used as an input to the RNN. Instead of extracting features from a top layer of the pretrained CNN, intermediate convolutional maps have been extracted for the video frames. Specifically, for a given video, X, with N frames

X = [X 1 , X 2 , • • • , X N ],

N convolutional maps of size R L×D are extracted where D is dimension of a feature corresponding to L locations in the input frame.

In order to let the network selectively focus on these L locations of each frame given the hidden state of RNN, we apply a soft attention model Bahdanau et al. (2015) ; Xu et al. (2015a) ; Sharma et al. (2015) , called "Location Attention (Latt)". More specifically, by using a softmax, each hidden state produces L probabilities to specify which part of the input is more important, and then creates an input map for the RNN using these probabilities. The f Latt is defined as follow:

ρ t j = exp((h t-1 v ) T W j p ) L k=1 exp((h t-1 v ) T W k p )

(2)

EQUATION

)

where

h t-1 v ∈ R K is hidden state of RNN at t -1, W p ∈ R K×L , and F t ∈ R D .

At each time step, TEM learns a vector representation for each frame, looking at the frame convolution map, and applying the location attention on this map conditioned on all previously seen frames.

EQUATION

EQUATION

where f v can be a vanilla RNN, LSTM, or GRU and Θ v is the parameters of the f v . Due to the fact that vanilla RNNs have gradient vanishing and exploding problems (Pascanu et al., 2013) , we use gradient clipping to deal with gradient exploding, and an LSTM with the following flow to deal with the gradient vanishing problem:

i t = σ(F t W x i + h t-1 v W h i ) f t = σ(F t W x f + h t-1 v W h f ) o t = σ(F t W xo + h t-1 v W ho ) g t = tanh(F t W xg + h t-1 v W hg ) c t v = f t c t-1 v + i t g t h t v = o t tanh(c t )

where

W h * ∈ R K×K , W x * ∈ R D×K , and we define Θ v = {W h * , W x * }.

3.2. HIERARCHICAL ATTENTION/MEMORY (HAM):

One problem with using sequence-to-sequence style architectures (Sutskever et al., 2014) , to model a task such as video language description, is how to find a mapping from a video space to a language space that can capture the relationship between a word and video, or more specifically, the connection between an entire video and an entire sentence where there might not be a clear alignment between the two sequences, as opposed to machine translation and speech recognition. Furthermore, the model should be able to identify which part of the video is more relevant to the description because captions normally focus on a tiny fraction of the facts present in the video. More importantly, once the model starts generating the description, it should still be reminded with the video frames to generate meaningful descriptions. In order to address these problems, we propose a memory-based attention that encodes a video into memory, built as a function of the state of the language generation network (a.k.a. Decoder) and the state of the TEM network. More specifically, our Hierarchical Attention/Memory can be formulated as two following steps:

• Attention update [ F(Θ a )]:

EQUATION

• Memory update:

EQUATION

where

H v ∈ R N ×K , W v and W g are ∈ R K×K , U ∈ R K , W m ∈ R M ×K , and Θ a = {W v , W g , W m , U }. N is number of frames in a given video, H v = [h 1 v , ..., h N v ], H t -1 g = [h t -1 g , ..., h t -1 g ], and H t -1 m = [h t -1 m , ..., h t -1 m ]

. α t is the set of probabilities in a given time step that specifies the attention over an input video state (H v ), memory state (H m ), and decoder state (H g ). In order to let the network remember what has been attended before and the temporal structure of a video, we propose f m to memorize the previous attention and encoded version of an input video with language model. Using f m not only enables the network to memorize previous attention and frames, but also to learn multi-layer attention over an input video and corresponding language. The output of the memory-attention is then used as input to the Decoder.

3.3. DECODER:

In order to generate a new word conditioned on all previous words and HAM states, a recurrent structure is modelled as follows:

EQUATION

where s t is a word vector at t , W e ∈ R K×C and C is the vocabulary size. In addition, ŝt assigns a probability to each word in the language. We use LSTMs for both f m and f g .

3.4. TRAINING AND OPTIMIZATION:

The goal in our network is to predict the next word given all previously seen words and an input video. In order to optimize our network parameters, Θ = {W p , Θ v , Θ a , Θ m , W e }, we minimize a negative log likelihood loss function, formulated as follow:

L(S, X; Θ) = - T j |V | i s j,i log(ŝ j,i ) + λ Θ 2 2 (12)

where |V | is the dictionary size. We fully train our network in an end-to-end fashion using first-order stochastic gradient-based optimization method with an adaptive learning rate. More specifically, in order to optimize our network parameters, we use Adam Kingma & Ba (2015) with learning rate 2 × 10 -5 and set β 1 , β 2 to 0.8 and 0.999, respectively. At the training, we use a batch size of 16. (2016) . It is worth noting that one major difference between this dataset and others is that they use a "Hollywood in Homes" approach to collecting the data (Sigurdsson et al., 2016) , where "actors" are crowdsourced, yielding a diverse scene and actor videos. One reason that we report results on this dataset is because each video has a specific action in it and would be a suitable testbed to evaluate our model.

4. EXPERIMENTS:

MSVD is a set of Youtube videos that are annotated by a Mechanical Turker,2 who was asked to pick a clip from a video that represents an activity. In this dataset, each clip is annotated by multiple workers with a single sentence. This dataset contains 1, 970 videos and about 80, 000 descriptions, where 1, 200 of the videos are training data, 670 are test data, and the rest (i.e., 100 videos) are assigned for validation. In order to make the results comparable with other papers, we follow the exact training/validation/test split provided by Venugopalan et al. (2015b) .

EVALUATION METRICS Below, we report results on the video caption generation task. In order to evaluate captions generated by our model, we use model-free automatic evaluation metrics. We adopt METEOR, BLEU@N, and CIDEr metrics available from the Microsoft COCO Caption Evaluation code3 to score the system.

VIDEO AND CAPTION PREPROCESSING We preprocess the captions for both datasets using the Natural Language Toolkit (NLTK)4 . Beyond this, no other type of preprocessing is used.

We extract sample frames for each video and pass each frame through VGGnet (Simonyan & Zisserman, 2014) without any fine-tuning. For the experiments in this paper, we use the feature maps from conv5 3 layer after applying ReLU. The feature map in this layer is 14 × 14 × 512. Our TEM component operates on the flattened 196 × 512 of this feature cubes. For the ablation studies, features from fully connected layer are used as well where the features in this layer have 4096 dimension.

HYPER-PARAMETER OPTIMIZATION We use random search (Bergstra & Bengio, 2012 ) on validation set to select hyper-parameters on both datasets. The word-embedding size, hidden layer size (for both TEM and Decoder), and memory size of the best model on Charades are: 237, 1316, and 437, respectively. These values are 402, 1479, and 797 for the model on MSVD dataset. A stack of two LSTMs are used in the Decoder and TEM.

4.1. VIDEO CAPTION GENERATION:

We first present an ablation analysis to elucidate the contribution of the different components of our proposed model. Next, we compare the overall performance of our model on video caption generation task to other models.

ABLATION ANALYSIS:

We first perform a series of ablation studies in order to show the contributions of the different components of our model. Specifically, we show that the importance of each components in our model in caption generation task on MSVD dataset. One ablation (denoted as Att + No TEM) corresponds to a simpler version of our model in which we remove the TEM component and instead we pass each frame of a video through a CNN and extract features from the last fully-connected hidden layer (e.g., fc7). In addition, we replace our HAM component with a simpler version where the model only memorizes the current step instead of all previous steps. In another ablation (denoted as No HAM + TEM), we remove the HAM component from our model and keep the rest of our model as it is. In the next variation (denoted as HAM + No TEM), we remove the TEM component and calculate features for each frame, similar to Att + No TEM. Finally, the last row in the table is our proposed model (denoted HAM + TEM) with all its components.

Table 1 reports the result of this study. In this experiment, we sample 40 frames per video and use them as the inputs to a network. As the results show, HAM plays a critical role in our proposed model, and removing it causes a drop in performance. On the other hand, removing TEM by itself does not drop performance as much as dropping the HAM. When we put the two together, they complement one another, resulting in better performance. (Tran et al., 2015) (denoted as C3D), or fine-tuned CNN features (denoted as FT) on the action recognition task with dataset such as UCF-101. The only exception happens when we compare our model with (Yu et al., 2016) , who uses C3D features. In this method, adding C3D features leads to a huge improvement in their results (compare row 4 with 11 in Table 2 ). On the other hand, our method without using any external features can achieve better results in comparison with all other methods. This is important because our proposed architecture can alone not only learn a representation for video that can model the temporal structure of a video sequence, but also a representation that can effectively map visual space to the language space.

Table 2 : Video captioning evaluation on the test set of 670 videos in MSVD. Method METEOR BLEU@1 BLEU@2 BLEU@3 BLEU@4 CIDEr In addition, we report results on the Charades dataset for video caption generation. This dataset is challenging because only a few captions per video (about 2 per video) are available. In this experiment, we use 16 frames per video as the inputs to the TEM module. Table 3 shows the performance of our method on this dataset. Our method can achieve 10% improvement over Venugopalan et al. (2015a) in the caption generation task. It is worth noting that a human can only achieve a score of 24 in METEOR for this dataset, which illustrated the level of difficulty in this dataset.

QUALITATIVE RESULTS:

We show some captions generated by our model in 2. The model mostly generates correct captions for cases where content and ground truth captions are consistent. There are some cases in which our model makes some mistakes. For example, in a 'a dog is on a trampoline' video, our model generated "a man is washing a bath" as a caption. This is interesting because the 'man' object only appears in a few frames (1 or 2), but our model can still recognize the man object in the video. 

5. CONCLUSION:

We introduce an end-to-end memory-based attention model to describe an input video using natural language description, similar to the central executive system proposed in human cognition. Our model utilizes memories of past attention when reasoning about where to attend to in the current time step. This allows the model to not only reason about local attention more effectively, but also allows it to consider the entire sequence of video frames while generating each word. Our experiments have confirmed that the memory components in our architecture play a significant role in improving the performance of the entire network. It is worth noting that in this paper, we consider the problem of video caption generation, but our architecture can be applied to any sequence learning problem, which we hope to explore in the future.

:

Only 16087 out of

27, 847 are used as captions for our evaluation since the 27, 847 refers to script of the video as well as captions.

https://www.mturk.com/mturk/welcome

https://github.com/tylin/coco-caption

http://www.nltk.org/

in the Table 2 means, that score was not reported by the corresponding paper.

Figures and Tables:

Figure 1: Our proposed architecture. Each component of our model is described in 3.1 through 3.3

Figure 2: Example captions generated by our model on the test video for MSVD. Incorrect caption cases are shown in red.

Ablation of our model with and without the HAM component on the test set of 670 videos extensively evaluate our model, we compare our model with state-of-the-art models and baselines for the video caption generation task on the MSVD dataset. In this experiment, we use 8 frames per video as the inputs to the TEM module. Table2 5shows the results for this experiment. As the results show, our model gets state-of-the-art scores either in BLEU-4 or METEOR, compared to other methods. This is particularly noteworthy because we do not use external features for the video, such as Optical Flow(Brox et al., 2004) (denoted as Flow in table), 3-Dimensional Convolutional Network features

Table content: <table><tr><td>Method</td><td colspan="6">METEOR BLEU@1 BLEU@2 BLEU@3 BLEU@4 CIDEr</td></tr><tr><td>Att + No TEM</td><td>31.20</td><td>77.90</td><td>65.10</td><td>55.3</td><td>44.90</td><td>63.90</td></tr><tr><td colspan="2">No HAM + TEM 30.5</td><td>78.10</td><td>65.20</td><td>55.10</td><td>44.60</td><td>60.50</td></tr><tr><td colspan="2">HAM + No TEM 31.0</td><td>78.70</td><td>66.90</td><td>57.40</td><td>47.0</td><td>62.10</td></tr><tr><td>HAM + TEM</td><td>31.70</td><td>79.0</td><td>66.20</td><td>56.0</td><td>45.6</td><td>62.20</td></tr><tr><td colspan="2">PERFORMANCE COMPARISON</td><td/><td/><td/><td/><td/></tr><tr><td>Next, to</td><td/><td/><td/><td/><td/><td/></tr></table>

Video captioning evaluation on the test set of 1863 videos in Charades. Method METEOR BLEU@1 BLEU@2 BLEU@3 BLEU@4 CIDEr

Table content: <table><tr><td colspan="2">Human(Sigurdsson et al., 2016) 24</td><td>62</td><td>43</td><td>29</td><td>20</td><td>53</td></tr><tr><td>Sigurdsson et al. (2016)</td><td>16</td><td>49</td><td>30</td><td>18</td><td>11</td><td>14</td></tr><tr><td>Our Model</td><td>17.6</td><td>50</td><td>31.1</td><td>18.8</td><td>11.5</td><td>16.7</td></tr><tr><td/><td/><td/><td colspan="2">Our Captions</td><td colspan="2">Ground Truth</td></tr><tr><td/><td/><td/><td colspan="2">A group of</td><td>A group of</td><td/></tr><tr><td/><td/><td/><td colspan="2">people are</td><td colspan="2">young children</td></tr><tr><td/><td/><td/><td colspan="2">dancing</td><td>performing</td><td/></tr><tr><td/><td/><td/><td/><td/><td>together</td><td/></tr><tr><td/><td/><td/><td colspan="2">A person is</td><td colspan="2">A woman is</td></tr><tr><td/><td/><td/><td colspan="2">cutting the</td><td colspan="2">cutting garlic</td></tr><tr><td/><td/><td/><td colspan="2">vegetable</td><td/><td/></tr><tr><td/><td/><td/><td colspan="2">A man is</td><td>A man is</td><td/></tr><tr><td/><td/><td/><td colspan="2">playing a</td><td>playing the</td><td/></tr><tr><td/><td/><td/><td colspan="2">guitar</td><td>guitar</td><td/></tr><tr><td/><td/><td/><td colspan="2">A young girl is</td><td colspan="2">A little girl is</td></tr><tr><td/><td/><td/><td colspan="2">playing the</td><td>talking on a</td><td/></tr><tr><td/><td/><td/><td>flute</td><td/><td>cordless</td><td/></tr><tr><td/><td/><td/><td/><td/><td>telephone</td><td/></tr><tr><td/><td/><td/><td colspan="2">A woman is</td><td colspan="2">A woman is</td></tr><tr><td/><td/><td/><td colspan="2">pouring eggs</td><td>pouring</td><td/></tr><tr><td/><td/><td/><td colspan="2">into a bowl</td><td colspan="2">ingredients into</td></tr><tr><td/><td/><td/><td/><td/><td>a bowl</td><td/></tr><tr><td/><td/><td/><td colspan="2">A man is</td><td>A man is</td><td/></tr><tr><td/><td/><td/><td colspan="2">playing a flute</td><td colspan="2">playing a large</td></tr><tr><td/><td/><td/><td/><td/><td>flute</td><td/></tr><tr><td/><td/><td/><td colspan="2">A woman is</td><td colspan="2">A woman is</td></tr><tr><td/><td/><td/><td colspan="2">applying</td><td>putting on</td><td/></tr><tr><td/><td/><td/><td colspan="2">makeup</td><td>makeup</td><td/></tr><tr><td/><td/><td/><td colspan="2">A man is</td><td>A guy is</td><td/></tr><tr><td/><td/><td/><td colspan="2">cutting a gun</td><td colspan="2">shooting a gun</td></tr><tr><td/><td/><td/><td colspan="2">A man is</td><td colspan="2">A dog is on a</td></tr><tr><td/><td/><td/><td colspan="2">washing a bath</td><td>trampoline</td><td/></tr></table>



Table content: <table/>

Peer review (iclr2017):
The paper proposes an attention-based approach for video description. The approach uses three LSTMs and two attention mechanisms to sequentially predict words from a sequence of frames. In the LSTM-encoder of the frames (TEM), the first attention approach predicts a spatial attention per frame, and computes the weighted average. The second LSTM (HAM) predicts an attention over the hidden states of the encoder LSTM. The third LSTM which run temporally in parallel to the second LSTM generates the sentence, one word at a time. Strength: =============== - The paper works on a relevant and interesting problem. - Using 2 layers of attention in the proposed way have to my knowledge not been used before for video description. The exact architecture is thus novel (but the work claims much more without sufficient attribution, see blow) - The experiments are evaluated on two datasets, MSVD and Charades, showing performance on the level of related work for MSVD and improvements for Charades. Weaknesses: =============== 1. Claims about the contribution/novelty of the model seem not to hold: 1.1. One of the main contributions is the Hierarchical Attention/Memory (HAM): 1.1.1. It is not clear to me how the presented model (Eq 6-8), are significantly different from the presented model in Xu et al / Yao et al. While Xu et al. attends over spatial image locations and Yao et al. attend over frames, this model attends over encoded video representations h_v^i. A slight difference might be that Xu et al. use the same LSTM to generate, while this model uses an additional LSTM for the decoding. 1.1.2. The paper states in section 3.2 “we propose f_m to memorize the previous attention”, however H_m^{t’-1} only consist of the last hidden state. Furthermore, the model f_m does not have access to the “attention” alpha. This was also discussed in comments by others, but remains unclear. 1.1.3. In the discussion of comments the authors claim that “attention not only is a function a current time step but also a function of all previous attentions and network states.”: While it is true that there is a dependency but that is true also for any LSTM, however the model does not have access to the previous network states as H_g^{t’-1} only consist of the last hidden state, as well as H_m^{t’-1} [at least that is what the formulas say and what Figure 1 suggests]. 1.1.4. The authors claim to have multi-layer attention in HAM, however it remains unclear where the multi-layer comes from. 1.2. The paper states that in section 3.1. “[CNN] features tend to discard the low level information useful in modeling the motion in the video (Ballas et al., 2016).” This suggests that the approach which follows attacks this problem. However, it cannot model motion as attention ho between frames is not available when predicting the next frame. Also, it is not clear how the model can capture anything “low level” as it operates on rather high level VGG conv 5 features. 2. Related work: The difference of HAM to Yao et al. and Xu et al. should be made more clear / or these papers should be cited in the HAM section. 3. Conceptual Limitation of the model: The model has two independent attention mechanisms, a spatial one, and a temporal one. The spatial (within a frame) is independent of the sentence generation. It thus cannot attend to different aspects of the frames for different words which would make sense. E.g. if the sentence is “the dog jumps on the trampoline”, the model should focus on the dog when saying “dog” and on the trampoline when saying “trampoline”, however, as the spatial attention is fixed this is difficult. Also, the encoder model does not have an explicitly way to look at different aspects in the frame during the encoding so might likely get stuck and always predict the same spatial attention for all frames (or it might e.g. always attend to the dog which moves around, but never on the scene). 4. Eq 11 contradicts Fig 1: How is the model exactly receiving the previous word as input. Eq. 11 suggests it is the softmax. If this is the case, the authors should emphasize this in the text as this is unusual. More common would be to use the ground truth previous word during training (which Fig 11 suggests) and the “hardmax”, i.e. the highest predicted previous word encoded as one-hot vector at test time. 5. Clarity: 5.1. It would be helpful if the same notation would be used in Eq 2-5 and 6-9. Why is a different notation required? 5.2. It would be helpful if Fig 1 could contain more details or additional figures for the corresponding parts would be added. If space is a problem, e.g. the well-known equations for LSTM, softmax (Eq 2), and log likelihood loss (Eq 12) could be omitted or inlined. 6. Evaluation: 6.1. The paper claims that the “the proposed architecture outperforms all previously proposed methods and leads to a new state of the art results”. 6.1.1. For the MSVD dataset this clearly is wrong, even given the same feature representation. Pan et al. (2016 a) in Table 2 achieve higher METEOR (33.10). 6.1.2. For this strong claim, I would also expect that it outperforms all previous results independent of the features used, which is also wrong again, Yu et al achieve higher performance in all compared metrics. 6.1.3. For Charades dataset, this claim is also too bold as hardly any methods have been evaluated on this dataset, so at least all the ablations reported in Table 1 should also be reported for the Charades dataset, to make for this dataset any stronger claims. 6.2. Missing qualitative results of attention: The authors should show qualitative results of the attention, for both attention mechanisms to understand if anything sensible is happening there. How diverse is the spatial and the temporal attention? Is it peaky or rather uniform? 6.3. Performance improvement is not significant over model ablations: The improvements over Att+No TEM is only 0.5 Meteor, 0.7 Blue@4 and the performance drops for CIDEr by 1.7. 6.4. Missing human evaluation: I disagree with the authors that a human evaluation is not feasible. 1. An evaluation on a subset of the test data is not so difficult. 2. Even if other authors do not provide their code/model [and some do], they are typically happy to share the predicted sentences which is sufficient and even better for human evaluation [if not I would explicitly mention that some authors did not share sentences, as this seems clearly wrong]. 3. For model ablations the sentences are available to the authors. 7. Several of the comments raised by reviewers/others have not yet been incorporated in a revised version of the paper and/or are still not clear from the explanations given. E.g. including SPICE evaluation and making fixes seems trivial. 8. Hyperparameters are inconsistent: Why are the hyperparemters inconsistent between the ablation analysis (40 frames are sampled) and the performance comparison (8 frames)? Should this not be selected on the validation set? What is the performance of all the ablations with 8 frames? Other (minor/discussion points) - Equation 10: what happens with h_m, and h_g, the LSTM formulas provided only handle two inputs. Are h_m and h_g concatenated. - There is a section 4.1 but no 4.2. - The paper states in section 4.1 “our proposed architecture can alone not only learn a representation for video that can model the temporal structure of a video sequence, but also a representation that can effectively map visual space to the language space.” However, this seems to be true also for many/most other approaches, e.g. [Venugopalan et al. 2015 ICCV] Summary: =============== While the paper makes strong claims w.r.t. to the approach and results, the approach lacks novelty and the results are not convincing over related work and ablations. Furthermore, improved clarity and visualizations of the model and attention results would benefit the paper.