Title: MEMORY-AUGMENTED ATTENTION MODELLING FOR VIDEOS

Abstract: Recent works on neural architectures have demonstrated the utility of attention mechanisms for a wide variety of tasks. Attention models used for problems such as image captioning typically depend on the image under consideration, as well as the previous sequence of words that come before the word currently being generated. While these types of models have produced impressive results, they are not able to model the higher-order interactions involved in problems such as video description/captioning, where the relationship between parts of the video and the concepts being depicted is complex. Motivated by these observations, we propose a novel memory-based attention model for video description. Our model utilizes memories of past attention when reasoning about where to attend to in the current time step, similar to the central executive system proposed in human cognition (Baddeley & Hitch, 1974) . This allows the model to not only reason about local attention more effectively, it allows it to consider the entire sequence of video frames while generating each word. Evaluation on the challenging and popular MSVD and Charades datasets show that the proposed architecture outperforms all previously proposed methods and leads to a new state of the art results in the video description.

1. INTRODUCTION:

Deep neural architectures have led to remarkable progress in computer vision and natural language understanding problems. Image captioning is one such application that has seen the combination of convolutional structures (Krizhevsky et al., 2012; LeCun et al., 1998) , which have been shown to be very effective for problems like object detection, with sequential recurrent structures, which have been shown to be very effective for problems like machine translation (Sutskever et al., 2014) . One of the key modelling paradigms shared by most models for image captioning is the notion of an attention mechanism that guide the model to attend to certain parts of the image while generating.

The attention models used for problems such as image captioning typically depend on the single image under consideration and the partial output generated so far, jointly capturing one region of an image and the words being generated. However, such models cannot capture the temporal reasoning necessary to effectively produce words that refer to actions and events taking place over multiple frames in a video. For example, in a video depicting "someone waving a hand", the "waving" action can start from any frame and can continue on for a variable number of following frames. More importantly, it is likely in a given video quite few frames do not contain any useful information or motion in regard to a given task. Given this, it is not surprising that even with recent advancements in image captioning Xu et al. (2015a) ; Johnson et al. (2016) ; Vinyals et al. (2015) , video captioning has remained challenging.

Motivated by these observations, we introduce a memory-based attention mechanism for video captioning and description. Our model utilizes memories of past attention in the video when reasoning about where to attend to in a current time step. This allows the model to not only effectively leverage local attention, but also to consider the entire video as it generates each word. This mechanism is similar to the proposed central executive system in human cognition, which is thought to permit human performance on two simultaneous tasks (e.g., seeing and saying) using two separate percep-tual domains (e.g., visual and linguistic) by binding information from both sources into coherent structure that enables coordination, selective attention, and inhibition.

Our work shares the same goals as recent work on attention mechanisms for sequence-to-sequence architectures, such as Rocktäschel et al. (2016) and Yang et al. (2016) . However, there are major differences between this work and our current work. Rocktäschel et al. (2016) considers the domain of entailment relations, where the goal is to determine entailment given two input sentences. They propose a soft attention model that is not only focused on the current state, but the previous as well. In our model, we explicitly store all previous attention into memory. In addition, our memory memorizes the encoded version of the input videos conditioned on previously seen words. Even though Yang et al. (2016) and our work both try to solve the problem of locality of attention, our work is different from them in how the memory architecture is modelled. More specifically, they incorporate discriminative supervision into their "reviewer" mechanism, which is not the case in our model. Further, their model is applied to image caption generation, which is to some extent simpler than video caption generation because there is no temporal structure to model. We apply our model on the video captioning problem and evaluate it on the MSVD (Chen & Dolan, 2011) and the Charades (Sigurdsson et al., 2016) datasets. Experimental results show that our proposed architecture outperforms all previous methods and leads to new state of the art results. While we have chosen the video captioning problem for our experiment, the model is general enough that it can be applied to other problems where attention models are used.

2. RELATED WORK:

One of the primary challenges in learning a mapping from a visual space (i.e., video or image) to a language space is learning a representation that not only effectively represents each of these modalities, but is also able to translate a representation from one space to the other. Rohrbach et al. (2013) developed a model that generates a semantic representation of visual content that can be used as the source language for the language generation module. Venugopalan et al. (2015b) proposed a deep method to translate a video into a sentence where an entire video is represented with a single vector based on the mean pool of frame features. However, representing a video by an average of its frames misses the temporal structure of the video. To address this problem, recent work (Yao et al., 2015; Pan et al., 2016a; Venugopalan et al., 2015a; Andrew Shin, 2016; Pan et al., 2016b; Xu et al., 2015b; Ballas et al., 2016; Yu et al., 2016) proposed methods to model temporal structure of video as well as language.

The majority of these methods are inspired by sequence to sequence (Sutskever et al., 2014) and attention (Bahdanau et al., 2015) models. Sequence learning (Sutskever et al., 2014) was originally proposed to map the input sequence of a source language to a target language. Even though applying this method with a combination of attention to the problem of translating a video to a description shows promising results, there are some shortcomings. First of all, modelling the video content with a fixed-length vector in order to map it to a language space is a much harder problem than mapping from a language to a language given the complexity of visual content. Since not all frames in a video are equally salient for a short description, and an event can happen in multiple frames, it is important for a model to identify which frames are most salient. Further, the model should be able to focus on points of interest within these frames to select what to talk about. Even using a variable-length vector to represent a video using attention (Yao et al., 2015) can have some problems. More specifically, current attention methods are local Yang et al. (2016) , since the attention mechanism works in a sequential structure, and lacks the ability to capture global structure. Moreover, combining a video and a language description as a sequence to sequence is usually done by some variant of a recurrent neural network (RNN) (Hochreiter & Schmidhuber, 1997) . Given the limited capacity of a recurrent network to model very long sequences, memory networks (Weston et al., 2014; Sukhbaatar et al., 2015) have been introduced to help the RNN memorize sequences. However, one problem these memory networks suffer from is the difficulty in training the model. The model in Weston et al. (2014) requires supervision at each layer which makes training with backpropagation a challenging task. Even though Sukhbaatar et al. (2015) proposed a memory network that can be trained endto-end, working with memory is still a challenging problem in deep learning especially with write operation (Graves et al., 2014) . To address these problems, we propose a memory-based attention sequence-to-sequence model that not only can learn hierarchical attention relationships, but provides a simple and effective memory structure. In the next section, we explain our model in more detail.

3. LEARNING TO ATTEND AND MEMORIZE:

Our goal is to design an architecture that learns where to look and what to look for in a video, in order to talk about it in the description. To achieve this goal, we formulate the problem as sequence learning to maximize the probability of generating a correct description given a video:

EQUATION

where S is the description, f 1 , f 2 , ..., f N are the input video frames, and Θ is the model parameter vector. The main modelling challenges in video description are to develop a system that can model the temporal structure of the video, learn to attend to the important parts of a video, how to memorize the video that was described given all the generated words so far, then generate a new word by looking at the entire video. To address these issues, we propose an end-to-end network that has three components (Figure 1 ): Temporal Model (TEM), Hierarchical Attention/Memory (HAM), and the Decoder. The goal in TEM is to capture the temporal structure and track motion in a video. The HAM component acts as a hierarchical attention or memory between an input video and the description. More specifically, the HAM learns a hierarchical attention structure that learns where to attend in video given all previously generated words and previous states. The HAM can be interpreted as a memory structure as well, where it learns to memorize an encoded version of a video with language. HAM provides the decoder with the ability to look at an entire video plus all the previously generated words before generating any new words. This is important because a single action normally exhibits multiple frames in the input video. By employing the HAM, the model can effectively model the action over these frames. One of the main contributions of this work is to use a global state to generate any new word. This global state aggregates information from previously generated words and all input frames. We will first describe each component of our model, then explain details of training and inference.

3.1. TEMPORAL MODELER (TEM):

One important question is how to encode the temporal structure of the input video for caption generation. Recently, it has been shown that Recurrent Neural Networks (RNN) has the ability to model the temporal structure in sequential data such as video (Ballas et al., 2016; Sharma et al., 2015; Venugopalan et al., 2015a) and speech (Graves & Jaitly, 2014) . Moreover, since frame-to-frame temporal variation tend to be local (Brox & Malik, 2011) and critical in the motion modeling (Ballas et al., 2016) , it is important to consider a frame representation that can preserve frame-to-frame temporal variation. Even though using features extracted from the fully connected layers of Convolutional Neural Networks (CNNs) have shown state of the art results in image classification and recognition (Simonyan & Zisserman, 2014; He et al., 2016) , these features tend to discard the low level information useful in modeling the motion in the video (Ballas et al., 2016) .

To address the temporal modeling and video representation problems, we use an RNN to model the temporal structure of the video where at each time step, a frame encoding with size of R D is used as an input to the RNN. Instead of extracting features from a top layer of the pretrained CNN, intermediate convolutional maps have been extracted for the video frames. Specifically, for a given video, X, with N frames

X = [X 1 , X 2 , • • • , X N ],

N convolutional maps of size R L×D are extracted where D is dimension of a feature corresponding to L locations in the input frame.

In order to let the network selectively focus on these L locations of each frame given the hidden state of RNN, we apply a soft attention model Bahdanau et al. (2015) ; Xu et al. (2015a) ; Sharma et al. (2015) , called "Location Attention (Latt)". More specifically, by using a softmax, each hidden state produces L probabilities to specify which part of the input is more important, and then creates an input map for the RNN using these probabilities. The f Latt is defined as follow:

ρ t j = exp((h t-1 v ) T W j p ) L k=1 exp((h t-1 v ) T W k p )

(2)

EQUATION

)

where

h t-1 v ∈ R K is hidden state of RNN at t -1, W p ∈ R K×L , and F t ∈ R D .

At each time step, TEM learns a vector representation for each frame, looking at the frame convolution map, and applying the location attention on this map conditioned on all previously seen frames.

EQUATION

EQUATION

where f v can be a vanilla RNN, LSTM, or GRU and Θ v is the parameters of the f v . Due to the fact that vanilla RNNs have gradient vanishing and exploding problems (Pascanu et al., 2013) , we use gradient clipping to deal with gradient exploding, and an LSTM with the following flow to deal with the gradient vanishing problem:

i t = σ(F t W x i + h t-1 v W h i ) f t = σ(F t W x f + h t-1 v W h f ) o t = σ(F t W xo + h t-1 v W ho ) g t = tanh(F t W xg + h t-1 v W hg ) c t v = f t c t-1 v + i t g t h t v = o t tanh(c t )

where

W h * ∈ R K×K , W x * ∈ R D×K , and we define Θ v = {W h * , W x * }.

3.2. HIERARCHICAL ATTENTION/MEMORY (HAM):

One problem with using sequence-to-sequence style architectures (Sutskever et al., 2014) , to model a task such as video language description, is how to find a mapping from a video space to a language space that can capture the relationship between a word and video, or more specifically, the connection between an entire video and an entire sentence where there might not be a clear alignment between the two sequences, as opposed to machine translation and speech recognition. Furthermore, the model should be able to identify which part of the video is more relevant to the description because captions normally focus on a tiny fraction of the facts present in the video. More importantly, once the model starts generating the description, it should still be reminded with the video frames to generate meaningful descriptions. In order to address these problems, we propose a memory-based attention that encodes a video into memory, built as a function of the state of the language generation network (a.k.a. Decoder) and the state of the TEM network. More specifically, our Hierarchical Attention/Memory can be formulated as two following steps:

• Attention update [ F(Θ a )]:

EQUATION

• Memory update:

EQUATION

where

H v ∈ R N ×K , W v and W g are ∈ R K×K , U ∈ R K , W m ∈ R M ×K , and Θ a = {W v , W g , W m , U }. N is number of frames in a given video, H v = [h 1 v , ..., h N v ], H t -1 g = [h t -1 g , ..., h t -1 g ], and H t -1 m = [h t -1 m , ..., h t -1 m ]

. α t is the set of probabilities in a given time step that specifies the attention over an input video state (H v ), memory state (H m ), and decoder state (H g ). In order to let the network remember what has been attended before and the temporal structure of a video, we propose f m to memorize the previous attention and encoded version of an input video with language model. Using f m not only enables the network to memorize previous attention and frames, but also to learn multi-layer attention over an input video and corresponding language. The output of the memory-attention is then used as input to the Decoder.

3.3. DECODER:

In order to generate a new word conditioned on all previous words and HAM states, a recurrent structure is modelled as follows:

EQUATION

where s t is a word vector at t , W e ∈ R K×C and C is the vocabulary size. In addition, ŝt assigns a probability to each word in the language. We use LSTMs for both f m and f g .

3.4. TRAINING AND OPTIMIZATION:

The goal in our network is to predict the next word given all previously seen words and an input video. In order to optimize our network parameters, Θ = {W p , Θ v , Θ a , Θ m , W e }, we minimize a negative log likelihood loss function, formulated as follow:

L(S, X; Θ) = - T j |V | i s j,i log(ŝ j,i ) + λ Θ 2 2 (12)

where |V | is the dictionary size. We fully train our network in an end-to-end fashion using first-order stochastic gradient-based optimization method with an adaptive learning rate. More specifically, in order to optimize our network parameters, we use Adam Kingma & Ba (2015) with learning rate 2 × 10 -5 and set β 1 , β 2 to 0.8 and 0.999, respectively. At the training, we use a batch size of 16. (2016) . It is worth noting that one major difference between this dataset and others is that they use a "Hollywood in Homes" approach to collecting the data (Sigurdsson et al., 2016) , where "actors" are crowdsourced, yielding a diverse scene and actor videos. One reason that we report results on this dataset is because each video has a specific action in it and would be a suitable testbed to evaluate our model.

4. EXPERIMENTS:

MSVD is a set of Youtube videos that are annotated by a Mechanical Turker,2 who was asked to pick a clip from a video that represents an activity. In this dataset, each clip is annotated by multiple workers with a single sentence. This dataset contains 1, 970 videos and about 80, 000 descriptions, where 1, 200 of the videos are training data, 670 are test data, and the rest (i.e., 100 videos) are assigned for validation. In order to make the results comparable with other papers, we follow the exact training/validation/test split provided by Venugopalan et al. (2015b) .

EVALUATION METRICS Below, we report results on the video caption generation task. In order to evaluate captions generated by our model, we use model-free automatic evaluation metrics. We adopt METEOR, BLEU@N, and CIDEr metrics available from the Microsoft COCO Caption Evaluation code3 to score the system.

VIDEO AND CAPTION PREPROCESSING We preprocess the captions for both datasets using the Natural Language Toolkit (NLTK)4 . Beyond this, no other type of preprocessing is used.

We extract sample frames for each video and pass each frame through VGGnet (Simonyan & Zisserman, 2014) without any fine-tuning. For the experiments in this paper, we use the feature maps from conv5 3 layer after applying ReLU. The feature map in this layer is 14 × 14 × 512. Our TEM component operates on the flattened 196 × 512 of this feature cubes. For the ablation studies, features from fully connected layer are used as well where the features in this layer have 4096 dimension.

HYPER-PARAMETER OPTIMIZATION We use random search (Bergstra & Bengio, 2012 ) on validation set to select hyper-parameters on both datasets. The word-embedding size, hidden layer size (for both TEM and Decoder), and memory size of the best model on Charades are: 237, 1316, and 437, respectively. These values are 402, 1479, and 797 for the model on MSVD dataset. A stack of two LSTMs are used in the Decoder and TEM.

4.1. VIDEO CAPTION GENERATION:

We first present an ablation analysis to elucidate the contribution of the different components of our proposed model. Next, we compare the overall performance of our model on video caption generation task to other models.

ABLATION ANALYSIS:

We first perform a series of ablation studies in order to show the contributions of the different components of our model. Specifically, we show that the importance of each components in our model in caption generation task on MSVD dataset. One ablation (denoted as Att + No TEM) corresponds to a simpler version of our model in which we remove the TEM component and instead we pass each frame of a video through a CNN and extract features from the last fully-connected hidden layer (e.g., fc7). In addition, we replace our HAM component with a simpler version where the model only memorizes the current step instead of all previous steps. In another ablation (denoted as No HAM + TEM), we remove the HAM component from our model and keep the rest of our model as it is. In the next variation (denoted as HAM + No TEM), we remove the TEM component and calculate features for each frame, similar to Att + No TEM. Finally, the last row in the table is our proposed model (denoted HAM + TEM) with all its components.

Table 1 reports the result of this study. In this experiment, we sample 40 frames per video and use them as the inputs to a network. As the results show, HAM plays a critical role in our proposed model, and removing it causes a drop in performance. On the other hand, removing TEM by itself does not drop performance as much as dropping the HAM. When we put the two together, they complement one another, resulting in better performance. (Tran et al., 2015) (denoted as C3D), or fine-tuned CNN features (denoted as FT) on the action recognition task with dataset such as UCF-101. The only exception happens when we compare our model with (Yu et al., 2016) , who uses C3D features. In this method, adding C3D features leads to a huge improvement in their results (compare row 4 with 11 in Table 2 ). On the other hand, our method without using any external features can achieve better results in comparison with all other methods. This is important because our proposed architecture can alone not only learn a representation for video that can model the temporal structure of a video sequence, but also a representation that can effectively map visual space to the language space.

Table 2 : Video captioning evaluation on the test set of 670 videos in MSVD. Method METEOR BLEU@1 BLEU@2 BLEU@3 BLEU@4 CIDEr In addition, we report results on the Charades dataset for video caption generation. This dataset is challenging because only a few captions per video (about 2 per video) are available. In this experiment, we use 16 frames per video as the inputs to the TEM module. Table 3 shows the performance of our method on this dataset. Our method can achieve 10% improvement over Venugopalan et al. (2015a) in the caption generation task. It is worth noting that a human can only achieve a score of 24 in METEOR for this dataset, which illustrated the level of difficulty in this dataset.

QUALITATIVE RESULTS:

We show some captions generated by our model in 2. The model mostly generates correct captions for cases where content and ground truth captions are consistent. There are some cases in which our model makes some mistakes. For example, in a 'a dog is on a trampoline' video, our model generated "a man is washing a bath" as a caption. This is interesting because the 'man' object only appears in a few frames (1 or 2), but our model can still recognize the man object in the video. 

5. CONCLUSION:

We introduce an end-to-end memory-based attention model to describe an input video using natural language description, similar to the central executive system proposed in human cognition. Our model utilizes memories of past attention when reasoning about where to attend to in the current time step. This allows the model to not only reason about local attention more effectively, but also allows it to consider the entire sequence of video frames while generating each word. Our experiments have confirmed that the memory components in our architecture play a significant role in improving the performance of the entire network. It is worth noting that in this paper, we consider the problem of video caption generation, but our architecture can be applied to any sequence learning problem, which we hope to explore in the future.

:

Only 16087 out of

27, 847 are used as captions for our evaluation since the 27, 847 refers to script of the video as well as captions.

https://www.mturk.com/mturk/welcome

https://github.com/tylin/coco-caption

http://www.nltk.org/

in the Table 2 means, that score was not reported by the corresponding paper.

Figures and Tables:

Figure 1: Our proposed architecture. Each component of our model is described in 3.1 through 3.3

Figure 2: Example captions generated by our model on the test video for MSVD. Incorrect caption cases are shown in red.

Ablation of our model with and without the HAM component on the test set of 670 videos extensively evaluate our model, we compare our model with state-of-the-art models and baselines for the video caption generation task on the MSVD dataset. In this experiment, we use 8 frames per video as the inputs to the TEM module. Table2 5shows the results for this experiment. As the results show, our model gets state-of-the-art scores either in BLEU-4 or METEOR, compared to other methods. This is particularly noteworthy because we do not use external features for the video, such as Optical Flow(Brox et al., 2004) (denoted as Flow in table), 3-Dimensional Convolutional Network features

Table content: <table><tr><td>Method</td><td colspan="6">METEOR BLEU@1 BLEU@2 BLEU@3 BLEU@4 CIDEr</td></tr><tr><td>Att + No TEM</td><td>31.20</td><td>77.90</td><td>65.10</td><td>55.3</td><td>44.90</td><td>63.90</td></tr><tr><td colspan="2">No HAM + TEM 30.5</td><td>78.10</td><td>65.20</td><td>55.10</td><td>44.60</td><td>60.50</td></tr><tr><td colspan="2">HAM + No TEM 31.0</td><td>78.70</td><td>66.90</td><td>57.40</td><td>47.0</td><td>62.10</td></tr><tr><td>HAM + TEM</td><td>31.70</td><td>79.0</td><td>66.20</td><td>56.0</td><td>45.6</td><td>62.20</td></tr><tr><td colspan="2">PERFORMANCE COMPARISON</td><td/><td/><td/><td/><td/></tr><tr><td>Next, to</td><td/><td/><td/><td/><td/><td/></tr></table>

Video captioning evaluation on the test set of 1863 videos in Charades. Method METEOR BLEU@1 BLEU@2 BLEU@3 BLEU@4 CIDEr

Table content: <table><tr><td colspan="2">Human(Sigurdsson et al., 2016) 24</td><td>62</td><td>43</td><td>29</td><td>20</td><td>53</td></tr><tr><td>Sigurdsson et al. (2016)</td><td>16</td><td>49</td><td>30</td><td>18</td><td>11</td><td>14</td></tr><tr><td>Our Model</td><td>17.6</td><td>50</td><td>31.1</td><td>18.8</td><td>11.5</td><td>16.7</td></tr><tr><td/><td/><td/><td colspan="2">Our Captions</td><td colspan="2">Ground Truth</td></tr><tr><td/><td/><td/><td colspan="2">A group of</td><td>A group of</td><td/></tr><tr><td/><td/><td/><td colspan="2">people are</td><td colspan="2">young children</td></tr><tr><td/><td/><td/><td colspan="2">dancing</td><td>performing</td><td/></tr><tr><td/><td/><td/><td/><td/><td>together</td><td/></tr><tr><td/><td/><td/><td colspan="2">A person is</td><td colspan="2">A woman is</td></tr><tr><td/><td/><td/><td colspan="2">cutting the</td><td colspan="2">cutting garlic</td></tr><tr><td/><td/><td/><td colspan="2">vegetable</td><td/><td/></tr><tr><td/><td/><td/><td colspan="2">A man is</td><td>A man is</td><td/></tr><tr><td/><td/><td/><td colspan="2">playing a</td><td>playing the</td><td/></tr><tr><td/><td/><td/><td colspan="2">guitar</td><td>guitar</td><td/></tr><tr><td/><td/><td/><td colspan="2">A young girl is</td><td colspan="2">A little girl is</td></tr><tr><td/><td/><td/><td colspan="2">playing the</td><td>talking on a</td><td/></tr><tr><td/><td/><td/><td>flute</td><td/><td>cordless</td><td/></tr><tr><td/><td/><td/><td/><td/><td>telephone</td><td/></tr><tr><td/><td/><td/><td colspan="2">A woman is</td><td colspan="2">A woman is</td></tr><tr><td/><td/><td/><td colspan="2">pouring eggs</td><td>pouring</td><td/></tr><tr><td/><td/><td/><td colspan="2">into a bowl</td><td colspan="2">ingredients into</td></tr><tr><td/><td/><td/><td/><td/><td>a bowl</td><td/></tr><tr><td/><td/><td/><td colspan="2">A man is</td><td>A man is</td><td/></tr><tr><td/><td/><td/><td colspan="2">playing a flute</td><td colspan="2">playing a large</td></tr><tr><td/><td/><td/><td/><td/><td>flute</td><td/></tr><tr><td/><td/><td/><td colspan="2">A woman is</td><td colspan="2">A woman is</td></tr><tr><td/><td/><td/><td colspan="2">applying</td><td>putting on</td><td/></tr><tr><td/><td/><td/><td colspan="2">makeup</td><td>makeup</td><td/></tr><tr><td/><td/><td/><td colspan="2">A man is</td><td>A guy is</td><td/></tr><tr><td/><td/><td/><td colspan="2">cutting a gun</td><td colspan="2">shooting a gun</td></tr><tr><td/><td/><td/><td colspan="2">A man is</td><td colspan="2">A dog is on a</td></tr><tr><td/><td/><td/><td colspan="2">washing a bath</td><td>trampoline</td><td/></tr></table>



Table content: <table/>

Peer review (iclr2017):
The authors propose a *hierarchical* attention model for video captioning. They introduce a model composed of three parts: the temporal modeler (TEM) that takes as input the video sequence and outputs a sequential representation of the video to the HAM; the hierarchical attention/memory mechanism (HAM) implements a soft-attention mechanism over the sequential video representation; and finally a decoder that generates a caption. Related to the second series of questions above, it seems as though the authors have chosen to refer to their use of an LSTM (or equivalent RNN) as the output of the Bahdanau et al (2015) attention mechanism as a hierarchical memory mechanism. I am actually sympathetic to this terminology in the sense that the recent popularity of memory-based models seems to neglect the memory implicit in the LSTM state vector, but that said, this seems to seriously misrepresent the significance fo the contribution of this paper. I appreciate the ablation study presented in Table 1. Not enough researchers bother with this kind of analysis. But it does show that the value of the contributions is not actually clear. In particular the case for the TEM is quite weak. Regarding the quantitative evaluation presented in Table 2, the authors are carving out a fairly specific set of features to describe the set of *fair* comparators from the literature. Given the variability of the models and alternate training datasets that are in use, I would find it more compelling if the authors just set about trying to achieve the best results they can, if that includes the fine-tuning of the frame model, so be it. The value of this work is as an application paper, so the discovery and incorporation of elements that can significantly improve performance would seems warranted. Overall, at this point, I do not see a sufficient contribution to warrant publication in ICLR.