Title: ADAPTIVE FEATURE ABSTRACTION FOR TRANSLATING VIDEO TO LANGUAGE

Abstract: A new model for video captioning is developed, using a deep three-dimensional Convolutional Neural Network (C3D) as an encoder for videos and a Recurrent Neural Network (RNN) as a decoder for captions. A novel attention mechanism with spatiotemporal alignment is employed to adaptively and sequentially focus on different layers of CNN features (levels of feature "abstraction"), as well as local spatiotemporal regions of the feature maps at each layer. The proposed approach is evaluated on the YouTube2Text benchmark. Experimental results demonstrate quantitatively the effectiveness of our proposed adaptive spatiotemporal feature abstraction for translating videos to sentences with rich semantic structures.

1. INTRODUCTION:

Accurately understanding the fast-growing number of videos poses a significant challenge for computer vision and machine learning. An important component of video analyasis involves generating natural-language video descriptions, i.e., video captioning. Inspired by the successful deployment of the encoder-decoder framework used in machine translation (Cho et al., 2014) and image caption generation (Vinyals et al., 2015; Pu et al., 2016; Gan et al., 2017) , most recent work on video captioning (Venugopalan et al., 2015; Yu et al., 2016) employs a 2-dimentional (2D) or 3-dimentional (3D) Convolutional Neural Network (CNN) as an encoder, mapping an input video to a compact feature vector representation; a Recurrent Neural Network (RNN) is typically employed as a decoder, unrolling this feature vector to generate a sequence of words of arbitrary length.

Despite achieving encouraging successes in video captioning, previous models suffer from important limitations. First, the rich contents in an input video is often compressed to a single compact feature vector for caption generation; this approach is prone to miss detailed spatiotemporal information. Secondly, the video feature representations are typically extracted from the output of a CNN at a manually-selected fixed layer, which is incapable of modeling rich context-aware semantics that requires focusing on different abstraction levels of features. As investigated in Zeiler & Fergus (2014) ; Simonyan et al. (2014) , the features from layers at or near the top of a CNN tends to focus on global semantic discriminative visual percepts, while low-layer feature provides more local, fine-grained information. It is desirable to select/weight features from different CNN layers adaptively when decoding a caption, selecting different levels of feature abstraction by sequentially emphasizing features from different CNN layers. In addition to focusing on features from different CNN layers, it is also desirable to emphasize local spatiotemporal regions in feature maps at particular layers.

To realize these desiderata, our proposed decoding process for generating a sequence of words dynamically emphasizes different levels (CNN layers) of 3D convolutional features, to model important coarse or fine-grained spatiotemporal structure. Additionally, the model employs different contexts and adaptively attends to different spatiotemporal locations of an input video. While some previous models use 2D CNN features to generate video representations, our model adopts the features from a pre-trained deep 3D convolutional neural network (C3D); such features have been shown to be natural and effective for video representations, action recognition and scene understanding (Tran et al., 2015) by learning the spatiotemporal features that can provide better appearance and motion information. In addition, the proposed model is inspired by the recent success of attention-based models that mimic human perception (Mnih et al., 2014; Xu et al., 2015) .

The principal contributions of this paper are as follows: (i) A new video-caption-generation model is developed by dynamically modeling context-dependent feature abstractions; (ii) New attention mechanisms to adaptively and sequentially emphasize different levels of feature abstraction (CNN layers), while also imposing attention within local spatiotemporal regions of the feature maps at each layer are employed; (iii) 3D convolutional transformations are introduced to achieve spatiotemporal and semantic feature consistency across different layers; (iv) The proposed model achieves state-ofthe-art performance on Youtube2Text benchmark. We call the proposed algorithm Adaptive Spa-tioTemporal representation with dynAmic abstRaction (ASTAR).

2. METHOD:

Consider N training videos, the nth of which is denoted X (n) , with associated caption Y (n) . The length-T n caption is represented

Y (n) = (y (n) 1 , . . . , y (n) Tn ), with y (n) t

a 1-of-V ("one hot") encoding vector, with V the size of the vocabulary.

For each video, the C3D feature extractor (Tran et al., 2015) produces a set of features

EQUATION

L } are feature maps extracted from L convolutional layers, and the fully connected layer at the top, responsible for a (n) L+1 , assumes that the input video is of the same size for all videos. To account for variable-length videos, we employ mean pooling to the video clips, based on a window of length 16 (as in (Tran et al., 2015) ) with an overlap of 8 frames.

2.1. CAPTION MODEL:

For notational simplicity, henceforth we omit superscript n. The t-th word in a caption, y t , is mapped to an M -dimensional vector w t = W e y t , where W e ∈ R M ×V is a learned wordembedding matrix, i.e., w t is a column of W e chosen by the one-hot y t . The probability of caption Y = {y t } t=1,T is defined as

p(Y|A) = p(y 1 |A) T t=2 p(y t |y <t , A) .

(1) Specifically, the first word y 1 is drawn from p(y 1 |A) = softmax(Vh 1 ), where h 1 = tanh(Ca L+1 ). Bias terms are omitted for simplicity throughout the paper. All the other words in the caption are then sequentially generated using an RNN, until the end-sentence symbol is generated. Conditional distribution p(y t |y <t , A) is specified as softmax(Vh t ), where h t is recursively updated as h t = H(w t-1 , h t-1 , z t ). V is a matrix connecting the RNN hidden state to a softmax, for computing a distribution over words. z t = φ(h t-1 , a 1 , . . . , a L ) is the context vector used in the attention mechanism, capturing the relevant visual features associated with the spatiotemporal attention (also weighting level of feature abstraction), as detailed in Sec. 2.2. The transition function H(•) is implemented with Long Short-Term Memory (LSTM) (Hochreiter & Schmidhuber, 1997) .

Given the video X (with features A) and associated caption Y, the objective function is the sum of the log-likelihood of the caption conditioned on the video representation:

EQUATION

Equation ( 2) is a function of all model parameters to be learned; they are not explicitly depicted in (2) for notational simplicity. Further, (2) corresponds to a single video-caption pair, and when training we sum over all such training pairs.

2.2. ATTENTION MECHANISM:

We introduce two attention mechanisms when predicting word y t : (i) spatiotemporal-localization attention, and (ii) abstraction-level attention; these, respectively, measure the relative importance of a particular spatiotemporal location and a particular CNN layer (feature abstraction) for producing y t , based on the word-history information y <t .

To achieve this, we seek to map a l → âl , where 4D tensors âl all have the same dimensions, are embedded into same semantic spaces, and are aligned spatialtemporally. Specifically, âl , l = 1, . . . , L-1 are aligned in the above ways with a L . To achieve this, we filter each a l , l = 1, . . . , L-1, and then apply max-pooling; the filters seek semantic alignment of the features (including feature dimension), and the pooling is used to spatiotemporally align the features with a L . Specifically, consider

EQUATION

for l = 1, . . . , L -1, and with âL = a L . a l (k) is the 3D feature map (tensor) for dictionary k ∈ {1, . . . , n l F } at layer l, and U k,l is a 4D tensor. The convolution * in (3) operates in the three shift dimensions, and a l (k) * U k,l manifests a 4D tensor. Function f (•) is an element-wise nonlinear activation function, followed by max pooling, with the pooling dimensions meant to realize final dimensions consistent with a L . Consequently, âi,l ∈ R n L F is a feature vector.

With {â l } l=1,L semantically and spatiotemporally aligned, we now seek to jointly quantify the value of a particular spatiotemporal region and a particular feature layer ("abstraction") for prediction of the next word. For each âi,l , the attention mechanism generates two positive weights, α ti and β tl , which measure the relative importance of location i and layer l for producing y t based y <t . Attention weights α ti and β tl and context vector z t are computed as e ti = w T α tanh(W aα âi + W hα h t-1 ),

EQUATION

b tl = w T β tanh(W sβ s tl + W hβ h t-1 ),

EQUATION

where âi is a vector composed by stacking {â i,l } l=1,L (all features at position i). e ti and b tl are scalars reflecting the importance of spatiotemporal region i and layer t to predicting y t , while α ti and β tl are relative weights of this importance, reflected by the softmax output. In (4) we provide attention in the spatiotemporal dimensions, with that spatiotemporal attention shared across all L (now aligned) CNN layers. In (5) the attention is further refined, focusing attention in the layer dimension.

3. EXPERIMENTS:

We present results on Microsoft Research Video Description Corpus (YouTube2Text) (Chen & Dolan, 2011) . The Youtube2Text contains 1970 Youtube clips, and each video is annotated with around 40 sentences. For fair comparison, we used the same splits as provided in Yu et al. (2016) , with 1200 videos for training, 100 videos for validation, and 670 videos for testing. We convert all captions to lower case and remove the punctuation, yielding vocabulary sizes V = 12594. We consider the RGB frames of videos as input, and all videos are resized to 112×112 spatially, with 2 frames per second. The C3D (Tran et al., 2015) is pretrained on Sports-1M dataset Karpathy et al. (2014) , consisting of 1.1 million sports videos belonging to 487 categories. We extract the features from four convolutional layers and one fully connected layer, named as pool2, pool3, pool4, pool5 and fc-7 in the C3D (Tran et al., 2015) , respectively.

The widely used BLEU (Papineni et al., 2002) , METEOR (Banerjee & Lavie, 2005) and CIDEr (Vedantam et al., 2015) metrics are employed to quantitatively evaluate the performance of our video caption generation model, and other models in the literature.

Results are summarized in Tables 1, and we outperform the previous state-of-the-art result on Youtube2Text. This demonstrates the importance of leveraging intermediate convolutional layer features. In addition, we achieve these results using a single model, without averaging over an ensemble of such models.

4. CONCLUSION AND FUTURE WORK:

We have proposed a novel video captioning model, that adaptively selects/weights the feature abstraction (CNN layer), as well as the location within a layer-dependent feature map. Our model achieves state-of-the-art video caption generation performance on Youtube2Text benchmark.

:



Figures and Tables:

Results on BLEU-4, METEOR and CIDEr met-

Table content: <table><tr><td colspan="4">rics compared to state-of-the-art results (Yu et al., 2016) on</td></tr><tr><td colspan="2">Youtube2Text. respectively.</td><td/><td/></tr><tr><td>Methods</td><td colspan="3">BLEU-4 METEOR CIDEr</td></tr><tr><td>h-RNN [4]</td><td>49.9</td><td>32.6</td><td>65.8</td></tr><tr><td>ASTAR</td><td>51.74</td><td>36.39</td><td>72.18</td></tr></table>

Peer review (iclr2017):
1) Summary This paper proposes a video captioning model based on a 3D (space+time) convnet (C3D) encoder and a LSTM decoder. The authors investigate the benefits of using attention mechanisms operating both at the spatio-temporal and layer (feature abstraction) levels. 2) Contributions + Well motivated and implemented attention mechanism to handle the different shapes of C3D feature maps (along space, time, and feature dimensions). + Convincing quantitative and qualitative experiments on three challenging datasets (Youtube2Text, M-VAD, MSR-VTT) showing clearly the benefit of the proposed attention mechanisms. + Interesting comparison of soft vs hard attention showing a slight performance advantage for the (simpler) soft attention mechanism in this case. 3) Suggestions for improvement Hypercolumns comparison: As mentioned during pre-review questions, it would be interesting to compare to the hypercolumns of https://arxiv.org/abs/1411.5752, as they are an alternative to the proposed attention mechanisms, with the same purpose of leveraging different feature abstraction levels. Minor clarifications in the text and figures as agreed with the authors in our pre-review discussions. 4) Conclusion Although the novelty with existing video captioning approaches is limited, the paper is relevant to ICLR, as the proposed simple but efficient implementation and benefits of spatio-temporal + feature abstraction attention are clearly validated in this work.