Multi-modal Dependency Tree for Video Captioning

1 Introduction . Video captioning has received extensive attention from researchers in both computer vision and natural language processing . It is a challenging task since it requires not only understanding the visual content but also generating fluent and relevant sentences . With the success of deep neural networks in natural language processing , many existing video captioning methods [ 1 , 2 ] use recurrent neural network ( RNN ) to generate sentences , which processes sequences by updating hidden states . Several recent studies [ 3–6 ] apply Transformer [ 7 ] , which relates the words in the sequences using a self-attention mechanism , to video captioning . All these methods treat each sentence as a word sequence and generate words in a predefined left-to-right order by capturing the relatively close contextual relationship between words in the sentence . In this paper , we investigate a graph structured model for video captioning to explicitly model the hierarchical structure in the sentence and capture the long-range dependency between words . With this in mind , we propose to generate a sentence by first constructing a multi-modal dependency tree in a top-down and depth-first order , and then traversing the tree in a recursive manner , as shown in Figure 1 . Each node of the tree is represented by a multi-modal embedding representation that integrates the information from both visual and textual modalities . During the tree construction , each ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . newly generated node receives the multi-modal embeddings from its parent and sibling nodes , and these multi-modal embeddings are used to predict the attention weights of the input features and the word . The attended visual feature and the predicted word are fused to generate the multi-modal embedding of the new node . Different from existing sequence models that tend to focus on the dependency between each word and its close preceding words , our tree model sufficiently captures the global dependency structure in the sentence to further improve the fluency of the generated sentences . In contrast to the unimodal dependency tree widely used in dependency parsing for language analysis , our multi-modal dependency tree effectively integrates both the visual and linguistic information , improving the relevance of the output sentence to the visual input . To effectively optimize the captioning model , we propose a tree-structured reinforcement learning algorithm and a novel node-level reward tailor-made for the tree construction process . By evaluating the consistency between the parent-child node pairs in the constructed tree and those in the ground truth trees , the tree node-level reward enables the model to capture the topological structure of the ground-truth dependency trees . Compared to the sequence-level rewards in existing methods , our node-level reward recognizes the contribution of each node to the multi-modal dependency tree and avoids the reward ambiguity problem [ 8 ] . To evaluate the effectiveness of the proposed method , we conduct experiments on two difficult video captioning datasets , the ActivityNet Captions dataset and Charades Captions dataset . We also perform experiments on two most widely-used datasets , namely the MSVD dataset and MSR-VTT dataset . Compared to MSVD and MSR-VTT , the sentences in the ActivityNet Captions dataset and Charades Captions dataset are typically longer and describe more complex activities in the videos . The experiments on these datasets demonstrate that our method not only generates long and complex sentences with high fluency and relevancy , but is also effective for relatively simple sentences . The main contributions of this paper are as follows : • We propose a multi-modal dependency tree construction method for video captioning . With the help of tree topology , our model generates more fluent and relevant sentences by effectively capturing the syntactic and semantic dependencies in natural language , especially when generating long and complex sentences . • We develop a novel tree-structured reinforcement learning algorithm that optimizes the captioning model using a node-level reward , which facilitates learning the topology of the dependency trees and alleviates the reward ambiguity problem . 2 Related Work . 2.1 Video Captioning . Thanks to the recent advances in computer vision and natural language processing , many video captioning methods based on the encoder-decoder framework have been proposed . The pioneers of video captioning methods [ 9 ] employ a convolutional neural network ( CNN ) to extract visual features and a recurrent neural network ( RNN ) to generate the words in the sentences in a sequential manner . To further improve the performance of video captioning , some methods [ 10 , 11 ] apply the attention mechanism in video captioning , which enables the model to focus on different temporal segments when generating the sentences . Inspired by the advances in natural language processing , some methods [ 12 , 13 ] propose to learn better syntax representations with the help of part-of-speech tags . In order to provide richer semantic information to video captioning models , Rahman et al . [ 14 ] attempt to utilize the audio information . Motivated by the advances in neural machine translation , several recent methods [ 3 , 4 , 6 ] investigate the application of Transformer [ 7 ] to video captioning , which uses multi-head self-attention mechanism to yield more expressive representations of the input . Some methods [ 15 , 16 ] focus on optimizing video captioning models using reinforcement learning , where the rewards are calculated based on the n-gram statistics . Instead of directly generating a word sequence , our method generates a sentence by first constructing a dependency tree and then traversing the tree . Compared to the sequence generation models , our method explicitly models the grammatical structure with tree topology and better preserves the semantic relationship by incorporating both visual and linguistic information into the dependency tree . 2.2 Tree-structured Language Generation . In recent years , tree-structured language generation methods are proposed for various natural language processing tasks , including program generation [ 17 ] , generating math equation [ 18 ] and machine translation [ 19 ] . Alvarez et al . [ 17 ] generate the tree nodes in depth-first search order using doublyrecurrent neural network , which combines the hidden states from the parent node and the last sibling node when predicting the label for a new node . A dependency tree generation method is proposed by [ 19 ] to generate target sentences in machine translation . Instead of directly constructing the tree structure , this method generates canonizalized ternary trees where each node has a fixed number of child nodes . Liu et al . [ 18 ] propose to generate abstract syntax trees of math expressions to solve math problems described in natural language . The work closest to our method is [ 20 ] that generates image captions by constructing canonicalized dependency trees . Different from this method that generates trees with only linguistic information , our method constructs multi-modal dependency trees that contain both visual and linguistic information to capture richer contextual information . 3 Our Method . 3.1 Overview . In this section , we introduce the proposed multi-modal dependency tree generation framework . The training data is formulated as D = { ( xj , Tj ) |j } , where xj denotes the j-th video , and Tj denotes the dependency tree of the corresponding sentence . Our model follows the encoder-decoder paradigm , where both the encoder and the decoder are stacks of N self-attention layers . Given an input video , the encoder of our model takes a set of d-dimensional visual features F = { f1 , f2 , ... , fM } as input , where fi ∈ Rd . The decoder firstly generates a multi-modal dependency tree T = { U , E } , where U and E denote the node set and edge set , respectively . Each node ui ∈ U corresponds to a word in the sentence , and a directed edge 〈uj , ui〉 ∈ E indicates that the node ui is dependent on its parent node uj . Then we use a recursive algorithm to traverse the tree T and recover the order of the words in the output sentence y = { w1 , w2 , ... , wL } , where L denotes the length of the sentence and wi denotes the i-th word . The overview of our framework is illustrated in Figure 2 . 3.2 Tree Generation Process . Our tree generation model follows a step-by-step generation process . In the i-th step , a node ui is added to the tree , and the corresponding word wi is predicted . Formally , given a video x , the probability of generating a tree T is given by p ( T |x ) = ∏ ui∈T p ( ui|x , Ti−1 ) , ( 1 ) where Ti−1 denotes the sub-tree that has been generated in the ( i− 1 ) -th step . Since the structure of the dependency tree is unknown in the tree generation process , our model also predicts the topological information related to each node . For each node ui , we define three topological labels : tsi ∈ { 0 , 1 } , tci ∈ { 0 , 1 } and tei ∈ { 0 , 1 } . tsi indicates whether node ui has another sibling node . If tsi = 1 , another sibling node of ui will be added to the tree in the following steps . tci decides whether ui have child nodes , and least one child node of ui will be added when t c i = 1. tei = 0 indicates that the node ui is the left child of its parent node uj , namely the word wi appears on the left of wj in the output sentence , and tei = 1 indicates that ui is the right child of uj . 3.3 Encoder . Given a set of visual features F = { f1 , f2 , ... , fM } , the encoder uses multiple self-attention operations to capture the relationships between video frames represented by the visual features . We first recall the scaled dot-product attention used by the self-attention layers , which operates on three sets of vectors Q , K and V : Attention ( Q , K , V ) = softmax ( QK > √ d ) V , ( 2 ) where Q ∈ Rnq×d is a matrix consisting of nq query vectors , K ∈ Rnk×d and V ∈ Rnk×d are matrices containing nk key vectors and nk value vectors , respectively . The vectors in Q , K and V are with the same dimensionality d. In addition to the self-attention layer , each encoder layer contains a feed-forward network ( FFN ) consisting of two fully connected layers , which can be formulated as FFN ( X ) = ReLU ( XW1 + b1 ) W2 + b2 , ( 3 ) where W1 , W2 ∈ Rd×d and b1 , b2 ∈ Rd are learnable parameters . In each layer of the encoder , the queries , keys and values are obtained by linearly mapping the input features , and the output of the n-th encoding layer Xn is given by X′n = Attention ( W q nXn−1 , W k nXn−1 , W v nXn−1 ) , Xn = FFN ( X ′ n ) , ( 4 ) where W qn ∈ Rd×d , W kn ∈ Rd×d and W vn ∈ Rd×d are learnable parameters , and Xn−1 is the output of the previous encoding layer . FFN represents the feed-forward network defined in Eq . 3 . The visual features V are used as the input of the first encoding layer .

Summary: This paper studies the task of video captioning. It proposes a structured approach where models generate a dependency tree for the caption, instead of just a sentence. The nodes of the dependency tree are words, and the tree is generated in depth-first order; thus, the sentence can be easily recovered by iterating through the tree.   The model is learned by a "pretraining stage"; where a left-to-right style loss is used (generating the next word, and the node in the tree, given previous nodes). In a "finetuning stage", the model is optimized using reinforcement learning with the improvement in CIDEr score as the reward (along with a reward as to whether the generated edge is in the ground truth tree or not).