Multi-modal Dependency Tree for Video Captioning

1 Introduction . Video captioning has received extensive attention from researchers in both computer vision and natural language processing . It is a challenging task since it requires not only understanding the visual content but also generating fluent and relevant sentences . With the success of deep neural networks in natural language processing , many existing video captioning methods [ 1 , 2 ] use recurrent neural network ( RNN ) to generate sentences , which processes sequences by updating hidden states . Several recent studies [ 3–6 ] apply Transformer [ 7 ] , which relates the words in the sequences using a self-attention mechanism , to video captioning . All these methods treat each sentence as a word sequence and generate words in a predefined left-to-right order by capturing the relatively close contextual relationship between words in the sentence . In this paper , we investigate a graph structured model for video captioning to explicitly model the hierarchical structure in the sentence and capture the long-range dependency between words . With this in mind , we propose to generate a sentence by first constructing a multi-modal dependency tree in a top-down and depth-first order , and then traversing the tree in a recursive manner , as shown in Figure 1 . Each node of the tree is represented by a multi-modal embedding representation that integrates the information from both visual and textual modalities . During the tree construction , each ∗Corresponding author . 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . newly generated node receives the multi-modal embeddings from its parent and sibling nodes , and these multi-modal embeddings are used to predict the attention weights of the input features and the word . The attended visual feature and the predicted word are fused to generate the multi-modal embedding of the new node . Different from existing sequence models that tend to focus on the dependency between each word and its close preceding words , our tree model sufficiently captures the global dependency structure in the sentence to further improve the fluency of the generated sentences . In contrast to the unimodal dependency tree widely used in dependency parsing for language analysis , our multi-modal dependency tree effectively integrates both the visual and linguistic information , improving the relevance of the output sentence to the visual input . To effectively optimize the captioning model , we propose a tree-structured reinforcement learning algorithm and a novel node-level reward tailor-made for the tree construction process . By evaluating the consistency between the parent-child node pairs in the constructed tree and those in the ground truth trees , the tree node-level reward enables the model to capture the topological structure of the ground-truth dependency trees . Compared to the sequence-level rewards in existing methods , our node-level reward recognizes the contribution of each node to the multi-modal dependency tree and avoids the reward ambiguity problem [ 8 ] . To evaluate the effectiveness of the proposed method , we conduct experiments on two difficult video captioning datasets , the ActivityNet Captions dataset and Charades Captions dataset . We also perform experiments on two most widely-used datasets , namely the MSVD dataset and MSR-VTT dataset . Compared to MSVD and MSR-VTT , the sentences in the ActivityNet Captions dataset and Charades Captions dataset are typically longer and describe more complex activities in the videos . The experiments on these datasets demonstrate that our method not only generates long and complex sentences with high fluency and relevancy , but is also effective for relatively simple sentences . The main contributions of this paper are as follows : • We propose a multi-modal dependency tree construction method for video captioning . With the help of tree topology , our model generates more fluent and relevant sentences by effectively capturing the syntactic and semantic dependencies in natural language , especially when generating long and complex sentences . • We develop a novel tree-structured reinforcement learning algorithm that optimizes the captioning model using a node-level reward , which facilitates learning the topology of the dependency trees and alleviates the reward ambiguity problem . 2 Related Work . 2.1 Video Captioning . Thanks to the recent advances in computer vision and natural language processing , many video captioning methods based on the encoder-decoder framework have been proposed . The pioneers of video captioning methods [ 9 ] employ a convolutional neural network ( CNN ) to extract visual features and a recurrent neural network ( RNN ) to generate the words in the sentences in a sequential manner . To further improve the performance of video captioning , some methods [ 10 , 11 ] apply the attention mechanism in video captioning , which enables the model to focus on different temporal segments when generating the sentences . Inspired by the advances in natural language processing , some methods [ 12 , 13 ] propose to learn better syntax representations with the help of part-of-speech tags . In order to provide richer semantic information to video captioning models , Rahman et al . [ 14 ] attempt to utilize the audio information . Motivated by the advances in neural machine translation , several recent methods [ 3 , 4 , 6 ] investigate the application of Transformer [ 7 ] to video captioning , which uses multi-head self-attention mechanism to yield more expressive representations of the input . Some methods [ 15 , 16 ] focus on optimizing video captioning models using reinforcement learning , where the rewards are calculated based on the n-gram statistics . Instead of directly generating a word sequence , our method generates a sentence by first constructing a dependency tree and then traversing the tree . Compared to the sequence generation models , our method explicitly models the grammatical structure with tree topology and better preserves the semantic relationship by incorporating both visual and linguistic information into the dependency tree . 2.2 Tree-structured Language Generation . In recent years , tree-structured language generation methods are proposed for various natural language processing tasks , including program generation [ 17 ] , generating math equation [ 18 ] and machine translation [ 19 ] . Alvarez et al . [ 17 ] generate the tree nodes in depth-first search order using doublyrecurrent neural network , which combines the hidden states from the parent node and the last sibling node when predicting the label for a new node . A dependency tree generation method is proposed by [ 19 ] to generate target sentences in machine translation . Instead of directly constructing the tree structure , this method generates canonizalized ternary trees where each node has a fixed number of child nodes . Liu et al . [ 18 ] propose to generate abstract syntax trees of math expressions to solve math problems described in natural language . The work closest to our method is [ 20 ] that generates image captions by constructing canonicalized dependency trees . Different from this method that generates trees with only linguistic information , our method constructs multi-modal dependency trees that contain both visual and linguistic information to capture richer contextual information . 3 Our Method . 3.1 Overview . In this section , we introduce the proposed multi-modal dependency tree generation framework . The training data is formulated as D = { ( xj , Tj ) |j } , where xj denotes the j-th video , and Tj denotes the dependency tree of the corresponding sentence . Our model follows the encoder-decoder paradigm , where both the encoder and the decoder are stacks of N self-attention layers . Given an input video , the encoder of our model takes a set of d-dimensional visual features F = { f1 , f2 , ... , fM } as input , where fi ∈ Rd . The decoder firstly generates a multi-modal dependency tree T = { U , E } , where U and E denote the node set and edge set , respectively . Each node ui ∈ U corresponds to a word in the sentence , and a directed edge 〈uj , ui〉 ∈ E indicates that the node ui is dependent on its parent node uj . Then we use a recursive algorithm to traverse the tree T and recover the order of the words in the output sentence y = { w1 , w2 , ... , wL } , where L denotes the length of the sentence and wi denotes the i-th word . The overview of our framework is illustrated in Figure 2 . 3.2 Tree Generation Process . Our tree generation model follows a step-by-step generation process . In the i-th step , a node ui is added to the tree , and the corresponding word wi is predicted . Formally , given a video x , the probability of generating a tree T is given by p ( T |x ) = ∏ ui∈T p ( ui|x , Ti−1 ) , ( 1 ) where Ti−1 denotes the sub-tree that has been generated in the ( i− 1 ) -th step . Since the structure of the dependency tree is unknown in the tree generation process , our model also predicts the topological information related to each node . For each node ui , we define three topological labels : tsi ∈ { 0 , 1 } , tci ∈ { 0 , 1 } and tei ∈ { 0 , 1 } . tsi indicates whether node ui has another sibling node . If tsi = 1 , another sibling node of ui will be added to the tree in the following steps . tci decides whether ui have child nodes , and least one child node of ui will be added when t c i = 1. tei = 0 indicates that the node ui is the left child of its parent node uj , namely the word wi appears on the left of wj in the output sentence , and tei = 1 indicates that ui is the right child of uj . 3.3 Encoder . Given a set of visual features F = { f1 , f2 , ... , fM } , the encoder uses multiple self-attention operations to capture the relationships between video frames represented by the visual features . We first recall the scaled dot-product attention used by the self-attention layers , which operates on three sets of vectors Q , K and V : Attention ( Q , K , V ) = softmax ( QK > √ d ) V , ( 2 ) where Q ∈ Rnq×d is a matrix consisting of nq query vectors , K ∈ Rnk×d and V ∈ Rnk×d are matrices containing nk key vectors and nk value vectors , respectively . The vectors in Q , K and V are with the same dimensionality d. In addition to the self-attention layer , each encoder layer contains a feed-forward network ( FFN ) consisting of two fully connected layers , which can be formulated as FFN ( X ) = ReLU ( XW1 + b1 ) W2 + b2 , ( 3 ) where W1 , W2 ∈ Rd×d and b1 , b2 ∈ Rd are learnable parameters . In each layer of the encoder , the queries , keys and values are obtained by linearly mapping the input features , and the output of the n-th encoding layer Xn is given by X′n = Attention ( W q nXn−1 , W k nXn−1 , W v nXn−1 ) , Xn = FFN ( X ′ n ) , ( 4 ) where W qn ∈ Rd×d , W kn ∈ Rd×d and W vn ∈ Rd×d are learnable parameters , and Xn−1 is the output of the previous encoding layer . FFN represents the feed-forward network defined in Eq . 3 . The visual features V are used as the input of the first encoding layer .

Summary: The paper addresses video captioning by generating captions in a tree structure. It stepwise constructs the multi-modal dependency tree and then traverses the constructed tree to obtain the captions. The supervision is performed on generated captions as well as the semantic consistency reward of the generated sub-trees. Experiments are conducted on several video captioning datasets.