Multi-modal Dependency Tree for Video Captioning
Authors: Wentian Zhao
Xinxiao Wu
Jiebo Luo
Venue: NeurIPS 2021 Poster (2021)
Keywords: video captioning
tree-structured decoding

Abstract: Generating fluent and relevant language to describe visual content is critical for the video captioning task. Many existing methods generate captions using sequence models that predict words in a left-to-right order. In this paper, we investigate a graph-structured model for caption generation by explicitly modeling the hierarchical structure in the sentences to further improve the fluency and relevance of sentences. To this end, we propose a novel video captioning method that generates a sentence by first constructing a multi-modal dependency tree and then traversing the constructed tree, where the syntactic structure and semantic relationship in the sentence are represented by the tree topology. To take full advantage of the information from both vision and language, both the visual and textual representation features are encoded into each tree node. Different from existing dependency parsing methods that generate uni-modal dependency trees for language understanding, our method construct s multi-modal dependency trees for language generation of images and videos. We also propose a tree-structured reinforcement learning algorithm to effectively optimize the captioning model where a novel reward is designed by evaluating the semantic consistency between the generated sub-tree and the ground-truth tree. Extensive experiments on several video captioning datasets demonstrate the effectiveness of the proposed method. 

Multi-modal Dependency Tree for Video Captioning
Wentian Zhao, Xinxiao Wu'
Jiebo Luo Department of Computer Science University of Rochester Rochester, NY 14627 USA jluoêes www.ddssrrcheeeer.oo edu
Beijing Laboratory of Intelligent Information Technology School of Computer Science
Beijing Institute of Technology
No. 5, Zhongguancun South Street, Beijing, China
{wentian taaetiinnnahoooo zhac, wuxinxiao]@bi: edu.
Abstract
Generating fluent and relevant language describe visual content is critical for the video captioning task. Many existing methods generate captions using sequence models that predict words in a left-to-right order. In this paper, we investigate a graph structured model by explicitly modeling the hierarchical structure in the sentences to further improve the fluency and relevance of the generated captions. To this end, we propose novel video captioning method that generates sentence by first constructing multi-mode modal dependency tree and then traversing the constructed tree, where the syntactic structure and semantic relationship in the sentence are represented by the tree topology. To take full advantage of the information from both vision and language, both the visual and textual representation features are encoded into each tree node. Different from existing dependency parsing methods method
that generate uni-modal dependency trees for language understanding, constructs multi-modal dependency trees for language generation of videos. We also propose
structurd reinforcement learning algorithm effectively optimize the captioning model, where a novel reward designed by evaluating the semantic consistency between the generated sub-trees and the ground-truth tree. Extensive experiments on several video captioning datasets demonstrate the effectiveness of the proposed method.
Introduction
Video captioning has received extensive attention from researchers in both computer vision and natural language processing. It is challenging task since it requires not only understanding the visual content but also generating fluent and relevant sentences. With the success of deep neural networks in natural language processing, many existing video captioning methods 21 use recurrent neural network (RNN) generate sentences, which processes sequences by updating hidden states. Several recent studies (3-6] apply Transformer (7), which relates the words in the sequences using a self-attention mechanism, to video captioning. All these methods treat each sentence as a word sequence and generate words in predefined left-to-right order by capturing the relatively close contextual relationship between words in the sentence.
In this paper, we investigate graph structured model for video captioning to explicitly model the hierarchical structure the sentence and capture the long-range dependency between words. With this in mind, we propose to generate sentence by first constructing multi-mode dependency tree top-down and depth-first order, and then traversing the tree in a recursive manner, shown in Figure] I Each node of the tree is represented by multi-modal embedding representation that integrates the information from both visual and textual modalities. During the tree construction, each
"Corresponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

newly generated node receives the multi- embeddings from its parent and sibling nodes, and these multi-modal embeddings are used predict the attention weights the input features and the word. The attended visual feature and the predicted word are fused to generate the multi-modal embedding of the new node.
throwing
T4
Tin
Chrowing
Ovewling
throwing
person
Showe
holdling
Eresion
(whie) person)
another
person
shoes
coffee
person throwing shoes while another person holding a cup of coffee
Vidco fromes
Figure I:
The sentence generation process of the proposed method. T; denotes the dependency tree generated in the i-th step, and the colored box denotes the multi-modal embedding of each node.
Different from existing sequence models that tend to focus on the dependency between each word and its close preceding words, tree model sufficiently captures the global dependency structure in the sentence to further improve the fluency of the generated sentences. In contrast to the uni- modal dependency tree widely used in dependency parsing for language analysis, our multi-modal dependency tree effectively integrates both the visual and linguistic information, improving the relevance of the output sentence to the visual input.
To effectively optimize the captioning model, we propose a tree-structured reinforcement learning algorithm and novel node-level reward tailor-made for the tree construction process. By evaluating the consistency between the parent-chile node pairs the constructed tree and those in the ground truth trees, the node-level reward enables the model to capture the topological structure of the ground-truth dependency trees. Compared to the sequence-level rewards in existing methods, our node-level reward recognizes the contribution of each node the multi-modal dependency tree and avoids the reward ambiguity problem (8). To evaluate the effectiveness of the proposed method, we conduct experiments two difficult video captioning datasets, the ActivityNet Captions dataset and Charades Captions dataset. We also perform experiments two most widely-used datasets, namely the MSVD dataset and MSR-VTI -TT dataset. Compared to MSVD and MSR- VTT, the sentences in the ActivityNet Captions dataset and Charades Captions dataset typically longer and describe more complex activities in the videos. The experiments on these datasets demonstrate that our method not only generates long and complex sentences with high fluency and relevancy, but is also effective for relatively simple sentences.
The main contributions of this paper are as follows:
We propose a multi-modal dependency tree construction method for video captioning. With the help of tree topology, our model generates more fluent and relevant sentences by effectively capturing the syntactic and semantic dependencies in natural language, especially when generating long and complex sentences.
We develop novel tree-structured reinforcement learning algorithm that optimizes the captioning model using a node-level reward, which facilitates learning the topology of the dependency trees and alleviates the reward ambiguity problem.
2 Related Work
2.1 Video Captioning
Thanks the recent advances in computer vision and natural language processing, many video captioning methods based on the encoder-decoder framework have been proposed. The pioneers of video captioning methods [9] employ convolutional neural network (CNN) extract visual features and recurrent neural network (RNN) to generate the words in the sentences in a sequential manner. further improve the performance video captioning, some methods 100 [ apply attention mechanism in video captioning, which enables the model to focus on different temporal segments when generating the sentences. Inspired by the advances natural language processing, some methods [22 [3 propose to leam better syntax representations with the help of part-of-

---

agent
Tj
throwing
action
person
holding
Decoder
video frames
dependency tree
reward
Encoder Layer
x N
Layer
r(w,
Iree-structured decoder
encoder
CNN
state
visual feature
environment
Figure 2: Overview of the proposed multi-modal dependency tree generation framework and the tree- structured reinforcement learning training algorithm. T; and w, denote the multi-modal dependency tree and the predicted word in the i-th step, respectively. The green-colored box in the middle stands for the environment, and the state of the environment is comprised of the perviously generated tree and the visual features. The bluc-colored box the right denotes the agent, namely the captioning model. During training, the agent executes an action by predicting a word w; in the i-th step, and receives reward r(w;) from the environment.
tags. In order provide richer semantic information to video captioning models, Rahman et al. attempt to utilize the audio information.
14
Motivated by the advances in neural machine translation, several recent methods 3 4 investi- gate the application of Transformer (7) to video captioning, which uses multi-head self attentinn mechanism to yield more expressive representations of the input. Some methods [[5 [6] focus on optimizing video captioning models using reinforcement learning, where the rewards are calculated based on the n-gram statistics.
Instead of directly generating a word sequence, our method generates a sentence by first constructing a dependency tree and then traversing the tree. Compared to the sequence generation models, our method explicitly models the grammatical structure with tree topology and better preserves the semantic relationship by incorporating both visual and linguistic information into the dependency tree.
2.2 Tree-structured Language Generation
In recent years, tree-structured language generation methods proposed for various natural language processing tasks, including program generation (17). generating math equation [18] and machine translation [19] Alvarez al. [17] generate the nodes in depth-first search order using doubly- recurrent neural network, which combines the hidden states from the parent node and the last sibling node when predicting the label for a new node. A dependency tree generation method proposed by [19] to generate target sentences in machine translation. Instead of directly constructing the tree structure, this method generates canonizalized ternary trees where each node has fixed number of child nodes. Liu et al. [18) propose to generate abstract syntax trees of math expressions to solve math problems described in natural language.
The work closest to our method is (0 that generates image captions by constructing canonicalized dependency trees. Different from this method that generates trees with only linguistic information, method constructs multi-modal dependency trees that contain both visual and linguistic information capture richer contextual information.
Our Method
3.1
Overview
In this section, we introduce the proposed multi-modal dependency tree generation framework. The training data is formulated as ((a,, T;)ls) where æy denotes j-th video, and Tj denotes the dependency tree of the corresponding sentence. Our model follows the encoder-decoder paradigm, where both the encoder and the decoder are stacks of N self-attention layers. Given an input video, the encoder of our model takes set of d-dimensional visual features F {fi, f2, ... fM input,

---

where f. E Rd. The decoder firstly generates a multi-modaa dependency tree T {U, E), where U and E denote the node set and edge set, respectively. Each node u, E U corresponds sentence, and a directed edge (uj. uz) E indicates that the node is dependent on its parent node Uj. Then we use a recursive algorithm to traverse the tree 7 and recover the order of the words the [W1, 1/2.... .... WL), where I. denotes the length of the sentence and uy denotes the i-th word. The overview of our framework is illustrated in Figure
word the
output sentence
3.2 Tree Generation Process
Our tree generation model follows a step-by-step generation process. In the i-th step, node added to the tree, and the corresponding word w/ predicted. Formally, given a video æ, the probability of generating a tree 7 is given by
p [T = II
p(ug [r, T:-1),
Mr ET
where T denotes the sub-tree that has been generated in the (i - 1)-th step.
Since the structure of the dependency tree unknown in the tree generation process, our model also predicts the topological information related to each node. For each node u,, we define three topological labels: t E {0, 1). & E {0, }} and t {0, 13. t indicates whether node u, has another sibling node. t 1, another sibling node of u, will be added to the tree in the following steps. tf decides whether u, have child nodes, and least one child node of uy will be added when t = 1. 0 indicates that the node u, is the left child of its parent node uj, namely the word w, appears on the left of w in the output sentence, and tf = 1 indicates that uN is the right child of uy.
3.3 Encoder
Given set of visual features F (2,.f2...... fM the encoder uses multiple self-attention opera- tions to capture the relationships between video frames represented by the visual features. We first recall the scaled dot-produce attention used by the self-attention layers, which operates on three sets of vectors Q, K and V:
QK T V
Attention K, V) = softmax
(2)
Vd
where Q Rd is matrix consisting of ng query vectors, K R and V R matrices containing n& key vectors and nıg value vectors, respectively. The vectors in Q. K and V with the same dimensionality d. In addition to the self-attention layer, each encoder layer contains a feed-for fordarr network (FFN) consisting of two fully connected layers, which can be formulated as
FFN(X) = ReLU(XW; + b,) + b2,
(3)
where W,, W2 E Raxd and b,, b2 E R2d are learnable parameters.
In each layer of the encoder, the queries, keys and values are obtained by linearly mapping the input features, and the output of the n-th encoding layer X,, is given by
X Autention W XY-1.,1111 W X, X W X X, FFN(X.),
(4)
where W E R dxd W E ddxd and W E d are learnable parameters, and X,-1 is the
output of the previous encoding layer. FFN represents the feed-forward network defined in Eq. The visual features V are used as the input of the first encoding layer.
3.4 Tree Structured Decoder
The decoder is conditioned on both the output of the encoder and the previously generated sub-tree. In the i-th step, the decoder takes the encoded visual feature XN, the words in the sub-tree T;-1 and the multi-modal embeddings of the nodes in T as input, and predicts the word w, together with the topological labels (ç, 15 and ( The tree generation process terminates if the predicted word a special token (eos) indicating the end of the sentence. Since the order of words may affect the semantics of the sentence, in each step, we traverse the tree T using the algorithm that will be

---

described in Section 3.5 and obtain node sequence uk, Uky .... Wk--1 before feeding the words that have been generated so far the decoder, where k; indicates the index of the nodes.
Let - ( denote the embeddings of the previously generated (i - 1) words. The decoder acquires the multi-modal embedding of the current node a, by firstly attending to the visual features and the embeddings of the previously generated words using self-attention layers, and then fusing the output of the self-attention. The n-th decoding layer formulated as:
Y, FY, Y,Wardd
(5)
= FFN Wattenttonnnn--.. XN, XN) + Altention(Y. (Yn-1, E, E;-
YN E IR' represents the output the last decoding layer and is used as the multi-modal embedding of the current node Uy i.c. a, Yn in the i-th step. The probability distribution of the word corresponding to u, is calculated using the multi-modal embedding:
Pw; Wworda;
(6)
where W word E RID xd learnable parameter, and |D] denotes the vocabulary size. After the word w. is determined by sampling strategy (c.g. greedy sampling or beam search), the topological labels of the current node are predicted by
p(t 1) sigmoid(w: [a;; Em..),
p(t 1) sigmoid w [a;; Em..)), p(t = 1) sigmoid w [a;; E...))
where 2,,, we ue E 2d2d are learnable parameters, Ew, E R.d denotes the embedding of word ws respectively, The operator [:] denotes the vector concatenation.
3.5 Tree Traversal Algorithm
We restore the word order and convert the tree human-readable word sequence by traversing the tree. Given generated dependency tree T, our tree traversal algorithm recursively processes each node in the tree, and restores the word order of the phrase represented by the node and its sub-trees. Since the tree is generated following the depth-first search, the generation order of the child nodes of each node is the same as the order of the corresponding words the sentence. Thus, for an arbitary node u; in the dependency tree T, we sequentially traverse its left child nodes, the node and its right child nodes.
3.6
Model Training
3.6.1
Pre-Training Stage
The training process of the multi-modal dependency generation model involves a pre-training stage and fine-tuning stage. Since the model predicts both the word and the topology labels, we use word loss Ly and a topological loss L, in the pre-training stage. Formally, the word loss is defined as
[T]
C.. = Σ logp( (ep((u u. (.:.TTTT..... TT,,,
where p(w; ((w:[Ti-1, x) denotes the probability of predicting word w; given the previously generated tree T: and the visual input x, and [T| denotes the number of nodes in the tree T. The topological loss is defined as
L. = Loce(t,, B2), + Lace (tS, EG) + Lbce (t),
Lace (t, f)
Σ f log(p(t)) + (1-f) log p(t)).
where iz. E and ? are ground-truth topological labels and Lace denotes the binary cross-entropy loss. The overall loss function in the pre-training stage is formulated by
Epretrain = -. + L,
(10)
5

---

3.6.2 Fine-tuning Stage
To effectively optimize the proposed model, we formulate the multi-modal dependency tree generation as a decision-making process and fine-tune the model with reinforcement learning. Specifically, the captioning model is regarded as the agent, the input video I and the generated dependency tree T are regarded as the state of the environment, and the prediction of the word w; is regarded as the action. Instead of using the same sequence-leve reward for all the actions, we design novel node-level reward that estimates the contribution of cach individual action. The reward r(ws) for the word W/ is given by
r(w) T1 (CIDEr(() - CIDE + (od up) E E),,
(11)
where CIDEr(T) denotes the CIDEr score of the word sequence represented by the nodes in T, up denotes the parent node of 04 and ((us, Up) E F) indicates whether the edge (u;, Up) is present in the ground-truth dependency tree. Xι and X2 are tunable hyper parameters. The discounted future reward for the agent is calculated by
80
R(w:)= = r(w:))
Fr(w+*))
(12)
where 7 denotes the discount factor. Let O denote the parameters of the captioning model, and then loss function in the fine-tuning stage is given by
L -Bmisa(0) R((w)),
(13)
where π (0) denotes the policy defined by
4
Experiments
4.1 Datasets
ActivityNet Captions is dense video captioning dataset that contains 10,030 training videos, 4,926
validation videos and 5,044 test videos. Each video is annotated with an average of 3.65 sentences together with temporal annotation and the average length of the sentences is 13.7 words. We conduct the experiments on ActivityNet Captions with ground-truth proposals and the results are reported on the validation set.
Charades Captions is composed of 9,223 videos indoor activities. Each video is annotated with 2-5 sentences, and the average length of the sentences is 24. words. Following [15), we split this dataset into 6,963 training videos, 500 validation videos and 1,760 test videos.
MSVD consists of 1,970 video clips collected from YouTube, each of which annotated with about 41 sentences and the average length of the sentences is 7.10 words. We follow 9 to split the dataset into 1,200 training videos, 100 validation videos and 670 testing videos.
MSR-VTT is a dataset collected for open-domain video captioning, which contains 10,000 video clips in total. Each video in the MSR-VTT dataset contains 20 human annotated captions. We use the splits provided by [15), where the training split, validatation split and test split contain 6,513 videos. 497 videos and 2,990 videos, respectively. The average length of the sentences in MSR-V VTT is 9.28 words.
4.2 Evaluation Metrics
We report several widely-used automatic evaluation metrics for video captioning, including Bleu-n 21], METEOR (22), ROUGE-L (23] and CIDEr (24). Nevertheless, these n-gram based evaluation metrics have some limitations, e.g., they penalize the phrases that are semantically correct but differ from ground-truth in the specific word choices
(25). To evaluate the quality the sentences more effectively, we also report Improved BERTScore metric proposed by (26). Compared with the n-gram based metrics, Improved BERTScore is computed using pre-trained BERT embeddings and has better correlation with human experts.

---

4.3 Implementation Details
In the training process, in order convert the sentences dependency trees, we use the dependency parser in the SpaCy toolkit [27]. For the ActivityNet Captions dataset, we extract the visual features of the 16-frame video segments using the pre-trained C3D network. To compare our method with 3. we also extract the frame features using ResNet-200 28 and extract the optical flow features using BN- IN--nncption [29]. For the Charades Captions dataset, we sample the video frames at fps and extract the feature of each frame using the pre-trained ResNet- 152 network ( following [5]] We use the Adam optimizer (30) in both pre-training and fine-tuning stages. The initial learning rate
set to 5 x 10-3 and decays 0.8 times for every 3 epochs. The experiments are conducted using one NVIDIA RTX 2080Ti GPU.
Table 1: Comparison with state-of-the-art methods and the results of ablation studies on the Ac- tivityNet Captions dataset using ground-truth proposals. B@n, M, C and BS are abbreviations for Bleu-n, METEOR, CIDEr and Improved BERTScore, respectively. Note that the last two rows are compared for different video features.
Feature
Model
B@1
B@2
B@3
B@4
M
BS
DCE
18.13
8.43
4.09
1.60
8.88
25.12
DVC
19.57
9.90
4.55
1.62
10.33
25.24
SDVC 33
28.02
12.05
4,41
1.28
13.07
43.48
WLT [4]
15.23
6.58
3.04
1.46
7.23
25.36
w/o tree
27.01
11.12
4.10
1.47
12,40
42.30
35.57
C3D
w/o
20.31
8.54
3.90
11.23
38.90
34.30
w/o tree, RL
20.56
9.45
4.48
1.26
9.97
37.80
30.20
w/o visual embedding
25.30
10.90
3.74
1.36
11.09
35.10
32.69
sequence reward
27.35
11.18
4.15
1.62
11.70
41.39
35.23
Ours
28.53
12.12
4.49
1.75
13.24
44.13
36.35
ResNet-220+
Masked
23.93
12.16
5.76
2.71
11.16
47.71
BNInception
Ours
24.10
12.26
5.57
2.72
11.20
47.80
Table 2: Experiment results the results of ablation studies on the Charades Captions dataset. the abbreviation for ROUGE-L.
Model
B@
@@2
B B@3
B B@4
M
R
BS
HRL [TS]
64.40
44.30
29.40
18.80
19.50
41.40
23.20
tree
62.10
43.50
28.49
17.68
18.65
40.20
23.50
34.13
w/o RL
58.40
39.30
25.31
17.08
18.19
38.97
22.53
33.71
w/o tree, RI.
57.63
38.32
24.83
16.50
18.21
38.80
21.45
30.41
w/o visual embedding
60.39
40.30
26.70
16.90
18.19
39.45
21.87
32.50
sequence reward
64.00
44.90
29.70
18.50
19.20
40.32
23.41
34.21
Ours
65.30
45.60
29.80
18.87
19.38
40.50
24.21
34.57
4.4 Comparison with State-of-the-Art Methods
The results on ActivityNet Captions shown in Table We compare method with DCE B1].
DVC (32), SDVC 33 and Masked (which uses different video features) 3
All these methods implemented
generate sentences using sequence models. The captioning models in 32
33
by RNN and the model proposed by 3 is based on Transformer.
All the results of these compared methods directly quoted from their original papers. Our method achieves the best performance on the ActivityNet Captions dataset, demonstrating that by explicitly modeling the hierarchical structure of the sentences, proposed method can generate long and complex sentences with higher fluency and relevance.
We compare our method with HRL [5] on the Charades Captions dataset. This method involves manager network that designs sub-goals and worker network that fulfills each sub-goal by predicting words, both of which are trained end-to-end using hierarchical reinforcement learning. The results are shown in Table 2 We observe that our method outperforms HRI., which evaluates the effectiveness of our method in generating complex video descriptions.
7

---

The results on the MSVD dataset and the MSR-VVIR VTT dataset are shown in Table 3 We compare our method with HRL [1]]. MARN 34, POS-CG [3], Joint [2] and RMN [35 .ll these methods generate sentences using sequence models implemented by RNN. All the results of these compared methods are directly quoted from their original papers. For fair comparison, we use the same visual features as these methods. From the results on MSVD and MSR- VTT, we observe that our method achieves comparable performance with the state-of-the-rr methods when using different visual features, which demonstrates that our method is also effective for generating relatively simple sentences. Note that such simple cases our proposed method is not expected make much difference perform better.
Table 3: Comparison with state-of of-thh-art methods on MSVD and MSR-VT VTT using different visual features. 13D(RGB) and I3D(OF) indicate using I3D network 366 to extract the features of raw video frames and optical flow, respectively. The mark denotes using additional object features.
MSVD
MSR- VTT
Feature
Model
B@4
R
C
@@4
R
HRL 15
41.3
61.7
28.7
48.0
ResNet152
Joint 12
52.1
69.8
33.7
80.6
41.4
62.0
28.9
48.1
Ours
52.2
70.0
34.0
81.2
41.6
62.0
29.1
48.4
ResNet101+
MARN 34
48.4
71.9
35.1
92.2
40.4
60.7
28.1
47.1
ResNext101
Ours
49.0
72.2
35.3
92.5
40.2
61.1
28.2
47.3
InceptionResnet2++
POS CSS-CGIB
52.5
71.3
34.1
92.0
42.0
61.6
28.2
48.7
[3D(FF)
Ours
51.7
71.6
34.9
92.4
41.8
61.4
28.3
49.0
RMN (H+1)*
54.6
73.4
36.5
94.4
42.5
61.6
28.4
49.6
IeceptionResNet+
RMN
R5
52.5
72.7
36.1
92.8
42.5
61.6
28.4
49.6
13D(RGB)
Ours
51.8
72.8
36.7
92.5
42.5
61.8
29.1
49.8
ORG-TRL* 33
54.3
73.9
36.4
95.2
43.6
62.1
28.8
50.9
InceptionResNet+
PMI-CAP
38
54.7
36.4
95.2
42.2
28.8
49.5
C3D
Ours
52.0
72.8
36.5
92.6
42.7
61.9
29.4
50.1
4.5
Ablation Studies
To analyze the effect of different components, we conduct ablation studies on the ActivityNet Captions dataset and the Charades Captions dataset. The following variants of our method are evaluated:
w/o tree: To evaluate the advantage the tree structure over the sequence structure, we replace the dependency trees with chain-structured trees, where the root node corresponds to the leftmost word in the sentence. For each non-leaf node, its only child node corresponds to the word its right. This variant of model is actually sequence model that generates the words the sentence left-to-right order.
w/o RL: To validate the effectiveness of reinforcement learning, we only optimize the model with the cross-entropy loss in pre- vre-trainin stage.
w/o tree, RL: evaluate the effect of using tree structure and reinforcement learning simultaneously, we optimize sequence model with only cross-entropy loss.
w/o visual embedding: To evaluate the effectiveness of the visual embeddings of the tree nodes, we replace the attended visual feature vi defined in Eq. with all-zero matrix, and the node representations contains only linguistic information.
sequence reward: To verify the contribution the tree node-level reward, we replace with the sequence-level reward, i.e. the reward of each node equals to the CIDEr score of the entire sentence.
The results of ablation studies on the ActivityNet Captions and Charades Captions datasets are shown in the middle part of Table and Table respectively. From these results, we make the following observations. First, by replacing the dependency tree with the chain-structured tree, the performance of our model degrades terms of all the metrics, indicating that the hierarchical structure of sentences is beneficial for generating sentences with high quality. Second, our full model outperforms "w/o visual embedding". validating the superiority by taking full advantage of both visual and linguistic information. Third, when the fine-tuning stage
multi-modal representation of the tree node

---

with reinforcement learning is removed, our model performs worse on all the metrics, which validates the contribution of our reinforcement learning algorithm. Our full model also outperforms "sequence reward", which verifies that the node-level reward effectively guides the training process.
4.6 Evaluation on Simple and Complex Subsets
To further evaluate the effect of the tree structure on generating simple sentences and complex sentences, we split the videos in the test sets of MSR-VTT and Charades Captions into a simple subset and complex subset according to the average length the ground-truth sentences. The data distributions of the two subsets and the evaluation results are shown in Table From these results, we observe that by utilizing the tree structure, the performance of our model significantly increases on the complex subset, which demonstrates the proposed model's capability in terms of generating complex sentences.
Table 4: Data distribution and evaluation results on the simple subset and complex subset of Charades
Captions and MSR-VTT.
Sentence length
Video number
w/o
Ours R
Subset
B @
R
B @
Charades (simple)
20
36.7
28.4
17.9
40.9
25.8
Charades (complex)
20
1210
17.1
33.3
20.8
18.8
39.8
24.4
MSR-VTI -TT (simple)
S 10
2005
38.3
62.3
54.8
45.8
66.2
59.8
MSR-VTT (complex)
10
985
31.0
48.6
25.3
35.1
50.8
27.7
4.7 Results of Using Different Evaluation Metric in the Reward
To assess the impact of the evaluation metric used in the reward the performance of the captioning model, we conduct additional experiments that replace the CIDEr score in the reward using other metrics (i.e. Bleu-4 and ROUGE-L) the Charades Captions dataset. From the results in Table we observe that when using Bleu-4 (or ROUGE-L) in the reward, the performance in terms of Bleu-4 (or ROUGE-L) increases when compared with using CIDEr in the reward. However, using CIDEr score in the reward leads to the best overall performance. Thus, we use CIDEr score in the reward in other experiments.
Table 5: The results of using different evaluation metrics the reward on the Charades Captions dataset.
Method
Reward
@@4
M
R
C
Ours
Bleu-4
19.02
18.50
40.53
22.95
Ours
ROUGE-L
17.06
18.32
41.79
22.32
Ours
CIDEr
18.87
19.38
40.50
24.21
4.8
Qualitative Results
We show some examples of the generated sentences on Charades Captions and MSR-VTT in Figure As shown in the figure, by utilizing the tree structure, our method generates sentences that not only describe the objects and the human actions more accurately but also possess correct grammatical structures.
3
4.9 Human Evaluation
To intuitively evaluate the effect of the tree structure on the quality of the generated sentences, we conduct human evaluation on the test splits of ActivityNet Captions and MSR-VTI VTT. We randomly choose 200 videos from the test splits of ActivityNet Captions and MSR-VTI VTT, respectively. The workers are given original video as well as two sentences generated by "Ours" and "w/o tree", and are asked to select the sentence that has better relevance to the video and the sentence with better fluency. For the two methods "Ours" and "w/o tree", we report the percentage of sentences that have better relevance and the percentage of sentences that have better fluency. From the results in Table

---

A person is standing in
trees A person is holding a book and phone. The person puls the book on the floor and begins tidying up the room. w/o treet A person is sitting on the floor reading book while reading a book.
gt: person walks to mirror and begins wiping it with towel. The person then losses the towel on couch and adjusts table.
front of a mirror holding
lowel. The person puts the
towel on a shelf and leaves.
Ours A person is sitting the
gl: She is sitting laundry room and reading book, and now she lying on floor while reading, and put the sheet under her head.
floor reading a book, The
person puts the book on shelf
picks up a book and leaves.
Ours A man is talking about
troer tree: There is a man talking about something.
gl: man talks about the henefits of defensive satellites.
space.
Ours A person is playing a golf game.
w/o tree: A man is in green shirt playing baseball game.
gt: golf player is trying to hit ball into the pit.
Figure 3: Qualitative results on Charades Captions (the first two rows) and MSR-VTT (the last two rows). "gt", "Ours" and "w/o tree" denote the ground-truth, the sentence generated by our method and the sentence generated by the variant "w/o tree" of our model, respectively.
we observe that the tree structure remarkably improves both the relevance and the fluency of the generated sentences.
Table 6: The results of human evaluation.
Relevance
Fluency
Dataset
w/o tree
Ours
w/o tree
Ours
ActivityNet MSR-VTT
44.63%
55.36%
48.02%
51.97%
45.65%
54.34%
46.20%
53.80%
5 Conclusions
We have presented a modti dependency tree generation method for video captioning. Our model can explicitly model the hierarchical structure of natural language using tree topology to better capture both the syntactic structure and semantic relationship in sentences. Both visual and linguistic information are incorporated into the node embeddings further improve the relevance of the generated sentence to the video. Moreover, we have developed a tree-structured reinforcement learning algorithm and designed novel tree node-level reward to effectively optimize the captioning model. Our multi-modal dependency tree is capable of generating complex sentences with high fluency and relevance for videos. Extensive experiment on multiple challenging video captioning datasets have demonstrated the effectiveness of our method.
Funding Transparency Statement
This work was supported in part by the Natural Science Foundation of China (NSFC) under Grant No 62072041.
References
[1] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show and tell: A neural image caption generator," in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. 2015.
[2] P. Anderson, X. He, C. Buchler, D. Teney, M. Johnson, S. Gould, and I.. Zhang, "Bottom-up and top-down attention for image captioning and visual question answering." in Proc. IEEE Conf. Compur. Vis. Pattern Recognit., 2018
[3] L. Zhou, Y. Zhou, J. Corso, R. Socher, and C. Xiong, "End-to-end dense video captioning with masked transformer." in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018
[4] L. Huang, W. Wang. J. Chen, and X. Wei, "Attention on attention for image captioning." in Proc. IEEE Int. Conf. Comput. Vis., 2019.
10

---

[5] G. Li, L. Zhu, P. Liu, and Y. Yang, "Entangled transformer for image captioning," in Proc. IEEE Int. Conf. Comput. Vis., 2019.
[6] M. Cornia, M. Stefanini, I.. Baraldi, and R. Cucchiara, "Meshed-memory transformer for image captioning." in Proc. IEEF Conf. Comput. Vis, Pattern Recognit., 2020.
[7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Proc. Int. Conf. Neural Inf. Process. Syst., 2017.
[8] A. Y. Ng. D. Harada, and S. Russell, "Policy invariance under reward transformations: Theory and application reward shaping." in Proc. 6th Int. Conf. Mach. Learn. (I. Bratko and S. Dzeroski, eds.), 1999.
[9] S. Venugopalan, M. Rohrbach, J. Donahuc, R. J. Mooney, T. Darrell and K. Saenko, "Sequence sequenee video text," in Proc. IEEE Int. Conf. Comput. Vis., 2015.
[10] L. Gao, Z. Guo, H. Zhang, X. Xu, and H. T. Shen, "Video captioning with attention-based LSTM and semantic consistency," IEEE Trans. Multimedia, vol. 19, no. 2017.
[11] J. Song, Z. Guo, L. Gao, W. Liu, D. Zhang, and H. T. Shen, "Hierarchical Istm with adjusted temporal attention for video captioning," arXiv preprint /706v 0/200660//31.. 2017.
[12] Hou, X. Wu, W. Zhao, Luo, and Y. Jia, "Joint syntax representation leamning and visual cue translation for vidco captioning." in Proc. IEEE Ins. Conf. Comput. Vis., pp. 9918-8927) 2019.
[13] B. Wang. L. Ma, W. Zhang, W. Jiang, Wang, and W. Liu, "Controllable video captioning with pos sequence guidance based on gated fusion network," " Proc. IEFF Int. Conf. Comput. Vis., pp. 2641-2650, 2019.
[14] T. Rahman, B. Xu, and L. Sigal, "Watch, listen and tell: Multi-modal weakly supervised dense event captioning," in Proc. IEEE Int. Conf. Comput. Vis., pp. 8908 8917, 2019.
[15] X. Wang. W. Chen, J. Wu, Y. Wang, and W. Y. Wang, "Video captioning via hierarchical reinforcement learning." in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018
[16] L. Li and B. Gong, "End-to-end video captioning with multitask reinforcement learning," in 2019 IEEE Winter Conference Applications of Computer Vision (WACV), pp. 339-3439-44433 348, 2019.
[17] D. Alvarez-Melis and T. S. Jaakkola, "Tree-structured decoding with doubly- neural networks," in 5th Int. Conf. Learn. Represent., 2017.
[18] Q. Liu, W. Guan, S. Li, and D. Kawahara, "Tree-structured decoding for solving math word prohlems," in Proc. Conf. Empirical Methods in Natural Language Processing, 2019.
[19] G. Zhou, P. Luo, R. Cao, Y. Xiao, F. Lin, B. Chen, and Q. Hc, "Tree- neural machine for linguistics-awar sentence generation," in Proc. AAAI Conf. Artif. Intell., 2018
[20] Z. Ma. C. Yuan, Y. Cheng, and X. Zhu, "Image-to-tree: A tree-structured decoder for image captioning," in IEEE International Conference on Multimedia and Expo, 2019.
[21] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "Bleu: method for automatic evaluation of machine translation." in Proceedings of the 40th annual meeting of the Association for Computational Linguistics. pp. 1111-18,220 2002.
[22] S. Banerjee and A. Lavie, "Meteor: An automatic metric for mt evaluation with improved correlation with human judgments," in Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization,, pp. 65-72, 2005.
[23] C.-Y. Lin, "Rouge: A package for automatic evaluation of summaries," in Text summarization branches oud, pp. 74-81, 2004.
[24] R. Vedantam, C. Lawrence Zitnick, and D. Parikh, "Cider: Consensus based image description evaluation." " in Proceedings of the IEEE conference on computer vision and pottern recognition, pp. 4566-4575, 2015.
[25] T. Zhang, V. Kishore, Wu, K. Q. Weinberger, and Y. Artzi, "Bertscore: Evaluating text generation with bert," arXiv preprint arXiv: 1904.09675 2019.
[26] Y. Yi, H. Deng, and J. Hu, "Improving image captioning evaluation by considering inter references variance." in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 2020.
11

---

[27] M. Honnibal and I. Montani, "spaCy 2: Natural language understanding with Bloom embeddings. convolu- tional neural networks and incremental parsing." To appear, 2017.
[28] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," in Proc. IEEE Conf. Comput. Vis. Pattern R
[29] S. loffe and C. Szegedy, "Batch normalization: Accelerating deep network training by reducing internal covariate shift," in International conference machine learning, pp- 448 448-456, PMLR, 2015.
[30] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in 3rd Int. Conf. Learn. Represent. 2015.
[31] R. Krishna, K. Hata, F. Ren, L. Fei-Fei, and J. C. Niebles, "Dense-captioning events in videos," in Proc. IEEE Int. Conf. Comput. Vis., 2017.
[32] Y. Li, T. Yao, Y. Pan, H. Chao, and T. Mci, "Jointly localizing and deseribing events for dense vidco captioning," " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018.
[33] J. Mun, L. Yang. 7. Ren, N. Xu, and B. Han, "Streamlined dense video captioning." in Proc. IEFE Conf. Comput. Vis. Pattern Recognit., 2019.
[34] W. Pei, J. Zhang, X. Wang, L. Ke, X. Shen, and Y.-W. Tai, "Memory-attendee recurrent network for vidco captioning," " in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 8347-8356, 2019.
[35] G. Tan, D. Liu, M. Wang, and 7.-J. Zha, "I carrnnn to discretely compose reasoning module networks for video captioning," arXiv preprint arXiv:2007.090// 2020.
[36] J. Carreira and A. Zisserman, "Quo vadis, action recognition? a new model and the kinetics dataset," in Proc. IEFE Conf. Comput. Vis. Pattern Recognit., pp. 6299-6308, 2017.
[37] Z. Zhang, Y. Shi, C. Yuan, B. Li, P. Wang, W. Hu, and Z.-J. Zha, "Object relational graph with teacher- recommended learning for video captioning." in Proceedings of the IEEE/CVF conference on computer vision and puttern recognition, pp. 13278-13288, 2020.
[38] S. Chen, W. Jiang, W. Liu, and Y.-G. Jiang, "Learning modality interaction for temporal sentence localiza- tion and event captioning videos," " European Conference on Computer Vision, pp. 333-351, Springer, 2020.
12


Reviews:
[{"review_id": "mvRk_hD-hP", "reviewer": "Reviewer_2Mir", "summary": "The paper proposes a new architecture that combines a multimodal dependency tree approach with a tree-structured reinforcement learning method to improve upon existing methods on the video captioning task. The authors conduct experiments to understand the impact of the two components on four datasets. Experiments are made with varying visual features, on settings of varying complexity, and based on human evaluation. The results and analysis show improvements over the state of the art and that adopting a multimodal tree structure is a promising area of research for this task.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Clairty:\nThe paper is clear and easy to follow despite multiple ideas being introduced. Images and the structured formulas are useful in understanding the concept and methodology is a significant part of the paper. One part that was a bit confusing on my first read was page 4 section 3.3, when the attention part is being described - Unless I misunderstood, the authors are using V as both the Visual features and the Value vectors. Although they might potentially be the same, it is unclear to me that this is always true since the authors also claim to use a multimodal representation per node. It might be better to use different variable names if they are not always the same or to clearly state that they will always be the same.\n\nOriginality:\n\nBy itself, the two contributions wouldn't much original. The multimodal tree structure would be too incremental if compared to previous work, mainly a mix of existing contributions. Also, incorporating CIDEr in the reward function has been tried before, although this specific reward function is different and incorporates the tree structure as well. That said, both contributions combined might make it sufficiently original for a publication.\n\nQuality:\nAt first I was not convinced that introducing CIDEr in the reward function would be fair with the other methods that are not directly optimized for the metric but the improvements are constant throughout several different (although correlated) metrics. One improvement on this aspect might be to share how the results would change if CIDEr was replaced by one of the other metrics like Rouge and see the effect of that choice on the reward function - that might show how robust the reward is to the quality of the metric.\n\nI would have liked to see significance scores for the results as well, I think these are lacking and might be relevant on some of the results that are too similar to each other. Also, for the ablation studies, it would be interesting to see the mean and standard deviation of multiple runs before making the claims the paper makes.\n\nDuring the experiments for section 4.6, the w/o tree method performs better on one of the short and simple scenarios, an interesting finding that is not discussed by the authors and could potentially 1- expose limitations of the method and 2- inform future research on how to improve the proposed method.\n\nFor the Qualitative results like the ones proposed on 4.7, it would be interesting to see some error analysis and understand where the attention + tree based methods fails. For example, on the second result shown the captioning system says the person left the room while the person never leaves. Maybe an analysis of the attention components could show where the problem was - was it the attention, tree structure?\n\nOverall, this is a well-written and interesting piece of work. I would appreciate more error analysis and significance tests to make it even stronger but I believe it is a good contribution to the video captioning task and reinforces directions related to reinforcement learning and tree-based text generation under this context.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't think there is a lot to add on negative societal impact since this paper is based on existing datasets and incremental contributions to the video captioning model. The error analysis I suggested before could help address some of the limitations and potentially surface societal problems related to the situations where the model fail but it is hard to make that claim before the initial error analysis is conducted.", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "RgN_AaaPT0U", "reviewer": "Reviewer_NzTM", "summary": "This paper studies the task of video captioning. It proposes a structured approach where models generate a dependency tree for the caption, instead of just a sentence. The nodes of the dependency tree are words, and the tree is generated in depth-first order; thus, the sentence can be easily recovered by iterating through the tree. \n\nThe model is learned by a \"pretraining stage\"; where a left-to-right style loss is used (generating the next word, and the node in the tree, given previous nodes). In a \"finetuning stage\", the model is optimized using reinforcement learning with the improvement in CIDEr score as the reward (along with a reward as to whether the generated edge is in the ground truth tree or not). ", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: This paper works on a challenging task that is of interest to many in the community, and it has competitive results. Additionally, many resaerchers in both language and vision are interested in combining syntax with deep-learning based approaches, so this learning paradigm could be interesting to them as well.\n\nIn addition to strong results (both human as well as automatic eval), an ablation study is shown. The ablations seem to suggest that using a tree structure helps, particularly when RL is optimized. Qualitative examples are also discussed.\n\nOverall, I would be willing to raise my score and vote to accept this paper if the weaknesses below are addressed.\n\nWeaknesses: The main concern to this reviewer concerns the overall premise of the paper. The argument expressed in L29-L30 is that graph-structure on the text allows the model to resolve long-term dependencies. However, the language in these datasets is pretty short -- 1 to 2 sentences. Thus, it seems very likely to this reviewer that there is something else going on for why in the ablation study, the tree model outperforms the baselines. I would like to be surprised, however, this paper would benefit from convincing evidence for why the tree model is better, along with helpful explanation about what the tree-structured losses/model achieves exactly.\n\nThe ablations are helpful, but one ablation that the paper could benefit from is disentangling the objective from the model in some way. In other words, if the model was trained to jointly generate trees, as well as sentences, and at test time generated only one of those, that might help provide evidence for why a tree structure is better. An alternate explanation for the ablation results might be that the losses proposed add more regularization to the model, even if they are not actually helpful for the end task.\n\nSection 4.6 claims that the proposed tree structure is better at generating complex sentences, however, this seems a bit unclear from Table 4 alone. Though the model seems to degrade less when compared with references that are longer, to this reviewer this is not by itself indicative of generating longer sentences, but rather being more similar to other longer sentences. One way the argument might be made stronger is if you evaluated the sentence length or complexity of generated sentences in some way, comparing \"w/o tree\" with the proposed model.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No section is provided, the paper could benefit from discussing this.", "review_text": "", "extra_scores": {"ethical_concerns": "None that come to mind to this reviewer.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}, {"review_id": "NhL6Qib2C_W", "reviewer": "Reviewer_c8JF", "summary": "This paper propose a video captioning method that generates a sentence by first constructing a multi-modal dependency tree and then traversing the constructed tree.Both the visual and textual representation features are encoded into each tree node to construct a multi-modal dependency trees for language generation of videos. The syntactic structure and semantic relationship in the sentence are represented by the tree topology.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "1.The amount of engineering for this paper is large.This paper has a large amount of experimentation，the author has done multiple experiments on both complex and simple datasets.There are many hyper-parameters, which I personally think is difficult to reproduce. Will the code be provided later?\n\n2.This paper develop a novel tree-structured reinforcement learning algorithm that optimizes the captioning model using a node-level reward, which facilitates learning the topology of the dependency trees and alleviates the reward ambiguity problem.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "1.The motivation to investigate a graph structured model is to capture the global dependency structure in the sentence which different from existing sequence models that tend to focus on the dependency between each word and its close preceding words. However,the encoder and decoder is based on Transformer，which can draw global dependencies in sentence. Therefore, I am a bit confused about the complex approach of this article.\n\n2.Table3 does not use the C3D feature to do the experiment, I think this paper should compare the result of C3D features in MSR-VTT with [1].On the MSR-VTT dataset, the performance of the method has not improved much on CIDEr, within 0.3%.\n\n3.In the ablation experiment, the performance without reinforcement learning dropped lower than without dependency tree.The two tables do not list the cases where dependency tree and RL are not used.\n\n4.Constructing the generation tree, constructing the ground-truth dependency trees, calculating loss and reward all consume many computing resources. Is the time cost of the training process large?\n\n[1] Zhang at al.Object Relational Graph with Teacher-Recommended Learning for Video Captioning.CVPR'2020\nTypos:e.g.The last two rows of table3 are wrongly blackened on the METEOR on dataset MSR-VTT.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": ["I don’t know"], "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "34xVBCWw_Zg", "reviewer": "Reviewer_xmz3", "summary": "The paper addresses video captioning by generating captions in a tree structure. It stepwise constructs the multi-modal dependency tree and then traverses the constructed tree to obtain the captions. The supervision is performed on generated captions as well as the semantic consistency reward of the generated sub-trees. Experiments are conducted on several video captioning datasets.  ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n\n+ The idea of tree-structured caption generation is novel and reasonable for video captioning, which could explicitly model the hierarchical structure in the sentence and capture the long-range dependency between words.  \n \n+ The paper is well-organized and clearly written, which can be followed easily. \n\n+ Experiments are conducted on several video captioning datasets, and the ablation study shows the effectiveness of the tree-structure generation and other modules.\n\n+ For a fair comparison with state-of-the-art methods, authors evaluate their model using different visual features. \n\nWeaknesses\n\n+ The authors do not compare their methods with some state-of-the-art methods, such as [1], ORG-TRL[2], PMI-CAP[3], RMN(H+L)[34], SBAT[4], etc. And those methods outperform the proposed model. Authors should include them and discuss the difference and reason.\n\n+ Authors do not provide the intermediate results of tree-structure caption generation, and those intermediate results are essential for readers to understand how the hierarchical structure contributes to captioning. \n\n+ The method could also be adapted to image captioning. The authors are suggested to provide the results for image captioning to demonstrate the effectiveness of the proposed method further.\n\n[1] Hou et al., Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning. ICCV2019 \n\n[2] Zhang et al., Object Relational Graph with Teacher-Recommended Learning for Video Captioning. CVPR2020\n\n[3] Chen et al., Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos. ECCV2020\n\n[4] Jin et al., SBAT: Video Captioning with Sparse Boundary-Aware Transformer. IJCAI2020\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Authors do not discuss the limitation and potential negative societal impact of their work. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "7 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers who took part in the post rebuttal discussion recommend or lean to accept the paper. In my opinion, the concerns of reviewer c8JF have also been addressed.

The paper contributes an interesting model of using dependency trees for video captioning, showing solid performance compared to SOTA, and clear experimental ablations. Additional results and ablations, including on image captioning in the author response strengthen the paper further.

I recommend accepting the paper under the expectation that the authors address the concerns of the reviewers as done in the author response, including, but not limited to the following
1) include additional results and ablations (if they don't fit include them in supplement).
2) promptly release code, data and models when the paper is published.
3) include clarifications and typos / errors (e.g. bolding in Table 3)

Decision: Accept (Poster)