Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper propose a video captioning method that generates a sentence by first constructing a multi-modal dependency tree and then traversing the constructed tree.Both the visual and textual representation features are encoded into each tree node to construct a multi-modal dependency trees for language generation of videos. The syntactic structure and semantic relationship in the sentence are represented by the tree topology.
main_review: 1.The amount of engineering for this paper is large.This paper has a large amount of experimentation，the author has done multiple experiments on both complex and simple datasets.There are many hyper-parameters, which I personally think is difficult to reproduce. Will the code be provided later?

2.This paper develop a novel tree-structured reinforcement learning algorithm that optimizes the captioning model using a node-level reward, which facilitates learning the topology of the dependency trees and alleviates the reward ambiguity problem.
limitations_and_societal_impact: 1.The motivation to investigate a graph structured model is to capture the global dependency structure in the sentence which different from existing sequence models that tend to focus on the dependency between each word and its close preceding words. However,the encoder and decoder is based on Transformer，which can draw global dependencies in sentence. Therefore, I am a bit confused about the complex approach of this article.

2.Table3 does not use the C3D feature to do the experiment, I think this paper should compare the result of C3D features in MSR-VTT with [1].On the MSR-VTT dataset, the performance of the method has not improved much on CIDEr, within 0.3%.

3.In the ablation experiment, the performance without reinforcement learning dropped lower than without dependency tree.The two tables do not list the cases where dependency tree and RL are not used.

4.Constructing the generation tree, constructing the ground-truth dependency trees, calculating loss and reward all consume many computing resources. Is the time cost of the training process large?

[1] Zhang at al.Object Relational Graph with Teacher-Recommended Learning for Video Captioning.CVPR'2020
Typos:e.g.The last two rows of table3 are wrongly blackened on the METEOR on dataset MSR-VTT.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper addresses video captioning by generating captions in a tree structure. It stepwise constructs the multi-modal dependency tree and then traverses the constructed tree to obtain the captions. The supervision is performed on generated captions as well as the semantic consistency reward of the generated sub-trees. Experiments are conducted on several video captioning datasets.  
main_review: Strengths

+ The idea of tree-structured caption generation is novel and reasonable for video captioning, which could explicitly model the hierarchical structure in the sentence and capture the long-range dependency between words.  
 
+ The paper is well-organized and clearly written, which can be followed easily. 

+ Experiments are conducted on several video captioning datasets, and the ablation study shows the effectiveness of the tree-structure generation and other modules.

+ For a fair comparison with state-of-the-art methods, authors evaluate their model using different visual features. 

Weaknesses

+ The authors do not compare their methods with some state-of-the-art methods, such as [1], ORG-TRL[2], PMI-CAP[3], RMN(H+L)[34], SBAT[4], etc. And those methods outperform the proposed model. Authors should include them and discuss the difference and reason.

+ Authors do not provide the intermediate results of tree-structure caption generation, and those intermediate results are essential for readers to understand how the hierarchical structure contributes to captioning. 

+ The method could also be adapted to image captioning. The authors are suggested to provide the results for image captioning to demonstrate the effectiveness of the proposed method further.

[1] Hou et al., Joint Syntax Representation Learning and Visual Cue Translation for Video Captioning. ICCV2019 

[2] Zhang et al., Object Relational Graph with Teacher-Recommended Learning for Video Captioning. CVPR2020

[3] Chen et al., Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videos. ECCV2020

[4] Jin et al., SBAT: Video Captioning with Sparse Boundary-Aware Transformer. IJCAI2020

limitations_and_societal_impact: Authors do not discuss the limitation and potential negative societal impact of their work.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper studies the task of video captioning. It proposes a structured approach where models generate a dependency tree for the caption, instead of just a sentence. The nodes of the dependency tree are words, and the tree is generated in depth-first order; thus, the sentence can be easily recovered by iterating through the tree. 

The model is learned by a "pretraining stage"; where a left-to-right style loss is used (generating the next word, and the node in the tree, given previous nodes). In a "finetuning stage", the model is optimized using reinforcement learning with the improvement in CIDEr score as the reward (along with a reward as to whether the generated edge is in the ground truth tree or not). 
main_review: Strengths: This paper works on a challenging task that is of interest to many in the community, and it has competitive results. Additionally, many resaerchers in both language and vision are interested in combining syntax with deep-learning based approaches, so this learning paradigm could be interesting to them as well.

In addition to strong results (both human as well as automatic eval), an ablation study is shown. The ablations seem to suggest that using a tree structure helps, particularly when RL is optimized. Qualitative examples are also discussed.

Overall, I would be willing to raise my score and vote to accept this paper if the weaknesses below are addressed.

Weaknesses: The main concern to this reviewer concerns the overall premise of the paper. The argument expressed in L29-L30 is that graph-structure on the text allows the model to resolve long-term dependencies. However, the language in these datasets is pretty short -- 1 to 2 sentences. Thus, it seems very likely to this reviewer that there is something else going on for why in the ablation study, the tree model outperforms the baselines. I would like to be surprised, however, this paper would benefit from convincing evidence for why the tree model is better, along with helpful explanation about what the tree-structured losses/model achieves exactly.

The ablations are helpful, but one ablation that the paper could benefit from is disentangling the objective from the model in some way. In other words, if the model was trained to jointly generate trees, as well as sentences, and at test time generated only one of those, that might help provide evidence for why a tree structure is better. An alternate explanation for the ablation results might be that the losses proposed add more regularization to the model, even if they are not actually helpful for the end task.

Section 4.6 claims that the proposed tree structure is better at generating complex sentences, however, this seems a bit unclear from Table 4 alone. Though the model seems to degrade less when compared with references that are longer, to this reviewer this is not by itself indicative of generating longer sentences, but rather being more similar to other longer sentences. One way the argument might be made stronger is if you evaluated the sentence length or complexity of generated sentences in some way, comparing "w/o tree" with the proposed model.

limitations_and_societal_impact: No section is provided, the paper could benefit from discussing this.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The paper proposes a new architecture that combines a multimodal dependency tree approach with a tree-structured reinforcement learning method to improve upon existing methods on the video captioning task. The authors conduct experiments to understand the impact of the two components on four datasets. Experiments are made with varying visual features, on settings of varying complexity, and based on human evaluation. The results and analysis show improvements over the state of the art and that adopting a multimodal tree structure is a promising area of research for this task.
main_review: Clairty:
The paper is clear and easy to follow despite multiple ideas being introduced. Images and the structured formulas are useful in understanding the concept and methodology is a significant part of the paper. One part that was a bit confusing on my first read was page 4 section 3.3, when the attention part is being described - Unless I misunderstood, the authors are using V as both the Visual features and the Value vectors. Although they might potentially be the same, it is unclear to me that this is always true since the authors also claim to use a multimodal representation per node. It might be better to use different variable names if they are not always the same or to clearly state that they will always be the same.

Originality:

By itself, the two contributions wouldn't much original. The multimodal tree structure would be too incremental if compared to previous work, mainly a mix of existing contributions. Also, incorporating CIDEr in the reward function has been tried before, although this specific reward function is different and incorporates the tree structure as well. That said, both contributions combined might make it sufficiently original for a publication.

Quality:
At first I was not convinced that introducing CIDEr in the reward function would be fair with the other methods that are not directly optimized for the metric but the improvements are constant throughout several different (although correlated) metrics. One improvement on this aspect might be to share how the results would change if CIDEr was replaced by one of the other metrics like Rouge and see the effect of that choice on the reward function - that might show how robust the reward is to the quality of the metric.

I would have liked to see significance scores for the results as well, I think these are lacking and might be relevant on some of the results that are too similar to each other. Also, for the ablation studies, it would be interesting to see the mean and standard deviation of multiple runs before making the claims the paper makes.

During the experiments for section 4.6, the w/o tree method performs better on one of the short and simple scenarios, an interesting finding that is not discussed by the authors and could potentially 1- expose limitations of the method and 2- inform future research on how to improve the proposed method.

For the Qualitative results like the ones proposed on 4.7, it would be interesting to see some error analysis and understand where the attention + tree based methods fails. For example, on the second result shown the captioning system says the person left the room while the person never leaves. Maybe an analysis of the attention components could show where the problem was - was it the attention, tree structure?

Overall, this is a well-written and interesting piece of work. I would appreciate more error analysis and significance tests to make it even stronger but I believe it is a good contribution to the video captioning task and reinforces directions related to reinforcement learning and tree-based text generation under this context.
limitations_and_societal_impact: I don't think there is a lot to add on negative societal impact since this paper is based on existing datasets and incremental contributions to the video captioning model. The error analysis I suggested before could help address some of the limitations and potentially surface societal problems related to the situations where the model fail but it is hard to make that claim before the initial error analysis is conducted.
review_title: null

Meta-review: All reviewers who took part in the post rebuttal discussion recommend or lean to accept the paper. In my opinion, the concerns of reviewer c8JF have also been addressed.

The paper contributes an interesting model of using dependency trees for video captioning, showing solid performance compared to SOTA, and clear experimental ablations. Additional results and ablations, including on image captioning in the author response strengthen the paper further.

I recommend accepting the paper under the expectation that the authors address the concerns of the reviewers as done in the author response, including, but not limited to the following
1) include additional results and ablations (if they don't fit include them in supplement).
2) promptly release code, data and models when the paper is published.
3) include clarifications and typos / errors (e.g. bolding in Table 3)

Decision: Accept (Poster)