Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper studies the language grounding aspect of video-language problems. It proposes a Neural Module Network (NMN) for explicit reasoning of visually-grounded object/action entities and their relationships. The proposed method is demonstrated to be somewhat effective in the audio-visual dialogue task and has been shown superior to existing works on video QA. Overall, the paper is motivated clearly and is delivered with good clarity. The followings need to be clarified.

i) The proposed model demonstrates impressive results on TGIF-QA but without any insightful justification. Since the questions in TGIF-QA are short and usually do not involve complicated reasoning, intuitively, a heavy reasoning scheme might not necessarily pay off. Please clarify the performance gain and possible reasons. Also, "soft label programs" lack the necessary context (and should be in bold instead in Tab. 4).

ii) Including intense model variants in the main result table (Tab. 2) gives this paper a somewhat unfair advantage, especially when the best performing method on each metric comes from different model variants. The validation set (from both AVSD and TGIF-QA) is supposed to serve the purpose of model architecture search and ablation studies. Besides, the underlines in the lower part of Tab. 2 should go to method VGD-GPT2.

========== Post-Rebuttal ==========

Concerns on paper/results clarity still persist. Lowering my rating to 5.
review_title: Interesting idea; results need further clarification

final_score: rating: 4: Ok but not good enough - rejection
review_content: The author present a novel neural moudalar network  for video grounding tasks, which can provide interpretable intermediate reasoing outcomes and show the model robustness. 
This model achieves competitive results on AVSD datasets and state-of-the-art performance on TGIF-QA datasets, which demonstrates the effectiveness of the model design.

Detailed comments are listed in the following
• The novelity of the NMN is limited in this paper. The similari idea have been used in many previous literatures. I am wondering that how you define the modular space? Is there any prinpicle guidelines to design module like "find, summarize, when, describe"?
• The reasoning struture in this papar is simple. The module "find, when, where" are more like signal detectors. There is no reasoning structure for how to get the final response. (in this paper, just fuse the detected information to get the final answer by a response decoder). So this methods cannot reveal the inner correlation between final response and detected visual/language entities.
• [Question] How do you train the program generation tasks from language? Is there any groundtruth program structure annotation to supervise this? How do you determine the hyper-parameters \alpha and \beta?
• The paper is written pooly and some expressions are confusing, like "Different from..., our model are trained to fully genrate the parameters of components in the text". The parameter here refer the input of each module, which is different from model parameters.
review_title: This paper present a novel NMN approach to solving video-grounding lanuguage tasks, which decompses all language into entity references and detect corresponding action-based visual feature, then instantiate NMN with those inputs to get the final response.

final_score: rating: 5: Marginally below acceptance threshold
review_content: Description:

This paper introduces the Visio-Linguistic Neural Module Network (VilNMN) consisting of a pipeline of dialogue and video understanding neural modules. Motivated by Hu et al. (2017), Kottur et al (2017), this paper extends the NMNs on video tasks for interpretable neural models. The model explicitly resolves entity references (dialog understanding) and detects actions from videos (video understanding) for response generation. Experiments show that NMNs achieve competitive results on AVSD (video-dialog) and TGIF-QA (video-QA) benchmarks. 

Strengths:
- New modules for video understanding (“where”, “when”) have been proposed. A step towards interpretability of compositional neural networks 
- Ablation studies have been provided to understand the importance of each module in the VILNMN model.  
- The breakdown of relative CIDEr/BLEU (Figure 7 in supplementary) for different context and video length is interesting. 
- SOTA results on TGIF-QA (Video QA) while competitive results on AVSD (dialog task). 

Weaknesses:
- Availability of code is not discussed which is essential for reproducibility. 
- It would help to specify whether sentence vs corpus level BLEU was used for evaluation
- Human evaluation is not provided. Limitations of automatic metrics and their reliability in language generation have been discussed repeatedly. See (Reiter and Belz, 2009; Novikova et al., 2017; Reiter, 2018). 
- Table 6 denotes that the evaluation results decrease with longer video or larger context modeling which is the main focus of the paper.   
- It might be argued that this approach would not generalize. How would this model scale when the dialog becomes challenging (in terms of disfluencies, ellipses or alignment, topic switch, etc apart from co-reference; see Haber et al 2019 Photobook dataset for brief summary of dialog phenomena)? Similarly when the videos become more complex, would action recognition suffice? 
- CorefNMN (Kottur et al 2017) was designed specifically for co-reference resolution in the dialog. The paper could be similarly improved by explicitly motivating the specific utility of the NMN modules compared to high-level description- eg. to capture co-reference (dialog) and action recognition (video). 
- An analysis of the AVSD dataset would help understand the importance of the dialog context in the dataset - focus of one of the modules in this paper. (See Agarwal et al 2020 study for Visual dialog)

Questions:
- Since AVSD is also posed as a retrieval task (Alamri et al. 2019), have the authors evaluated the system on ranking based metrics?  
- Could the authors clarify why accuracy is also not reported for CountQA in Table 4? 
- It would help to explicitly mention the neural modules previously defined and the novel modules, eg how the “find” module differs from Kottur et al. 2018. 
- Have the authors experimented with the dialog-based modules from Kottur et al. 2018 - eg. “refer” module? 
- In Fig 2 (as well as the main text), it would help to clarify if the underlying text encoder is shared for the dialog history, question, and caption. 
- How are the audio signals incorporated in the VilNMN mentioned in Table 2? Pardon if I missed this. 
- Have the authors experimented with pre-trained weights (decoders)? 

Suggestions/Comments:
- Previously Johnson et al. (2017a); Hu et al. (2017), Kottur et al. (2018) have all explored NMN for visio-linguistic tasks (such as VQA, Visual dialog), the nomenclature “Visio-Linguistic Neural Module Network (VLNMN)” seems too broad. Something on the grounds of “ActionNMN/ActNMN” would do justice to the work. 
- Model descriptions of ablations in Table 3 could be improved for clarity. 
- Implementation details could be further specified - the framework, all other hyperparameters to ease reproducibility. 

--------------------------------------------------------------------------------------------------------------------------------------------------------
Post Rebuttal update:

I would like to thank the authors for answering the questions. I believe that an updated version addressing all the concerns in detail will find its place in other future conferences. Original rating is maintained.



review_title: Adapting Neural Module Network (NMN) for Video dialog and QA

final_score: rating: 5: Marginally below acceptance threshold
review_content: Summary:

The paper studies the application of neural module network to video-grounded language tasks. They propose a method dubbed Visio-Linguistic Neural Module Network (VilNMN) to retrieve spatio-temporal information in a video through a linguistic-based parsed program. In particular, VilNMN first extracts entity references and their corresponding actions in linguistic cues. This information is then being used to locate relevant information in the visual cue to arrive at the correct answer. The proposed method is evaluated on two large scales benchmarks AVSD and TGIF-QA, demonstrating competitive performance with state-of-the-art methods.

Comments (Technical, Major Flaws of this paper): 

(1) Overall, it is interesting to see neural module network works in such a complex setting as in video-grounded language tasks and I appreciate the efforts of the authors trying to explain the method as detailed as possible. 

(2) The use of the mathematical symbols in the paper is very confusing. Normally, upper cases (capital letters) are used to denote matrices/sets of vectors while lower cases (non-capital letters) are often used to denote vectors.

(3) How did the authors subsample F frames/clips from a video? In my understanding, there are a lot of frames in a video are blurry or distorted so if you sample them in a random manner, it would greatly affect the performance of object detection (Faster RCNN in this case). Please elaborate more on this.

(4) Subsection 3.1, in the description of "when" module: A_when,i is a vector, not a matrix. Same in the supplementary document. In addition, I am wondering since there are many objects/entities in a video, how would the "when" module be able to localize the same object/entity over time given no additional supervision? I think object tracking would greatly be beneficial in this case.

(5) In the experiments on TGIF-QA:
- Why the authors only used ResNet features instead of using similar features as in AVSD? Spatial-based level features are fine in terms of computations but are less intuitive as I expect the object-based features counterpart and the linguistic entity references represent things at the same level of abstraction. 
- Honestly, I am skeptical about the results on the TGIF-QA datasets as the gap between VilNMN and the existing methods is very significant. From Table 4, it looks like motion features (optical flow, C3D, and the likes) play a role in tasks containing repetition of actions (such as count and action). I am not sure the reason why VilNMN without the use of any of those motion features could manage to outperform the existing methods with large margins?
- Please provide more analysis on the results on the TGIF-QA as in my understanding even when a model correctly links entities in a question with their visual representations, it does not guarantee that it can arrive at correct answers. For example, in counting task, I would say most of the questions have the same parsed program. How does your model work in this scenario?


Some other concerns:

(6) At the end of the related works, the authors wrote "In video represented as sequence of images,...., e.g though average pooling, resulting in potential loss of information". This is a big assumption as using average pooling over object proposals is a bad idea as object appearance may vary very little over time in a video and the average pooling would smash the temporal information in the video. If one can properly model object tubelets via object tracking, what written in the paper wouldn't make sense.

(7) Section 3.2: The sentence "...calculated as softmax scores between an entity P_i and each token in dialogue history" is mathematically incorrect. What drives the attention weights? Softmax is a function applying over a set of elements.

I would be okay to raise the rating if the authors sufficiently address my concerns during the rebuttal phase.
review_title: I am glad to see the progress in applying NMN in video-grounded language tasks but the paper could be improved on its current form.

Meta-review: The authors propose a neural module based approach for reasoning about video grounding.  The goal is to provide performance and interpretability.  Unfortunately, the reviewers found the paper opaque, the results confusing, and expressed repeated concerns about the novelty, fairness of comparisons and concerns that the surprising results were not sufficiently well justified by the paper (or the author's response).
Decision: Reject