Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros
1.	This paper extends the semantic modular network on images to the video setting.
2.	By making use of the weak supervision from the question answer, it can learn different kinds of concepts including physical objects, attributes and events. 
3.	The paper conduct experiments on different evaluation settings including video causal reasoning, video grounding and generalization experiments.
Cons
1.	The paper lacks some details on the training procedure.
2.	It’s better to show the learned concepts visually as one of the strengthens of is explain ability.
3.	How’s the model’s sample complexity

Comments
1.	Is the trajectory refinement and model learning an iterative process?
2.	What does the RGB patches mean and how do you combine the RGB patches into the predicted locations in the dynamic predictions?
3.	Is this paper using some pre-trained object recognizer and attribute classifier which acts as some kind of pseudo labeling?
4.	In the generalization experiments, does the model trained and test using the same domain dataset?

review_title: In this paper, the authors study the problem of dynamic visual reasoning on raw videos. Compared with the previous work which require dense supervision of the video objects and events. Instead they proposed DCL which grounds objects and events without ground truth labels.  The proposed method achieves state of the art performance on CLEVRER and other tasks including video-retrieval and event localization.

final_score: rating: 7: Good paper, accept
review_content: > Summary:

The paper studies the problem of dynamic visual reasoning on the recently proposed synthetic video QA dataset CLEVRER for understanding visual properties, physical events, the causal relationship between events, and making future and counterfactual predictions. The paper proposes a joint framework called Dynamic Concept Learner which contains five modules (object trajectory detector, video feature extractor, dynamic predictor, language program parser, and neural symbolic executor) and follows a multi-step training paradigm to train the model purely from the question-answer pairs in CLEVRER train split.  The object trajectory detector module, the concept embedding/quantization, and the dynamic predictor module play a very important role in DCL. These modules enable the dynamic & causal reasoning abilities for DCL. The neural program parsing & execution part of DCL is similar to that in Neuro-Symbolic Concept Learner (NS-CL) (Mao et al., 2019) and thus DCL can be regarded as the dynamic extended version of  NS-CL.

> Strengths:

*  The paper is well-written and easy to follow.  A lot of details about the DCL model, neural programs, and dataset are provided in the appendix.

* The experiment on CLEVRER shows the effectiveness of DCL over previous methods.  Additional experiments on new CLEVRER Grounding & Retrieval applications demonstrate the generalization ability of DCL trained with the original VQA task on other CLEVRER related tasks.

* DCL is trained directly from QA pairs without additional labels for object attributes and events but still outperforms the previous methods, which is a big strength.

* During inference, DCL can acquire attribute/event concept perception, object trajectory prediction at the same time. Once the answer is generated, the whole reasoning path for the answer is also ready, which exhibits the good model interpretability of DCL.

* From *VirTex: Learning Visual Representations from Textual Annotations (Desai et al., 2020)* and *LXMERT: Learning Cross-Modality Encoder Representations from Transformers (Tan et al., 2020)*, we can see that actually textual annotations such captions and question-answer pairs contain high-quality information and are helpful for learning visual or concept representations. I think this paper is another good example of discovering such high-quality information without using additional fine-grained labels.

> Some concerns and suggestions:

- My major concern is that the evaluation is based on a synthetic toy video dataset and it is simplistic compared to real-world videos. Although the author demonstrates that DCL has transferability between different vision-language tasks on CLEVRER, it is still skeptical whether DCL can be generalized to real-world video QA datasets with more complex visual scenes such as TGIF-QA, MSVD-QA, or MSRVTT-QA since the dynamic attributes and events in CLEVRER is only limited to (moving, stationary) and (in, out, collision). If possible, evaluating on more datasets would further demonstrate the effectiveness of DCL across different visual scenes.

* Besides NS-DR baseline which was proposed together with CLEVRER dataset, Memory (Fan et al., 2019) is the most recent visual-language reasoning approach for videos. It is evaluated on TGIF-QA, MSVD-QA, and  MSRVTT-QA in the original paper. I think the author should find some newer approaches evaluated on these datasets to make comparisons on CLEVRER. For instance, [Hierarchical Conditional Relation Networks for Video Question Answering (CVPR 20’)](https://openaccess.thecvf.com/content_CVPR_2020/papers/Le_Hierarchical_Conditional_Relation_Networks_for_Video_Question_Answering_CVPR_2020_paper.pdf) can be a candidate baseline.

* Although the author introduces all the five modules and describes the training steps, it is still not clear for readers to understand how to train the five modules jointly without providing an explicit loss function or training objective. I think the author should specify such a training objective explicitly.

- The paper states that DCL does not require additional fine-grained labels for attributes and events. However, DCL requires additional 1000 programs to train the program parser for all question types. Can we call these 1000 programs additional annotations? For a new visual scene that is different from the current 3D synthetic domain, to make DCL suitable for the new visual domain,  we need to define new sets of programs and concepts, which may require additional annotation labors.

- To make the experiments more convincing on Retrieval and Grounding tasks, the author could further compare with some recent video-text retrieval/grounding approaches. To name a few: 1) [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_Fine-Grained_Video-Text_Retrieval_With_Hierarchical_Graph_Reasoning_CVPR_2020_paper.pdf) 2) [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://arxiv.org/pdf/2001.09099.pdf) 3) [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://papers.nips.cc/paper/8344-semantic-conditioned-dynamic-modulation-for-temporal-sentence-grounding-in-videos.pdf)

review_title: Good paper on dynamic visual reasoning but with concerns in generalization to real-world videos

final_score: rating: 7: Good paper, accept
review_content: =Summary
The paper proposes a new framework, Dynamic Concept Learner (DCL), which learns by watching videos and reading questions/answers.  It is inspired by prior work which combines symbolic representations with video dynamics modeling, but unlike prior work, here the authors do not use any additional supervision except for question/answer pairs. DCL is quiet complex, involving multiple components, including program parser (transforms the question/answer into executable symbolic programs), object detection&tracking, object-centric feature extractor (with static and dynamic components), object/event concept embeddings (share the same latent space as the visual features), trajectory refinement (based on static cues) and dynamic predictor (for future and counterfactual scenes), and, finally, symbolic executor (to predict whether the answer is correct), which runs on the latent features. DCL achieves state-of-the-art results on the CLEVRER dataset. The learned representations are also shown to be useful for other tasks, such as grounding and retrieval.

=Strengths

The paper is overall well written and the approach is novel to the best of my knowledge.

DCL achieves state-of-the-art results on the CLEVRER dataset, w/o accessing visual labels from simulation during training.
The biggest improvement is notably achieved on the predictive and counterfactual questions.

The authors also show that DCL learns to ground object/event concepts w/o explicit labels.

Finally, the authors construct CLEVRER-Grounding and CLEVRER-Retrieval datasets to test the generalization ability of DCL. They show that it significantly improves over a baseline on both tasks. 

=Weaknesses/High-level comments

Counterfactual questions remain particularly hard to answer, do the authors have some intuition as to what may be missing in their model? What are some common failure modes?

Table 1: it is somewhat surprising that the difference between the DCL and DCL-Oracle is so small, and DCL even improves over DCL-Oracle in one case. Why is that?

What do the authors believe is the main limitation of their proposed approach? Currently it operates on synthetic images as well as synthetic language, what will be necessary to move to more realistic domains?

Nothing was stated about making the code available.

=Minor comments (P# - page number)
- P1 that what the concepts => what the concepts
- P3, Sec 3: DYNAMICS CONCEPT LEARNER => DYNAMIC CONCEPT LEARNER
- P4: an program parser => a program parser
- The paper heavily relies on the supplemental material.
review_title: An effective approach to learning dynamic concepts without expensive supervision

final_score: rating: 6: Marginally above acceptance threshold
review_content: Summary:

The paper presents a neural-symbolic approach for video question answering. In particular, the authors propose the Dynamic Concept Learner (DCL), a network architecture to localize object trajectories over time and further build a graph network to model the physical interaction between objects and events. Finally, a standard symbolic executor is used to reason across visual and language cues for question answering. The proposed method is evaluated on CLEVRER dataset in which it achieves new state of the art performance.

Pros:

(1) The main contribution of this paper is that it is able to ground object and event concepts to model the dynamic scene of the interactions between objects/events without the use of extra supervision. This is well motivated as we do not always have access to explicit labels such as object attributes, masks, and collision events in real-world applications.

(2) The idea of dynamics prediction to support predictive and counterfactual questions is new. This makes the proposed framework as a whole interesting IMHO.

(3) The proposed method sets new state of the art performance on CLEVRER dataset in all question types. On video grounding and video-text retrieval tasks, DCL seems to achieve interesting results.

Cons:

Despite the pros above, I have some comments on the paper:

(1) I found the paper is quite hard to follow. Figure 2 and its description represent the process of DCL (object trajectory detector -> dynamic predictor -> feature extractor -> symbolic executor). However, the multi-step training paradigm in Sec. 1, 3.1, and 3.2 seems to follow a different structure. In addition, some details are missing in the paper. For example, in the ‘Grounding Object and Event Concepts’ subsection, where did you get the vector embedding for the concept "moving" (s_moving) from?

(2) The motivation in the subsection ‘Trajectory refinement’ is exactly as what explained in DeepSORT [1] – a common tracking algorithm. In addition, I believe the use of Kalman Filters in DeepSORT makes it better in handling occlusion than the Eq. 2. I suggest the author discuss the object tracking literature for better judgments. 

(3) One of the main weaknesses of this paper is they only evaluate their method on one synthetic dataset which is CLEVRER. It seems to me that the proposed method is particularly designed to solve this dataset. While I find the results interesting, I think physical interactions in those synthetic data are too simple and do not reflect much what happens in real-world data. Not to mention the limit in terms of linguistic variants of those synthetic datasets compared to human natural language. I would appreciate if the author can evaluate the proposed model on other datasets such as CATER [2] for temporal reasoning and Video QA datasets (TGIF-QA [3], TVQA [4], etc).

(4) Similar to my concerns in (3), it would be more convincing to see if the DCL works on video grounding datasets such as ActivityNet-Entities or YouCook2, MSR-VTT for video-text retrieval to support their claims on the generalization of the method. 

I am willing to raise the rating if the author could properly address my concerns.

References:
- [1] Wojke, Nicolai, Alex Bewley, and Dietrich Paulus. "Simple online and realtime tracking with a deep association metric." 2017 IEEE international conference on image processing (ICIP). IEEE, 2017.
- [2] Girdhar, Rohit, and Deva Ramanan. "Cater: A diagnostic dataset for compositional actions and temporal reasoning." arXiv preprint arXiv:1910.04744 (2019).
- [3] Jang, Yunseok, et al. "Tgif-qa: Toward spatio-temporal reasoning in visual question answering." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.
- [4] Lei, J., Yu, L., Bansal, M., & Berg, T. L. (2018). Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696.

review_title: I find the method is interesting but the paper could be improved on its current form

Meta-review: This paper received 4 reviews with mixed initial ratings: 6,7,7,5. The main concerns of R1, who gave an unfavorable score, included lack of clarity (the manuscript is hard to follow) and limited empirical evaluation (the method is tested on a single synthetic dataset, CLEVRER). The latter point is echoed in other reviews as well. In response to that, the authors submitted a new revision and provided detailed responses to each of the reviews separately, which seemed to have addressed these concerns. R1 upgraded the rating and recommended acceptance.
As a result, the final recommendation is to accept this submission for presentation at ICLR as a poster.
Decision: Accept (Poster)