Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning
Authors: Zhenfang Chen
Jiayuan Mao
Jiajun Wu
Kwan-Yee Kenneth Wong
Joshua B. Tenenbaum
Chuang Gan
Venue: ICLR 2021 Poster (2021)
Keywords: Concept Learning
Neuro-Symbolic Learning
Video Reasoning
Visual Reasoning

Abstract: We study the problem of dynamic visual reasoning on raw videos. This is a challenging problem; currently, state-of-the-art models often require dense supervision on physical object properties and events from simulation, which are impractical to obtain in real life. In this paper, we present the Dynamic Concept Learner (DCL), a unified framework that grounds physical objects and events from video and language. DCL first adopts a trajectory extractor to track each object over time and to represent it as a latent, object-centric feature vector. Building upon this object-centric representation, DCL learns to approximate the dynamic interaction among objects using graph networks. DCL further incorporates a semantic parser to parse question into semantic programs and, finally, a program executor to run the program to answer the question, levering the learned dynamics model. After training, DCL can detect and associate objects across the frames, ground visual properties and physical events, understand the causal relationship between events, make future and counterfactual predictions, and leverage these extracted presentations for answering queries. DCL achieves state-of-the-art performance on CLEVRER, a challenging causal video reasoning dataset, even without using ground-truth attributes and collision labels from simulations for training. We further test DCL on a newly proposed video-retrieval and event localization dataset derived from CLEVRER, showing its strong generalization capacity.

Published as a conference paper at ICLR 2021
GROUNDING PHYSICAL CONCEPTS OF OBJECTS AND EVENTS THROUGH DYNAMIC VISUAL REASONING
Zhenfang Chen The University of Hong Kong
Jiayuan Mao MIT CSAIL
Jiajun Wu Stanford University
Kwan-Yee Yee K. Wong
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL
Chuang Gan MIT-IBM Watson Al Lab
The University of Hong Kong
ABSTRACT
We study the problem of dynamic visual reasoning on raw videos. This is chal- lenging problem; currently, state-of-the-art models often require dense supervision on physical object properties and events from simulation, which are impractical obtain in real life. In this paper, we present the Dynamic Concept Learner (DCL), a unified framework that grounds physical objects and events from dynamic scenes and language. DCL first adopts a trajectory extractor to track each object over time and to represent it as a latent, object-centric feature vector. Building upon this object-centric representation, DCL learns to approximate the dynamic interac- tion among objects using graph networks. DCL further incorporates semantic parser to parse question into semantic programs and, finally, a program executor to run the program to answer the question, levering the learned dynamics model. After training, DCL can detect and associate objects across the frames, ground visual properties and physical events, understand the causal relationship between events, make future and counterfactual predictions, and leverage these extracted presentations for answering queries. DCL achieves state-of- of--th--rtttttt performance on CLEVRER, a challenging causal video reasoning dataset, even without using ground-truth attributes and collision labels from simulations for training. We fur- ther test DCL on newly proposed video-retrieval and event localization dataset derived from CLEVRER, showing its strong generalization capacity.
INTRODUCTION
Visual reasoning in dynamic scenes involves both the understanding of compositional properties, relationships, and events of objects, and the inference and prediction of their temporal and causal structures. As depicted in Fig. 1, to answer the question "What will happen next?" based on the observed video frames, one needs to detect the object trajectories, predict their dynamics, analyze the temporal structures, and ground visual objects and events to get the answer "The blue sphere and the yellow object collide".
Recently, various end-to-end neural network- beasork-based approaches have been proposed for joint un- derstanding of video and language (Lei et al., 2018; Fan et al., 2019). While these methods have shown great success in learning to recognize visually complex concepts, such as human activities (Xu et al., 2017; Ye et al., 2017), they typically fail on benchmarks that require the understanding of compositional and causal structures in the videos and text (Yi et al., 2020). Another line of research has been focusing on building modular neural networks that can represent the compositional structures in scenes and questions, such as object-centric scene structures and multi-hop reasoning (Andreas et al., 2016; Johnson al., 201 2017b; Hudson & Manning, 2019). However, these methods are designed for static images and do not handle the temporal and causal structure in dynamic scenes well, leading to inferior performance on video causal reasoning benchmark CLEVRER (Yi et al., 2020).
To model the temporal and causal structures in dynamic scenes, Yi et al. (2020) proposed an oracle model to combine symbolic representation with video dynamics modeling and achieved of the-art performance on CLEVRER. However, this model requires videos with dense annotations for visual attributes and physical events, which are impractical extremely fabor-intensive in real scenes.
Project page: http:// /dal..a...//// csail 11 mi mit.edu

---

Published as a conference paper at ICLR 2021
Observed Frames
Predicted Dynamics
Object Grounding > Event Grounding Answer: The blue sphere and the ycllow object collide. The process to handle visual reasoning in dynamic seenes. The trajectories of the target blue and yellow spheres are marked by the sequences of bounding boxes. Object attributes of the blue sphere and ycllow sphere and the collision event marked by blue, yellow and purple colors. Stroboscopic imaging is applied for motion visualization.
Question: What will happen next?
Figure I:
We argue that such dense explicit video annotations are unnecessary for video reasoning, since they are naturally encoded in the question answer pairs associated with the videos. For example, the question answer pair and the video in Fig. 1 can implicitly inform model what the concepts "sphere", "blue", "yellow" and "collide" really mean. However, a video may contain multiple fast-moving occluded objects and complex object interactions, and the questions and answers have diverse forms. remains an open and challenging problem to simultaneously represent objects over time, train an accurate dynamic model from raw videos, and align objects with visual properties and events for accurate temporal and causal reasoning, using vision and language as the only supervision.
Our main ideas object and event concept grounding, and dynamics prediction. We first detect objects in the video, associating them into object tracks across the frames. We can then ground various object and event concepts from language, train a dynamic model on top of object tracks for future and counterfactual predictions, analyze relationships between events, and answer queries based on these extracted representations. All these modules can be trained jointly by watching videos and reading paired questions and answers.
factorize video perception and reasoning into several modules: object tracking,
To achieve this goal, we introduce Dynamic Concept Learner (DCL), unified neural-symbolic framework for recognizing objects and events videos and analyzing their temporal and causal structures, without explicit annotations on visual attributes and physical events such collisions during training. To facilitate model training, a multi-step training paradigm has been proposed. We first run object detector individual frames and associate objects across frames based motion- based correspondence. Next, our model learns concepts about object properties, relationships, and events by reading paired questions and answers that describe explain the events in the video. Then, we leverage the acquired visual concepts in the previous steps to refine the object association across frames. Finally, we train dynamics prediction network (Li al., 2019b) based the refined object trajectories and optimize it jointly with other learning parts in this unified framework. Such training paradigm ensures that all neural modules share the same latent space for representing concepts and they can bootstrap the learning of each other.
We evaluate DCL's performance on CLEVRER, a video reasoning benchmark that includes descrip- tive, explanatory, predictive, and counterfactual reasoning with uniform language interface. DCL achieves state-of-the-art performance on all question categories and requires no scene supervision such as object properties and collision events. To further examine the grounding accuracy and transferability of the acquired concepts, we introduce two new benchmarks for video-text retrieval grounding and localization the CLEVRER videos, namely CLEVRER- Retrieval and CLEVRER LLEVVRER-Grounding Without any further training, our model generalizes well to these benchmarks, surpassing the baseline by noticeable margin.
and
2 RELATED WORK
Our work is related to reasoning and answering questions about visual content. Early studies like (Wu et al. 2016; Zhu et al., 2016; Gan et al., 2017) typically adopted monolithic network architectures and mainly focused on visual understanding. To perform deeper visual reasoning, neural module networks were extensively studied in recent works (Johnson et al., 2017a; Hu et al., 2018; Hudson & Manning, 2018; Amizadeh et al., 2020), where they represent symbolic operations with small neural networks and perform multi-hop reasoning. Some previous research has also attempted learn
2

---

Published as a conference paper at ICLR 2021
Input Question and Choice
Parsed Programs
Concept Embeddings
Filter
following will not huppen the yellow cabe remowod??
Question:
Program Parser
O- Filter
Ujb & Yellow
Ceunterfact
Symholic
Exceutor
Vellow
Culbe
Scatonny
Callisins
QRA
Yes
Sihore
Choice: collision between the sphere and the sylinder.
o- Filter Cylinder
Colliirn
Static Concepts
Dynamic Concepts
Event
Concepts
Trajectory Refinement
Symbolic Exceutor
(
Dynamic Predictor
IB Oh:i Olez
Obil Chty
Extractor
Okji.
Trajectory Detector
(it Yeea
[b New
Average Visual Feature
Temporal Sequence Hearure
Interactive Feature
(BF
Input video
Object Trajectories
Counterfactual Object Trajectorics
Representation for objects
events
Figure 2: DCL'ss architecture for counterfactual questions during inference. Given input video and its corresponding question and choice, first program parser to parse the question and the choice into executable programs. We adopt an object trajectory detector to detect trajectories all objects. Then, the extracted objects are sent to a dynamic predictor to predict their dynamics. Next, the extracted objects are sent to the feature extractor to extract latent representations for objects and events. Finally, feed the parsed programs and latent representation the symbolic executor answer the question and optimize concept learning.
visual concepts through visual question answering (Mao al., 2019). However, it mainly focused learning static concepts in images, while our DCL aims at learning dynamic concepts like moving and collision in videos and at making use of these concepts for temporal and causal reasoning.
Later, visual reasoning was extended to more complex dynamic videos (Lei et al., 2018; 18; Fan et al., 2019: Li al., 2020; 2019a; Huang al., 2020). Recently, Yi al. (2020) proposed CLEVRER, new video reasoning benchmark for evaluating computational models' comprehension of the causal structure behind physical object interaction. They also developed an oracle model, combining the neuro-symbolic visual
model (Yi et al., 2018) with the dynamics prediction model (Li al., 2019b), showing competitive performance. However, this model requires explicit labels for object attributes, masks, and spatio-temporal localization of events during training. Instead, our DCL has no reliance on any labels for objects and events and can learn these concepts through natural supervision (i.e., videos and question guestion-answer pairs).
Our work also related to temporal and relational reasoning in videos via neural networks (Wang & Gupta, 2018; Materzynska al., 2020; Ji et al., 2020). These works typically rely specific action annotations, while our DCL learns to ground object and event concepts and analyze their temporal relations through question answering. Recently, various benchmarks (Riochet et al., 2018; Bakhtin et al., 2019; Girdhar & Ramanan, 2020; Baradel et al., 2020; Gan et al., 2020) have been proposed to study dynamics and reasoning physical scenes. However, these datasets mainly target at pure video understanding and do not contain natural language question answering. Much research has been studying dynamic modeling for physical scenes (Lerer et al., 2016; Battaglia al., 2013; Mottaghi et al.. 2016; Finn et al., 2016; Shao et al., 2014; Fire & Zhu, 2015; Ye et al., 2018; Li et al., 2019b). We adopt PropNet (Li al., 2019b) for dynamics prediction and feed the predicted scenes to the video feature extractor and the neuro-symbolic executor for event prediction and question answering.
While many works (Zhou et al., 2019; 2018; Gan al., 2015) have been studying on the problems of understanding human actions and activities (e.g., running, cooking, cleaning) in videos, our work's primary goal is to design a unified framework for learning physical object and event concepts (e.g.. collision, falling, stability). These tasks are of great importance in practical applications such as industrial robot manipulation which requires AI systems with human-like physical common sense.
DYNAMIC CONCEPT LEARNER
In this section, we introduce a new video reasoning model, Dynamic Concept Learner (DCL), which learns to recognize video attributes, events, and dynamics and to analyze their temporal and causal structures, all through watching videos and answering corresponding questions. DCL contains five modules, 1) an object trajectory detector, 2) vidco feature extractor, 3) a dynamic predictor, 4) a language program parser, and 5) neural symbolic executor. As shown in Fig. 2, given an input
3

---

Published as a conference paper at ICLR 2021
video, the trajectory detector detects objects in each frame and associates them into trajectories; the feature extractor then represents them as latent feature vectors. After that, DCL quantizes the objects' static concepts (i.e., color, shape, and material) by matching the latent object features with the corresponding concept embeddings in the executor. As these static concepts are motion-independent they can be adopted as an additional criteria refine the object trajectories. Based on the refined trajectories, the dynamics predictor predicts the objects' movement and interactions in future and counterfactual scenes. The language parser parses the question and choices into functional programs, which are executed by the program executor on the latent representation space to get answers.
The object and event concept embeddings and the object-centric representation share the same latent space; answering questions associated with videos can directly optimize them through backpropa- gation. The object trajectories and dynamics can be refined by the object static attributes predicted by DCL. Our framework enjoys the advantages of both transparency and efficiency, since it enables step-by-step investigations of the whole reasoning process and has no requirements for explicit annotations of visual attributes, events, and object masks.
3.1 MODEL DETAILS
Object Detection and Tracking. Given video, the object trajectory detector detects object proposals in each frame and connects them into object trajectories O = In" where o" {b, (=) and N is the number of objects in the video. b. = [x] y2²; w h is an object proposal at frame / and T is the frame number, where (x? y²) denotes the normalized proposal coordinate center and We and fy denote the normalized width and height, respectively.
The object detector first uses a pre-trained region proposal network (Ren et al., 2015) to generate object proposals in all frames, which are further linked across connective frames get all objects' trajectories. Let {b and 1 by Gkioxari & Malik (2015), we define connection score 81 between bj and b/42 to be
) be two sets of proposals in two connective frames. Inspired
(b b/41)) se (b)) + se (b/+1) + A (bU b/41))
(1)
where sb() is the confidence score of the proposal by, ToU the intersection over union and À is scalar. Gkioxari & Malik (2015) adopts greedy algorithm to connect the proposals without global optimization. Instead, we assign boxes (b/+1 1 at the t + 1 frame {b) by linear sum assignment.
Video Feature Extraction. Given input video and its detected object trajectories, we extract three kinds of latent features for grounding object and event concepts. It includes 1) the average visual feature f ND for static attribute prediction, 2) temporal sequence feature fe Nx1T for dynamic attribute and unary event prediction, and 3) interactive feature f E collision event prediction, where Dj and D2 denote dimensions of the features and K is the number of sampled frames. We give more details on how to extract these features in Appendix B.
for
Grounding Object and Event Concepts. Video Reasoning requires a model ground object and event concepts in videos. DCL achieves this by matching object and event representation with object and event embeddings in the symbolic executor. Specifically, DCL calculates the confidence score that the n-th object is moving by [coois maa (f:)) - 5] /A, where fü denotes the temporal sequence feature for the n-th object, s moving denotes vector embedding for concept moving, and mdo denotes a linear transformation, mapping fü into the dynamic concept representation space. and À are the shifting and scaling scalars, and cos() calculates the cosine similarity between two vectors. DCL grounds static attributes and the collision event similarly, matching average visual features and interactive features with their corresponding concept embeddings in the latent space. We give more details on the concept and event quantization in Appendix E.
Trajectory Refinement. The connection score in Eq. I ensures the continuity of the detected object trajectories. However, it does not consider the objects' visual appearance; therefore, it may fail to track the objects and may connect inconsistent objects when different objects are close to each other and moving rapidly. To detect better object trajectories and to ensure the consistency of visual appearance along the track, we add a new term to Eq. 1 and re-define the connection score be
si((b 1'm o b(41) (1)) (b(1)) d TOU 1b41) + d2 Sappeurr ( ((b,, }t = b(41); (2)
4

---

Published as a conference paper at ICLR 2021
where fappear ((bm I m= 1(+1) measures the attribute similarity between the newly added proposal b/m and all proposals ((b Jm=o in previous frames. We define fuppear as
Sappear [appear((Uin (o: 3x1
Saur B(41):
where attr E fcolor, material, shape). (bmr b(+1) equals to 1 when by, and b/4+, have the same attribute, and otherwise. In Eq. 2, fuppeur ensures that the detected trajectories have consistent visual appearance and helps distinguish the correct object when different objects are close to each other in the same frame. These additional static attributes, including color, material, and shape, are extracted without explicit annotation during training. Specifically, we quantize the attributes by choosing the concept whose concept embedding has the best cosine similarity with the object feature. We iteratively connect proposals at the t frame to proposals at the frame and get set of object trajectories O = [o"] Jn=1 where o = [b; It-1-
attr m=0
Dynamic Prediction. Given an input video and the refined trajectories of objects, we predict the locations and RGB patches of the objects in future counterfactual scenes with Propagation Network (Li et al., 2019b). We then generate the predicted scenes by pasting RGB patches into the predicted locations. The generated scenes are fed to the feature extractor extract the corresponding features. Such a design enables the question answer pairs associated with the predicted scenes to optimize the concept embeddings and requires no explicit labels for collision prediction, leading to better optimization. This is different from Yi et al. (2020), which requires dense collision event labels to train a collision classifier.
To predict the locations and RGB patches, the dynamic predictor maintains a directed graph (V, D) N.N (un)) {dn, n2 |nj=1, The n-th vertex Un is represented by a concatenation of tuple (l') over a small time window w. where by = [=] wf', wy, hf] is the n-th object's normalized coordinates and Pt is cropped RGB patch centering at (xy v()). The edge d=y,ng denotes the relation between the n1 n1-th and n2-thh objects and is represented by the concatenation of the normalized coordinate difference 6," b"a. The dynamic predictor performs multi-step message passing to simulate instantaneous propagation effects.
During inference, the dynamies predictor predicts the locations and patches frame h using the features of the last w observed frames in the original video. We get the predictions at frame k + 2 by autoregressively feeding the predicted results at frame get the counterfactual scenes where the n-th object is removed, we remove the n-th vertex and its associated edges from the input to predict counterfactual dynamics. Iteratively, we get the predicted
the input to the predictor. To
N.K'' normalized coordinates (ōg In=1,k'= and RGB patches {pA Jnm1,k'm at all predicted K' frames. We give more details on the dynamic predictor at Appendix C.
N.KK
Language Program Parsing. The language program parser aims to translate the questions and choices into executable symbolic programs. Each executable program consists of series of opera- tions like selecting objects with certain properties, filtering events happening at a specific moment, finding the causes of an event, and eventually enabling transparent and step-by-step visual reasoning. Moreover, these operations are compositional and can be combined to represent questions with various compositionallty and complexity. We adopt seq2seq model (Bahdanau et al., 2015) with an attention mechanism to translate word sequences into a set of symbolic programs and treat questions and choices, separately. We give detailed implementation of the program parser in Appendix D.
Symbolic Execution. Given a parsed program, the symbolic executor explicitly runs it on the latent features extracted from the observed and predicted scenes to answer the question. The executor consists of a series of functional modules to realize the operators in symbolic programs. The last operator's output is the answer to the question. Similar to Mao et al. (2019), we represent all object states, events, and results of all operators in probabilistic manner during training. This makes the whole execution process differential w.r.t. the latent representations from the observed and predicted scenes. It enables the optimization of the feature extractor and concept embeddings in the symbolic exccutor. We provide the implementation of all the operators in Appendix E.
3.2 TRAINING AND INFERENCE
Training. We follow a multi-step training paradigm to optimize the model: 1) We first extract object trajectories with the scoring function in Eq. and optimize the video feature extractor and
5

---

Published as a conference paper at ICLR 2021
Extra Labels Descriptive
Explanatory per opt. per ques. per opt. per ques. per opt. per ques.
Predictive
Counterfactual
Methods
Attr. Prog.
CNN+MLP
48.4
54.9
18.3
50.5
13.2
55.2
9.0
CNN+LSTM
51.8
62.0
17.5
57.9
31.6
61.2
14.7
Memory
No
No
54.7
53.7
13.9
50.0
33.1.
54.2
7.0
HCRN
55.7
63.3
21.0
54.1
21.0
57.1
11.5
MAC (V)
85.6
59.5
12.5
51.0
16.5
54.6
13.7
TVQA+
72.0
63.3
23.7
70.3
48.9
53.9
Yes
No
MAC (V+)
86.4
70.5
22.3
59.7
42.9
63.5
25.1
IEP (V)
52.8
52.6
14.5
50.0
9.7
53.4
3.8
TbD-net (V)
No
Yes
79.5
61.6
3.8
50.3
6.5
56.1
4.4
DCL (Ours)
90.7
89.6
82.8
90.5
82.0
80.4
46.5
NS-DR
88.1 1
87.6
79.6
82.9
68.7
74.1
42.4
NS-DR (NE)
Yes
Yes
85.8
85.9
74.3
75.4
54.1
76.1
42.0
DCL-Oracle (Ours)
91.4
89.8
82.0
90.6
82.1
80.7
46.9
Table Question answering accuracy on CLEVRER. The first and the second parts of the table show the models without and with visual attribute and event labels during training, respectively. Best performance is highlighted in boldface. DCL and DCL-Orace denote models trained without and with labels of visual attributes and events, respectively.
concept embeddings in the symbolic executor with only descriptive and explanatory questions; 2) We quantize the static attributes for all objects with the feature extractor and the concept embeddings learned in Step 1) and refine object trajectories with the scoring function Eq. 2: 3) Based on the refined trajectories,
train the dynamic predictor and predict dynamics for future and counterfactual scenes; 4) We train the full DCL with all the question answer pairs and get the final model. The program executor is fully differentiable w.r.t. the feature extractor and concept embeddings. We use cross-entropy loss to supervise open-ended questions and use mean square error loss to supervise counting questions. We provide specific loss functions for each module in Appendix
Inference. During inference, given an input video and question, we first detect the object trajectories and predict their motions and interactions in future and counterfactual scenes. We then extract object and event features for both the observed and predicted scenes with the feature extractor. We parse the questions and choices into executable symbolic programs. We finally execute the programs on the latent feature space and get the answer to the question.
EXPERIMENTS
To show the proposed DCL's advantages, conduct extensive experiments the video reasoning benchmark CLEVRER. Existing other video datasets either ask questions about the complex visual context (Tapaswi et al., 2016; Lei al., 2018) () study dynamics and reasoning without question answering (Girdhar& & Ramanan, 2020; Baradel et al., 2020). Thus, they are unsuitable for evaluating video causal reasoning and learning object and event concepts through question answering. We first show its strong performance on video causal reasoning. Then, we show DCL's ability on concept learning, predicting object visual attributes and events happening in videos. We show DCL's generalization capacity to new applications, including CLEVRER-GGrond Retrieval. We finally extend DCL to real block tower video dataset (Lerer al., 2016).
and CLEVRER
4.1 IMPLEMENTATION DETAILS
Following the experimental setting in Yi et al. (2020), train the language program parser with 1000 programs for all question types. We train all our models without attribute and event labels. Our models for video question answering are trained on the training set, tuned on the validation set, and evaluated in the test set. To show DCL's generalization capacity, we build CLEVRER-Grounding Grounding and CLEVRER- Retrieval datasets from the original CLEVRER videos and their associated video annotations. We provide more implementation details in Appendix A.
4.2 COMPARISONS ON TEMPORAL AND CAUSAL REASONING
We compare our DCL with previous methods on CLEVRER, including Memory (Fan et al., 2019), IEP (Johnson et al., 2017b), TbD-net (Mascharka et al., 2018), TVQA+ (Lei et al., 2018), NS-

---

Published as a conference paper at ICLR 2021
Static Attributes
Dynamic Attributes
Events
Methods
Color
Shape
Material
Moving
Stationary
In
Out
Collision
DCL
99.7
99.2
99.6
89.7
93.3
99.2
98.9
96.9
Table 2: Evaluation of vidco concept learning on the validation set.
DR (Yi et al., 2020), MAC (V) (Hudson & Manning, 2018) and its attribute-aware variant, MAC (V+). We refer interested readers to CLEVRER (Yi et al., 2020) for more details. Additionally, we also include a recent state-of-the-arr VQA model HCRN (Le et al., 2020) for performance comparison, which adopts a conditional relation network for representation and reasoning over videos. To provide more extensive analysis, we introduce DCL-Oracle by adding object attribute and collision supervisions into DCL's training. We summarize their requirement for visual labels and language programs in the second and third columns of Table 1.
According to the results in Table 1, we have the following observations. Although HCRN achieves state-of-the-art performance on human-centric action datasets (Jang et al., 2017; 7; Xu et al., 2017; 2016), it only performs slightly better than Memory and much worse than NS-DR on CLEVRER. We believe the reason is that HCRN mainly focuses on motion modeling across frames while CLEVRER requires models to perform dynamic visual reasoning on videos and analyze its temporal and causal structures. NS-DR performs best among all the baseline models, showing the power of combining symbolic representation with dynamics modeling. Our model achieves the state-of-the-an question answering performance on all kinds of questions even without visual attributes and event labels from simulations during training, showing its effectiveness and label-efficiency. Compared with NS-DR, model achieves more significant gains on predictive and counterfactual questions than that on the descriptive questions. This shows DCL's s effectiveness in modeling for temporal and causal reasoning. Unlike NS-DR, which directly predicts collision event labels with its dynamic model, DCL quantizes concepts and executes symbolic programs in an end-to-end training manner, leading to better predictions for dynamic concepts. DCL--racle shows the upper-bount bound performance of the proposed model to ground physical object and event concepts through question answering.
4.3 EVALUATION OF OBJECT AND EVENT CONCEPT GROUNDING IN VIDEOS
Previous methods like MAC (V) and TbD-net (V) did not learn explicit concepts during training, and NS-DR required intrinsic attribute and event labels as input. Instead, DCL can directly quantize video concepts, including static visual attributes (i.e. color, shape, and material), dynamic attributes (i.e. moving and stationary) and events (i.e. in, out, and collision). Specifically, DCL quantizes the concepts by mapping the latent object features into the concept space by linear transformation and calculating their cosine similarities with the concept embeddings in the neubol-symbolic executor.
We predict the static attributes of each object by averaging the visual object features at each sampled frame. We regard an object to be moving if it moves at any frame, and otherwise stationary. We consider there is collision happening between a pair of objects if they collide at any frame of the video. We get the ground-truth labels from the provided video annotation and report the accuracy in table 2 on the validation set.
We observe that DCL can learn to recognize different kinds of concepts without explicit concept labels during training. This shows DCL's 's effectiveness to learn object and event concepts through natural question answering. We also find that DCL recognizes static attributes and events better than dynamic attributes. We further find that DCL may misclassify objects to be "stationary" if they are missing for most frames and only move slowly at specific frames. We suspect the reason is that we only learn the dynamic attributes through question answering and question answering pairs for such slow-moving objects rarely appear in the training set.
4.4 GENERALIZATION
We further apply DCL to two new applications, including CLEVRER-Grounding spatio-temporal localization of objects or events in a video, and GLEVRER-Retrieee finding semantic-related videos for the query expressions and vice versa.
We first build datasets for video grounding and video-text retrieval by synthesizing language ex- pressions for videos in CLEVRER. We generate the expressions by filling visual contents from the video annotations into a set of pre-defined templates. For example, given the text template, "The <static attribute that is <dynamic attribute> <time identifier>": we can fill it and generate "The
7

---

Published as a conference paper at ICLR 2021
Query: The collision that happens after the blue sphere exits the scene.
Query: A video that contains a gray metal cube that enters the scene.
Appl: CLEVRER CLEVRER-Grounding
App2: CLEVRER -LEVRER--etrrival.
Retrieval.
Examples of CLEVRER-Grounding Grounding and CLEVRER-Retrieval Datasets. The target region are marked by purple boxes and stroboscopic imaging is applied for visualization purposes. In LLEVRER-Reeriewe mark randomly-selected positive and negative gallery videos with green and red borders, respectively.
Figure 3:
Text-to- Videot
Spatial Acc.f
Spatial mloU.1
Frame Diff.
Methods
Video-to-Texti
Methods
Obj.
Out Col.
Obj.
Col.
Obj.
Col.
In Out Col.
WSSTG 2.2 1.3 3.4 3.3
7.7
WSSTG 34.4
10.3
34.9
15.6
21.4
50.4
37.5 4,4 4.5
HGR
16.9 17.2 22.2
15.5
DCL
91.9
88.3
90.0
79.0
5.5
DCL
73.1 81.9 88.5 85.4
78.6
Table 3: Evaluation of video grounding. For spatial grounding, we consider it to be accurate if the loU between detected trajectory and the ground-truth trajectory is greater than 0.5.
Table 4: Evaluation of CLEVRER -ettrvlll Mean average precision (mAP) is adopted the metric.
metal cube that is moving when the video ends.". Fig. shows examples for the generated datasets, and provide statistics and examples in Appendix G. We transform the grounding and retrieval expressions into executable programs by training new language parsers on the expressions of the synthetic training set. To provide more extensive comparisons, we adopt the representative video grounding/ retrieval model WSSTG (Chen et al., 2019) as baseline. We provide more details of the baseline implementation in Appendix A.
SLEVRER-Groundin contains object grounding and event grounding. For video object grounding, we localize each described object's 's whole trajectory and compute the mean intersection over union (IoU) with the ground-truth trajectory. For event grounding, including collision, in and out, we temporally localize the frame that the event happens at and calculate the frame difference with the ground-truth frames. For collision event, we also spatially localize the collided objects' the union box and compute it's IoU with the ground-truth. We don't It perform spatial localization for in and out events since the target object usually appears be too small localize at the frame it enters or leaves the scene.
CLEVRER SNEER-Grounding
Table lists the results. From the table, we can find that our proposed DCL transforms to the new CLEVRER-Grone frame differences for temporal localization. On the contrary, the traditional video grounding method WSSTG performs much worse, since it mainly aligns simple visual concepts between text and images and has difficulties in modeling temporal structures and understanding the complex logic.
task well and achieves high accuracy for spatial localization and low
For CLEVRER- Retrrieeval, expression-video pair is considered positive pair if the video contains the objects and events described by the expression and otherwise negative. Given a video, we define its matching similarity with the query expression be the maximal similarity between the query expression and all the object or event proposals. Additionally, also introduce recent state-of-the-art video-text retrieval model HGR (Chen et al., 2020) for performance comparison, which decomposes video-text matching into global-to-local levels and performs cross- cross-modal matching with attention-bau based graph reasoning. We densely compare every possible expression-vide pair and use mean average precision (mAP) as the retrieval metric.
LLEVERR-Rerievva
We report the retrieval mAP Table Compared with CLEVRER- SeNVREERGrounding.. CLEVRER-Retriee is more challenging since it contains many more distracting objects, events and expressions. WSSTG performs worse on the retrieval setting because it does not model temporal structures and understand its logic. HGR achieves better performance than the previous baseline WSSTG since it performs hierarchical modeling for events, actions and entities. However, it performs worse than DCL since doesn't 't explicitly model the temporal structures and the complex logic behind the video-text pairs in CLEVRER Retrieell. On the other hand, DCL is much more robust since it can explicitly ground object and event concepts, analyze their relations and perform step-by-step visual reasoning.
8

---

Published as a conference paper at ICLR 2021
Question Type
Methods
Average
Method
Static Color
Dynamic "falling"
Query
Exist
Count
DCL (ours)
98.5
91.8
MAC (V)
92.8
95.5
75,0
87.7
Table 6: Evaluation of concept learning on the block tower dataset. Our DCL can learn to quantize the new concept "falling" real videos through QA.
DCL (ours) 97.0
95.5
84.1
92.6
Table 5: QA results on the block tower datasct.
Q1: How many objects are falling? A1: 2.
Q1: What is the color of the block that is at the bottom? A1: Blue.
Q2: Are there any falling red objects? A2: No.
Q3: Are there any falling blue objects? A3: Yes.
Q2: Are there any falling yellow objects? A2: No.
Falling block tower
Stable block tower.
Figure 4: Typical videos and question-answer pairs of the block tower dataset. Stroboscopic imaging is applied for motion visualization.
4.5 EXTENSION TO REAL VIDEOS AND THE NEW CONCEPT
We further conduct experiments on real block tower video dataset (Lerer et al., 2016) to learn the new physical concept "falling". The block tower dataset has 493 videos and each video contains stable or falling block tower. Since the original dataset aims study physical intuition and doesn't contain question-ansse pairs, we manually synthesize question-answe answer pairs in a similar way to CLEVRER (Yi et al., 2020). We show examples of the new dataset in Fig 4. We train models on randomly-selected 393 videos and their associated question-answer pairs and evaluate their performance on the rest 100 videos.
Similar to the setting in CLEVRER, we use the average visual feature from ResNet-34 for static attribute prediction and temporal sequence feature for the prediction of the new dynamic concept "falling". Additionally, we train a visual reasoning baseline MAC (V) (Hudson & Manning, 2018) for performance comparison. Table 5 lists the results. Our model achieves better question-ansee answering performance the block tower dataset especially are falling?'' We believe the reason is that counting questions require a model to estimate the states of each object. MAC (V) just simply adopts MLP classifier predict each answer's probability and doesn't 't model the object states. Differently. DCL answers the counting questions by accumulating the probabilities of each object and more transparent and accurate. We also show the accuracy of color and "falling" concepl prediction on the validation set in Table 6. Our DCL can naturally learn to ground the new dynamic concept "falling" in the real videos through question answering. This shows DCL's 's effectiveness and strong generalization capacity.
the counting questions like "How many objects
DISCUSSION AND FUTURE WORK
We present a unified neural symbolic framework, named Dynamic Concept Learner (DCL), to study temporal and causal reasoning videos. DCL, learned by watching videos and reading question- answers, is able to track objects across different frames, ground physical object and event concepts, understand the causal relationship. make future and counterfactual predictions and combine all these abilities to perform temporal and causal reasoning. DCL achieves state-of-the-arr performance on the video reasoning benchmark CLEVRER. Based on the learned object and event concepts, DCL generalizes well to spatial-tempon object and event grounding and video-text retrieval. We also extend DCL to real videos to learn new physical concepts.
Our DCL suggests several future research directions. First, it still requires further exploration for dynamic models with stronger long-term dynamic prediction capability to handle some counterfactual questions. Second, it will be interesting extend our DCL to general videos to build stronger model for learning both physical concepts and human-centric action concepts.
Ack Acknowledgement This work is in part supported by ONR MURI N00014-16-1-2000 the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF- 1231112)),)), the Samsung Global Research Outreach (GRO) Program, Autodesk, and IBM Research.

---

Published as a conference paper at ICLR 2021
REFERENCES
Saeed Amizadeh, Hamid Palangi, Oleksandr Polozov, Yichen Huang, and Kazuhito Koishida. Neuro-symbolic visual reasoning: Disentangling" visual" from" reasoning". In ICMI, 2020. 2
Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Dan Klein. Neural module networks. In CVPR, 2016.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. In ICLR, 2015. 5,
Anton Bakhtin. Laurens der Maaten, Justin Johnson, Laura Gustafson, and Ross Girshick. Phyre: A new benchmark for physical reasoning. In NeurIPS, 2019.
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counterfactual learning of physical dynamics. In ICLR, 2020. 3,6 6
Peter W Battaglia, Jessica B Hamrick. and Joshua B Tenenbaum. Simulation as an engine of physical scene understanding. Proceedings of the National Academy Sciencex, 2013. 3
Alex Bewley, Zongyuan Ge, Lionel Ott, Fabio Ramos, and Ben Uperoft. Simple online and realtime tracking. In ICIP, 2016. 15, 16
Shizhe Chen, Yida Zhao, Qin Jin, and Qi Wu. Fine-grained video-text retrieval with hierarchical graph reasoning. In CVPR, 2020. 8, 13
Zhenfang Chen, Lin Ma, Wenhan Luo, and Kwan- Yee Kenneth Wong. Weakly-supervised seatio-temporall grounding natural sentence in video. In ACL, 2019. 8, 13
Chenyou Fan, Xiaofan Zhang, Shu Zhang. Wensheng Wang, Chi Zhang, and Heng Huang. Heterogeneous memory enhanced multimodal attention model for video question answering. In CVPR, 2019. 1, 3,
Chelsea Finn, Goodfellow, and Sergey Levine. Unsupervised learning for physical interaction through video prediction. NeurlPS, 2016. 3
Amy Fire and Song-Chun Zhu. Learning perceptual causality from video. ACM Transactions on Intelligent Systems and Technology, 2015.
Chuang Gan, Naiyan Wang, Yi Yang. Dit-Yan Yeung, and Alex G Hauptmann. Devnet: A deep event network for multimedia event detection and evidence recounting. In CVPR, pp. 2568-2577, 2015. 3
Chuang Gan, Yandong Li, Haoxiang Li, Chen Sun, and Boging Gong. Vqs: Linking segmentations to questions and answers for supervised attention in vqa and question-focusee semantic segmentation. In ICCV, pp. 1811-1820, 2017.
Chuang Gan, Jeremy Schwartz, Seth Alter, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, al. Threedworld: A platform for interactive multi-modal physical simulation. arXiv preprint MXXi::007700455. 2020. 3
Rohit Girdhar and Deva Ramanan. Cater: diagnostic dataset for compositional actions and temporal reasoning. In ICLR, 2020. 3,6
Georgia Gkioxari and Jitendra Malik. Finding action tubes. In CVPR, 2015. 4, 1.5
Alex Graves, Santiago Fernández, and Jürgen Schmidhuber. Bidirectional Istm networks for improved phoneme classification and recognition. In ICANN, 2005. 14
Kaiming Hc, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 13
Ronghang Hu. Jacob Andreas, Trevor Darrell, and Kate Saenko. Explainable neural computation via stack neural module networks. In ECCV, 2018.
Deng Huang. Peihao Chen, Runhao Zeng, Qing Du, Mingkui Tan, and Chuang Gan. Location aware graph convolutional networks for video question answering. In AAAI, volume 34, pp. 11021-11028, 2020.
Drew Hudson and Christopher Manning. Learning by abstraction: The neural state machine. In NeurIPS. 2019.
Drew A Hudson and Christopher D Manning. Compositional attention networks for machine reasoning. In ICLR, 2018. 2,7,9
10

---

Published as a conference paper at ICLR 2021
Yunseok Jang, Yale Song. Youngjae Yu, Youngjin Kim, and Gunhee Kim. Tgif-qa: Toward spatio-temporal reasoning in visual question answering. In CVPR, 2017.
Jingwei Ji, Ranjay Krishna, Li Fei-Fei, and Juan Carlos Niebles. Action genome: Actions compositions of spatio-temporal scene graphs. CVPR, 2020.
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. CVPR, 2017a.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Judy Hoffman, Fei-Fei. Lawrence Zitnick, and Ross Girshick. Inferring and executing programs for visual reasoning. In ICCV, 2017b. 1,6
Rudolph Emil Kalman, A new approach linear filtering and prediction problems. Basic Eng., Trans. ASME, D, 82:35 45, 1960. 15
Diederik P Kingma and Jimmy Ba, Adam: A method for stochastic optimization. arXiv: 1412.9980, 2014, 13
Thao Minh Le, Vuong Le, Svetha Venkatesh, and Truyen Tran. Hierarchical conditional relation networks for video question answering. In CVPR, 2020. 7, 13
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara Berg. Tvqa: Localized, compositional video question answering. In EMNLP, 2018. 1,
Adam Lerer, Sam Gross, and Rob Fergus. Learning physical intuition of block towers by example. In ICML,, 2016. 3, 6,9
Qing Li, Siyuan Huang, Yining Hong, Yixin Chen, Ying Nian Wu, and Song-Chun Zhu, Closed loop neural- symbolic learning via integrating neural perception, grammar parsing, and symbolic reasoning, In ICML, 2020. 3
Xiangpeng Li. Jingkuan Song, Lianli Gao, Xianglong Liu, Wenbing Huang, Xiangnan He, and Chuang Gan. Beyond rass: Positional self-attention with co-attention for video question answering. In AAAI, volume 33, pp. 8658-866668 8665, 2019a. 3
Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation networks for model-based control under partial observation. ICRA, 2019b. 3.5
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua Tenenbaum, and Jiajun Wu. The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision. ICLR, 2019. 3,5
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by design: Closing the gap between performance and interpretability in visual reasoning. In CVPR, 2018. 6
Joanna Materzynska, Tete Xiao, Roei Herzig, Huijuan Xu, Xiaolong Wang, and Trevor Darrell. Something-else: Compositional action recognition with spatial-temporal interaction networks. In CVPR, 2020. 3
Roozheh Mottaghi, Mohammad Rastegari, Abhinav Gupta, and Ali Farhadi, "what happens if..." learning to predict the effect of forces in images. In ECCV. Springer, 2016. 3
Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In NeurlPS, 2015. 4
Ronan Riochet, Mario Ynocente Castro, Mathieu Bernard, Adam Lerer, Rob Fergus, Véronique Izard, and Emmanuel Dupoux. Iniphys: A framework and benchmark for visual intuitive physics reasoning. arXiv: 1803.07666 2018.
Tianjia Shao, Aron Monszpart, Youyi Zheng, Bongjin Koo, Weiwei Xu, Kun Zhou, and Niloy Mitra. Imagining the unseen: Stability- based cuboid arrangements for seene understanding. ACM TOG, 2014. 3
Makarand Tapaswi, Yukun Zhu, Rainer Sticfelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: Understanding Stories in Movies through Question-Answering. In CVPR, 2016.
Xiaolong Wang and Abhinav Gupta. Videos space-time region graphs. In ECCV, 2018. 3
Nicolai Wojke, Alex Bewley, and Dietrich Paulus. Simple online and realtime tracking with a deep association
metric. In ICIP, 2017. 15, 16
Qi Wu, Chunbua Shen, Linggiao Liu, Anthony Dick, and Anton van den Hengel. What value do explicit high level concepts have in vision language problems? In CVPR, 2016. 2
11

---

Published as a conference paper at ICLR 2021
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. ACM MM, 2017. 7,
Jun Xu, Tao Mei, Ting Yao. and Yong Rui. Msr-vtt: large video description dataset for bridging video and language. In CVPR, pp- 9288-5296, 2016.
Tian Ye, Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. In ECCV, 2018.3 3
Yuman Ye. Zhou Zhao, Yimeng Li. Long Chen, Jun Xiao, and Yueting Zhuang. Video question answering via attribute- aumbbettd attention network learning. In ICLR, 2017.
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Josh Tenenbaum. Neural-symbolic
vqa: Disentangling reasoning from vision and language understanding. In NeurIPS, 2018. 3,
Kexin Yi, Chuang Gan, Yunzhu Li, Pushmeet Kohli, Jiajun Wu, Antonio Torralba, and Joshua B Tenenbaum.
Clevrer: Collision events for video representation and reasoning. ICLR, 2020. 1, 3, 5, 6, 7, 14
Luowei Zhou, Chenliang Xu, and Jason J Corso. Towards automatic learning of procedures from web instruc- tional videos. AAAI, 2018.
Luowei Zhou, Yannis Kalantidis, Xinlei Chen, Jason J Corso, and Marcus Rohrbach. Grounded video description. CVPR, 2019.
Yuke Zhu Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. CVPR, 2016."
12

---

Published as a conference paper at ICLR 2021
A IMPLEMENTATION DETAILS
DCL Implementation Details. Since it's extremely computation-intenstt to predict the object states and events at every frame, evenly sample 32 frames for each video. All models are trained using Adam (Kingma & Ba, 2014) for 20 epochs and the learning rate is set to 10- We adopt a two-stage training strategy for training the dynamic predictor. For the dynamic predictor, we set the time window size w, the propogation step L and dimension of hidden states to be 3, 2 and 512, respectively. Following the sample rate at the observed frames, we sample a frame for prediction every 4 frames. We first train the dynamic model with only location prediction and then train it with both location and RGB patch prediction. Experimentally, we find this training strategy provides a more stable prediction. We train the language parser with the same training strategy as Yi et al. (201 8) for fair comparison.
Baseline Implementation. We implement the baselines HCRN (Le et al., 2020), HGR Chen et al. (2020) and WSSTG (Chen et al., 2019) carefully based on the public source code. WSSTG first generate a set of object or event candidates and match them with the query sentence. We choose the proposal candidate with the best similarity as the grounding result. For object grounding, we use the same tube trajectory candidates we use for implementing DCL. For grounding event concepts in and out, we treat each object at each sampled frame as a potential candidate for selection. For grounding event concept collision, we treat the union regions of any object pairs as candidates. For CLEvRER-Retrieval, we treat the proposal candidate with the best similarity as the similarity score between the video and the query sentence. We train WSSTG with a synthetic training set generated from the videos of CLEVRER-VE -QA training set. A fully-supervised triplet loss is adopted to optimize the model.
B FEATURE EXTRACTION
We evenly sample K frames for each video and use a ResNet-34 (He al., 2016) to extract visual features. For the n-th object the video, we define its average visual feature to be f" = K Σx= Sk. where f is the concatenation of the regional feature and the global context feature frame. We define its temporal sequence feature f to be the contenation of [x?, .2"" wo, hT] at all T frames, where (xf", yt') denotes the normalized object coordinate centre and w/ and hy denote the normalized width and height, respectively. For the collision feature between the n1-th object and n2-th objet at the k-th frame, we define it to be f2 & f",, i where f 22.k is the ResNet feature of the union region of the 1ı-th and n objects at the k-th frame and plac my,k is a spatial embedding for correlations between bounding box trajectories. We define
the k-th
flee mai. loU(sn) 8xa (s 8xz * X 8mg),, which is the concatenation of the intersection over union (IoU), difference (-) and multiplication (x) of the normalized trajectory coordinates for the ni nt-th and n2g objects centering at the k-th frame. We padding fia,n,,, & with a zero vector either the n1 or the n2 objects doesn' It appear at the k-th frame.
C DYNAMIC PREDICTOR
To predict the locations and RGB patches, the dynamic predictor maintains a directed graph (V, D) = (v, J. (dm,, 1 N.NNN /=j=1,m==1/- The n-th vertex on is represented by the concatenation of its normalized coordinates 67 [="' vi, wp, hP] and RGB patches pº. The edge day,ng is represented by the concatenation of the normalized coordinate difference b," bb2 To capture the object dynamics, we concatenate the features over small history window. To predict the dynamics the k ++ 1 frame, we first encode the vertexes and edges
c (1 (b7 (ll"))
e = (11 bb()))
(4)
where indicates concatenation, w is the history window size, f and f are CNN-based encoders for objects and relations. w is set to 3. We then update the object influences ( I and
13

---

Published as a conference paper at ICLR 2021
N.N relation influences (ong, I'j 1,a==1 through L propagation steps. Specifically, we have en ,1g, = fR(eng mg
h mikk Mrak)) by, fo(en,k
Σ mn (111)
(5)
where I E [1, LJ, denoting the I-th step, Jo and ÍR denote the object propagator and relation propagator, respectively. We initialize hy, = O. We finally predict the states of objects and relations at the k+1 frame to be
0 - Son (c7,30 hy,)), ph+ (o k(r,k)
where /pred and 10, are predictors for the normalized object coordinates and RGB patches the next frame. We optimize this dynamic predictor by mimizing the L2 distance between the predicted bg pE+, and the real future locations b and extracted patches Pk++1
During inference, the dynamics predictor predicts the locations and patches at k+1 frames by using the features of the last observed frames in the original video. We get the predictions at the k+2 frames by feeding the predicted results at the k+1 frame to the encoder in Eq. 4. To get the counterfactual scenes where the n-thh object is removed, we use the first frames of the original video the start point and remove the n-th vertex and its associated edges of the input to predict counterfactual ,. dynamics. Iteratively, we get the predicted normalized coordinates (6), Jni,k'' and RGB patches N.K (i% I==1,k=1 at all predicted K" frames.
D PROGRAM PARSER
Following Yi et al. (2020), we use seq2seq model (Bahdanau et al., 2015) with attention mechanism to word sequences into a set of symbolic programs and treat questions and choices, separately. The model consists of Bi-LSTTM (Graves et al., 2005) to encode the word sequences into hidden states and a decoder to attentively aggregate the important words to decode the target program. Specifically, to encode the word uubeddings((::e into the hidden states, we have
2 R4 LSTM( (w;), h;-1), E h (FSM((F""""))) T.+1),
(7)
where I is the number of words and fore is an encoder for word embeddings. To decode the encoded vectors (e: (ei) into symbolic programs {pン} we have
explo e))
9 LSTM(Je done (p,-1)), as E. caplafa))
PS softmax(W
(g: Easee)))
(8)
where 6; क द and is the number of programs, The dimension of the word embedding and all the hidden states is set to 300 and 256, respectively.
E CLEVRER OPERATIONS AND PROGRAM EXECUTION
We list all the available data types and operations for CLEVRER VQA (Yi et al., 2020) in Table and Table 7. In this section, we first introduce how we represent the objects, events and moments in the video. Then, we describe how we quantize the static and dynamic concepts and perform temporal and causal reasoning. Finally, summarize the detailed implementation of all operations in Table 9.
Representation for Objects, Events and Time. We consider a video with N objects and T frames and we sample K frames for collision prediction. The objects in Table 8 can be represented by vector objects s of length N, where objects, [0, 1] represents the probability of the n-h object being referred to. Similarly, we use vector events of length N to representing the probability of objects coming into the visible scene. additionally store frame indexes f for event in, where in indicates the moment when the n-th object first appear in the visual scene. We represent event out in a similar way as we represent event in. For event collision, we represent it with a matrix events col IR NXNNK where event col m.. eventss represents the n1 and n22-th objects collide at the k-th frame. Since CLEVRER requires temporal relations of events, we also maintain time mask M E RT to annotate valid time steps, where M2 = indicates the t-th valid at the current step and M, 0 indicating invalid. In CLEVRER, Uni que also involves transformation from objects (object set) to object (a single object). We achieve by selecting the object with the largest probability. We perform in similar way to transform events event.
14

---

Published as a conference paper at ICLR 2021
Object and Event Concept Quantization. We first introduce how DCL quantizes different con- cepts by showing an example how DCL quantizes the static object concept cube. Let fx denote the latent visual feature for the n-th object in the video, SA denotes the set of all static attributes. The concept cube is represented by a semantic vector 5Cube and an indication vector goube, goube is of length [SA] and L--1 normalized, indicating concept Cube belongs to the static attribute shape. We compute the confidence scores that an object is Cube by
Cube cobee mm (f:)) - 6
poube
(9)
SOESA
where 8 and X denotes the shifting and scaling scalars and are set to 0.15 and 0.2, respectively. cos() calculates the cosine similarity between two vectors and m denotes linear transformation, mapping object features into the concept representation space. We get a vector of length N by applying this concept filter to all objects, denoted as (ebber
We perform similar quantization to temporal dynamic concepts. For event in and out, we simply replace fü with temporal sequence features fä E g to get events For event collision, we replace fü with fny mn. to predict the confidence that the n1-th and the nz2thh objects collide at the k-frame and get events m2 For moment-specific dynamic concepts moving and stationary, we adopt frame-specific feature fü, E RR" for concept prediction. We denote the filter result on all objects as Obj Filter Filler(mooiing (moving, 1*). Specifically, we generate the sequence feature fū at the ("-th frame by only concatenating [as] ;/' up: hT] from t" T to t" frames and padding other dimensions with 0.
Temporal and causal Reasoning. One unique feature for CLEVRER is that it requires model to reason over temporal and causal structures of the video to get the answer. We handle Filter be fore and Fi lter a afee by updating the valid time mask M. For example, filter events happening after a target event. We first get the frame t that the target event happens at and update valid time mask M by setting M, 1 if t > t* else M2- O. We then ignore the events happening at the invalid frames and update the temporal sequence features to be fú fū o Mezp- where denotes the component multiplication and Mezp [MM; M: M; M] E EAT
For Filter_order of events we first filter all the valid events by find events who cupe > n) nj simply set to 0 and type E {in, out, collision). We then sort all the remain events based on (uye to find the target event.
For Fi 1 ttetta of collision event, we first predict valid events by finding eventStype >> 17- We then return all valid events that are in the causal graphs of the given collision event.
We summarize the implementation of all operations in Table 9.
F TRAJECTORY PERFORMANCE EVALUATION.
In this section, we compare different kinds of methods for generating object trajectory proposals. Creedy+IcU denotes the method used in (Gkioxari & Malik, 2015), which adopts a greedy Viterbi algorithm to generate trajectories based on IoUs of image proposals in connective frames. Greedy+ FreedyyIcU+AAte IcU+Attr. denotes the method adopts the greedy algorithm to generate trajectory proposals based on the IoUs and predicted static attributes. L.SM+ ToU denotes the method that we linear sum assignment to connect the image proposals based on IoUs. LSM+ IoU+Attz> denotes the method we use linear sum assignment connect image proposals based on IoUs and predicted static attributes. LSM+ IoU+At t r +KE denotes the method that we apply additional Kalman filtering (Kalman, 1960; Bewley et al., 2016; Wojke et al., 2017) to L.M+IOUUAtEr.. We evaluate the performance of different methods by compute the IoU between the generated trajectory proposals and the ground-truth trajectories. We consider it a "correct" trajectory proposal the IoU between the proposal and the ground-truth is larger than a threshold. Specifically, two metrics are used evaluation, precision = Numas and recall Norms where Nermect, N, and Ngt denotes the number of correct Ngt proposals, the number of proposals and the number of ground- ground-truth objects, respectively.
N
Table 10 list the performance of different thresholds. We can see that Greedy +IoU achieve bad performance when the loU threshold is high while our method based on linear sum assignment and
15

---

Published as a conference paper at ICLR 2021
static attributes robust. Empirically, find that linear sum assignment and static attributes can help distinguish close object proposals and make the correct image proposal assignments. Similar to normal object tracking algorithms (Bewley al., 2016; Wojke al., 2017), we also find that adding additional Kalman filter can further slightly improve the trajectory quality.
G STATISTICS FOR CLEVRER-GROUNDEE AND CLEVRER-RETRIEVE
We simply use the videos from original CLEVRER training set as the training videos for CLEVRER-Grounding and CLEVRER- Retrieell and evaluate their performance the validation set. CLEVERER- LEVERERR-Grunninn contains 10.2 expressions for each video on average. CLEVERER Rettreevaeeie contains 7.4 expressions for each video in the training set. We for evaluating the video retrieval task on the validation set. We evaluate the performance of CLLEVRER-rrunddie videos from original CLEVRER validation set. For generate 1,129 unique expressions from the validation set as query and treat the first 1,000 videos from CLEVRER validation set as the gallery. We provide more examples for CLEVRER -eVoEiinG and CLEVRER Retrieval datasets in Fig. 5, Fig. 6 and Fig. 7. It can be seen from the examples that the newly proposed CLEVRER DeYVEE-Grrunding and CLEVRER-Retriewe datasets contain delicate and compositional expressions for objects and physical events. It can evaluate models' ability to perform compositional temporal and causal reasoning.
task all 5,000 -eerree--eerieall, We additionally
H TRAINING OBJECTIVES
In this section, we provide the explicit training objectives for each module. We optimize the feature extractor and the concept embeddings in the executors by question answering. We treat each option of multiple-choiee question as an independent boolean question during training and we use different loss functions for different question types. Specifically, we use cross-entropy loss supervise open-ended questions and use mean square error loss to supervise counting questions. Formally, for open-ended questions, we have
C Lop.open = - 1(.. = log(pe),
(10)
where the size the pre-defined answer set, Pe is the probability for the e-th answer and yo is the ground-truth answer label. For counting questions, we have
LQA.count (Va - =))²
(11)
where z the predicted number and ya is the ground-truth number label.
We train the program parser with program labels using cross-entropy loss,
Eprogram
Σ [up j log(p,);
(12)
where J is the size of the pre-defined program set, Pj is the probability for the j-th program and yp is the ground-truth program label.
We optimize the dynamic predictor with mean square error loss. Mathematically, we have
Ldynamic Σ b 8"
1 Pl 11
(13)
where 6d is the object coordinates for the n-th object, Ph; is the pixel value of the n-th object's cropped patch (11, and Np is the cropped size. 6" and ph are the dynamic predictor's predictions for and ph
16

---

Published as a conference paper at ICLR 2021
Type
Operation
Signature
Cb Cbjctt
objects
Returns all objects in the video
Events
0 events
Returns all events happening in the video
Input
UnseenEvents
Modules
Returns all future events happening the video Start
Returns the special "start" event end
event
Returns the special "end" event
Filler.s c.sssa paaaiia Lic ccconnepl
(onjects,concep)) - objects
Object
Select objects from the imput list with the input static concept
Filter
F 1ll welter_diee setterrynaatt c.concept
(ohjects, cracept objects
Modules
Selects objects in the input frame with the dynamic concept
Filter_in
anvents,objectts - events
Select incoming events of the input objects
Filter.our
(events, objects) + events
Select existing events of the input objects
Filter_col lel_collll s on
(events, objects) + events
Select all collisions that involve of the imput objects
Get col partner
(event, object) + object
Return the collision partner of the input object
Event Filter
welter.befor
(events, events)
Select all events before the target event
Modules
after
(events, events
Select all after the target
Fill ter_oddrr
(events, order) event
Select event the specific time order
Filter, Filter_ances ancestor
eventss) - events
all ancestors the input event the causal graph
Cet-frane
(event) + frame
Return the frame of the input event in the video
Query Attribute
(object) concept
Returns the query attribute of the input object Count
(objects) - int
Returns the number of input objects/ events Exist
(events) -
Output Modules
(objects) - bool
Returns "yes" inpul objects not emply
Be Dalond lelong_Lo
(event, events) + bool
Returns "yes" if the input event belongs to the input event sets
Negate
(bool) - bool
Returns the negation of the input boolean
Unique
(events) event (objects)
Return the only event /object in input list
object
Table 7: Operations available on CLEVRER dataset.
Type
Semantics
object
A single object in the video.
objects
A set of objects in the video.
event
A single event in the video.
events
A set of events in the video.
order
The chronological order of event, e.g. "First", "Second" and "Last". Object-level static concepts like "Red", "Sphere" and "Mental".
static concept
dynamic concept
Object-level dynamic concepts like "Moving" and "Stationary".
attribute
Static attributes including "Color", "Shape" and "Material".
frame
The frame number of an event.
int
A single integer like "0" and "1".
bool
A single boolean value, "True" or "False".
Table 8: The data type system of CLEVRER-VQA.
17

---

Published as a conference paper at ICLR 2021
Type
Operation/ Signature
Implementation
Objects
objects
- objects
Events
events for type E {in, out, col.)
+ events
Input Modules
UnseenEvents
eventy and events
+ events
Start
if 5 clse Mr
- M
end M
M, 1 if t (T 5) else M, =
Filter static concept
min (injs (bjjilses,,)))lle((ssssssa))))))
Object Filter Modules
(objs: objects, sa: concept) objects
Filter dynamic concept
(objs: objects, da: concept, t: frame) + objects
Filter-in
min min(objs, events(")
(events"" events, objs: objects) - events
Filter.out
min(objs, (objsseeenntssu
(events""C : events, ohjs: objects) - events
Filter collisio
min(objs events
(events evennts,, objs: objects) + events
pet.coer
maxic[1.x][evennt)
Event Filter
(events events, objn object) -> objects
Filter.before
events if t geventt
Modules
(events events, eventl: event)
Filter.after
events, if +in geventl
(cvents pvents,eventie event) - events Filter.order
events R
0 if order
(events events, or: order) + event
Pilter.ancestor.
[eventl,, 0 and eventsl, in the causal graph of eventl geventl
(eventl: event, eventsl; events) events
Get frame
(eventl: event) >> frame
Query Attribute
pop
Oby Pddter(op) SP Dnd ObjFitter(op')
(obj: object, altribute) - concept
Count
(objs, > >0)
(objs: objects) + int
Exist
(objs, > 0)) >
(objs: objects) - bool
Output Modules
Belong.to
True if eventl E events else False Falsc if bl clsc True
(eventl; event, eventsl: events) + bool
Negate
(bl: bool) bool
Table 9; Neural operations in DCL. eventsoal denotes the collision events happening the unseen future frames. "P E N x K and objs cop " .us.k max(objsng obj$m2). events oot denotes the collision events that the 1-th object get involved at the kitth frame.
x
0.5
0.6
0.7
0.8
0.9
pree.
recall
ргсc.
recall
prec.
recall
pгee.
recall
prec.
recall
Gre
87.2
88.3
71.4
72.3
57.3
58.0
46.9
47.5
39,6
40.0
Groody+loU+Atr..
97.0
97.4
95.1
95.5
93.1
93.5
89.9
90.3
83.6
83.9
LSM+
97.6
98.8
96.9
98.1
96.0
97.2
93.8
95.0
88.5
89.6
LSM+HoU+Atrr
99.1
98.4
98.6
97.9
97.9
97.2
96.2
95.5
91.5
90.8
AFF
99.1
98.4
98.6
97.9
98.0
97.3
96.2
95.6
91.6
90.9
Table 10: The evaluation of different methods for object trajectory generation.
18

---

Published as a conference paper at ICLR 2021
Query: The collision that happens before the gray object enters the scene.
Query: The green object enters the before the rubber sphere enters the scene
Query: The cube exits the scene after the sphere enters the scene
Query:
The metal cylinder that is stationary when the sphere enters the scene
Figure 5: Typical examples of CLEVRER-Grounding datasets. The target regions bounded with purple boxes.
19

---

Published as a conference paper at ICLR 2021
Query: A video that contains a collision that happens after the yellow metal cylinder enters the scene.
Positive video sample (a)
Positive video sample (b)
Positive video sample (c)
Positive video sample (d)
Figure A exemplar query expression and of its associated positive vidcos from CLEVRER-Retrioval datasct. The target regions in vidcos are bounded with purple boxes.
1. A video that contains object that collides with the brown metal cube.
video that contains an object that collides with the gray metal sphere.
3. A video that contains object to collide with the brown metal cube.
video that contains an object to collide with the gray metal sphere.
5. A video that contains collision that happens after the yellow metal cube enters the scene.
A video that contains a collision that happens after the brown metal cube enters the scene.
7. video that contains collision that happens before the yellow metal cube enters the scene.
Figure 7: An typical query video and its associated positive expressions from CLEVRER- dataset.
20


Reviews:
[{"review_id": "qPHZB0LrulM", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Pros\n1.\tThis paper extends the semantic modular network on images to the video setting.\n2.\tBy making use of the weak supervision from the question answer, it can learn different kinds of concepts including physical objects, attributes and events. \n3.\tThe paper conduct experiments on different evaluation settings including video causal reasoning, video grounding and generalization experiments.\nCons\n1.\tThe paper lacks some details on the training procedure.\n2.\tIt’s better to show the learned concepts visually as one of the strengthens of is explain ability.\n3.\tHow’s the model’s sample complexity\n\nComments\n1.\tIs the trajectory refinement and model learning an iterative process?\n2.\tWhat does the RGB patches mean and how do you combine the RGB patches into the predicted locations in the dynamic predictions?\n3.\tIs this paper using some pre-trained object recognizer and attribute classifier which acts as some kind of pseudo labeling?\n4.\tIn the generalization experiments, does the model trained and test using the same domain dataset?\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "p6m6x7IUmVW", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThe paper presents a neural-symbolic approach for video question answering. In particular, the authors propose the Dynamic Concept Learner (DCL), a network architecture to localize object trajectories over time and further build a graph network to model the physical interaction between objects and events. Finally, a standard symbolic executor is used to reason across visual and language cues for question answering. The proposed method is evaluated on CLEVRER dataset in which it achieves new state of the art performance.\n\nPros:\n\n(1) The main contribution of this paper is that it is able to ground object and event concepts to model the dynamic scene of the interactions between objects/events without the use of extra supervision. This is well motivated as we do not always have access to explicit labels such as object attributes, masks, and collision events in real-world applications.\n\n(2) The idea of dynamics prediction to support predictive and counterfactual questions is new. This makes the proposed framework as a whole interesting IMHO.\n\n(3) The proposed method sets new state of the art performance on CLEVRER dataset in all question types. On video grounding and video-text retrieval tasks, DCL seems to achieve interesting results.\n\nCons:\n\nDespite the pros above, I have some comments on the paper:\n\n(1) I found the paper is quite hard to follow. Figure 2 and its description represent the process of DCL (object trajectory detector -> dynamic predictor -> feature extractor -> symbolic executor). However, the multi-step training paradigm in Sec. 1, 3.1, and 3.2 seems to follow a different structure. In addition, some details are missing in the paper. For example, in the ‘Grounding Object and Event Concepts’ subsection, where did you get the vector embedding for the concept \"moving\" (s_moving) from?\n\n(2) The motivation in the subsection ‘Trajectory refinement’ is exactly as what explained in DeepSORT [1] – a common tracking algorithm. In addition, I believe the use of Kalman Filters in DeepSORT makes it better in handling occlusion than the Eq. 2. I suggest the author discuss the object tracking literature for better judgments. \n\n(3) One of the main weaknesses of this paper is they only evaluate their method on one synthetic dataset which is CLEVRER. It seems to me that the proposed method is particularly designed to solve this dataset. While I find the results interesting, I think physical interactions in those synthetic data are too simple and do not reflect much what happens in real-world data. Not to mention the limit in terms of linguistic variants of those synthetic datasets compared to human natural language. I would appreciate if the author can evaluate the proposed model on other datasets such as CATER [2] for temporal reasoning and Video QA datasets (TGIF-QA [3], TVQA [4], etc).\n\n(4) Similar to my concerns in (3), it would be more convincing to see if the DCL works on video grounding datasets such as ActivityNet-Entities or YouCook2, MSR-VTT for video-text retrieval to support their claims on the generalization of the method. \n\nI am willing to raise the rating if the author could properly address my concerns.\n\nReferences:\n- [1] Wojke, Nicolai, Alex Bewley, and Dietrich Paulus. \"Simple online and realtime tracking with a deep association metric.\" 2017 IEEE international conference on image processing (ICIP). IEEE, 2017.\n- [2] Girdhar, Rohit, and Deva Ramanan. \"Cater: A diagnostic dataset for compositional actions and temporal reasoning.\" arXiv preprint arXiv:1910.04744 (2019).\n- [3] Jang, Yunseok, et al. \"Tgif-qa: Toward spatio-temporal reasoning in visual question answering.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2017.\n- [4] Lei, J., Yu, L., Bansal, M., & Berg, T. L. (2018). Tvqa: Localized, compositional video question answering. arXiv preprint arXiv:1809.01696.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "lTqMTgNROIR", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "=Summary\nThe paper proposes a new framework, Dynamic Concept Learner (DCL), which learns by watching videos and reading questions/answers.  It is inspired by prior work which combines symbolic representations with video dynamics modeling, but unlike prior work, here the authors do not use any additional supervision except for question/answer pairs. DCL is quiet complex, involving multiple components, including program parser (transforms the question/answer into executable symbolic programs), object detection&tracking, object-centric feature extractor (with static and dynamic components), object/event concept embeddings (share the same latent space as the visual features), trajectory refinement (based on static cues) and dynamic predictor (for future and counterfactual scenes), and, finally, symbolic executor (to predict whether the answer is correct), which runs on the latent features. DCL achieves state-of-the-art results on the CLEVRER dataset. The learned representations are also shown to be useful for other tasks, such as grounding and retrieval.\n\n=Strengths\n\nThe paper is overall well written and the approach is novel to the best of my knowledge.\n\nDCL achieves state-of-the-art results on the CLEVRER dataset, w/o accessing visual labels from simulation during training.\nThe biggest improvement is notably achieved on the predictive and counterfactual questions.\n\nThe authors also show that DCL learns to ground object/event concepts w/o explicit labels.\n\nFinally, the authors construct CLEVRER-Grounding and CLEVRER-Retrieval datasets to test the generalization ability of DCL. They show that it significantly improves over a baseline on both tasks. \n\n=Weaknesses/High-level comments\n\nCounterfactual questions remain particularly hard to answer, do the authors have some intuition as to what may be missing in their model? What are some common failure modes?\n\nTable 1: it is somewhat surprising that the difference between the DCL and DCL-Oracle is so small, and DCL even improves over DCL-Oracle in one case. Why is that?\n\nWhat do the authors believe is the main limitation of their proposed approach? Currently it operates on synthetic images as well as synthetic language, what will be necessary to move to more realistic domains?\n\nNothing was stated about making the code available.\n\n=Minor comments (P# - page number)\n- P1 that what the concepts => what the concepts\n- P3, Sec 3: DYNAMICS CONCEPT LEARNER => DYNAMIC CONCEPT LEARNER\n- P4: an program parser => a program parser\n- The paper heavily relies on the supplemental material.", "extra_scores": {}, "extra_text": {}}, {"review_id": "VtiRgTG8etn", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "> Summary:\n\nThe paper studies the problem of dynamic visual reasoning on the recently proposed synthetic video QA dataset CLEVRER for understanding visual properties, physical events, the causal relationship between events, and making future and counterfactual predictions. The paper proposes a joint framework called Dynamic Concept Learner which contains five modules (object trajectory detector, video feature extractor, dynamic predictor, language program parser, and neural symbolic executor) and follows a multi-step training paradigm to train the model purely from the question-answer pairs in CLEVRER train split.  The object trajectory detector module, the concept embedding/quantization, and the dynamic predictor module play a very important role in DCL. These modules enable the dynamic & causal reasoning abilities for DCL. The neural program parsing & execution part of DCL is similar to that in Neuro-Symbolic Concept Learner (NS-CL) (Mao et al., 2019) and thus DCL can be regarded as the dynamic extended version of  NS-CL.\n\n> Strengths:\n\n*  The paper is well-written and easy to follow.  A lot of details about the DCL model, neural programs, and dataset are provided in the appendix.\n\n* The experiment on CLEVRER shows the effectiveness of DCL over previous methods.  Additional experiments on new CLEVRER Grounding & Retrieval applications demonstrate the generalization ability of DCL trained with the original VQA task on other CLEVRER related tasks.\n\n* DCL is trained directly from QA pairs without additional labels for object attributes and events but still outperforms the previous methods, which is a big strength.\n\n* During inference, DCL can acquire attribute/event concept perception, object trajectory prediction at the same time. Once the answer is generated, the whole reasoning path for the answer is also ready, which exhibits the good model interpretability of DCL.\n\n* From *VirTex: Learning Visual Representations from Textual Annotations (Desai et al., 2020)* and *LXMERT: Learning Cross-Modality Encoder Representations from Transformers (Tan et al., 2020)*, we can see that actually textual annotations such captions and question-answer pairs contain high-quality information and are helpful for learning visual or concept representations. I think this paper is another good example of discovering such high-quality information without using additional fine-grained labels.\n\n> Some concerns and suggestions:\n\n- My major concern is that the evaluation is based on a synthetic toy video dataset and it is simplistic compared to real-world videos. Although the author demonstrates that DCL has transferability between different vision-language tasks on CLEVRER, it is still skeptical whether DCL can be generalized to real-world video QA datasets with more complex visual scenes such as TGIF-QA, MSVD-QA, or MSRVTT-QA since the dynamic attributes and events in CLEVRER is only limited to (moving, stationary) and (in, out, collision). If possible, evaluating on more datasets would further demonstrate the effectiveness of DCL across different visual scenes.\n\n* Besides NS-DR baseline which was proposed together with CLEVRER dataset, Memory (Fan et al., 2019) is the most recent visual-language reasoning approach for videos. It is evaluated on TGIF-QA, MSVD-QA, and  MSRVTT-QA in the original paper. I think the author should find some newer approaches evaluated on these datasets to make comparisons on CLEVRER. For instance, [Hierarchical Conditional Relation Networks for Video Question Answering (CVPR 20’)](https://openaccess.thecvf.com/content_CVPR_2020/papers/Le_Hierarchical_Conditional_Relation_Networks_for_Video_Question_Answering_CVPR_2020_paper.pdf) can be a candidate baseline.\n\n* Although the author introduces all the five modules and describes the training steps, it is still not clear for readers to understand how to train the five modules jointly without providing an explicit loss function or training objective. I think the author should specify such a training objective explicitly.\n\n- The paper states that DCL does not require additional fine-grained labels for attributes and events. However, DCL requires additional 1000 programs to train the program parser for all question types. Can we call these 1000 programs additional annotations? For a new visual scene that is different from the current 3D synthetic domain, to make DCL suitable for the new visual domain,  we need to define new sets of programs and concepts, which may require additional annotation labors.\n\n- To make the experiments more convincing on Retrieval and Grounding tasks, the author could further compare with some recent video-text retrieval/grounding approaches. To name a few: 1) [Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning](https://openaccess.thecvf.com/content_CVPR_2020/papers/Chen_Fine-Grained_Video-Text_Retrieval_With_Hierarchical_Graph_Reasoning_CVPR_2020_paper.pdf) 2) [TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval](https://arxiv.org/pdf/2001.09099.pdf) 3) [Semantic Conditioned Dynamic Modulation for Temporal Sentence Grounding in Videos](https://papers.nips.cc/paper/8344-semantic-conditioned-dynamic-modulation-for-temporal-sentence-grounding-in-videos.pdf)\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received 4 reviews with mixed initial ratings: 6,7,7,5. The main concerns of R1, who gave an unfavorable score, included lack of clarity (the manuscript is hard to follow) and limited empirical evaluation (the method is tested on a single synthetic dataset, CLEVRER). The latter point is echoed in other reviews as well. In response to that, the authors submitted a new revision and provided detailed responses to each of the reviews separately, which seemed to have addressed these concerns. R1 upgraded the rating and recommended acceptance.
As a result, the final recommendation is to accept this submission for presentation at ICLR as a poster.
Decision: Accept (Poster)