CLEVRER: Collision Events for Video Representation and Reasoning
Authors: Kexin Yi*
Chuang Gan*
Yunzhu Li
Pushmeet Kohli
Jiajun Wu
Antonio Torralba
Joshua B. Tenenbaum
Venue:  (2020)
Keywords: Neuro-symbolic
Reasoning

Abstract: The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER) dataset, a  diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks.  Motivated by the theory of human casual judgment, CLEVRER includes four types of question:  descriptive (e.g., ‘what color’), explanatory (‘what’s responsible for’), predictive (‘what will happen next’), and counterfactual (‘what if’).  We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perception-based task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations. 

Published as a conference paper at ICLR 2020
CLEVRER: COLLISION EVENTS FOR VIDEO REPRESENTATION AND REASONING
Kexin Yi Harvard University
Chuang Gan' MIT-IBM Watson Al Lab
Yunzhu Li MIT CSAIL
Pushmeet Kohli DeepMind
Jiajun Wu MIT CSAIL
Antonio Torralba MIT CSAIL
Joshua B. Tenenbaum MIT BCS, CBMM, CSAIL
ABSTRACT
The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER) dataset, a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human causal judgment, CLEVRER includes four types of question: descriptive (e.g.. 'what color'), explanatory ('what's 's responsible for'), predictive ('what will happen next'), ), and counterfactual ('what if"). We evaluate various state-of-the-an models for visual reasoning on our benchmark. While these models thrive on the perception- based task (descriptive), they perform poorly the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.
INTRODUCTION
The ability to recognize objects and reason about their behaviors in physical events from videos lies the core of human cognitive development (Spelke, 2000). Humans, even young infants, group segments into objects based on motion, and use concepts of object permanence, solidity, and continuity to explain what has happened, infer what is about to happen, and imagine what would happen in counterfactual situations. The problem of complex visual reasoning has been widely studied in artificial intelligence and computer vision, driven by the introduction of various datasets on both static images (Antol et al., 2015; Zhu et al., 2016; Hudson & Manning, 2019) and videos (Jang et al., 2017; Tapaswi et al., 2016; Zadeh et al., 2019). However, despite the complexity and variety of the visual context covered by these datasets, the underlying logic, temporal and causal structure behind the reasoning process less explored.
In this paper, study the problem of temporal and causal reasoning in videos from complementary perspective: inspired by a recent visual reasoning dataset, CLEVR (Johnson et al., 2017a), we simplify the problem of visual recognition, but emphasize the complex temporal and causal structure behind the interacting objects. We introduce a video reasoning benchmark for this problem, drawing inspirations from developmental psychology (Gerstenberg et al., 2015; Ullman, 2015). We also evaluate and assess limitations of various current visual reasoning models on the benchmark.
Our benchmark, named CoLlision Events for Video REpresentation and Reasoning (CLEVRER), is a diagnostic video dataset for temporal and causal reasoning under a fully controlled environment. The design of CLEVRER follows two guidelines: first, the posted tasks should focus logic reasoning in the temporal and causal domain while staying simple and exhibiting minimal biases on visual scenes and language; second, the dataset should be fully controlled and well-annotated in order to host the
indicates equal contributions. Project page: http:/ /tct::////eeeer....//mmm csail aall 1mmii e.du//

---

Published as a conference paper at ICLR 2020
First collision
(b) Cyan cube enters
(c) Second collision
Video ends
Descriptive
III. Predictive
Q: What shape is the object that collides with the ejan sylinder?
At gylönder
Q: Whick event will huppees next
Q How man motal objerte moving when video oudo?
cube collides with the rod object
The eyee cyfinder callider with the red object
At a)
II Explanatory
IV. Counterfactual
Q: Which of followinng responsible for the gray cylinder's coiliding the cube?
Q: Without gray object, event happen?
The presence the sphore
a) The cyun cylinder collider with the sphere rod object and sphere cullide
collision betneen gray eyllnder and the ejon sylindor
As
Figure I: Sample video, questions, and answers from our CoLlision Events for Video REpresentation and Rea- soning (CLEVRER) dataset. CLEVRER is designed to evaluate whether computational models can understand what is in the video (I, deseriptive questions), explain the cause of events (II, explanatory), predict what will happen in the future (III, predictive), and imagine counterfactual scenarios (IV, counterfactual) In the four images (a-d), only for visualization purposes. we apply stroboscopic imaging reveal object motion. The captions (e.g.. 'First collision') )) are for the readers to better understand the frames, not part of the dataset.
complex reasoning tasks and provide effective diagnostics for models on those tasks. CLEVRER includes 20,000 synthetic videos of colliding objects and more than 300,000 questions and answers (Figure 1). We focus on four specific elements of complex logical reasoning on videos: descriptive (e.g.. 'what color'), explanatory ('what's responsible for'), predictive ('what will happen next'), and counterfactual ("what if"). CLEVRER comes with ground-truth motion traces and event histories of each object in the videos. Each question is paired with a functional program representing its underlying logic. As summarized in table 1, CLEVRER complements existing visual reasoning benchmarks on various aspects and introduces several novel tasks.
We also present analysis of various state-of-the-ar visual reasoning models on CLEVRER. While these models perform well on descriptive questions, they lack the ability to perform causal reasoning and struggle on the explanatory, predictive, and counterfactual questions. We therefore identify three key elements that are essential to the task: recognition of the objects and events the videos; modeling the dynamics and causal relations between the objects and events; and understanding of the symbolic logic behind the questions. As first-step exploration of this principle, we study an oracle model, feerro-Symbole Dynamic Reasoning (NS-DR), that explicitly joins these components via symbolic video representation, and assess its performance and limitations.
RELATED WORK
Our work can be uniquely positioned in the context of three recent research directions: video understanding, visual question answering. and physical and causal reasoning.
Video understanding. With the availability of large-scale video datasets (Caba Heilbron et al., 2015; Kay et al., (017), video and language understanding tasks have received much interest. This includes video captioning (Guadarrama et al., 2013; Venugopalan et al., 2015; Gan et al., 2017) 1), localizing video segments from natural language queries (Gao al., 2017; Hendricks al., 2017), and video question answering. In particular, recent papers have explored different approaches to acquire and ground various reasoning tasks to videos. Among those, MovieQA (Tapaswi al., 2016), TGIF-QA (Jang et al., 2017), TVQA (Lei et al., 2018) are based on real-world videos and numan-generated questions. Social-IQ (Zadeh et al., 2019) discusses causal relations in human social interactions based on real videos. COG (Yang et al., 2018) and MarioQA (Mun et al., 2017) use simulated environments generate synthetic data and controllable reasoning tasks. Compared to them, CLEVRER focuses on the causal relations grounded in object dynamics and physical interactions, and introduces wide range of tasks including description, explanation, prediction and counterfactuals. CLEVRER also emphasizes compositionality in the visual and logic context.
Visual question answering. Many benchmark tasks have been introduced in the domain of visual question answering. The Visual Question Answering (VQA) dataset (Antol et al., 2015) marks an important milestone towards top-down visual reasoning, based on large-scale cloud-sourced real images and human- genueratd questions. The CLEVR dataset (Johnson et al., 2017a) follows bottom- up approach by defining the tasks under controlled close-domain setup of synthetic images and questions with compositional attributes and logic traces. More recently, the GQA dataset (Hudson & Manning, 2019) applies synthetic compositional questions to real images. The VCR dataset (Zellers
2

---

Published as a conference paper at ICLR 2020
Diagnostic Annotations
Temporal Relation
Dataset
Video
Explanation
Prediction
Counterfactual
VQA (Antol al., 2015)
CLEVR (Johnson al., 2017a)
X
X
COG (Yang et al., 2018)
X
X
X
VCR (Zellers ct al., 2019)
X
V
X
GQA (Johnson al., 701
X
V
X
X
TGIF-QA QA (Jang et al., 2017)
X
V
X
X
MovieQA (Tapaswi et al., 2016)
4
V
X
MarioQA (Mun al., 2017)
X
X
TVQA (Lii et al., 2018)
X
X
X
X
Social-IQ (Zadeh et al., 2019)
X
X
X
CLEVRER (ours)
Table 1:
Comparison between CLEVRER and other visual reasoning benchmarks on images and videos. CLEVRER well-annotated video reasoning dataset created under a controlled environment. introduces wide range of reasoning tasks including description, explanation, prediction and counterfactuals
al., 2019) discusses explanations and hypothesis judgements based on common sense. There have also been numerous visual reasoning models (Hudson & Manning, 2018; Santoro et al., 2017; Hu al., 2017; Perez et al., 201 8; Zhu al., 2017; Mascharka al., 2018; Suarez al., 2018; Cao et al., 2018; Bisk et al., 201 8; Misra et al., 2018; 8; Aditya et al., 2018), 8). Here we briefly review few. The stacked attention networks (SAN) (Yang et al., 2016) introduce a hierarchical attention mechanism for end-to-end VQA models. The MAC network (Hudson & Manning, 2018) combines visual and language attention for compositional visual reasoning. The IEP model (Johnson al., 2017b) proposes to answer questions via neural program execution. The NS- VQA model (Yi et al., 2018) disentangles perception and logic reasoning by combining an object-based abstract representation of the image with symbolic program execution. In this work study a complementary problem of causal reasoning and assess the strengths and limitations of these baseline methods.
Physical and causal reasoning. Our work is also related to research on learning scene dynamics for physical and causal reasoning (Lerer al., 2016; Battaglia et al., 2013; Mottaghi et al., 2016; Fragkiadaki et al., 2016, Battaglia et al., 2016; Chang et al., 2017; Agrawal et al., 2016; Finn et al., 2016; Shao al., 2014; Fire & Zhu, 2016; Pearl, 2009; Ye et al., 2018), either directly from im- ages (Finn et al., 2016; Ebert et al., 2017; Watters et al., 2017: Lerer et al., 2016; Mottaghi et al., 2016; Fragkiadaki et al., 2016), or from a symbolic, abstract representation of the environment (Battaglia et al., 2016; Chang et al., 2017). Concurrent to our work, CoPhy (Baradel et al., 2020) studies physical a counterfatual setting. CATER (Girdhar Ramanan, 2020) introduces a synthetic video dataset for temporal reasoning associated with compositional actions. CLEVRER complements these works by incorporating dynamics modeling with compositional causal reasoning, and grounding the reasoning tasks to the language domain.
dynamics prediction
THE CLEVRER DATASET
The CLEVRER dataset studies temporal and causal reasoning on videos. It is carefully designed in fully-controlled synthetic environment, enabling complex reasoning tasks, providing effective diagnostics for models while simplifying video recognition and language understanding. The videos describe motion and collisions of objects on a flat tabletop (as shown in Figure 1) simulated by a physics engine, and are associated with the ground-truth motion traces and histories of all objects and events. Each video comes with four types of questions generated by machine, including descriptive ('what color', 'how many'), explanatory ( ('What is responsible for'), ), predictive ("What will happen next'), and counterfactual ("what if"). Each question is paired with a functional program.
3.1 VIDEOS
CLEVRER includes 10,000 videos for training, 5,000 for validation, and 5,000 for testing. All videos last for 5 seconds. The videos generated by physics engine that simulates object motion plus graphs engine that renders the frames. Extra examples from the dataset can found in supplementary material C.
Objects in CLEVRER videos adopt similar compositional intrinsic attributes as CLEVR (Johnson et al., 2017a) 7a), including three shapes (cube, sphere, and cylinder), two materials (metal and rubber), and eight colors (gray, red, blue, green, brown, cyan, purple, and
Objects and events.
3

---

Published as a conference paper at ICLR 2020
How mony metal objects moving when the video ands?
Without the gray object, which event will happen?
The cube collides with the red object,
Objects
Fikerr materal
Tiller
Objeccts
Filter
Ger cousteriacta
Belona
Events
Hilter coltaiio
Fiter enulition
Qusation grograms
mroding
color
Gasy
Choice
Cube
Fiber shupe
Filer
Red
Events
Fiiner
cod
franse
Giet
Obbjects
Objjects
Figure 2: Sample questions and programs from CLEVRER. Left: Descriptive question. Middle and right: multiple-choice question and choice. Each choice can pair with the question to form a joint logic trace.
ycllow). All objects have the same size so no vertical bouncing occurs during collision. In each video, we prohibit identical objects, such that each combination of the three attributes uniquely identifies one object. Under this constraint, all intrinsic attributes for each object are sampled randomly. We further introduce three types of events: enter, exit and collision, each of which contains a fixed number of object participants: 2 for collision and for enter and exit. The objects and events form an abstract representation of the video. These ground-truth annotations, together with the object motion traces, enable model diagnostics, one of the key advantages offered by fully controlled environment.
Causal structure. Objects and events in CLEVRER videos exhibit rich causal structures. An event can be either caused by an object if the event the first one participated by the object, or another event if the cause event happens right before the outcome event on the same object. For example, if sphere collides with a cube and then a cylinder, then the first collision and the cube jointly "cause" the second collision. The object motion traces with complex causal structures are generated by the following recursive process, We start with one randomly initialized moving object, and then add another object whose initial position and velocity are set such that will collide with the first object. The same process is then repeated to add more objects and collisions to the scene. All collision pairs and motion trajectories are randomly chosen. We discard simulations with repetitive collisions between the same pair of objects.
Video generation. CLEVRER videos are generated from the simulated motion traces, including each object's 's position and pose at each time step. We use the Bullet (Coumans, 2010) physics engine for motion simulation. Each simulation lasts for seven seconds. The motion traces first down-sampled to fit the frame rate the output video (25 frames per second). Then the motion of the first five seconds are sent to Blender (Blender Online Community, 2016) render realistic video frames of object motion and collision. The remaining two seconds are held-out for predictive tasks. We further note CLEVRER adopts the same software and parameters for rendering as CLEVR.
3.2 QUESTIONS
Predictive
Counter-
We pair each video with machine-generated questions for descriptive, explanatory, predic- tive, and counterfactual reasoning. Sample ques- tions of the four types can be found in Fig- ure Each question is paired with functional program executable on the video's 's dynamical scene. Unlike CLEVR (Johnson et al., 2017a), our questions focus on the temporal and causal aspects of the objects and events in the videos. We exclude all questions erties, which can be answered by looking at single frame. CLEVRER consists of 219,918 descriptive questions, 33,811 explanatory questions, 14,298 predictive questions and 37,253 counterfactual questions. Detailed distribution and split of the questions be found in Figure 3 and supplementary material A.
fucual
Query Color
Count
Query Material
Explenatery
Descippive
Query
Shape
Exist
Figure 3: Distribution of CLEVRER question types.
Left: distribution of four main questions types. Right: distribution of descriptive sub-types.
static object prop-
Descriptive. Descriptive questions evaluate a model's capability to understand and reason about a video's 's dynamical content and temporal relation. The reasoning tasks are grounded to the com- positional space of both object and event properties, including intrinsic attributes (color, material, shape), motion, collision, and temporal order. All descriptive questions are 'open-ended' and can be answered by a single word. Descriptive questions contain multiple sub-types including count, exist, query color, query material, and query shape. Distribution of the sub-types is shown in Figure 3. We evenly sample the answers within each sub-type to reduce answer bias.
Explanatory. Explanatory questions query the causal structure of a video by asking whether an object event is responsible for another event. Event is responsible for event B if A is among
4

---

Published as a conference paper at ICLR 2020
B's ancestors in the causal graph. Similarly, object O is responsible for event A if O participates in A or any other event responsible for A. Explanatory questions are multiple choice questions with at most four options, each representing event or object in the video. Models need to select all options that match the question's 's description. There can be multiple correct options for each question. We sample the options balance the number of correct and wrong ones, and minimize text-only biases. Predictive. Predictive questions test a model's capability of predicting possible occurrences of future events after the video ends. Similar to explanatory questions, predictive questions are multiple- choice, whose options represent candidate events that will or will not happen. Because post-video events are sparse, we provide two options for each predictive question to reduce bias.
Counterfactual. Counterfactual questions query the outcome of the video under certain hypothet- ical conditions (e.g- removing one of the objects). Models need to select the events that would or would not happen under the designated condition. There are at most four options for each question. The numbers of correct and incorrect options are balanced. Both predictive and counterfactual questions require knowledge of object dynamics underlying the videos and the ability to imagine and reason about unobserved events.
Program representation. In CLEVRER, each question is represented by tree-structured func- tional program, as shown in Figure 2. A program begins with a list of objects or events from the video. The list is then passed through a sequence of filter modules, which select entries from the list and join the tree branches to output a set of target objects and events. Finally, an output module is called to query designated property of the target outputs. For multiple choice questions, each question and option correspond to separate programs, which can be jointly executed to output a yes/no token that indicates if the choice is correct for the question. A list of all program modules can be found in the supplementary material B.
Questions in CLEVRER are generated by a multi-step procedure. For each question type, a pre-defined logic template is chosen. The logic template can be further populated by attributes that are associated with the context of the video (i.c. the color, material, shape that identifies the object to be queried) and then turned into natural language. We first generate exhaustive list of all possible questions for each video. To avoid language bias, we sample from that list to maintain balanced answer distribution for each question type and minimize the correlation between the questions and answers over the entire dataset.
Question generation.
4 BASELINE EVALUATION
In this section, we evaluate and analyse the performances of a wide range of baseline models for video reasoning on CLEVRER. For descriptive questions, the models treat each question as a multi-class classification problem over all possible answers. For multiple choice questions, each question-choice pair is treated as a binary classification problem indicating the correctness of the choice.
4.1 MODEL DETAILS
The baseline models we evaluate fall into three families: language-only models, models for video question answering, and models for compositional visual reasoning.
Language-only models. This model family includes weak baselines that only relies on question input to assess language biases in CLEVRER. Q-type (random) uniformly samples an answer from the answer space or randomly select each choice for multiple-choice questions. Q-type (frequent) chooses the most frequent answer in the training set for each question type. LSTM uses a pretrained word embedding trained on the Google News corpus (Mikolov et al., 2013) to encode the input question and processes the sequence with LSTM (Hochreiter & Schmidhuber, 1997). A MLP is then applied to the final hidden state to predict a distribution over the answers.
Video question answering. We also evaluate the following models that relies both video and language inputs. CNN+MLP extracts features from the input video via a convolutional neural network (CNN) and encodes the question by taking the average of the pretrained word embeddings (Mikolov et al., 2013). The video and language features are then jointly sent to MLP for answer prediction. CNN+LSTM relies on the same architecture for video feature extraction but uses the final state of a LSTM for answer prediction. TVQA (Lei et al., 2018) introduces a multi-strean end-to-end neural model that sets the state of the art for video question answering. We apply attribute-awe object-centric features acquired by video frame parser (TVQA+). We also include a recent model that incorporates heterogeneous memory with multimodal attention work (Memory) (Fan et al., 2019) that achieves superior performance on several datasets.
5

---

Published as a conference paper at ICLR 2020
Explanatory per ques.
Predictive
Counterfactual
Methods
Descriptive
per opt.
per opt.
per ques.
per opt.
per ques.
Q-type (random)
29.2
50.1
8.1
50.7
25.5
50.1
10.3
Qttype (frequent)
33.0
50.2
16.5
50.0
0.0
50.2
1.0
LSTM
34.7
59.7
13.6
50.6
23.2
53.8
3.1
CNN+MLP
48.4
54,9
18.3
50.5
13.2
55.2
9.0
CNN+LSTM
51.8
62.0
17.5
57.9
31.6
61.2
TVQA+
72.0
63.3
23.7
70.3
48.9
53,9
4.1
Memory
54.7
53.7
13.9
50.0
33.1
54.2
7.0
IEP (V)
52.8
52.6
14.5
50.0
9.7
53,4
3.8
TeD-net (V)
79.5
61.6
3.8
50.3
6.5
56.1
4.4
MAC (V)
85.6
59.5
12.5
51.0
16.5
54.6
13.7
MAC (V+)
86.4
70.5
22.3
59.7
42.9
63.5
25.1
Table 2: Question-answeriin accuracy of visual reasoning baselines CLEVRER. All models are trained on the full training set. The IEP (V) model and TbD-net (V) 1000 programs to train the program generator.
Compositional visual reasoning, The CLEVR dataset (Johnson et al., 2017a) opened up new direction of compositional visual reasoning, which emphasizes complexity and compositionality in the logic and visual context. We modify several best- cest-performing models and apply them to our video benchmark. The IEP model (Johnson et al., 2017b) applies neural program execution for images. We apply the same approach to program-bases video reasoning task (IEP (V)) by substituting the program primitives by the ones from CLEVRER, and applying the execution modules the video features extracted by convolutional LSTM (Shi et al., 2015). TbD-net (Mascharka et al., 2018) follows a similar approach by parsing the input question into a program, which is then assembled into neural network that acts the attention map the image features. The final attended image feature is then sent to an output layer for classification. We adopt the same approach through spatial-temporal attention over the video feature space (TbD-net (V)). MAC (Hudson & Manning, 2018) incorporates joint attention mechanism on both the image feature map and the question, which leads to strong performance on CLEVR without program supervision. We modify the model by applying a temporal attention unit across the video frames to generate latent encoding
visual reasoning
the video (MAC (V)). The video feature then input to the MAC network to output an answer distribution. We study an augmented approach: we construct object-aware video features by adding the segmentation masks of all objects in the frames and labeling them by the values of their intrinsic attributes (MAC (V+)).
Implementation details. We use a pre-trained ResNet-50 (He et al., 2016) to extract features from pool5 layer output for CNN-based methods, and
the video frames. We use the 2,048-dimension the x4x 14 feature maps for MAC, IEP and TbD-net. The program generators of IEP and TbD-net trained 1000 programs. We uniformly sample 25 frames for each video as input. Object segmentation masks and attributes are obtained by a video parser consisted of an object detector and an attribute network.
4.2 RESULTS
We summarize the performances of all baseline models in Table 2. The fact that these models achieve different performances over the wide spectrum of tasks suggest that CLEVRER offers powerful assessment the models' strength and limitations various domains. All models are trained on the training set until convergence, tuned on the validation set and evaluated on the test set.
Evaluation metrics. For descriptive questions, calculate the accuracy by comparing the pre- dicted answer token to the ground-truth. For multiple choice questions, we adopt two metries: per-option accuracy measures the model's overall correctness single options across all questions; per-question accuracy measures the correctness the full question, requiring all choices to be selected correctly.
Descriptive reasoning. Descriptive questions query the content of the video from various aspects. In order to do well on this question type, model needs to both accurately recognize the objects and events that happen the video, as well as understanding the compositional logic pattern behind the questions. In other words, descriptive questions require strong perception and logic operations on both visual and language signals. As shown in Table 2, the LSTM baseline that relies only on question input performs poorly the descriptive questions, only outperforming the random baselines

---

Published as a conference paper at ICLR 2020
Question
objects
Answer
Whot shope the second object to callide with the gray object?
ISTM Eusolle
LSTM
rilter color (gray)
IV. Program Executor
Cibe
Query_shape
Putches
Madks
Postions
Prediction
III. Question Parser
(en ac)
Mask RACNN
Learned
Dynamies
Re-a..))
Video
I. Video Frame Parser
II. Dynamics Predictor
Figure 4: Our model includes four components: video frame parser that generates an object- representation of the video frames; question parser that turns question into functional program; dynamics predictor that extracts and predicts the dynamic of the video; and symbolic program executor that the program the dynamic scene obtain an answer.
by small margin. This suggests that CLEVRER has very small bias the questions. Video QA models, including the state of the art model TVQA+ (Lei et al., 2018) achieve better performances. But because of their limited capability of handling the compositionality in the question logic and visual context, these models are still unable to thrive on the task. In contrast, models designed for compositional reasoning, including TbD-net that operates on neural program execution and MAC that introduces a joint attention mechanism, able to achieve more competitive performances.
Causal reasoning. Results on the descriptive questions demonstrate the power models that combine visual and language perception with compositional logic operations. However, the causal reasoning tasks (explanatory, predictive, counterfactual) require further understanding beyond per- ception. Our evaluation results (Table show poor performance of most baseline models these questions. The compositional reasoning models that performs well on the descriptive questions (MAC (V) and TbD-net (V)) only achieve marginal improvements over the random and language-only baselines on the causal tasks. However, we do notice a reasonable gain in performance on models that inputs object-awar aware representations: TVQA+ achieves high accuracy on the predictive questions. and MAC (V+) improves upon MAC (V) on all tasks.
We identify the following messages suggested by our evaluation results. First, object-centric repre- sentations are essential for the causal reasoning tasks. This is supported by the large improvement on MAC (V+) over MAC (V) after using features that are aware object instances and attributes, as well as the strong performance of TVQA+ the predictive questions. Second, all baseline models lack a component to explicitly model the dynamics of the objects and the causal relations between collision events. As result, they struggle on the tasks involving unobserved scenes and in particular performs poorly on the counterfactual tasks. The combination object-centric representation and dynamics modeling therefore suggests promising direction for approaching the causal tasks.
NEURO-SYMBOLIE DYNAMIC REASONING
Bascline evaluations CLEVRER have revealed two key elements that are essential to causal reasoning: an object-centric video representation that aware the temporal and causal relations between the objects and events; and dynamics model able to predict the object dynamics under unobserved or counterfactual scenarios. However, unifying these elements with video and language understanding posts the following challenges: first, all the disjoint model components should operate on a common set of representations of the video, question, dynamics and causal relations; second, the representation should be aware of the compositional relations between the objects and events. We draw inspirations from Yi et al. (2018) and study an oracle model that operates on a symbolic representation join video perception, language understanding with dynamics modeling. Our model Neuro-Symbolic Dynamic Reasoning (NS-DR combines neural nets for pattern recognition and dynamics prediction, and symbolic logic for causal reasoning. shown in Figure 4, NS-DR consists of a video frame parser (Figure 4-1), neural dynamics predictor (Figure 4-II), question parser (Figure 4-III), and a program executor (Figure 4-IV). We present details of the model components below and in supplementary material B.
7

---

Published as a conference paper at ICLR 2020
Video frame parser. The video frame parser serves as a perception module that is disentangled from other components of the model, from which we obtain an object-centric representation of the video frames. The parser is a Mask R-CNN (He et al., 2017) that performs object detection and seene de-rendering on each video frame (Wu et al., 2017). We use a ResNet-50 FPN (Liin et al., 2017; He al., 2016) as the backbone. For each input video frame, the network outputs the object mask proposals, a label that corresponds to the intrinsic attributes (color, material, shape) and a score representing the confidence level of the proposal used for filtering. Please refer to He al. (He et al., 2017) for more details. The video parser is trained on 4000 video frames randomly sampled from the training set with object masks and attribute annotations.
Neural dynamics predictor. We apply the Propagation Network (PropNet) (Li et al., 2019) for dynamics modeling, which is a learnable physics engine that, extending from the interaction net- works (Battaglia et al., 2016), performs object- and relation-centric updates on a dynamic scene. The model inputs the object proposals from the video parser, and learns the dynamics of the objects across the frames for predicting motion traces and collision events. PropNet represents the dynamical system as directed graph, G -= (O, Rt), where the vertices O {o.) represent objects and edges R = [rk} represent relations. Each object (vertex) o; and relation (edge) Thk can be further written as 0 (s), T = (ux, Ukk af), where s; the state of object as denotes its intrinsic attributes; Uk. Uk are integers denoting the index of the receiver and sender vertices joined by edge r's og represents the state of edge rk. indicating whether there is collision between the two objects. In our case, 8; is concatenation of tuple (G.,,, Ps) over small history window to encode motion history, where C and m, are the corresponding image and mask patches cropped at p;. which the T-y position of the mask in the original image (please see the cyan metal cube in Figure 4 for an example). PropNet handles the instantaneous propagation of effects via multi-step message passing. Please refer to supplementary material B for more details.
Question parser. We use an -based seq2seq model (Bahdanau et al., 2015) parse the input questions into their corresponding functional programs. The model consists of a bidirectional LSTM encoder plus an LSTM decoder with attention, similar to the question parser in (Yi al., 2018). For multiple choice questions, we use two networks to parse the questions and choices separately. Given an input word sequence (z1, (21,22, F2,).. the encoder first generates bi-directional latent encoding at each step:
cf h/ LSTM L, h/1),
c 4 = LSTM (mrs), M4+1):
(2)
e; [c], e
(3)
The decoder then generates sequence of program tokens (y1, y2, U)) from the latent encodings using attention:
" = (S)M()0()))))),
(4)
Ogz x exp(v es),
Cj c- Σ agies
(5)
ú softmax(W Iaj,,,)))
(6)
At training time, the input program tokens {y, are used for generating the predicted labels 3/-1 + 9; at each step j = 1, 3,.,,... J. The generated label Jj is then compared with yj to compute loss for training. At test time, the decoder rolls out by feeding the sampled prediction at the previous step to the input of the current step új Vj- The roll-out stops when the end token appears or when the sequence reaches certain maximal length. For both the encoder and decoder LSTM, use the same parameter setup of two hidden layers with 256 units, and a 300-dimensional word embedding vector.
Program executor. The program executor explicitly executes the program on the motion and event traces extracted by the dynamics predictor and output an answer to the question. It consists of a collection of functional modules implemented in Python. Given an input program, the executor first assembles the modules and then iterate through the program tree. The output of the final module is the answer to the target question. There are three types of program modules: input module, filter module, and output module. The input modules are the entry points of the program trees and can directly query the output of the dynamics predictor. For example, the Events and Ob jects s modules that commonly appear in the programs of descriptive questions indicate inputting all the observed events and objects from video. The filter modules perform logic operations on the input objects/events
8

---

Published as a conference paper at ICLR 2020
Explanatory
Predictive
Counterfactual
Methods
Descriptive
per opt.
per ques.
per opt.
per ques.
per opt.
per ques.
NS-DR
88.1
87.6
79.6
82.9
68.7
74.1
42.2
NS-DR (NE)
85.8
85.9
74.3
75,4
54.1
76.1
42.0
Table 3: Quantitative results of NS-DR CLEVRER. We evaluate model all four question types. We also study a variation our model NS-DR (NE) with "no events" but only motion traces from the dynamics predictor. Our question parser is trained with 1000 programs.
based on a designated intrinsic attribute, motion state, temporal order, or causal relation. The output modules return the answer label. For open-ended descriptive questions, the executor will return token from the answer space. For multiple choice questions, the executor will first execute the choice program and then send the result to the question program to output yes/no token. A comprehensive list of all modules in the NS-DR program executor is presented in supplementary material B.
Results. We summarize the performance of NS-DR on CLEVRER in Table 3. On descriptive questions, our model achieves an 88. 1% accuracy when the question parser is trained under 1,000 programs, outperforming other baseline methods. On explanatory, predictive, and counterfactual questions, our model achieves more significant gain. We also study a variation of our model, NS-DR (No Events/NE), which uses the dynamics predictor to generate only the predictive and counterfactual motion traces, and collision events are identified by the velocity change of the colliding objects. NS-DR (NE) performs comparably to the full model, showing that our disentangled system adaptable to alternative methods for dynamics modeling. We also conduct ablation study on the number of programs for training the question parser. As shown in Figure 5, NS-DR reaches full capability at 1,000 programs for all question types.
We highlight the following contributions of the model. First, NS-DR incorporates a dynamics planner into the visual reasoning task, which directly enables predictions of unobserved motion and events, and enables the model for the predictive and counterfactual tasks. This suggests that dynamics planning has great potential for uaguage-groundee takes preliminary step towards this direction. Second, symbolic representation provides powerful common ground for vision, language, dynamics and causality. By design, it empowers the model to explicitly capture the compositionality behind the video's 's causal structure and the question logic.
visual reasoning tasks and NS-DR
We further discuss limitations of NS-DR
Descriptive
Explanatory
and suggest possible directions for future research. First, training of the video and question parser relies extra supervisions such as object masks, attributes, and ques- tion programs. Even though the amount of data required for training is minimal com- pared to end-to-end approaches (i.e. thou- sands of annotated frames and programs), these data is hard to acquire in real-world applications. This constraint could be re- laxed by applying unsupervised/weedl weakly- supervised methods for scene decomposi- tion and concept discovery (Burgess et al., 2019; Mao et al., 2019). Second, our model performance decreases on tasks that require long-term dynamics prediction such as the counterfactual questions. This suggests that we need a better dynamics model capable of generating more stable and accurate trajectories. CLEVRER provides benchmark for assessing the predictive power of such dynamics models.
100
100
uestuceurcy (%)
75
50
50
25
25
p2r 0gt.
por ques.
200 500
1000
2000
200
500
1000
2000
Predictive
Counterfactunl
100
100
Text accuracy (%6
75
75
50
50
per opt.
per ques
25
per opt.
ques.
200 500
1000
2000
200
1000
2000
Programs
Programs
Figure 5: Performance of model under different number of programs used for training question parser.
CONCLUSION
We present a systematic study of temporal and causal reasoning in videos. This profound and challenging problem deeply rooted to the fundamentals of human intelligence has just begun to be studied with 'modern' Al tools. We introduce set of benchmark tasks to better facilitate the research in this area. We also believe video understanding and reasoning should go beyond passive know anwweegge extraction, and focus building internal understanding of the dynamics and causal relations,

---

Published as a conference paper at ICLR 2020
which is essential for practical applications such dynamic robot manipulation under complex causal conditions. Our newly introduced CLEVRER dataset and the NS-DR model are preliminary steps toward this direction. We hope that with recent advances in graph networks, visual predictive models, and neuro-symbolic algorithms, the deep learning community can now revisit this classic problem in more realistic setups in the future, capturing true intelligence beyond pattern recognition.
Acknowledgment. We thank Jiayuan Mao for helpful discussions and suggestions. This work is in part supported by ONR MURI 500014416-112000 the Center for Brain, Minds, and Machines (CBMM, funded by NSF STC award CCF- (CF-1231216), and IBM Research.
REFERENCES
Somak Aditya, Yezhou Yang, and Chitta Baral. Explicit Reasoning Over End-to-End Neural Archi- tectures for Visual Question Answering. In AAAI, 2018. 3
Pulkit Agrawal, Ashvin Nair, Pieter Abbeel, Jitendra Malik, and Sergey Levine. Learning to Poke by Poking: Experiential Learning of Intuitive Physics. In NeurIPS, 2016. 3
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C. Lawrence Zit- nick, and Devi Parikh. VQA: Visual Question Answering. In ICCV, 2015. 1, 2,
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural Machine Translation by Jointly Learning to Align and Translate. In ICLR, 2015. 8
Fabien Baradel, Natalia Neverova, Julien Mille, Greg Mori, and Christian Wolf. Cophy: Counterfac- tual learning of physical dynamics. In ICLR, 2020. 3
Peter W. Battaglia, Jessica B. Hamrick, and Joshua B. Tenenbaum. Simulation As an Engine of Physical Scene Understanding. PNAS, (10(45):13377-183333 2013. 3
Peter W. Battaglia, Razvan Pascanu, Matthew Lai, Danilo Rezende, and Koray Kavukcuoglu. In- teraction Networks for Learning about Objects, Relations and Physics. NeurIPS, 2016.
Yonatan Bisk, Kevin J. Shih, Yejin Choi, and Daniel Marcu. Learning Interpretable Spatial Operations in a Rich 3D Blocks World. In AAAI, 2018.
Blender Online Community. Blender 3D modelling and rendering package. Blender Foundation, Blender Institute, Amsterdam, 2016. URL http: /www blleerr................. org. 4
Christopher P Burgess, Loie Matthey, Nicholas Watters, Rishabh Kabra, Irina Higgins, Matt Botvinick, and Alexander Lerchner. Monet: Unsupervised scene decomposition and representation. arXiv preprint arXiv:: 1390, 2019. 9
Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large-scale video benchmark for human activity understanding. In CVPR, 2015.
Qingxing Cao, Xiaodan Liang, Bailing Li, Guanbin Li, and Liang Lin. Visual Question Reasoning on General Dependency Tree. CVPR, 2018.
Michael B. Chang, Tomer Ullman, Antonio Torralba, and Joshua B. Tenenbaum. A Compositional Basect-Based Approach to Learning Physical Dynamics. In ICLR, 2017.
Erwin Coumans. Bullet Physics Engine. Open Source Software: http://bullelphysicc.. org, 2010. 4
Frederik Ebert, Chelsea Finn, Alex X Lee, and Sergey Levine. Self-supervised visual planning with temporal skip connections. In CoRL, 2017.
Chenyou Fan, Xiaofan Zhang, Shu Zhang, Wensheng Wang, Chi Zhang, and Heng Huang. Hetero- gencous memory enhanced multimodal attention model for video question answering. In CVPR, 2019. 5
Chelsea Finn, Ian Goodfellow, and Sergey Levine. Unsupervised Learning for Physical Interaction through Video Prediction. In NeurIPS, 2016.
10

---

Published as a conference paper at ICLR 2020
Amy Fire and Song-Chin -hun Zhu. Learning perceptual causality from video. 7IST, 7(2)23, 2016. 2016. 3
Katerina Fragkiadaki, Pulkit Agrawal, Sergey Levine, and Jitendra Malik. Learning Visual Predictive Models of Physics for Playing Billiards. In ICLR, 2016. 3
Zhe Gan, Chuang Gan, Xiaodong He, Yunchen Pu, Kenneth Tran, Jianfeng Gao, Lawrence Carin, and Deng. Semantic compositional networks for visual captioning. In CVPR, 2017. 2
Jiyang Gao, Chen Sun, Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. ICCV, 2017.
Tobias Gerstenberg, Noah Goodman, David A Lagnado, and Joshua B Tenenbaum. How, whether, why: Causal judgments as counterfactual contrasts. In CogSci, 2015.
Rohit Girdhar and Deva Ramanan. Cater: diagnostic dataset for compositional actions and temporal
reasoning. 2020. 3
Sergio Guadarrama, Niveda Krishnamoorthy, Girish Malkamennarr, Subhashini Venugopalan, Ray- mond Mooney, Trevor Darrell, and Kate Saenko. Youtube2text: Recognizing and describing arbitrary activities using semantic hierarchies and zero-shot recognition. In ICCV, 2013. 2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In CVPR, 2016. 6, 8
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. ICCV, 2017. 8
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In ICCV, 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long Short-Term Memory. Neural Comput., 9(8): 1735-1780, 1997. 5
Ronghang Hu, Jacob Andreas, Marcus Rohrbach, Trevor Darrell, and Kate Saenko. Learning to Reason: End-to-Ent Module Networks for Visual Question Answering. In CVPR, 2017.
Drew A. Hudson and Christopher D. Manning, Compositional Attention Networks for Machine Reasoning. ICLR, 2018. 3,
Drew Hudson and Christopher D Manning. GQA: new dataset for real-world visual reasoning
and compositional question answering. In CVPR, 2019. 1,2
Yunseok Jang, Yale Song, Youngjae Yu, Youngjin Kim, and Gunhee Kim. TGIF-QA: Toward Temporal Reasoning in Visual Question Answering. In CVPR, 2017. 1,
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Li Fei-Fei, C. Lawrence Zitnick, and Ross Girshick. CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning, CVPR, 2017a. 1, 2, 3,
Justin Johnson, Bharath Hariharan, Laurens van der Maaten, Judy Hoffman, Li Fei-Fei, Lawrence Zitnick, and Ross Girshick. Inferring and Executing Programs for Visual Reasoning, In ICCV, 2017b. 3.6
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan. Fabio Viola, Tim Green, Trevor Back, Paul Natsev, al. The kinetics human action video dataset. /7Xiv:1005.06950 2017. 2
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In ICLR, 2015. 15
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. TVQA: Localized, compositional video question answering. In EMNLP, 2018. 2, 3,7,
Adam Lerer, Sam Gross, and Rob Fergus. Learning Physical Intuition of Block Towers by Example. In ICMI, 2016.
11

---

Published as a conference paper at ICLR 2020
Yunzhu Li, Jiajun Wu, Jun-Yan Zhu, Joshua B. Tenenbaum, Antonio Torralba, and Russ Tedrake. Propagation Networks for Model-Based Control under Partial Observation. In ICRA, 2019. 8
Tsung-Yi Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.
Feature pyramid networks for object detection. CVPR, 2017.
Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, and Jiajun Wu. The Neuro- Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences from Natural Supervision. In ICLR, 2019. 9
David Mascharka, Philip Tran, Ryan Soklaski, and Arjun Majumdar. Transparency by Design: Closing the Gap between Performance and Interpretability in Visual Reasoning. In CVPR, 2018.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In NewrIPS, 2013.
Ishan Misra, Ross Girshick, Rob Fergus. Martial Hebert, Abhinav Gupta, and Laurens van der Maaten. Learning by asking questions. In CVPR, 2018.
Roozbeh Mottaghi, Mohammad Rastegari, Abhinav Gupta, and Ali Farhadi. "What happens if..." learning to predict the effect of forces in images. In ECCV, 2016. 3
Jonghwan Mun, Paul Hongsuck Seo, Ilchae Jung, and Bohyung Han. MarioQA: Answering Questions by Watching Gameplay Videos. In ICCV, 2017. 2,3
Judea Pearl. Causality. Cambridge University Press, 2009. 3
Ethan Perez, Florian Strub, Harm De Vries, Vincent Dumoulin, and Aaron Courville. FiLM: Visual Reasoning with a General Conditioning Layer. In AAAI, 2018. 3
Adam Santoro, David Raposo, David G. T. Barrett, Mateusz Malinowski, Razvan Pascanu, Peter Battaglia, and Timothy Lillicrap. A Simple Neural Network Module for Relational Reasoning. In NeurIPS, 2017. 3
Tianjia Shao, Aron Monszpart, Youyi Zheng, Bongjin Koo, Weiwei Xu, Kun Zhou, and Niloy J. Mitra. Imagining the Unseen: Stability-Based Cuboid Arrangements for Scene Understanding. ACM TOG, 33(6), 2014. 3
Xingjian Shi, Zhourong Chen, Hao Wang, Dit- Yan Yeung, Wai-Kin Wong, and Wang-chun Woo. Convolutional Istm network: A machine learning approach for precipitation nowcasting. In NeurIPS, 2015.
Elizabeth S. Spelke. Core Knowledge. Am. Psychol., 55(11):1233) 2000.
Joseph Suarez, Justin Johnson, and Fei-Fei Li. DDRprog: A Clevr Differentiable Dynamic Reasoning Programmer. vrXiv:1803.1136/ 2018. 3
Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. MovieQA: Understanding Stories in Movies through Question-Answering. In CVPR, 2016. 2,
Tomer David Ullman. On the Nature and Origin of Intuitive Theories: Learning, Physics and Psychology. PhD thesis, Massachusetts Institute of Technology, 2015.
Subhashini Venugopalan, Marcus Rohrbach, Jeffrey Donahue, Raymond Mooney, Trevor Darrell, and Kate Saenko. Sequence sequence-video to text. In ICCV, 2015. 2
Nicholas Watters, Andrea Tacchetti, Theophane Weber, Razvan Pascanu, Peter Battaglia, and Daniel Zoran. Visual Interaction Networks: Learning a Physics Simulator from Video. In NewrlPS, 2017.
Jiajun Wu, Joshua B. Tenenbaum, and Pushmeet Kohli. Neural Scene De-rendering. In CVPR, 2017.
12

---

Published as a conference paper at ICLR 2020
Guangyu Robert Yang, Igor Ganichev, Xiao-Jing Wang. Jonathon Shlens, and David Sussillo. A Dataset and Architecture for Visual Reasoning with a Working Memory. ECCV, 2018. 2,
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked Attention Networks for Image Question Answering. In CVPR, 2016. 3
Tian Ye, Xiaolong Wang, James Davidson, and Abhinav Gupta. Interpretable intuitive physics model. In ECCV, 2018. 3
Kexin Yi, Jiajun Wu, Chuang Gan, Antonio Torralba, Pushmeet Kohli, and Joshua B. Tenenbaum. Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understanding. In NeurIPS, 2018. 7,8
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis-Philippe Morency. Social-IQ: A question answering benchmark for artificial social intelligence. In CVPR, 2019. 1,
Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. CVPR, 2019. 2.3
Chen Zhu, Yanpeng Zhao, Shuaiyi Huang, Kewei Tu, and Yi Ma. Structured Attentions for Visual Question Answering. In ICCV, 2017.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, 2016.
13

---

Published as a conference paper at ICLR 2020
SUPPLEMENTARY MATERIALS
A DESCRIPTIVE QUESTION SUB-TYPES AND STATISTICS
Descriptive questions in CLEVRER consists of five sub-types. When generating the questions, balance the frequency of the answers within each question sub-type to reduce bias. The sub-types and their answer space distribution are shown in Figure
Sub-type
Answers
Count
2, 4, yes, no rubber, metal sphere, cube, cylinder gray, brown, green, red, blue, purple, yellow, cyan
metal
Exist
Query material Query-shape
rabler
golore
Query.color
Exist
8
Shape
Color
Figure 6: Descriptive question sub-type and answer space statistics.
B NS-DR MODEL DETAILS AND TRAINING PARADIGM
Here we present further details of the NS-DR model and the training paradigm. A running example of the model can be found in Figure 7.
Video frame parser. Our frame parser trained on 4,000 frames randomly selected from the training videos plus ground-truth masks and attribute annotations of each object in the frames. We train the model for 30,000 iterations with stochastic gradient decent, using a batch size of 6 and learning rate 0.001. At test time, we keep the object proposals with confidence score of 0.9 and use those to form the object-level abstract representation of the video.
Neural dynamics predictor. The object encoder fo and the relation encoder f in PropNet are instantiated as convolutional neural networks and output Deme-dim vector the representation. We add skip connections between the object encoder for and the predictor ffreed for each object to generate higher quality images. We also include a relation predictor fk to determine whether two objects will collide not in the next time step.
At time f, we first encodes the objects and relations c,, = 10 (oi,t) Ch. = J (Our, i Oue ag)) where og, denotes object at time
then denotes the propagating effect from relation & at propagation step as ct.,, and the effect from object i as h4,-- For step s < L, the propagation procedure can be described as
Step
h.: = 0, i=
Step / L....,L:
cf. = fulch, M h == 1-..|||)
h/, = folch. Σ ch hie
11-1),
i= 1........ 10),
(8)
ReN:
Output:
Pk,t+1 = spred
f (ce ek,),
1........ |R).
(9)
p (c 111)))
1...10
where fo denotes the object propagator, fA denotes the relation propagator, and Ae denotes the relation set where object is the receiver. The neural dynamics predictor is trained by minimizing the Le distance between the predicted fk, d and the real future observation Tkst1s 0 using stochastic gradient descent.
14

---

Published as a conference paper at ICLR 2020
Objects
Events
Cyan
Gray
Frime
Rabber
Metal
Rubber
Rubber
Motal
Type
Collision
Enter
Collision
Collision
Collision
Shape
Cylinder
Cylinder
Sphere
Sphere
Sphere
Object ID
1,4
5
1,2
4,5
2,4
Question:
What shope the object collide with the cyan object?
Program:
query wwwpeeeeeeeteollperrrereeeeee
fitter_colorObjjots. Cyan)),
Answer: Sphere
First), cileer_cler(Obbjets Cyanl))
Question: Which will happen next?
Choice: red rubber sphere collides with the metal sphere
Program: belong bellnggtoofilteer htttteercolliioonttteee..o celliee__olllsinnfeeeett Sittershape(filee htt.:lllerraatrrtffffm www.mmmmaaaltteeeeooooo _ooloot
Answer:
Objects. Red), Rubber), Sphere)), filter_ citeerssapp(iittrrrrr. (orrrfilllermmttrtrllhhyeeeee Metal), Sphere)), UnsemnVentt)
Question: Which event will hoppen without the cyen cylinder?
Choice: The red rubber sphere collides with the yellow sphere
Program: belong Ioffileer collisionfillrr cellisionnEvents filer_shapeliiiee stappeliller.mmtttiaaiiii wwt.mmattaallll.ccm crr Objects, Red), Rubber), Spherb)), filter serr_shppfflterr. shapoffiltr_ccooorOheees Yellow), Sphere)), get_coumterfact filer_ stoopssseeeeeeeeerrrrre poliiiOOoeo.. Cysn), Cylinder)l)
Answer: X
Sample results of NS-DR CLEVRER. 'Pred.' and 'CF' indicate predictive and counterfactual events extracted by model's dynamics predictor. The counterfactual condition shown in this example to remove the cyan cylinder.
Figure 7:
The output of the neural dynamics predictor is ((0,) (Re.rr.Reee)..)). the collection of object states and relations across all the observed and rollout frames. From this representation, one can recover the full motion and event traces of the objects under different rollout conditions and generate dynamic scene representation of the video. For example, if we want to know what will happen if an object is removed from the scene, we just need to erase the corresponding vertex and associated edges from the graph and rollout using the learned dynamics obtain the traces.
Our neural dynamics predictor is trained on the proposed object masks and attributes from the frame parser. We filter out inconsistent object proposals by matching the object intrinsic attributes across different frames and keeping the ones that appear in more than 10 frames. For video of length 125, will sample 5 rollouts, where each rollout contains 25 frames that are uniformly sampled from the original video. We normalize the input data to the range of [-1, 1] and concatenate them over a time window of size 3. We set the propagation step T. to and the dimension of the propagating effects Dene to 512. We use the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 10-1 and decay factor of 0.3 per epochs. The model is trained for 9 epochs with batch size Question parser. For open-ended descriptive questions, the question parser is trained various numbers of randomly selected question program pairs. For multiple choice questions, we separately train the question parser and the choice parser. Training with n samples means training the question parser with questions randomly sampled from the multiple choice questions (all three types together), and then training the choice parser with 4n choices sampled from the same pool. All models are trained using Adam (Kingma & Ba, 2015) for 30,000 iterations with batch size 64 and learning rate x 10-4
prestion-program
Program executor. A comprehensive list of modules in the NS-DR program executor sum- marized in Table 4 and Table 5. The input and output data types of each module summarized in Table
C EXTRA EXAMPLES FROM CLEVRER
We show extra examples from CLEVRER in Figure 8. We also present visualizations of ques- tion/choice programs of different question types in Figure 9 through Figure 11. The full dataset will be made available for download.
15

---

Published as a conference paper at ICLR 2020
Deseriptive
Predictive
Wiet
collide with gray
Whink will hagoon
Q2: How many collisions happen??
A2:
a) The gray sphore and cube collide The cahe cullides with the cylinder
yuhoves that the before brow object enters the scene?
Are there
A:
A3:
Counterfactual
Explanatory
Q: Whh will happos withhost the gray sphere? eyou b) The gress sphere and the ejan sphere collide The brown objest and the cyun sphore collide object with cabe
Q: Which the following not responstle for the collitinn betwees the gray
cahe cullides
sphere and the brows object?
cullition
groy quhove grvwn spherv
The presence the green rubber sphere
A:
The presence the ejan motal sphere
Descriptive
Predictive
Ql: Are there any muving objects when
Q: eveut will happos
Q2: color the object that enters the scene?
A2: purple
0) The green cylinder and the sphere collide grewe cylinder rollider with cuhe.
Q3:
of the fiest object
uphore?
A3: green
Counterfactual
Explauatory
Without the green cylinder, what will not happen?
Q: Which of the following responstble for the sphere's cxiting the scene?
a) The sphere and cube collide
prosueee the object
The sphere and the eyon cylinder collidle cube cyan cyfinder collide
The presence the purpic rubber cube
The presence the green metal cylinder
collition
sphere cyan cylinder
b
Descriptive
Predictive
many objects
will hoppen al The cube and bise sphere collide
Q2: What shape the lert object collide with the cylinder? What volor the object natiouary when video onde?
Al: sphere
blu
The neul object collider with the blue
a)
Explanatory
Counterfactual
O: Which of the following respoustbie for the collision between the sylinder
Q: What will happen the metul objest is removed?
objont?
cylinder the sphere collide
0) The cube's enteing the scone
b) The cube collides with blue sphore
5) The presence the blue rubber cube
The cube and the cylinder eallide
As (sb)).))
colliding with cylinder
d) The prevence of the blue rubber sphere
As b).c)
Figure 8: Sample videos and questions from CLEVRER. Stroboscopic imaging is applied for motion visualiza-
tion.
16

---

Published as a conference paper at ICLR 2020
Module Type
Name
/ Description
Input Type
Output Type
Objects
objects
Returns all objects in the video Events
events
Returns all events that happen the video UnseenFvents
events
Input Modules
Returns all future events after the video ends A1 1 EEvents
events
Returns all possible events on / between any objects Start
event
Returns the special "start" event
End
event
Returns the special "end" event
Filter-color
(objects, color)
objects
Selects objects from the input list with the input color Filter_mater
(objects, material)
objects
Selects objects from the input list with the input material F ltterssappe
Object Filter Modules
(objects, shape)
objects
Selects objects from the input list with the input shape
tter moving
cohjects,frame,
objects
Selects all moving objects in the input frame or the entire video (when input frame is "null") Filter-sted serr-ssttioaaryy
(abjects, frame)
objects
Selects all stationary objects in the input frame or the entire video (when input frame is "null")
Filter-in
(events, objects)
events
Selects all incoming events of the input objects
F Filter_out
(events, objects)
events
Selects all exiting events of the input objects
Fi 1tte_colll ision
(events, objects)
events
Selects all collisions that involve any of the input objects F wilter before
(events, event)
events
Selects all events before the target event
Filter-after
(events, event)
events
Selects all events after the target event F Filter_ordde
Event Filter Modules
(events, order)
event
Selects the event at the specific time order Fill aateraanestoo
(events, event)
events
Selects all causal ancestors of the input event Get_frame
event
frume
Returns the frame the input event the video
Get.count fact
(events, object)
events
Selects all events that would happen if the object is removed
Get_cola ret.col-partnes
(event, object)
object
Returns the collision partner of the input object (the input event must be a collision)
Get ob ject
event
object
Returns the object that participates in the input event (the input event must be incoming outgoing event)
Unique
events / objects
event object
Returns the only event / object in the input list
Table 4: Functional modules of NS-DR's program executor.
17

---

Published as a conference paper at ICLR 2020
Module Type
Name / Description
Input Type
Output Type
Query corloo
object
color
Returns the color of the input object
Query_ma warymmateiiiaa
object
material
Returns the material of the input object
Query_shape
object
shape
Output Modules
Returns the shape of the input object
Count
objects
int
Returns the number the input objects
Exist
objects
bool
Returns "yes" if the input objects is not empty
Belong_to
(event, events)
bool
Returns "yes" if the input event belongs the input of events
Negate
bool
bool
Returns the negation of the input boolean
Table 5: Functional modules NS-DR's program exccutor (continued).
Data type
Description
object
dictionary storing the intrinsic attributes of object in video
objects
list of objects
event
dictionary storing the type, frame, and participating objects of an event
events
A list of events
order
string indicating the chronological order of event out of "first", "second", "last"
color
"gray", "brown", "green", "red", "blue", "purple", "yellow", "cyan" string indicating material out of "metal", "rubber"
string indicating color
material
shape
string indicating shape out of "cube", "cylinder", "sphere"
frume
An integer representing the frame number of an event
Table 6: Input/output data types of modules in the program executor.
Question: "How many spheres are moving when the video ends?"
Question: "What shape is the second object to collide with the green object?'
Count(
Query_ sshey_shape( Get_col_partnert fiiteroorder(
Filter Filter_move( movel
Filter Shaeeesshpee(Ojjet... "Sphere"),
Get (rnd
Filte_ccolission
Events,
Filter 'Green")
"Second"
"Are there any stationary spheres when the metal sphere enters the scene?"
Question:
colter_color(Ohjectss. "Green")
Exist(
Filler_stalie _tt_ssttaiooary(
Filter_ (Objjjss,, "Sphere"),
Query_framee
Question: "What color is the object collide with the blue object?"
Pilter_in(
Events,
Query_colore
Filter Filter_shape(
Set__colpaartee
Filter litter_materiilOObectsss.. "Metal"),
Filter collisioo
"Sphere"
Events,
Filter_color(Ohieccs,, "Blue")
Pilter cole_cccolrOOssccss. "Bluc")
Figure 9: Example of descriptive question programs.
18

---

Published as a conference paper at ICLR 2020
Question: "Which of the following responsible for the metal sphere's colliding with the rubber sphere?" Negate(
Choice: "The collision between the green cube and the
rubber sphere"
Pfiter_collisions Filter_collision
Belong_tol
Event, // Output of the Choice program
Events,
Filter_ance anrrrancceorr
Filter_shape(
Filte_cculission
Filler_cole caller_colorOObjeuts, "Cube"),
Filter_col (itterccolissoo(
"Cube"
Events,
Filler shhep_sspeeee
)
Filler_materialOObjeee "Metal"),
Filter shapel
"Sphere"
Filter_ma matteriaaaaaaieetss "Rubber"), "Sphere"
Filter shape(
Filter_ tttter_materialOOjectts "Rubber"),
Choice: "The green cube's entering scene"
"Sphere"
Filter_in iin
Filer_shapee(
Filter_col citter_color(Objects, "Green"), "Cube"
Figure 10: Example of explanatory question and choice programs.
rubber sphere removed?" "Which event will happen Predictive question: "What will happen next?" Belong_tof
Counterfactual question:
Belong_tol
Event, // Output of the choice program UnseenEvents
Event, Output of the choice program coleettttrrfc
Filter Filter_shape(
Filter_matd mbteriil "Rubber"),
"Sphere"
Choice:
"The cube collides with the metal sphere"
Choice:
"The metal sphere and the cylinder collide"
coll__collssio((
Filter_collision
Filter_col collisooo
AllEvents,
AllEvents,
Filter Silter_shape(Objeett "Cube")
Filter silter_shape(
).
Filter material(Objects, "Metal"), "Sphere"
Siter_shapp((
Filter tttirrtaaaraiiiiittts "Metal"),
"Sphere"
Filter Siter_sshppeOOheers. "Cylinder")
Figure 11: Example of predictive counterfactual question and choice programs.
19


Reviews:
[{"review_id": "SJl5uAax5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Authors propose a new Dataset CLEVRER, a simulated video dataset involving interaction between objects. It is discussed, the existing state-of-the-art models for visual question answering, doesn’t capture the causal structure between the objects and their claim is supported by their experiments.  Authors also proposed a model which captures the dynamics of the objects involved in the video, through experiments they have shown their model performs better than the existing models. \n\nThe CLEVRER dataset is designed to test a model capability to answer the queries involving causal relationship between the objects involved in the video. \n\nDetails.\nThis work begins with a well motivated problem by pointing out the drawback in existing VQA(Visual Question Answering)  models, that existing works focus on visual and input language patterns to answer the queries and doesn’t tackle the task involving causal structure.  It  explores the current literature around the problem related to visual question answering(both real world data and simulated data). Through experiments they have shown the existing state-of-the-art work on visual question answering doesn’t perform well on the dataset CLEVRER. \n\nTo prove the incompetence of existing models to capture causal structure, authors designed an artificial dataset with questions which can be answered only when the model is capable of capturing the causal structure between the objects.  \n\nThe process involving the creation of CLEVRER dataset is well explained. But, it is unclear how the questions are generated. \n\nThrough experiments authors revealed the drawbacks of existing models on capturing the dynamics between the objects and  proposed a model which is said to be inspired by previous VQA[1] model. An important modification by incorporating neural dynamics predictor module to the existing model is key, and also achieves good performance on the dataset. \n\nComments:\n\n- The paper is well written, but it is unclear how the questions are generated during dataset creation process.\n- The main contribution of the paper is to show the incompetence of existing models to capture dynamics. Which is a form of analysis.\n- It is shown that,  learning dynamics of the objects the model can achieve better performance.\n- This dataset is created in more restricted environment like height of the objects should be same. How can this be generalized to a more real world setting ?\n\nSome minor issues:\nIn few places causal is misspelled as casual(page 2,8).\nEquation 2, in the appendix the subscripts are not proper. \n\n[1] Yi, Kexin, et al. \"Neural-symbolic vqa: Disentangling reasoning from vision and language understanding.\" Advances in Neural Information Processing Systems. 2018.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "H1lez5_nFr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies the temporal and causal structures in videos. Specifically, the authors first introduce a new dataset called CLEVRER drawing motivation from CLEVR, a well-known visual reasoning dataset. They further evaluate a set of state-of-the-art methods on the newly introduced dataset to confirm their initial beliefs of the challenges posed by causal reasoning. Based on empirical clues, they also suggest neural-symbolic based framework for causal reasoning.\nOn the whole, I think this is a good paper and it addresses one of the most challenging and exciting tasks in visual reasoning. The introduction of the CLEVRER dataset is likely valuable for the community to facilitate research in this area. I have some concerns as follows:\n(1) As the fact that questions are generated algorithmically, I believe many of them are either ill-posed or degenerate similar to what described in the CLEVR. Did you manage to filter out those questions? Please provide more details on the question generation process.\n(2) I have a doubt on the reported results in table 2 as you extract visual features not fairly between all methods. Outputs of pool5 feature basically kill all spatial information while 14x14 feature map used as input of MAC, IEP and TbD-net keep the spatial information. I also not sure if the temporal attention is a better option than vectorizing the whole video features (Some pooling layers might be helpful) before feeding into those methods. It would be fairer to evaluate the state of the art methods on the CLEVRER more carefully.\n(3) As the number of objects in CLEVRER is limited, using class labels from Mask R-CNN may introduce noises to the model due to incorrect detections. The authors may need to explain more clearly at this point in the implementation details.\n(4) Have you tried to incorporate flow features, for example, C3D/I3D feature?\n\nMinor comments:\nThere are some typos in the paper in both main paper and supplementary document -causal vs. casual. Please fix these.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BJesxs8dYS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A new benchmark is presented, which requires to reason on spatio-temporal data (videos) and incorporates physics (videos of physical dynamics), language (questions are formulated through language) and causality. The benchmark builds on the well-known CLEVR benchmark and adds several interesting contributions, in particular reasoning over time.\n\nCompared to other benchmarks proposed on reasoning on physical dynamics, this benchmark adds to the language component (which is presented in classical VQA benchmarks), and an interesting counterfactual component known from the causal inference literature. This is fairly new for the computer vision and statistical machine learning literature (a few papers on counterfactual reasoning exist), but in contrast to the causal inference literature, here the do-operator is observable during training: we do have access to do-interventions and even the outcome during training and therefore we can even learn counterfactuals using supervised learning. This statement is not meant as criticism, as learning counterfactuals without supervision from high-dimensional input like images seems to be currently out of a reach, or at least has not yet been demonstrated up to my knowledge.\n\nOn the downside, compared to other physical reasoning benchmarks, the answers here are multiple choice instead of regressions of the future motion, which requires more fine-grained reasoning. This will guide solutions to a certain type and will also favor solutions, requiring the network to detect certain binary concepts and combine them instead of regressing complex functions. It also favors solutions of the type presented in the paper.\n\nAs for other benchmarks, the dataset is quite large (20 000 videos) and, given its synthetic nature, is accompanied by functional programs. As for CLEVR, the simulations have been performed by a physics engine and then separately rendered with Blender to maximize visual quality. The result is a very interesting benchmark, and I have no doubt that it will be very useful for us (the learning reasoning community).\n\nAnother positive point is the number of baselines tested on the benchmark, among which we can find strong papers on VQA/VQA2\n\nI have a couple of questions:\n\nHow is the causal graph created? Are the experiments rendered and outcomes examined, creating the causal graph for the counterfactual answers, or are the experiments selected with a given outcome already decided?\n\nThe distribution of question types has been provided, but how about the biases? Distributions of the answers would have been helpful. How did the authors avoid biases during construction of the dataset, in particular in the counterfactual case (see remarks on the causal graph).\n\nThe paper also comes with a method for solving, which is very similar to existing methods on learning through functional programs. The method itself is unfortunately described only very briefly and the reader needs to look it up almost entirely in the appendix of the paper, which is surprising, as the paper is only 8 pages long instead of 10.\n\nAs for CLEVR, one of the downsides of this type of benchmark is the synthetic nature of the images and the limited range of different objects in the scene. Of course this comes with the advantage of being able to study compositional reasoning in detail, as a scene graph can be calculated easily (and is available during simulation). However, it also makes reasoning through functional programs much easier, as the proposed filters are limited in number and can strong respond to the small number of shapes and colors available in the data. I have strong doubts that this kind of approach extends to real life scenarios.\n\nFor VQA type of scenarios, GQA is a nice compromise between natural looking images and the availability of scene graphs and the restriction of questions to compositional reasoning. The optimal choice would be a similar compromise for spatio-temporal data, but of course this would be a huge effort and it would be up to impossible to have access to counterfactuals.\n\nLast point, and this question is not restricted to this paper, as the name came up elsewhere, why is the model called neuro-symbolic reasoning? While it could be argued that the questions require a sort of “symbolic reasoning”, I am not sure that the reasoning method itself is symbolic even partially. Other than selecting functional programs out of a discrete set, the reasoning itself is connectionist and performed with graph networks.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are unanimous in their opinion that this paper offers a novel approach to causal learning.  I concur.
Decision: Accept (Spotlight)