Title: CLEVRER: COLLISION EVENTS FOR VIDEO REPRESENTATION AND REASONING

Abstract: The ability to reason about temporal and causal events from videos lies at the core of human intelligence. Most video reasoning benchmarks, however, focus on pattern recognition from complex visual and language input, instead of on causal structure. We study the complementary problem, exploring the temporal and causal structures behind videos of objects with simple visual appearance. To this end, we introduce the CoLlision Events for Video REpresentation and Reasoning (CLEVRER) dataset, a diagnostic video dataset for systematic evaluation of computational models on a wide range of reasoning tasks. Motivated by the theory of human causal judgment, CLEVRER includes four types of question: descriptive (e.g., 'what color'), explanatory ('what's responsible for'), predictive ('what will happen next'), and counterfactual ('what if'). We evaluate various state-of-the-art models for visual reasoning on our benchmark. While these models thrive on the perceptionbased task (descriptive), they perform poorly on the causal tasks (explanatory, predictive and counterfactual), suggesting that a principled approach for causal reasoning should incorporate the capability of both perceiving complex visual and language inputs, and understanding the underlying dynamics and causal relations. We also study an oracle model that explicitly combines these components via symbolic representations.

1. INTRODUCTION:

The ability to recognize objects and reason about their behaviors in physical events from videos lies at the core of human cognitive development (Spelke, 2000) . Humans, even young infants, group segments into objects based on motion, and use concepts of object permanence, solidity, and continuity to explain what has happened, infer what is about to happen, and imagine what would happen in counterfactual situations. The problem of complex visual reasoning has been widely studied in artificial intelligence and computer vision, driven by the introduction of various datasets on both static images (Antol et al., 2015; Zhu et al., 2016; Hudson & Manning, 2019) and videos (Jang et al., 2017; Tapaswi et al., 2016; Zadeh et al., 2019) . However, despite the complexity and variety of the visual context covered by these datasets, the underlying logic, temporal and causal structure behind the reasoning process is less explored.

In this paper, we study the problem of temporal and causal reasoning in videos from a complementary perspective: inspired by a recent visual reasoning dataset, CLEVR (Johnson et al., 2017a) , we simplify the problem of visual recognition, but emphasize the complex temporal and causal structure behind the interacting objects. We introduce a video reasoning benchmark for this problem, drawing inspirations from developmental psychology (Gerstenberg et al., 2015; Ullman, 2015) . We also evaluate and assess limitations of various current visual reasoning models on the benchmark.

Our benchmark, named CoLlision Events for Video REpresentation and Reasoning (CLEVRER), is a diagnostic video dataset for temporal and causal reasoning under a fully controlled environment. The design of CLEVRER follows two guidelines: first, the posted tasks should focus on logic reasoning in the temporal and causal domain while staying simple and exhibiting minimal biases on visual scenes and language; second, the dataset should be fully controlled and well-annotated in order to host the Figure 1 : Sample video, questions, and answers from our CoLlision Events for Video REpresentation and Reasoning (CLEVRER) dataset. CLEVRER is designed to evaluate whether computational models can understand what is in the video (I, descriptive questions), explain the cause of events (II, explanatory), predict what will happen in the future (III, predictive), and imagine counterfactual scenarios (IV, counterfactual). In the four images (a-d), only for visualization purposes, we apply stroboscopic imaging to reveal object motion. The captions (e.g., 'First collision') are for the readers to better understand the frames, not part of the dataset. complex reasoning tasks and provide effective diagnostics for models on those tasks. CLEVRER includes 20,000 synthetic videos of colliding objects and more than 300,000 questions and answers (Figure 1 ). We focus on four specific elements of complex logical reasoning on videos: descriptive (e.g., 'what color'), explanatory ('what's responsible for'), predictive ('what will happen next'), and counterfactual ('what if'). CLEVRER comes with ground-truth motion traces and event histories of each object in the videos. Each question is paired with a functional program representing its underlying logic. As summarized in table 1, CLEVRER complements existing visual reasoning benchmarks on various aspects and introduces several novel tasks.

We also present analysis of various state-of-the-art visual reasoning models on CLEVRER. While these models perform well on descriptive questions, they lack the ability to perform causal reasoning and struggle on the explanatory, predictive, and counterfactual questions. We therefore identify three key elements that are essential to the task: recognition of the objects and events in the videos; modeling the dynamics and causal relations between the objects and events; and understanding of the symbolic logic behind the questions. As a first-step exploration of this principle, we study an oracle model, Neuro-Symbolic Dynamic Reasoning (NS-DR), that explicitly joins these components via a symbolic video representation, and assess its performance and limitations.

2. RELATED WORK:

Our work can be uniquely positioned in the context of three recent research directions: video understanding, visual question answering, and physical and causal reasoning. Video understanding. With the availability of large-scale video datasets (Caba Heilbron et al., 2015; Kay et al., 2017) , joint video and language understanding tasks have received much interest. This includes video captioning (Guadarrama et al., 2013; Venugopalan et al., 2015; Gan et al., 2017) , localizing video segments from natural language queries (Gao et al., 2017; Hendricks et al., 2017) , and video question answering. In particular, recent papers have explored different approaches to acquire and ground various reasoning tasks to videos. Among those, MovieQA (Tapaswi et al., 2016) , TGIF-QA (Jang et al., 2017) , TVQA (Lei et al., 2018) are based on real-world videos and human-generated questions. Social-IQ (Zadeh et al., 2019) discusses causal relations in human social interactions based on real videos. COG (Yang et al., 2018) and MarioQA (Mun et al., 2017) use simulated environments to generate synthetic data and controllable reasoning tasks. Compared to them, CLEVRER focuses on the causal relations grounded in object dynamics and physical interactions, and introduces a wide range of tasks including description, explanation, prediction and counterfactuals. CLEVRER also emphasizes compositionality in the visual and logic context. Visual question answering. Many benchmark tasks have been introduced in the domain of visual question answering. The Visual Question Answering (VQA) dataset (Antol et al., 2015) marks an important milestone towards top-down visual reasoning, based on large-scale cloud-sourced real images and human-generated questions. The CLEVR dataset (Johnson et al., 2017a ) follows a bottomup approach by defining the tasks under a controlled close-domain setup of synthetic images and questions with compositional attributes and logic traces. More recently, the GQA dataset (Hudson & Manning, 2019) applies synthetic compositional questions to real images. The VCR dataset (Zellers 

3. THE CLEVRER DATASET:

The CLEVRER dataset studies temporal and causal reasoning on videos. It is carefully designed in a fully-controlled synthetic environment, enabling complex reasoning tasks, providing effective diagnostics for models while simplifying video recognition and language understanding. The videos describe motion and collisions of objects on a flat tabletop (as shown in Figure 1 ) simulated by a physics engine, and are associated with the ground-truth motion traces and histories of all objects and events. Each video comes with four types of questions generated by machine, including descriptive ('what color', 'how many'), explanatory ('What is responsible for'), predictive ('What will happen next'), and counterfactual ('what if'). Each question is paired with a functional program.

3.1. VIDEOS:

CLEVRER includes 10,000 videos for training, 5,000 for validation, and 5,000 for testing. All videos last for 5 seconds. The videos are generated by a physics engine that simulates object motion plus a graphs engine that renders the frames. Extra examples from the dataset can be found in supplementary material C. Objects and events. Objects in CLEVRER videos adopt similar compositional intrinsic attributes as in CLEVR (Johnson et al., 2017a) , including three shapes (cube, sphere, and cylinder), two materials (metal and rubber), and eight colors (gray, red, blue, green, brown, cyan, purple, and

How many metal objects are moving when the video ends?:

Filter material

Objects:



Metal:



Filter moving:



Events:

Filter end

Get frame:



Count:

Without the gray object, which event will happen?

Filter color:





Gray:



Get counterfacts:

Belong to

Choice program:

The cube collides with the red object.

Cube Objects Objects:

Filter shape yellow). All objects have the same size so no vertical bouncing occurs during collision. In each video, we prohibit identical objects, such that each combination of the three attributes uniquely identifies one object. Under this constraint, all intrinsic attributes for each object are sampled randomly. We further introduce three types of events: enter, exit and collision, each of which contains a fixed number of object participants: 2 for collision and 1 for enter and exit. The objects and events form an abstract representation of the video. These ground-truth annotations, together with the object motion traces, enable model diagnostics, one of the key advantages offered by a fully controlled environment. Causal structure. Objects and events in CLEVRER videos exhibit rich causal structures. An event can be either caused by an object if the event is the first one participated by the object, or another event if the cause event happens right before the outcome event on the same object. For example, if a sphere collides with a cube and then a cylinder, then the first collision and the cube jointly "cause" the second collision. The object motion traces with complex causal structures are generated by the following recursive process. We start with one randomly initialized moving object, and then add another object whose initial position and velocity are set such that it will collide with the first object. The same process is then repeated to add more objects and collisions to the scene. All collision pairs and motion trajectories are randomly chosen. We discard simulations with repetitive collisions between the same pair of objects. Video generation. CLEVRER videos are generated from the simulated motion traces, including each object's position and pose at each time step. We use the Bullet (Coumans, 2010) physics engine for motion simulation. Each simulation lasts for seven seconds. The motion traces are first down-sampled to fit the frame rate of the output video (25 frames per second). Then the motion of the first five seconds are sent to Blender (Blender Online Community, 2016) to render realistic video frames of object motion and collision. The remaining two seconds are held-out for predictive tasks. We further note CLEVRER adopts the same software and parameters for rendering as CLEVR. We pair each video with machine-generated questions for descriptive, explanatory, predictive, and counterfactual reasoning. Sample questions of the four types can be found in Figure 1 . Each question is paired with a functional program executable on the video's dynamical scene. Unlike CLEVR (Johnson et al., 2017a) , our questions focus on the temporal and causal aspects of the objects and events in the videos. We exclude all questions on static object properties, which can be answered by looking at a single frame. CLEVRER consists of 219,918 descriptive questions, 33,811 explanatory questions, 14,298 predictive questions and 37,253 counterfactual questions. Detailed distribution and split of the questions can be found in Figure 3 and supplementary material A.



Red:



Filter collision:







Question program:



3.2. QUESTIONS:



Descriptive. Descriptive questions evaluate a model's capability to understand and reason about a video's dynamical content and temporal relation. The reasoning tasks are grounded to the compositional space of both object and event properties, including intrinsic attributes (color, material, shape), motion, collision, and temporal order. All descriptive questions are 'open-ended' and can be answered by a single word. Descriptive questions contain multiple sub-types including count, exist, query color, query material, and query shape. Distribution of the sub-types is shown in Figure 3 . We evenly sample the answers within each sub-type to reduce answer bias.

Explanatory. Explanatory questions query the causal structure of a video by asking whether an object or event is responsible for another event. Event A is responsible for event B if A is among B's ancestors in the causal graph. Similarly, object O is responsible for event A if O participates in A or any other event responsible for A. Explanatory questions are multiple choice questions with at most four options, each representing an event or object in the video. Models need to select all options that match the question's description. There can be multiple correct options for each question. We sample the options to balance the number of correct and wrong ones, and minimize text-only biases.

Predictive. Predictive questions test a model's capability of predicting possible occurrences of future events after the video ends. Similar to explanatory questions, predictive questions are multiplechoice, whose options represent candidate events that will or will not happen. Because post-video events are sparse, we provide two options for each predictive question to reduce bias.

Counterfactual. Counterfactual questions query the outcome of the video under certain hypothetical conditions (e.g. removing one of the objects). Models need to select the events that would or would not happen under the designated condition. There are at most four options for each question.

The numbers of correct and incorrect options are balanced. Both predictive and counterfactual questions require knowledge of object dynamics underlying the videos and the ability to imagine and reason about unobserved events.

Program representation. In CLEVRER, each question is represented by a tree-structured functional program, as shown in Figure 2 . A program begins with a list of objects or events from the video. The list is then passed through a sequence of filter modules, which select entries from the list and join the tree branches to output a set of target objects and events. Finally, an output module is called to query a designated property of the target outputs. For multiple choice questions, each question and option correspond to separate programs, which can be jointly executed to output a yes/no token that indicates if the choice is correct for the question. A list of all program modules can be found in the supplementary material B. Question generation. Questions in CLEVRER are generated by a multi-step procedure. For each question type, a pre-defined logic template is chosen. The logic template can be further populated by attributes that are associated with the context of the video (i.e. the color, material, shape that identifies the object to be queried) and then turned into natural language. We first generate an exhaustive list of all possible questions for each video. To avoid language bias, we sample from that list to maintain a balanced answer distribution for each question type and minimize the correlation between the questions and answers over the entire dataset.

4. BASELINE EVALUATION:

In this section, we evaluate and analyse the performances of a wide range of baseline models for video reasoning on CLEVRER. For descriptive questions, the models treat each question as a multi-class classification problem over all possible answers. For multiple choice questions, each question-choice pair is treated as a binary classification problem indicating the correctness of the choice.

4.1. MODEL DETAILS:

The baseline models we evaluate fall into three families: language-only models, models for video question answering, and models for compositional visual reasoning.

Language-only models. This model family includes weak baselines that only relies on question input to assess language biases in CLEVRER. Q-type (random) uniformly samples an answer from the answer space or randomly select each choice for multiple-choice questions. Q-type (frequent) chooses the most frequent answer in the training set for each question type. LSTM uses a pretrained word embedding trained on the Google News corpus (Mikolov et al., 2013) to encode the input question and processes the sequence with a LSTM (Hochreiter & Schmidhuber, 1997) . A MLP is then applied to the final hidden state to predict a distribution over the answers. Video question answering. We also evaluate the following models that relies on both video and language inputs. CNN+MLP extracts features from the input video via a convolutional neural network (CNN) and encodes the question by taking the average of the pretrained word embeddings (Mikolov et al., 2013) . The video and language features are then jointly sent to a MLP for answer prediction. CNN+LSTM relies on the same architecture for video feature extraction but uses the final state of a LSTM for answer prediction. TVQA (Lei et al., 2018 ) introduces a multi-stream end-to-end neural model that sets the state of the art for video question answering. We apply attribute-aware object-centric features acquired by a video frame parser (TVQA+). We also include a recent model that incorporates heterogeneous memory with multimodal attention work (Memory) (Fan et al., 2019 ) that achieves superior performance on several datasets. ) follows a similar approach by parsing the input question into a program, which is then assembled into a neural network that acts on the attention map over the image features. The final attended image feature is then sent to an output layer for classification. We adopt the same approach through spatial-temporal attention over the video feature space (TbD-net (V)). MAC (Hudson & Manning, 2018) incorporates a joint attention mechanism on both the image feature map and the question, which leads to strong performance on CLEVR without program supervision. We modify the model by applying a temporal attention unit across the video frames to generate a latent encoding for the video (MAC (V)). The video feature is then input to the MAC network to output an answer distribution. We study an augmented approach: we construct object-aware video features by adding the segmentation masks of all objects in the frames and labeling them by the values of their intrinsic attributes (MAC (V+)). Implementation details. We use a pre-trained ResNet-50 (He et al., 2016) to extract features from the video frames. We use the 2,048-dimensional pool5 layer output for CNN-based methods, and the 14×14 feature maps for MAC, IEP and TbD-net. The program generators of IEP and TbD-net are trained on 1000 programs. We uniformly sample 25 frames for each video as input. Object segmentation masks and attributes are obtained by a video parser consisted of an object detector and an attribute network.

4.2. RESULTS:

We summarize the performances of all baseline models in Table 2 . The fact that these models achieve different performances over the wide spectrum of tasks suggest that CLEVRER offers powerful assessment to the models' strength and limitations on various domains. All models are trained on the training set until convergence, tuned on the validation set and evaluated on the test set. Evaluation metrics. For descriptive questions, we calculate the accuracy by comparing the predicted answer token to the ground-truth. For multiple choice questions, we adopt two metrics: per-option accuracy measures the model's overall correctness on single options across all questions; per-question accuracy measures the correctness of the full question, requiring all choices to be selected correctly. Descriptive reasoning. Descriptive questions query the content of the video from various aspects.

In order to do well on this question type, a model needs to both accurately recognize the objects and events that happen in the video, as well as understanding the compositional logic pattern behind the questions. In other words, descriptive questions require strong perception and logic operations on both visual and language signals. As shown in Table 2 , the LSTM baseline that relies only on question input performs poorly on the descriptive questions, only outperforming the random baselines < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 f q X a U z 0 A 3 x a i J y 7 q 2 K 5 

u Q U 1 8 Y 7 y X V T f N F v v 1 3 Q e 5 z 8 = " > A A A C F 3 i c b V C 7 T s M w F H V 4 l v I K M L J Y V E g M U C U V E o w V L G w U R B 9 S U 1 W O 6 7 R W H S e y b 5 C q q H / B w q + w M I A Q K 2 z 8 D W 6 a g b Y c y d L x O f d e + x 4 / F l y D 4 / x Y S 8 s r q 2 v r h Y 3 i 5 t b 2 z q 6 9 t 9 / Q U a I o q 9 N I R K r l E 8 0 E l 6 w O H A R r x Y q R 0 B e s 6 Q + v J 3 7 z k S n N I / k A o 5 h 1 Q t K X P O C U g J G 6 d t k T R P Y F w 7 f d F M 4 q X i 8 C j W F 8 i u 9 n 7 t h T W V n X L j l l J w N e J G 5 O S i h H r W t / m x E 0 C Z k E K o j W b d e J o Z M S B Z w K N i 5 6 i W Y x o U P S Z 2 1 D J Q m Z 7 q T Z X m N 8 b J Q e D i J l j g S c q X 8 7 U h J q P Q p 9 U x k S G O h 5 b y L + 5 7 U T C C 4 7 K Z d x A k z S 6 U N B I j B E e B I S 7 n H F K I i R I Y Q q b v 6 K 6 Y A o Q s F E W T Q h u P M r L 5 J G p e w a f n d e q l 7 l c R T Q I T p C J 8 h F F 6 i K b l A N 1 R F F T + g F v a F 3 6 9 l 6 t T 6 s z 2 n p k p X 3 H K A Z W F + / m M i e 8 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 R h p u Q U 1 8 Y 7 y X V T f N F v v 1 3 Q e 5 z 8 = " > A A A C F 3 i c b V C 7 T s M w F H V 4 l v I K M L J Y V E g M U C U V E o w V L G w U R B 9 S U 1 W O 6 7 R W H S e y b 5 C q q H / B w q + w M I A Q K 2 z 8 D W 6 a g b Y c y d L x O f d e + x 4 / F l y D 4 / x Y S 8 s r q 2 v r h Y 3 i 5 t b 2 z q 6 9 t 9 / Q U a I o q 9 N I R K r l E 8 0 E l 6 w O H A R r x Y q R 0 B e s 6 Q + v J 3 7 z k S n N I / k A o 5 h 1 Q t K X P O C U g J G 6 d t k T R P Y F w 7 f d F M 4 q X i 8 C j W F 8 i u 9 n 7 t h T W V n X L j l l J w N e J G 5 O S i h H r W t / m x E 0 C Z k E K o j W b d e J o Z M S B Z w K N i 5 6 i W Y x o U P S Z 2 1 D J Q m Z 7 q T Z X m N 8 b J Q e D i J l j g S c q X 8 7 U h J q P Q p 9 U x k S G

u Q U 1 8 Y 7 y X V T f N F v v 1 3 Q e 5 z 8 = " > A A A C F 3 i c b V C 7 T s M w F H V 4 l v I K M L J Y V E g M U C U V E o w V L G w U R B 9 S U 1 W O 6 7 R W H S e y b 5 C q q H / B w q + w M I A Q K 2 z 8 D W 6 a g b Y c y d L x O f d e + x 4 / F l y D 4 / x Y S 8 s r q 2 v r h Y 3 i 5 t b 2 z q 6 9 t 9 / Q U a I o q 9 N I R K r l E 8 0 E l 6 w O H A R r x Y q R 0 B e s 6 Q + v J 3 7 z k S n N I / k A o 5 h 1 Q t K X P O C U g J G 6 d t k T R P Y F w 7 f d F M 4 q X i 8 C j W F 8 i u 9 n 7 t h T W V n X L j l l J w N e J G 5 O S i h H r W t / m x E 0 C Z k E K o j W b d e J o Z M S B Z w K N i 5 6 i W Y x o U P S Z 2 1 D J Q m Z 7 q T Z X m N 8 b J Q e D i J l j g S c q X 8 7 U h J q P Q p 9 U x k S G

u Q U 1 8 Y 7 y X V T f N F v v 1 3 Q e 5 z 8 = " > A A A C F 3 i c b V C 7 T s M w F H V 4 l v I K M L J Y V E g M U C U V E o w V L G w U R B 9 S U 1 W O 6 7 R W H S e y b 5 C q q H / B w q + w M I A Q K 2 z 8 D W 6 a g b Y c y d L x O f d e + x 4 / F l y D 4 / x Y S 8 s r q 2 v r h Y 3 i 5 t b 2 z q 6 9 t 9 / Q U a I o q 9 N I R K r l E 8 0 E l 6 w O H A R r x Y q R 0 B e s 6 Q + v J 3 7 z k S n N I / k A o 5 h 1 Q t K X P O C U g J G 6 d t k T R P Y F w 7 f d F M 4 q X i 8 C j W F 8 i u 9 n 7 t h T W V n X L j l l J w N e J G 5 O S i h H r W t / m x E 0 C Z k E K o j W b d e J o Z M S B Z w K N i 5 6 i W Y x o U P S Z 2 1 D J Q m Z 7 q T Z X m N 8 b J Q e D i J l j g S c q X 8 7 U h J q P Q p 9 U x k S G

I L v W N 0 = " > A A A C F X i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 U Q L C U R Q Z d F N + 6 s Y h / Q h D C Z T t u h k 0 m Y u R F K 6 E + 4 8 V f c u F D E r e D O v 3 H a Z q G t B

+ v G N I 2 Y B C q I 1 h 3 X S c D P i A J O B R u X v F S z h N A h 6 b O O o Z J E T P v Z 9 K o x P j J K F / d i Z Z 4 E P F V / T 2 Q k 0 n o U h a Y z I j D Q 8 9 5 E / M / r p N C 7 8 D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I 0 M I V d z 8 F d M B U Y S C C b J k Q n D n T 1 4 k z d O q a / j t W b l 2 m c d R R A f o E B 0 j F 5 2 j G r p G d d R A F D 2

i Z / S K 3 q w n 6 8 V 6 t z 5 m r Q U r n 9 l H f 2 B 9 / g D o P p 6 k < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 f q X a U z 0 A 3 x a i J y 7 q 2 K 5 

I L v W N 0 = " > A A A C F X i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 U Q L C U R Q Z d F N + 6 s Y h / Q h D C Z T t u h k 0 m Y u R F K 6 E + 4 8 V f c u F D E r e D O v 3 H a Z q G t B

+ v G N I 2 Y B C q I 1 h 3 X S c D P i A J O B R u X v F S z h N A h 6 b O O o Z J E T P v Z 9 K o x P j J K F / d i Z Z 4 E P F V / T 2 Q k 0 n o U h a Y z I j D Q 8 9 5 E / M / r p N C 7 8 D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I 0 M I V d z 8 F d M B U Y S C C b J k Q n D n T 1 4 k z d O q a / j t W b l 2 m c d R R A f o E B 0 j F 5 2 j G r p G d d R A F D 2

i Z / S K 3 q w n 6 8 V 6 t z 5 m r Q U r n 9 l H f 2 B 9 / g D o P p 6 k < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 f q X a U z 0 A 3 x a i J y 7 q 2 K 5 

I L v W N 0 = " > A A A C F X i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 U Q L C U R Q Z d F N + 6 s Y h / Q h D C Z T t u h k 0 m Y u R F K 6 E + 4 8 V f c u F D E r e D O v 3 H a Z q G t B

+ v G N I 2 Y B C q I 1 h 3 X S c D P i A J O B R u X v F S z h N A h 6 b O O o Z J E T P v Z 9 K o x P j J K F / d i Z Z 4 E P F V / T 2 Q k 0 n o U h a Y z I j D Q 8 9 5 E / M / r p N C 7 8 D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I 0 M I V d z 8 F d M B U Y S C C b J k Q n D n T 1 4 k z d O q a / j t W b l 2 m c d R R A f o E B 0 j F 5 2 j G r p G d d R A F D 2

i Z / S K 3 q w n 6 8 V 6 t z 5 m r Q U r n 9 l H f 2 B 9 / g D o P p 6 k < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 f q X a U z 0 A 3 x a i J y 7 q 2 K 5 < l a t e x i t s h a _ b a s e = " f f q X a U z A x a i J y q 

I L v W N 0 = " > A A A C F X i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 U Q L C U R Q Z d F N + 6 s Y h / Q h D C Z T t u h k 0 m Y u R F K 6 E + 4 8 V f c u F D E r e D O v 3 H a Z q G t B

G V t s u v B + X D o a J E s g U y u A G i F g U K H k p y G g e S t Y H w q c m X B s R q U e c x t w P V C J g W A U r e V R x T T O O u l e O F l v X L F r b p z k V X w c q h A r n q v / N X t R y w J u U I m q T E d z R T l G w S T P S t E J i y M R y j k V F Q d H R s s y e D S N u n k M z d x M p D Y Z h o H t D C m O z H J t Z v X y Q u P F T o e I E u W K L R Y N E E o z I L A H S F o z l F M L l G l h b y V s R D V l a H M q R C S + v Q v O y l l + u K r U b v M i n A C p A O

G V t s u v B + X D o a J E s g U y u A G i F g U K H k p y G g e S t Y H w q c m X B s R q U e c x t w P V C J g W A U r e V R x T T O O u l e O F l v X L F r b p z k V X w c q h A r n q v / N X t R y w J u U I m q T E d z R T l G w S T P S t E J i y M R y j k V F Q d H R s s y e D S N u n k M z d x M p D Y Z h o H t D C m O z H J t Z v X y Q u P F T o e I E u W K L R Y N E E o z I L A H S F o z l F M L l G l h b y V s R D V l a H M q R C S + v Q v O y l l + u K r U b v M i n A C p A O

G V t s u v B + X D o a J E s g U y u A G i F g U K H k p y G g e S t Y H w q c m X B s R q U e c x t w P V C J g W A U r e V R x T T O O u l e O F l v X L F r b p z k V X w c q h A r n q v / N X t R y w J u U I m q T E d z R T l G w S T P S t E J i y M R y j k V F Q d H R s s y e D S N u n k M z d x M p D Y Z h o H t D C m O z H J t Z v X y Q u P F T o e I E u W K L R Y N E E o z I L A H S F o z l F M L l G l h b y V s R D V l a H M q R C S + v Q v O y l l + u K r U b v M i n A C p A O

G V t s u v B + X D o a J E s g U y u A G i F g U K H k p y G g e S t Y H w q c m X B s R q U e c x t w P V C J g W A U r e V R x T T O O u l e O F l v X L F r b p z k V X w c q h A r n q v / N X t R y w J u U I m q T E d z R T l G w S T P S t E J i y M R y j k V F Q d H R s s y e D S N u n k M z d x M p D Y Z h o H t D C m O z H J t Z v X y Q u P F T o e I E u W K L R Y N E E o z I L A H S F o z l F M L l G l h b y V s R D V l a H M q R C S + v Q v O y l l + u K r U b v M i n A C p A O

K I L v W N = " > A A A C F X i c b V D L S s N A F J U V v q E s g U Q L C U R Q Z d F N + s Y h / Q h D C Z T t u h k m Y u R F K E + V f c u F D E r e D O v H a Z q G t B

+ v G N I Y B C q I h X S c D P i A J O B R u X v F S z h N A h b O O o Z J E T P v Z K o x P j J K F / d i Z Z E P F V / T Q k n o U h a Y z I j D Q E / M / r p N C D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I M I V d z F d M B U Y S C C b J k Q n D n T k z d O q a / j t W b l m c d R R A f o E B j F j G r p G d d R A F D i Z / S K q w n V t z m r Q U r n l H f B / g D o P p k < / l a t e x i t > < l a t e x i t s h a _ b a s e = " f f q X a U z A x a i J y q K I L v W N = " > A A A C F X i c b V D L S s N A F J U V v q E s g U Q L C U R Q Z d F N + s Y h / Q h D C Z T t u h k m Y u R F K E + V f c u F D E r e D O v H a Z q G t B

+ v G N I Y B C q I h X S c D P i A J O B R u X v F S z h N A h b O O o Z J E T P v Z K o x P j J K F / d i Z Z E P F V / T Q k n o U h a Y z I j D Q E / M / r p N C D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I M I V d z F d M B U Y S C C b J k Q n D n T k z d O q a / j t W b l m c d R R A f o E B j F j G r p G d d R A F D i Z / S K q w n V t z m r Q U r n l H f B / g D o P p k < / l a t e x i t > < l a t e x i t s h a _ b a s e = " f f q X a U z A x a i J y q K I L v W N = " > A A A C F X i c b V D L S s N A F J U V v q E s g U Q L C U R Q Z d F N + s Y h / Q h D C Z T t u h k m Y u R F K E + V f c u F D E r e D O v H a Z q G t B

+ v G N I Y B C q I h X S c D P i A J O B R u X v F S z h N A h b O O o Z J E T P v Z K o x P j J K F / d i Z Z E P F V / T Q k n o U h a Y z I j D Q E / M / r p N C D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I M I V d z F d M B U Y S C C b J k Q n D n T k z d O q a / j t W b l m c d R R A f o E B j F j G r p G d d R A F D i Z / S K q w n V t z m r Q U r n l H f B / g D o P p k < / l a t e x i t > < l a t e x i t s h a _ b a s e = " f f q X a U z A x a i J y q K I L v W N = " > A A A C F X i c b V D L S s N A F J U V v q E s g U Q L C U R Q Z d F N + s Y h / Q h D C Z T t u h k m Y u R F K E + V f c u F D E r e D O v H a Z q G t B

+ v G N I Y B C q I h X S c D P i A J O B R u X v F S z h N A h b O O o Z J E T P v Z K o x P j J K F / d i Z Z E P F V / T Q k n o U h a Y z I j D Q E / M / r p N C D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I M I V d z F d M B U Y S C C b J k Q n D n T k z d O q a / j t W b l m c d R R A f o E B j F j G r p G d d R A F D i Z / S K q w n V t z m r Q U r n l H f B / g D o P p k < / l a t e x i t > h Ôt+1, Rt+1i

< l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 f q X a U z 0 A 3 x a i J y 7 q 2 K 5 

I L v W N 0 = " > A A A C F X i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 U Q L C U R Q Z d F N + 6 s Y h / Q h D C Z T t u h k 0 m Y u R F K 6 E + 4 8 V f c u F D E r e D O v 3 H a Z q G t B

+ v G N I 2 Y B C q I 1 h 3 X S c D P i A J O B R u X v F S z h N A h 6 b O O o Z J E T P v Z 9 K o x P j J K F / d i Z Z 4 E P F V / T 2 Q k 0 n o U h a Y z I j D Q 8 9 5 E / M / r p N C 7 8 D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I 0 M I V d z 8 F d M B U Y S C C b J k Q n D n T 1 4 k z d O q a / j t W b l 2 m c d R R A f o E B 0 j F 5 2 j G r p G d d R A F D 2 i

Z / S K 3 q w n 6 8 V 6 t z 5 m r Q U r n 9 l H f 2 B 9 / g D o P p 6 k < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 f q X a U z 0 A 3 x a i J y 7 q 2 K 5 

I L v W N 0 = " > A A A C F X i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 U Q L C U R Q Z d F N + 6 s Y h / Q h D C Z T t u h k 0 m Y u R F K 6 E + 4 8 V f c u F D E r e D O v 3 H a Z q G t B

+ v G N I 2 Y B C q I 1 h 3 X S c D P i A J O B R u X v F S z h N A h 6 b O O o Z J E T P v Z 9 K o x P j J K F / d i Z Z 4 E P F V / T 2 Q k 0 n o U h a Y z I j D Q 8 9 5 E / M / r p N C 7 8 D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I 0 M I V d z 8 F d M B U Y S C C b J k Q n D n T 1 4 k z d O q a / j t W b l 2 m c d R R A f o E B 0 j F 5 2 j G r p G d d R A F D 2 i

Z / S K 3 q w n 6 8 V 6 t z 5 m r Q U r n 9 l H f 2 B 9 / g D o P p 6 k < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 f q X a U z 0 A 3 x a i J y 7 q 2 K 5 

I L v W N 0 = " > A A A C F X i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 U Q L C U R Q Z d F N + 6 s Y h / Q h D C Z T t u h k 0 m Y u R F K 6 E + 4 8 V f c u F D E r e D O v 3 H a Z q G t B

+ v G N I 2 Y B C q I 1 h 3 X S c D P i A J O B R u X v F S z h N A h 6 b O O o Z J E T P v Z 9 K o x P j J K F / d i Z Z 4 E P F V / T 2 Q k 0 n o U h a Y z I j D Q 8 9 5 E / M / r p N C 7 8 D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I 0 M I V d z 8 F d M B U Y S C C b J k Q n D n T 1 4 k z d O q a / j t W b l 2 m c d R R A f o E B 0 j F 5 2 j G r p G d d R A F D 2 i

Z / S K 3 q w n 6 8 V 6 t z 5 m r Q U r n 9 l H f 2 B 9 / g D o P p 6 k < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " f 2 f q X a U z 0 A 3 x a i J y 7 q 2 K 5 by a small margin. This suggests that CLEVRER has very small bias on the questions. Video QA models, including the state of the art model TVQA+ (Lei et al., 2018 ) achieve better performances. But because of their limited capability of handling the compositionality in the question logic and visual context, these models are still unable to thrive on the task. In contrast, models designed for compositional reasoning, including TbD-net that operates on neural program execution and MAC that introduces a joint attention mechanism, are able to achieve more competitive performances.

I L v W N 0 = " > A A A C F X i c b V D L S s N A F J 3 U V 6 2 v q E s 3 g 0 U Q L C U R Q Z d F N + 6 s Y h / Q h D C Z T t u h k 0 m Y u R F K 6 E + 4 8 V f c u F D E r e D O v 3 H a Z q G t B

+ v G N I 2 Y B C q I 1 h 3 X S c D P i A J O B R u X v F S z h N A h 6 b O O o Z J E T P v Z 9 K o x P j J K F / d i Z Z 4 E P F V / T 2 Q k 0 n o U h a Y z I j D Q 8 9 5 E / M / r p N C 7 8 D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I 0 M I V d z 8 F d M B U Y S C C b J k Q n D n T 1 4 k z d O q a / j t W b l 2 m c d R R A f o E B 0 j F 5 2 j G r p G d d R A F D 2 i Z / S K

Causal reasoning. Results on the descriptive questions demonstrate the power of models that combine visual and language perception with compositional logic operations. However, the causal reasoning tasks (explanatory, predictive, counterfactual) require further understanding beyond perception. Our evaluation results (Table 2 ) show poor performance of most baseline models on these questions. The compositional reasoning models that performs well on the descriptive questions (MAC (V) and TbD-net (V)) only achieve marginal improvements over the random and language-only baselines on the causal tasks. However, we do notice a reasonable gain in performance on models that inputs object-aware representations: TVQA+ achieves high accuracy on the predictive questions, and MAC (V+) improves upon MAC (V) on all tasks.

We identify the following messages suggested by our evaluation results. First, object-centric representations are essential for the causal reasoning tasks. This is supported by the large improvement on MAC (V+) over MAC (V) after using features that are aware of object instances and attributes, as well as the strong performance of TVQA+ on the predictive questions. Second, all baseline models lack a component to explicitly model the dynamics of the objects and the causal relations between the collision events. As a result, they struggle on the tasks involving unobserved scenes and in particular performs poorly on the counterfactual tasks. The combination of object-centric representation and dynamics modeling therefore suggests a promising direction for approaching the causal tasks.

5. NEURO-SYMBOLIC DYNAMIC REASONING:

Baseline evaluations on CLEVRER have revealed two key elements that are essential to causal reasoning: an object-centric video representation that is aware of the temporal and causal relations between the objects and events; and a dynamics model able to predict the object dynamics under unobserved or counterfactual scenarios. However, unifying these elements with video and language understanding posts the following challenges: first, all the disjoint model components should operate on a common set of representations of the video, question, dynamics and causal relations; second, the representation should be aware of the compositional relations between the objects and events. We draw inspirations from Yi et al. (2018) and study an oracle model that operates on a symbolic representation to join video perception, language understanding with dynamics modeling. Our model Neuro-Symbolic Dynamic Reasoning (NS-DR) combines neural nets for pattern recognition and dynamics prediction, and symbolic logic for causal reasoning. As shown in Figure 4 , NS-DR consists of a video frame parser (Figure 4 -I), a neural dynamics predictor (Figure 4 -II), a question parser (Figure 4 -III), and a program executor (Figure 4 -IV). We present details of the model components below and in supplementary material B.

Video frame parser. The video frame parser serves as a perception module that is disentangled from other components of the model, from which we obtain an object-centric representation of the video frames. The parser is a Mask R-CNN (He et al., 2017 ) that performs object detection and scene de-rendering on each video frame (Wu et al., 2017) . We use a ResNet-50 FPN (Lin et al., 2017; He et al., 2016) as the backbone. For each input video frame, the network outputs the object mask proposals, a label that corresponds to the intrinsic attributes (color, material, shape) and a score representing the confidence level of the proposal used for filtering. Please refer to He et al. (He et al., 2017) for more details. The video parser is trained on 4000 video frames randomly sampled from the training set with object masks and attribute annotations. Neural dynamics predictor. We apply the Propagation Network (PropNet) (Li et al., 2019) for dynamics modeling, which is a learnable physics engine that, extending from the interaction networks (Battaglia et al., 2016) , performs object-and relation-centric updates on a dynamic scene. The model inputs the object proposals from the video parser, and learns the dynamics of the objects across the frames for predicting motion traces and collision events. PropNet represents the dynamical system as a directed graph, G = O, R , where the vertices O = {o i } represent objects and edges R = {r k } represent relations. Each object (vertex) o i and relation (edge) r k can be further written as

o i = s i , a o i , r k = u k , v k , a r k

, where s i is the state of object i; a o i denotes its intrinsic attributes; u k , v k are integers denoting the index of the receiver and sender vertices joined by edge r k ; a r k represents the state of edge r k , indicating whether there is collision between the two objects. In our case, s i is a concatenation of tuple c i , m i , p i over a small history window to encode motion history, where c i and m i are the corresponding image and mask patches cropped at p i , which is the x-y position of the mask in the original image (please see the cyan metal cube in Figure 4 for an example). PropNet handles the instantaneous propagation of effects via multi-step message passing. Please refer to supplementary material B for more details. Question parser. We use an attention-based seq2seq model (Bahdanau et al., 2015) to parse the input questions into their corresponding functional programs. The model consists of a bidirectional LSTM encoder plus an LSTM decoder with attention, similar to the question parser in (Yi et al., 2018) . For multiple choice questions, we use two networks to parse the questions and choices separately. Given an input word sequence (x 1 , x 2 , ...x I ), the encoder first generates a bi-directional latent encoding at each step:

EQUATION

EQUATION

EQUATION

The decoder then generates a sequence of program tokens (y 1 , y 2 , ..., y J ) from the latent encodings using attention:

EQUATION

EQUATION

EQUATION

At training time, the input program tokens {y j } are used for generating the predicted labels y j-1 → ŷj at each step j = 1, 2, 3, ..., J. The generated label ŷj is then compared with y j to compute a loss for training. At test time, the decoder rolls out by feeding the sampled prediction at the previous step to the input of the current step ŷj = y j . The roll-out stops when the end token appears or when the sequence reaches certain maximal length. For both the encoder and decoder LSTM, we use the same parameter setup of two hidden layers with 256 units, and a 300-dimensional word embedding vector.

Program executor. The program executor explicitly executes the program on the motion and event traces extracted by the dynamics predictor and output an answer to the question. It consists of a collection of functional modules implemented in Python. Given an input program, the executor first assembles the modules and then iterate through the program tree. The output of the final module is the answer to the target question. There are three types of program modules: input module, filter module, and output module. The input modules are the entry points of the program trees and can directly query the output of the dynamics predictor. For example, the Events and Objects modules that commonly appear in the programs of descriptive questions indicate inputting all the observed events and objects from a video. The filter modules perform logic operations on the input objects/events based on a designated intrinsic attribute, motion state, temporal order, or causal relation. The output modules return the answer label. For open-ended descriptive questions, the executor will return a token from the answer space. For multiple choice questions, the executor will first execute the choice program and then send the result to the question program to output a yes/no token. A comprehensive list of all modules in the NS-DR program executor is presented in supplementary material B.

Results. We summarize the performance of NS-DR on CLEVRER in Table 3 . On descriptive questions, our model achieves an 88.1% accuracy when the question parser is trained under 1,000 programs, outperforming other baseline methods. On explanatory, predictive, and counterfactual questions, our model achieves a more significant gain. We also study a variation of our model, NS-DR (No Events/NE), which uses the dynamics predictor to generate only the predictive and counterfactual motion traces, and collision events are identified by the velocity change of the colliding objects. NS-DR (NE) performs comparably to the full model, showing that our disentangled system is adaptable to alternative methods for dynamics modeling. We also conduct ablation study on the number of programs for training the question parser. As shown in Figure 5 , NS-DR reaches full capability at 1,000 programs for all question types.

We highlight the following contributions of the model. First, NS-DR incorporates a dynamics planner into the visual reasoning task, which directly enables predictions of unobserved motion and events, and enables the model for the predictive and counterfactual tasks. This suggests that dynamics planning has great potential for language-grounded visual reasoning tasks and NS-DR takes a preliminary step towards this direction. Second, symbolic representation provides a powerful common ground for vision, language, dynamics and causality. By design, it empowers the model to explicitly capture the compositionality behind the video's causal structure and the question logic.

# Programs # Programs:

Test accuracy (%)

Test accuracy (%) 

Descriptive Explanatory Predictive Counterfactual

6. CONCLUSION:

We present a systematic study of temporal and causal reasoning in videos. This profound and challenging problem deeply rooted to the fundamentals of human intelligence has just begun to be studied with 'modern' AI tools. We introduce a set of benchmark tasks to better facilitate the research in this area. We also believe video understanding and reasoning should go beyond passive knowledge extraction, and focus on building an internal understanding of the dynamics and causal relations, which is essential for practical applications such as dynamic robot manipulation under complex causal conditions. Our newly introduced CLEVRER dataset and the NS-DR model are preliminary steps toward this direction. We hope that with recent advances in graph networks, visual predictive models, and neuro-symbolic algorithms, the deep learning community can now revisit this classic problem in more realistic setups in the future, capturing true intelligence beyond pattern recognition. Answer: Sphere Question: Which event will happen without the cyan cylinder? Choice: The red rubber sphere collides with the yellow sphere Program: belong_to(filter_collision(filter_collision(Events, filter_shape(filter_material(filter_color( Objects, Red), Rubber), Sphere)), filter_shape(filter_color(Objects, Yellow), Sphere)), get_counterfact( filter_shape(filter_color(Objects, Cyan), Cylinder)))

Answer: The output of the neural dynamics predictor is { Ôt } t=1...T , { Rt } t=1...T , the collection of object states and relations across all the observed and rollout frames. From this representation, one can recover the full motion and event traces of the objects under different rollout conditions and generate a dynamic scene representation of the video. For example, if we want to know what will happen if an object is removed from the scene, we just need to erase the corresponding vertex and associated edges from the graph and rollout using the learned dynamics to obtain the traces.

Our neural dynamics predictor is trained on the proposed object masks and attributes from the frame parser. We filter out inconsistent object proposals by matching the object intrinsic attributes across different frames and keeping the ones that appear in more than 10 frames. For a video of length 125, we will sample 5 rollouts, where each rollout contains 25 frames that are uniformly sampled from the original video. We normalize the input data to the range of [-1, 1] and concatenate them over a time window of size 3. We set the propagation step L to 2, and the dimension of the propagating effects D enc to 512. We use the Adam optimizer (Kingma & Ba, 2015) with an initial learning rate of 10 -4 , and a decay factor of 0.3 per 3 epochs. The model is trained for 9 epochs with batch size 2. Question parser. For open-ended descriptive questions, the question parser is trained on various numbers of randomly selected question-program pairs. For multiple choice questions, we separately train the question parser and the choice parser. Training with n samples means training the question parser with n questions randomly sampled from the multiple choice questions (all three types together), and then training the choice parser with 4n choices sampled from the same pool. All models are trained using Adam (Kingma & Ba, 2015) for 30,000 iterations with batch size 64 and learning rate 7 × 10 -4 . Program executor. A comprehensive list of all modules in the NS-DR program executor is summarized in Table 4 and Table 5 . The input and output data types of each module are summarized in Table 6 .

C EXTRA EXAMPLES FROM CLEVRER:

We show extra examples from CLEVRER in Figure 8 . We also present visualizations of question/choice programs of different question types in Figure 9 through Figure 11 . The full dataset will be made available for download. Published as a conference paper at ICLR 2020 

Data type Description:



object:

A dictionary storing the intrinsic attributes of an object in a video objects A list of objects event A dictionary storing the type, frame, and participating objects of an event events A list of events order A string indicating the chronological order of an event out of "first", "second", "last" color A string indicating a color out of "gray", "brown", "green", "red", "blue", "purple", "yellow", "cyan" material A string indicating a material out of "metal", "rubber" shape A string indicating a shape out of "cube", "cylinder", "sphere" frame An integer representing the frame number of an event Table 6 : Input/output data types of modules in the program executor.

Question: "How many spheres are moving when the video ends?" Count( Filter_move( Filter_shape(Objects, "Sphere"), Get_frame(End) ) )

Question: "What color is the object to collide with the blue object?" Query_color( Get_col_partner( Filter_collision( Events, Filter_color(Objects, "Blue") ), Filter_color(Objects, "Blue") ) )

Question: "What shape is the second object to collide with the green object?" Query_shape( Get_col_partner( Filter_order( Filter_collision( Events, Filter_color(Objects, "Green") ), "Second" ), Filter_color(Objects, "Green") ) )

Question: "Are there any stationary spheres when the metal sphere enters the scene?" Exist( Filter_stationary( Filter_shape(Objects, "Sphere"), Query_frame( Filter_in( Events, Filter_shape( Filter_material(Objects, "Metal"), "Sphere" ) ) ) ) ) 

:



Figures and Tables:

Figure 2: Sample questions and programs from CLEVRER. Left: Descriptive question. Middle and right: multiple-choice question and choice. Each choice can pair with the question to form a joint logic trace.

Figure 3: Distribution of CLEVRER question types. Left: distribution of four main questions types. Right: distribution of descriptive sub-types.

hOt 2...t, Rt 2...ti < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 R h p

O h 5 b y L + 5 7 U T C C 4 7 K Z d x A k z S 6 U N B I j B E e B I S 7 n H F KI i R I Y Q q b v 6 K 6 Y A o Q s F E W T Q h u P M r L 5 J G p e w a f n d e q l 7 l c R T Q I T p C J 8 h F F 6 i K b l A N 1 R F F T + g F v a F3 6 9 l 6 t T 6 s z 2 n p k p X 3 H K A Z W F + / m M i e 8 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 R h p

O h 5 b y L + 5 7 U T C C 4 7 K Z d x A k z S 6 U N B I j B E e B I S 7 n H F KI i R I Y Q q b v 6 K 6 Y A o Q s F E W T Q h u P M r L 5 J G p e w a f n d e q l 7 l c R T Q I T p C J 8 h F F 6 i K b l A N 1 R F F T + g F v a F3 6 9 l 6 t T 6 s z 2 n p k p X 3 H K A Z W F + / m M i e 8 A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " 9 R h p

O h 5 b y L + 5 7 U T C C 4 7 K Z d x A k z S 6 U N B I j B E e B I S 7 n H F KI i R I Y Q q b v 6 K 6 Y A o Q s F E W T Q h u P M r L 5 J G p e w a f n d e q l 7 l c R T Q I T p C J 8 h F F 6 i K b l A N 1 R F F T + g F v a F3 6 9 l 6 t T 6 s z 2 n p k p X 3 H K A Z W F + / m M i e 8 A = = < / l a t e x i t > h Ôt+1, Rt+1i

w b O P e d e 7 t w T J o J r c J x v q 7 C 0 v L K 6 V l w v b W x u b e / Y u 3 t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K 3 H p j S P J b 3 M E q Y H 5 G + 5 D 1 O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q 7 v M S e m j Y F d t m p O l P g R e L m p I x y 1 A P 7 y

w b O P e d e 7 t w T J o J r c J x v q 7 C 0 v L K 6 V l w v b W x u b e / Y u 3 t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K 3 H p j S P J b 3 M E q Y H 5 G + 5 D 1 O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q 7 v M S e m j Y F d t m p O l P g R e L m p I x y 1 A P 7 y

w b O P e d e 7 t w T J o J r c J x v q 7 C 0 v L K 6 V l w v b W x u b e / Y u 3 t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K 3 H p j S P J b 3 M E q Y H 5 G + 5 D 1 O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q 7 v M S e m j Y F d t m p O l P g R e L m p I x y 1 A P 7 y

w b O P e d e 7 t w T J o J r c J x v q 7 C 0 v L K 6 V l w v b W x u b e / Y u 3 t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K 3 H p j S P J b 3 M E q Y H 5 G + 5 D 1 O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q 7 v M S e m j Y F d t m p O l P g R e L m p I x y 1 A P 7 y+ v G N I 2 Y B C q I 1 h 3 X S c D P i A J O B R u X v F S z h N A h 6 b O O o Z J E T P v Z 9 K o x P j J K F / d i Z Z 4 E P F V / T 2 Q k 0 n o U h a Y z I j D Q 8 9 5 E / M / r p N C 7 8 D M u k x S Y p L N F v V R g i P E k I t z l i l E Q I 0 M I V d z 8 F d M B U Y S C C b J k Q n D n T 1 4 k z d O q a / j t W b l 2 m c d R R A f o E B 0 j F 5 2 j G r p G d d R A F D 2 i Z / S K 3 q w n 6 8 V 6 t z 5 m r Q U r n 9 l H f 2 B9 / g D o P p 6 k < / l a t e x i t > t e x i t s h a _ b a s e = " + J u u E x A Q z I j M h N R F B a I D L n Y = " > A A A B H i c b Z B N S N A E I Y n a v W r p H L t F E I S S i K D H o h e P F e w H t K F s t t t W Y T d y e F E v I v H h Q x K s / x p v / x m b g a + s P D w z g w z + w a x F A Z d s p r K v b

H l x D D e h D g g A T P A p v z s R c d d j V r w c l n j u G P n M f N y S J A = = < / l a t e x i t > < l a t e x i t s h a _ b a s e = " + J u u E x A Q z I j M h N R F B a I D L n Y = " > A A A B H i c b Z B N S N A E I Y n a v W r p H L t F E I S S i K D H o h e P F e w H t K F s t t t W Y T d y e F E v I v H h Q x K s / x p v / x m b g a + s P D w z g w z + w a x F A Z d s p r K v b

H l x D D e h D g g A T P A p v z s R c d d j V r w c l n j u G P n M f N y S J A = = < / l a t e x i t > < l a t e x i t s h a _ b a s e = " + J u u E x A Q z I j M h N R F B a I D L n Y = " > A A A B H i c b Z B N S N A E I Y n a v W r p H L t F E I S S i K D H o h e P F e w H t K F s t t t W Y T d y e F E v I v H h Q x K s / x p v / x m b g a + s P D w z g w z + w a x F A Z d s p r K v b

H l x D D e h D g g A T P A p v z s R c d d j V r w c l n j u G P n M f N y S J A = = < / l a t e x i t > < l a t e x i t s h a _ b a s e = " + J u u E x A Q z I j M h N R F B a I D L n Y = " > A A A B H i c b Z B N S N A E I Y n a v W r p H L t F E I S S i K D H o h e P F e w H t K F s t t t W Y T d y e F E v I v H h Q x K s / x p v / x m b g a + s P D w z g w z + w a x F A Z d s p r K v b

H l x D D e h D g g A T P A p v z s R c d d j V r w c l n j u G P n M f N y S J A = = < / l a t e x i t > pt+1 < l a t e x i t s h a 1 _ b a s e 6 4 = " + J u u E x 5 5 A Q z I j M h N R 1 F B a I D L 2 n Y = " > A A A B 9 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F E I S S i K D H o h e P F e w H t K F s t t t 2 6 W Y T d y e F E v I 7 v H h Q x K s / x p v / x m 2 b g 7 a + s P D w z g w z + w a x F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 6 a J E s 1 4 g 0 U y 0 u 2 A G i 6 F 4 g 0 U K H k 7 1 p y G g e S t Y H w 3 q 7 c m X B s R q U e c x t w P 6 V C J g W A U r e V 3 R x T T O O u l e O F l v X L F r b p z k V X w c q h A r n q v / N X t R y w J u U I m q T E d z 4 3 R T 6 l G w S T P S t 3 E 8 J i y M R 3 y j k V F Q 2 7 8 d H 5 0 R s 6 s 0 y e D S N u n k M z d 3 x M p D Y 2 Z h o H t D C m O z H J t Z v 5 X 6 y Q 4 u P F T o e I E u W K L R Y N E E o z I L A H S F 5 o z l F M L l G l h b y V s R D V l a H M q 2 R C 8 5 S + v Q v O y 6 l l + u K r U b v M 4 i n A C p 3 A O H l x D D e 6 h D g 1 g 8 A T P 8 A p v z s R 5 c d 6 d j 0 V r w c l n j u G P n M 8 f 3 N y S J A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + J u u E x 5 5 A Q z I j M h N R 1 F B a I D L 2 n Y = " > A A A B 9 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F E I S S i K D H o h e P F e w H t K F s t t t 2 6 W Y T d y e F E v I 7 v H h Q x K s / x p v / x m 2 b g 7 a + s P D w z g w z + w a x F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 6 a J E s 1 4 g 0 U y 0 u 2 A G i 6 F 4 g 0 U K H k 7 1 p y G g e S t Y H w 3 q 7 c m X B s R q U e c x t w P 6 V C J g W A U r e V 3 R x T T O O u l e O F l v X L F r b p z k V X w c q h A r n q v / N X t R y w J u U I m q T E d z 4 3 R T 6 l G w S T P S t 3 E 8 J i y M R 3 y j k V F Q 2 7 8 d H 5 0 R s 6 s 0 y e D S N u n k M z d 3 x M p D Y 2 Z h o H t D C m O z H J t Z v 5 X 6 y Q 4 u P F T o e I E u W K L R Y N E E o z I L A H S F 5 o z l F M L l G l h b y V s R D V l a H M q 2 R C 8 5 S + v Q v O y 6 l l + u K r U b v M 4 i n A C p 3 A O H l x D D e 6 h D g 1 g 8 A T P 8 A p v z s R 5 c d 6 d j 0 V r w c l n j u G P n M 8 f 3 N y S J A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + J u u E x 5 5 A Q z I j M h N R 1 F B a I D L 2 n Y = " > A A A B 9 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F E I S S i K D H o h e P F e w H t K F s t t t 2 6 W Y T d y e F E v I 7 v H h Q x K s / x p v / x m 2 b g 7 a + s P D w z g w z + w a x F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 6 a J E s 1 4 g 0 U y 0 u 2 A G i 6 F 4 g 0 U K H k 7 1 p y G g e S t Y H w 3 q 7 c m X B s R q U e c x t w P 6 V C J g W A U r e V 3 R x T T O O u l e O F l v X L F r b p z k V X w c q h A r n q v / N X t R y w J u U I m q T E d z 4 3 R T 6 l G w S T P S t 3 E 8 J i y M R 3 y j k V F Q 2 7 8 d H 5 0 R s 6 s 0 y e D S N u n k M z d 3 x M p D Y 2 Z h o H t D C m O z H J t Z v 5 X 6 y Q 4 u P F T o e I E u W K L R Y N E E o z I L A H S F 5 o z l F M L l G l h b y V s R D V l a H M q 2 R C 8 5 S + v Q v O y 6 l l + u K r U b v M 4 i n A C p 3 A O H l x D D e 6 h D g 1 g 8 A T P 8 A p v z s R 5 c d 6 d j 0 V r w c l n j u G P n M 8 f 3 N y S J A = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " + J u u E x 5 5 A Q z I j M h N R 1 F B a I D L 2 n Y = " > A A A B 9 H i c b Z B N S 8 N A E I Y n 9 a v W r 6 p H L 4 t F E I S S i K D H o h e P F e w H t K F s t t t 2 6 W Y T d y e F E v I 7 v H h Q x K s / x p v / x m 2 b g 7 a + s P D w z g w z + w a x F A Z d 9 9 s p r K 1 v b G 4 V t 0 s 7 u 3 v 7 B + X D o 6 a J E s 1 4 g 0 U y 0 u 2 A G i 6 F 4 g 0 U K H k 7 1 p y G g e S t Y H w 3 q 7 c m X B s R q U e c x t w P 6 V C J g W A U r e V 3 R x T T O O u l e O F l v X L F r b p z k V X w c q h A r n q v / N X t R y w J u U I m q T E d z 4 3 R T 6 l G w S T P S t 3 E 8 J i y M R 3 y j k V F Q 2 7 8 d H 5 0 R s 6 s 0 y e D S N u n k M z d 3 x M p D Y 2 Z h o H t D C m O z H J t Z v 5 X 6 y Q 4 u P F T o e I E u W K L R Y N E E o z I L A H S F 5 o z l F M L l G l h b y V s R D V l a H M q 2 R C 8 5 S + v Q v O y 6 l l + u K r U b v M 4 i n A C p 3 A O H l x D D e 6 h D g 1 g 8 A T P 8 A p v z s R 5 c d 6 d j 0 V r w c l n j u G P n M 8 f 3 N y S J A = = < / l a t e x i t > h Ôt+1, Rt+1i

w b O P e d e t w T J o J r c J x v q C v L K V l w v b W x u b e / Y u t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K H p j S P J b M E q Y H G + D O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q v M S e m j Y F d t m p O l P g R e L m p I x y A P y

w b O P e d e t w T J o J r c J x v q C v L K V l w v b W x u b e / Y u t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K H p j S P J b M E q Y H G + D O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q v M S e m j Y F d t m p O l P g R e L m p I x y A P y

w b O P e d e t w T J o J r c J x v q C v L K V l w v b W x u b e / Y u t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K H p j S P J b M E q Y H G + D O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q v M S e m j Y F d t m p O l P g R e L m p I x y A P y

w b O P e d e t w T J o J r c J x v q C v L K V l w v b W x u b e / Y u t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K H p j S P J b M E q Y H G + D O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q v M S e m j Y F d t m p O l P g R e L m p I x y A P y

w b O P e d e 7 t w T J o J r c J x v q 7 C 0 v L K 6 V l w v b W x u b e / Y u 3 t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K 3 H p j S P J b 3 M E q Y H 5 G + 5 D 1 O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q 7 v M S e m j Y F d t m p O l P g R e L m p I x y 1 A P 7 y

w b O P e d e 7 t w T J o J r c J x v q 7 C 0 v L K 6 V l w v b W x u b e / Y u 3 t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K 3 H p j S P J b 3 M E q Y H 5 G + 5 D 1 O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q 7 v M S e m j Y F d t m p O l P g R e L m p I x y 1 A P 7 y

w b O P e d e 7 t w T J o J r c J x v q 7 C 0 v L K 6 V l w v b W x u b e / Y u 3 t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K 3 H p j S P J b 3 M E q Y H 5 G + 5 D 1 O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q 7 v M S e m j Y F d t m p O l P g R e L m p I x y 1 A P 7 y

w b O P e d e 7 t w T J o J r c J x v q 7 C 0 v L K 6 V l w v b W x u b e / Y u 3 t N H a e K s g a N R a z a I d F M c M k a w E G w d q I Y i U L B W u H w a u K 3 H p j S P J b 3 M E q Y H 5 G + 5 D 1 O C R g p s C u e I L I v G P Y G B L K b c Z D B i T u u z M q 7 v M S e m j Y F d t m p O l P g R e L m p I x y 1 A P 7 y

Figure 4: Our model includes four components: a video frame parser that generates an object-based representation of the video frames; a question parser that turns a question into a functional program; a dynamics predictor that extracts and predicts the dynamic scene of the video; and a symbolic program executor that runs the program on the dynamic scene to obtain an answer.

Figure 5: Performance of our model under different number of programs used for training the question parser.

Figure 7: Sample results of NS-DR on CLEVRER. 'Pred.' and 'CF.' indicate predictive and counterfactual events extracted by the model's dynamics predictor. The counterfactual condition shown in this example is to remove the cyan cylinder.

What shape is the first object to collide with the gray sphere? Descriptive Q2: How many collisions happen? Explanatory Q: Which of the following is not responsible for the collision between the gray sphere and the brown object? a) The collision between the gray sphere and the green sphere b) The presence of the green rubber sphere c) The presence of the cyan metal sphere A: c) Predictive Q: Which event will happen next a) The gray sphere and the cube collide b) The cube collides with the cylinder A: a) Counterfactual Q: What will happen without the gray sphere? a) The cube collides with the cyan sphere b) The green sphere and the cyan sphere collide c) The brown object and the cyan sphere collide d) The brown object collides with the cube A: d) A1: sphere A2: 2 Q3: Are there any metal spheres that enter the scene before the brown object enters the scene? Are there any moving red objects when the video ends? Descriptive Q2: What color is the object that enters the scene? Explanatory Q: Which of the following is responsible for the sphere's exiting the scene? a) The presence of the cyan object b) The presence of the purple rubber cube c) The presence of the green metal cylinder d) The collision between the sphere and the cyan cylinder A: a), c), d) Predictive Q: Which event will happen next a) The green cylinder and the sphere collide b) The green cylinder collides with the cube A: b) Counterfactual Q: Without the green cylinder, what will not happen? a) The sphere and the cube collide b) The sphere and the cyan cylinder collide c) The cube and the cyan cylinder collide A: b), c) A1: no A2: purple Q3: What is the color of the first object to collide with the rubber sphere? How many blue objects enter the scene? Descriptive Q2: What shape is the last object to collide with the cylinder? Explanatory Q: Which of the following is responsible for the collision between the cylinder and the metal object? a) The cube's entering the scene b) The presence of the blue rubber cube c) The cube's colliding with the cylinder d) The presence of the blue rubber sphere A: a), b), c) Predictive Q: Which event will happen next a) The cube and the blue sphere collide b) The metal object collides with the blue sphere A: a) Counterfactual Q: What will happen if the metal object is removed? a) The cylinder and the blue sphere collide b) The cube collides with the blue sphere c) The cube and the cylinder collide A: b), c) A1: 2 A2: sphere Q3: What color is the object that is stationary when the video ends? A3: blue

Figure 8: Sample videos and questions from CLEVRER. Stroboscopic imaging is applied for motion visualization.

Figure 9: Example of descriptive question programs.

Comparison between CLEVRER and other visual reasoning benchmarks on images and videos. CLEVRER is a well-annotated video reasoning dataset created under a controlled environment. It introduces a wide range of reasoning tasks including description, explanation, prediction and counterfactuals

Table content: <table><tr><td>Dataset</td><td>Video</td><td colspan="5">Diagnostic Temporal Explanation Prediction Counterfactual Annotations Relation</td></tr><tr><td>VQA (Antol et al., 2015)</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td><td>×</td></tr><tr><td>CLEVR (Johnson et al., 2017a)</td><td>×</td><td/><td>×</td><td>×</td><td>×</td><td>×</td></tr><tr><td>COG (Yang et al., 2018)</td><td>×</td><td/><td/><td>×</td><td>×</td><td>×</td></tr><tr><td>VCR (Zellers et al., 2019)</td><td>×</td><td/><td>×</td><td/><td>×</td><td/></tr><tr><td>GQA (Johnson et al., 2017a)</td><td>×</td><td/><td>×</td><td>×</td><td>×</td><td>×</td></tr><tr><td>TGIF-QA (Jang et al., 2017)</td><td/><td>×</td><td/><td>×</td><td>×</td><td>×</td></tr><tr><td>MovieQA (Tapaswi et al., 2016)</td><td/><td>×</td><td/><td/><td>×</td><td>×</td></tr><tr><td>MarioQA (Mun et al., 2017)</td><td/><td>×</td><td/><td/><td>×</td><td>×</td></tr><tr><td>TVQA (Lei et al., 2018)</td><td/><td>×</td><td>×</td><td/><td>×</td><td>×</td></tr><tr><td>Social-IQ (Zadeh et al., 2019)</td><td/><td>×</td><td>×</td><td/><td>×</td><td>×</td></tr><tr><td>CLEVRER (ours)</td><td/><td/><td/><td/><td/><td/></tr></table>

Question-answering accuracy of visual reasoning baselines on CLEVRER. All models are trained on the full training set. The IEP (V) model and TbD-net (V) use 1000 programs to train the program generator.

Table content: <table><tr><td>Methods</td><td>Descriptive</td><td colspan="2">Explanatory</td><td colspan="2">Predictive</td><td colspan="2">Counterfactual</td></tr><tr><td/><td/><td colspan="6">per opt. per ques. per opt. per ques. per opt. per ques.</td></tr><tr><td>Q-type (random)</td><td>29.2</td><td>50.1</td><td>8.1</td><td>50.7</td><td>25.5</td><td>50.1</td><td>10.3</td></tr><tr><td>Q-type (frequent)</td><td>33.0</td><td>50.2</td><td>16.5</td><td>50.0</td><td>0.0</td><td>50.2</td><td>1.0</td></tr><tr><td>LSTM</td><td>34.7</td><td>59.7</td><td>13.6</td><td>50.6</td><td>23.2</td><td>53.8</td><td>3.1</td></tr><tr><td>CNN+MLP</td><td>48.4</td><td>54.9</td><td>18.3</td><td>50.5</td><td>13.2</td><td>55.2</td><td>9.0</td></tr><tr><td>CNN+LSTM</td><td>51.8</td><td>62.0</td><td>17.5</td><td>57.9</td><td>31.6</td><td>61.2</td><td>14.7</td></tr><tr><td>TVQA+</td><td>72.0</td><td>63.3</td><td>23.7</td><td>70.3</td><td>48.9</td><td>53.9</td><td>4.1</td></tr><tr><td>Memory</td><td>54.7</td><td>53.7</td><td>13.9</td><td>50.0</td><td>33.1</td><td>54.2</td><td>7.0</td></tr><tr><td>IEP (V)</td><td>52.8</td><td>52.6</td><td>14.5</td><td>50.0</td><td>9.7</td><td>53.4</td><td>3.8</td></tr><tr><td>TbD-net (V)</td><td>79.5</td><td>61.6</td><td>3.8</td><td>50.3</td><td>6.5</td><td>56.1</td><td>4.4</td></tr><tr><td>MAC (V)</td><td>85.6</td><td>59.5</td><td>12.5</td><td>51.0</td><td>16.5</td><td>54.6</td><td>13.7</td></tr><tr><td>MAC (V+)</td><td>86.4</td><td>70.5</td><td>22.3</td><td>59.7</td><td>42.9</td><td>63.5</td><td>25.1</td></tr><tr><td colspan="8">Compositional visual reasoning. The CLEVR dataset (Johnson et al., 2017a) opened up a new</td></tr><tr><td colspan="8">direction of compositional visual reasoning, which emphasizes complexity and compositionality</td></tr><tr><td>in the logic and</td><td/><td/><td/><td/><td/><td/><td/></tr></table>

Quantitative results of NS-DR on CLEVRER. We evaluate our model on all four question types. We also study a variation of our model NS-DR (NE) with "no events" but only motion traces from the dynamics predictor. Our question parser is trained with 1000 programs.

Table content: <table><tr><td>Methods</td><td>Descriptive</td><td colspan="2">Explanatory</td><td colspan="2">Predictive</td><td colspan="2">Counterfactual</td></tr><tr><td/><td/><td colspan="6">per opt. per ques. per opt. per ques. per opt. per ques.</td></tr><tr><td>NS-DR</td><td>88.1</td><td>87.6</td><td>79.6</td><td>82.9</td><td>68.7</td><td>74.1</td><td>42.2</td></tr><tr><td>NS-DR (NE)</td><td>85.8</td><td>85.9</td><td>74.3</td><td>75.4</td><td>54.1</td><td>76.1</td><td>42.0</td></tr></table>

" if the input objects is not empty Belong to (event, events) bool Returns "yes" if the input event belongs to the input set of events Negate bool bool Returns the negation of the input boolean Functional modules of NS-DR's program executor (continued).

Table content: <table><tr><td colspan="2">Module Type Name / Description</td><td>Input Type</td><td>Output Type</td></tr><tr><td/><td>Query color</td><td>object</td><td>color</td></tr><tr><td/><td>Returns the color of the input object</td><td/><td/></tr><tr><td/><td>Query material</td><td>object</td><td>material</td></tr><tr><td/><td>Returns the material of the input object</td><td/><td/></tr><tr><td/><td>Query shape</td><td>object</td><td>shape</td></tr><tr><td>Output</td><td>Returns the shape of the input object</td><td/><td/></tr><tr><td>Modules</td><td>Count</td><td>objects</td><td>int</td></tr><tr><td/><td>Returns the number of the input objects</td><td/><td/></tr><tr><td/><td>Exist</td><td>objects</td><td>bool</td></tr><tr><td/><td>Returns "yes</td><td/><td/></tr></table>

Peer review (iclr2020):
A new benchmark is presented, which requires to reason on spatio-temporal data (videos) and incorporates physics (videos of physical dynamics), language (questions are formulated through language) and causality. The benchmark builds on the well-known CLEVR benchmark and adds several interesting contributions, in particular reasoning over time. Compared to other benchmarks proposed on reasoning on physical dynamics, this benchmark adds to the language component (which is presented in classical VQA benchmarks), and an interesting counterfactual component known from the causal inference literature. This is fairly new for the computer vision and statistical machine learning literature (a few papers on counterfactual reasoning exist), but in contrast to the causal inference literature, here the do-operator is observable during training: we do have access to do-interventions and even the outcome during training and therefore we can even learn counterfactuals using supervised learning. This statement is not meant as criticism, as learning counterfactuals without supervision from high-dimensional input like images seems to be currently out of a reach, or at least has not yet been demonstrated up to my knowledge. On the downside, compared to other physical reasoning benchmarks, the answers here are multiple choice instead of regressions of the future motion, which requires more fine-grained reasoning. This will guide solutions to a certain type and will also favor solutions, requiring the network to detect certain binary concepts and combine them instead of regressing complex functions. It also favors solutions of the type presented in the paper. As for other benchmarks, the dataset is quite large (20 000 videos) and, given its synthetic nature, is accompanied by functional programs. As for CLEVR, the simulations have been performed by a physics engine and then separately rendered with Blender to maximize visual quality. The result is a very interesting benchmark, and I have no doubt that it will be very useful for us (the learning reasoning community). Another positive point is the number of baselines tested on the benchmark, among which we can find strong papers on VQA/VQA2 I have a couple of questions: How is the causal graph created? Are the experiments rendered and outcomes examined, creating the causal graph for the counterfactual answers, or are the experiments selected with a given outcome already decided? The distribution of question types has been provided, but how about the biases? Distributions of the answers would have been helpful. How did the authors avoid biases during construction of the dataset, in particular in the counterfactual case (see remarks on the causal graph). The paper also comes with a method for solving, which is very similar to existing methods on learning through functional programs. The method itself is unfortunately described only very briefly and the reader needs to look it up almost entirely in the appendix of the paper, which is surprising, as the paper is only 8 pages long instead of 10. As for CLEVR, one of the downsides of this type of benchmark is the synthetic nature of the images and the limited range of different objects in the scene. Of course this comes with the advantage of being able to study compositional reasoning in detail, as a scene graph can be calculated easily (and is available during simulation). However, it also makes reasoning through functional programs much easier, as the proposed filters are limited in number and can strong respond to the small number of shapes and colors available in the data. I have strong doubts that this kind of approach extends to real life scenarios. For VQA type of scenarios, GQA is a nice compromise between natural looking images and the availability of scene graphs and the restriction of questions to compositional reasoning. The optimal choice would be a similar compromise for spatio-temporal data, but of course this would be a huge effort and it would be up to impossible to have access to counterfactuals. Last point, and this question is not restricted to this paper, as the name came up elsewhere, why is the model called neuro-symbolic reasoning? While it could be argued that the questions require a sort of “symbolic reasoning”, I am not sure that the reasoning method itself is symbolic even partially. Other than selecting functional programs out of a discrete set, the reasoning itself is connectionist and performed with graph networks.