CLEVRER: Collision Events for Video Representation and Reasoning

1 INTRODUCTION . The ability to recognize objects and reason about their behaviors in physical events from videos lies at the core of human cognitive development ( Spelke , 2000 ) . Humans , even young infants , group segments into objects based on motion , and use concepts of object permanence , solidity , and continuity to explain what has happened , infer what is about to happen , and imagine what would happen in counterfactual situations . The problem of complex visual reasoning has been widely studied in artificial intelligence and computer vision , driven by the introduction of various datasets on both static images ( Antol et al. , 2015 ; Zhu et al. , 2016 ; Hudson & Manning , 2019 ) and videos ( Jang et al. , 2017 ; Tapaswi et al. , 2016 ; Zadeh et al. , 2019 ) . However , despite the complexity and variety of the visual context covered by these datasets , the underlying logic , temporal and causal structure behind the reasoning process is less explored . In this paper , we study the problem of temporal and causal reasoning in videos from a complementary perspective : inspired by a recent visual reasoning dataset , CLEVR ( Johnson et al. , 2017a ) , we simplify the problem of visual recognition , but emphasize the complex temporal and causal structure behind the interacting objects . We introduce a video reasoning benchmark for this problem , drawing inspirations from developmental psychology ( Gerstenberg et al. , 2015 ; Ullman , 2015 ) . We also evaluate and assess limitations of various current visual reasoning models on the benchmark . Our benchmark , named CoLlision Events for Video REpresentation and Reasoning ( CLEVRER ) , is a diagnostic video dataset for temporal and causal reasoning under a fully controlled environment . The design of CLEVRER follows two guidelines : first , the posted tasks should focus on logic reasoning in the temporal and causal domain while staying simple and exhibiting minimal biases on visual scenes and language ; second , the dataset should be fully controlled and well-annotated in order to host the ∗indicates equal contributions . Project page : http : //clevrer.csail.mit.edu/ complex reasoning tasks and provide effective diagnostics for models on those tasks . CLEVRER includes 20,000 synthetic videos of colliding objects and more than 300,000 questions and answers ( Figure 1 ) . We focus on four specific elements of complex logical reasoning on videos : descriptive ( e.g. , ‘ what color ’ ) , explanatory ( ‘ what ’ s responsible for ’ ) , predictive ( ‘ what will happen next ’ ) , and counterfactual ( ‘ what if ’ ) . CLEVRER comes with ground-truth motion traces and event histories of each object in the videos . Each question is paired with a functional program representing its underlying logic . As summarized in table 1 , CLEVRER complements existing visual reasoning benchmarks on various aspects and introduces several novel tasks . We also present analysis of various state-of-the-art visual reasoning models on CLEVRER . While these models perform well on descriptive questions , they lack the ability to perform causal reasoning and struggle on the explanatory , predictive , and counterfactual questions . We therefore identify three key elements that are essential to the task : recognition of the objects and events in the videos ; modeling the dynamics and causal relations between the objects and events ; and understanding of the symbolic logic behind the questions . As a first-step exploration of this principle , we study an oracle model , Neuro-Symbolic Dynamic Reasoning ( NS-DR ) , that explicitly joins these components via a symbolic video representation , and assess its performance and limitations . 2 RELATED WORK . Our work can be uniquely positioned in the context of three recent research directions : video understanding , visual question answering , and physical and causal reasoning . Video understanding . With the availability of large-scale video datasets ( Caba Heilbron et al. , 2015 ; Kay et al. , 2017 ) , joint video and language understanding tasks have received much interest . This includes video captioning ( Guadarrama et al. , 2013 ; Venugopalan et al. , 2015 ; Gan et al. , 2017 ) , localizing video segments from natural language queries ( Gao et al. , 2017 ; Hendricks et al. , 2017 ) , and video question answering . In particular , recent papers have explored different approaches to acquire and ground various reasoning tasks to videos . Among those , MovieQA ( Tapaswi et al. , 2016 ) , TGIF-QA ( Jang et al. , 2017 ) , TVQA ( Lei et al. , 2018 ) are based on real-world videos and human-generated questions . Social-IQ ( Zadeh et al. , 2019 ) discusses causal relations in human social interactions based on real videos . COG ( Yang et al. , 2018 ) and MarioQA ( Mun et al. , 2017 ) use simulated environments to generate synthetic data and controllable reasoning tasks . Compared to them , CLEVRER focuses on the causal relations grounded in object dynamics and physical interactions , and introduces a wide range of tasks including description , explanation , prediction and counterfactuals . CLEVRER also emphasizes compositionality in the visual and logic context . Visual question answering . Many benchmark tasks have been introduced in the domain of visual question answering . The Visual Question Answering ( VQA ) dataset ( Antol et al. , 2015 ) marks an important milestone towards top-down visual reasoning , based on large-scale cloud-sourced real images and human-generated questions . The CLEVR dataset ( Johnson et al. , 2017a ) follows a bottomup approach by defining the tasks under a controlled close-domain setup of synthetic images and questions with compositional attributes and logic traces . More recently , the GQA dataset ( Hudson & Manning , 2019 ) applies synthetic compositional questions to real images . The VCR dataset ( Zellers et al. , 2019 ) discusses explanations and hypothesis judgements based on common sense . There have also been numerous visual reasoning models ( Hudson & Manning , 2018 ; Santoro et al. , 2017 ; Hu et al. , 2017 ; Perez et al. , 2018 ; Zhu et al. , 2017 ; Mascharka et al. , 2018 ; Suarez et al. , 2018 ; Cao et al. , 2018 ; Bisk et al. , 2018 ; Misra et al. , 2018 ; Aditya et al. , 2018 ) . Here we briefly review a few . The stacked attention networks ( SAN ) ( Yang et al. , 2016 ) introduce a hierarchical attention mechanism for end-to-end VQA models . The MAC network ( Hudson & Manning , 2018 ) combines visual and language attention for compositional visual reasoning . The IEP model ( Johnson et al. , 2017b ) proposes to answer questions via neural program execution . The NS-VQA model ( Yi et al. , 2018 ) disentangles perception and logic reasoning by combining an object-based abstract representation of the image with symbolic program execution . In this work we study a complementary problem of causal reasoning and assess the strengths and limitations of these baseline methods . Physical and causal reasoning . Our work is also related to research on learning scene dynamics for physical and causal reasoning ( Lerer et al. , 2016 ; Battaglia et al. , 2013 ; Mottaghi et al. , 2016 ; Fragkiadaki et al. , 2016 ; Battaglia et al. , 2016 ; Chang et al. , 2017 ; Agrawal et al. , 2016 ; Finn et al. , 2016 ; Shao et al. , 2014 ; Fire & Zhu , 2016 ; Pearl , 2009 ; Ye et al. , 2018 ) , either directly from images ( Finn et al. , 2016 ; Ebert et al. , 2017 ; Watters et al. , 2017 ; Lerer et al. , 2016 ; Mottaghi et al. , 2016 ; Fragkiadaki et al. , 2016 ) , or from a symbolic , abstract representation of the environment ( Battaglia et al. , 2016 ; Chang et al. , 2017 ) . Concurrent to our work , CoPhy ( Baradel et al. , 2020 ) studies physical dynamics prediction in a counterfatual setting . CATER ( Girdhar & Ramanan , 2020 ) introduces a synthetic video dataset for temporal reasoning associated with compositional actions . CLEVRER complements these works by incorporating dynamics modeling with compositional causal reasoning , and grounding the reasoning tasks to the language domain . 3 THE CLEVRER DATASET . The CLEVRER dataset studies temporal and causal reasoning on videos . It is carefully designed in a fully-controlled synthetic environment , enabling complex reasoning tasks , providing effective diagnostics for models while simplifying video recognition and language understanding . The videos describe motion and collisions of objects on a flat tabletop ( as shown in Figure 1 ) simulated by a physics engine , and are associated with the ground-truth motion traces and histories of all objects and events . Each video comes with four types of questions generated by machine , including descriptive ( ‘ what color ’ , ‘ how many ’ ) , explanatory ( ‘ What is responsible for ’ ) , predictive ( ‘ What will happen next ’ ) , and counterfactual ( ‘ what if ’ ) . Each question is paired with a functional program . 3.1 VIDEOS . CLEVRER includes 10,000 videos for training , 5,000 for validation , and 5,000 for testing . All videos last for 5 seconds . The videos are generated by a physics engine that simulates object motion plus a graphs engine that renders the frames . Extra examples from the dataset can be found in supplementary material C. Objects and events . Objects in CLEVRER videos adopt similar compositional intrinsic attributes as in CLEVR ( Johnson et al. , 2017a ) , including three shapes ( cube , sphere , and cylinder ) , two materials ( metal and rubber ) , and eight colors ( gray , red , blue , green , brown , cyan , purple , and yellow ) . All objects have the same size so no vertical bouncing occurs during collision . In each video , we prohibit identical objects , such that each combination of the three attributes uniquely identifies one object . Under this constraint , all intrinsic attributes for each object are sampled randomly . We further introduce three types of events : enter , exit and collision , each of which contains a fixed number of object participants : 2 for collision and 1 for enter and exit . The objects and events form an abstract representation of the video . These ground-truth annotations , together with the object motion traces , enable model diagnostics , one of the key advantages offered by a fully controlled environment . Causal structure . Objects and events in CLEVRER videos exhibit rich causal structures . An event can be either caused by an object if the event is the first one participated by the object , or another event if the cause event happens right before the outcome event on the same object . For example , if a sphere collides with a cube and then a cylinder , then the first collision and the cube jointly “ cause ” the second collision . The object motion traces with complex causal structures are generated by the following recursive process . We start with one randomly initialized moving object , and then add another object whose initial position and velocity are set such that it will collide with the first object . The same process is then repeated to add more objects and collisions to the scene . All collision pairs and motion trajectories are randomly chosen . We discard simulations with repetitive collisions between the same pair of objects . Video generation . CLEVRER videos are generated from the simulated motion traces , including each object ’ s position and pose at each time step . We use the Bullet ( Coumans , 2010 ) physics engine for motion simulation . Each simulation lasts for seven seconds . The motion traces are first down-sampled to fit the frame rate of the output video ( 25 frames per second ) . Then the motion of the first five seconds are sent to Blender ( Blender Online Community , 2016 ) to render realistic video frames of object motion and collision . The remaining two seconds are held-out for predictive tasks . We further note CLEVRER adopts the same software and parameters for rendering as CLEVR . 3.2 QUESTIONS We pair each video with machine-generated questions for descriptive , explanatory , predictive , and counterfactual reasoning . Sample questions of the four types can be found in Figure 1 . Each question is paired with a functional program executable on the video ’ s dynamical scene . Unlike CLEVR ( Johnson et al. , 2017a ) , our questions focus on the temporal and causal aspects of the objects and events in the videos . We exclude all questions on static object properties , which can be answered by looking at a single frame . CLEVRER consists of 219,918 descriptive questions , 33,811 explanatory questions , 14,298 predictive questions and 37,253 counterfactual questions . Detailed distribution and split of the questions can be found in Figure 3 and supplementary material A. Descriptive . Descriptive questions evaluate a model ’ s capability to understand and reason about a video ’ s dynamical content and temporal relation . The reasoning tasks are grounded to the compositional space of both object and event properties , including intrinsic attributes ( color , material , shape ) , motion , collision , and temporal order . All descriptive questions are ‘ open-ended ’ and can be answered by a single word . Descriptive questions contain multiple sub-types including count , exist , query color , query material , and query shape . Distribution of the sub-types is shown in Figure 3 . We evenly sample the answers within each sub-type to reduce answer bias . Explanatory . Explanatory questions query the causal structure of a video by asking whether an object or event is responsible for another event . Event A is responsible for event B if A is among B ’ s ancestors in the causal graph . Similarly , object O is responsible for event A if O participates in A or any other event responsible for A. Explanatory questions are multiple choice questions with at most four options , each representing an event or object in the video . Models need to select all options that match the question ’ s description . There can be multiple correct options for each question . We sample the options to balance the number of correct and wrong ones , and minimize text-only biases . Predictive . Predictive questions test a model ’ s capability of predicting possible occurrences of future events after the video ends . Similar to explanatory questions , predictive questions are multiplechoice , whose options represent candidate events that will or will not happen . Because post-video events are sparse , we provide two options for each predictive question to reduce bias . Counterfactual . Counterfactual questions query the outcome of the video under certain hypothetical conditions ( e.g . removing one of the objects ) . Models need to select the events that would or would not happen under the designated condition . There are at most four options for each question . The numbers of correct and incorrect options are balanced . Both predictive and counterfactual questions require knowledge of object dynamics underlying the videos and the ability to imagine and reason about unobserved events . Program representation . In CLEVRER , each question is represented by a tree-structured functional program , as shown in Figure 2 . A program begins with a list of objects or events from the video . The list is then passed through a sequence of filter modules , which select entries from the list and join the tree branches to output a set of target objects and events . Finally , an output module is called to query a designated property of the target outputs . For multiple choice questions , each question and option correspond to separate programs , which can be jointly executed to output a yes/no token that indicates if the choice is correct for the question . A list of all program modules can be found in the supplementary material B . Question generation . Questions in CLEVRER are generated by a multi-step procedure . For each question type , a pre-defined logic template is chosen . The logic template can be further populated by attributes that are associated with the context of the video ( i.e . the color , material , shape that identifies the object to be queried ) and then turned into natural language . We first generate an exhaustive list of all possible questions for each video . To avoid language bias , we sample from that list to maintain a balanced answer distribution for each question type and minimize the correlation between the questions and answers over the entire dataset .

Summary: This paper studies the temporal and causal structures in videos. Specifically, the authors first introduce a new dataset called CLEVRER drawing motivation from CLEVR, a well-known visual reasoning dataset. They further evaluate a set of state-of-the-art methods on the newly introduced dataset to confirm their initial beliefs of the challenges posed by causal reasoning. Based on empirical clues, they also suggest neural-symbolic based framework for causal reasoning.