Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoning

1 INTRODUCTION . Visual reasoning in dynamic scenes involves both the understanding of compositional properties , relationships , and events of objects , and the inference and prediction of their temporal and causal structures . As depicted in Fig . 1 , to answer the question “ What will happen next ? ” based on the observed video frames , one needs to detect the object trajectories , predict their dynamics , analyze the temporal structures , and ground visual objects and events to get the answer “ The blue sphere and the yellow object collide ” . Recently , various end-to-end neural network-based approaches have been proposed for joint understanding of video and language ( Lei et al. , 2018 ; Fan et al. , 2019 ) . While these methods have shown great success in learning to recognize visually complex concepts , such as human activities ( Xu et al. , 2017 ; Ye et al. , 2017 ) , they typically fail on benchmarks that require the understanding of compositional and causal structures in the videos and text ( Yi et al. , 2020 ) . Another line of research has been focusing on building modular neural networks that can represent the compositional structures in scenes and questions , such as object-centric scene structures and multi-hop reasoning ( Andreas et al. , 2016 ; Johnson et al. , 2017b ; Hudson & Manning , 2019 ) . However , these methods are designed for static images and do not handle the temporal and causal structure in dynamic scenes well , leading to inferior performance on video causal reasoning benchmark CLEVRER ( Yi et al. , 2020 ) . To model the temporal and causal structures in dynamic scenes , Yi et al . ( 2020 ) proposed an oracle model to combine symbolic representation with video dynamics modeling and achieved state-ofthe-art performance on CLEVRER . However , this model requires videos with dense annotations for visual attributes and physical events , which are impractical or extremely labor-intensive in real scenes . Project page : http : //dcl.csail.mit.edu We argue that such dense explicit video annotations are unnecessary for video reasoning , since they are naturally encoded in the question answer pairs associated with the videos . For example , the question answer pair and the video in Fig . 1 can implicitly inform a model what the concepts “ sphere ” , “ blue ” , “ yellow ” and “ collide ” really mean . However , a video may contain multiple fast-moving occluded objects and complex object interactions , and the questions and answers have diverse forms . It remains an open and challenging problem to simultaneously represent objects over time , train an accurate dynamic model from raw videos , and align objects with visual properties and events for accurate temporal and causal reasoning , using vision and language as the only supervision . Our main ideas are to factorize video perception and reasoning into several modules : object tracking , object and event concept grounding , and dynamics prediction . We first detect objects in the video , associating them into object tracks across the frames . We can then ground various object and event concepts from language , train a dynamic model on top of object tracks for future and counterfactual predictions , analyze relationships between events , and answer queries based on these extracted representations . All these modules can be trained jointly by watching videos and reading paired questions and answers . To achieve this goal , we introduce Dynamic Concept Learner ( DCL ) , a unified neural-symbolic framework for recognizing objects and events in videos and analyzing their temporal and causal structures , without explicit annotations on visual attributes and physical events such as collisions during training . To facilitate model training , a multi-step training paradigm has been proposed . We first run an object detector on individual frames and associate objects across frames based on a motion-based correspondence . Next , our model learns concepts about object properties , relationships , and events by reading paired questions and answers that describe or explain the events in the video . Then , we leverage the acquired visual concepts in the previous steps to refine the object association across frames . Finally , we train a dynamics prediction network ( Li et al. , 2019b ) based on the refined object trajectories and optimize it jointly with other learning parts in this unified framework . Such a training paradigm ensures that all neural modules share the same latent space for representing concepts and they can bootstrap the learning of each other . We evaluate DCL ’ s performance on CLEVRER , a video reasoning benchmark that includes descriptive , explanatory , predictive , and counterfactual reasoning with a uniform language interface . DCL achieves state-of-the-art performance on all question categories and requires no scene supervision such as object properties and collision events . To further examine the grounding accuracy and transferability of the acquired concepts , we introduce two new benchmarks for video-text retrieval and spatial-temporal grounding and localization on the CLEVRER videos , namely CLEVRERRetrieval and CLEVRER-Grounding . Without any further training , our model generalizes well to these benchmarks , surpassing the baseline by a noticeable margin . 2 RELATED WORK . Our work is related to reasoning and answering questions about visual content . Early studies like ( Wu et al. , 2016 ; Zhu et al. , 2016 ; Gan et al. , 2017 ) typically adopted monolithic network architectures and mainly focused on visual understanding . To perform deeper visual reasoning , neural module networks were extensively studied in recent works ( Johnson et al. , 2017a ; Hu et al. , 2018 ; Hudson & Manning , 2018 ; Amizadeh et al. , 2020 ) , where they represent symbolic operations with small neural networks and perform multi-hop reasoning . Some previous research has also attempted to learn visual concepts through visual question answering ( Mao et al. , 2019 ) . However , it mainly focused on learning static concepts in images , while our DCL aims at learning dynamic concepts like moving and collision in videos and at making use of these concepts for temporal and causal reasoning . Later , visual reasoning was extended to more complex dynamic videos ( Lei et al. , 2018 ; Fan et al. , 2019 ; Li et al. , 2020 ; 2019a ; Huang et al. , 2020 ) . Recently , Yi et al . ( 2020 ) proposed CLEVRER , a new video reasoning benchmark for evaluating computational models ’ comprehension of the causal structure behind physical object interaction . They also developed an oracle model , combining the neuro-symbolic visual question-answering model ( Yi et al. , 2018 ) with the dynamics prediction model ( Li et al. , 2019b ) , showing competitive performance . However , this model requires explicit labels for object attributes , masks , and spatio-temporal localization of events during training . Instead , our DCL has no reliance on any labels for objects and events and can learn these concepts through natural supervision ( i.e. , videos and question-answer pairs ) . Our work is also related to temporal and relational reasoning in videos via neural networks ( Wang & Gupta , 2018 ; Materzynska et al. , 2020 ; Ji et al. , 2020 ) . These works typically rely on specific action annotations , while our DCL learns to ground object and event concepts and analyze their temporal relations through question answering . Recently , various benchmarks ( Riochet et al. , 2018 ; Bakhtin et al. , 2019 ; Girdhar & Ramanan , 2020 ; Baradel et al. , 2020 ; Gan et al. , 2020 ) have been proposed to study dynamics and reasoning in physical scenes . However , these datasets mainly target at pure video understanding and do not contain natural language question answering . Much research has been studying dynamic modeling for physical scenes ( Lerer et al. , 2016 ; Battaglia et al. , 2013 ; Mottaghi et al. , 2016 ; Finn et al. , 2016 ; Shao et al. , 2014 ; Fire & Zhu , 2015 ; Ye et al. , 2018 ; Li et al. , 2019b ) . We adopt PropNet ( Li et al. , 2019b ) for dynamics prediction and feed the predicted scenes to the video feature extractor and the neuro-symbolic executor for event prediction and question answering . While many works ( Zhou et al. , 2019 ; 2018 ; Gan et al. , 2015 ) have been studying on the problems of understanding human actions and activities ( e.g. , running , cooking , cleaning ) in videos , our work ’ s primary goal is to design a unified framework for learning physical object and event concepts ( e.g. , collision , falling , stability ) . These tasks are of great importance in practical applications such as industrial robot manipulation which requires AI systems with human-like physical common sense . 3 DYNAMIC CONCEPT LEARNER . In this section , we introduce a new video reasoning model , Dynamic Concept Learner ( DCL ) , which learns to recognize video attributes , events , and dynamics and to analyze their temporal and causal structures , all through watching videos and answering corresponding questions . DCL contains five modules , 1 ) an object trajectory detector , 2 ) video feature extractor , 3 ) a dynamic predictor , 4 ) a language program parser , and 5 ) a neural symbolic executor . As shown in Fig . 2 , given an input video , the trajectory detector detects objects in each frame and associates them into trajectories ; the feature extractor then represents them as latent feature vectors . After that , DCL quantizes the objects ’ static concepts ( i.e. , color , shape , and material ) by matching the latent object features with the corresponding concept embeddings in the executor . As these static concepts are motion-independent , they can be adopted as an additional criteria to refine the object trajectories . Based on the refined trajectories , the dynamics predictor predicts the objects ’ movement and interactions in future and counterfactual scenes . The language parser parses the question and choices into functional programs , which are executed by the program executor on the latent representation space to get answers . The object and event concept embeddings and the object-centric representation share the same latent space ; answering questions associated with videos can directly optimize them through backpropagation . The object trajectories and dynamics can be refined by the object static attributes predicted by DCL . Our framework enjoys the advantages of both transparency and efficiency , since it enables step-by-step investigations of the whole reasoning process and has no requirements for explicit annotations of visual attributes , events , and object masks . 3.1 MODEL DETAILS . Object Detection and Tracking . Given a video , the object trajectory detector detects object proposals in each frame and connects them into object trajectories O = { on } Nn=1 , where on = { bnt } Tt=1 and N is the number of objects in the video . bt = [ xnt , ynt , wnt , hnt ] is an object proposal at frame t and T is the frame number , where ( xnt , y n t ) denotes the normalized proposal coordinate center and wnt and h n t denote the normalized width and height , respectively . The object detector first uses a pre-trained region proposal network ( Ren et al. , 2015 ) to generate object proposals in all frames , which are further linked across connective frames to get all objects ’ trajectories . Let { bit } Ni=1 and { b j t+1 } Nj=1 to be two sets of proposals in two connective frames . Inspired by Gkioxari & Malik ( 2015 ) , we define a connection score sl between bit and b j t+1 to be sl ( b i t , b j t+1 ) = sc ( b i t ) + sc ( b j t+1 ) + λ1 · IoU ( bit , b j t+1 ) , ( 1 ) where sc ( bit ) is the confidence score of the proposal b i t , IoU is the intersection over union and λ1 is a scalar . Gkioxari & Malik ( 2015 ) adopts a greedy algorithm to connect the proposals without global optimization . Instead , we assign boxes { bjt+1 } Nj=1 at the t + 1 frame to { bit } Ni=1 by a linear sum assignment . Video Feature Extraction . Given an input video and its detected object trajectories , we extract three kinds of latent features for grounding object and event concepts . It includes 1 ) the average visual feature fv ∈ RN×D1 for static attribute prediction , 2 ) temporal sequence feature fs ∈ RN×4T for dynamic attribute and unary event prediction , and 3 ) interactive feature f c ∈ RK×N×N×D2 for collision event prediction , where D1 and D2 denote dimensions of the features and K is the number of sampled frames . We give more details on how to extract these features in Appendix B. Grounding Object and Event Concepts . Video Reasoning requires a model to ground object and event concepts in videos . DCL achieves this by matching object and event representation with object and event embeddings in the symbolic executor . Specifically , DCL calculates the confidence score that the n-th object is moving by [ cos ( smoving , mda ( f s n ) ) − δ ] /λ , where fsn denotes the temporal sequence feature for the n-th object , smoving denotes a vector embedding for concept moving , and mda denotes a linear transformation , mapping fsn into the dynamic concept representation space . δ and λ are the shifting and scaling scalars , and cos ( ) calculates the cosine similarity between two vectors . DCL grounds static attributes and the collision event similarly , matching average visual features and interactive features with their corresponding concept embeddings in the latent space . We give more details on the concept and event quantization in Appendix E. Trajectory Refinement . The connection score in Eq . 1 ensures the continuity of the detected object trajectories . However , it does not consider the objects ’ visual appearance ; therefore , it may fail to track the objects and may connect inconsistent objects when different objects are close to each other and moving rapidly . To detect better object trajectories and to ensure the consistency of visual appearance along the track , we add a new term to Eq . 1 and re-define the connection score to be sl ( { bim } tm=0 , b j t+1 ) = sc ( b i t ) + sc ( b j t+1 ) + λ1 · IoU ( bit , b j t+1 ) + λ2 · fappear ( { bim } tm=0 , b j t+1 ) , ( 2 ) where fappear ( { bim } tm=0 , b j t+1 ) measures the attribute similarity between the newly added proposal bjt+1 and all proposals ( { bim } tm=0 in previous frames . We define fappear as fappear ( { bim } tm=0 , bjt+1 ) = 1 3× t ∑ attr t∑ m=0 fattr ( b i m , b j t+1 ) , ( 3 ) where attr ∈ { color , material , shape } . fattr ( bm , bt+1 ) equals to 1 when bim and b j t+1 have the same attribute , and 0 otherwise . In Eq . 2 , fappear ensures that the detected trajectories have consistent visual appearance and helps to distinguish the correct object when different objects are close to each other in the same frame . These additional static attributes , including color , material , and shape , are extracted without explicit annotation during training . Specifically , we quantize the attributes by choosing the concept whose concept embedding has the best cosine similarity with the object feature . We iteratively connect proposals at the t + 1 frame to proposals at the t frame and get a set of object trajectories O = { on } Nn=1 , where on = { bnt } Tt=1 . Dynamic Prediction . Given an input video and the refined trajectories of objects , we predict the locations and RGB patches of the objects in future or counterfactual scenes with a Propagation Network ( Li et al. , 2019b ) . We then generate the predicted scenes by pasting RGB patches into the predicted locations . The generated scenes are fed to the feature extractor to extract the corresponding features . Such a design enables the question answer pairs associated with the predicted scenes to optimize the concept embeddings and requires no explicit labels for collision prediction , leading to better optimization . This is different from Yi et al . ( 2020 ) , which requires dense collision event labels to train a collision classifier . To predict the locations and RGB patches , the dynamic predictor maintains a directed graph 〈V , D〉 =〈 { vn } Nn=1 , { dn1 , n2 } N , N n1=1 , n2=1 〉 . The n-th vertex vn is represented by a concatenation of tuple 〈bnt , pnt 〉 over a small time window w , where bnt = [ xnt , ynt , wnt , hnt ] is the n-th object ’ s normalized coordinates and pnt is a cropped RGB patch centering at ( x n t , y n t ) . The edge dn1 , n2 denotes the relation between the n1-th and n2-th objects and is represented by the concatenation of the normalized coordinate difference bn1t − b n2 t . The dynamic predictor performs multi-step message passing to simulate instantaneous propagation effects . During inference , the dynamics predictor predicts the locations and patches at frame k + 1 using the features of the last w observed frames in the original video . We get the predictions at frame k + 2 by autoregressively feeding the predicted results at frame k + 1 as the input to the predictor . To get the counterfactual scenes where the n-th object is removed , we remove the n-th vertex and its associated edges from the input to predict counterfactual dynamics . Iteratively , we get the predicted normalized coordinates { b̂nk′ } N , K′ n=1 , k′=1 and RGB patches { p̂nk′ } N , K n=1 , k′=1 at all predicted K ′ frames . We give more details on the dynamic predictor at Appendix C. Language Program Parsing . The language program parser aims to translate the questions and choices into executable symbolic programs . Each executable program consists of a series of operations like selecting objects with certain properties , filtering events happening at a specific moment , finding the causes of an event , and eventually enabling transparent and step-by-step visual reasoning . Moreover , these operations are compositional and can be combined to represent questions with various compositionality and complexity . We adopt a seq2seq model ( Bahdanau et al. , 2015 ) with an attention mechanism to translate word sequences into a set of symbolic programs and treat questions and choices , separately . We give detailed implementation of the program parser in Appendix D. Symbolic Execution . Given a parsed program , the symbolic executor explicitly runs it on the latent features extracted from the observed and predicted scenes to answer the question . The executor consists of a series of functional modules to realize the operators in symbolic programs . The last operator ’ s output is the answer to the question . Similar to Mao et al . ( 2019 ) , we represent all object states , events , and results of all operators in a probabilistic manner during training . This makes the whole execution process differential w.r.t . the latent representations from the observed and predicted scenes . It enables the optimization of the feature extractor and concept embeddings in the symbolic executor . We provide the implementation of all the operators in Appendix E .

Summary: The paper proposes a new framework, Dynamic Concept Learner (DCL), which learns by watching videos and reading questions/answers.  It is inspired by prior work which combines symbolic representations with video dynamics modeling, but unlike prior work, here the authors do not use any additional supervision except for question/answer pairs. DCL is quiet complex, involving multiple components, including program parser (transforms the question/answer into executable symbolic programs), object detection&tracking, object-centric feature extractor (with static and dynamic components), object/event concept embeddings (share the same latent space as the visual features), trajectory refinement (based on static cues) and dynamic predictor (for future and counterfactual scenes), and, finally, symbolic executor (to predict whether the answer is correct), which runs on the latent features. DCL achieves state-of-the-art results on the CLEVRER dataset. The learned representations are also shown to be useful for other tasks, such as grounding and retrieval.