Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language

1 Introduction . Dynamic visual reasoning about objects , relations , and physics is essential for human intelligence . Given a raw video , humans can easily use their common sense of intuitive physics to explain what has happened , predict what will happen next , and infer what would happen in counterfactual situations . Such human-like physical scene understanding capabilities are also of great importance in practical applications such as industrial robot control [ 2 , 53 ] . 1Project page : http : //vrdp.csail.mit.edu/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Previous works have made great efforts to build artificial intelligence ( AI ) models with such physical reasoning capabilities . One popular strategy is to develop pure neural-network-based models [ 63 , 19 , 40 ] . These methods typically leverage end-to-end neural networks [ 32 , 35 ] with powerful attention modules such as Transformer [ 69 , 21 ] to extract attended features from both video frames and question words , based on which they answer questions directly . Despite their high question-answering accuracy on CLEVRER [ 79 ] , a challenging dynamic visual question-answering benchmark , these black-box models neither learn concepts nor model objects ’ dynamics . Therefore , they lack transparency , interpretability , and generalizability to new concepts and scenarios . Another common approach to dynamic visual reasoning is to build graph neural networks ( GNNs ) [ 47 ] to capture the dynamics of the scenes . These GNN models [ 54 , 79 , 16 ] treat objects in the video as nodes and perform objectand relation-centric updates to predict objects ’ dynamics in future or counterfactual scenes . Such systems achieve decent performance with good interpretability on CLEVRER by combining the GNN-based dynamics models with neural-symbolic execution [ 58 , 80 ] . However , these dynamic models do not explicitly consider laws of physics or use concepts encoded in the question-answer pairs associated with the videos . As a result , they show limitations in counterfactual situations that require long-term dynamics prediction . Although ( graph- ) neural-network-based approaches have achieved competitive performance on CLEVRER , dynamic visual reasoning is still far from being solved perfectly . In particular , due to the lack of explicit physics models , existing models [ 79 , 19 , 16 ] typically struggle to reason about future and counterfactual events , especially when training data is limited . For this reason , one appealing alternative is to develop explicit physics-based methods to model and reason about dynamics , as highlighted in the recent development of differentiable physics engines [ 9 , 17 , 68 , 18 , 66 ] and their applications in robotics [ 9 , 17 , 68 ] . However , these physics engines typically take as input a full description of the scene ( e.g. , the number of objects and their shapes ) which usually requires certain human priors , limiting their availability to applications with well-defined inputs only . In this work , we take an approach fundamentally different from either network-based methods or physics-based methods . Noting that deep learning based methods excel at parsing objects and learning concepts from videos and language , and physics laws are good at capturing object dynamics , we propose Visual Reasoning with Differentiable Physics ( VRDP ) , a unified framework that combines a visual perception module , a concept learner , and a differentiable physics engine . VRDP jointly learns object trajectories , language concepts , and objects ’ physics models to make accurate dynamic predictions . It starts with a perception module running an object detector [ 31 ] on individual frames to generate object proposals and connect them into trajectories based on a motion heuristic . Then , a concept learner learns object- and event-based concepts , such as ‘ shape ’ , ‘ moving ’ , and ‘ collision ’ as in DCL [ 16 , 58 ] . Based on the obtained object trajectories and attributes , the differentiable physics engine estimates all dynamic and physical properties ( e.g. , velocity , angular velocity , restitution , mass , and the coefficient of resistance ) by comparing the simulated trajectories with the video observations . With these explicit physical parameters , the physics engine reruns the simulation to reason about future motion and causal events , which a program executor then executes to get the answer . The three components of VRDP cooperate seamlessly : the concept learner grounds physical concepts needed by the physics engine like ‘ shape ’ onto the objects detected by the perception module ; the differentiable physics engine estimates all physical parameters and simulates accurate object trajectories , which in turn help the concept learning process in the concept learner . Compared with existing methods , VRDP has several advantages thanks to its carefully modularized design . First , it achieves the state-of-the-art performance on both synthetic videos ( CLEVRER [ 79 ] ) and real-world videos ( Real-Billiard [ 63 ] ) without sacrificing transparency or interpretability , especially in situations that require long-term dynamics prediction . Second , it has high data efficiency thanks to the differentiable physics engine and symbolic representation . Third , it shows strong generalization capabilities and can capture new concepts with only a few examples . 2 Related Work . Visual Reasoning Our model is related to reasoning on vision and natural language . Existing works can be generally categorized into two streams as end-to-end approaches [ 40 , 83 , 74 , 41 , 5 ] and neurosymbolic approaches [ 80 , 29 , 30 , 24 , 4 , 59 , 45 , 58 , 36 , 3 ] . The end-to-end methods [ 40 , 83 , 74 , 60 ] typically tackle the visual question answering ( VQA ) problem by designing monolithic multi-modal deep networks [ 32 , 35 ] . They directly output answers without explicit and interpretable mechanisms . Beyond question answering , neuro-symbolic methods [ 80 , 29 , 59 , 45 , 58 ] propose a set of visualreasoning primitives , which manifest as an attention mechanism capable of performing complex reasoning tasks in an explicitly interpretable manner . Dynamic visual reasoning in videos has attracted much research attention . Many video question answering datasets [ 27 , 49 , 42 , 67 , 81 ] and the methods [ 82 , 51 , 38 , 75 , 78 , 22 ] built on them mainly focus on understanding diverse visual scenes , such as human actions ( MovieQA [ 67 ] ) or 3D object movements without physical and language cues ( CATER [ 27 ] ) . Differently , CLEVRER [ 79 ] targets the physical and causal relations grounded in dynamic videos of rigid-body collisions and asks a range of questions that requires the modeling of long-term dynamic predictions . For this reason , we evaluate our method and compare it with other state-of-the-arts on CLEVRER . Both end-to-end [ 63 , 19 ] and neuro-symbolic methods [ 79 , 16 ] have been explored on CLEVRER . However , they either lack transparency or struggle for long-term dynamic prediction . In this paper , we perform high-performance and interpretable reasoning by recovering the physics model of objects and their interactions ( e.g. , collisions ) from visual perception and language concepts . Physical Models Physical models are widely used in video prediction [ 50 , 23 , 76 , 77 ] , neural simulation and rendering [ 52 , 54 ] , and dynamic reasoning [ 10 , 71 ] . For example , PhysNet and its variants [ 50 , 23 , 76 , 77 ] leverage global or object-centric deep features to predict the physical motion in video frames . Some other related works [ 61 , 1 , 65 , 46 ] extend physical models to predict the effect of forces and infer the geometric attributes and topological relations of cuboids . In this work , we focus on dynamic visual reasoning about object interactions , dynamics , and physics with question answering , which is central to human intelligence and a key goal of artificial intelligence . Solving such tasks requires a good representation and understanding of physics models . A common choice is to train a deep neural network for physical property estimation ( e.g. , location and velocity ) based on learned visual and dynamic priors [ 15 , 10 , 71 , 70 , 43 , 54 , 55 , 34 , 33 ] . However , since these neural networks do not model physics laws , generalizing them to unseen events or counterfactual scenarios could result in unexpected results . Our work is different and more physics-based : Inspired by the recent advances in differentiable physics [ 9 , 17 , 68 , 18 , 66 , 37 ] , we implement an impulsebased differentiable rigid-body simulator and leverage the power of its gradients to infer dynamics information about the scene . Physical Scene Understanding Our work is also relevant to studies on physical scene understanding [ 7 , 73 , 66 , 64 , 8 , 25 , 26 , 20 , 50 , 39 ] , most of which propose pure neural-network solutions without explicitly incorporating physics models . Benchmarks like PHYRE [ 7 ] study physical understanding and reasoning based on pure videos without concept learning and language inference . Based on such benchmarks , some works [ 72 , 11 , 50 , 54 ] learn compositional scene structure and estimate states through physical motions and visual de-animation . Recently , two papers propose pure physics-based methods [ 44 , 39 ] that make heavy use of differentiable physics simulators , but they typically assume concepts in the scene are given as input . Our work is unique in that we learn video and language concepts from raw videos and infer dynamics information from a differentiable simulator , combining the benefits of both learning and physics . 3 Method . By integrating differentiable physics into the dynamic reasoning framework , VRDP jointly learns visual perception , language concepts , and physical properties of objects . The first two provide prior knowledge for optimizing the third one to reason about the physical world , and the optimized physical properties in turn help to learn better concepts . In the following , we first give an overview of our framework and then describe each of its components in detail . 3.1 Framework Overview . An overview of VRDP is illustrated in Fig . 1 . It contains three components : a visual perception module , a concept learner , and a physics model . The input to the framework is a video and reasoning questions , where the former is processed by the visual perception module to get object trajectories and corresponding visual features , and the latter is parsed into executable symbolic programs with language concepts by the concept learner . Similar to DCL [ 58 , 16 ] , the concept learner first grounds object properties ( e.g. , color and shape ) and event concepts ( e.g. , collision and moving ) by aligning the visual features and the corresponding concept embeddings in the ( explanatory or descriptive ) program that does not require dynamic predictions , e.g. , “ what is the shape of ... ” . With those perceptually grounded object trajectories and properties , the physical model then performs differentiable simulation to learn all physical parameters of the scene and objects by comparing the simulated trajectories with the video observations . After that , the physics engine simulates unseen trajectories for predictive and counterfactual scenarios and generates their features , in turn enabling the concept learner to finetune event concepts from the program that requires dynamic predictions , e.g. , “ what will happen ... ” and “ what if ... ” . Finally , a symbolic executor executes the parsed programs with the dynamic predictions to get the answer .

Summary: This paper introduces VRDP, a method that learns to answer predictive, counterfactual, explanatory and descriptive yes or no questions about videos depicting the evolution of physical scenes. VRDP is composed of three modules, an object detector, a neuro-symbolic concept learner and differentiable physics model.  The paper shows that VRDP surpasses or closely matches the SOTA on two datasets, a synthetic dataset CLEVERER and a dataset of real world videos.