Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Language

1 Introduction . Dynamic visual reasoning about objects , relations , and physics is essential for human intelligence . Given a raw video , humans can easily use their common sense of intuitive physics to explain what has happened , predict what will happen next , and infer what would happen in counterfactual situations . Such human-like physical scene understanding capabilities are also of great importance in practical applications such as industrial robot control [ 2 , 53 ] . 1Project page : http : //vrdp.csail.mit.edu/ 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . Previous works have made great efforts to build artificial intelligence ( AI ) models with such physical reasoning capabilities . One popular strategy is to develop pure neural-network-based models [ 63 , 19 , 40 ] . These methods typically leverage end-to-end neural networks [ 32 , 35 ] with powerful attention modules such as Transformer [ 69 , 21 ] to extract attended features from both video frames and question words , based on which they answer questions directly . Despite their high question-answering accuracy on CLEVRER [ 79 ] , a challenging dynamic visual question-answering benchmark , these black-box models neither learn concepts nor model objects ’ dynamics . Therefore , they lack transparency , interpretability , and generalizability to new concepts and scenarios . Another common approach to dynamic visual reasoning is to build graph neural networks ( GNNs ) [ 47 ] to capture the dynamics of the scenes . These GNN models [ 54 , 79 , 16 ] treat objects in the video as nodes and perform objectand relation-centric updates to predict objects ’ dynamics in future or counterfactual scenes . Such systems achieve decent performance with good interpretability on CLEVRER by combining the GNN-based dynamics models with neural-symbolic execution [ 58 , 80 ] . However , these dynamic models do not explicitly consider laws of physics or use concepts encoded in the question-answer pairs associated with the videos . As a result , they show limitations in counterfactual situations that require long-term dynamics prediction . Although ( graph- ) neural-network-based approaches have achieved competitive performance on CLEVRER , dynamic visual reasoning is still far from being solved perfectly . In particular , due to the lack of explicit physics models , existing models [ 79 , 19 , 16 ] typically struggle to reason about future and counterfactual events , especially when training data is limited . For this reason , one appealing alternative is to develop explicit physics-based methods to model and reason about dynamics , as highlighted in the recent development of differentiable physics engines [ 9 , 17 , 68 , 18 , 66 ] and their applications in robotics [ 9 , 17 , 68 ] . However , these physics engines typically take as input a full description of the scene ( e.g. , the number of objects and their shapes ) which usually requires certain human priors , limiting their availability to applications with well-defined inputs only . In this work , we take an approach fundamentally different from either network-based methods or physics-based methods . Noting that deep learning based methods excel at parsing objects and learning concepts from videos and language , and physics laws are good at capturing object dynamics , we propose Visual Reasoning with Differentiable Physics ( VRDP ) , a unified framework that combines a visual perception module , a concept learner , and a differentiable physics engine . VRDP jointly learns object trajectories , language concepts , and objects ’ physics models to make accurate dynamic predictions . It starts with a perception module running an object detector [ 31 ] on individual frames to generate object proposals and connect them into trajectories based on a motion heuristic . Then , a concept learner learns object- and event-based concepts , such as ‘ shape ’ , ‘ moving ’ , and ‘ collision ’ as in DCL [ 16 , 58 ] . Based on the obtained object trajectories and attributes , the differentiable physics engine estimates all dynamic and physical properties ( e.g. , velocity , angular velocity , restitution , mass , and the coefficient of resistance ) by comparing the simulated trajectories with the video observations . With these explicit physical parameters , the physics engine reruns the simulation to reason about future motion and causal events , which a program executor then executes to get the answer . The three components of VRDP cooperate seamlessly : the concept learner grounds physical concepts needed by the physics engine like ‘ shape ’ onto the objects detected by the perception module ; the differentiable physics engine estimates all physical parameters and simulates accurate object trajectories , which in turn help the concept learning process in the concept learner . Compared with existing methods , VRDP has several advantages thanks to its carefully modularized design . First , it achieves the state-of-the-art performance on both synthetic videos ( CLEVRER [ 79 ] ) and real-world videos ( Real-Billiard [ 63 ] ) without sacrificing transparency or interpretability , especially in situations that require long-term dynamics prediction . Second , it has high data efficiency thanks to the differentiable physics engine and symbolic representation . Third , it shows strong generalization capabilities and can capture new concepts with only a few examples . 2 Related Work . Visual Reasoning Our model is related to reasoning on vision and natural language . Existing works can be generally categorized into two streams as end-to-end approaches [ 40 , 83 , 74 , 41 , 5 ] and neurosymbolic approaches [ 80 , 29 , 30 , 24 , 4 , 59 , 45 , 58 , 36 , 3 ] . The end-to-end methods [ 40 , 83 , 74 , 60 ] typically tackle the visual question answering ( VQA ) problem by designing monolithic multi-modal deep networks [ 32 , 35 ] . They directly output answers without explicit and interpretable mechanisms . Beyond question answering , neuro-symbolic methods [ 80 , 29 , 59 , 45 , 58 ] propose a set of visualreasoning primitives , which manifest as an attention mechanism capable of performing complex reasoning tasks in an explicitly interpretable manner . Dynamic visual reasoning in videos has attracted much research attention . Many video question answering datasets [ 27 , 49 , 42 , 67 , 81 ] and the methods [ 82 , 51 , 38 , 75 , 78 , 22 ] built on them mainly focus on understanding diverse visual scenes , such as human actions ( MovieQA [ 67 ] ) or 3D object movements without physical and language cues ( CATER [ 27 ] ) . Differently , CLEVRER [ 79 ] targets the physical and causal relations grounded in dynamic videos of rigid-body collisions and asks a range of questions that requires the modeling of long-term dynamic predictions . For this reason , we evaluate our method and compare it with other state-of-the-arts on CLEVRER . Both end-to-end [ 63 , 19 ] and neuro-symbolic methods [ 79 , 16 ] have been explored on CLEVRER . However , they either lack transparency or struggle for long-term dynamic prediction . In this paper , we perform high-performance and interpretable reasoning by recovering the physics model of objects and their interactions ( e.g. , collisions ) from visual perception and language concepts . Physical Models Physical models are widely used in video prediction [ 50 , 23 , 76 , 77 ] , neural simulation and rendering [ 52 , 54 ] , and dynamic reasoning [ 10 , 71 ] . For example , PhysNet and its variants [ 50 , 23 , 76 , 77 ] leverage global or object-centric deep features to predict the physical motion in video frames . Some other related works [ 61 , 1 , 65 , 46 ] extend physical models to predict the effect of forces and infer the geometric attributes and topological relations of cuboids . In this work , we focus on dynamic visual reasoning about object interactions , dynamics , and physics with question answering , which is central to human intelligence and a key goal of artificial intelligence . Solving such tasks requires a good representation and understanding of physics models . A common choice is to train a deep neural network for physical property estimation ( e.g. , location and velocity ) based on learned visual and dynamic priors [ 15 , 10 , 71 , 70 , 43 , 54 , 55 , 34 , 33 ] . However , since these neural networks do not model physics laws , generalizing them to unseen events or counterfactual scenarios could result in unexpected results . Our work is different and more physics-based : Inspired by the recent advances in differentiable physics [ 9 , 17 , 68 , 18 , 66 , 37 ] , we implement an impulsebased differentiable rigid-body simulator and leverage the power of its gradients to infer dynamics information about the scene . Physical Scene Understanding Our work is also relevant to studies on physical scene understanding [ 7 , 73 , 66 , 64 , 8 , 25 , 26 , 20 , 50 , 39 ] , most of which propose pure neural-network solutions without explicitly incorporating physics models . Benchmarks like PHYRE [ 7 ] study physical understanding and reasoning based on pure videos without concept learning and language inference . Based on such benchmarks , some works [ 72 , 11 , 50 , 54 ] learn compositional scene structure and estimate states through physical motions and visual de-animation . Recently , two papers propose pure physics-based methods [ 44 , 39 ] that make heavy use of differentiable physics simulators , but they typically assume concepts in the scene are given as input . Our work is unique in that we learn video and language concepts from raw videos and infer dynamics information from a differentiable simulator , combining the benefits of both learning and physics . 3 Method . By integrating differentiable physics into the dynamic reasoning framework , VRDP jointly learns visual perception , language concepts , and physical properties of objects . The first two provide prior knowledge for optimizing the third one to reason about the physical world , and the optimized physical properties in turn help to learn better concepts . In the following , we first give an overview of our framework and then describe each of its components in detail . 3.1 Framework Overview . An overview of VRDP is illustrated in Fig . 1 . It contains three components : a visual perception module , a concept learner , and a physics model . The input to the framework is a video and reasoning questions , where the former is processed by the visual perception module to get object trajectories and corresponding visual features , and the latter is parsed into executable symbolic programs with language concepts by the concept learner . Similar to DCL [ 58 , 16 ] , the concept learner first grounds object properties ( e.g. , color and shape ) and event concepts ( e.g. , collision and moving ) by aligning the visual features and the corresponding concept embeddings in the ( explanatory or descriptive ) program that does not require dynamic predictions , e.g. , “ what is the shape of ... ” . With those perceptually grounded object trajectories and properties , the physical model then performs differentiable simulation to learn all physical parameters of the scene and objects by comparing the simulated trajectories with the video observations . After that , the physics engine simulates unseen trajectories for predictive and counterfactual scenarios and generates their features , in turn enabling the concept learner to finetune event concepts from the program that requires dynamic predictions , e.g. , “ what will happen ... ” and “ what if ... ” . Finally , a symbolic executor executes the parsed programs with the dynamic predictions to get the answer .

Summary: This paper proposes to do visual question answering by learning a differentiable physics model from video. The pipeline is divided to several modular and interpretable parts. Firstly, a visual perception module parse the input video to several objects’ features. Secondly, a concept learner parse the input questions to a set of programs and grounded concepts. The output of the above two modules are used to estimate the objects’ physical parameters and fed to a differentiable physical engine, which is used to do future trajectory predictions. Finally, a symbolic execution part is used to output answer given the future predictions.