Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes to do visual question answering by learning a differentiable physics model from video. The pipeline is divided to several modular and interpretable parts. Firstly, a visual perception module parse the input video to several objects’ features. Secondly, a concept learner parse the input questions to a set of programs and grounded concepts. The output of the above two modules are used to estimate the objects’ physical parameters and fed to a differentiable physical engine, which is used to do future trajectory predictions. Finally, a symbolic execution part is used to output answer given the future predictions.
main_review: This paper presents a new way to do visual reasoning by incorporating a differentiable physics engine to make future prediction. I think this is a novel introduction of an advanced method to the VQA field. And as the authors claim, the advantage of modularized design is that the intermediate outputs are interpretable so it’s easier to identify the reason of failure. More importantly, thanks to the differentiable simulator, the method does not lose the ability to end-to-end optimization.

In addition, incorporating the differentiable physics engine can greatly improve the sample efficiency due to the introduction of human prior. The author verifies this idea by showing the proposed method is several times better than previous state-of-the-art under the 20% training data regime and the few-shot learning setting.

The paper is also well-written and very easy to follow.

Questions:
1. Have the authors considered how to apply this to more complex environments? For example, most of the objects in CLEVERER are blocks and spheres and the current physics simulation model assume to be a mass-point. How would the proposed method be extended to more complex objects’ shapes?
2. Have the authors considered another ablation experiments: using learned dynamics prediction model to replace the differentiable physics simulator? For example, what if we use the models from [51, 59] to do simulation?
limitations_and_societal_impact: I didn't find any potential negative societal impact of this work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The paper tackles the problem of dynamic visual reasoning by using a modular unit that is composed of three components: visual perception, concept learner, and physics simulator. This system is capable of performing system identification from visual and language modalities with assistance from a physics model. Having access to the physics model, it can produce imagined future trajectories for reasoning about possible future outcomes. This gave them an advantage of outperforming competitive methods, especially in counterfactual tasks.

main_review: Overall:

It’s a well-written paper, that formulates a problem, proposes a solution, and does a reasonable analysis of the results. The results are strong and competitive especially in the scenarios such as counterfactual where they do argue why their method should perform better. 

While the current system could have a great many applications in a controlled robotics environments, the system looks a bit brittle and specialized and it’s not clear how this system could be generalized to other datasets and applications.

Details:

Is understanding physical laws and having a physics engine a necessary condition for physics reasoning? The intuitive physics in humans suggests otherwise. While it’s very reasonable to have a physical simulator for robotics applications, it might not necessarily be a requirement for AGI.

Would this model be capable to expand more complicated physical models, such as complicated contact dynamics that happens in manipulation? 

By just reading the main text alone the mechanism of “Symbolic programs” and “Symbolic excitation” are not clear. While the authors cite relevant work and have a section in the appendix, the text needs to be self-sufficient and it would be great to have a section to explain those in more detail.

How does the computation efficiency of VRDP compare to the other methods? 
limitations_and_societal_impact: It has been addressed in the paper.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: - The paper presents a composite model for visual question answering in scenarios which require dynamic visual reasoning.
- The model pipeline consists of a visual perception module which extracts object-centric trajectories from an input video, a concept learner which parses the natural language QA pair into a set of neurosymbolic concept vectors, a differentiable physics model which can simulate the evolution of rigid body trajectories and a neurosymbolic program executor which computes or selects the answer for a given input question using the intermediate concept and trajectory features.
- As a contribution over prior art in neurosymbolic visual reasoning, the authors propose the utilisation of a differentiable physics module for accurate trajectory prediction based on physics parameters which are inferred from the video input sample. Since explicit physical properties like mass, friction and restitution are inferred in the model, the computed output is very amenable to human interpretation.
- The proposed approach is comprehensively evaluated on the synthetic CLEVRER and realistic Real-billiard benchmarks and compared against related neurosymbolic and end-to-end VQA models. It demonstrates remarkable boosts in data efficiency on CLEVRER, sets a new state-of-the-art performance on counterfactual questions and significantly outperforms prior art in a few-shot setup where generalisation to modified physical attributes like 'heavier' or 'lighter' is required.
main_review: TL;DR: The paper proposes a clean neurosymbolic pipeline approach for visual question answering involving dynamical reasoning. The introduction of a differential physics module seeded from object collisions massively increases data efficiency and few-shot generalization compared to prior art. The approach has been clearly presented and comprehensively evaluated on two established benchmarks featuring synthetic and real videos depicting rigid body collisions. All claims are corroborated by experiments, the limitations of the approach are outlined. A very good paper which is recommended for acceptance. It would be nice, if the authors provided code and data used in their experiments upon publication.


### Pros and Cons
- Pros
  - The utilisation of a differentiable physics engine affords massive boost in data efficiency during learning and improves performance, especially in counterfactual QAs.
  - The approach is comprehensively evaluated on synthetic and real data; a small-scale dataset extension has been collected to evaluate the model for few-shot generalization to new physical concepts like 'heavier' or 'lighter'.
- Cons
  - The paper only provides sparse details on concept embedding and neurosymbolic program execution -> Maybe some details about those parts of the pipeline should be added to appendix to make the paper more self-contained for the unfamiliar reader?
  - The proposed approach relies heavily on collision events and exact modelling of physics surrounding it.
  - There is no clear evidence presented that the model performance delta over end-to-end models can hold in high-data regimes.
  - The small scale dataset for generalization experiments should have been provided as supplementary material or reported in the appendix to gauge the data distribution of the QAs collected.

### Originality
- Introducing prior knowledge about rigid body physics (especially collision events) affords massive data efficiency boosts on the physical reasoning side since parameters can be fitted well per sample input video and accurate forward prediction can be achieved.
- While this observation about the benefits of prior knowledge injection into a QA model is not novel per se, the comprehensive evaluation of the approach in low-data regimes and counterfactual scenarios is a worthwhile contribution to this field.

### Quality
- typos, formatting errors:
  - l. 61: **a** motion heuristic
  - l. 140: **physics** engine
  - l. 152, 153, 197: coordinate **frame**
  - l. 202: **is** acting on
  - l. 218-219: from ~~less~~ fewer to more frames
  - l. 231: program **outputs** the number
  - l. 263: follow the experimental setting in [75, 16] ~~that~~ using a
  - l. 267, 282: object attribute **supervision**

### Clarity
- The paper is consistently well written and the model pipeline is presented in a clean and understandable way.
- All claims, especially w.r.t. data efficiency and few-shot generalization are sufficiently corroborated in the experiments section.
- There are a few open questions, please clarify during the discussion phase:
  - l. 155, 162: Is `b^n_t` a coordinate since it is used for distance computation?
  - l. 171: How is the concept space being trained?
  - ll. 173-174: What happens to unassigned concepts which aren't visually grounded?
  - How are trajectory features being computed?

### Significance
- Physical reasoning from visual input is highly relevant for visual understanding and applications such as robotic manipulation or navigation, doubly so when paired with natural language descriptions and data-efficient training and inference techniques.
- The neuro-symbolic grounding and forward predictive aspects of the proposed VRDP model make it highly relevant for applications in object-centric scene inference or model-based planning.

### Remarks
- The approach relies heavily on collision events to estimate sample-dependent physics parameters. -> How could this be done in collision-free videos?
- The physics engine only helps to generalize physical reasoning, but embeddings of visual and textual content can mainly be improved by data (as recently shown in multi-modal transformers). -> Are there forseeable cases where more data wouldn't be beneficial to better trajectory prediction, e.g. where [19] would saturate but VRDP keeps improving?
- How does the model fare in very crowded scenes where many enter/exit events occur? Which enhancements to the visual perception module and dynamics heuristic would be needed in such cases?
limitations_and_societal_impact: - The authors have provided a comprehensive list of failure modes of their model in the 'Failure Analysis' paragraph of section 4.2.
- The foreseeable societal impact of the method proposed has been discussed adequately in the Broader Impact section.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper introduces VRDP, a method that learns to answer predictive, counterfactual, explanatory and descriptive yes or no questions about videos depicting the evolution of physical scenes. VRDP is composed of three modules, an object detector, a neuro-symbolic concept learner and differentiable physics model.

The paper shows that VRDP surpasses or closely matches the SOTA on two datasets, a synthetic dataset CLEVERER and a dataset of real world videos.
main_review: This paper introduces VRDP, a system that learns to perform question answering on videos of physical scenes.

The paper is well written, well organized and it properly places the contributions presented in the context of existing and related work.

I like the idea of combining distinct modules for visual perception, concept learning and physics simulation into a question answering system. I am impressed with the results and I would love to make some suggestions, which I hope can addressed in a revised version, that I think might help make this a more impactful paper.

1) The manuscript claims that VRDP is more interpretable than alternative approaches, however this is never defined or quantified. In particular, there is a large body of work on interpreting transformer based systems and the original Object Based Attention paper itself has an extensive attention analysis. Could the authors clarify what is meant by interpretability and provide qualitative or quantitative comparisons with alternative approaches? These could include e.g. linear readouts from transformer embeddings.

2) VRDP is substantially more data efficient than Object Based Attention. However, VRDP uses a pre-trained object detection module, whereas Object Based Attention trains its system end to end. It would be good to maybe include a discussion of this, it's certainly a positive to not have to reinvent the wheel with every new experiment, but one could imagine situations where end to end training could be beneficial. One possible way to take a crack at this would be to re-train an Object Based Attention system based on the output of VRDP's visual perception module and see if the sample complexity gap is still there.

3) The comparison to the current state of the art is limited to a single dataset , could the two additional datasets used in Object Based Attention be included as well? If not, could the authors please provide a discussion of what makes the comparison hard or impossible?

I would like to thank the authors for sharing these cool ideas and for writing a clear and easy to follow paper.
Physics based question answering directly from perception is an open, interesting and relevant problem. The results presented here on the CLEVERER dataset are promising, and the interpretability and reduced sample complexity are exciting prospects for a system such as this.

I hope the authors agree that my suggestions might make the paper stronger and take them into consideration. In its current form my impression is that this paper is not ready to be shared for the wider community because the results section is limited and it does not substantiate claims of improved sample complexity and interpretability.

I think these could easily be addressed in a revised version.

Thank you again for sharing these cool ideas, and I look forward to your rebuttal. -- All the best.
limitations_and_societal_impact: Limitations and societal impact are properly addressed.
review_title: null

Meta-review: The paper proposes a new task and method for dynamic visual question answering based on neuro-symbolic reasoning and the usage of a differentiable physics engine. It has received reviews from four experts, who appreciated the new formulation with the differentiable engine (and its effect on counterfactual tasks), the modular and interpretable design, and a well written and presented paper. 

Minor issues were raised on the model-based nature of the method --- the advantage of having a physics engine is also a shortcoming, since the method requires a model of the physical processes, and on evaluation, comparisons.

The authors' responses were convincing, in particular some requested experiments (eg re: pre-training on objects) where appreciated by the reviewers, and a consensus for acceptance emerged.

The AC recommends acceptance.
Decision: Accept (Poster)