Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies an interesting problem of counterfactual video prediction, which aims to predict the future frame (D) based on the initial frame (C) and an observed video sequence (AB). AB can be seen as a demonstration that is driven by the same confounders, i.e., the physics parameters such as mass and initial velocities. 

The paper follows the previous models from CoPhyNet for dynamics modeling and confounder estimation, and has two improvements over the work of CoPhy, in my point of view: 
- First, it improves the CoPhy benchmark.
- Second, it presents a new model based on the representations of high-dimensional features, 2D keypoints, and corresponding coefficients. The new form of representation allows the model to be trained in an unsupervised manner only with the supervision of RGB images, as opposed to the training procedure with the supervision of object positions in CoPhyNet. 


main_review: Strengths:
- The paper is well-written.
- The proposed encoding-decoding framework and the corresponding two-stage learning process are reasonable.
- the newly designed representation (i.e, high-dimensional features + 2D keypoints + coefficients) is shown to facilitate unsupervised learning from pixel space.
- The paper provides good empirical results and extensive ablation studies for the effectiveness of each model component.

Weaknesses: 
- The authors may give more empirical analyses about the impact of using different types of interventions. 
- The proposed method is only compared with two existing approaches, including PhyDNet and a modified version of V-CDN. It would be better if the authors could include more existing approaches into the experimental comparison, for example, the keypoint-based model [Minderer et al., 2019], the VAE-based stochastic model [Ref1], and the ConvLSTM-based deterministic model [Ref2]. 
- All experiments are conducted in a simulated environment based on CoPhy, which is somewhat insufficient compared with previous video prediction literature (though I understand the paper studies a new problem). Since the proposed method does not require the supervision of confounders, I wonder if it can be generalized to real-world datasets such as Human3.6M, KTH, or BAIR robot pushing.

[Ref1] Stochastic Video Generation with a Learned Prior. ICML 2018.

[Ref2] PredRNN: Recurrent Neural Networks for Predictive Learning Using Spatiotemporal LSTMs. NIPS 2017.


summary_of_the_review: This paper explores an interesting problem of learning counterfactual physics from pixels. It presents a solution by extending a previous work, CoPhyNet, with new forms of unsupervised video representations. 

One of my concerns is that it does not provide sufficient empirical comparisons with existing video prediction models other than PhyDNet and the modified V-CDN. Furthermore, I would increase my score if the effectiveness of the method can be validated in a real-world dataset.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This work extends CoPhy, in which the author(s) address the problem of predicting counterfactual outcomes of physics-based tasks from pixel space (CoPhy used ground truth object positions). To do this, they learn a keypoint representation of the scene and use them to extract the confounders (such as velocities of objects, masses, etc). The author(s) also propose a new benchmark (built upon CoPhy's benchmark) for counterfactual prediction (after intervening the initial set of objects in the scene) which satisfies the Identifiability and the Counterfactuality constraints of causality.
main_review: Strengths:
1) Filtered-CoPhy seems to be the next intuitive step from CoPhy from a modeling perspective. It is incremental work and a crucial one to get counterfactual predictions (for physics-based simulations) working in an unsupervised fashion.

2) The new benchmark proposal considering Identifiability and Counterfactuality constraints is systematic and alleviates the issues in the CoPhy benchmark.

3) I acknowledge the code released during the review period by the author(s). I have had a chance to briefly look over it and hope that the authors document the code (with a README and instructions on how to run the code) if the paper is accepted.


Weakness:

1) Keypoint based encoder/decoder seems to be a good choice. Have the author(s) tried/given a thought how other object-centric representations such as slot-attention would affect the counterfactual performance?

2) It would make the paper stronger if the CoPhy like baseline is reported i.e one using ground truth object positions. This would give at least an upper bound for the proposed benchmark.

3) It would be beneficial to add a quantitative metric based on the predicted and ground truth location of the center of mass (for both Table 1 and Table 3).

4) Also, since the keypoint points can be tracked, it would add more credibility if tracking metrics such as MOT are added as well. Here the author(s) can assume N=number of objects in the scene (otherwise this metric cannot be computed).

5) During the training stage, can the sequences AB be sent to the CoDy (i.e dynamics model) as well? Since $u_k$ should be the same for AB and CD, this acts as a consistency check.
summary_of_the_review: Overall I feel that the paper is a strong contribution towards visual counterfactual reasoning (making the framework unsupervised as opposed to CoPhy) and would strongly urge the author(s) to add the experiments mentioned in the Weaknesses section. Based on the novelty of the paper, I am voting for a weak acceptance of the paper.

=======
Post-rebuttal decision:
After reading the comments of the authors as well as reviews of the other reviewers, I'm very much satisfied with the additional experiments added in the paper and I believe this paper would be a good contribution towards unsupervised visual counterfactual prediction. Hence, I've increased my score to 8 (Accept).
review_title: null

final_score: rating: 10: strong accept, should be highlighted at the conference
review_content: summary_of_the_paper: The paper introduces a testing approach, dataset, and method for counterfactual video predictions, using 3d physics simulation videos.  Importantly, the approach predicts directly from pixel space rather than requiring spoon-fed keypoints.  It employs a counterfactual approach to establish a network's capacity to learn causal relations.  Separating the problem into one of parsing the inputs into keypoints and additional coefficients, inferring object attributes, and sequential prediction from an input frame + object attributes, it proposes an architecture which is based on combining modules for each of these learning tasks (but where keypoints are learned in an unsupervised way).  Besides this key architectural innovation, it adds an inductive bias by applying directional Gaussian filters to the keypoint maps.  The paper checks that the network actually works as intended by empirically examining the effect of changing object coefficients.  Three other ablation analyses, one that combines two of the modules (rather than separating them via stop-grad) and one that removes coefficients, and one comparing the handcrafted filter bank with learnable ones, add confidence to the approach.  In the supplement, the paper also analyzes in detail how the method compares to a previous method (Transporter).  It uses sensible benchmarks including previous methods and common-sense baselines where either the original or counterfactual input is used as a prediction, and does relatively well on this challenging problem, although the video on the website shows that there is still plenty of room for improvement.
main_review: Strengths
 - Introduces and justifies an important new benchmark task for video predictions involving inanimate objects
 - Introduces a sophisticated approach that is well-described and justified
 - Strong empirical evaluation that includes ablation studies and exploration of the learned representations
 - Thorough comparison with a previous approach that sheds light on reasons for the improvement
 - The counterfactual style of evaluation may be used in other domains of AI

Weaknesses
 - The paper would be even better if the diversity of the benchmark dataset could be improved.
 - Perhaps more could be said about the visible failures of the approach.  The network does not seem to succeed at learning the structure of rigid 3d bodies, as we see from the videos where the cubes visibly distort and lose their edges over successive frames.

Other comments
 - Many incomplete sentences in the supplement.  I would be happy to list them if this would be beneficial to the authors; I assume this was due to time limitations.
summary_of_the_review: This paper introduces an ambitious new baseline for counterfactual 3d physics predictions in pixel space.  It introduces a sophisticated method based on separate modules for image parsing, parameter inference, and prediction, that outperforms two previous approaches V-CDN and PhyDNet.
review_title: null

Meta-review: This paper introduces the Filtered-CoPhy method, an approach for learning counterfactual reasoning of physical processes in pixel space. The approach enables forecasting raw videos over long horizons, without requiring strong supervision, e.g. object positions or scene properties. 

The paper initially received one strong accept, one weak accept, and one weak reject recommendations. The main reviewers' concerns relate to clarifications and consolidations in experiments, including stronger baselines, experiments on real data, or more diversity on the datasets. The rebuttal did a good job in answering reviewers' concerns, especially by providing new experimental results and analysis. Eventually, all reviewers recommended a clear acceptance after authors' feedback. 

The AC's own readings confirmed the reviewers' recommendations. The proposed approach is a meaningful extension of CoPhy for the unsupervised prediction at the pixel level. The proposed approach is solid, clearly described, and overcomes important limitations of previous methods. The dataset is also an important outcome for the community. Causality and counterfactual reasoning are of primary importance for the design of effective and explainable AI prediction models: this paper brings therefore an important contribution to the ICLR community.
Decision: Accept (Oral)