Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper proposes a pipeline for predicting the future steps of the physical simulation starting from the video input from multiple cameras. The model builds a Nerf model from several input frames, samples the particles from Nerf and runs the particle-based simulator using the extracted particle representation.

strength_and_weaknesses: 
## Strengths

This paper is one of the few approaches that are able to perform the dynamics from the video inputs. Unlike the previous approach Nerf-DY, this paper uses a more structured representation of the simulation. That allow the model to generalise to new sizes of containers and different amount of fluid, which is consistent with the findings in the previous papers. The paper connects the two approaches that are known to work well from the previous works: PixelNerf (for a high-res 3D model) and particle-based simulators (for fluid modelling and generalisation). The past works have already shown the strong generalisation to new scenes due to locality of the particle interactions, so it is not surprising to see the similar results in this paper too.

## Weaknesses

There are a few weaknesses in the method that limit the applicability of the work to different systems.

The method requires a major knowledge about the underlying simulation states, which defeats the purpose of learning from video. Specifically:
- Knowledge of the 3D meshes of the robot/container/pusher and their positions/velocities at every time step. This is a strong assumption, because these meshes constitute a large part of the state. The meshes are also notoriously hard to extract from the video in the real-world scenarios.
- Knowledge of fluid density to set the minimum distance for the spacing loss.
- The model relies on the fact that the fluid is coloured and is easy to distinguish from other objects only based on the blue color. Similarly, the objects need to be of different colors, always visible and are easy to track across time.

Another major drawback is that the approach still requires to have the particle simulation as the ground-truth. The mesh/particles and the initial state are notoriously hard to obtain, preventing the previous simulators to train on the real-world data. Therefore, learning directly from videos is desirable. However, the proposed model unable to learn entirely from the video data. Moreover, it requires a corresponding ground-truth particle simulation for every step. It defeats the whole point of learning simulation from videos. 

### Representing the correct fluid dynamics.

*Supplementary Section B.4*: “If the particle belongs to the water, then we have no history states, so the input of Qv is all-zero. …. Compared with human intuition, we can get an intuitive prediction of the movement of water by simply knowing the past movement of the cup without knowing the past movement of water”

I don’t think this is true. Consider the case when the we start moving the cup back and forth, and we start recording the simulation when the cup is already moving. The fluid particles already have the prior velocity that is not the same as the velocity of the cup. It is not possible to infer the correct fluid dynamics without the history of fluid particles or velocities.

Another example from the paper is the fluid falling from one container to another.  Velocities of the falling particles depend on gravity and the previous particle velocity. Here we might be able to infer the fluid velocities **only** if we assume that fluid velocities is zero in the beginning of the simulation and we have access to the **entire** history of states. In all other cases, it is not possible to represent the correct dynamics of the fluid.

The fact that the videos look pretty good without the velocity information may indicate that the model exploits some biases from the data that allow to roughly imitate the dynamics under the point-cloud Chamfer loss. Specifically, I suspect that the model misuses the heuristic that the fluid particles have roughly the same as the velocities as the pusher/constrainer. It is visible in the attached videos of the ablations with and without the spacing loss: the particles that do not interact with the pusher are moving with the same speed as the pusher itself. The spacing loss also introduces a strong inductive bias that the fluid density remains constant. 

It would be helpful to add the comparison of the dynamics component to the ground truth, starting from the known particle state and evaluated under the Chamfer distance, to compare the correctness of the predicted fluid dynamics. This should be possible to do, as the paper assumes having the ground truth particle simulation.

Authors should be honest about the very limited range of applicability of their model: the fluid velocity can be inferred from the velocities of the surrounding objects, which is a rare case. Even a simple case of the falling block of fluid cannot be represented by the current model. 
clarity,_quality,_novelty_and_reproducibility: The idea of sampling the particle representation from the NeRF model is new. The paper is well-motivated, because being able to learn the simulation from the video data without knowing the ground-truth meshes/particles high desirable. However, the paper does not deliver on this motivation.  

The paper does not emphasise that the approach requires a lot of knowledge about the 3D simulation state. The paper also does not clearly mention that actually still requires the ground-truth particle simulations and compute the loss, and therefore the model cannot learn purely from image data. The simulation has to be specifically constructed, such that we can distinguish fluid from other objects (fluid is colored and objects that have to be of different colors)

Additionally, the paper has severe limitations in terms of the systems that it can represent. As stated in section B.4, the fluid particles do not have the history of states, and their velocity is set to zero, and thus the model relies on the *velocities of the objects* to infer the dynamics of the fluid, leading to the incorrect fluid dynamics, as mentioned above. It is unacceptable to put this detail into the last page of the supplementary. The authors should be honest about the applicability and limitations of their model and adequately assess whether the predicted dynamics is, in fact, correct.

Additional questions:

1. Figure 5 shows the comparison of the Chamfer distance between Nerf-DY and the proposed method. How was the Chamfer (point-cloud) distance computed for Nerf-DY, if Nerf-DY performs the dynamics in the latent space and does not have the notion of particles?
2. Table 1 and Figure 4 provide the comparison between the perception modules. Does “Our model” provide only the comparison to PixelNerf? Or are there other parts of the model besides PixelNerf that are included into the “our model” row?
3. Is the particle simulator pre-trained or trained together with the model.
4. Is the proposed model trained only on 1-step prediction or multi-step prediction?
summary_of_the_review: This paper is one of the few approaches that performs the simulation from the video inputs, which is novel.However, this approach is severely limited in the environments that it can model (namely, simulated environments with colored fluids and differently-coloured objects), assumes to know that full mesh and state of the robot/container and relies on the velocities of the known objects to infer the velocities of the fluid. The authors need to be clear about the limitations of their work and the environments where the method is applicable: because fluid particles do not have velocities or past history, it would not be able to produce the correct fluid dynamics. I suggest rejecting the paper due to severe limitations of the approach that are not clearly stated.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes to learn a visual dynamics prediction model in the 3D space. The 3D point cloud representation is learned only from a few images from multiple views (with known camera pose). After that, they use instance segmentation to parse individual objects from the point cloud. Then it trains a graph-based prediction model on the 3D representations, without relying on the ground-truth correspondence, using the chamfer and spacing loss. The proposed method is evaluated using the prediction loss on a few simulation tasks including Pour, Shake, and Push.
strength_and_weaknesses: Strength:
1) The idea of combining NeRF and graph-based dynamics prediction is interesting. Results are also visually good.
2) The writing is pretty clear and complete. It’s easy to read and understand the main message of the paper. The figures are also nice and well-designed.
3) Code is provided. This could be a good asset to the community.

Weakness:
1) Most of components are adopted from previous works, including image-conditioned nerf and training dynamics prediction using chamfer distance. Getting instance point cloud from segmentation is new but this is relatively easy (color segmentation) and this is not as “imperfect” as the authors claimed in the abstract given the relatively simple scene. It remains unclear if this pipeline can be used in other complex scenarios or even in the real-world.
2) The evaluation is purely based on visual quality while previous work (Li et al. 2021b) also evaluate on control accuracy.
3) What is the merge loss in Figure 2? Does that mean the spacing loss?
clarity,_quality,_novelty_and_reproducibility: I think the paper is well-written. Novelty may not be its advantage but there is still enough contribution to the community. The code is provided so I’m not concerned about its reproducibility.
summary_of_the_review: In summary, I think some aspects of this paper is worth reading to the community. There are still some discussion and experiments missing as I pointed out in the weakness section and it may still need some work. I’m also happy to discuss and increase my score if the above concerns are addressed.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper presents a method for learning visual intuitive physics models solely from unlabeled multi-view images. Specifically, the method first leverages a conditional NeRF to learn the 3D geometry for the scene, which are then segmented and clustered to produce the 3D point cloud representation for the target fluid, rigid bodies and granular materials. A graph neural network is then applied to learn the dynamics of 3D points. The experiments demonstrate that the methods can produce good visual dynamics and generalize to simple unseen scenes.
strength_and_weaknesses: Strengths:
- The paper provides a possible direction for learning visual dynamics by advocating explicitly utilizing the geometry information.
- The experiments are clear. The proposed method outperforms existing methods.

Weaknesses:
- The proposed two modules are simple combination of existing works. The Perception Module uses the existing PixelNeRF to learn underlying geometry from sparse views. The dynamics simulator applies standard graph-based backbone. There are no much new in the network design.

- The proposed methods use the color information to obtain the masks, which is a very strong assumption and not reasonable. Although the author claims it could also be solved with object segmentation, I think the problem of segmenting fluid, rigid bodies and granular materials is non-trivial (such as labeling, generalization etc) while the proposed method has a very strong reliance on it. This is even more severe when the author claims the methods can achieve "strong generalization". It would be better if the author can show the robustness of the proposed methods with different learned segmentation results. Otherwise I don't think the paper has enough contribution to the problem of learning visual dynamics. It's more like combining two existing methods with a strong and unreasonable assumption to solve a problem.


clarity,_quality,_novelty_and_reproducibility: The paper is clear and interesting. I think it could be reproduced. However, I question the novelty of the work (see Weaknesses)
summary_of_the_review: I think the assumption of known segmentation is too strong in order to combine the two existing modules. It's unknown how the methods would perform without such an unreasonable assumption. I think more experiments (learned segmentation) and more analysis (robustness to worse segmentation) should be added. I think the paper is below the bar of ICLR.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper is about dynamics prediction. Given multiple video streams taken from different view points of the same scene, for instance cubes floating in a liquid, the task is to predict future states of the scene. This is a very interesting problem. The proposed solution has a bit of an engineering approach but is reasonable. I generally enjoyed reading this paper and have only a few questions or suggestions how to improve the paper.
strength_and_weaknesses: This paper presents a consistent pipeline how to infer future states of physical systems given images only. The pipeline to rely on a NeRF model and uplift instance segmentations into 3D space is a neat idea.

How to infer dynamics from raw pixels is a problem not addressed by prior works.

I have 2 questions remaining: In Sec 3.2, how do you obtain estimates for the velocities since there does not exist explicit correspondences between 3D points across frames? Secondly, what are the point attributes a_i^v? Are these the class labels from the instance segmentation? 
clarity,_quality,_novelty_and_reproducibility: The paper is very well written and pleasant to read. It was the best paper on my stack - thank you.

I agree with the other reviewers that the authors should be upfront with the limitations of their work.
summary_of_the_review: The proposed pipeline is reasonable and the problem addressed by the authors is very important and interesting. I am not very familiar with the topic itself, though, so my confidence is not high.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper tackles the problem of learning a 3D dynamical physics model from multi-view observations of a scene. It follows a multi-stage approach, involving training a NeRF model, sample particles / 3D points from the NeRF representation, and running a particle-based GNN simulator using the extracted particles. It avoids requiring ground-truth correspondence between particles of different time steps by using a Chamfer loss together with a spacing loss. The method, 3D-IntPhys, is evaluated on simulated 3D environments involving manipulation of fluids, rigid bodies and granular materials.

All reviewers agree that this paper is tackling an important and very difficult problem (learning physics from videos) for which currently no general solutions exist – and that the paper is taking small steps toward that direction. The combination of NeRF-based particle representations and a GNN-based simulator is novel and interesting. The experimental results demonstrate that the proposed method addresses the considered tasks well. The writing is very clear and the paper is generally of high quality.

The main concerns highlighted in the original reviews and the discussion period are that the paper makes very strong and limiting assumptions about the considered environments; some of these assumptions are not clearly discussed upfront, or are completely missing from the main text of the paper. These assumptions include: known velocity of the fluid (experiments start from nearly static fluid), object segmentation (by color), known meshes/state for some aspects of the environment (container, robot, pusher), and knowledge of fluid density for the spacing loss. Overall, concerns were raised that assumptions make the method very impractical for use beyond the demonstrated synthetic settings.

I agree with the reviewers that the paper needs to be reworked to be much clearer about the addressed scope of the contributions, the limited practicality of the method, and the exact assumptions that are built into the method. It would require another round of reviews to assess whether these changes have been satisfactorily incorporated into the paper. With the current set of assumptions, the scope/applicability of the method is rather limited. If the authors could demonstrate some real-world results or results for an environment where the ground-truth state of part of the system is not known, while being upfront about any remaining assumptions, the paper would be a great fit for a conference like ICLR/ICML/NeurIPS. In its current form, my recommendation is to reject the paper.

justification_for_why_not_higher_score: As mentioned above, the paper needs to be reworked to be much clearer about the addressed scope of the contributions, the limited practicality of the method, and the exact assumptions that are built into the method. It would require another round of reviews to assess whether these changes have been satisfactorily incorporated into the paper.
Decision: Reject