Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a way of combining NERF-based volumetric renderer and eulerian fluid simulation to reconstruct fluid dynamics from videos. Given synthesized fluid videos, it can use NERF to estimate the volume of fluid from the images and then simulate it with a differentiable Euler simulator with a ConvNet as the projection solver. The volume in the simulator is fed as an additional input to the volumetric renderer, enabling supervising dynamics with videos directly. Authors conduct experiments on synthetic data and demonstrate effectiveness against NERF-based approaches without physical priors.
strength_and_weaknesses: Strengths

- I appreciate the differentiable physics and the differentiable renderer, where the simulator is further augmented with a neural network. It provides the framework with the ability to generalize across different scenes. This approach is effective and will inspire future work.
- Learning fluid dynamics will be of practical relevance.
- I like the explanation of the Lagrangian v.s. Eulerian representation. The baseline comparison is convincing. Eulerian representation is easier for implementing a fluid simulator and is very natural to combine with NERF.

Weakness

- It seems that the method requires an initial volume field for pretraining the NERF so that it can start the simulator. Will this limit the application in the real world? After all, obtaining ground truth volume in the real world is very hard. Can the volume be estimated through NERF directly?
- Though Eulerian representation simplifies the fluid simulator, it's a costly representation compared with particles (for training), and one has to predefine the volume's resolution and size. What if we combine the Langrangian and Eulerian views as in MPM?
- The lack of real-world experiments kind of weakens the experiment. The authors have explained this limitation in the appendix, which will largely reduce the paper's contribution.
clarity,_quality,_novelty_and_reproducibility: As far as I see, the paper provides sufficient details to reproduce the experiments. The paper’s novelty is a little bit limited because of previous works that combined differentiable physics and differentiable rendering.
summary_of_the_review: This paper provides a simple yet effective approach to combine NERF-based differentiable rendering and an Eulerian fluid simulation. However, the methods are not very applicable in the real world, and the novelty is limited. From my perspective, the paper is marginally below the acceptance threshold.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presents an approach to inferring 3D fluid dynamics from 2D videos. The main idea is to embed a fluid simulator and explicitly represent fluid as a volume, augmented by a NeRF-like volume rendering framework to bridge the 3D representation to 2D videos. Experiments show good results on synthetic liquid datasets.
strength_and_weaknesses: + It is in general an interesting idea to incorporate explicit simulator with differentiable rendering to allow inference from videos.
+ Approximating pressure solve with a CNN looks promising.
+ It is interesting to see plausible future prediction results from novel views.

- My main concern is on technical clarity. It is unclear how the generalization and test-time inference is done. In evaluation, it is emphasized that the proposed method is trained on training set only and directly used on test set, while other methods need test-time optimization. But I don't see an inference module in the architecture. It seems like the approach needs a conditional NeRF and two volumes for velocity/occupancy. How are these components inferred given a test-time video without any optimization?
- How is the joint training done? Does it train on separate frames, or train on the last frame only?
- How is the pre-training for NeRF done? It seems that the NeRF needs volume as input, and if it is trained on first several frames, then how can it be trained "without any groundtruth fluid dynamics"?
- How to generate the velocity field for training and testing? Since it is claimed that the model is trained without dynamics, I assume it is randomly initialized at the very beginning of training/testing each scene instance, and it gets evolved across time?
- Since the proposed model already has an explicit representation of the fluid, why does it need a NeRF for rendering instead of directly rendering the volume? Is resolution the main concern?
- It is slightly confusing that in Figure 3 the GT frames have a different color for BG/FG compared to the results (also this happens for later figures), while the PSNR in Table 2 looks high. Is it an issue with visualization in figures?
- Please use \citep{} for references.
clarity,_quality,_novelty_and_reproducibility: Clarity: Fair. My main concern is technical clarity. See weakness above.

Quality: Overall the quality is good.

Novelty: I believe incorporating simulator with differentiable rendering is novel.

Reproducibility: Fair.
summary_of_the_review: I think the paper presents an interesting attempt to incorporate differentiable simulator with differentiable rendering for inferring fluid dynamics. However, my main concern is the technical unclarity that makes it difficult to evaluate the overall quality. I'm willing to raise my score if the technical clarity can be substantially improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper approaches the problem of learning fluid dynamics with a neural network. To do this, the paper proposes combining a NeRF with the dynamics network to do joint training. The fluid is represented on a grid (the Eulerian way) as volume and velocity. First, the fluid is advected and gravity applied via the standard fluid equations. Next, the fluid is fed into a 3D ConvNet to compute the pressure projection. Finally, the scene is rendered with a NeRF, which takes the volume of fluid at a given location as an additional input. The results show that the joint system is able to accurately model fluid dynamics and render realistic images of the fluid.
strength_and_weaknesses: The biggest strength of this paper is that the proposed methodology is able to learn fluid dynamics from videos alone! Recent work has shown how this can be done for static scenes and even small amounts of movement, but fluids are high dimensional and have quite complicated dynamics. It is remarkable that this is able to be done, and done well according to the results, from only video data. What this paper is missing is a video attachment, which I suspect would get the results across much better than the static images. An additional strength of this paper is both the analysis of the pressure ConvNet in isolation of the NeRF (by using the GT fluid state, which is available because this is all in sim), and the comparison to T-NeRF and D-NeRF.

The paper does have a few areas for improvement, however. First of all, the separation of responsibilities between the advection method used and the learned pressure projection is unclear. This makes it hard to assess what the network is learning. Is it a hard task? Or is the advection algorithm doing most of the heavy lifting? This can be fixed by 1) putting a summary of the advection algorithm in the methodology section, and 2) putting a detailed mathematical description of the algorithm in the appendix. Second, the apparent train/test data is fairly simplistic. Most of it appears to be fluid in a cube enclosure, with the main variation being the starting configuration. Note the “apparent.” The figures in the paper show only these settings. If there are more complex settings, it is not obvious. This can be fixed by either featuring already existing complex settings in figures, or by adding such settings to the datasets. Finally, the biggest baselines to the proposed methodology are buried in the appendix (A.4). These are the main alternatives to what the paper proposes, their comparisons should be featured prominently in the results section (if space is an issue, the details of the comparison can remain in the appendix).

Also, one additional comment: The data used in this paper is all sim data. Given that the methodology only requires video, why not try it on a real world dataset? I think those results would be very interesting.

clarity,_quality,_novelty_and_reproducibility: While there exist some other papers that propose solutions to similar problems, the solution proposed by this paper is quite novel. However, this paper has some major clarity issues.

First of all, there are significant grammatical errors in the paper. For example, section 4.1 sentence 5 reads “We generate two versions datasets…” Section 4.2 sentence 5 reads in part “...to unseen data … due to that our method does learn fluid dynamics…” A thorough proofreading pass will be necessary before publication.

Second, it would be difficult to reproduce this paper because the methodology isn’t explained in enough detail. For example, the advection process only cites another paper, but doesn’t say how this paper implements it. Furthermore, section 3.3 says that the volume field is advected based on the velocity field, but doesn’t describe how that is done. Given that section 3.1 paragraph 3 makes it sound as if the use of a volume field is unique to this paper, knowing how it is updated is critical to reproducibility. 

Finally, the figures in the paper are difficult to understand. For example, Figure 1 doesn’t make sense. There are 2 advection boxes on the volume field path and 1 on the velocity field path. Are they doing the same thing? Presumably not, but that’s not clear. Also, the velocity field is an input to the advection box between V_t and V_{t+1}, but not an input to the advection box between V_{t-1} and V_t? Also, the gradient flow seems to show the gradients going straight from the NeRF to the pressure ConvNet when they actually pass back through the advection step. Furthermore, Figures 2, 3 (left), and 9 have alternating rows GT and Ours. Presumably each pair of rows are just more examples, but the way they’re presented makes the reader think they are different methods being compared. It would make the figures more clear if the GT-Ours comparison rows could somehow be grouped, e.g., put a line between rows that aren’t the same scene.

summary_of_the_review: Despite my above concerns, this is actually a very good paper. It solves a difficult task from limited data, and does so well. The paper could use some work on clarity and details, but overall the proposed methodology is novel and the evaluation is thorough.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes to infer the dynamics of fluid by integrating differentiable simulation and rendering. Reviewers all agree that the problem is worth studying, but have concerns regarding the presentation and the limited experiments. After the rebuttal, there was extensive discussion about the paper among all reviewers and the AC. Two reviewers are negative, and one is positive. The main point of disagreement is whether the assumption of the initial state is realistic. The negative reviewers argued that if such a perception module exists, the value of learning the dynamics model becomes moot. At the same time, the positive reviewer suggests that the entire framework may be considered in a Bayesian filtering framework.

The AC had discussions with the reviewers and agreed on the concern. To be considered a Bayesian filtering framework, this submission should demonstrate how the learned dynamics model can be integrated with updates from the perception model, which is currently missing. If the main argument of the paper is to learn the dynamics model directly from video via neural rendering, then the assumption of the initial state greatly undermines the value of the paper, as the introduction of differentiable neural rendering is supposed to address the limitation. Therefore, the AC agrees with the negative reviewers that this is a significant concern, making the paper below the bar to be accepted.  The authors are encouraged to revise the paper based on the reviews for the next venue.
justification_for_why_not_higher_score: Two reviewers raised the
Decision: Reject