Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper proposes to 'invert' graph neural networks (GNNs) in the search for optimal solutions in the context of engineering design. The paper uses stochastic gradient descent for optimization and observes that the gradients can be reliably propagated in a coarse trajectory. The paper evaluates the method in different CFD tasks and shows significant advantages compared to sampling-based techniques used in model-based control. An important advantage of the method is its generalization to unseen inverse-design scenarios. 
strengths_and_weaknesses: STRENGTHS: 
- The important observation of suitability of using GNNs for inverse design. 
- The solid execution of the idea and an extensive set of applications in different scenarios. 
- The writing. 


WEAKNESSES: 
- The lack of evaluation against a fully data-driven approach to inverse design. These models, which are learned from pairs of data in a supervised fashion (for example, MO-PaDGAN: Reparameterizing Engineering Designs for Augmented Multi-Objective Optimization), are arguably as important as the sampling methods used for comparison in the paper.  
- The method is incremental in a straightforward way in the sense it observes the general behavior of GNNs and proposes gradient-based optimization in the context of inverse design. The strengths of the method (long trajectories, generalization) are native to the forward surrogate models (GNNs) introduced in earlier works and not because of special treatment of the inverse problem per se. This makes the paper more an application of the GNN simulations. However, given the broad implication, this should not prevent the paper from a presentation to a broad audience. 
- The tile is unnecessarily broad. "Differentiable learned simulator" could pertain to a lot of different ML models (MLP, CNN, GP, etc.). Also, the applications seem well suited to computational fluid dynamics (CFD) tasks as shown in all 3 examples. In a more focused approach, this paper could have been called something along the lines: Inverse design in CFD using graph neural networks. 
questions: 
- Line 108: The abbreviation GNS is not defined. 
- Generalization in the 2D Fluid Tools is well explained (Lines 265-269). I could not find the same explanation for the airfoil setup. It is unlikely, but it appears that (Lines 198-205) the GNN is learned in a typical supervised way. I checked the supplementary materials and was left a bit confused. For example, speaking of a rollout of length 1 (supplementary materials Line 766) means the whole learning has been carried out without transitions specific to GNNs? 
limitations: - I think the generality argument could have been better supported using a more systematic approach. 
- The suitability of the forward model toward CFD tasks could be highlighted.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper approaches the problem of modeling fluids using GNNs. The paper then shows how using GNNs as differentiable simulators enables the design of components that help solve fluid manipulation tasks. 
strengths_and_weaknesses: ## Strengths:

* The paper is well-written and follows a clear structure.
* The experiments section is particularly well-written. The Figures are clear and really show the strength of the approach and also highlight a good choice of experiments. The additional supplementary materials on the webpage were helpful.
* The GNN does seem to do a good job of modeling the dynamics of the fluids for the three different domains.

## Weaknesses:

* The strength of this paper is as an applications paper. As a result the novelty is limited to the applications and not to the approach. It is not clear how this particular surrogate model is novel when compared to GNS and MeshGraphNets. It is not necessarily the case that just submitting an applications paper is a problem, it is just that the novelty of the approach must be taken into consideration.
* As an applications paper with valuable examples, it would be important to the community to make these examples available. The code does not seem to be provided with the paper and this would seem like a serious issue when considering whether the community would benefit from this paper.


## Minor Weaknesses:

* The title seems a little misleading as it is especially general. The abstract is also very general “a task-agnostic approach to inverse design…”. There is no mention of the fact that all experiments are within the domain of computational fluid dynamics. The reason the approach is so successful seems to be because the GNNs nicely match the simulators and the relationships between the fluid particles. It would probably make sense to make this clearer with a more specific title/abstract.
* Line 280-283: This seems to be an artifact of the simulator and not necessarily a strength/robustness of the approach. This part of the paper reads as if the fact that the ML model did some “smoothing” means that the ML model manages to avoid predicting the edge cases where the simulator fails and therefore is a more robust model choice. However, the reality is the ML model is actually failing when it does not predict these simulator edge cases as the ML model has only seen data from the “flawed” simulator and has only learnt information about the real-world through the “flawed” simulator. Perhaps this point should be removed.


questions: * Just to confirm, a different pretrained model is used per domain, but the same one is used across tasks?  E.g. all tasks for the 3D Watercourse use the same pretrained model but not the same one as for the airfoil.
* While the GNN is learnt independent of the reward function, it does seem like knowledge of the final application is included in the trained data set. E.g. for the waterworks model, the data includes a randomized obstacle plane that was sampled from random rotations and sine wave deformations. This seems like the deformations were purposely made to be smooth. What would happen if you added shot noise to the training data? Also how do you know when your model is sufficiently general? This seems vital to the motivation of the paper.

limitations: A limitation is that the paper focuses on fluid dynamics but references the area domain as a more general physical design problem. This should be highlighted.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: In this work, the authors applied a learned physical simulator MESHGRAPHNETS [55] to physical design. 

MESHGRAPHNETS is a graph network that can learn and predict the interaction among local particles (in fluids) or nodes (in meshes). Since the dynamics are described by a neural network, the simulation sequence is naturally differentiable when the time series is rolled out. Therefore, people can define some task-related objective functions on the trajectory and optimize the boundary condition using gradient-based methods.

Experiments demonstrate that this strategy works work even with the learned physical simulator. Although neural networks are prone to overfitting, and the long timestep could easily result in error accumulation, the experiments show that designs optimized on the MESHGRAPHNETS simulator work pretty well on ground-truth simulators.


strengths_and_weaknesses: I honestly like the message and ideology in this paper very much. However, the biggest concern for me is that the novelty of this paper itself is not that much.

Strengths

First, the paper is well-written and easy to follow. The motivation for conducting gradient-based design optimization using the learned physics simulator makes total sense. There have been a lot of previous methods that did this as well.

Second, although the experiment settings are simple, the results of the optimization results are impressive. It is known that prior learned surrogate physics models (I have also played with some of them) usually generalize poorly to out-of-domain data, let alone in a long simulation sequence where errors accumulation can be significant. But this paper demonstrates that MESHGRAPHNETS [55] performs pretty well and can optimize a design that is reasonably well in the ground-truth simulator.

Weaknesses

Nevertheless, in my view, the good results demonstrated in this paper should be credited to MESHGRAPHNETS. This paper simply wraps MESHGRAPHNETS into an optimizer package and then gets good results. As mentioned in L72, utilizing a differentiable learned module to do design optimization is very common. For the tasks in this submission, it is the design of MESHGRAPHNETS that makes this end-to-end optimization stable, generalizable, scalable, and efficient. The authors did a good verification for MESHGRAPHNETS but I hope to see more original and novel techniques.

For example, it would be good if the authors can show more challenging tasks, for example, training a control policy in the learned simulation which can instead be deployed in the ground-truth simulator or even the real world.


questions: Generally, this paper writes well and I only have several minor questions.

1. In L279, 

"However, since the learned simulator was trained on simpler data where these effects are unobserved, it picks up only on the appropriate collision performance and not the unrealistic edge cases. Thus, the learned simulator produces more plausible rollouts than the classical simulator in these cases, and might therefore be a better candidate for producing designs that would transfer to the real world."

It is common that learned physics can have low fidelity in uncovered corner cases. The corner cases are a major issue for other domains like autonomous cars. I wonder in the experiments whether the inaccuracy helps more or causes more problems for the designing process. 

2. What is the definition of theta_R in L142? Is it defined somewhere before?
limitations: As I said in the Strengths And Weaknesses part, I think the technical contribution of this paper is limited. I suggest, if possible, doing some more challenging experiments or proposing some new algorithms.

1. Is it possible to involve some control policy networks within the open loop of design? Some new RL algorithms could be introduced in such a pipeline.

2. Can this method deal with larger-scale or more realistic scenarios? Like 3D Airfoil or more realistic fluids experiments (the particles in the fluids videos looks very toy demo).

3. Can this method be applied to the real world? A 3D printed watercourse or a 2D 'fluid' tool on a table with marbles can be very cool and much more convincing.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors investigate the performance of learned GNN simulators for inverse design.
strengths_and_weaknesses: **Strengths:**

* The approach is appealing in its simplicity: using a fixed, trained surrogate solvers for inverse problems, rather than relying on devising "in-the-loop" strategies to fine-tune the solver during solution of downstream tasks.
* The experiments are varied and interesting. The authors evaluate the model across particle-based fluid simulations as well as field-based steady-state flow around airfoils.

**Weaknesses:**

* From the perspective of advancing state-of-the-art (empirical or theoretical), the paper is not particularly satisfying due to a lack of contextualization of the proposed results. Part of it is certainly due to the fact that the tasks chosen involve a large amount of moving pieces. The authors provide ablations and helpful comments; however, the experimental section of the paper reads more like a post-hoc description of "we did this specific thing and it worked". 
* I agree that showcasing GNN surrogate solvers on inverse problems is interesting. I also agree that this is a high-impact and open area of research. I do not think the experiments provided here do a good enough job at motivating the choice of models and baselines. The paper could benefit from adding targeted, designed "small scale" experiments aimed at clarifying why a specific choice made in the approach is appropriate. 
* Only a single baseline (CEM) is considered. 
* It would be of great value to the community if the code was open-sourced, particularly for an empirical work such as this one. Even with a detailed appendix, I do not believe the results are completely reproducible as is, especially given the amount hyperparameters involved (in both models and ground-truth simulators such as OpenFOAM). The authors attempt, wherever possible, to provide rules of thumb and empirical takeaways, which is appreciated.
questions: * GNNs simulators are chosen here without much explanation other than "they work".  Do you think the good performance of GNNs here might be due to tasks and dynamics involved, rather than some intrinsic (unexplained) superiority of the method? As the core focus of the  paper is performance in downstream design with learned simulators, an analysis of how other architectures used as surrogate solvers (for example, neural operators (1, 2, 3, 4, 5) ) compare would have been appreciated, particularly for the airfoil example.
* Why has CEM been chosen as the only baseline? The empirical performance of GNN simulators is the entire focus of this work, and any claim made would be stronger with at least another gradient-free optimization technique. I would appreciate comparisons with at least a single simple baseline from reinforcement learning, since it is mentioned in the introduction.

(1) Li et al., Fourier Neural Operator for Parametric Partial Differential Equations

(2) Guibas et al., Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers

(3) Gupta et al., Multiwavelet-based Operator Learning for Differential Equations

(4) Poli et al., Transform Once: Efficient Operator Learning in Frequency Domain

(5) Tran et al., Factorized Fourier Neural Operators
limitations: The authors discuss some limitations of the method. Some suggestions are provided in **questions** above (lack of baseline comparisons, no comparison with other types of learned simulators, and thus limited analysis on which tasks are appropriate for the approach and which require different models).
review_title: null

Meta-review: metareview: This work proposes to use GNNs in acting as good simulators of fluid dynamics. It is a solid example of using neural networks as differentiable simulators that can then be used for solving for design of components for fluid manipulation tasks. Overall, this work has been well received and should inspire related work in inverse design. The authors are encouraged to take the detailed reviews into account for camera-ready and especially make sure to make high quality reproducible code available to the community. There has also been discussion about the naming of the paper to reflect proper scope and that should be considered as well. 
confidence: Certain
Decision: Accept