Title: Nonparametric Boundary Geometry in Physics Informed Deep Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: PINNs

physics informed neural networks

geometric deep learning

neural operator

PDEs
Abstract: Engineering design problems frequently require solving systems of
partial differential equations with boundary conditions specified on
object geometries in the form of a triangular mesh. These boundary
geometries are provided by a designer and are problem dependent.
The efficiency of the design process greatly benefits from fast turnaround
times when repeatedly solving PDEs on various geometries. However,
most current work that uses machine learning to speed up the solution
process relies heavily on a fixed parameterization of the geometry, which
cannot be changed after training. This severely limits the possibility of
reusing a trained model across a variety of design problems.
In this work, we propose a novel neural operator architecture which accepts
boundary geometry, in the form of triangular meshes, as input and produces an
approximate solution to a given PDE as output. Once trained, the model can be
used to rapidly estimate the PDE solution over a new geometry, without the need for
retraining or representation of the geometry to a pre-specified parameterization.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe Authors introduce a cross-attention (without self-attention)  based decoder architecture for neural operators for solving PDEs. The architecture contains an encoder component that has the boundary defined by a triangular mesh as an input. The mesh encoder uses graph convolutions on the edges with uniform 4-nearest neighbour architecture across  the triangular mesh to define the features of the boundaries, the decoder attention layers propagate the information to create the PDE solution in the interior domain. The decoder uses ReLU, the transformer SiLU non-linearities. The latter to make sure the automatic differentiation for the PDE solution is smooth enough everywhere.\n\nThe system is trained with 12000 example meshes (geometries), validated by FEM solutions. The lowest validation error parameters from each epoch is used.  The methods is demonstrated for a set of simple geometries and equations with mostly encouraging results.\n\nThe capability of solving PDEs with varying boundaries in an efficient was is valuable for many industrial applications, and is an impactful addition to the  physical system simulation toolbox.\n\nSTRENGTHS:\nThe introduction is clear and easy to follow, also  the manuscript brings clearly up the benefits of using PINNS in the industrial domain.\n\nThe neural network operator that can take a triangularisation define boundary as an input for the trained system, and hence allows for solution of gPDEs with geometries using only an inference of a neural network, with convolution and a layers of cross-attention has the potential to provide solution much faster than traditional  FEM solvers.\n\nWEAKNESSES:\nThe architecture of the solution, especially its hyper parameters are not described.  Figures showing the components of the architecture are missing, and verbal explanation with references to similar architectures MeshNet and decoder from the original attention is all you need makes it hard to read. As introducing the novel architecture is the main result of the paper, one should concentrate on describing it in simple visual manner that allows a wider audience to grasp the essential.\n\nAlso, regularisation of PDE solutions when geometries are given with triangularisation have singularities. The manuscript should address this.\n\nQUESTIONS:\nPoisson  equation solutions have corner singularities in certain geometries. Now, analytically the PDE solution of triangulation based geometric behave badly on all vertices of the boundary mesh. It is possible to solve the problem by smoothing the surface, but this would require a different parametrisation of the surface, for example splines - as often used in industry. How would the current neural operator architecture address this problem?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a neural operator methods that can take different boundary geometry, in the representation of a mesh, as input to solve different PDEs. To the best of my understanding, the proposed method can take geometry represented in different triangular meshes as input and predict PDE solutions (i.e. a function of the geometry). The proposed method used MeshCNN to extract geometry features, and use cross attention transformer as the decoder to produce the solution. The proposed model needs to be trained only once for a specific PDE, then it can be used to produce solution for different conditions.\n\nThis work is trying to address a very important problem, but due to the presentation issue and some concern with the results, I am leaning toward rejection. I think the paper can be greatly strengthen with an iteration with better presentation and more evaluation.\n\nSTRENGTHS:\n1. I think the problem this paper set-up to solve is of very important. When the engineers are using a model to speed up their simulation pipelines, they usually require solving the same problem with different geometries. If the trained model can only work on one specific geometry, then it might not be the best fit of such pipeline. With this said, if this paper is successful, it can enable neural operator to be used in more practical scenario.\n2. The proposed method (if I understand it correctly) can indeed end handle different geometry while maintaining respected invariances required for the problem set-up.\n\nWEAKNESSES:\n1. Presentation issues. I found the paper (especially the method section) difficult to understand as it lacks of rigorous definition of the problem set-up (e.g. input/output of the model; how do we represent it; what’s the network architecture;). As the paper is proposing a specific way to achieve this goal, I think including a more rigorous definition of the model can improve reproducibility and help understand why specific model design choice is necessary.\n2. The results does not necessarily support the claims. A important claim of the paper is that their proposed model can be trained over different geometries of the same PDE, thus provide advantage by amortizing the training compute to benefit multiple simulation rounds. To show this, I think it’s necessary to compare to the baseline where for each geometry, we train a neural operator (which we might not be able to do without data) or we solve for a PINN for each geometry (this is valid as PINNs can be solved for scratch). I expect this comparison turns out to be that the proposed method is comparable with the baseline, while being faster since PINNs requires optimization while the proposed method requires only forward pass. Right now the results only show comparison with the ground truth, which makes it difficult to see 1) whether the proposed method is accurate enough, and 2) whether the hypothesized advantage in amortizing the training across different geometry exists. \n3. Agnostic to discrete action. While the paper cite MeshCNN to argue that in practice the feature extraction pipeline is not adversely affected by the discretization quality, I think this argument is not sufficient as simulation using FEM usually requires a careful discretization and it’s simulation is different from the task MeshCNN were designed to do (classification and segmentation). The fact that the method section mentioned the manifold assumption in L88-100 strengthen my concern. While this can be better fit into the future-works scope, I would recommend the authors to stress-test how different discretization affects the performance, and probably discuss how this method can be applied to non-curated meshes such as mesh obtained from real-world scans.\n\nQUESTIONS:\nSee the weakness section.\n\nLIMITATIONS:\nYes, the author has a good discussion of the limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPhysics Informed Neural Nets (PINN) have gained considerable attention lately. However, they are significantly expensive as compared to FEM or other classical PDE solvers. Moreover, any trained PINN is specific to the object geometry it has been trained upon. This paper proposes a solution to reuse the trained model to various object geometries by learning a PDE solution without restricting to a certain object parametrisation. It takes into account the bounding conditions on the object in terms of meshes and deploys MeshCNN instead of regular CNN to encode local geometric properties under an attention mechanish [17] to learn the solutions to PDE.\n\nSTRENGTHS:\nThe proposed method addresses an important problem of making the physics-based learning more generic.\n\nThe paper discusses an important aspect of learning on meshes: contravariance/invariance of geometric properties under mesh formulations. It enforces the preservation of geometric quantities in order to solve the PDE.\n\nThe experiments show various well-known PDE systems are solved quite accurately using the proposed approach.\n\nWEAKNESSES:\nThe writing is a bit difficult to follow. The use of footnotes is weird and interrupts the reader's flow.\n\nSome design aspects of the paper are not clear. Lines 118-120 suggest that the self-attention block of the Transformer is removed.  The authors argue that the value of learned parameter at a point x should be independent of whether or not one is simultaneously calculating its value at another point x′. This argument needs to be explained further as geometrically close points manifest similar properties and learning them together (or simultaneously) can be beneficial to obtain the solution to the PDE. It is understandable that the geometrically distant points need not to considered simultaneously. Choosing to remove the self-attention layer seems to be a compromise and should be explained in more details.\n\nQUESTIONS:\nsee weakness section\n\nLIMITATIONS:\nThe limitations have been discussed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The article describes a method to obtain the solution of a PDE given just the boundary mesh as an input. A variety of edge features are first transformed using MeshCNN, which are then used with a Transformer decoder to obtain the solution. The method is demonstrated to work for a few different PDEs on relatively simple domains.

STRENGTHS:
The proposed framework is potentially powerful considering it is supposed to work off of the mesh manifold, which is typically the starting point of the computational analyses.

WEAKNESSES:
The method is demonstrated to work qualitatively, and only on toy problems.
The framework presentation should be improved to demonstrate the framework architecture and how one goes from input (boundary mesh) to output (solution at a physical point in the domain).. I have gone through the text several times and it is still not clear to me. While the authors refer to other papers, I think it is important to make the article self-sufficient to an extent for the reader.
Without sufficient evidence, it is hard to see how this framework would work for complex domains which require large number of elements to accurately solve the PDE of interest.

QUESTIONS:
It is not clear to me how one goes from the input (mesh manifold) to the solution at any physical point in the domain.

LIMITATIONS:
It is not clear whether the method can be effective for large scale problems (think 10k+ elements) where the benefits of a fast solution estimator would actually be helpful.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07