Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an application of reinforcement learning for adaptive mesh refinement in large-scale finite element simulations of complex physical systems. The authors suggest to formulate the mesh refinement problem as a MDP and propose different policy architectures for scalable application of reinforcement learning. Experiment results demonstrate that the proposed RL approaches outperform existing baselines, and can generalize well to situations of different finement budgets and larger meshes.
main_review: Pros:
- A MDP formulation for mesh refinement
- Different policy architectures to capture mesh geometry
- Experiment on different settings: time-dependent, state and action size changes, and generalization to unseen mesh sizes and varied budget.

Cons: 

- Unclear MDP formulation
- Policy network architectures are not novel.

 
In overall, the paper pursues an interesting direction. Dynamical mesh refinement is an important feature in numerical simulation of PDEs. It makes sense to formulate the process of dynamical mesh refinement as a MDP where state and action are well defined. It's also a complex problem if the dimensionality of state and action is allowed to vary over time. 

1. At first, the description in section 3 can cover a bit more background of AMR so that the readers can follow and get ideas on how the formulation in 3.2 works. 

2. Regarding the formulation in 3.2, as the authors define a MDP for a new sequential decision making problem, this formulation can be more complete, if it includes detailed definition of transition. It's hard to understand how transition is computed and defines the dynamics based on only short description. The same issue for the reward defection is hard to understand how those values are computed. It be helpful if the authors consider providing more description of AMR background.

3. Three policy architectures are good. However I could not find where in the paper the policy is designed to capture time-dependency and the change of state and action sizes.

4. The experiments are interesting which can show the benefit of the proposed approach. I have only one point that is hard to follow. Why the TrueError is worse than other methods. What is the implication of this baseline? Does it mean the metrics used to define rewards is not optimal? If so, can the formulation be different, e.g. taking into the downstream task's reward instead of direct mesh refinement's error?

summary_of_the_review: See above
===========================
After the rebuttal: Thanks the authors for the detailed response. 
Though I found the proposed idea of using RL for AMR is interesting, the technical contribution is still limited. 
Therefore I like to keep my original rating.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: For various complicated problems governed by PDEs (e.g. solid/fluid interactions, aerodynamics, elasticity, backscattering, etc.) the computational cost can become prohibitive even for one inquiry, let alone parametric study. In the same time, mesh refinement is crucial to achieve acceptable accuracy. To mitigate such challenges, one solution is to use adaptive mesh refinement (AMR), for which the mesh refined only in the regions that numerically are sensitive to error propagation. For example for boundary layer models or shock-boundary layer interactions, one much capture the dynamics in high-gradients region of the solution, which are typically in vicinity of the walls or are part of the solution, while in very far regions of the domain, a coarse mesh is sufficient. 
Authors recognize that the process of AMR, refinement at each step, can be formulated a Markov decision process (MDP) and hence utilize reinforcement learning (RL) to train refinement policies directly from simulation. But this in turn poses a new challenge, at each step, the dimension of state (number of elements) and action space may (and should) alter. They propose suitable policy updates to overcome this challenge and come up with three different architectures for the implementation. Three test cases are used for experiments (static, advection and Burger) and authors compare all three architectures with each other as well as some traditional AMR methods to demonstrate the performance of the proposed method. They also carry out extra tests on the same set of PDEs to show generalization and out of distribution capabilities of the method for both static and transient PDEs. The paper is well-written and two tricks for RL are impressive (using Nmax to take care of varying dimension of state/action space and use of surrogate reward for training). However, I have some major reservations that I'll explain below in the Main Review. 

main_review: Article review and position the paper in the large context of AMR:
(major) Authors write in the abstract: "Existing scalable AMR methods make heuristic refinement decisions based on instantaneous error estimation and thus do not aim for long-term optimality over an entire simulation." This is a bit strong statement. For example, people in computational science for all finite-element, finite-volume, and spectral finite element (SEM) use mesh sensitivity method for AMR. In such cases, adjoint solver is used to measure the sensitivity of the solution. Although this approach employs some coefficients that are hard to quantify a-priori, however, there is a very solid and concrete argument on how to choose them, as well as their interpretability. Since, tuning the parameters is also part of RL and all three architectures used, I think this part should be removed, which brings me to another discussion. What is the fundamental reason of using RL for AMR compared to established AMR methods in finite-element or SEM community? There exists several differentiable FE, FV, or SEM solvers that would readily give the user the adjoint variables using AD, which in turn, can be used for AMR. The algorithm is automated, and based on calculus of variation so mathematically sound. Also the results are either as good as other AMR methods or even exceeding them (please see Adjoint error estimators and adaptive mesh refinement in Nek5000, among many others). When using adjoint-based AMR, it is also possible to define the cost function for mesh refinement based on the terminal solution instead of instantaneous refinement. So that aspect of the paper is also not necessarily novel in a sense that other class methods can also accomplish that. Furthermore, when it comes to optimal control, the mesh refinement can be done with a specific task of achieving the cost function related to the performance (see Rannacher: model reduction by adaptive discretization in optimal control or Apel et al Graded meshes in optimal control for elliptic partial differential equations: an overview, among others). Such approaches are also very scalable, which based solely on the results of this paper, it's hard to claim. Overall, I think i) some claims are not true, ii) some traditional AMR methods are absent in the analysis and introduction of this paper. 

Discussion on test cases and practical aspect of the algorithm:
(major) The test cases considered in this manuscript are toy models that can be served as "proof of concept". However, I'm not convinced that such models can be really considered a proof-of concept. The geometry is simple 2D periodic domain and in almost all practical PDE analysis, this is not the case. One beautiful aspect of the work is Out-of-distribution testing, which for the Burger PDE is done based on IC. This is an important test but what about the cases that BCs are changed? The Burger equation is inviscid, what about viscous Burger, for which the Reynolds number (or viscosity) can be changed. A parametric study of PDEs is always challenging and if authors can show the algorithm is robust to such changes, that would a strong argument for the RL-based AMR. Also, the geometry is so simple and for mesh generation this is really a challenge. One of the main concerns of numerical scientists is the complex domain/field analysis. For example, when there is a boundary layer in vicinity of airfoils. The method should demonstrate itself when it comes to more complex geometries. Otherwise, it hard to justify why use RL instead of traditional methods. Finally, MDP transition P consists of one step in PDE solution. Considering RL may require a large number of trajectories, for complicated PDEs, I imagine the training for all relevant parameters (IC, BCs, parameters such as viscosity, etc.) may be intractable. How do authors respond to that?
(minor) One nice trick in the paper is the use of so-called surrogate reward for training. However, it's not clear to me why such reward is superior to exact reward in results of Figure 4. Can authors clarify that? Why not such comparison for other cases?
(minor) Seems like the GreedyOptimal approach is superior in Fig 3 results. Why not comparing with this AMR technique for the rest of figures?

Discussion on the solution space and mathematical definitions: 
(major) The finite element requires a bit of discussion on the mathematical formalism. For example, authors discuss "test function", which has a established meaning in finite element community, but seems like they don't refer to the same concept. The finite element method is a discretization of the weak formulation, for which one use the inner product with test function. The next step is the transformation of such weak formulation from the functional space with infinite basis into the functional space with finite basis Such test functions, all of which belong to appropriate Hilbert spaces that should be mentioned (for example, it could be H1 space). I understand that authors are relating to the ML community, for whom the "test functions" may have a separate meaning but this causes confusion. 
(minor) Furthermore, authors use "finite element space", which again is not formally defined.
(major) On a similar note, how do authors enforce continuity within elements, if they belong to H1 space? If this is not necessary, then authors should clarify. 

Discussion on the algorithm:
(minor) From Fig 5-d, and the manuscript, I'd say authors use non-conformal elements, which is fine; however, how to enforce conforming mesh? Is it possible to enforce not having hanging nodes? Some finite element solvers can't handle the non-conforming mesh and should be modified. 
(major) How about mesh quality check? This is specially true if triangular elements are going to be used, which is likely for complex geometries. How to avoid de-generative elements, or irregular mesh skewness? Seems like a constrained RL framework should be used but that would require changing the three architectures proposed. 
(major) Authors mentioned de-refinement (or coarsening) is a future direction of this research. However, almost all traditional AMR algorithms use both refinement (in regions with high sensitivity) and de-refinement (in regions with low sensitivity) to achieve not only an accurate solution but also low computational cost. Dropping the de-refinement in AMR algorithm makes it much less competitive compared to state-of-the-art.



summary_of_the_review: The paper is well-written and two tricks for RL are impressive (using Nmax to take care of varying dimension of state/action space and use of surrogate reward for training). However, I have some major reservations: 1) Article review and position the paper in the large context of AMR (the adjoint-based AMR and goal-oriented mesh refinement in the PDE-optimal control community can achieve all goals proposed in the paper), 2) Discussion on test cases and practical aspect of the algorithm (other BCs, parameter should be explored, geometries are too simple for mesh generation whose main backlog is dealing with complex problems, scalability for complicated PDEs like Navier Stokes over complex geometries, etc.), 3) Discussion on the solution space and mathematical definitions (this is rather lighter comment but I think authors should clarify some concepts better in the manuscript), 4) Discussion on the algorithm (what about other constraints of AMR that traditional algorithms can handle, de-refinement, etc.). 
Overall, when it comes to RL for AMR I think it is novel and authors use some nice tricks to handle the challenges. But the overall use of RL for AMR over traditional methods is not demonstrated and there remain major concerns regarding the pragmatism of the method.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper presented an RL-based method to perform adaptive mesh refinement to solve PDEs more accurately. 
main_review: The problem of mesh refinement is critical in engineering, and current practices, as the authors stated, are often designed from some heuristics. The application of RL to this mesh refinement, as far as I know, is novel, promising, and may provide new ideas to solving this open problem. Nevertheless, the authors tested their framework on some toy examples with low resolution and barely sufficient validations. 

The PDEs to be solved in practice, as the authors indicated, usually have millions or billions of degrees of freedom. The authors claimed to "show that an RL refinement policy can generalize to higher refinement budgets and larger meshes." However, the "larger meshes" only contain 64x64 (4096) nodes, which is only a tiny fraction of the PDEs to be solved in practice. Hence, it is questionable whether the method may apply to PDEs on a practical scale, showing that the policy is "efficient". In other words, the paper would be more convincing if the authors demonstrate a case where their method is applied to a mesh containing millions of degrees of freedom and still outperform the baselines.

Another concern is about validation. The authors compared their policy with some trivial policy but did not compare with the state-of-the-art in adaptive mesh refinement. Hence it is not convincing that the RL-based method may outperform the traditional "heuristic" methods on this problem. The paper would be more grounded if the authors compare the RL-based method with the latest, non-RL-based method on the same problem set.
summary_of_the_review: The paper proposed a promising direction for future research, but it would be more convincing if the authors tested their method on practical scales with more solid validations.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes using reinforcement learning algorithms to train a few variants of neural network-based models to perform adaptive mesh refinement. These models predict for each node in a mesh where a refinement operation should be performed.
main_review: This paper is mostly clearly written, and has a conceptually simple yet clear idea behind it. At its core, it proposes to use reinforcement learning to train neural networks to propose adaptive mesh refinement operations in order to reduce simulation error (in contrast to traditional methods, which tend to use heuristics to perform the adaptive refinement). Therefore, this work's evaluation rests mainly in the empirical demonstration of the benefits of the proposed method.

However, the practical implementation of this simple becomes more convoluted, and is one of the weak points of the paper. Three variants for the neural network architecture are proposed, a simple "per-node" architecture, a hypernetwork based one and a graph-net based one. It is not necessarily an issue to propose diverse variants of a single method. However, in this case there is not much justification as to why different methods are tried, or how one would choose one over the other when applying the proposed approach in practice. 
In the end, the results are presented in such a way that the best performing of the 3 proposed variants is presented as better than baselines, though no single one of the 3 is able to consistently perform best. Given that there is no analysis or proposed method to pick a priori which one of the variants is more appropriate for a given task, the results become muddied.

Nevertheless, the so called "Independent Policy Network" (IPN) seems to be the generally best performing variant, usually performing better or on par with the baselines, though not always. It is important to take into account that training the neural network model incurs some considerable cost, and as such performing on par with a traditional method might not be enough.

To this end it would also be important to have comparisons of run-times between the methods and baselines, not only error reductions. This is required in order to evaluate if the proposed mesh refinements are actually reducing error by being "smart" about the refinement, instead of simply proposing "more" refinement, or more intense computation.

Some additional questions and comments:

- Even though the authors state it should be simple to implement, the policy networks in this work do not have a "coarsen" or "de-refine" action. Does this mean the meshes simply get finer and finer throughout the simulation? How does this impact long simulations, and ones where for example a shock wave is moving (and thus the region that requires refinement is moving)? Would this lead to an overall very fine mesh? This is important in light of the comment above, regarding comparisons of run time of the proposed method and baselines

- For the IPN, the description is a bit confusing. At one point it is stated that it does not depend on other elements, yet at a later point it is said that information of surrounding elements is included. It was also unclear to me how the convolutions are applied here. It would be good to improve this description.

----------

After reading the authors response and discussing the issues above with the authors (see discussion in the thread below), I am still not fully convinced of the robustness and strength of the experimental evaluation. As such, I will maintain my previous evaluation.
summary_of_the_review: Overall, given the concerns above, I do not feel like this work is ready for acceptance at this point. This is a work that rests mostly on its empirical results, and yet these are not completely robust at this point, given the issues described previously. As such, I am classifying as marginally below acceptance for now. Nevertheless, I am interested in reading the authors responses to the points raised above.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The current study formulates adaptive mesh refinement (AMR) as a Markov decision process and applies deep reinforcement learning (RL) to train refinement policies directly from simulation. The proposed method achieved high accuracy while reducing the computational time. 
main_review: Q1. "Nonetheless, to our knowledge, such variation in state-action spaces does not occur in any existing RL application." This comment is not valid. There are many RL formulations and applications that can generalize over a varying dimension of states and actions. Refers RL with neural graph networks and includes these references. For example, many RL approaches seek to solve the solution of various optimization problems formulated as graphs. These approaches train the policy using RL with the small-sized problem and then apply trained policy to solve large-scale unseen problems. 

Q2. Is action defined as selecting a computational grid among all the grid points under the current mesh strategy? The proposed action definition sounds inefficient since only one grid point can be selected at each iteration. Is there any reason or justification for using this action definition? 

Q3. Why is the simple policy gradient algorithm used? Is there any room for improving the performance by employing more advanced or problem-specific learning algorithms?

Q4. Why independent policy networks (section 4.1) and Hypernetwork policy (section 4.2) are discussed in the main text? Obliviously these policy structures should be inferior to Graph neural network-based policy, and the primary proposed model is GNN based model. 

Q5. Which modeling component contributes the most to the generalization capability? For example, state representation and the representation learning, specific reward form, or specific learning strategy? 

Q6. How are the computational times compared both in training and test phases?

summary_of_the_review: Formulating AMR as an MDP sounds reasonable, and structuring meshing policy using the graph neural network is also straightforward. Formulating a specific component (step) of an iterative algorithm using a learning-based module (i.e., RL) and GNN to learn a policy that can generalize over varying sizes and actions is a pervasive idea. Thus, the technical novelties of the proposed method are limited. The only novelty would be employing these ideas to finite element analysis.
review_title: null

Meta-review: This work formulates the Adaptive Mesh Refinement (AMR) problem used in solving Finite Element Method (FEM) as an MDP, and suggests an RL-based solution for it. Most reviewers agree that this is a novel problem and the solution is promising. There are, however, several issues raised by our reviewers, who have expertise ranging from ML to computational methods to solve PDEs. Some of the concerns are:

- As this is not a theoretical work, the burden of proof is on the empirical evaluations. Some reviewers found the experiments very small and not convincing enough.
- The paper does not compare with the state of the art AMR methods.
- The detail of how the problem is formulated as an MDP can be improved.

Given that four out of five reviewers are on the negative side, unfortunately I cannot recommend acceptance of this paper in its current form. Nevertheless, I believe this is a promising application of RL. I'd like to encourage the authors to consider the reviews in order to improve their work, and resubmit it to another venue.
Decision: Reject