Reinforcement Learning for Adaptive Mesh Refinement
Authors: Jiachen Yang
Tarik Dzanic
Brenden K. Petersen
Jun Kudo
Ketan Mittal
Jean-Sylvain Camier
Vladimir Tomov
Tuo Zhao
Hongyuan Zha
Tzanio Kolev
Robert Anderson
Daniel faissol
Venue: ICLR 2022 Submitted (2022)
Keywords: reinforcement learning
adaptive mesh refinement
finite element method

Abstract: Large-scale finite element simulations of complex physical systems governed by partial differential equations (PDE) crucially depend on adaptive mesh refinement (AMR) to allocate computational budget to regions where higher resolution is required. Existing scalable AMR methods make heuristic refinement decisions based on instantaneous error estimation and thus do not aim for long-term optimality over an entire simulation. We propose a novel formulation of AMR as a Markov decision process and apply deep reinforcement learning (RL) to train refinement {\it policies} directly from simulation. AMR poses a new problem for RL as both the state dimension and available action set changes at every step, which we solve by proposing new policy architectures with differing generality and inductive bias. The model sizes of these policy architectures are independent of the mesh size and hence can be deployed on larger simulations than those used at train time. We demonstrate in comprehensive experiments on static function estimation and time-dependent equations that RL policies can be trained on problems without using ground truth solutions, are competitive with a widely-used error estimator, and generalize to larger, more complex, and unseen test problems.

Under review as a conference paper at ICLR 2022
REINFORCEMENT LEARNING FOR ADAPTIVE MESH REFINEMENT
Anonymous authors Paper under double-bling review
ABSTRACT
Large-scale finite element simulations of complex physical systems governed by partial differential equations (PDE) crucially depend on adaptive mesh refinement (AMR) to allocate computational budget to regions where higher resolution is required. Existing scalable AMR methods make heuristic refinement decisions based on instantaneous error estimation and thus do not aim for long-term op- timality over an entire simulation. We propose a novel formulation of AMR as Markov decision process and apply deep reinforcement learning (RL.) to train refinement policies directly from simulation. AMR poses new problem for RI. as both the state dimension and available action set changes at every step, which we solve by proposing new policy architectures with differing generality and in- duetive bias. The model sizes of these policy architectures are independent of the mesh size and hence can be deployed on larger simulations than those used at train time. We demonstrate in comprehensive experiments on static function estimation and time-dependent equations that RL policies can be trained on problems without using ground truth solutions, are competitive with a widely-used error estimator, and generalize to larger, more complex, and unseen test problems.
INTRODUCTION
Numerical simulation of PDEs via the finite element method (FEM) (Brenner & Scott, 2007) plays an integral role in computational science and engineering (Reddy & Gartling, 2010; Monk et al., 2003). Given fixed set of basis functions, the resolution of the finite element mesh determines the trade-off between solution accuracy and computational cost. For complex systems with large variations in local solution characteristies, uniform meshes can be computationally inefficient due to their suboptimal distribution of mesh density, reselviesolving regions with complex features such discontinuities large gradients and over-resolving regions with smoothly varying solutions. For systems with multi-scale properties in particular, attempting to resolve these features with uniform meshes can be challenging even on the largest supercomputers. To achieve more efficient numerical simulations, adaptive mesh refinement (AMR), a class of methods that dynamically adjust the mesh resolution during simulation maintain equidistribution of error, is used to significantly increase accuracy relative to computational cost.
Existing methods for AMR share the same iterative process of computing solution on the current mesh estimating refinement indicators, marking element(s) to refine, and generating new mesh by refining marked elements (Bangerth & Rannacher, 2013; Červený et al., 2019). The optimal algorithms for error estimation and marking in many problems, especially evolutionary PDEs, are not known (Bohn & Feischl, 2021), and deriving them is difficult for complex refinement schemes such as hpprefinement (Zienkiewicz et al., 1989). As such, the current of-te-of-the-ar is guided largely by heuristic principles that are derived by intuition and expert knowledge (Zienkiewicz. & Zhu, 1992), but choosing the best combination of heuristics is complex and not well understood.
We advance the novel notion that adaptive mesh refinement is fundamentally a sequential decision- making problem in which a sequence of greedy decisions based on instantancous error indicators does not constitute optimal sequence of decisions for the actual goal of achieving high cumulative or terminal accuracy. In dimendependent problems for example, an error estimator by itself cannot preemptively refine elements which would encounter complex features in the next time step. This means that the optimality of a refinement decision depends on the accuracy of the future solution and that selecting element which yields the largest reduction in error at the current time step may

---

Under review as a conference paper at ICLR 2022
((6)
Figure 1: AMR viewed as a Markov decision process.
not be the optimal decision over the entire simulation. Whether and how optimal AMR strategies can be found by directly optimizing a long-term performance objective are open questions.
Given this perspective, we formulate AMR as a Markov decision process (MDP) (Puterman, 2014) (Figure 1) and propose reinforcement learning (RL) (Sutton & Barto, 2018) approach that explic- itly trains mesh refinement policy to optimize a performance metric, such as final solution error In contrast most, if not all. benchmark problems and complex applications 2015; Brockman et al., 2016; Osband et al., 2019; Berner et al., 2019; Vinyals et al., 2019), AMR poses new challenge as the sizes of both the state and the set of available actions depend on the current number of mesh elements, which changes with each refinement action at every MDP time step. While one may define a fixed and bounded state and action space given a finite refinement bud- get, doing is very inefficient as the policy's input-output dimensions would have to accommodate the full exponentially large space but only subspaces (with increasing size) are encountered during simulation. In many practical applications, order of millions or billions of degrees of freedom. This motivates the design of efficient policy architectures that leverage the correspondence between the current mesh state and valid action set.
RL (Mnih al.,
would routinely encounter input dimensions the
In this paper, we make the following conceptual, methodological, and experimental contributions: 1) We formally define MDP with effective variable-size state and action spaces for AMR (Sec- tion 3.2); 2) We propose three policy urchitectureswwei -with differing generality, inductive bias, and capacity for modeling interaction-that operate such variable-size spaces (Section 4); 3) As path toward potentially solving large and complex problems on which RL cannot tractably be trained, we investigate the generalizability of policies trained on small representative features with known analytic solutions and the effectiveness of policies trained using a novel reward formulation that can be applied to problems without known analytic solutions (Section 5); 4) Our experiments demonstrate for the first time that RL can be competitive with, and sometimes outperform, greedy refinement strategy based on the widely-used Zienkiewicz-Zhu--ypa error estimator; moreover, we show that RL refinement policy can generalize to higher refinement budgets and larger meshes, transfer effectively from static to time- dependennn problems, and can be effectively trained on more complex problems without readily- avvilalle ground truth solutions. (Section 6).
2 RELATED WORK
The formulation of problems in numerical analysis as statistical learning problems can be traced at least as far in time to Poincaré (Poincaré, 1912; Diaconis, 1988). Contemporary works have employed neural networks as powerful function approximators existing numerical PDE and linear system solvers achieve faster convergence rates, generalize different boundary conditions larger problems, and approximate underresolved features in coarse-grained simulations (Hsieh et al., 2018; Luz al., 2020; Bar-Sinai et al., 2019). Our work focuses on optimizing finite element space rather than components of numerical solver.
To the best of knowledge, prior work has formulated adaptive mesh refinement as a sequential decision-making problem and proposed a reinforcement learning approach (Sutton & Barto, 2018).
Previous work at the intersection of neural networks and mesh-based simulation trained neural net- works to predict mesh densities, sizes, or eттor fields for use by downstream mesh generators (Dyck et al., 1992; Chedid Najjar, 1996; Zhang et al., 2020; Pfaff et al., 2020; Chen & Fidkowski, 2020). Brevis et al. (2020) apply supervised learning to find an optimal parameterized test space without modifying the degrees of freedom. Bohn & Feischl (2021) show theoretically that the estimation and marking steps of AMR for an elliptic PDE can be represented optimally by a recurrent neural network, but model optimization was left as an open question. Recent studies have leveraged the effectiveness of graph neural networks (GNN) (Sperduti & Starita, 1997; Gori al., 2005; Scarselli et al., 2008) at representing relational structure to predict PDE dynamics on general unstructured and non-uniform meshes (Alet et al., 2019; Belbute-Peres al., 2020; Pfaff et al., 2020). Previous
2

---

Under review as a conference paper at ICLR 2022
work graph generation and formation have employed GNNs as the policy model in an RL context with applications to biological and social network datasets (You et al., 2018; Trivedi et al., 2020).
Learning& a policy for unbounded variable-size state and action spaces is care-if not new- problem for RL, which has been typically applied to environments with fixed-size observation and small bounded action spaces in almost all benchmark problems (Mnih et al., 2015; Brockman al., 2016; Osband et al., 2019). While there are notable applications where the available action set varies with state (Berner et al., 2019; Vinyals et al., 2019), they do not face the challenge of potentially millions of possible actions that arises in large-scale AMR. The technique of growing action spaces (Farquhar et al., 2020) maintains a fixed action space size within each episode, whereas both state and action space sizes change at every time step within an episode in AMR.
BACKGROUND AND FORMULATION
3.1 FINITE ELEMENT METHOD
Our mesh adaptation strategy is implemented in a FEM-based framework (Brenner & Scott, 2007)- In FEM, the domain S RD is modeled with mesh that union of E nonoverlapping subsets (elements) such that f == U2t where k E N: : k < E. The solution on these elements is repre- sented using polynomials (basis functions) which are used to transform the governing equations into a system of algebraic equations via the weak formulation. AMR is a commonly used approach to im- prove the trade-off between the solution accuracy, which depends on the shape and sizes of elements, and the computational cost, which depends on the number of elements. The most ubiquitous method for AMR is h-reffinemnt,, whereby elements are split into smaller elements (refinement) or multiple elements coalesee to form single element (derefinement). In practical applications with unknown true solutions, the conventional AMR approach is to take greedy refinement decisions based on a posteriori error estimators, which rely on the numerical solution and its derived quantities the current mesh without regard to long-term optimality.
3.2 AMR AS A MARKOV DECISION PROCESS
We formulate AMR with spatial h-rrefinmmntt as Markov decision process M (O, Nmax: A, R, P.7) with each component defined as follows. Each episode consists of T RI. time steps: for time-dependent PDEs, T spans the entire simulation and there may be multiple underlying PDE evolution steps per RL step: for static problems, T is an arbitrary number of steps at which RL can act. Consider time step t when the current mesh has N₂ Nuux N elements. Each element i is associated with an observation of E O and the global state is 8, :- [o] lo!....,.f"1 of"] E ON:, We define O Ed such that each element's observation is ten- sor of shape d := 1 X w x that includes the values and refinement depths of a local window centered on itself. For brevity, let S, denote the current global state space O%, We denote an action Naxx ......,aa.,,, J. where 0 means "do-nothing" and * O means refine element Given the current state and action, the MDP transition P consists of:
by 0 E A, == {0, 1........ N. C A == {0, 1, 1,...,
1) refining the selected element into multiple finer elements (which increases N₂ if a refinement budget B not exceeded and the selected element not at the maximum refinement depth dmuxi 2) stepping the finite element simulation forward in time (for time-dependent PDEs only);
3) computing a solution on the new finite element space.
When true solution is available at training time, the reward at step t is defined as the change in error from the previous step, normalized by the initial error to reduce variation across function classes:
Tr == [|le:-ll2 - ||ez (l2)) //loll/llll
where error e is computed relative to the true solution. With abuse of notation, we shall use e to indicate the error norm. The ground truth is not needed to deploy a trained policy on test problems. When the true solution is not readily available, as is the case for most non-trivial PDEs, one may run reference simulation on a highly-resolved mesh compute equation 1, but this approach can be prohibitively expensive for training on large-scale simulations. Instead, we propose the use of a surrogate reward r. := ||et, Uu..o-reeime 2|,, the normed difference between the estimated solution with and without executing the chosen refinement action. This surrogate, which is an upper bound
Polynomial p-refinement can be formulated in similar way. r-refinement (Huang & Russell, 2010; Do- brev al., 2019) can he formulated as an RL problem but is not treated in this work.
3

---

Under review as a conference paper at ICLR 2022
on the true reward and effectively acts as estimate of the error reduction, is only used training time to minimize computational effort, whereas at test time, the effectiveness of trained policies is evaluated using the error computed with respect to a highly-resolved reference simulation.
Our objective to find a stochastic policy n: S, - ((A)) to maximize the objective
((a) == (0)
Σ
Aside from y E (0, 1), this objective is equivalent to maximizing total error reduction: eo Efinal
Although the size of the state vector and set of valid actions changes with each time step due to the varying N₁, this MDP is well-defined since one can define the global state space as the set of all possible ON N Nmax. and likewise for the action space. Hence, the policy is navigating through subspaces of increasing size during an episode. Moreover, the exact 1:1 correspondence between the number of observation components and the number of valid actions calls for designing dedicated policy architecture for AMR, which we present below in Section 4.
We work with the class of policy optimization methods as they naturally admit stochastic policies that could benefit AMR at test time: a stochastic refinement action could reveal the need for further refinement in a region that appears flat on coarse mesh. We build the policy gradient algorithm (Sutton et al., 2000; Schulman et al., 2017) to train policy (parameterized by f) using batches of trajectories {To = {(s,, at, F2) T+)x 1 k=1 generated by the current policy.
POLICY ARCHITECTURES FOR VARIABLE STATE- SPACES
We propose three policy architectures, each with different inductive biases, that address the chal- lenge of variable size state vector s E N xd and action set {0, ,,..,,.,,N NN), both whose sizes changes with number of elements N₂ within episode. These architectures any stochastic policy gradient algorithm. We focus on the special case of 1:1 correspondence be- tween the number of observations that compose each global state and the number of available actions at that state. Although not treated in this work, these policy architectures can be easily extended to the general case of 1:k correspondence e.g., i = 2 to include derefinement actions.
compatible with
4.1 INDEPENDENT POLICY NETWORK
The Independent Policy Network (IPN) handles the 1:1 correspondence by mapping each observa- tion probability for the corresponding action. Let fo: IRd R be a function parameterized by 0. Given a matrix of observations 8 := [o' [o JNo E R N.x xd we define the policy as
(-|s) softmax (folo'). ..... fo("))))""")
(3)
For example, using a neural network with hidden layer W E Rdxh with h nodes, output layer H E Rax1, and activation function a,, the discrete probability distribution over N actions conditioned on 8 is defined by softmax (o(sW)H)
IPN applies to meshes of any size since the set of trainable parameters is independent of N, but it has two main limitations. Firstly, it makes a strong assumption of locality as the action proba- bility an element does not depend on the observations at other elements. This assumption also appears in existing AMR methods that estimate error independently at each element; in fact, the output probabilities of IPN may be viewed as normalized error estimates. Secondly, the permutation equivariance of this architecture-i.e,, =aa((() (o0(1), ,...., (a())) <<alls) for any permutation operator μ: [N] + [N] means that one cannot use the ordering of inputs to represent spatial re- lations among elements, which would be necessary for refining an element based on neighboring conditions. We mitigate this problem by defining each element's observation as an image tensor that includes neighborhood information and using a convolutional network layer, but this may face difficulties on unstructured meshes with non conquudrilateral elements (Cervený et al., 2019).
4.2 HYPERNETWORK POLICY
The hypernetwork policy captures higher-order interaction among inputs via the function form
([+ss) so ftmax (Jos(e)(o"),... .... (a.)ee
(4)
4

---

Under review as a conference paper at ICLR 2022
The main policy network weights 9 are now the output of hypernetwork (Ha et al., 2017) 9:: IR-xxd - Rdim(f), parameterized by Φ, which produces mixing among the inputs 8 E RR Continuing with the example in IPN, where the policy network's 's first layer is W E RdxA, hy-
pernetwork with two layers can be instantiated as E (sU) V = W where U E TRdx and
V E A (dxh)) are the trainable parameters Φ, and M, denotes the i-th row of matrix M. The output W can then used as part of 9 in equation 3.
This increased generality comes with more difficulty in the choice of architecture (specific form of go), which affects the extent to which it captures interaction among inputs. It does not contain an inductive bias for the local nature of interactions seen in classical applications of AMR. In fact, the use of a summation from = I to N in the example above means that complete global information affects each local refinement decision, which is extremely strong inductive bias.
4.3 GRAPH NET NETWORK POLICY
We build on graph networks (Scarselli et al., 2008; Battaglia et al., 2018) to address both the issue of interaction terms and spatial relation among elements. Specifically, we construct a policy based on Interaction Networks (Battaglia et al., 2016), which is special case without global attributes At each step, the mesh is represented as a graph g = EV. Each vertex of in V = {o" h= corresponds to element and is initialized to be the observation o'. E {(c* 1hh s²) k==INe is a set of edges with attributes e* between sender vertex 8 and receiver vertex r* An edge exists between two vertices if and only if they are spatially adjacent. We define the initial edge attribute e* as one-hot vector indicator of the difference in refinement depth between r* and s².
Graph networks capture the relations between nodes and edges via the inductive bias of its internal update rules. A single forward pass through the graph policy involves one or many rounds of message passing (Algorithm 1 in the Appendix). Each round is defined by the following sequence of computations: 1) Each edge attribute et is updated by learned function & using local node information via e* < yo (e* wn". v 2) For each node i, we denote by E = {(et, rd )'t) the set of all edges with node as the receiver, and all updated edge attributes are aggregated into a single feature e² + pr-t"(E" by aggregation function pr-"" (e.g., element-wise sum); 3) Then, each node attribute is updated by 6² < (' ut) using learned function sp Each additional round increases the size of the local neighborhood that determines node attributes. Finally, we map each node attribute to a scalar using learned function vi, apply a global softmax over all nodes, and interpret the value at each node i as the probability of choosing element i for refinement.
These update rules allow the graphnet policy to address both limitations of the IPN and the hypemet- work policy. Cross terms arise in the forward pass due to mutual updates of edge and node attributes using local information. The order of cross terms increases with each messsaee-passngg round. Local spatial relations between mesh elements are included by construction in the initial edge attributes, so there is no need to include numerical spatial information in each element's 's observation vector.
5 EXPERIMENTAL SETUP
Our experiments assess the ability of RL, using the proposed policy architectures, to find AMR strategies that generalize to test function classes that differ from the training class, generalize to variable mesh sizes and refinement budgets, and extend to more complex problems without readily- available ground truth solutions. We define the FEM environment in Section 5.1, the train-test procedure in Section 5.2, and the implementation of our method and baselines in Section 5.3.
5.1 AMR ENVIRONMENT
MFEM. We use MFEM (Anderson et al., 2021; MFEM), a modular open-source C++ library for FEM, to implement the MDP for AMR. We ran experiments on two classes of AMR problems: static and -meedeepedeent. In the static case, the objective of mesh refinement is to minimize the I error norm of projecting a variety of test functions onto a two-dimensional finite element space. In the time-dependent case, the functions are projected onto the finite element space, and PDE of the form du
σt + V.F(u) (u) = solved on periodic domain using the finite element framework.
While not demonstrated in this work, it is conceivable to use function coefficients and initial/boundarry conditions global attributes to improve generalization in a graphnet policy for AMR.
5

---

Under review as a conference paper at ICLR 2022
In contrast to the static case, the numerical error accumulated at each time step propagates with the physical dynamics and determines future ror. Two types of PDEs were used: the lin- advection equation, where F(u)= cu with c = [1, 0), and the nonlinear Burgers equation, where F(u) cu² with c [1,0.3 0.3). The ad- vection equation is used as there is an analytic solution to provide ground truth, whereas the Burgers equation is used as the resulting solu- tions are representative of more complex phys- ical systems that include shock and rarefaction waves, albeit without avvillla analytic solution. The solution is represented using continuous (or discontinuous) second-order Bernstein polynomials for the static (or time-dependent) case, and the initial mesh is partitioned into n2 X ny quadrilateral elements.
(a) Bumps
(b) Circles
(c) Steps
(d) Steps2
Individual samples from each true lution function class. Each function sampled in bumps and circles is superposition of a random number of such features in general. Refinements shown here are produced by IPN.
Figure 2:
True solutions. We aim to show that RL policies have the potential to be deployed on larger and more complex problems without a priori known solutions by either training these policies on small representative features with known solutions utilizing the proposed surrogate reward at training time. As such, we defined a collection of parameterized function classes, each exhibiting features such as sharp discontinuities and smooth variations, from which we randomly sample ground truth functions f: [0, 1]2 + E to initialize each episode. The collection, shown in Figure 2 and defined precisely Appendix A.1. includes: bumps, circles, steps, and steps2 (a combination of two steps). These functions with closed form allow us to compute the error and reward at train time for static and advection problems. In the case of Burgers equation where the exact solution is not we either use reference simulations on highly-resolved mesh to act "ground truth" or employ the surrogate reward (defined Section 3.2) to compute the reward for training.
In the static case, the true solution is fixed and each simulation time step is an RL step. For the time-dependent PDE cases, the initial solution is transported through the periodic domain and the ratio of simulation time steps to RL steps is set such that a feature advecting at unit velocity returns to its original position after 10 RL steps. We set refinement budget B = 10 for static problems, B = 20 for advection, and B 50 for Burgers; episode length at train time equals B. Due to the Gibbs phenomena in FEM, using smooth polynomial approximations to solve hyperbolic systems containing discontinuities can introduce spurious oscillations which, in turn, can cause the simulation to become unstable. Therefore, we limit the true solutions to smooth functions (e.g.. bumps, circles) for the advection and Burgers cases. Due to the nonlinearity of Burgers equation, initially smooth solutions can develop discontinuities in finite time. This behavior is resolved using the flux correcere transport (FCT) approach (Boris & Book, 1997).
5.2 EXPERIMENTS AND PERFORMANCE METRIC
We conducted the following experiments to compare RL policies with baselines:
In-distribution: Train and test on true solutions sampled from the same function class.
:ittr--disttibution::: For problems such as Burgers equation where training on multiple initial con- ditions (ICs) may be expensive, we show the effectiieness of policies trained a single initial condition (IC) when tested on multiple random ICs, either with or without fine-tuning.
Policies trained on static functions are tested on advection. 2) Budget Policies trained with a small refinement budget (20 on static and 10 on advection) are test with B- = 50, 100. 3) Sizet: Policies trained on an 8 x 8 initial mesh are tested on 16 x 16 and 64 x 64 initial meshes with and without preserving the relative solution and mesh length scales.
Generalization: 1)
We define the performance of given refinement policy in an episode in the static case as (Cinitial /emma)/ where Emnttill (or Efina) is the error norm at the beginning (or end) of an episode, to remove the variation in the error due to different true solution classes and random function ini- tialization within each class. In the dependeennnt case, without any refinement, the error may increase over the course of the simulation due to the accumulation of discretization error. Hence, given a refinement policy that achieves Gúial at episode termination, we define its performance as (Cro refine, final Gfmal) Cinitial, where ren-ne final is the final error without any refinement.

---

Under review as a conference paper at ICLR 2022
We tuned hyperparameters by training multitask scenario for static and advection, and individu- ally for Burgers; the procedure and chosen values are given in Appendix A.3. For every experiment and every policy architecture, For each test case, we report the mean and standard error-over the four independent policies and with different simulator seeds of the mean performance metric over 100 test episodes. At each test episode, we ensured that all methods faced the same initial condition (which differs across episodes).
trained four independent policies with different random seeds.
5.3 IMPLEMENTATION AND BASELINES
We describe the high-level implementation here and provide complete details in Appendix A.2. All policy architectures use a convolutional neural network with the same architecture as the input layer. The IPN has two fully-connecten by so softmax output layer. Its action on input states is described in Section 4.1. The Graphnet policy is implemented with the Graph Nets library (Battaglia et al., 2018). Each input state consists of node observation tensors, all edge vectors, and the adjacency matrix. Node tensors first passed through an Independent block, after which multiple Interaction networks (Battaglia et al., 2016) act on both node and edge emheddings produce a probability each node (see Section 4.3). The Hypernet policy is parameterized by matrices U E _dxhi, V 1 ((dxh), and Y E Rdxh, where h and are design choices. and V act on input state 8 to produce the main policy weights W RdxA (see Section 4.2) while Y acts b). Output probabilities are computed in the same way as IPN.
hidden layers with h and h2 nodes and ReLU activation, followed
produce bias b E RA, that the main policy's
first hidden layer is ReLU(s
Baselines. The ZZ policy uses Zienkiewicz-ZMhe Zhu-ty--ehtteeeee recovery- baseed error estimator (Zienkiewiez & Zhu, 1992) and refines the element with the largest estimated error. The TrueError policy refines the element where the error of the numerical solution with respect to the true solution is largest. It is not the theoretical upper bound on performance because refining the element with largest error does necessarily result largest reduction of error but it does effectively pose an upper bound on the performance of an instantaneous error estimator. The GreedyOptimal policy performs one-step lookahead by checking all possible outcomes of refining each element individually and chooses the element whose refinement would result in the lowest eтror at the next step. In comparison to RL policies, only the 7Z. policy can feasibly deployed at test time as TrueError cannot be deployed on systems without known solutions and GreedyOptimal is intractable for real applications.
RESULTS
We find that the proposed methods achieve performance that is competitive with basclines and, more importantly, generalize well to larger refinement budgets and mesh sizes, and transfer effectively from static problem to time-dependent problem. Videos of policies advection and Burgers can be viewed at https:://// / ssitestt....//ee.///eees.eeetese google. cowmviiww/ccrr2222222
6.1 IN-DISTRIBUTION
Static functions (Figure 3a). RL policies either meet or significantly exceed the performance of ZZ on all function classes. Notably, both IPN and Graphnet outperform ZZ significantly on steps by spending the limited refinement budget only on regions with discontinuities (Figure 2c). On the smoother function classes such as bumps where ZZ is known to perform well, all three policy architectures have comparable performance to ZZ. Overall, IPN outperforms both Graphnet and Hypemnet, while Hypernet performed poorly (albeit still better than random) on all classes except
recuction
dormllized entor redlcction
Bomaino etror reduction
PN
0.125
0.125
Graghnet
03
0.100
0.100
Hyperwet
Random
22
0.2
0.075-
0.050
0.050
0.025
0.025
0.0
DUTDO
Crdine Jest
PEDS 20062
0.000
DUITES DC
crdss
0.000
dirlles
Test class
(a) Static
(b) Advection
(c) Static-radvection
Figure 3: In-distribution and Static work policies versus baselines. Higher values are better. (a.b) RL policies were trained and tested on the same function class, for static and advection cases independently. (c) Static-trained policies on a function class are tested on advection of the same function class.
Performance of IPN, Graphnet and Hypernet-
7

---

Under review as a conference paper at ICLR 2022
bumps. This suggests that capturing higher-order interaction among observations, each of which already contains local neighborhood information, is unnecessary for estimation of static functions as they only have local domain of influence. Hypernetwork policies converged to the behavior of making no refinements on at least one out of four independent runs on all classes except humps. This could be attributed to the inherent difficulty of choosing and training highly nonlinear model.
Advection (Figure 3b). As explained above, we limit the true solutions to smooth functions (bumps and circles) in the advection case. Graphnet significantly outperformed ZZ on circles and is com- parable to TrueError on bumps, while IPN is comparable to ZZ on both functions. Hypernet is comparable ZZ on circles but has high variance across independent runs, Graphnet's higher per- formance than other methods indicates that its inductive bias can better represent the local geometric relations between neighboring mesh elements along the circle.
Burgers equation. In experiments with a single bump function (visualized in Figure 4c) as the fixed initial condition (IC) in both train and test, IPN trained with both the exact and surrogate rewards outperformed all baselines (Figure 4a). The policy trained using the surrogate reward resulted in similar performance to the policy trained with the exact reward, indicating that the surrogate reward can be effectively used to train policies without the need for a ground truth solution, as is necessary for general random ICs.
No erter reducion
Puesii sumber
Tacks
Nomatzed
1.3
0,3
(c) Mesh by IPN (Fixed (d) Mesh by IPN (pre- trained, SR) at T'= 50 Figure 4: Burgers equation and surrogate reward. Solid bars/ER denote exact reward, striped bars/SR denote surrogate reward. (a) IPN trained and tested on a fixed IC. (b) IPN tested on random ICs using policies trained on fixed IC (from Figure 4a). policies pretrained on fixed IC and fine- tuned on random ICs, and policies only trained on random ICs. (c/d) Visualization of resulting meshes for Burgers equation with a fixed bump IC.
(a) Test on fixed IC
(b) Test on random ICs
IC. SR) at 7 = 50
6.2 OUT-OF-DISTRIBUTION (OOD)
Figure 4b shows the performance of IPN RL policies and baselines on Burgers equation with random ICs. We observe that policies trained on fixed IC using either the exact or surrogate reward (labeled "Fixed IC, ER" and "Fixed IC, SR") generalize well to random unseen ICs and still outperform baselines. Moreover, policies that were pretrained with the surrogate reward on the fixed IC for 2k episodes and fine-tuned with the surrogate reward random ICs for another 2k episodes performed the best ("pretrained, SR"). We truncated training around 4-5k episodes due to computational limits, but we see that performance can improve with more training episodes. Policies trained only on random ICs with the surrogate reward were not as performant, indicating that the training time was not sufficient and that pretraining on a fixed IC is more efficient approach. We do not envision that one would train and test on highly different function classes in applications involving static and advection problems, but we include experimental results of all combinations of train and test function classes in Appendix B.
6.3 GENERALIZATION
sadvec-advectioon (Figure 3c). All static-trained policies demonstrated comparable performance to 77. and TrueError when tested on advection-bumpe while both IPN and Graphnet significantly outperformed ZZ on advection-circles. Surprisingly, static-trained IPN significantly outperforms advection-trained IPN when tested on advection-circles, and the static-trained Hypernet does so as well on advection-bumps, while static-trained Graphnet maintains comparable performance to its policy
advection-traiine counterpart (Figure 3b vs. Figure 3c). Figure 12 shows that a on bumps with B- 10 correctly refines the region of propagation advection -umpss with B = 50.
Budgett (Figure 5). RL policies trained with low refinement budget generalize to test cases with higher budget. In the static case, comparing Figure 3a (B 10) with Figure 5a (B 50) shows that
8

---

Under review as a conference paper at ICLR 2022
Coppent
Deedon
PM
syperec
03
0.30
64
0.13
0.35
Lumco
39d0) 300
Llass
(a) Statie
(b) Advection
(c) IPN on static circle (d) Graphnet on advection
Figure 5: Budgett- (a-b) Policies trained with budget B 10 (static) and B 20 (advection)
tested with B = 50. (c) IPN trained with B = 10 generalizes to B = 100. (d) Graphnet trained on
advecting bump with B = 20 generalizes to B 50.
the performance of RL policies relative to ZZ is generally preserved by the increase in refinement budget. Figures 5 and show that an IPN trained with B = 10 makes qualitatively correct refinement decisions when allowed B 100 during test. In the advection case (Figure 5b), Graphnet trained with B = 20 significantly outperforms both ZZ and TrueError when tested with B = 50 on bumps and comes within the margin of error of TrueError
circles. Figure 7 shows that an IPN trained with B = 20 correctly allocates a higher budget B = 100 to the limited region of propagation.
Gracteet:
Farcon
TueDrr
Graptret
0:25
8.00
013
Nermateed
8.30
0.30
8.00
0.30
T5st dlsss
Test class
Best class
Test class
(a) Statie
(b) Advection
(c) Advection 64x64
(d) Advection 200x200
Figure 6: Sizet. (a-b) Policies trained with initial 8 x 8 mesh were tested on initial 16 x 16 mesh. (c-d) Policies trained on 8 x 8 initial mesh were tested on initial meshes of size 64 64 (c) and 200 x 200 (d) with approximately constant feature-to-mesh length scale ratio.
S (Figure 6). In the static case, the relative performance of RL policies that were trained with an 8 x 8 initial mesh (Figure 3a) is generally preserved when deployed on a 16 x 16 initial mesh (Figure 6a). All policy architectures outperform ZZ on bumps, while IPN and Graphnet still out- perform ZZ on steps. IPN and Graphnet were comparable to ZZ on 8 x 8 but underperformed on 16 x 16 on circles. Nonetheless, Figure 10 shows that IPN makes qualitatively correct refine- ments. On advection, relative performance is preserved on circles while IPN and Graphnet deproved slightly on humps (Figure 3b vs. Figure 6b). Without preserving the solution-to-mest length scales, the prior tests emulate deploying a policy trained on coarser versions of the simulations. When tested on 64 x 64 and 200 x 200 initial meshes that preserves the relative solution-to-mer length scales, which emulates deploying a policy trained on small subset of highly-resolved simulation, Figures 6c and 6d shows that IPN is competitive with baselines on bumps and even outperforms TrueError circles.
7 CONCLUSION
We present a novel formulation of adaptive mesh refinement as a Markov decision process and pro- pose new policy architectures for scalable application of reinforcement learning. Our experiments on static and time-dependent problems demonstrate that RL policies can outperform a policy based on the widely-used ZZ-type eπror estimator, and in some cases even outperform a policy based on the exact true eгror, suggesting that RL can potentially provide efficiency gains beyond the reach of existing AMR approaches. We demonstrated that these RL. policies generalize to different refine- ment budgets and larger meshes, transfer from static to time-dependent settings, and generalize to more complex problems even when trained without ground truth rewards. Moreover, because these RL policies do not use problem-specife knowledge or domain expertise as input, our results provide a path for learning novel AMR strategies for cases such as hp-refinement that currently lack effec- tive solutions. Future work can extend our RL methods to include derefinement actions, sampling multiple elements at each time step, and taking a multi-agent perspective that views each element as an agent who acts concurrently with all other agents.

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We have described the experimental protocol involving tuning, training, and testing in Section 5.2; the high-level implementation of the proposed method in Section 5.3; the specific implementation details the proposed method in Appendix A.2; the experimental test cases in Appendix A.1.2 and Table I; the detailed tuning procedure in Appendix A.3; and the complete specification of all hyperparameters in Appendix A.3, Table 2, and Table
REFERENCES
Ferran Alet, Adarsh Keshav Jeewajee, Maria Bauza Villalonga, Alberto Rodriguez, Tomas Lozano- Perez, and Leslie Kaelbling, Graph element networks: adaptive, structured computation and memory. In International Conference Machine Learning, pp. 212-222. PMLR, 2019.
Robert Anderson, Julian Andrej, Andrew Barker, Jamie Bramwell, Jean- SealvSylvain Camier, Jakub Cer- veny, Veselin Dobrev, Yohann Dudouit, Aaron Fisher, Tzanio Kolev, Will Pazner, Mark Stowell, Vladimir Tomov, Ido Akkerman, Johann Dahm, David Medina, and Stefano Zampini. MFEM: A modular finite element methods library. Computers & Mathematics with Applications, 81:42 74, 2021. ISSN 0898-1221, Development and Application of Open-source Software for Problems with Numerical PDEs.
Wolfgang Bangerth and Rolf Rannacher. Adaptive finite element methods for differential equations. Birkhäuser, 2013.
Yohai Bar-Sinai, Stephan Hoyer, Jason Hickey, and Michael P Brenner. Learning data-driven dis- cretizations for partial differential equations. Proceedings of the National Academy of Sciences, 116631):13344-5544 2019.
Peter Battaglia, Razvan Pascanu, Matthew Lai, Danilo Jimenez Rezende, et al. Interaction networks for learning about objects, relations and physics. In Advances in neural information processing systems, pp. 4502-4510 2016.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sanchez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:/806.00 1806.0/261, 2018.
Filipe de Avila Belbute-Peres, Thomas Economon, and Zico Kolter. Combining differentiable PDE solvers and graph neural networks for fluid flow prediction. In International Conference on Ma- chine Learning, pp. 22122411. PMLR, 2020.
Christopher Berner, Greg Brockman, Brooke Chan, Vicki Cheung, Przemyslaw Debiak, Christy Dennison, David Farhi, Quirin Fischer, Shariq Hashme, Chris Hesse, al. Dota with large scale deep reinforcement learning. arXiv preprint arXiv: 19/2.06600, 2019.
Jan Bohn and Michael Feischl. Recurrent neural networks as optimal mesh refinement strategies. Computers & Mathematics with Applications, 97:61-76, 2021.
Jay P. Boris and David L.. Book. Flux-corrected transport. Journal of Computational Physics, 135 (2):172-186, August 1997. doi: 10. 10.1006/jjhh..997.5700
Susanne Brenner and Ridgway Scott. The mathematical theory finite element methods, volume 15. Springer Science & Business Media, 2007.
Ignacio Brevis, Ignacio Muga, and Kristoffer G van der Zee. Data-driven finite elements methods: Machine learning acceleration of goal-oriented computations. arXiv preprint arXiv:2003.0444 2020.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. OpenAl Gym. arXiv preprint arXiv: 1606:16600015400 2016.
10

---

Under review as a conference paper at ICLR 2022
Jakub Červený, Veselin Dobrev, and Tzanio Kolev. Nonconforming mesh refinement for high-orde order finite elements. SIAM Journal on Scientific Computing, 41(4):C367-C392, January 2019. doi: 00.1137/18m11933
R Chedid and NN Najjar. Automatic finite-element mesh generation using artificial neural networks- Part I: Prediction of mesh density. IEEE Transactions on Magnetics, 32(5):5133-5178 1996.
Guodong Chen and Krzysztof Fidkowski. Output-based error estimation and mesh adaptation using convolutional neural networks: Application scalar ddvection-diffusiie problem. In AIAA Scitech 2020 Forum, pp. 1143, 2020.
Persi Diaconis. Bayesian numerical analysis. Statistical decision theory and related topics IV, 1: 163-175, 1988.
Veselin Dobrev, Patrick Knupp, Tzanio Kolev, Ketan Mittal, and Vladimir Tomov. The Target- Matrix Optimization Paradigm for high-order meshes. SIAM Journal on Scientific Computing, 41 ()):550-B68, 2019.
DN Dyck, DA Lowther, and S McFee. Determining an approximate finite element mesh density using neural network techniques. IEEE transactions magneticy, 28(2):1777-1770, 1992.
Gregory Farquhar, Laura Gustafson, Zeming Lin, Shimon Whiteson, Nicolas Usunier, and Gabriel Synnaeve. Growing action spaces, International Conference on Machine Learning, pp. 3040- 3051. PMLR, 2020.
Marco Gori, Gabricle Monfardini, and Franco Scarselli. A new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 729-734. IEEE, 2005.
David Ha, Andrew Dai, and Quoe V Le. Hypernetworks. In International Conference Learning Representations, 2017.
Jun- Ting Hsieh, Shengjia Zhao, Stephan Eismann, Lucia Mirabella, and Stefano Ermon. Learn- ing neural pde solvers with convergence guarantees. In International Conference on Learning Representations, 2018
Weizhang Huang and Robert D Russell. Adaptive moving mesh methods, volume 174. Springer Science & Business Media, 2010.
Ilay Luz, Meirav Galun, Haggai Maron, Ronen Basri, and Irad Yavneh. Learning algebraic multigrid using graph neural networks. In International Conference on Machine Learning, pp. 6489-6499. PMLR, 2020.
MFEM. MFEM: Modular finite element methods [Software]. https: //ff://///mmmm// org, 2020.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare Belle- mare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(75 2015.
Peter Monk al. Finite element methods for Maxwell's equations. Oxford University Press, 2003.
Ian Oshand, Yotam Doron, Matteo Hessel, John Aslanides, Eren Sezener, Andre Saraiva, Katrina McKinney, Tor Lattimore, Csaba Szepesvari. Satinder Singh, et al. Behaviour suite for reinforce- ment learning. In International Conference on Learning Representations, 2019.
Tobias Pfaff, Meire Fortunato, Alvaro Sanchez- Gonzalzz, and Peter Battaglia. Learning mesh-based simulation with graph networks. In International Conference Learning Representations, 2020.
Henri Poincaré. Calcul des probabilités, volume 1. Gauthier-Villars, 1912.
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014.
Junuthula Narasimha Reddy and David K Gartling. The finite element method in heat transfer and fluid dynamics. CRC press, 2010.
11

---

Under review as a conference paper at ICLR 2022
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2008.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXtv:1707.06347, 2017.
Alessandro Sperduti and Antonina Starita. Supervised neural networks for the classification of structures. IEEE Transactions on Neural Networks, 8(3):714-735, 1997.
Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in neural informa- fion processing systems, pp. 1057-1063, 2000.
Rakshit Trivedi, Jiachen Yang, and Hongyuan Zha. Graphopt: Learning optimization models of graph formation. In International Conference on Machine Learning, pp. 9603-9613, PMLR, 2020.
Oriol Vinyals, Igor Babuschkin, Wojciech Czarnecki, Michaël Mathicu, Andrew Dudzik, Juny- oung Chung, David H Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature, 575(7782) 2019.
Jiaxuan You, Bowen Liu, Zhitao Ying, Vijay S Pande, and Jure Leskovec, Graph convolutional policy network for goal-directed molecular graph generation. In NeurIPS, 2018.
Zheyan Zhang, Yongxing Wang, Peter K Jimack, and He Wang- Meshingnet: a new mesh generation method based on deep learning. In International Conference on Computational Science, pp. 186- 198. Springer, 2020.
OC Zienkiewicz, JZ Zhu, and NG Gong. Effective and practical h h-p-version adaptive analysis proce- dures for the finite element method. International Journal for Numerical Methods in Engineeringg 28(4):879-B8) 891, 1989.
Olgierd Cecil Zienkiewicz and Jian Zhong Zhu. The superconvergent patch recovery and posterioni еттоr estimates. Part 1: The recovery technique. International Journal for Numerical Methods in Engineering, 33(7):1331-1366 1992.
12

---

Under review as a conference paper at ICLR 2022
Algorithm Graphnet policy forward pass
1: for each pessaag--passing round do
2: for [l,...,Ne do
3: 2t t +--0((e (e(et, wr u" # Update edge attribute
4:
end for
5:
for i E {l...., N do
6:
E' = ((Et ph (h,ss)))rs )/== # Edge set for of Eº #+p p # Aggregation for of o' + ("',", (²)) # Update vertex attribute
7:
8:
9:
end for
10: e* 4- Eh wk: E [Ne] o vi [N]
end for
12:
R 3 x² ((v)), vi E [N]
13: T(alss) is the i-th entry of softmax (x) xx"
A EXPERIMENTAL SETUP
A.1 ENVIRONMENT DETAILS
A.1.1 MFEMCTRI.
To interface between the MFEM framework and the RL environment, we developed MFEMCtrl, C++/Python wrapper for AMR and FEM capabilities MFEM. MFEMCtrl used to convert solutions to observations, apply refinement decisions, and calculate errors.
The initial mesh partitioned into n₂ fly 8 x elements for static and advection experiments and 10x 10 for Burgers equation. Generalization experiments on larger initial mesh used x nyy 16 x 16 or 64 64. The true solution is projected onto the finite element space by interpolation to the nodes of the Bernstein basis functions. After each refinement action, the solution is projected again onto the refined mesh (for the static case) or integrated in time until the next refinement action (for the ime--eppedentt case). The maximum refinement depth is fixed by the parameter dmaa such that the maximally-refined mesh consists of 20uas x 2d ny elements. dmax was set to 3 for static experiments, whereas dmax 2 for advection and dmax for Burgers equation due the time step restrictions imposed by the Courant-Friedrichit
(CFL) condition of the finest elements.
Observation. The observation consisted of the solution and the depth of each element. Since the gradients of the solution are, by definition, a function of the solution. the observation does not include the gradients observed by interpolating the functions to a local equispaced mesh (image) centered around each element, shown by the white box Figure 1. Each element's observation is
they can be implicitly learned. The solution/depth of each element was
tensor where I = w Telement + 21comeett is the spatial observation window with /element 16 sampled points inside the element and lcomtext
sampled points in coordinate direction outside the element. We chose 2 channels that estimated function values and element depths are observed, while gradients are omitted since the policy network can in principle estimate gradients from the value channel. To impose 1:1 map between each observation and possible action, we append dummy o" to state corresponding to action 0. At most one refinement is allowed per MDP step.
13

---

Under review as a conference paper at ICLR 2022
Table 1: Parameterized true solutions
Steps
n Uniform Mmins Rmaax]
Parameter
[min, max]
0 Uniform m m
Cr
[0.2, 0.9]
Bumps (static)
Cy
[0.2, 0.9]
~ Uniform Omin Omax i 1,.....
w
[0.05, 0.2]
{1,...,6)
f(x, y)
tanh [100(0r - (x + tan 0))
Cr
[0.3, 0.71
Bumps (advection)
Cy
[0.3, 0.7]
Steps2
[0.005, 0.05]
n
(1....,4)
Uniform mmin Plmax.]
0.5
Uniform m mmax]
1.... n
Bumps (Burgers) single IC
Cy
0.5
Uniform Ominn O
1, .....
0,05
81 == (x 0.5) cos 0g (y 0.5) cos 0j
n
I
Ca
[0.3, 0.7]
f(x,8)
Σ + tanh [100(s; o;)
Bumps (Burgers) random IC
Cy
[0.3, 0.7]
w
[0.005, 0.05]
n
(1....,4)
Cr
[0.2, 0.8]
Cy
[0.2, 0.8]
Circles (static)
r
[0.05, 0.2]
[0.1, 1.0]
(1,...,6)
n
Cy
[0.3, 0.71
Cy
[0.3, 0.71
Circles (advection)
r
[0.05, 0.2]
w
[0.03, 0.05]
(1,...,4)
[0. 1.0]
Steps and Steps2
[0, a/2]
(1....,6) 6}
A.1.2 GROUND TRUTH FUNCTIONS
Bumps
n Uniform| min: mmax
Cz. Uniform/ Cr,min; Ca,max]
1,
Gyi Uniform| Cy.mins ma,
.....
wi Uniform Wmin; Wmax],
i = 1 .
(xr Ca
) + (y Cy,,)
f(x v) = exp
W0
Circles
n Uniform Mminn 7mmux]
Cr.i: Cy.á Uniform Cmin Cmax]
1, .....
r; Uniform Tmin- 7max]:
1,...,77 72
W/ Uniform Winin: Wmax]]
1 ........
(v(x Cx,s) + (y Cy. ) T2)2
f(x,)) Σ exp
W:
14

---

Under review as a conference paper at ICLR 2022
A.1.3 REWARD
Let u, denote the true solution at time let ūn denote the estimated solution on the mesh time t. For a given mesh at time t - 1, a given time evolution of the true solution from t -- to t, and a refinement action a (which may be "do-nothing"", let aetinee denote the estimated solution on the mesh at time t that has undergone refinement action de, and let यt no-rriinee denote the estimated solution on the mesh at time without that refinement. We have two reward definitions:
1. Delta norm reward with true solution
ra (et-1 cco/co
(5)
C2 = 11 - 01112
(6)
2. Surrogate reward
refine úe no-refine
We used the first reward definition for static and advection experiments where analytic true solutions are available, and for Burgers experiments involving a single initial condition and pre-computed reference data that acts as the true solution. We used the second reward definition for all other Burgers experiments.
A.2 IMPLEMENTATION
We used standard policy gradient (Sutton et al., 2000) for all experiments except for experiments on Burgers equation and generalization of 8x8-trained advection policies to 64x64 test meshes. We used PPO (Schulman al., 2017) for the latter two cases. We trained for 20k episodes on static problems, 10k episodes on advection problems, 2k episodes on Burgers equation with a single IC, and 4k episodes on Burgers equation with random ICs. The Burgers experiment with pretraining used 2k episodes on single IC and further 2k on random ICs. Each episode is initialized with refinement budget B, where Burgers.
for static problems,
for advection, and B = 50 for
IPN. For efficient computation on a batch of trajectories, where each trajectory consists of T environment steps and each step to consists of a variable-sized global state R,V,, xd merge the variable dimension with the batch and time dimension to form an input matrix whose dimensions are DE 1- Et= Nr,, d]. The output is reshaped into a "ragged" matrix of logits with dimensions [ T.NNN. where the row lengths vary for each batch and time step. A softmax operation over each row produces the final action probabilities at each step.
Graphnet policy. The first graph layer is an Independent recurrent block that passes the input node tensors through a convolutional layer followed by fully-connected layer, to arrive at node embeddings. This is followed by two recurrent passes through an InterationNetwork (Battaglia et al., 2016) where fully-connected layers are used for edge and node update functions. A final InteractionNetworkk output layer followed by a global softmax over the graph produces a sealar at each node, which is interpreted as the probability of selecting the corresponding element for refinement. Except for the input node feature
E IRd and output node scalar, all internal node (edge) embeddings have the same size, denoted as dim(c) (dim(e)). We fixed dim(e) 16 for both static and advection and tuned dim(z) (Table 2).
Hypernet policy. We fixed the main network's hidden layer dimension at h 64 and tuned the hypernetwork's hidden layer dimension h1 (Table 2).
A.3 HYPERPARAMETERS
For both static and advection problems, we tuned a subset of all hyperparameters for all methods by the following procedure to handle the large set of policy architectures and ground truth functions. Chosen values of tuned hyperparameters are given in Table 2; all other hyperparameters have the same values for all methods and are listed below. We conducted tuning in multi-task setup, where we train a single policy on functions randomly sampled from all ground truth classes, with randomly sampled parameters according to Appendix A.1.2. This is done separately static and advection
15

---

Under review as a conference paper at ICLR 2022
problems. The tuning process is coordinate descent where the best parameter from sweep is used for the next sweep. We start with exploration decay cav E {100, 500, 1000, 5000} (a lower bound on exploration with e decaying linearly from start to fend by édiv episodes). Next we tune the size of hidden layers in the policy network (over (h1, E (128, 64), (256, 64), (128, 128), (256. 256)] for IPN, node representation dimension dim(v) E {32, 64, 128, 256} for Graphnet, and h1 E {16.11116 32, 64, 128] for Hypernet). Lastly, we tune the learning rate E {5 10 3 10 10.110...00 55 10 - 10 5 10-33. For Graphnet and Hypernet, we inherit the best Caiv from IPN because optimal exploration depends in large part on the complexity of the environment, which is the same across all policy architectures.
enforced by using behavioral policy #[ag|s,)
(1 () (s,) + e/Ne
Separately for the static and advection cases, all three policy architectures have the same values for all other hyperparameters. These are: policy gradient batch size 8, initial exploration lower bound Sxtat 0.5, final exploration lower bound Cend = 0.05, discount factor 7 0.99, convolutional neural network layer with 6 filters of size (5, 5) and stride (2,
Table 2: Hyperparameters for IPN, Graphnet and Hypernet policies on static and advection AMR.
Static
Advection
Parameter
IPN
Graphnet
Hypernet
IPN
Graphnet
Hypernet
Ediv
500
500
500
100
100
100
IPN (hi, h2)
(128, 64)
(256,256)
Graphnet dim(v)
64
256
Hypernet h1
128
32
a
10
10
4 5
10
10-4
10 10-4
10 10-4
For Burgers experiments and advection experiments on generalization from 8 x 8 to 64 x 64 initial mesh sizes, we used a more comprehensive population-based hyperparameter search with successive elimination for all methods. We start with batch of Tbarch tuples, where each tuple is combination of hyperparameter values, with each value sampled either log-unifornly from a continuous range or uniformly from discrete set. We train independently with each tuple for Repisode episodes, eliminate the lower half of the batch based on their final performance, then initialize the next set of Repisade episodes with the current models for the remaining tuples. We use the hyperparameters of the last surviving model. Chosen values are shown in Table 3.
The hyperparameter ranges are: discount factor in {0.1, 0.5, 0.99), policy entropy coefficient in (10-3, 1.0), GAE A in {0.85, 0.90, 0.95]. learning rate in (10 10-3, PPO in (0.01, 0.5), value loss coefficient in {0.1, 0.5, 1.0]. IPN h2 in {128, 256). and IPN h2 in {64, 128, 256).
Table 3: Hyperparameters for advection size t and Burgers experiments
Advection (IPN)
Burgers (IPN)
Parameter
Bumps
Circles
1 IC
Random IC
Discount
0.99
0.99
0.1
0.5
Entropy coefficient
0.0133
0.0689
8.84
10-3
1.17
10-3
GAE A
0.95
0.9
0.85
0.85
Learning rate
1.18
10-3
4.8
10-3
1.59
10-3
2.11
10
PPO
0.0113
0.195
0.128
0.169
Value loss coefficient
0.1
0.5
0.5
0.1
IPN h1
128
256
256
128
IPN h2
128
128
128
256
A.4 COMPUTING INFRASTRUCTURE AND RUNTIME
Experiments were on Intel 8-core Xeon E5-2670 CPUs, using one core for each independent policy training session. Average training time with 20k episodes in the static case was approximately 6 hours for IPN and Hypernet, and 9 hours for Graphnet. Average training time with 10k episodes
16

---

Under review as a conference paper at ICLR 2022
in the advection approximately 14 hours for IPN and Hypernet, and 18 hours for Graphnet. Policy decision times are shown in Table 4.
Table 4: Mean (standard error) time in milliseconds per refinement decision on 8 and 16 16 initial mesh partitions.
8 x
16 16
24 24
IPN
3.22 (0.07)
5.85 (0.05)
9.64 (0.28)
Graphnet
7.74 (0.33)
13.9 (0.43)
23.7 (0.19)
Hypernet
8.08 (0.08)
10.7 (0.05)
14.3 (0.17)
ZZ
1.96 (0.01)
6.94 (0.01)
15.5 (0.05)
1= 0
t= 32
68
( 100
Figure 7: Advection of a bump function. RL policy trained with budget B 20 generalizes to B = 100.
(a) Steps
(b) Steps2
Figure 8: Generalization of policies trained with refinement bud- Figure 9: Example test case on x mesh
get = 10 to test case with B = 100.
B ADDITIONAL RESULTS
OOD, In the static case (Figures 13a to 13c), IPN policies trained on circles transfer well to bumps (and vice versa). Hypernet policies performed poorly overall even in the case of in-distribution. and consequently does not show comparable performance when transferring across function classes. In the advection case (Figures 13d 13l), both IPN and Graphnet policies trained bumps signif- icantly outperformed ZZ when tested on circles (compare to ZZ in Figure 3b).
17

---

Under review as a conference paper at ICLR 2022
2.0
340
235
250
220
Furth
5.0
heep
(a) IPN
(b) ZZ
(a) 1-bump Burgers
(b) Multi-bumps Burgers
Figure 10: IPN trained on 8 x initial mesh underperformed ZZ when tested on 16 x 16 initial mesh but makes qualitatively correct refinements.
Figure 11: OOD. (a) IPN (OOD) was trained Burgers with multi-bumps IC and tested on Burgers with 1-bump IC. (b) IPN (OOD) was trained 1-bump IC and tested with multi-bumps IC.
f=0
t= 22
t
38
Figure 12: Static-radvection and Budget IPN trained on static bumps (B 10) transfers to advection (B 50).
bumps
steps
fell
hamoo
sieoN #2e002
2x,
hamoo
www. s7e002
>> fvll
cirlls
sboo52
drries
dircies
Normalized ertor onduction
Normelized lieed ertor reduct.on
Normalized lized error oeducttonn
0.3-
0.1
0.1
riries
steo?
27
humps
circles
Meok Mepo
77
humas
Mieo Meppp
Train class
Train class
Train class
(a) Static IPN
(b) Static Graphnet
(c) Static Hypernet
bumns
circtes
bumps
cicless
bumps
Cirless
Normalized etror reduccion
Normalized error reduction
Normalized error reductinn
0.08
0.06
0.08
0.06
0.06
0.04
0.04
0.02
0.02
0.00
Train
(d) Advection IPN
(c) Advection Graphnet
(f) Advection Hypernet
Figure 13: All train-test combinations. Normalized error reduction of IPN, Graphnet and Hyper- network policies on (a-c) Static AMR and (d-f) Advection PDE. Higher values are better. Legend (colors) shows test classes. RL policies were trained and tested each combination of true solu- tions. Mean and standard error over four RNG seeds of mean final error over 100 test episodes per method.
18


Reviews:
[{"review_id": "n2kBJiWGx7D", "reviewer": "Reviewer_sdD7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- A MDP formulation for mesh refinement\n- Different policy architectures to capture mesh geometry\n- Experiment on different settings: time-dependent, state and action size changes, and generalization to unseen mesh sizes and varied budget.\n\nCons: \n\n- Unclear MDP formulation\n- Policy network architectures are not novel.\n\n \nIn overall, the paper pursues an interesting direction. Dynamical mesh refinement is an important feature in numerical simulation of PDEs. It makes sense to formulate the process of dynamical mesh refinement as a MDP where state and action are well defined. It's also a complex problem if the dimensionality of state and action is allowed to vary over time. \n\n1. At first, the description in section 3 can cover a bit more background of AMR so that the readers can follow and get ideas on how the formulation in 3.2 works. \n\n2. Regarding the formulation in 3.2, as the authors define a MDP for a new sequential decision making problem, this formulation can be more complete, if it includes detailed definition of transition. It's hard to understand how transition is computed and defines the dynamics based on only short description. The same issue for the reward defection is hard to understand how those values are computed. It be helpful if the authors consider providing more description of AMR background.\n\n3. Three policy architectures are good. However I could not find where in the paper the policy is designed to capture time-dependency and the change of state and action sizes.\n\n4. The experiments are interesting which can show the benefit of the proposed approach. I have only one point that is hard to follow. Why the TrueError is worse than other methods. What is the implication of this baseline? Does it mean the metrics used to define rewards is not optimal? If so, can the formulation be different, e.g. taking into the downstream task's reward instead of direct mesh refinement's error?\n", "summary_of_the_paper": "This paper proposes an application of reinforcement learning for adaptive mesh refinement in large-scale finite element simulations of complex physical systems. The authors suggest to formulate the mesh refinement problem as a MDP and propose different policy architectures for scalable application of reinforcement learning. Experiment results demonstrate that the proposed RL approaches outperform existing baselines, and can generalize well to situations of different finement budgets and larger meshes.", "summary_of_the_review": "See above\n===========================\nAfter the rebuttal: Thanks the authors for the detailed response. \nThough I found the proposed idea of using RL for AMR is interesting, the technical contribution is still limited. \nTherefore I like to keep my original rating.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "f04avUDEY1F", "reviewer": "Reviewer_Gif3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Article review and position the paper in the large context of AMR:\n(major) Authors write in the abstract: \"Existing scalable AMR methods make heuristic refinement decisions based on instantaneous error estimation and thus do not aim for long-term optimality over an entire simulation.\" This is a bit strong statement. For example, people in computational science for all finite-element, finite-volume, and spectral finite element (SEM) use mesh sensitivity method for AMR. In such cases, adjoint solver is used to measure the sensitivity of the solution. Although this approach employs some coefficients that are hard to quantify a-priori, however, there is a very solid and concrete argument on how to choose them, as well as their interpretability. Since, tuning the parameters is also part of RL and all three architectures used, I think this part should be removed, which brings me to another discussion. What is the fundamental reason of using RL for AMR compared to established AMR methods in finite-element or SEM community? There exists several differentiable FE, FV, or SEM solvers that would readily give the user the adjoint variables using AD, which in turn, can be used for AMR. The algorithm is automated, and based on calculus of variation so mathematically sound. Also the results are either as good as other AMR methods or even exceeding them (please see Adjoint error estimators and adaptive mesh refinement in Nek5000, among many others). When using adjoint-based AMR, it is also possible to define the cost function for mesh refinement based on the terminal solution instead of instantaneous refinement. So that aspect of the paper is also not necessarily novel in a sense that other class methods can also accomplish that. Furthermore, when it comes to optimal control, the mesh refinement can be done with a specific task of achieving the cost function related to the performance (see Rannacher: model reduction by adaptive discretization in optimal control or Apel et al Graded meshes in optimal control for elliptic partial differential equations: an overview, among others). Such approaches are also very scalable, which based solely on the results of this paper, it's hard to claim. Overall, I think i) some claims are not true, ii) some traditional AMR methods are absent in the analysis and introduction of this paper. \n\nDiscussion on test cases and practical aspect of the algorithm:\n(major) The test cases considered in this manuscript are toy models that can be served as \"proof of concept\". However, I'm not convinced that such models can be really considered a proof-of concept. The geometry is simple 2D periodic domain and in almost all practical PDE analysis, this is not the case. One beautiful aspect of the work is Out-of-distribution testing, which for the Burger PDE is done based on IC. This is an important test but what about the cases that BCs are changed? The Burger equation is inviscid, what about viscous Burger, for which the Reynolds number (or viscosity) can be changed. A parametric study of PDEs is always challenging and if authors can show the algorithm is robust to such changes, that would a strong argument for the RL-based AMR. Also, the geometry is so simple and for mesh generation this is really a challenge. One of the main concerns of numerical scientists is the complex domain/field analysis. For example, when there is a boundary layer in vicinity of airfoils. The method should demonstrate itself when it comes to more complex geometries. Otherwise, it hard to justify why use RL instead of traditional methods. Finally, MDP transition P consists of one step in PDE solution. Considering RL may require a large number of trajectories, for complicated PDEs, I imagine the training for all relevant parameters (IC, BCs, parameters such as viscosity, etc.) may be intractable. How do authors respond to that?\n(minor) One nice trick in the paper is the use of so-called surrogate reward for training. However, it's not clear to me why such reward is superior to exact reward in results of Figure 4. Can authors clarify that? Why not such comparison for other cases?\n(minor) Seems like the GreedyOptimal approach is superior in Fig 3 results. Why not comparing with this AMR technique for the rest of figures?\n\nDiscussion on the solution space and mathematical definitions: \n(major) The finite element requires a bit of discussion on the mathematical formalism. For example, authors discuss \"test function\", which has a established meaning in finite element community, but seems like they don't refer to the same concept. The finite element method is a discretization of the weak formulation, for which one use the inner product with test function. The next step is the transformation of such weak formulation from the functional space with infinite basis into the functional space with finite basis Such test functions, all of which belong to appropriate Hilbert spaces that should be mentioned (for example, it could be H1 space). I understand that authors are relating to the ML community, for whom the \"test functions\" may have a separate meaning but this causes confusion. \n(minor) Furthermore, authors use \"finite element space\", which again is not formally defined.\n(major) On a similar note, how do authors enforce continuity within elements, if they belong to H1 space? If this is not necessary, then authors should clarify. \n\nDiscussion on the algorithm:\n(minor) From Fig 5-d, and the manuscript, I'd say authors use non-conformal elements, which is fine; however, how to enforce conforming mesh? Is it possible to enforce not having hanging nodes? Some finite element solvers can't handle the non-conforming mesh and should be modified. \n(major) How about mesh quality check? This is specially true if triangular elements are going to be used, which is likely for complex geometries. How to avoid de-generative elements, or irregular mesh skewness? Seems like a constrained RL framework should be used but that would require changing the three architectures proposed. \n(major) Authors mentioned de-refinement (or coarsening) is a future direction of this research. However, almost all traditional AMR algorithms use both refinement (in regions with high sensitivity) and de-refinement (in regions with low sensitivity) to achieve not only an accurate solution but also low computational cost. Dropping the de-refinement in AMR algorithm makes it much less competitive compared to state-of-the-art.\n\n\n", "summary_of_the_paper": "For various complicated problems governed by PDEs (e.g. solid/fluid interactions, aerodynamics, elasticity, backscattering, etc.) the computational cost can become prohibitive even for one inquiry, let alone parametric study. In the same time, mesh refinement is crucial to achieve acceptable accuracy. To mitigate such challenges, one solution is to use adaptive mesh refinement (AMR), for which the mesh refined only in the regions that numerically are sensitive to error propagation. For example for boundary layer models or shock-boundary layer interactions, one much capture the dynamics in high-gradients region of the solution, which are typically in vicinity of the walls or are part of the solution, while in very far regions of the domain, a coarse mesh is sufficient. \nAuthors recognize that the process of AMR, refinement at each step, can be formulated a Markov decision process (MDP) and hence utilize reinforcement learning (RL) to train refinement policies directly from simulation. But this in turn poses a new challenge, at each step, the dimension of state (number of elements) and action space may (and should) alter. They propose suitable policy updates to overcome this challenge and come up with three different architectures for the implementation. Three test cases are used for experiments (static, advection and Burger) and authors compare all three architectures with each other as well as some traditional AMR methods to demonstrate the performance of the proposed method. They also carry out extra tests on the same set of PDEs to show generalization and out of distribution capabilities of the method for both static and transient PDEs. The paper is well-written and two tricks for RL are impressive (using Nmax to take care of varying dimension of state/action space and use of surrogate reward for training). However, I have some major reservations that I'll explain below in the Main Review. \n", "summary_of_the_review": "The paper is well-written and two tricks for RL are impressive (using Nmax to take care of varying dimension of state/action space and use of surrogate reward for training). However, I have some major reservations: 1) Article review and position the paper in the large context of AMR (the adjoint-based AMR and goal-oriented mesh refinement in the PDE-optimal control community can achieve all goals proposed in the paper), 2) Discussion on test cases and practical aspect of the algorithm (other BCs, parameter should be explored, geometries are too simple for mesh generation whose main backlog is dealing with complex problems, scalability for complicated PDEs like Navier Stokes over complex geometries, etc.), 3) Discussion on the solution space and mathematical definitions (this is rather lighter comment but I think authors should clarify some concepts better in the manuscript), 4) Discussion on the algorithm (what about other constraints of AMR that traditional algorithms can handle, de-refinement, etc.). \nOverall, when it comes to RL for AMR I think it is novel and authors use some nice tricks to handle the challenges. But the overall use of RL for AMR over traditional methods is not demonstrated and there remain major concerns regarding the pragmatism of the method.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "dxIVAmC4yEO", "reviewer": "Reviewer_rRgy", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The problem of mesh refinement is critical in engineering, and current practices, as the authors stated, are often designed from some heuristics. The application of RL to this mesh refinement, as far as I know, is novel, promising, and may provide new ideas to solving this open problem. Nevertheless, the authors tested their framework on some toy examples with low resolution and barely sufficient validations. \n\nThe PDEs to be solved in practice, as the authors indicated, usually have millions or billions of degrees of freedom. The authors claimed to \"show that an RL refinement policy can generalize to higher refinement budgets and larger meshes.\" However, the \"larger meshes\" only contain 64x64 (4096) nodes, which is only a tiny fraction of the PDEs to be solved in practice. Hence, it is questionable whether the method may apply to PDEs on a practical scale, showing that the policy is \"efficient\". In other words, the paper would be more convincing if the authors demonstrate a case where their method is applied to a mesh containing millions of degrees of freedom and still outperform the baselines.\n\nAnother concern is about validation. The authors compared their policy with some trivial policy but did not compare with the state-of-the-art in adaptive mesh refinement. Hence it is not convincing that the RL-based method may outperform the traditional \"heuristic\" methods on this problem. The paper would be more grounded if the authors compare the RL-based method with the latest, non-RL-based method on the same problem set.", "summary_of_the_paper": "This paper presented an RL-based method to perform adaptive mesh refinement to solve PDEs more accurately. ", "summary_of_the_review": "The paper proposed a promising direction for future research, but it would be more convincing if the authors tested their method on practical scales with more solid validations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "8PQ8ciuzgeU", "reviewer": "Reviewer_aatv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is mostly clearly written, and has a conceptually simple yet clear idea behind it. At its core, it proposes to use reinforcement learning to train neural networks to propose adaptive mesh refinement operations in order to reduce simulation error (in contrast to traditional methods, which tend to use heuristics to perform the adaptive refinement). Therefore, this work's evaluation rests mainly in the empirical demonstration of the benefits of the proposed method.\n\nHowever, the practical implementation of this simple becomes more convoluted, and is one of the weak points of the paper. Three variants for the neural network architecture are proposed, a simple \"per-node\" architecture, a hypernetwork based one and a graph-net based one. It is not necessarily an issue to propose diverse variants of a single method. However, in this case there is not much justification as to why different methods are tried, or how one would choose one over the other when applying the proposed approach in practice. \nIn the end, the results are presented in such a way that the best performing of the 3 proposed variants is presented as better than baselines, though no single one of the 3 is able to consistently perform best. Given that there is no analysis or proposed method to pick a priori which one of the variants is more appropriate for a given task, the results become muddied.\n\nNevertheless, the so called \"Independent Policy Network\" (IPN) seems to be the generally best performing variant, usually performing better or on par with the baselines, though not always. It is important to take into account that training the neural network model incurs some considerable cost, and as such performing on par with a traditional method might not be enough.\n\nTo this end it would also be important to have comparisons of run-times between the methods and baselines, not only error reductions. This is required in order to evaluate if the proposed mesh refinements are actually reducing error by being \"smart\" about the refinement, instead of simply proposing \"more\" refinement, or more intense computation.\n\nSome additional questions and comments:\n\n- Even though the authors state it should be simple to implement, the policy networks in this work do not have a \"coarsen\" or \"de-refine\" action. Does this mean the meshes simply get finer and finer throughout the simulation? How does this impact long simulations, and ones where for example a shock wave is moving (and thus the region that requires refinement is moving)? Would this lead to an overall very fine mesh? This is important in light of the comment above, regarding comparisons of run time of the proposed method and baselines\n\n- For the IPN, the description is a bit confusing. At one point it is stated that it does not depend on other elements, yet at a later point it is said that information of surrounding elements is included. It was also unclear to me how the convolutions are applied here. It would be good to improve this description.\n\n----------\n\nAfter reading the authors response and discussing the issues above with the authors (see discussion in the thread below), I am still not fully convinced of the robustness and strength of the experimental evaluation. As such, I will maintain my previous evaluation.", "summary_of_the_paper": "This paper proposes using reinforcement learning algorithms to train a few variants of neural network-based models to perform adaptive mesh refinement. These models predict for each node in a mesh where a refinement operation should be performed.", "summary_of_the_review": "Overall, given the concerns above, I do not feel like this work is ready for acceptance at this point. This is a work that rests mostly on its empirical results, and yet these are not completely robust at this point, given the issues described previously. As such, I am classifying as marginally below acceptance for now. Nevertheless, I am interested in reading the authors responses to the points raised above.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5ijqn6ZqY6t", "reviewer": "Reviewer_LhZp", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Q1. \"Nonetheless, to our knowledge, such variation in state-action spaces does not occur in any existing RL application.\" This comment is not valid. There are many RL formulations and applications that can generalize over a varying dimension of states and actions. Refers RL with neural graph networks and includes these references. For example, many RL approaches seek to solve the solution of various optimization problems formulated as graphs. These approaches train the policy using RL with the small-sized problem and then apply trained policy to solve large-scale unseen problems. \n\nQ2. Is action defined as selecting a computational grid among all the grid points under the current mesh strategy? The proposed action definition sounds inefficient since only one grid point can be selected at each iteration. Is there any reason or justification for using this action definition? \n\nQ3. Why is the simple policy gradient algorithm used? Is there any room for improving the performance by employing more advanced or problem-specific learning algorithms?\n\nQ4. Why independent policy networks (section 4.1) and Hypernetwork policy (section 4.2) are discussed in the main text? Obliviously these policy structures should be inferior to Graph neural network-based policy, and the primary proposed model is GNN based model. \n\nQ5. Which modeling component contributes the most to the generalization capability? For example, state representation and the representation learning, specific reward form, or specific learning strategy? \n\nQ6. How are the computational times compared both in training and test phases?\n", "summary_of_the_paper": "The current study formulates adaptive mesh refinement (AMR) as a Markov decision process and applies deep reinforcement learning (RL) to train refinement policies directly from simulation. The proposed method achieved high accuracy while reducing the computational time. ", "summary_of_the_review": "Formulating AMR as an MDP sounds reasonable, and structuring meshing policy using the graph neural network is also straightforward. Formulating a specific component (step) of an iterative algorithm using a learning-based module (i.e., RL) and GNN to learn a policy that can generalize over varying sizes and actions is a pervasive idea. Thus, the technical novelties of the proposed method are limited. The only novelty would be employing these ideas to finite element analysis.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work formulates the Adaptive Mesh Refinement (AMR) problem used in solving Finite Element Method (FEM) as an MDP, and suggests an RL-based solution for it. Most reviewers agree that this is a novel problem and the solution is promising. There are, however, several issues raised by our reviewers, who have expertise ranging from ML to computational methods to solve PDEs. Some of the concerns are:

- As this is not a theoretical work, the burden of proof is on the empirical evaluations. Some reviewers found the experiments very small and not convincing enough.
- The paper does not compare with the state of the art AMR methods.
- The detail of how the problem is formulated as an MDP can be improved.

Given that four out of five reviewers are on the negative side, unfortunately I cannot recommend acceptance of this paper in its current form. Nevertheless, I believe this is a promising application of RL. I'd like to encourage the authors to consider the reviews in order to improve their work, and resubmit it to another venue.
Decision: Reject