Swarm Reinforcement Learning for Adaptive Mesh Refinement
Authors: Niklas Freymuth
Philipp Dahlinger
Tobias Daniel Würth
Simon Reisch
Luise Kärger
Gerhard Neumann
Venue: NeurIPS 2023 poster (2023)
Keywords: Adaptive Mesh Refinement
Finite Element Method
Swarm Reinforcement Learning
Graph Neural Networks
TL;DR: We view a mesh as a swarm of agents and use Swarm Reinforcement Learning to find strategies for adaptive mesh refinement using a novel spatial reward formulation.

Abstract: The Finite Element Method, an important technique in engineering, is aided by Adaptive Mesh Refinement (AMR), which dynamically refines mesh regions to allow for a favorable trade-off between computational speed and simulation accuracy. Classical methods for AMR depend on task-specific heuristics or expensive error estimators, hindering their use for complex simulations. Recent learned AMR methods tackle these problems, but so far scale only to simple toy examples. We formulate AMR as a novel Adaptive Swarm Markov Decision Process in which a mesh is modeled as a system of simple collaborating agents that may split into multiple new agents. This framework allows for a spatial reward formulation that simplifies the credit assignment problem, which we combine with Message Passing Networks to propagate information between neighboring mesh elements. We experimentally validate the effectiveness of our approach, Adaptive Swarm Mesh Refinement (ASMR), showing that it learns reliable, scalable, and efficient refinement strategies on a set of challenging problems. Our approach significantly speeds up computation, achieving up to 30-fold improvement compared to uniform refinements in complex simulations. Additionally, we outperform learned baselines and achieve a refinement quality that is on par with a traditional error-based AMR strategy without expensive oracle information about the error signal.

Swarm Reinforcement Learning for Adaptive Mesh Refinement
Niklas Freymuth" 1- Philipp Dahlinger' Tobias Würth Simon Reisch' !
Luise Kärger Gerhard Neumann'
Autonomous Learning Robots, Karlsruhe Institute of Technology, Karlsruhe Institute of Vehicle Systems Technology, Karlsruhe Institute of Technology, Karlsruhe
Abstract
Adaptive Mesh Refinement (AMR) enhances the Finite Element Method, im- portant technique for simulating complex problems in engineering, by dynamically refining mesh regions, enabling favorable trade-off between computational speed and simulation accuracy. Classical methods for AMR depend on heuristics or expensive error estimators, hindering their use for complex simulations. Recent baaming-based AMR methods tackle these issues, but so far scale only to simple toy examples. We formulate AMR as a novel Adaptive Swarm Markov Decision Process in which a mesh is modeled as a system of simple collaborating agents that may split into multiple new agents. This framework allows for spatial reward formulation that simplifies the credit assignment problem, which we combine with Message Passing Networks propagate information between neighboring mesh elements. We experimentally validate our approach, Adaptive Swarm Mesh Re- finement (ASMR), on challenging refinement tasks. Our approach learns reliable and efficient refinement strategies that can robustly generalize to different domains during inference. Additionally, achieves speedup of up to 2 orders of magnitude compared to uniform refinements in more demanding simulations. We outperform learned baselines and heuristics, achieving a refinement quality that is on par with costly error-based oracle AMR strategies.
Introduction
The Finite Element Method (FEM) is widely used numerical technique in engineering and applied sciences for solving complex partial differential equations [1, 2, 3, 4J. The method discretizes the continuous problem domain into smaller, finite elements, allowing for an efficient numerical solution. A key aspect the FEM for complex systems is Adaptive Mesh Refinement (AMR), which dynamically refines regions of high solution variability, allowing for favorable trade-off between computational speed and simulation accuracy [5, 6, 7]. As problems in engineering grow more complex, the FEM and especially Adaptive Mesh Refinement (AMR) techniques have become increasingly important tools in providing computationally tractable yet precise solutions. Applications of AMR include fluid dynamics [8, 10. 11. 12. 13], structural mechanics [14, 15, 16, 17], and astrophysics [18, 19, 20J. Yet, classical approaches for AMR usually rely on either problem-dependee or more general but potentially suboptimal error indicators, or require expensive error estimates [21, 22, 23, 24, 25, 26, 12). In either case, they can be cumbersome to use in practice.
To address this issue, formalize AMR Reinforcement Learning (RL) [27] problem. Following previous work [28, 29, 30], each refinement step encodes the state the current simulation as local observations that feed RL agents, who then determine which elements of mesh refine. However, previous work has issues with scalability due to an expensive inference process [28),
"correspondence niklas. wtw.lass.rrrymtttktt...o edu
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Next Step [=++
Reward per Agent F(2;)
Observation Graph Gnc
Refine
Marked
ASMR Pollcy xt (a Gnz
Elemants
Refine? Yes No
Current Mesh ft
Next Mesh Ω
Figure 1: Given a mesh Ω2', an observation graph 9p' encodes the elements as graph nodes and the neighborhood between elements as edges. The graph given learned policy x, which marks mesh elements for refinement. A remesher refines the mesh, and spatial rewards r(22)) are calculated for all agents based on the quality of the refinement. This process iterated for several steps until the mesh is fully refined.
misaligned objectives and high variance in the state transitions [30], and noisy reward signals [29]. To mitigate these shortcomings and scale to more complex problems, we formulate AMR as Swarm RL [31, 32] problem. We extend the Swarm RL. framework to per-agent rewards, a shared observation space, and the option of splitting agents into new agents as required in the AMR process. Additionally, we introduce a novel spatial reward formulation that provides a dense reward signal for the refinement of each mesh element, simplifying the credit assignment problem for swarm systems. Our policy based on Message Passing Networks (MPNs) [33], a class Graph Neural Networks (GNNs) [34, 35, 36, 37] that has proven to be effective for physical simulations [38, 39]. The resulting method, Adaptive Swarm Mesh Refinement (ASMR), consistently produces highly refined meshes with thousands of elements while applying to arbitrary Partial Differential Equations (PDEs). A high-level overview is given in Figure 1.
Experimentally, we show the effectiveness of our approach on suite of PDEs that require complex and challenging refinement strategies, including a non-stationary heat diffusion problem and linear elasticity task. We implement our tasks OpenAl gym [40] environments. The experiments use static meshes with conforming triangular elements and corresponding h-adaptive refinements [41, 42] due to their importance in engineering applications [43, 44, 45, 46). Here, accurate meshes require multiple precise refinement steps and thousands of elements. We implement and compare to current state-of-the-e thr--e- RL methods for AMR [28, 29, 30] that have been shown to work well dynamic tasks where shallow mesh refinement and coarsening is sufficient'. We observe that these methods struggle with finer static meshes, whereas ASMR yields stable and consistent refinements across tasks. To further evaluate our method's effectiveness, we compare it to the popular Zienkiewicz-Zhu Error Estimator (ZZ Error) estimate and traditional AMR heuristic requiring oracle estimates. ASMR not only outperforms both learned and traditional methods that lack oracle data but also achieves performance comparable oracle-based heuristics. Furthermore, ASMR is 2 100 times faster than computing the uniform mesh on which the oracle information is based, and demonstrates robust generalization capabilities across different domains and initial conditions. We conduct series of ablations to show which parts of the approach make it uniquely effective, finding that spatial rewards per agent are preferable to a shared global reward signal for all agents.
To summarize our contributions, we (1) propose a novel Markov Decision Process (MDP) formulation that naturally integrates local rewards for swarms where agents may split into new agents time; (2) combine this formulation with MPNs and a novel spatial reward formulation to reliable and efficiently scale leamedd AMR static meshes with thousands of elements multiple levels of refinement; (3) showcase our approach's 's effectiveness on a suite of PDEs with challenging refinement
'ee publish the first codebase on RL for AMR, including all methods and tasks presented in this paper, to facilitate research in this direction. The code is available https: : // //tpthgttttttttt/// www//oco/////e/eeee http//ikkasFFeymuth//// ASMR.

---

problems, Our method surpasses state-of-the-arr RL methods and the popular ZZ Error heuristic. and achieves refinement quality comparable to oracle-based AMR strategies, all without requiring costly emor estimates during inference.
2 Related Work
Learned Physics Simulation. considerable body of work deals with directly learning simulate physical systems with neural networks. These approaches typically learn from data generated by some underlying ground-truth simulator and train the network predict the (change in) quantities of interest during a simulation. Such learned physics simulators are fully differentiable and often orders of magnitude faster than their classical counterparts [35, 38]. lending them to use cases such as Inverse Design [47, 48, 49]. Researchers have developed simulators based on simple corwadd networks [50, 51] and Convolutional Neural Networks [52, 53, 54, 55, 56, 57, 58, 59, 60]. Closely related to our method are Graph Network Simulators (GNSs) [33, 38, 61, 62, 63, 64, 39, 65], which utilize GNNs to encode physical problems as a graph on which to compute quantities of interest per node. Here, a recent method [66] jointly learns a GNS and an AMR strategy on mesh edges to allow for simulation on different resolutions.
Physics-Informed Neural Networks [67, 68, 69] are mesh-free methods designed to directly train neural networks to satisfy the governing equations of a physical system. They share the goal of using deep neural networks to solve PDEs, yet differ in their approach in that they directly approximate the equations rather than providing a mesh for a classical solver. Thus, AMR strategies provide a more robust, flexible, and risk-averse approach to solving complex physics problems that demand high precision and accuracy [11, 70, 13). As Physics-Ifformed Neural Networks also operate on geometric domains, they have been extended to GNN architectures [71, 72, 73]. In this work, we do not learn to solve a system of equations directly, but rather propose an efficient mesh refinement for a classical solver.
Supervised Learning for AMR. Applications of supervised learning for AMR include directly calculating an eπгоr per mesh element with Multilayer Perceptron (MLP) [11] and predicting mesh densities from domain images [70]. Additionally, recurrent networks have been used to find optimal marking strategies for second-orde elliptical PDEs [74]. Another body of work speeds up the computation of Dual Weighted Residual [75, 76] error estimators by substituting expensive parts of the procedure with neural networks. Here, recent methods [77, 78] consider learning metric tensor from solution information that can then be used in existing refinement procedures [24]. Other approaches [79, 13] employ neural networks to solve the strong form of the adjoint problem and use hand-crafted features to compute error estimates directly. We instead leverage the fact that RL can optimize non-differentiable rewards, enabling us to directly learn refinement strategy in an iterative manner instead of learning error estimators or other specific facets of AMR.
Reinforcement Learning for AMR. Current research in Reinforcement Learning for AMR involves various approaches. such as optimizing the positions of mesh elements [80], predicting a global thresh- old for heuristic-based refinement using existing eтror estimates [81]. and generating quadrilateral meshes by iteratively extracting elements from the problem domain [82].
We instead directly manipulate the mesh elements themselves. This approach presents a unique challenge in that the size of the observation and action spaces is constantly changing during refinement. Existing methods [28, 29, 30] derive their observation spaces from the mesh geometry and the solution computed on the mesh. These methods include mesh coarsening operations that are required for time-dependent problems. While our method can easily be extended to coarsening and time-dependent problems, this work instead considers static meshes and mostly stationary problems due to their prevalence in engineering [43, 44]. Here, the challenge lies in finding multiple levels of accurate refinements rather than shallow or local time- -ieeddepeddent refinement and coarsening. The earliest of these methods [28] treats the entire mesh as an action and observation space for a Single Agent, which uses a GNN-based policy to provide a categorical action that selects single element for refnement. Single Agent demands solving the system of equations after each refinement step, significantly increasing inference time while reducing the amount of information per environment sample. Another approach [30] iteratively selects random element during training and uses an MI. policy to determine its marking based on local and global features. During inference, the method performs a Sweep for all mesh elements in parallel.
generally designed for non-stationary PDEs and thus
3

---

Laplace
Poisson
Stokes Flow
Linear Elasticity
Heat Diffusion
Exemplary ASMR refinements. The heatmaps represent the normalized quantities of interest. ASMR provides complex and accurate refinements for different tasks. From left to right: Laplace's 's equation requires a homogeneous refinement near the source at the inner boundary. For Poisson's 's equation, multi-moda load function causes multiple distinct regions of interest. The Stokes flow uses more complex shape functions and requires high precision near the inlet on the left and the rhomboid holes. For the linear elasticity task, we consider both the deformation and the resulting stress as quantities interest. The heat diffusion task requires accurate refinements on the path of the moving heat source predict accurate solution of the final step.
Figure 2:
This procedure speeds up inference but causes misalignment in the environment transition between training and inference [29]. Additionally, during training, the agent randomly assigned new element after each action, leading to high variance in the state transitions.
method are Value Decomposition Graph Networks (VDGN) [29] which frame AMR as a cooperative multi-agent problem by setting maximum refinement depth and number of agents. VDGN employs Value Decomposition Network [83] circumvent the posthumous credit assignment problem [84] of vanishing agents. Though theoretically efficient in training and inference, the method's 'ss performance depends on the quality of the value decomposition, which becomes more difficult for larger meshes. In summary, existing RL methods do not utilize the spatial nature of AMR and thus only scale to either simple or comparatively shallow refinements. We instead formulate AMR as Swarm Reinforcement Learning problem with spatial rewards, naturally integrating changing observation and action spaces while also providing strong feedback signal to all agents.
Most similar
Adaptive Swarm Mesh Refinement
In the following, we introduce the individual components of ASMR, including our novel Adaptive Swarm Markov Decision Process (ASMDP) and spatial reward function. We consider each element Of E 2 of a mesh 52' to be an agent in a swarm system. The agent's state is its position in the mesh, as well as boundary conditions and other PDE-dependent quantities. The agent's 's observation consists of a local view of a graph Go where each node represents mesh element and each edge the neighborhood of two elements. We train a simple MPN-based policy ( Ga+ that computes joint action vector a E A for each mesh element by passing messages along the observation graph, as detailed in Appendix with the refined mesh for given number of steps. Since our policy uses a GNN, it is equivariant to permutation and can handle varying numbers of agents by construction. Figure provides schematic overview of our method.
The action vector used for refinement, and the process is repeated
Adaptive Swarm Markov Decision Process. We adapt the SwarMDP framework [31, 32] to incorporate action and observation spaces of changing size as necessary for AMR, agent-wise rewards and mappings between agents over time. The resulting framework conceptually simpler than, e.g., a decentralized partially obeservable MDP with dummy states [29], makes the permutation- cquivariance of the agents explicit and naturally integrates both the agent-dependent reward and the mapping between agents. Formally, we define an Adaptive Swarm Markov Decision Process (ASMDP) tuple (S, 0, A, T, M). Here, $ is the state space of the system, O is the space of observations for this state space, and & is the action space for the system of agents. Let S c S. ON C 0, N C A. denote the subsets of the state, observation, and action spaces with exactly N agents. The transition function T: SNN A + S maps to new system state with a potentially different number of agents, and r: SN A + R is per-agent reward function. The observation graph of the agents is calculated from their states via the observation function S. + ON, To

---

accommodate changing numbers of agents throughout an episode, we define an agent mapping M' E [0. 1| with Σ,M; step t. Each entry M describes whether agent i at step & progresses into agent j at step + 1. The influence of each agent step & on the reward, in terms of all successor agents it is responsible for up to step t + k, can then be computed via the matrix multiplication M == ... ...M+tk-1,
for all j = L..... K that specifies how agents evolve time
The usual objective in RL is to find a policy π ON x A + [0, 1] that maximizes the return, i.e., the expected discounted cumulative future reward J := (x((((()) [E 3 r(s a(**)] for discount factor y and scalar reward r(s'+h, a(**) at step + k. Adapting this to varying numbers of agents within a single episode, as necessary for e.g., the refinement of mesh elements, yields
J == (all(())
"r a *(++));;
for agent at step t. Intuitively, this return represents the discounted sum of rewards of all agents that agent i is responsible for. We set Vi(s*) r(s', a²); + M3 MY Vj (ilss,aa)) a²)) with a ~ ((((s) for training the value function and derive the targets for Q-functions analogously.
Agents and Observations. Given a domain Ω and a mesh Ω == (2) S 2 Ü, 0 = 2). we view each mesh element Q as agent. Each element's action space comprises binary decision to mark it for refinement. These markings are provided to a remesher, which refines all marked elements, yielding a finer mesh 2 = {2] "}-- Here, 2/+1 == O for no refinement and 2f41 & S with Ü, 22 if 2 is refined. The remesher may also refine unmarked elements to assert a conforming solution [41], i.e., make sure that elements of the mesh align with each other at the boundaries and interfaces to ensure continuity of solution variables between adjacent elements. We define the mapping for an agent to its successor agents as the indicator function Mi 1( S st). In other words, an agent maps all future agents that it spawns, or equivalently, an element is responsible for all sub-elements that it refines into over time. While we focus on mesh refinement in this work, this mapping can be extended to coarsening by setting, e.g..
My = I(2)+1 C 2)
(I(S)) _ 2 (Ex I(22 Ç 1))))).
an observation graph Gy- = G (V,E,y, Xz), which is a bidirectional directed graph with mesh elements as nodes V and their neighborhood relation as edges& & V x V. Node and edge features of dimensions dy and de are given as Xv V - Bd and Xx : E + Rde, Further details can be found in Appendix B.
For encoding the observations,
Reward. A good refinement strategy trades off the accuracy of the solution of the mesh L2 with its total number of elements 12 E .2'.. We define an error per element the difference in the solution of this element compared to a solution using a fine-grained reference mesh J2 [28]. We consider Q to be optimal, but very slow to compute due to a large number of elements. However, we only require the reference mesh Ω for the reward calculation, not during inference. For each element S2 we then integrate over the evaluated differences of all midpoints po:: E Q reference elements D that fall into it, scaling each by the area Area(22;,) of its respective element. This procedure results in an error estimate
cir
Σ Area(( |ua- (po;:) uug (poz,)
(2)
R CR
where ue- denotes the solution on the fine mesh and uig the solution on the current mesh. We note that this error estimate be efficiently calculated using k-d tree [85] and that it is gen- erally applicable for a large range of PDEs. Problem-specific error estimates may be used in- stead include domain knowledge. To get an error estimate that is consistent across different geometries, we normalize the error with the total error of the elements of the initial mesh Ω", i.c., err(st)) = eir( (e0l//ngeen Engeor efr()))). We then formulate local reward per element as
(EML-I) -1).
r(2)) = (2 erT(N)) EM.yeme er((( - EM, M
Σ
(3)
where a is hyperparameter that penalizes adding new elements.
This reward function evaluates whether a refinement decreases the overall error by enough to justify the extra resources required, with reward of 0 for unrefined elements. Thus, the reward maximizes
5

---

SCCC
0.1
a = 0.05
C' = 0.03
a = 0.01
a 0.005
Figure 3: Final refinements of ASMR on Heat Diffusion problem for different values of the element penalty a. All refinements focus on the relevant parts of the problem, and lower element penalties lead to more fine-grained meshes.
error reduction, rather than simply encouraging a refinement of areas with a high existing error regardless of the resulting mesh improvement. Further, incorporating novel area scaling term encourages the policy to focus on smaller elements with high potential reduction in error rather than larger elements with low average error reduction, up to some threshold depending on the element penalty a. We find that the combination of local formulation and the area scaling term allows for simpler credit assignment for the RL agents, as ensures that each agent gets rewarded for its own actions and that rewards of elements of different sizes are on the same scale. Similarly, the area scaling term of the reward effectively cancels out the area of the integration points in Equation causing the policies optimized on this reward to implicitly minimize the maximum remaining error of the mesh, while also making sure that the mean error stays sufficiently low. We compare this to directly minimizing the maximum error in Appendix C.2.
Since the effects of mesh refinement be non-local for elliptical PDEs, we optimize the average of the local and global returns, i.c.
J - 2
where Jí is the return of agent at step as shown in Equation 1, and J is the global return calculated using the average reward r = +/2 Σ; Ty. In multi-quantity systems of equations, it is important for the mesh to be suitable for all the quantities of interest. For this, we calculate individual errors ( for each solution dimension and then use a norm or convex sum of these as the overall depending
error, depending on the application.
Experiments
Setup. All learned methods are trained on 100 PDEs and their corresponding initial and reference meshes Ω2, O to limit the number of required reference meshes during training. We experiment with 10 different target mesh resolutions per method to produce wide range of solutions, as detailed in Appendix F.3. We repeat each experiment for 10 random seeds and always report the average performance on 100 randomly sampled but fixed evaluation PDEs. These PDEs are disjoint from the training PDEs, and both sets of PDEs consist of randomly sampled domains as well as boundary and initial conditions as detailed below. Details on the setup and the computational budget for our experiments are provided in Appendix C.I. The reference mesh Ω² is created by uniformly refining the initial mesh 6 times. An environment episode consists of drawing one of the 100 training PDE without replacement, and iteratively refining the coarse initial mesh Ω2 a total of T = 6 times unless mentioned otherwise. Since the maximum number of clements scales exponentially with the refinement depth, we additionally evaluate simpler task setup with T = 4 refinements to roughly replicate the task complexity of existing work [28, 63, 29]. We experiment with Deep Q-Network off-policy and Proximal Policy Optimization (PPO) [88] with discrete actions
(DQN) [86, 87] as an on-policy RL algorithm for all RL-based methods.
We evaluate mesh quality by calculating the squared error at each point in the high-resolution reference Ω*, i.c., as Ex,, en' Area( (un- (po:.) ugr (pp:)) ))² and normalize the resulting value by that of the initial mesh for comparability across PDEs. This metric captures both the maximum localized errors by punishing outliers, and the overall error across the domain. We evaluate both the mean error and an approximation of the maximum error over the mesh as additional metries in Appendix D.6. Appendix F.1 lists all further algorithm and neural network hyperparameters.

---

Graph Features. The features X, of each node v E V consist of the environment timestep, the element area, the distance to the closest boundary, and the mean and standard deviation of the solution on the element's & vertices. Edge features X. are defined as Euclidean distances between element midpoints. We omit absolute positions to ensure that the observations are equivariant under the Fuclidean group [38, 37] to utilize the underlying symmetry of the task. We use additional task-dependent node features for some considered systems of equations, as described in Appendix B.
Systems of Equations We experiment on various 2D elliptical PDEs, namely the Laplace equation, the Poisson equation, a Stokes flow task, a linear elasticity example, and a non stationary heat diffusion equation. The domains are L-shapes, rectangles with square hole or multiple rhomboid holes, and convex polygons. Figure 2 shows exemplary ASMR refinements on all tasks and briefly explains the challenge associated with each task. The PDEs and the FEM are implemented using scikit-fem [89], and we use conforming triangular meshes and linear elements unless mentioned otherwise. The code provides OpenAl gym [40] environments for all tasks. We define the systems of equations and their specific features in Appendix B.
Baselines. We adapt several recent RL methods [28, 29, 30] that were originally designed for non-stationary AMR as baselines for our application focusing on stationary refinements. We use our error estimates as the basis of all reward calculations for comparability but otherwise calculate the rewards as described in the respective papers. Single Agent [28] predicts categorical action over the mesh to mark the next element for refinement. Sweep [30] trains a single-agent policy by randomly sampling an element on the mesh and deciding its refinement based on local features and global resource budget. During inference, each timestep consists of a sweep over the full mesh that may mark each element. Finally, VDGN [29] estimates global Q-function as the sum of agent-wise local Q-functions. As the PPO version of VDGN has no Q-Function, we decompose the value function as the sum of value functions of the individual elements, yielding a VDGN-like baseline in the case of the PPO version. We use MPN policy for Single Agent and VDGN, while Sweep utilizes simple MLP. Hyperparameters and further details are provided in Appendix F.2
We also compare to traditional urror-based Oracle Error Heuristic [90. 91, 30J. Given re- finement threshold 0, the Oracle Error Heuristic iteratively refines all elements D for which e > 8 max, err(S))) As we are also interested in the reduction of the maximum error, we analogously define the Maximum Oracle Error Heuristic, which uses the maximum error per element max Cnt [M2- (por.) ugr (poz) as surrogate error estimate. Note that these base- lines require the fine ine-grained reference mesh Ω which is expensive to compute and thus usually unavailable during inference. As a substitute, we consider the commonly used ZZ Error, which uses the superconvergent patch recovery process to estimate an eттor per mesh element [21]. Similar to the Oracle Error Heuristic, these estimates are combined with a refinement threshold 0 to iteratively refine the mesh. The 77. Error generally produces smooth error estimates as the recovery process requires averaging over neighboring mesh elements, which can in some cases lead to more coherent refinements when compared to the Oracle Error Heuristic. The hcuristics act on local element information and greedily refine elements with a high error rather than elements for which refinement would lead to a high reduction in error. As such, they may select sub-optimal refinements for globally propagating errors, which a well-known issue for elliptic PDEs [92, 30J. In contrast, RL methods learn to directly maximize the decrease in error, allowing them to find long-term strategies that also take the local receptive fields of the individual agents into account.
We conduct a series of ablation experiments to determine which parts of ASMR make it uniquely effective. We look at both the area scaling and the spatial decomposition of the reward in Equation 3 and ablate different node features and the number of training PDEs that are used. Additionally, we consider an alternate reward formulation that uses the maximum error per element instead of its average detailed in Appendix C.2. Due to their importance for practical applications, we further experiment with both generalization to unseen and larger domains and the improvements in runtime for ASMR compared to uniform refinement.
Additional Experiments.
Results
Quantitative Results. We visualize the mesh quality quantitatively with Pareto plot of the number of elements and the remaining error. We plot one point per trained policy, which represents the interquartile mean [93] of this policy when evaluated on 100 evaluation environments. We further
7

---

ASMR (Ours)
Single Agent
VDGN-like (PPO)
Sweap
Heuristic (Oracle Error)
Houristic (Max. Oracle Er.)
Heuristic (ZZ Error)
Uniform Refinement
Laplace's Equation 4 Refinements
Laplace's Equation 6 Refinements
10a
10
Squared Error
10"
10
10-2
10
10-1
10-3
0.0
0.2
0.4
0.6
0.8
1.0
1.2 0.0 2.0
4,0
6.0
8.0
Elements (x10")
Elements (x10")
Figure 4: Pareto plot of normalized squared errors and number of final mesh elements for Laplace's Equation. (Left) For only 4 refinement steps, all learned methods perform well and significantly improve upon uniform refinement. (Right) Scaling to refinement steps, the learned baselines become less stable and in some cases fail to provide refinements that are better than uniform. In contrast, ASMR consistently provides high-quality refinements and is on par with or better than the heuristics in both cases.
provide a log-log quadratic regression over the aggregated results of each method as a general trend-line. To enhance visibility and focus typical behavior, exclude sporadic outliers from the baseline methods that produce degenerate meshes with an excessively high number of elements. For all learned methods, we experiment with both PPO and DQN the RL backbone on the Poisson equation in Appendix D. All learned methods, including ASMR yield better results with PPO, indicating that an on-policy algorithm is favorable when dealing with action and observation spaces of varying size. Similarly, we compare the Graph Attention Network (GAT)) like [94] architecture proposed by VDGN [29] MPNs in Appendix D.2, finding that ASMR works well for both architectures, while VDGN performs better with MPNs. We consequently use PPO and MPNs in all other experiments. Appendix D.3 compares the ZZ Error estimator for different initial refinement levels. As the results show that sufficiently fine initial mesh is important, we start each refinement procedure for the ZZ Error Heuristic with two uniform refinements. We note that this initial tuning prevents coarse refinements and is not needed for our method.
Using these results, Figure 4 compares the different approaches on Laplace's equation. The left side of Figure 4 shows that all methods work in simple setup on par with experiments from previous work. Here, 4 refinement steps are used for all methods except for Single Agent, which instead refines 4 times fewer elements. On the right side, scaling to 6 refinement steps and significantly more elements, only ASMR effectively handles larger instances while learned methods falter. Notably, ASMR also outperforms the Oracle, Maximum Oracle, and ZZ Error Heuristics. These results demonstrate the effectiveness of our Swarm RL framework for learning non-greedy refinmentt strategies for static meshes. Specifically. ASMR refines elements with a high potential for error reduction over the heuristics' strategy of targeting elements with high error. Figure 5 provides results on the remaining tasks. Appendix D.6 presents additional results using mean error and smooth version of a maximum error metric. ASMR clearly outperforms the learned baselines on all tasks and is generally competitive with or better than the Heuristics. Both heuristics improve over the RL. methods on the Stokes flow task, likely because the task requires high precision for both the inlet and on inner boundaries near regions of high flow velocity.
Qualitative Results. Figure 2 shows refinements of ASMR on randomly sampled systems of equations for all considered tasks. The refinement strategy adapts to the given task, providing an efficient trade-off between simulation accuracy and the number of elements used. Figure visualizes the refinements of ASMR on randomly sampled heat diffusion problem. ASMR refines based on the element penalty a, yet always focuses on the heat source and its path. Appendix G. provides additional ASMR visualizations for all tasks, and Appendix G.2 visualizes all methods on Poisson's 's equation to showcase common refinement behaviors. Appendix G.3 presents the iterative marking procedure of our approach on exemplary Poisson problem.
8

---

ASMR (Ours)
Single Agent
VDGN-like (PPO)
Sweep
Heuristic (Oracle Error)
Houristic (Max. Oraele Err))
Heuristic (ZZ Error)
Uniform Refinement
Poisson's 's Equation
Stokes Flow
10
10
10
Squared Error
10
1
10
10 3
10 3
10
10 0
10
0.0
2.0
4.0
6.0
3.0
0.0
2.0
4.0
6.0
8.0
10.0
Linear Elasticity
Heat Diffusion
16
100
10-1
Squared Eroo
10
10-2
10
2
10 8
10 0
10-4
10-
10-3
0.0
2,0
4.0
6.0
8.0
10.0
12.0
0.0 1.0
2.0
3.0
4,0
5.0
6.0
Elements (x 10
Elements (x (x10")
Figure 5: Pareto plot of normalized squared errors and number of final mesh elements across different tasks. All methods generally work well for relatively small instances, but Single Agent, VDGN- -ike and Sweep break down for larger meshes. Our method uniquely scales meshes thousands of elements and consistently outperooms these learned baselines on all tasks and while performing on par with or better than the Orucle, Maximum Oracle and ZZ. Error Heuristics most cases.
Ablations. The reward proposed in Equation 3 combines an area scaling per element with spatial allocation of the decrease in error the individual mesh elements. Figure investigates these decisions. We find that combining both features is uniquely responsible for the effectiveness of our method, suggesting that the spatial reward's limited expressiveness for small elements is compensated by area scaling. However, the area scaling can only be leveraged if it is allocated to individual mesh elements, as may introduce excessive reward noise on the full mesh. The maximum reward variant of Appendix C.2 explicitly minimizes the maximum error of the mesh, resulting to refinements of similar quality than those created by ASMR using the reward in Equation We thus use the latter as it simplifies the error estimate in the reward function and better aligns with existing work.
We evaluate the effect of different parameters for the target mesh resolution in Appendix D.4, finding that ASMR provides meshes with considerably more consistent numbers of elements for given target resolution than the other learned methods. Additional ablations in Appendix D.5 show that 100 training PDEs are sufficient and that adding absolute positions in the node features is detrimental, while providing solution information and load function evaluations improves performance.
Generalization Capabilities and Runtime Experiments. Table compares the wall-clock time of ASMR trained on a variant of the Poisson task with that of the reference O showing that our method provides a speedup of more than factor 100 compared to computing uniform mesh for large domains. The evaluation uses load functions with 16 Gaussian modes and spiral-shaped of varying sizes for the same average initial element size. Appendix E. provides details for the training environments and the spiral- shral- evaluation domain, as well as results on larger domains and the associated improvement in runtime. These results includes an ASMR visualization of refinement of 20 x 20 spiral domain with more than 000 elements. Appendix E.2 additionally shows the exceptional generalization capabilities of ASMR across various domains and load functions for Poisson's 's equation on 1 x 1 domains. Appendix E.3 presents further runtime comparisons for all tasks and shows that ASMR provides a task-dependent speedup of factor 2 to 30 over uniform refinement.

---

ASMR (Area Scaling, Spatial Reward)
Arca Scaling, Global Reward
Maximum Roward Uniform Refinement
No Area Scaling, Spatial Reward
No Area Scaling, Global Reward
Linear Elasticity
Domain
Elements
Time[s]
Speedup[x]
10
x
4208
0.19
14.6
3 x
4185
0.23
29.6
Squared Error
10
4 x 4
4391
0.28
47.1
10
5120
0.35
67.3
x
5462
0.41
90.1
10-3
7 7
6515
0.51
114.8
8 x
7506
0.60
131.4
Table I: Elements and speedup versus uni- form refinement for achieving normalized squared error of 0.001 for different domain sizes on Poisson's 's Equation. The elements required for the error threshold grow slower than the domain size, indicating fewer ar- eas of significant error in larger domains. Thus, ASMR offers increasing speedups as the size of the domain increases.
0.0
2.0
4.0
6.0
S.
10.0
12.0
Elements (x10²)
Figure 6: Pareto plot of normalized squared errors and number of final mesh elements for the Linear Elasticity task for different reward ablations. ASMR benefits from the area scaling term of Equation 3 and spatial reward formulation. The maximum reward of Appendix C.2 performs similar to that of Equation 3.
The generalization capabilities in combination with the fast runtime of our method offer substantial advantages in practical engineering applications. A policy can be trained on small. cost-effective environments and then deployed on much larger and dynamically changing setups during inference. These generalization traits arise from the MPN architecture and the utilized observation graphs, which both lead to refinement strategies based on local element neighborhoods rather than global meshes.
6 Conclusion
We present novel Adaptive Mesh Refinement method that uses Swarm Reinforcement Learning to iteratively refine meshes for efficient solutions of Partial Differential Equations. Our approach, Adaptive Swarm Mesh Refinement (ASMR), treats each mesh element an agent and trains all agents under a shared policy using Graph Neural Networks and a novel per-agent reward formulation. ASMR gracefully scales to meshes with thousands of elements without requiring error estimate during inference. our experiments focused on static meshes, ASMR demonstrates strong performance in handling complex refinements. The method significantly outperforms both existing Reinforeement Learning-baa based approaches and traditional refinement strategies, achieving a mesh quality comparable to expensive cracle-based eтror heuristics. Once trained, the ASMR policy generalizes well to different forcing functions and significantly larger problem domains. In terms runtime, our method outperforms uniform refinements by up 30 times on domains similar in scale the training set, and by over 100 times in larger evaluation setups.
Broader Impact Our proposed Adaptive Mesh Refinement technique can positively impact various fields relying on computational modeling and simulation. By reducing simulation times while maintaining high precision, this technology enables researchers to explore wider range of scenarios. However, like any powerful tool, there are potential negative impacts, such as the development of advanced weapon models exploitation of resources.
Limitations and Future Work Our approach solves the partial differential equation after each refinement step, which requires a considerable amount of computation time. In future work, we will explore using Swarm RL for refinement strategies from the raw geometry and boundary conditions to further speed up our approach. We currently use relatively simple message passing networks for our policy, and want to optimize the network architecture to include, e.g., long-range message passing. Lastly, this work only considers 2D problems, static meshes with triangular elements, and comparatively simple domains. Here, we want to extend and modify our approach to quadrilateral meshes, time-dependent refinement and coarsening operations, and 3-dimensional domains.
10

---

Acknowledgments and Disclosure of Funding
NF was supported by the BMBF project Davis (Datengetricbene Vernezzung für die ingenicurtechnis- che Simulation). This work is also part of the DFG AI Resarch Unit 5339 regarding the combination of physsis-bbssee based simulation with Al-based methodologies for the fast maturation manufacturing processes. The financial support by German Research Foundation (DFG, Deutsche Forschungsge- meinschaft) is gratefully acknowledged. The authors acknowledge support by the state of Baden- Württemberg through bwHPC, well as the HoreKa supercomputer funded by the Ministry of Science, Research and the Arts Baden-Württemben Württemberg and by the German Federal Ministry of Educa- tion and Research.
References
[1] Susanne C Brenner and L Ridgway Scott. The mathematical theory of finite element methods, volume Springer, 2008.
[2] Junuthula Narasimha Reddy and David K Gartling. The finite element method in heat transfer and fluid dynamics. CRC press, 2010.
[3] Junuthula Narasimha Reddy. Introduction to the finite element method. McGraw-Hi Educa- tion, 2019.
[4] Robert Anderson, Julian Andrej. Andrew Barker, Jamie Bramwell, Jean-Sylvain Camier, Jakub Cerveny, Veselin Dobrev, Yohann Dudouit, Aaron Fisher, Tzanio Kolev, al. Mfem: A modular finite element methods library. Computers & Mathematics with Applications, 81:42-74, 2021.
[5] Tomasz Plewa, Timur Linde, V Gregory Weirs, et Adaptive mesh refinement-theon and applications. Springer, 2005.
[6] Weizhang Huang and Robert D Russell. Adaptive moving mesh methods, volume 174. Springer Science & Business Media, 2010.
[7] Krzysztof J Fidkowski and David L Darmofal. Review of nutput-based error estimation and mesh adaptation in computational fluid dynamics. AIAA journal, 49(4):673 694, 2011.
[8] Marsha Berger and Phillip Colella. Local adaptive mesh refinement for shock hydrodynamics. Journal of computational Physics, 82(1):64 84, 1989.
[9] Timothy J Baker. Mesh adaptation strategies for problems in fluid dynamics. Finite Elements in Analysis and Design, 25(3-4):243-271 1997.
[10] Raunak Borker, Daniel Huang, Sebastian Grimberg, Charbel Farhat, Philip Avery, and Jason Rabinovitch. Mesh adaptation framework for embedded boundary methods for computational fluid dynamics and fluid-structure interaction. International Journal for Numerical Methods in Fluids, 90(8):389 424, 2019.
[1]] Zheyan Zhang, Yongxing Wang, Peter Jimack, and He Wang. Meshingnet: A new mesh generation method based deep learning. In Computational Science-ICCS 2020: 20th International Conference, Amsterdam, The Netherlands, June 3-5, 2020, Proceedings. Part III 20, pages 186-198. Springer, 2020.
[12] Joseph Gregory Wallwork.
Mesh adaptation and adjoint methods for finite element coastal ocean modelling. PhD thesis, Imperial College London, 2021.
[13] Joseph Gregory Wallwork, Jingyi Lu, Mingrui Zhang, and Matthew D Piggott. E2n: Error estimation networks for goal-oriented mesh adaptation. arXiv preprint arXiv:2207.11333. 2022.
[14] M Ortiz and JJ Quigley Iv. Adaptive mesh refinement in strain localization problems. Computer Methods in Applied Mechanics and Engineering, (0(1-3):781-800 1991.
[15] Nikolas Provatas. Nigel Goldenfeld, and Jonathan Dantzig. Efficient computation of dendritic microstructures using adaptive mesh refinement. Physical Review Letters, 80(15):3308, 1998.
[16] Erwin Stein. Adaptive finite elements in linear and nonlinear solid and structural mechanics, volume 416. Springer Science & Business Media, 2007.

---

[17] Gaël Gibert, Benoit Prabel, Anthony Gravouil, and Clémentine Jacquemoud. A 3d automatic mesh refinement x-fem approach for fatigue crack propagation. Finite Elements in Analysis and Design, 157:21-37, 2019.
[18] Andrew J Cunningham, Adam Frank, Peggy Varnière, Sorin Mitran, and Thomas W Jones. Simulating magnetohydrodynnmice flow with constrained transport and adaptive mesh re- finement: algorithms and tests of the astrobear code. The Astrophysical Journal Supplement Series, 182(2):519, 2009.
[19] Greg L Bryan, Michael L Norman, Brian W O'Shea, Tom Abel, John H Wise, Matthew J Turk, Daniel R Reynolds, David C Collins, Peng Wang, Samuel W Skillman, et al. Enzo: An adaptive mesh refinement code for astrophysics. The Astrophysical Journal Supplement Series, 21 211(2):19, 2014.
[20] Thomas Guillet, Rüdiger Pakmor, Volker Springel, Praveen Chandrashekar, and Christian Klingenberg. High-order magnetohydrodynamies for astrophysics with an adaptive mesh refinement discontinuous galerkin scheme. Monthly Notices of the Royal Astronomical Society, 485(3):4209-42266 2019.
[21] Olgierd Cecil Zienkiewicz and Jian Zhong Zhu. The superconvergent patch recovery and a posteriori error estimates. part I: The recovery technique. International Journal for Numerical Methods in Engineering. 33(7):1331-1364 1992.
[22] Arup Mukherjee. An adaptive finite element code for elliptic boundary value problems in three dimensions with applications in numerical relativity. The Pennsylvania State University, 1996.
[23] Eisuke Kita and Norio Kamiya. Error estimation and adaptive mesh refinement in boundary element method, an overview. Engineering Analysis with Boundary Elements, 25(7):479 2001.
[24] Masayuki Yano and David L Darmofal. An optimization- based framework for anisotropic simplex mesh adaptation. Journal of Computational Physics, 231 231(22)7626-764) 2012.
[25] Wolfgang Bangerth and Rolf Rannacher. Adaptive Finite Element Methods for Differential Equations. Birkhäuser, 2013.
[26] Jakub Cerveny, Veselin Dobrev, and Tzanio Kolev. Nonconforming mesh refinement for high-order finite elements. SIAM Journal on Scientific Computing, C134))336,-332,, 2019.
[27] Richard S Sutton and Andrew G Barto. Reinforcement learning: An introduction. MIT press, 2018.
[28] Jiachen Yang, Tarik Dzanic, Brenden Petersen, Jun Kudo, Ketan Mittal, Vladimir Tomov, Jean-Sylvain Camier, Tuo Zhao, Hongyuan Zha, Tzanio Kolev, Robert Anderson, and Daniel Faissol. Reinforcement learning for adaptive mesh refinement. 26th International Conference on Artificial Intelligence and Statistics (AISTATS), 2023.
[29] Jiachen Yang, Ketan Mittal, Tarik Dzanic, Socratis Petrides, Brendan Keith, Brenden Petersen, Daniel Faissol, and Robert Anderson. Multi-agent reinforcement learning for adaptive mesh refinement. 22nd International Conference on Autonomous Agents and Multiagent Systems (AAMAS), 2023.
[30] Corbin Foucart, Aaron Charous, and Pierre FJ Lermusiaux. Deep reinforcement learning for adaptive mesh refinement, arXiv preprini arXiv:2209. 12351, 2022.
[31] Adrian Šošić, Wasiur R KhudaBukhsh, Abdelhak M Zoubir, and Heinz Koeppl. Inverse rein- forcement learning in swarm systems. In Proceedings of the 16th Conference on Autonomous Agents and MultiAgem Systems, pages 1413- 1421, 2017.
[32] Maximilian Hüttenrauch, Šošié Adrian, and Gerhard Neumann. Deep reinforcement learning for swarm systems. Journal of Machine Learning Research, 20(54):1-31, 2019.
[33] Alvaro Sanzaez-Gonzalez. Jonathan Godwin, Tobias Pfaff, Rex Ying, Jure Leskovec, and Peter Battaglia. Learning to simulate complex physics with graph networks. In Proceedings of the 37th International Conference on Machine Learning, pages 8468. PMLR, 2020.
[34] Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfar- dini. The graph neural network model. IEFE Transactions on Neural Networks, 20(1):61-80, 2009.
12

---

[35] Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Maachez-Gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv//80.0//61,, 2018.
[36] Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. IEEE transactions neural networks and learning systems, 32(1):4-24, 2020.
[37] Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. arXiv preprint arXiv:2104. 13478, 2021.
[38] Tobias Pfaff, Meire Fortunato, Alvaro Gonzaez- and Peter W. Battaglia. Learn- ing mesh-based simulation with graph networks. In International Conference on Learning Representations, 2021.
[39] Jonas Linkerhägner, Niklas Freymuth, Paul Maria Scheikl, Franziska Mathis Ullrich and Gerhard Neumann. Grounding graph network simulators using physical sensor observations. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
[40] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym. arXiv preprint arXiv: 01540, 2016.
[41] Douglas N Amold, Arup Mukherjee, and Luc Pouly. Locally adapted tetrahedral meshes using bisection. SIAM Journal on Scientific Computing, 22(2):431-448, 2000.
[42] Rob Stevenson. The completion locally refined simplicial partitions created by bisection. Muthematics of computation,
2008.
[43] Anand Nagarajan and Soheil Soghrati. Conforming interface structured adaptive mesh refinement: 3d algorithm and implementation. Computational Mechanics, 62:1213-1238, 2018.
[44] K Ho-Le. Finite element mesh generation methods: a review and classification, Computer- aided design, 20(1):27-38, 1988.
[45] Mark T Jones and Paul E Plassmann. Adaptive refinement of unstructured finite-element meshes. Finite Elements in Analysis and Design, 25(1-2):41-60, 1997.
[46] Christophe Geuzaine and Jean- -rançoos Remacle. Gmsh: A 3-d finite element mesh generator with built-in pre-and prst-procssiing facilities. International journal for numerical methods in engineering, 79(1 13(11):130--1311 2009.
[47] Pierre Baqué, Edoardo Remelli, François Fleuret, and Pascal Fua. Geodesic convolutional shape optimization. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmåsssan Stockholm, Sweden, July 10- 15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 481-490 PMLR, 2018
[48] Nikita Durasov, Artem Lukoyanov, Jonathan Donier, and Pascal Fua. Debosh: Deep bayesian shape optimization. arXiv preprint arXiv:2109, 2021.
[49] Kelsey R Allen, Tatiana Lopez-Guevara, Kimberly Stachenfeld, Alvaro Sanchez-Gone Gonzalez, Peter Battaglia, Jessica Hamrick, and Tobias Pfaff. Physical design using differentiable learned simulators. arXiv preprint arXiv:2202.0077 007288 2022.
[50] Kiwon Um. Xiangyu Hu, and Nils Thuerey. Liquid splash modeling with neural networks. Computer Graphics Forum, 37(8):171-182, 2018.
[51] Steffen Wiewel, Moritz Becher, and Nils Thuerey. Latent Space Physics: Towards Learning the Temporal Evolution of Fluid Flow. Computer Graphics Forum, 2019.
[52] Xiaoxiao Guo, Wei Li, and Francesco lorio. Convolutional neural networks for steady flow approximation. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, page 48 481-490, New York, NY, USA, 2016. Association for Computing Machinery.
[53] Mengyu Chu and Nils Thuerey. Data-driven synthesis smoke flows with enn-based feature descriptors. ACM Trans. Graph., 36(4), jul 2017.
13

---

[54] You Xie, Erik Franz, Mengyu Chu, and Nils Thuerey. tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow. ACM Transactions on Graphics (TOG), 37(4))95, 2018.
[55] Yao Zhang, Woong Je Sung, and Dimitri N. Mavris. Application of convolutional neural network predict airfoil lift coefficient. In 2018 A/AN/SCCE/AHS/SSC Structures, Structural Dynamics, and Materials Conference, 2018.
[56] Clemens Zimmerling, Daniel Trippe, Benedikt Fengler, and Luise Kärger. An approach for rapid prediction of textile draping results for variable composite component geometries using deep neural networks. AIP Conference Proceedings, 2113(1), 07 2019. 020007.
[57] Byungsoo Kim, Vinicius C. Azevedo, Nils Thuerey, Theodore Kim, Markus Gross, and Barbara Solenthaler. Deep Fluids: A Generative Network for Parameterized Fluid Simulations. Computer Graphics Forum (Proc. Eurographics), 38(2). 2019.
[58] Saakaar Bhatnagar, Yaser Afshar, Shaowu Pan, Karthik Duraisamy, and Shailendra Kaushik. Prediction of aerodynamic flow fields using convolutional neural networks. Computational Mechanics, 64(2):525-4 54055 jun 2019.
[59] Benjamin Ummenhofer, Lukas Prantl, Nils Thuerey, and Vladlen Koltun. Lagrangian fluid simulation with continuous convolutions. In International Conference on Learning Represen- tations, 2020.
[60] Clemens Zimmerling, Christian Poppe, Oliver Stein, and Luise Kärger. Optimisation of manufacturing process parameters for variable component geometries using reinforcement learning. Materials & Design, 10423, 2022.
[61] Zehang Weng, Fabian Paus, Anastasiaa Varava, Hang Yin, Tamim Asfour, and Danica Kragic. Graph-based spec-fpecific prediction models for interactions between deformable and rigid objects. In 2021 JEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 5741-5748, 2021.
[62] Xu Han, Han Gao, Tobias Pffaf, Jian-Xun Wang, and Li-Ping Liu. Predicting physics in mesh-reduced space with temporal attention. CoRR, abs/2201.001 2022.
[63] Meire Fortunato, Tobias Pfaff, Peter Wimsserger, Alexander Pritzel, and Peter Battaglia. Multiscale meshgraphnets. In ICMI. 2022 2nd Al for Science Workshop, 2022.
[64] Kelsey R Allen, Tatiana Lopez Guevara, Yulia Rubanova, Kimberly Stachenfeld, Alvaro Sanchez-Gonzaler Peter Battaglia, and Tobias Pfaff. Graph network simulators can learn discontinuous, rigid contact dynamics. Conference on Robot Learning (CoRL)., 2022.
[65] Kelsey R Allen, Yulia Rubanova, Tatiana LueezaG William Whitney, Alvaro Sanchez- Gonzalez, Peter Battaglia, and Tobias Pfaff. Learning rigid dynamics with face interaction graph networks. The Eleventh International Conference on Learning Representations (ICLR), 2023.
[66] Tailin Wu, Takashi Maruyama, Qingqing Zhao, Gordon Wetzstein, and Jure Leskovec. Learn- ing controllable adaptive simulation for multi-resolution physics. The Eleventh International Conference on Learning Representations, 2023.
[67] Maziar Raissi, Paris Perdikaris, and George E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. Journal of Computational physics, 378:686-701 2019.
[68] Shengze Cai, Zhiping Mao, Zhicheng Wang, Minglang Yin, and George Em Karniadakis. Physies-informed neural networks (pinns) for fluid mechanies: A review. Acta Mechanica Sinica, 37(12):177--1338 2021.
[69] Tobias Würth, Constantin KrauB, Clemens Zimmerling, and Luise Kärger. neural networks for data-free surrogate modelling and engineering optimization-an example from composite manufacturing. Materials & Design, 231:1 1034, 2023.
[70] Keefe Huang, Moritz Krügener, Alistair Brown, Friedrich Menhorn, Hans-Joachim Bungartz, and Dirk Hartmann. Machine learning based optimal mesh generation in computational fluid dynamics. arXiv preprint arXiv:2102.1200
2021.
[71] Balthazar Donon, Zhengying Liu, Wenzhuo LIU, Isabelle Guyon, Antoine Marot, and Marc Schoenauer. Deep statistical solvers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan,
14

---

and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 7910-7921. Curran Associates, Inc., 2020.
[72] Han Gao, Matthew J Zahr, and Jian-Xun Wang. Physics-informed graph neural galerkin networks: A unified framework for solving pde-govemned forward and inverse problems. Computer Methods in Applied Mechanics and Engineering, 390:1 14502, 2022.
[73] Masanobu Horie and NAOTO MITSUME. Physics-embedded neural networks: Graph neural pde solvers with mixed boundary conditions. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho. and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 23218-22222 2-2229.. Curran Associates, Inc., 2022.
[74] Jan Bohn and Michael Feischl. Recurrent neural networks as optimal mesh refinement strategies. Computers & Mathematics with Applications, 97:61-76, 2021.
[75] Roland Becker and Rolf Rannacher. Weighted a posteriori error control in FE methods. IWR, 1996.
[76] Roland Becker and Rolf Rannacher. An optimal control approach to a posteriori error estima- tion in finite element methods. Acta numerica, 10:1-102, 2001.
[77] Krzysztof J Fidkowski and Guodong Chen. Metric-based, gral-ore mesh adaptation using
machine learning. Journal of Computational Physics, 426: 2021.
[78] Guodong Chen and Krzysztof J Fidkowski. Output-based adaptive aerodynamic simulations using convolutional neural networks. Computers & Fluids, 223:104947, 2021.
[79] Julian Roth, Max Schröder, and Thomas Wick. Neural network guided adjoint computations in dual weighted residual error estimation. SN Applied Sciences, 4(2):62, 2022.
[80] Filipe De Avila Belbute-Peres, Thomas Economon, and Zico Kolter. Combining differentiable pde solvers and graph neural networks for fluid flow prediction. In international conference on machine learning, pages 2402-2411. PMLR, 2020.
[81] Andrew Gillette, Brendan Keith, and Socratis Petrides. Learning robust marking policies for adaptive mesh refinement. arXiv preprint arXiv:2207.0m 066399, 2022.
[82] Jie Pan. Jingwei Huang, Gengdong Cheng, and Yong Zeng. Reinforcement learning for automatic quadrilateral mesh generation: A soft actor-critic approach. Neural Networks, 157:288-304 2023.
[83] Peter Sunehag. Guy Lever, Audrunas Gruslys. Wojciech Marian Czarnecki, Vinicius Zambaldi, Max Jaderberg, Mare Lanctot, Nicolas Sonnerat, Joel Z Leibo, Karl Tuyls, et al. dalue-decomposition networks for cooperative multi-agent learning. arXiv preprint arXiv:/706.005 052966 2017.
[84] Andrew Cohen, Ervin Teng, Vincent- Berges, Ruo-Ping Dong, Hunter Henry, Marwan Mattar, Alexander Zook, and Sujoy Ganguly. On the use and misuse of absorbing states in multi-agent reinforcement learning. arXiv preprint arXiv:2111.0599 2021.
[85] Jon Louis Bentley. Multidimensional binary search trees used for associative searching. Commanications of the ACM, 1899):509-517, 1975.
[86] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5660 2013.
[87] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, al. Human-level control through deep reinforcement learning. nature, 51 318(7540):599-533 2015.
[88] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: /70.0067777.6444 2017.
[89] Tom Gustafsson and Geordie Drummond Mcbain. scikit-fem: A python package for finite element assembly. Journal of Open Source Software, 5(52):2369, 2020.
[90] Peter Binev, Wolfgang Dahmen, and Ron Vorer. Adaptive finite element methods with convergence rates. Numerische Mathematik, 97:219-268, 2004.
[91] Wolfgang Bangerth, Carsten Burstedde, Timo Heister, and Martin Kronbichler. Algorithms and data structures for massively parallel generic adaptive finite element codes. ACM Tronsactions Mathematical Software (TOMS), 38(2):1-28, 2012.
15

---

[92] Walter A Strauss. Partial differential equations: An introduction. John Wiley & Sons, 2007.
[93] Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement learning the edge of the statistical precipice. Advances in neural information processing systems, 34:29304-29520, 2021.
[94] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. stat, 1050:4, 2018.
[95] Alfio Quarteroni and Silvia Quarteroni. Numerical models for differential problems, volume 2. Springer, 2009.
[96] Volker John al. Finite element methods for incompressible flow problems, volume 51. Springer, 2016.
[97] Olek C Zienkiewicz, Robert Leroy Taylor, and Jian Z Zhu. The finite element method: its basis and fundamentals. Elsevier, 2005.
[98] Marcin Andrychowicz, Anton Raichuk, Piotr Stanccyyk, Manu Orsini, Sertan Girgin, Raphaël Marinier, Leonard Hussenot, Matthieu Geist, Olivier Pietquin, Marcin Michalski, Sylvain Gelly, and Olivier Bachem. What matters for on-policy deep actor-critic methods? large-scale study. In International Conference on Learning Representations, 2021.
[99] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv:1506.. D.XY::5506.02388. 2015.
[100] Mattco Hessel, Joseph Modayil, Hado Van Hasselt, Tom Schaul, Georg Ostrovski, Will Dabney, Dan Horgan, Bilal Piot, Mohammad Azar, and David Silver. Rainbow: Combining improvements in deep reinforcement learning. In Proceedings of the AAAI conference artificial intelligence, volume 32, 2018.
[101] Hado Van Hasselt, Arthur Guez, and David Silver. Deep reinforcement learning with double q-learming. In Proceedings of the AAAI conference artificial intelligence, volume 30, 2016.
[102] Ziyu Wang, Tom Schaul, Matteo Hessel, Hado Hasselt, Mare Lanctot, and Nando Freitas. Dueling network architectures for deep reinforcement learning. In International conference on machine learning, pages 1995-2003. PMLR, 2016.
[103] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, In Yoshua Bengio and Yann LeCun, editors, 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
[104] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high- performarfe deep learning library. Advances in neural information processing systems, 32, 2019.
[105] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprini 1rXiv::442.600 2014.
[106] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. stat, 1050:21, 2016.
[107] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
16

---

A Message Passing Network Architecture
Given a graph = (V,E, Xy, XE), Message Passing Networks (MPN) [33, 38, 39] are GNNs consisting of L Message Passing Steps. Each step ( receives the output of the previous step and and edges E &. Using linear embeddings x and
updates the features Xy. Xg for all nodes x2 of the initial node and edge features, the I-th step is given
1+1 = [x x xx), with e = (u,
x = (x,,
(0,))EE
The operator @ a permutation-invariann aggregation such as a sum, max, or mean operator. Each f. is a learned function that generally parameterize as simple MLP. The network's final output is learned representation x for each node V.
B Systems of Equations
In its most general form, the FEM is used to approximate the solution u(x) that satisfies the weak formulation Væ Vu(z): a(u(x), v(x)) ((v(x)) of the underlying system of equations for the set of test functions v. In the following, we describe the specific equations and boundary conditions used for our experiments.
B.1 Laplace's Equation
Let Ω be a domain with an inner boundary 0n, and an outer boundary 02g We seek a solution u(æ) that satisfies the weak formulation of the Laplace Equation
/ Vu(w)) Vu(ar) dæ
0
for all test functions vz)). Additionally, the solution has satisfy the Dirichlet boundary conditions
u(x) O. æ E an, and u(x) 1, x a21.
We use a unit square (0, 1)2 for the outer boundary a2, of the domain and add a randomly sampled square hole, whose borders are considered to be the inner boundary Mn,, The size of the hole sampled from the uniform distribution (0.05, 0.25)², and mean position is sampled from U (0.2, 0.
We add the closest distance to the inner boundary an additional node feature.
B.2 Poisson's Equation
The weak formulation of the considered Poisson problem is given as
/ Vu(x) Vu(x) dx = / f f(x)v(() dæ Ve.
Here, f(x) Ω + IR denotes the load function and v(x)) the test function. In addition to the weak formulation, the solution must be zero on the boundary O2 of the domain Q. We model Poisson's Equation on L--haped domains Ω2 using rectangular cutoff whose lower left corner is sampled from Po ~ (0.2, 0.95)2, resulting we sample a Gaussian Mixture Model with 3 components. Each component's 's mean is sampled from U 00.1, and use rejection sampling to ensure that all means lie within the domain. The components' covariances are determined by first drawing diagonal covariances. where each dimension drawn independently from a -on-ffrrrmm distribution exp(U(log(O 0003,000 0.003))). The diagonal covariances are then rotated by random angle in (0, 180) to produce Gaussians with full covariance matrix. The component weights are drawn from the distribution exp( NN 1)) + and subsequently normalized, where the 1 in the end is used to ensure that all components have relevant weight. The evaluation of the load function the respective face midpoint is added as node feature.
domain l (0, () (1). On this domain,
17

---

B.3 Stokes flow
Let u(x) be the velocity field and p(z) the pressure field. We consider a Stokes flow of fluid through a channel. Therefore, we seek solution and p. which satisfy the weak formulation of the Stokes flow without forcing term
V / Vo Vu da 1
- / (V u)p dx = 0 We
12
I (V. u) u)
44..
wherein v(x) and g(x)) denote the test functions [95]. We define the inlet-profile as
u(x 0, upy(1 y) + sin (4 + 2xy).
At the outlet, the gradient of velocity Vu(r = 1, = is set to zero. Additionally, we assume a no-slip condition 0 at all boundaries except for the inlet and the outlet. For stability purposes, we use Pi/P3 Taylor-Hood-elements i.e., quadratic shape functions for the velocity and linear shape functions for the pressure [96]. We sample the quadratic part up of the velocity inlet from log-uniform distribution (log(() 2))). The class of domains uses a unit square for the outer boundary and 3 rhomboid holes with length 0.4 and height 0.2 whose centers are set to E {0.2, 0.5, 0.8} in y-direction and randomly sampled from (0.3, 0.7) in x-direction. We optimize the meshes for the prediction accuracy the velocity in and y direction and calculate the overall error the norm of these errors.
B.4 Linear Elasticity
We looking for the steady-state deformation of solid under stress, due to displacements the boundary of the part A. Here, we are interested in both the norm of the deformation and the norm of the stress. The weak formulation of the considered problem the domain Ω without body forces is given as [97]
I a (s (u)) e (v) dx 0.
a
Here, u(x) is the displacement field, v(x) is the test function, and (u) (Vu + (Vu) T is the strain tensor. a (e) is the stress tensor, which is given as a (e) = 2ue + Xtr(e) in a linear-elastic and isotropic case. The Lamé parameters
[[+=][1-0] 2nj and μ π[-v) can be calculated with the problem specific Young's modulus E = 1 and the Poisson ratio // = 0.3. The displacement (x 0, ug the left side of the boundary is specified by dess-dependent parameter un. whereas the displacement u(x = L, y) = 0 is set to zero on the right boundary. The stress 0 is normal the boundary at both the top and bottom of the part. We the same class of L-shaped domains as the Poisson problem in Section B.2 and set up by drawing a random angle from (0, л) pull on the domain from different angles, and add random magnitude from (0.2, 0.8). We add the task-dependent displacement up as feature to all nodes. We are interested in the norm of the displacement field u and the resulting Von-Mises stress, giving us 2-dimensional objective. We weight both dimensions equally in the reward.
Ev
B.5 Non-stationary Heat Diffusion
We consider non-stationary thermal diffusion problem defined by the weak formulation
[ du dæ dæ ( aVu Ve dv dr = =f fv fz dz Ve.
Öt
wherein u denotes the temperature, the test function, a the thermal diffusivity and / a heat distribution, given as
q exp (- ((x (T))) + (v ()(r))))
The position of the maximum heat entry P+(7) = (z, (T) is changing over time, while its magnitude is scaled by factor q. The temperature u E 02 is set zero on all boundaries. For the
18

---

time-integration, the implicit Euler method is applied. We use a total of Tmax 20 time steps in {0.5..... 10), scaling factor of q 1000 and a diffusivity a 0.001. The position of the heat source at step is linearly interpolated as P- = Po + Tan (Pr..x Po). where the start and goal positions Po and Pr... are randomly drawn from the domain. To create our domains, we start with 10 points that are equidistantly placed on a circle with center (0.5, 0.5) and radius 0.4. Each point is distorted by random value drawn from U (00.2, 0.2) We then normalize the resulting points to be in (0, 1)² and calculate the convex hull. The result a family of convex polygons with up to 10 vertices. We measure the error and solution of the final simulation step, and provide the distance to the start and end position of the heat source as additional node features for each element.
C Further Experiments
C.1 Experiment Details
All experiments are repeated for 10 random seeds with randomized PDEs and network parameters. All domains are normalized to be in (0, 1)2 unless mentioned otherwise. The initial meshes created using meshpy?. For practical purposes, we add an element threshold Amux in our environments, and terminate episode with large negative reward when this threshold is exceeded. We train all policies on 100 training PDEs and evaluate the resulting final policies on 100 different evaluation PDEs that we keep consistent across random seeds for better comparability. All experiments are run for up to 2 days on cores of an Intel Xeon Platinum 8358 CPU. In terms of total compute, we train 4 different learned methods, namely the RL baselines and our method, on separate tasks. Each experiment is repeated for 10 different target mesh resolutions and 10 repetitions, resulting in 5 4 10- 10 2000 main experiments. Additionally, we use a similar amount of compute for the combined ablations, preliminary experiments and heuristics.
C.2 Maximum Reward
Equation scales the reduction in error of each element by its area. This modification encourages the policy to focus on smaller elements, effectively shifting the objective from an reduction in average error across the mesh to a minimization of error densities. An alternate way to phrase this objective to make the reward depend on the reduction in maximum error per element. For this, we modify the error estimate per element of Equation read
eir eir(
max ugr (po:;)) uge f Cot
ugg (poz.)) )
and subsequently drop the area scaling and replace the sum in Equation 3 with a maximum, i.c.,
[ EM (
r' ((22)) = eer(())) max M (m((((t))))
While conceptually simpler than our reward formulation, evaluating the decrease in maximum enror only optimizes this objective, which may result in worse meshes when looking at, e.g., the mean error. The left side of Figure compares this alternate reward formulation to that of ASMR. We find that both reward schemes perform similarly. Therefore, we opt for the reward function defined in Equation 3 for simplicity and casier comparison with the baseline methods.
D Extended Results
D.1 Proximal Policy Optimization and Deep Q-Networks.
The left side of Figure shows results Poisson's Equation for PPO and DQN the RL. backbone for all learned methods. We find that PPO outperforms DQN, suggesting that an on-policy objective favorable for the changing observation and action spaces of AMR. We use mean instead of sum for the agent mapping of the targets of the Q-values for the DQN experiments with ASMR as this
"https://gitaan httpp/////gicuub..oo//e/eeeeeeee com//nnducor//reehhy
19

---

ASMR (Ours, PPO)
ASMR (DQN)
VIGN-like (PPO)
VDGN (DQN)
Single Agent (PPO)
Single Agent (DQN)
VDGN-like (PPO, GAT)
VDGN (DQN, GAT)
Sweep (PPO)
Sweep (DQN)
ASMR (PPO, GAT)
Uniform Refinement
Poisson PPO and DQN
Poisson
MPN and GAT
10
10
10-2
Squared
10
10-0
3.0
4.0
5.0
8.0
2,0
3.0
4,0
5.0
6.0
7.0
8.0
(x10")
Elements (x10º)
Figure 7: Pareto plot normalized squared errors and number final mesh elements on Poisson's equation for (Left) PPO and DQN for all RL baselines and (Right) GAT and MPN for ASMR and VDGN(-like). PPO generally results better performance. ASMR works well for both MPNs and GATs, while the VDGN-liks like baseline is more stable when using MPNs.
seems experimentally increase training stability. For the VDGN-like variant that uses PPO, factorize the value function instead of the Q-function, i.e., we define the value function of the full mesh the sum of value functions of the individual mesh elements. We choose PPO for all other experiments as it leads to better performance for all methods.
D.2 Message Passing and Graph Attention Networks.
The right side of Figure 7 compares MPNs and GATs for ASMR and VDGN. For ASMR, the performance between MPNs and GATs is comparable, while VDGN seems to be more stable and produce better refinements when using MPNs. We use MPNs for the other experiments as it seems to benefit VDGN while decreasing the performance of our method.
D.3 Initial Meshes for the ZZ Error.
Figure 8 compares the ZZ Error Heuristic when directly applied to the initial mesh to variants that instead start each refinement procedure by uniformly refining either once or twice. We find that the method greatly benefits from two initial uniform refinements, likely because the heuristic may not detect gradients for interesting parts of the domain if the corresponding elements are too coarse. Given these results, use the twice refined version for all experiments, noting that the RL based methods avoid having to tune the initial mesh by design.
D.4 Target Mesh Resolutions
All RL methods use some parameter to control the number of target elements of the final refined mesh. ASMR and VDGN use an element penalty a, Sweep uses budget Nmax and Single Agent different numbers rollout steps T. visualize evaluations for different target resolutions in Figure 9. The results indicate that ASMR provides meshes with consistent numbers of elements for given target resolution, while the other RL methods produce meshes with inconsistent numbers elements over target resolutions. The concrete target resolution parameters for all experiments found in Table 2.
D.5 Ablations.
Node Features. ASMR utilizes both task- information, such as the evaluation of the load function for Poisson's Equation, and the local solution u(x:) per mesh element as part of its observation graph. Here, we experiment how the performance is affected if either of these features left out. Additionally, we consider a variant where we include explicit (x, y) positions of each element midpoint as node features. The results are shown on the left of Figure 10. We find that
20

---

ASMR (Ours)
lluuisstic (ZZ Ervo, No Unilorm Refinements)
Heuristic (ZZ. Eror, 1 Uniform Refinement)
Heuristic (ZZ Ervor 2 Uniform Refinements)
Uniform Refinement
Poisson ZZ Error Heuristic
10
1
10"
Squared
10
10-5
Elements (x 10
Figure 8: Pareto plot of normalized squared errors and number of final mesh elements on Poisson's 's equation for the Zienkiewiez- Zhu Error Estimator (ZZ Error) Heuristic when using either no, initial uniform refinements. The ZZ Error Heuristic produces better refinements when provided with finer initial mesh the cost of not being able to produce meshes with very few elements.
Low Target Resolution
Intermodiate Target Resolution
High Target Resolution
Uniform Refinement
ASMR
VDGN-liK like (PPO)
16
Squared Eror
10
20-2
10
10
Single Agent
Sweep
10a
Squared Ervor
1011
1
x
ox
10-3
10
Elements (x 100))
Elements (x10*)
Figure 9: Pareto plot of normalized squared errors and number of final mesh elements for the linear elasticity task for all RL methods. Small blue dots indicate a policy trained on a coarse target mesh resolution, which corresponds to large element penalties for ASMR and VDGN-like, small budget Niaxx for Sweep and low number of rollout steps for Single Agent. Large dots correspond to finer target meshes, and the medium-sized purple dots interpolate between the two. Details on the target resolution parameters are found in Table 2. We find that ASMR provides high-quality refinements with consistent numbers of final mesh elements for any given target resolution, whereas the other methods yield poor-quality refinements (Sweep) or inconsistent results (VDGN-like, Single Agent) for similar target resolutions and different random seeds.
21

---

ASMR (Oars, 100 PDE)
ASMR (1 PDE)
ASMR (10 PDEs) ASMR (No Solution)
ASMR (1000 PDEs)
ASMR (No Load Function)
ASMR (Node Positions)
Uniform Refinement
Poisson Node Features
Poisson Num. Training PDEs
10
10
x
10-
Squared
X
10
10-
10-5
0.0
1.0
2.0
3.0
4.0
5.0
6.0
7.0
8.0
9.0
0.0
2.0
4.0
6.0
8.0
Bloments (x16")
Elements (x 10
Figure 10: Pareto plot of normalized Top 0.1 % of errors and number of final mesh elements for Poisson's s Equation. (Left) Omitting either the solution or evaluation of the load function per element leads to a decrease in refinement quality. Explicit node positions slightly decreases performance, likely because they cause the observation to lose equivariance w.r.t., e.g., reflection. (Right) Performance reduced for fewer training PDEs, but stabilizes around 100 PDEs. Interestingly, ASMR achieves better-than performance when using single training PDE, meaning that it can generalize from single training example. This ability is likely a result of our spatial problem formulation.
both the task-dependent features and the solution are important for the performance of our approach. Omitting positional features slightly improves performance, presumably because the features assign a fixed position to each mesh element, causing the observation graph to no longer be equivariant to rotation, translation and reflection. Interestingly, ASMR provides reasonable refinements even without solution information, suggesting that the RL algorithm is able detect relevant regions of the PDE from just an encoding of the domain and the boundary conditions and forcing functions.
Number of Training PDEs Since calculating the fine-grained reference I² is slow for large meshes and complex tasks, we want to minimize the number of unique PDEs that we need during training. We use 100 PDEs in our other experiments, and additionally visualize results for 1, 10 and 1000 training PDEs on the right of Figure 10. We find that fewer than 100 PDEs lead to less stable and reliable results, and that there only a minor advantage in using 1000 PDEs compared to our 100. Noticeably, a single training PDE results in suitable refinements, which hints at significant generalization capabilities that are likely granted by our spatial treatment of the underlying task.
D.6 Alternate Error Metrics
Section evaluates all approaches on the normalized squared error of the mesh. This metric captures both the average error across the domain, leading low mean error, and outliers, thus punishing high maximum error. Here, we additionally present normalized mean and maximum error metrics to provide more thorough nuanced evaluation. The first directly quantifies the average absolute error of the mesh, which makes it easy to interpret and less sensitive to outliers. The maximum error metric measures the worst-case performance, which is crucial for applications where a single high-error prediction could be costly. Since the maximum remaining error is susceptible to outliers, we approximate it as the average of the Top 0.1 % of errors of all integration points po:. For comparability across PDEs, we normalize both metrics by the respective error of the initial mesh Ω2.
Figure 11 displays the results for all tasks and both alternate metrics. The general trends for both metrics are consistent with that of the squared error in Section 5, with ASMR outperforming all learned baselines while being on par with or better than the Heuristies in most cases. The Oracle Error Heuristic performs particularly well on the mean error metric, as it selects elements with high integrated error for refinement. Conversely, the Maximum Oracle Error Heuristic excels on the top 0.1 % error metric, eтror metric tends to favor more uniform meshes due to its lower sensitivity to outliers, enabling baselines like Sweep, which generally produce relatively uniform meshes, yield better performance here when compared to the other metrics.
specifically targets elements with high maximum error. Notably, the mean
22

---

ASMR (Ours)
Single Agent
VDGN- like (PPO)
Sweep
Heuristic (Oracle Error)
Heuristic (Max. Oraele Em.)
Heuristic (ZZ Error)
Uniform Refinement
Laplace's 's Equation
Laplace's Equation
10
10
Top 0. 1% Eiror
10
Meun
10
2.0
2.0
4,0
6.0
8.0
Poisson's Equation
Poisson's Equation
100
10
Top O. 9% Firrrr
Moun Error
10
100
10
10
0.0
6.0
8.0
0.0
2.0
4.0
5.0
8.0
Stokes Flow
Stokes Flow
100
10
Top 0.1 9 Flmor
Mean Erro
10
10
10
10
10
0.0
2.0
4.0
6.0
8.0
10.0
0.0
2.0
6.0 Linear Elasticity
4.0
8.0
10.0
Linear Elasticity
100
10
Top 0. 9 Frro
Mcan Error
10
12
0.0
2.0
4.0
6.0
8.0
10.0
12.0
0.0
2.0
4.0
6.0
8.0
1000
12.0
Heat Diffusion
Heat Diffusion
109
10a
Top 0. % Frnor
Mean lirox
10
10
1
10
1.0
3.0
5.0
7.0
0.0
1.0
Blements (x10")
Elements (x 10"))
Figure 11: Pareto plot of (left) normalized mean errors and (right) normalized top 0.1 % errors compared to number of final mesh elements across different tasks. Performance on both metrics is highly correlated and generally consistent with that of the squared error in Figures 4 and 5. ASMR outperforms all learned baselines on both metrics and all tasks.
23

---

E
Generalization Capabilities and Runtime Experiments
E.1
Domain Size Generalization
Figure Random training PDEs and ASMR refinements for the domain size generalization experiments. The solutions are on different scales as indicated by the colorbars on the left of the meshes.
We experiment with the abilities of ASMR generalize larger domains during inference for Poisson's s equation. Such generalization is non-trivial due to the varying boundary conditions and complexities arising from domain scaling, yet extremely useful in practical scenarios where a policy is trained on small and relatively cheap training domains, and then applied to much larger setups during inference. To generalize to larger domains, we modify the training PDEs to mimic larger mesh segments by altering boundary conditions and load functions. The means of the load function are sampled from a centered unit Gaussian, allowing components outside the mesh. We use domains with random holes for varied initial meshes and apply random Gaussian loads to selected boundary parts as 'inlets'. Examplary training PDEs and ASMR refinements can be seen in Figure 12. We further add an L2 norm of 3e-4 to combat overfitting and omit the per-domain normalization in favor of a constant normalization factor of 100, i.e., use 100 eir(St) instead of err(S2) in Equation These modifications can be seen as data augmentation and only affect the training environments without changing the ASMR algorithm. We evaluate the resulting policy on larger, spiral-shaped domains with initial clements of the same size as the evaluation domains. Figure 13 shows that ASMR consistently provides high-quality refinements as the domain increases size (left) while leading to more and more significant speedups when compared to the reference uniform refinement (right). Here, we use a spiral-shaped domain and load functions with 16 randomly placed components, A slice of these figures for normalized squared error of 0.001 is provided in the main paper in Table 1.
Figure 14 shows how the same procedure scales inference on a spiral mesh of size 20 20 with load function with 81 components. The refined mesh has more than 50 000 elements, which is several times larger than any refinement shown by previous work. Creating and solving this mesh using ASMR is roughly 100 times faster than solving the fine-grained reference Ω A close-up for the marked region is shown on the right side Figure 15. The left side of Figure compares ASMR trained the generalization environments with the setup used throughout the paper, showing that the additional generalization capabilities only lead to marginal decrease in performance on the original evaluation PDEs.
E.2 Same-scale Generalization Capabilities
We additionally visualize ASMR on Poisson's 's equation on domains of size 1 x i.e., of the same size that is seen during training. Here, we utilize the regular training environments without the above augmentation and perform inference on 3 different domain types used in throughout the main experiments, plus simple rectangular domain = (0, 1)² and randomly generated trapezoids. We sample 3 random domains per class, and use Gaussian Mixture Model load functions with 1. 3 and 5 components respectively. Figure 16 shows refinements of an ASMR policy with a = 0.0075 for the resulting 3 x 5 problems. We find that ASMR generalizes across domains and load functions, which is likely a result of the Swarm RL setting, where each mesh element is governed by its own agent.
E.3 Runtime Comparison.
Finally, we compare the wallclock-time of our approach with that of directly computing the fine- grained uniform mesh ! on the evaluation PDEs. For ASMR, measure the cumulative time of

---

times. For the uniform Poisson Domain Size Generalization
2
3 X
1
X
5
6 x 6
7x 7
Uniform Retinement
Reference Meshes
109
100
30
Squared Erroe
10-2
Time [e]
10-3
1
0.3
0.1
10
0.03
10a
10
10
10a
10
10a
Number of Elements
Number of Elements
Figure 13: (Left) Pareto plot the normalized squared error for ASMR evaluated on domains size N x N. While the initial number of elements increases with the size of the domain, fewer relative elements are needed to achieve a given error threshold, suggesting that there are fewer elements with significant error in larger domains. (Right) Wallclock-time in seconds of ASMR (points, solid lines) for different numbers of elements compared to the uniform reference Q (dashed lines). ASMR provides high-quality refinements across different domain sizes, achieving larger and larger speedups when compared to the uniform mesh as the size of the domain increases.
creating an initial coarse mesh, iteratively solving the problem on this mesh, computing the resulting observation graphs after every step, feeding each observation graph to the policy to obtain set of actions, and using each set of actions to refine the mesh a total of mesh, we simply measure the time it takes to refine the coarse mesh 6 times and to subsequently solve the problem on the resulting mesh. We use single 8-Core AMD Ryzen 3700X Processor for all measurements, Figure 17 shows the results for all tasks. We find that approach is always significantly faster than computing the fine-grained mesh despite the comparatively large computational overhead. Further, the final resolution of the refined mesh produced by method trades off the wallclock-ting of the method, meaning that ASMR can be trained to generate coarser task-specific computational budgets. Notably, for the Stokes flow
finer meshes depending equations, which use Pi/Ps gyyloo-Hood--lemeene our method is more than 30 times faster than S for highly refined final meshes. Since the Local Oracle baseline requires the calculation of Q and otherwise follows a similar iterative refinement procedure, its runtime dominated by Ω
25

---

Figure 14: Visualization of a refinement produced by the policy of Figure 12 on a 20 x 20 spiral domain. The presented mesh has 53 189 elements and ASMR produces it in about 10 seconds on a regular CPU, whereas uniform refinement marked region is shown the right side of Figure 15. ASMR provides highly accurate refinements for domains that are significantly larger than those seen during training.
takes more than 20 minutes. close-up of the
F Hyperparameters
F1 General Hyperparameters
We use the same hyperparameters across all methods and environments unless mentioned otherwise.
PPO. We largely follow the suggestions of [98] for our PPO parameters. We train each PPO policy for total of 400 iterations. each iteration, algorithm samples 256 environment transitions and then trains them for 5 epochs with batch size of 32. The value function loss is multiplied with a factor 0.5 and we clip the gradient norm to 0.5. The policy and value function clip ranges are chosen be 0.2. We normalize the observations with running mean and standard deviation. The discount factor is = 0.99 and advantages are estimated via Generalized Advantage Estimate [99] with À = 0.95. We compute agent's ss advantage by subtracting the agent-wisg wise value estimates from the combination local and global returns in Equation
DQN. For DQN-based approaches, we instead train for 24 * 400 = 9600 steps, where each step consists executing an environment transition and then drawing batch of 32 samples from the replay buffer for single gradient update. We additionally draw 500 initial random replay buffer samples before the first training step. We keep 10000 transitions in the replay buffer, since each transition represents full mesh and action on each graph element. We experimented with both larger replay
26

---

ASMR (Ours)
ASMR (Generalization Environments)
Houristic (ZZ Error)
Houristic (Oracle Eror))
Heuristic (Max. Oracle Err.)
Uniform Refinement
Poisson Squared error
10
10
Squared ettor
10
10
10
0.0
2.0
4.0
6.0
80
Elements (x10")
Figure 15: (Left) Pareto plot of the normalized squared error for ASMR trained on regular (blue) and generalizing (green) environments evaluated on 1 x evaluation PDEs. (Right) A local region of the mesh in Figure 14. Adding data augmentation in the training environments allows generalization to significantly larger meshes during inference at the cost of slightly decreased refinement quality.
buffers and more training steps in preliminary experiments, finding that neither significantly improve performance, but may lead to very long runtimes and large memory requirements. During training, we draw actions using Boltzmann distribution over the predicted Q-values per agent, where we linearly decrease the temperature of the distribution from to 0.01 in the first 4800 steps. We find that this action selection strategy leads to more correlated actions when compared to an epsilon greedy action sampling, which stabilizes the training for our iterative mesh refinement problems. We update the target networks using Polyak averaging at a rate of 0.99 per step. Further, we follow previous work [100] and include number of common improvements for DQNs in our implementation. These are double Q-learning [101], dueling Q-networks [102] and prioritized experience replay [103].
Neural Networks. All networks are implemented in PyTorch [104] and trained using the ADAM optimizer [105] with a learning rate of 3.0e-4 unless mentioned otherwise. All MLPs use 2 hidden layers and latent dimension of 64. We separate MPNs for the policy and the value function. Each MPN consists of 2 message passing steps, where each update function represented as an MLP with LeakyReLU activation functions. The policy and value function heads are additional MLPs with tanh activation functions acting on the final latent node features of the MPN. All message aggregations 4 are mean aggregations. Additionally, we apply Layer Normalization [106] and Residual Connections [107] independently for the node and edge features after each message passing step.
F.2 Baseline-Specific Parameters
For Single Agenf, we use a maximum refinement depth 10 refinements per element to avoid numerical instabilities during simulation, skipping actions that try refine elements that have been refined too often. We consider environment sequences of up to T' = 400 steps since the method marks only one element time. For Sweep, the agent placed on a random mesh element for each training step and may decide not to refine this element, resulting in no change in the mesh. Here, we follow the proposed hyperparameters for this approach and train each rollout for 200 steps. As this approach is based on purely local agents, we adapt our input features per element to consist of our regular node features, the global resource budget proposed by the authors, the mean solution and area of the element's ss neighbors and the average distance to them. The global budget is controlled via a maximum number of elements Nmux. allowing to get refinements of different granularity. To accommodate for less overall changes in the mesh increase the number of environment transitions
27

---

Component
3 Components
Components
Shape
Rhomboid Holes
Square Hole
Convex Polygon
Square
Figure 16: ASMR refinements for different domains and Poisson's 's equation with a Gaussian Mixture Model load with and 5 components. Even though the ASMR policy only trained on L-shaped domains with 3 components in the load function and no data augmentation, generalizes to different domains and loads.
of PPO to 512, and the number of DQN steps to 96 400 38400. Finally, use learning rate of 1.0c-5 instead 3.0c-4 for the DQN variant of VDGN to stabilize its training.
F.3 Refinement Hyperparameters
The AMR methods considered in this work use different parameters to control the granularity of the final refined mesh. ASMR and VDGN use an element penalty a, while Sweep considers an element budget Nmx. Single Agent varies the number of rollout steps T. For each learned method and task, we choose 10 different values for the refinement parameter that showcase wide range of final mesh resolutions. For the Oracle, Maximum Oracle, and ZZ Error Heuristics we instead cover range of up to 100 parameter thresholds 0, yielding one aggregated evaluation result per threshold.
Table 2 lists the different ranges for these parameters for the different tasks. For stability purposes, we set maximum number of 20 000 elements during training for experiments except for the Sweep baseline, as this baseline uses its own element budget instead. If this number is surpassed, constant penalty of 1000 is subtracted from the reward and the episode terminates early.
28

---

Laplace's Equation
Poisson's Equation
Stokes Flow
Linear Elasticity
Heat Diffusion
Reference Meshes
Uniform Refinement
10a
10
Time
10
10 0.0
2.0 4.0
6.0
8.0 10.0 12.0
Elements (x 10")
Figure 17: Wallclock-time in seconds of ASMR (points, solid lines) for different numbers of elements compared to the uniform reference ! (dashed lines). Each task is denoted by its color, and the full lines represent quadratic regression of the wallclock-time of our method for different numbers of final elements. On average, Ω contains about 10" elements, with concrete numbers varying depending on the domain. Our approach significantly faster than the reference for all tasks, achieving speedups of factor 2-30 for reasonable numbers of elements.
Table 2: Ranges for the different refinement hyperparameters for all tasks. ASMR and VDGN apply an element penalty a. but only ASMR scales the area of each element with area in Equation Sweep uses an element budget Nmax Single Agent varies the number of rollout steps I', and the Oracle Error, Maximum Oracle Error and ZZ Error Heuristies use of different error thresholds O.
Method
Task
Laplace
Poisson
Stokes Flow
Lin. Elast.
Heat Diff.
ASMR (a)
00.01. 0.3]
0.002, 0.1]
[0.006, 0.15)
0.01, 0.15]
0.003, 0.3]
VDGN(-like) (a)
2c-5, 5c
2c-5, 5e-2)
3e-4, 5c-3)
lc-5, 1e-2)
5c-6, 5e-3)
Sweep (Nmx)
200, 3000]
200, 3000)
(200, 3500]
500, 6000)
400, 5000]
Single Agent (T)
25. 400]
25, 400
25, 400]
25, 400]
25, 400)
Oracle Error (0)
0.25, 1.00
1.0)
0.16, 1.0)
0.02, 1.0
0.03, 1.0)
Max. Oracle Err. (0)
0.20, 1.0)
0.2, 1.0
0.1, 1.0]
0.01,
0.02, 1.0
ZZ Error (0)
0.001,
0.002,
00.001, 1
0.001, 1]
0.001,
G
Visualizations
We provide additional visualizations for method all tasks, and for all methods the Poisson task. All visualizations show the final refined mesh the respective method for 5 different refinement levels randomly selected PDEs. For the RL methods, all policies are taken from the first repetition of the 10 random seeds conducted for the respective experiment.
G.1 ASMR Refinements
visualize exemplary refinements of ASMR policies for considered tasks in Figures 18 (Laplace's s equation), 19 (Poisson's 's equation), 20 (Stokes equation), 21 (Linear Elasticity), and 22 (Heat Diffusion). Across all tasks, ASMR able to provide highly accurate refinements for different numbers of total clements.
29

---

G.2 Baseline Comparisons
Figure 23 shows refinements for Single Agent for different total timesteps T. Figure 24 presents VDGN with different values. Figure 25 visualizes refinements of Sweep for varying number of maximum elements Nmax Figures 26, 27 and 28 show refinements of the Oracle, Maximum Oracle and 77. Error Heuristics for different values of the threshold 0.
The visualizations show that the RL baselines struggle provide consistent high-quality refinements for different mesh resolutions. The Single Agens baseline sometimes focuses on uninteresting regions of the mesh refines the same area too often. The VDGN-like baseline performs well in some cases, but collapses to no refinements or fully uniform refinements for some evaluation PDE. Sweep provides almost uniform refinements for most PDEs and element budgets, likely result of the misalignment in the environment transitions between training and inference. The Heuristics greedily refine the elements with the largest error estimates in their respective metric, regardless of the resulting decrease in error. This behavior generally leads to locally accurate refinements, but fails to effectively decrease the mesh error some cases. Additionally, the heuristics act locally, which causes potential issues for PDEs with global dependencies [92] and conforming refinements. The 7Z. Error Heuristic sometimes misses regions of interest, but provides much smoother refinement than the Oracle Error Heuristic, which can be beneficial for the error reduction in some cases.
G.3 Element Markings
Figure 29 visualizes full rollout our method, including the markings of the elements after every step.
0.2
0.1
0.075
0.03
0.015
Figure 18: Final refined meshes ASMR for randomly sampled PDEs for the Laplace equation for different element penalties a.
30

---

a= 0.1
0.05
0.03
0.01
0.005
Figure 19: Final refined meshes of ASMR for randomly sampled PDEs for the Poisson equation for different element penalties a.
0.2
0.1
0.05
0.03
0.02
Figure 20: Final refined meshes of ASMR for randomly sampled PDEs for the Stokes flow task for different element penalties a.
31

---

a 0.1
0.075
0.025
0.009
a 0.007
Figure 21: Final refined meshes of ASMR for randomly sampled PDEs for the linear elasticity task for different element penalties a. The visualizations show the deformed meshes, which are originally L-shaped.
DOCCO
a= 0.1
Q 0.05
0.03
0.01
a: 0.005
Figure 22: Final refined meshes of ASMR for randomly sampled PDEs for the non-stationary heat diffusion task for different element penalties a.
32

---

T = 50
T = 100
T = 200
T 300
T = 400
Figure 23: Final refined meshes of the Single Agent baseline for the Poisson equation randomly sampled PDEs for different environment rollout lengths T.
0.05
0.01
0.005
0.0005
5e
05
Figure 24: Final refined meshes of the VDGN-like baseline for the Poisson equation randomly sampled PDEs for different element penalties a.
33

---

N Mmx 200
N 500
N 750
N 1500
Nmax 2000
Figure 25: Final refined meshes of the Sweep baseline for the Poisson equation randomly sampled PDEs for different maximum numbers of elements N Max-
0 0.9
0 = 0.6
0
0.5
0.4
0 0.2
Figure 26: Final refined meshes of the Local Oracle baseline for the Poisson equation randomly sampled PDEs for different error thresholds
34

---

0 = 0.9
0 = 0.6
0 = 0.5
0 0.4
= 0.2
Figure 27: Final refined meshes of the Local Maximum Oracle baseline for the Poisson equation on randomly sampled PDEs for different error thresholds 0.
0 0.5
0 0.25
0.125
0 0.0625
0 0.03125
Figure 28: Final refined meshes of the 7Z. Error for the Poisson equation randomly sampled PDEs for different error thresholds 0.
35

---

Figure 29: Visualization of full rollout of our method on Poisson task, including the markings of the elements after every step. The figures in the first and third row show the markings, and the second and fourth row show the resulting refined meshes.
36


Reviews:
[{"review_id": "fXnxWkgMXe", "reviewer": "Reviewer_Tf6s", "summary": "The article presents an RL-based framework for adaptive mesh refinement in FEM. In this framework, each element in the mesh is perceived as an agent from RL's perspective. The observation for each agent is then constructed using a variety of local and global features relevant to the PDE of interest. The reward function is setup to account for the impact of agent's action in terms of error reduction and increase in compute cost, and also for future agents that are a result of a decision at current time (due to mesh refinement). The method is demonstrated to work for a variety of PDEs and be somewhat competitive with traditional error-based strategies.", "questions": "None", "limitations": "The authors only talk about how the formulation can be used for de-refinement, but do not actually include it in their method.\nThe authors do not compare their method to existing estimators such as the one by Zienkiewicz-Zhu, which is fairly inexpensive and commonly used.\nThe demonstrated generalizations are for simple PDEs for relatively small meshes. Thus, I find the use of the phrase \"complex simulations\" in the abstract arguable.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Use of graph neural networks helps tackle unstructured meshes.\nThe agent mapping $M$ helps account for future agents that result from an agent due to mesh refinement.\nReward formulation helps account for error reduction and increase in compute cost due to mesh refinement.", "weaknesses": "While there is some novelty in terms of reward formulation and agent mapping, I find that the biggest weakness of the method is that it is only demonstrated to work for simple problems on meshes with only thousands of elements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their insightful feedback, particularly on the significance of our reward formulation and agent mapping. We acknowledge their concerns about the scalability of our method and the necessity for a comparison with existing error estimators. In the following, we address each point raised by the reviewer in detail.\n\n> While there is some novelty in terms of reward formulation and agent mapping […]\n> \n\nWe thank the reviewer for recognizing the innovative aspects of our work. We want to highlight that the novelty in our approach, particularly in reward formulation and agent mapping, plays a fundamental role in the success of our method. This is evidenced by the performance of our method when compared to the baselines, which mainly differ from our method in these key aspects.\n\n> The biggest weakness of the method is that it is only demonstrated to work for simple problems on meshes with only thousands of elements.\n> \n\nWe provide new generalization experiments in our general answer and the accompanying PDF that show how ASMR generalizes to meshes with tens of thousands of elements when slightly adapting its training setup. The experiments in the main paper also show that existing RL-based methods struggle on the presented tasks, suggesting that the novel reward formulation and agent mapping in the present work mark an important step towards learning mesh refinement strategies for more complex problems.\n\n> The authors do not compare their method to existing estimators such as the one by Zienkiewicz-Zhu.\n> \n\nWe sincerely thank the reviewer for suggesting to compare to additional established methods for AMR such as the Zienkiewicz-Zhu (ZZ) error estimator. We apply this error estimator to a percentage-based heuristic that refines all elements that are within the top $k\\,\\%$ of errors, and compare the results with ASMR on the Poisson problem in Figure 3 of the uploaded PDF.\n\nWe find that the ZZ is significantly outperfomed by ASMR for the Poisson problem when applied as is. We find on preliminary visualizations that this is likely due to the ZZ error estimate missing entire modes of the load function. To compensate, we also compare to variants that uniformly refine the initial mesh one or two times (ZZ Error ({1,2}x Uniform)) before starting to apply the ZZ-error-based heuristic. When applying two initial refinements, the error estimate is competitive to ASMR on the top $0.1\\,\\%$ error, but significantly worse on the mean error metric. These results suggest that the ZZ error can provide competitive refinement when applied to suitable meshes, but that ASMR is preferable for general tasks. We will revise the paper to include the ZZ error estimate for all considered tasks.\n\n> The authors only talk about how the formulation can be used for de-refinement, but do not actually include it in their method.\n> \n\nWe acknowledge that mesh de-refinement is important for general AMR strategies and aim to explore this direction further in future work. In the present submission, we focus on AMR for conforming triangular meshes and consider problems where de-refinement and coarsening operations are not crucial. We provide additional discussion in the general answer.\n\n> The method is demonstrated to work for a variety of PDEs and be somewhat competitive with traditional error-based strategies.\n> \n\nWhile traditional error-based strategies outperform our method in select scenarios, a considerable advantage of our approach lies in its enhanced computational speed. We show in Appendix D.3. that computing a reference mesh, which is required for the local oracles, is $2$ to $30$ times slower than our method. This advantage in computational speed is further amplified when considering larger meshes such as the one shown in Figure 2 of the uploaded PDF. Here, the refined mesh consists of slightly more than $50\\,000$ elements, and computing it is more than $100$ times faster than generating a uniform refinement.\n\nWe earnestly hope that our clarifications and provided materials address the concerns raised. If there are any lingering reservations or questions about the paper, we encourage the reviewer to reach out to us for further clarification."}, {"review_id": "EGfcNgqXq3", "reviewer": "Reviewer_fwaD", "summary": "This paper presents a novel framework for adaptive mesh refinement for solving PDEs describing physical systems. The refinement is done as a Markov Decision Process in a Swarm RL setting, with each element of the mesh being an agent in the swarm. The agents' action space is binary and a learned policy decides whether an element should be refined or not. The system is supervised by a task-agnostic reward.", "questions": "Can the method be extended/adapted for Mesh Movement instead of AMR? It'd extend the applicability of the proposed framework.", "limitations": "Same as those written in the weakness section - primarily, the results are only shown on 2D domains.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The problem of neural-AMR for efficient simulation is an important one, and the proposed solution is novel and promising.\n2. The ability to have variable number of agents as the process progresses allows for finer refinements with more degrees of freedom.\n3. The proposed refinement strategy doesn't require task-specific heuristics/rewards.\n4. The improvement in speed as observed in the supplemental is impressive, especially for Fluid Flow.\n5. Code is provided.", "weaknesses": "1. While the method handles complex PDEs, it does so only for 2D domains (as observed by the authors). it'd be interesting to see the results in 3D domains.\n\n2. While Message Passing Networks do the job, it might be that Message Passing Attention Networks/ Graph Attention Networks as recently proposed might improve the results. I think an experiment in that direction is important to justify MPNs over more recent techniques.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable insight and suggestions, and particularly for highlighting the significance of our work's potential in addressing complex PDEs for efficient simulation. In the following, we want to address the individual points raised by the reviewer.\n\n> While the method handles complex PDEs, it does so only for 2D domains (as observed by the authors). it'd be interesting to see the results in 3D domains.\n> \n\nThe present work focuses on 2D domains and conforming triangular meshes. We fully agree with the reviewer that extending the approach to 3D domains is an important next step, and briefly touch on the subject in the `Limitations and Future Work' section of the submission. While extending the experiments to 3D is out of the scope of this rebuttal period, we showcase the generalization capabilities of our method to significantly larger 2D domains in the PDF attached to the general answer.\n\n> Message Passing Attention Networks/ Graph Attention Networks as recently proposed might improve the results\n> \n\nWe thank the reviewer for the suggestion. We use Message Passing Networks instead of e.g., Graph Attention Networks or Graph Convolutional Networks as they are the most general graph neural network architecture and have been used in related work for physical simulations (see e.g., Ref. 32, 37, 38) and learned adaptive mesh refinement (Ref. 27). The contributions of the present work lie in viewing mesh refinement as a swarm problem and providing spatial rewards and appropriate agent mappings for this problem. As such, the method is agnostic to the specific kind of GNN used. Still, we agree with the reviewer that a more optimized network architecture may improve downstream performance, which is important for practical applications of the presented method. We will add a discussion on the choice of network architecture to the revised paper. \n\n> Can the method be extended/adapted for Mesh Movement instead of AMR?\n> \n\nWe thank the reviewer for the intriguing question. Mesh movement operations or R-Refinement could be implemented by extending the action space to include a velocity for each element’s barycenter and adapt the agent mapping based on overlapping element regions. Relatedly, we briefly discuss in lines 163-165 of the paper how the method can be extended to e.g., mesh coarsening operations. We find this to be a promising direction for future work, and provide a more detailed explanation for why this is not done in the present paper in the general answer.\n\nWe hope that our responses address the reviewer's concerns, especially with respect to future extensions of the method. Should there be any further queries or unresolved issues, we encourage them to reach out to us."}, {"review_id": "DzYFYtGMKN", "reviewer": "Reviewer_ymhS", "summary": "This paper builds on recent advances in learned adaptive mesh refinement method to scale to complex physical simulations. Instead of formulating AMR as a reinforcement learning problem with a single agent, the authors formulate AMR as a swarm reinforcement learning problem, in which multiple agents collaborate and share observations each other. By allowing the agents to be split into new agents in AMR process, a dense reward signal is efficiently feed to policy function to decide cells to refine. The experiments demonstrate that the method achieves competitive error against traditional error-based refinement strategies and generalizes well across different problem domains.", "questions": "It would be interesting to see a study of how the hyperparameters of MPN (for policy function) impact the wall-clock time and error estimate. For example,\n* Does large message passing steps decrease the error estimate and improves the scalability to much larger number of mesh elements? \n* How much does the wall-clock time increase as increasing the message passing steps? \n", "limitations": "Yes, the paper discusses technical limitations in the main text. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper is placed in the current growing literature on ML-based Adaptive Mesh Refinement (AMR), to enable scaling to large and complex systems. The authors propose formulating AMR as swarm reinforcement learning problem. The paper conducts various range of experiments and compare to RL-based baselines. The proposed method outperforms existing RL-based methods in most of the cases. The experiments are thorough, covering all the important ablations I could think of while reading the paper.", "weaknesses": "While the paper is generally well written, notations introduced in Section 3 are ambiguous. Below I list a few which made the paper harder to understand\n* $M^{t,k}$ seems not to meet $\\forall$ $j, $ $\\sum M^{t, k}_{i, j} = 1$ since it seems to be defined by Hadmard product. \n* Although definitions for state and observation are provided, they do not look well-defined; a local view of observation graph can have infinitely many choices. \n* The definition of $J_{I}^{t}$ is also ambiguous; the domain of the policy comprises the observation and action spaces while reward is defined on state space and action space.  \n* What do $V$ and $E$ in the observation graph represent? \n\nAs someone not familiar with this field, I found the subsection on \"Systems of Equations\" in Section 4 described very clearly PDE systems defined on various types of (maybe complex) shapes. However it is still unclear to me how challenging solving AMR for those PDEs is, although they are claimed as being challenging in the introduction section. Can you expand on the details of PDEs and give some reasons that make solving AMR for those types of PDEs difficult?\n\nFor the evaluation of the wall-clock time of ASMR, I believe I don’t see plot of the proposed method in the figure in Appendix D.3. If so, it is difficult to compare the proposed method against other baselines. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's constructive feedback. We are pleased to hear that they found our experiments and ablations to be conclusive and convincing. We also thank them for their remarks on the notations in Section 3 and their inquiries about the challenges of solving AMR for specific types of PDEs.\n\n> While the paper is generally well written, notations introduced in Section 3 are ambiguous. Below I list a few which made the paper harder to understand\n> \n\nWe thank the reviewer for pointing out that the paper is currently unclear in terms of notation. We will clarify it during the revision, and briefly explain the notation in its current form in the following.\n\n> $M^{t,k}$ seems not to meet $\\forall_j \\sum M^{t,k}_{i,j}=1$ since it seems to be defined by Hadmard product.\n> \n\n$\\mathbf{M}^{t,k}$ is defined by the (regular) matrix multiplication $\\mathbf{M}^{t,k}=\\mathbf{M}^t\\mathbf{M}^{t+1}\\dots\\mathbf{M}^{t+k}$ in Line 150 in the paper. We will clarify that this is a matrix multiplication rather than a Hadamard product in the revision. As the individual matrices satisfy $\\forall_j\\sum_i \\mathbf{M}_{ij}=1$ , so does their product.\n\n> Although definitions for state and observation are provided, they do not look well-defined; a local view of observation graph can have infinitely many choices.\n> \n\nThe state of the system includes the mesh, its boundary conditions and its current solution. The observations encode this state as an observation graph (c.f. lines 166-169). The features for this graph depend on the current PDE, mesh and solution, which we detail in lines 222-229. We will adapt the paper to make the connection between the definition of the observation graph and its instantiation clearer. The local observation for each agent stems from its position in the observation graph. Our method is permutation equivariant due to the use of GNNs, ensuring consistent outputs from agents with identical local observations. We hope that this addresses the concern regarding the “infinitely many choices” and would appreciate the opportunity to provide further clarification if it does not. \n\n> The definition of $J_I^t$ is also ambiguous; the domain of the policy comprises the observation and action spaces while reward is defined on state space and action space.\n> \n\nThe reward is defined as a function of state and action, while the policy is a function of the deterministic local observations induced by the mesh/graph. The missing link here is the observation function $\\xi(s)$. The full equation above line 150 thus reads $J_i^t:=E_{\\pi(\\mathbf{a}|\\xi(\\mathbf{s}))}[\\sum_{k=0}^{\\infty}\\gamma^k(\\mathbf{M}^{t,k}\\mathbf{r}(\\mathbf{s}^{t+k},\\mathbf{a}^{t+k}))_i]\\text{,}$ which we will adapt in the revision.\n\n> What do $V$ and $E$ in the observation graph represent?\n> \n\n$V$ and $E$ are the vertices and edges in the observation graph respectively, and represent individual mesh elements and their neighborhood. This is briefly explained in Lines 166-167. We will clarify this notation in the revision.\n\n> Can you expand on the details of PDEs and give some reasons that make solving AMR for those types of PDEs difficult?\n> \n\nWe thank the reviewer for the insightful question and the opportunity to expand on the details of the used PDEs. The general answer discusses which aspects make finding a good AMR strategy difficult for a particular PDE. On a very high level, finding a good refinement strategy for a PDE is difficult if there are certain areas of the domain that are much more interesting for the solution than others. A good AMR strategy must identify these areas and adapt their local resolution to a degree that depends on the local complexity of the problem\n\n> I believe I don’t see plot of the proposed method in the figure in Appendix D.3. If so, it is difficult to compare the proposed method against other baselines.\n> \n\nRegarding the evaluation of the wall-clock time of ASMR, Figure 10 on Page 22 of the appendix compares our method (represented by solid lines) with the calculation of the reference solution on a uniform refinement $\\Omega^*$ (dashed lines). Other baselines are not part of this figure. The other RL-based methods with the exception of the *Argmax* baselines have a runtime that is comparable to ours due to their iterative refinement process but produce significantly worse refinements. Meanwhile, the local oracles use the reference mesh $\\Omega^*$ which dominates their runtime. We will add a section in the revised version of the paper to clarify this relationship.\n\n> Does large message passing steps decrease the error estimate and improves the scalability to much larger number of mesh elements?\n> \n\nOur preliminary experiments with larger network architectures suggest a very small but consistent performance improvement when increasing the latent dimension from $32$ to $64$, whereas more than $2$ message passing steps do not significantly improve performance. For a Poisson problem with approximately $10\\,000$ final mesh elements, the total runtime during inference increases by slightly over $10\\,\\%$ when either the latent dimension is increased to $64$ or the number of message passing steps is increased to $4$. While concrete numbers depend on the sizes of the individual meshes, executing the policy for the above example takes about $25$ to $30\\,\\%$ of the total runtime of a rollout, whereas calculating the solutions of the intermediate meshes takes up about $40$ to $45\\,\\%$ of time. The remaining time is taken up by the construction of the initial mesh and PDE and the calculation of the observation graph. We will provide more detailed results over different mesh resolutions in the revised version.\n\nWe hope that our answer clarifies some of the details of our method. Should any issues persist or new ones arise, including ones regarding the notation in Section 3, we would greatly appreciate further communication."}, {"review_id": "t0jKqIoqX6", "reviewer": "Reviewer_LCc4", "summary": "This paper proposes a novel MDP formulation and policy architecture for adaptive mesh refinement.\nThe MDP formulation (ASMDP) defines the components of the MDP in order to account for the changing number of agents across timesteps, and the reward is formulated in a manner to make credit assignment easier and to account for particular aspects of the AMR task (e.g., accounting for the area associated with an element).\nThe paper applies RL algorithms (PPO, DQN) with a graph neural network policy (with a discrete action space of whether or not to refine a particular node).\nThe experiments consider a set of 2d mesh refinement problems and evaluate primarily with top 0.1% error against other learned and oracle baselines, showing that ASMR outperforms the other learned baselines.\n", "questions": "Problem formulation / algorithm\n1. Does it make sense to formulate the problem as a SwarMDP / special case of a DecPOMDP? It seems that in practice (both at training and inference time) the problem is fully observable and decision making is performed in a centralized fashion (which would make it an MDP). Is that not the case? Perhaps this is sort of a semantics question about what constitutes the problem formulation vs solution.\n2. Relatedly, assuming the SwarMDP formulation, in what sense is the problem partially observable to individual agents/elements? It seems that each agent makes decisions based on an observation space capturing information about the full state of the system (i.e., the observation graph taken as input seems to fully capture the state of the system). Is that not the case?\n3. Given that $\\Omega^{\\*}$ and $\\Omega^{0}$ are available, it seems like the problem could be formulated as a supervised learning problem (SL for AMR is discussed in the related work, but I don’t think the references answer the following question). Given this, why is it necessary to formulate the problem as an RL problem? Is it simply that mapping from $\\Omega^{0}$ to $\\Omega^{*}$ is best done in an iterative manner (due to the variable number of nodes) (thereby making it a sequential decision making problems best addressed by RL)?\n\nEvaluation\n1. In the runtime comparison of section D.3, it seems like either of the processes being compared could be parallelized effectively on the GPU. Is that not the case? Why is the comparison performed on CPUs?\n2. Related to that runtime comparison, what is the additional runtime cost of executing the local oracle approaches? Does it make sense to consider that in the comparison?\n3. In figure 14 (right), would you elaborate on why ASMR underperforms the baselines? \n", "limitations": "See weaknesses and questions sections\n", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "### Significance\n- Problem setting: FEM plays an important role in many fields and (given the potential scale of such problems) improvements in the accuracy / computational efficiency could have a significant impact on the rate of progress or computation required\n- Contribution: The proposed problem formulation / algorithm significantly improves performance and reliability on a sample of AMR tasks as demonstrated in the experiments and appendix, which constitutes a significant contribution.\n### Originality\n- Aspects of the MDP formulation (reward function, aspects of the handling of a time-varying number of agents) appear to be novel contributions.\n- The use of the particular policy architecture appears to be novel (though the use of GNNs for the policy isn’t novel)\n- Extensive experiments investigating the effectiveness of the proposed formulation/algorithm in a variety of of AMR setting (demonstrating sota performance of the algorithm)\n### Quality\nApproach\n- The application of RL in this setting is reasonable. The problem formulation / algorithm are formulated reasonably.\nEvaluation\n- The main experimental claim is that ASMR outperforms existing learned methods for AMR, and this is evaluated extensively through experiments across different tasks with appropriate metrics, and the results support the claim.\n- A number of important additional questions are addressed experimentally by the paper (e.g., ablations, use of top 0.1% error directly as reward, runtime comparison w/ just computing $\\Omega^{*}$)\n### Clarity\n- The paper is exceptionally well-written and clear. Figures and tables are explained clearly and contribute to the understanding of the method / experiments.\n- Sufficient details are given in the appendix to reproduce the paper, and code is provided in the supplement, which appears to be reasonably well-designed / implemented.\n", "weaknesses": "### Originality\nAlgorithm\n- Ultimately many aspects of the problem formulation / algorithm are based on prior work, and the novel contributions are (1) specific aspects of the problem formulation (e.g., reward function), (2) the application of existing methods to the problem (some of which are novel) (e.g., policy architecture, RL algorithms), and (3) extensive experimental evaluation.\n### Quality\nAlgorithm\n- See questions section\nEvaluation\n- Further evaluation of the generalization abilities of the algorithm would be beneficial (e.g., building on D.4 with quantitative metrics) because (I assume) that an RL algorithm for AMR would in practice be applied outside its training distribution.\n- Relatedly, it’s not clear to me based on the experimental results that practitioners using AMR for FEM would prefer ASMR to the local oracle or local maximum oracle approaches in practice (perhaps ASMR is faster excluding training, but is the reduction in simulation time worth the less consistent performance?). Does increasing the amount of training data (or other parameters) improve the performance of ASMR beyond these oracle baselines across (more of) the tasks? \n### Clarity\n- The setting / setup of section D.4 on the generalization capabilities is unclear to me\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their detailed evaluation of our paper, particularly emphasizing our method's significant contributions and robust experimental claims. We will now respond to the individual concerns raised by the reviewer, including the generalization capabilities of the method and the importance of framing AMR as a Swarm Reinforcement Learning problem.\n\n> Further evaluation of the generalization abilities of the algorithm would be beneficial (e.g., building on D.4 with quantitative metrics) […].\n> \n\nWe recognize the importance of robust generalization for learned AMR methods and show further qualitative experiments in Figure 2 of the general PDF. Initial quantitative evaluations suggest that generalization is consistent across methods for all domains shown in Figure 11, likely due to the local perspective that all methods use. Given ASMR's superior performance, it also seems to perform best in these generalization tests. We will detail these quantitative evaluations in the revision.\n\n> perhaps ASMR is faster excluding training, but is the reduction in simulation time worth the less consistent performance?\n> \n\nFigure 10 in the Appendix shows that our method is up to $30$ times quicker than uniform refinement, while the main experiments underscore its improved reliability compared to other RL-driven approaches. To iterate on this, the large mesh in Figure 2 of the general PDF takes $10$-$12$ seconds to produce, whereas the uniform refinement needs $~20$ *minutes*. Training ML models such as ASMR in early stages of engineering development thus allows to speed up subsequent phases due to faster inference times. Generalizing well also allows for future problems to be predicted, making the training time negligible.\n\n> Does increasing the amount of training data (or other parameters) improve the performance of ASMR beyond these oracle baselines across (more of) the tasks?\n> \n\nFigure 9 in Appendix D.2 illustrates that more training PDEs marginally boost performance. Our preliminary results indicate that larger latent dimensions offer slight performance gains. Since our main goal is to introduce and assess an effective RL framework for AMR, we do not fine-tune the specific architecture to surpass the oracle heuristics. Still, we acknowledge the importance of this optimization for practical usage and will discuss it in more detail in our paper's revised version.\n\n> The setting / setup of section D.4 on the generalization capabilities is unclear to me\n> \n\nIn Appendix D.4, we apply a singular trained policy to problems outside the training distribution. The policy's training data comprises 100 L Shapes with three-component load functions, as shown in the middle-left part of Figure 11. The rest of Figure 11 depicts ASMR's generalization across various shapes (left to right) and load functions (top to bottom). We'll offer more clarity in our paper's updated version.\n\n> Does it make sense to formulate the problem as a SwarMDP / special case of a DecPOMDP?  It seems that […] the problem is fully observable […] in what sense is the problem partially observable to individual agents/elements?\n> \n\nWe discuss the importance of the adaptive swarm setting in the general answer. While the problem is fully observable, each agent's observation space is localized, with the receptive field depending on the number of message passing steps of the GNN. Each agent can access a shared global vector of limited size (32 in our experiments), keeping the observation space predominantly local.\n\n> Given that Ω∗ and Ω0  are available, it seems like the problem could be formulated as a supervised learning problem […] why is it necessary to formulate the problem as an RL problem?\n> \n\nWe present AMR as an RL problem because of its sequential nature, noting that it additionally allows for non differentiable refinement and PDE-solving processes, which increase the methods applicability. This approach aligns with existing literature (Refs. 27-29), whereas supervised methods often target error estimators (Refs.71-72) or specific AMR facets (Refs.73-74) instead of actual refining strategies directly. We'll clarify this distinction in our revised related work section.\n\n> […] it seems like either of the processes being compared could be parallelized effectively on the GPU.\n> \n\nSince our experiments utilize Scikit-FEM (Ref. 39) which is CPU-only, we perform all runtime tests on a CPU for fairness. In general, both the policy and solving the PDE could be efficiently parallelized on a GPU. Broadly speaking, our model's scalability is approximately linear with the refined mesh's elements, while solving a uniform refinement becomes progressively costlier as the problem size increases.\n\n> What is the additional runtime cost of executing the local oracle approaches?\n> \n\nThe cost of executing the local oracle heuristics is dominated by the calculation of the reference refinement $\\Omega^*$. Since only computing $\\Omega^*$ already takes longer than our policy does, we chose not to include them in Figure 10. We will clarify this in the revision.\n\n> In figure 14 (right), would you elaborate on why ASMR underperforms the baselines?\n> \n\nIn Figure 14, ASMR shows good performance for large meshes but lags behind some baselines for smaller instances. This is mainly because ASMR skips refining larger elements far from the flow's inlet. While these elements don't show a significant maximum error (as seen in Figure 5's left side), they contribute to a considerable total error mass. Many baselines offer relatively uniform refinements, explaining their performance and similarity with the uniform comparison. Crucially, ASMR recognizes that errors on the left side intensify and accumulate as information flows to the right, explaining its effectiveness for larger element counts.\n\nWe hope that our answer clarifies some of the concerns of the reviewer, and would be grateful for further communication if any issues persist or new ones arise."}, {"review_id": "wCRt9kHUaJ", "reviewer": "Reviewer_qjoP", "summary": "This paper formulates h-adaptive mesh refinement (AMR) as a decentralized partially-observable Markov decision process (Dec-POMDP), and proposes a methods ASMR with parameter-sharing among agents and individual rewards to find refinement strategies. Refinement policies are parameterized by message-passing graph neural networks (MPN) and trained by Q-learning (specifically, DQN) and proximal policy optimization (PPO). Evaluation was conducted on various 2D elliptic partial differential equations (PDEs) with different domains, showing that the proposed approach is competitive with traditional error-based AMR heuristics and outperforms modified versions of existing learning-based approaches.", "questions": "Suggestions:\n- Demonstrate that trained policies generalize to larger meshes and more time steps than those in training. This is necessary for policies trained on tractable small problems to be deployed in real FEM simulations.\n- some of the learning-based baselines implemented in this paper do not accurately reflect the methods in the referenced papers, so they should not be described as though they represent those works.\n- Include empirical evidence that the approach extends to coarsening actions.", "limitations": "The statement of limitations should include the absence of empirical results on coarsening, and the lack of other classes of PDEs such as hyperbolic equations.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "Originality: One main novelty of this paper is the use of independent multi-agent reinforcement learning with individual rewards/returns for AMR. The proposed individual return is a novel solution to the posthumous credit assignment problem that arises in the AMR context, whereby an agent vanishes upon taking a refinement action. It also has the advantage of avoiding the difficulties of the regular multi-agent credit assignment problem that arises from using a single team reward.\n\nQuality: Overall, this paper covers most of the aspects of a high-quality applied paper, as it provides sufficient motivation and background for the application and provides a rigorous formulation and empirical evaluation.\n\nClarity: The description of the problem, the proposed method, the notation, and experimental setup and results are all clearly written. See below for more comments about clarity.\n\nSignificance: The results in this paper are significant as they provide a path to improving AMR for the finite element method, which is a critical tool in engineering and applied sciences for solving PDEs.", "weaknesses": "The characterization of the scale of experiments in this paper, in contrast to previous work, should be made more precise. The abstract of this paper says that previous work \"scale only to simple toy examples\", which leads readers to expect that this paper deals with real-world problems on large domains. However, the experiments in this paper were conducted only on simple elliptic PDEs on meshes with up to thousands of elements, which appears to be on the same level of complexity and scale as previous work (e.g., AMR on simple hyperbolic equations with up to thousands of elements). The paper's Limitation section also acknowledges this fact, so the abstract and main paper should be made consistent.\n\nThe proposed approach seems to be extendable to include coarsening actions, but this is not demonstrated in this paper and hence it is hard to judge whether the proposed approach holds promise if extended. There is existing work that support coarsening (refs 28,29), so one would expect this to be a requirement for this application area. One would expect that an optimal strategy for the moving heat source problem requires coarsening actions, since regions that were refined due to proximity to the source may need to be coarsened once the source has moved away from them. \n\nExperimental evaluation includes randomization over domains, boundary conditions and initial conditions. However, for trained policies to be useful in applications to real-world problems, one should show that trained policies can work on larger meshes and longer step counts than those seen in training. The proposed method uses global features such as the number of mesh elements and environment time step, which this leads to concerns about what would happen if the trained policies are run on test cases where the number of elements and step count go out of the training distribution.\n\nIt was confusing to read the phrase \"PPO version of VDGN\" (line 245) and \"VDGN...produce better results with PPO\" (line 265), since PPO is a policy optimization method that is fundamentally different from value-based methods, so it is unclear how exactly PPO can be used in combination with VDGN. Upon looking closer, it appears that some of the learning-based baselines described in Section 4 do not match the methods in the previous work (refs 27,28), so it is misleading to label them as such. For the \"Argmax\" method, (ref 27) trains a stochastic policy on a discrete action space consisting of all the available elements, but the implementation in this paper \"predicts a continuous action for each mesh element.\" It is not clear what is that continuous action and what RL algorithm is used to train this baseline. For the baseline labeled as \"VDGN\", it appears that the implementation in this paper is an actually an on-policy policy optimization method based on PPO, using a decomposed value function. But that is fundamentally different from VDGN in (ref 28), which looks like an off-policy method based on Q-learning.\n\nAdditional suggestions:\n- line 37: \"shared observations\" imply that agents have the same observation, but that is not the case since each agent is a node in the graph and have different observations. Perhaps the authors mean to say agents have the same observation space.\n- It is hard to see what is gained by calling the formulation an \"Adaptive Swarm Markov Decision Process\", as opposed to the more standard terminology of a Dec-POMDP with parameter-sharing among agents.\n- line 257: not clear what is meant by \"holistic\" in this context. Perhaps the authors mean that RL can find optimal strategies for long-term objectives", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their detailed feedback, particularly noting the innovative use of swarm reinforcement learning for AMR and the comprehensive clarity and rigor with which we presented our findings. We now address individual concerns, including the clarification needed in the experimental section and the accurate representation of related work.\n\n> […] the experiments in this paper were conducted only on simple elliptic PDEs on meshes with up to thousands of elements, which appears to be on the same level of complexity and scale as previous work. […] the abstract and main paper should be made consistent\n> \n\nOur experiments use conforming meshes and more refinement levels than existing work. While the considered PDEs are relatively simple, the difficulty of finding good refinements for them is evidenced by the performance of the baseline methods. We provide further discussion in the general answer and will address this distinction in the revision. We also refer to Figure 2 of the accompanying PDF for a refinement with more than $50\\,000$ elements.\n\n> One would expect that an optimal strategy for the moving heat source problem requires coarsening actions […] Include empirical evidence that the approach extends to coarsening actions.\n> \n\nWe solve the Heat Diffusion equation in an inner loop and optimize the mesh with respect to the error of the last step of the equation as described in Appendix B.5. Respecting the movement of the heat source is still important due to the propagation of errors, and this task showcases that our method is able to create a single refined mesh that provides accurate results for the full simulation.\n\nWe agree that mesh coarsening is important for general AMR strategies and aim to explore this direction further in future work. We believe that iterative refinement from a coarse initial mesh is sufficient for the problems considered in this work, which we discuss in more detail in the general answer.\n\n> […] one should show that trained policies can work on larger meshes and longer step counts than those seen in training. The proposed method uses global features […] Demonstrate that trained policies generalize to larger meshes and more time steps than those in training.\n> \n\nFigure 2 of the general PDF shows that ASMR, with minor modifications to the training environment and observation space (excluding, in particular, the global information), can generalize to meshes with over $50\\,000$ elements on the Poisson problem, achieving a speedup of more than $100$ times when compared to $\\Omega^*$. We equate the initial size of elements in this larger mesh to those seen during training so that the six refinement steps that are used during training are sufficient.\n\n> It was confusing to read the phrase \"PPO version of VDGN\" (line 245) […] For the \"Argmax\" method, (ref 27) trains a stochastic policy on a discrete action space consisting of all the available elements […] Some of the learning-based baselines implemented in this paper do not accurately reflect the methods in the referenced papers […]\n> \n\nIn Figure 8 in Appendix D.1, we compare VDGN (Ref. 28) with DQN and our “VDQN+PPO” variant. The latter is a policy gradient method and can not directly apply the value decomposition in the Q function. However, since PPO uses a value function as a baseline to approximate the advantage function, we apply a similar value decomposition there (c.f. Line 245). We find in Figure 8 that VDGN (DQN) performs significantly worse than our PPO variant, which is why we use the latter for the main experiments in the paper.\n\n*Argmax* (Ref. 27) trains a discrete policy that chooses one of the mesh elements for refinement. This is realized via a categorical distribution over all elements, which is computed via a softmax over the policy outputs for each element. We experiment with both DQN and PPO for our *Argmax* baseline in Figure 8 in Appendix D.1. For the PPO version, we take the maximum over a continuous scalar per element (c.f. Line 239). Since the softmax is order-preserving, this corresponds to acting on a categorical distribution. We directly use a categorical distribution for the DQN variant and apologize that the paper does not state this clearly.\n\nIn summary, “VDGN+PPO” uses the value decomposition idea of VDGN for its advantage estimation, while the Argmax baseline essentially uses a categorical distribution to select the next element to be refined. We thank the reviewer for pointing out the issues with the phrasing and nomenclature for the baselines and will adapt both in the revision.\n\n> line 37: \"shared observations\" imply that agents have the same observation […]. Perhaps the authors mean to say agents have the same observation space.\n> \n\nWe agree that “shared observation space” in Line 37 is technically more correct. We originally chose to use “shared observations” here due to the overlapping receptive fields of the individual agents and will adapt this in the revision.\n\n> It is hard to see what is gained by calling the formulation an \"Adaptive Swarm Markov Decision Process\" […]\n> \n\nWe avoid the DEC-POMDP formulation due to agent-wise rewards, changing action and observation spaces and crucially mappings between the agents over time, all of which can be more naturally represented when formulating AMR as a swarm problem. We refer to the general answer for further details.\n\n> line 257: not clear what is meant by \"holistic\" in this context. Perhaps the authors mean that RL can find optimal strategies for long-term objectives\n> \n\n“Hollistic” refers to both the optimization of long-term objectives, and to finding an optimal refinement w.r.t. the full receptive field of the agent rather than just the element itself.\n\nWe hope that this answer addresses the concerns of the reviewer. If any reservations or questions about the paper remain, we encourage the reviewer to reach out to us for further clarification."}]

Author rebuttal:
We thank all reviewers for their valuable feedback on our submission. We are delighted to hear that the reviewers found our reward function and agent mapping innovative and useful (Tf6s), the speedup impressive (fwaD), the experiments convincing (ymhS), the improved stability of our method significant (LCc4), and the overall presentation to be of high quality and well-written (qjoP). We want to use this global response to address common concerns.

> Can the method generalize to larger and different problems? (Tf6s, LCc4, qjoP)

The most common concern raised by the reviewers was that of generalization to larger problems. We agree that it is crucial for the method to scale to larger (Tf6s, qjoP) and different (LCc4) problems and thus provide more significant speedups compared to the oracle heuristics.

In Appendix D.4, we illustrate our approach's effective generalization to new load functions and similarly sized domains. To generalize to larger domains, we adjust the observation space by removing global features (as suggested by qjoP), messages, and boundary node distances. We employ data augmentation: modifying training PDEs to mimic larger mesh segments by altering boundary conditions and load functions. The means of the load function are sampled from a centered unit Gaussian, allowing modes outside the mesh. We use domains with random holes (Appendix B.1) for varied initial meshes and apply random Gaussian loads to selected boundary parts as 'inlets'. See examples in Figure 1. We add an L2 norm of $3e$-$4$ to combat overfitting and omit the per-domain normalization of Lines 181-183. Due to time constraints, we only train for $200$ instead of $800$ iterations, reducing training time to a few hours on CPU. We emphasize that these changes only affect the observation space and the environment and leave the underlying algorithm unchanged. 

We evaluate the resulting policy on a larger, spiral-shaped domain. This domain lies in $(0,20)^2$ compared to the $(0,1)^2$ domains seen during training and uses initial meshes with comparable element sizes. ASMR creates highly refined meshes with tens of thousands of elements, as showcased in Figure 2 in the attached PDF. The shown mesh has $50236$ elements, which is multiple times larger than any refinement shown by an existing RL-based method. Creating and solving this mesh takes $10$-$12$ seconds, whereas computing a uniform reference on the same domain takes over $20$ minutes and significantly more memory, or more than $100$ times as long. The error compared to the uniform mesh lies at $3$-$4\,\%$, depending on the metric, making it useful for most engineering applications.

We want to thank the reviewers for pointing us towards these findings and will incorporate these insights into the paper revision. We believe that these additional results significantly strengthen the claims of our paper and the applicability of our method.

> Mesh coarsening, 3-D domains and more complex problems are important for this line of work. (Tf6s, fwaD, qjoP)

While mesh coarsening is an important aspect of AMR, is it not essential for stationary problems with a unique solution. In the present submission, we prioritize conforming refinements due to their increased precision and numerical stability over non-conforming refinements [1]. For these refinement types, coarsening is complex due to dependencies on neighboring elements, as briefly discussed in lines 159-162. We do not consider 3D meshes here, as including them would not contribute to our core methodology but plan including them in future work. We choose problems from diverse fields to evaluate the methods across different solutions, but keep the underlying PDEs simple to ensure manageable computational demand. The difficulty of finding good refinements is evidenced by the comparatively poor performance of existing RL-based methods as mentioned in the previous answer.

> How does the method improve over the baselines, and why are the proposed tasks difficult? (Tf6s, ymhS, qjoP)

Our experiments include various elliptic PDEs on conforming triangular meshes with up to $6$ refinement steps across different domains. In contrast, existing work utilizes simpler domains and non-conforming quadrilateral meshes, often with only $2$ refinement levels (Ref. 27, 28) or problems that can be refined from local information (Ref. 29). The challenges in our tasks arise from the need for attention on multiple parts of the mesh (e.g., the Poisson problem) and pinpointed refinements (e.g., Linear Elasticity problem). This complexity is evidenced by the baseline methods significantly underperforming ASMR. Further, ASMR can provide high-quality refinements for significantly larger instances than any existing RL-based method, as shown in Figure 2 of the provided PDF. We recognize the need to differentiate between PDE complexity and refinement intricacy (see also [1]) and will address this in the revised paper.

> Why use a SwarMDP instead of a Dec-POMDP? (LCc4, qjoP)

We adapt the SwarMDP framework (Ref. 31) to agent-wise rewards, changing action and observation spaces to accommodate the varying number of agents, and crucially to mappings between these varying agents over time. While related work (Ref. 28) fits the varying action and observation spaces into a DEC-POMDP through the use of dummy states, viewing the mesh as a swarm system is conceptually simpler, makes the permutation-equivariance of the agents explicit, and allows for a more natural integration of both the agent-dependent reward and the mapping between agents. For the revision, we will add a brief discussion on this framework and its advantages.

We want to thank the reviewers again for their time and feedback and appreciate further communication if any concerns remain or appear during the discussion period.

[1] Nagarajan, A., & Soghrati, S. (2018). Conforming to interface structured adaptive mesh refinement: 3D algorithm and implementation. *Computational Mechanics*

Meta-review: This paper presents a fairly novel approach to handling variable number of nodes in the meshing procedure using MARL and is an interesting contribution to the community. Furthermore, the results are extensively benchmarked and demonstrate an interesting scaling to larger examples. However, the reviewers were quite concerned that statements about the performance relative to the baselines did not adequately reflect that the baselines were intended for non-stationary PDEs and changes to the wording in the paper should be made to reflect this. We expect the author to make the following changes to the final paper
- The large scale generalization example and results on ZZ should be included
- It should be made clearer what the relationship between the methods in [27, 28] and the methods labeled Argmax and VGDN are.
- The related work section should make clear that the compared baselines are intended to work on non-stationary PDEs and the limitations of the current work wrt other classes of PDEs should be clear.
Decision: Accept (poster)