Learning to Discretize: Solving 1D Scalar Conservation Laws via Deep Reinforcement Learning
Authors: Yufei Wang*
Ziju Shen*
Zichao Long
Bin Dong
Venue:  (2020)
Keywords: Numerical Methods
Conservation Laws
Reinforcement Learning

Abstract: Conservation laws are considered to be fundamental laws of nature. It has broad application in many fields including physics, chemistry, biology, geology, and engineering. Solving the differential equations associated with conservation laws is a major branch in computational mathematics. Recent success of machine learning, especially deep learning, in areas such as computer vision and natural language processing, has attracted a lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods. In this paper, we are the first to explore the possibility and benefit of solving nonlinear conservation laws using deep reinforcement learning. As a proof of concept, we focus on 1-dimensional scalar conservation laws. We deploy the machinery of deep reinforcement learning to train a policy network that can decide on how the numerical solutions should be approximated in a sequential and spatial-temporal adaptive manner. We will show that the problem of solving conservation laws can be naturally viewed as a sequential decision making process and the numerical schemes learned in such a way can easily enforce long-term accuracy. 
Furthermore, the learned policy network is carefully designed to determine a good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach.
In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts. Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-art numerical solvers such as WENO schemes, and how well it generalizes. Our code is released anomynously at \url{https://github.com/qwerlanksdf/L2D}.

Under review as a conference paper at ICLR 2020
LEARNING TO DISCRETIZE: SOLVING ID SCALAR CONSERVATION LAWS VIA DEEP REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Conservation laws are considered to be fundamental laws of nature. It has broad application in many fields including physics, chemistry. biology, geology, and engineering. Solving the differential equations associated with conservation laws is a major branch in computational mathematics. Recent success of machine learning, especially deep learning, in areas such as computer vision and natural language processing, has attracted lot of attention from the community of computational mathematics and inspired many intriguing works in combining machine learning with traditional methods. In this paper, we are the first to view numerical PDE (deep) RL to learn new solvers. As a proof of concept,
solvers as MDP and we focus on 1-dimensional scalar conservation laws. We deploy the machinery of deep reinforcement learning to train policy network that can decide on how the numerical solutions should be approximated in sequential and spatial-tempore adaptive manner. We will show that the problem of solving conservation laws can be naturally viewed schemes learned in such a way can easily enforce long-term accuracy. Furthermore, the learned policy network is carefully designed to determine good local discrete approximation based on the current state of the solution, which essentially makes the proposed method a meta-learning approach. In other words, the proposed method is capable of learning how to discretize for a given situation mimicking human experts. Finally, we will provide details on how the policy network is trained, how well it performs compared with some state-of-the-ar numerical solvers such WENO schemes, and how well it generalizes. Our code is released anomynously at ht hhttps://githaa com/ qwer waw.ralnnssfff//
sequential decision making process and the numerical
INTRODUCTION
Conservation laws are considered to be one the fundamental laws nature, and has broad applications in multiple fields such physics, chemistry, biology, geology, and engineering. For example, Burger's equation, a very classic partial differential equation (PDE) in conservation laws, has important applications in fluid mechanics, nonlinear acoustics, gas dynamics, and traffic flow.
Solving the differential equations associated with conservation laws has been a major branch of computational mathematics (Leveque 1992 2002), and a lot of effective methods have been proposed, from classic methods such as the upwind scheme, the -ax-Fiiessssss scheme, to the advanced ones such as the ENO/WENO schemes (Liu et al. (Jerez Galiano & Uh Zapata| 2010), and etc. In the past few decades, these traditional methods have been proven successful in solving conservation laws. Nonetheless, the design of some of the high-end methods heavily relies on expert knowledge and the coding of these methods can be laborious process. To ease the usage and potentially improve these traditional algorithms, machine learning, especially deep learning, has been recently incorporated into this field. For example, the ENO scheme requires lots of 'if/else" logical judgments when used to solve complicated system of equations or nigh-dimensional equations. This very much resembles the old-fashionel expert systems. The recent trend in artificial intelligence (AI) is to replace the expert systems by the so-called "connectionism" e.g., deep neural networks, which leads the recent bloom of AI. Therefore,
1994 Shu 1998), the flux-limiter methods

---

Under review as a conference paper at ICLR 2020
is natural and potentially beneficial to introduce deep learning in traditional numerical solvers of conservation laws.
1.1 RELATED WORKS
In the last few years, neural networks (NNs) have been applied solving ODEs/PDEs or the associated inverse problems. These works can be roughly classified into three categories according to the way that the NN is used.
The first type of works propose to harness the representation power of NNs, and are irrelevant to the numerical discretization based methods. For example, Raissi al. 7a b): Yohai Bar-Sinai (2018) treated the NNs as new ansatz to approximate solutions of PDEs. It was later generalized by Wei al. (0199) to allow randomness in the solution which is trained using policy gradient. More recent works along this line include (Magiera et al. 2019 Michoski et al. 2019 Both et al. 2019). Besides, several works have focused on using NNs establish direct mappings between the parameters of the PDEs (e.g. the coefficient field or the ground state energy) and their associated solutions (Khoo et al.
Khoo & Ying. 2018 Li et al. 2019 Fan et al.) 2018b). Furthermore, Han al. (2018): Beck et al. 2017) proposed a method to solve very high-dimensional PDEs by converting the PDE to stochastic control problem and use NNs to approximate the gradient of the solution.
2017
The second type of works focus the connection between deep neural networks (DNNs) and dynamic systems (Weinan 2017] Chang et al., 2017: Lu et al. 2018 Long et al. 201 8b Chen et al. 2018). These works observed that there connections between DNNs and dynamic systems (e.g- differential equations or unrolled optimization algorithms) so that we can combine deep learning with traditional tools from applied and computational mathematics to handle challenging tasks in inverse problems (Long et al. 2018b a Qin et al. 2018). (018).Tho The main focus of these works, however, is to solve inverse problems, instead of learning numerical discretizations of differential equations. Nonetheless, these methods are closely related to numerical differential equations since learning a proper discretization is often an important auxiliary task for these methods to accurately recover the form of the differential equations.
The third type of works, which target using NNs learn new numerical schemes, are closely related to our work. However, we note that these works mainly fall in the setting of supervised learning (SL). For example, Discacciati et al. (2019) proposed to integrate NNs into high-order numerical solvers to predict artificial viscosity; Ray & Hesthaven (2018) trained multilayer perceptron to replace traditional indicators for identifying troubled-cells in high-resolution schemes for conservation laws. These works greatly advanced the development in machine learning based design of numerical schemes for conservation laws. Note that in Discacciati et al.] (2019), the authors only utilized the one-step error to train the artificial viscosity networks without taking into account the long- term accuracy of the learned numerical scheme. Ray & Hesthaven (201)) first constructed several functions with known regularities and then used them to train neural network to predict the location of discontinuity, which was later used to choose a proper slope limiter. Therefore, the training of the NNs is separated from the numerical scheme. Then, natural question is whether we can learn discretization of differential equations in an end-to-end fashion and the learned discrete scheme also takes long-term accuracy into account. This motivates us to employ reinforcement learning to learn good solvers for conservation laws.
1.2 OUR APPROACH
The main objective of this paper is to design new numerical schemes in an autonomous way. We propose to use reinforcement learning (RL) to aid the process of solving the conservation laws. To our best knowledge, we are the first to regard numerical PDE solvers as a MDP and to use (deep) RL to learn new solvers. We carefully design the proposed RL-based method so that the learned policy can generate high accuracy numerical schemes and can well generalize in varied situations. Details will be given in section 3
Here, we first provide a brief discussion on the benefits of using RL to solve conservation laws (the arguments apply to general evolution PDEs well):
Most of the numerical solvers of conservation law can be interpreted naturally as a sequential decision making process (e.g., the approximated grid values at the current time instance definitely
2

---

Under review as a conference paper at ICLR 2020
affects all the future approximations). Thus, it can be easily formulated as Markov Decision Process (MDP) and solved by RL.
In almost all the RL algorithms, the policy स (which is the Al agent who decides on how the solution should be approximated locally) is optimized with regards to the values Q= ag) r(se, Go) + _ 0o 7'r(st, at), which by defimition considers the long-term accumulated reward (or, error of the learned numerical scheme). thus could naturally guarantee the long-term accuracy of the learned schemes, instead of greedily deciding the local approximation which is the case for most numerical PDEs solvers. Furthermore, it can gracefully handle the cases when the action space is discrete, which is in fact one of the major strength of RL.
By optimizing towards long-term accuracy and effective exploration, believe that RL has good potential in improving traditional numerical schemes, especially in parts where no clear design principles exist. For example, although the WENO-5 scheme achieves optimal order of accuracy at smooth regions of the solution (Shu 1998), the best way of choosing templates near singularities remains unknown. Our belief that RL could shed lights on such parts is later verified in the experiments: the trained RL policy demonstrated new behaviours and is able to select better templates than WENO and hence approximate the solution better than WENO near singularities.
Non-smooth norms such as the infinity norm of the error is often used to evaluate the performance of the learned numerical schemes. As the norm of the error serves the loss function for the learning algorithms, computing the gradient of the infinity norm can be problematic for supervised learning, while RL does not have such problem since it does not explicitly take gradients of the loss function (i.e. the reward function for RL).
Learning the policy within the RL framework makes the algorithm meta-learning-like (Schmid) huber 1987 [Bngiio et al., 1992 Andrychowicz et al., 2016 & Malik 2016] Finn et 2017).
The learned policy can decide on which local numerical approximation to use by judging from the current state of the solution (e.g. local smoothness, oscillatory patterns, dissipation, etc). This is vastly different from regular (non-meta-) learning where the algorithms directly make inference on the numerical schemes without the aid of an additional network such as т. As subtle the difference as it may seem, meta- meta-learning--. like methods have been proven effective in various applications such as in image restoration (Jin et al.] 2017]] Fan et al., 20188] ZZhag et al. 2019). See (Vanschoren) 2018) for comprehensive survey on meta-learning.
Another purpose of this paper to raise an awareness of the connection between MDP and numerical PDE solvers, and the general idea of how to use RL to improve PDE solvers or even finding brand new ones. Furthermore, in computational mathematics, lot of numerical algorithms are sequential, and the computation at each step is expert-designed and usually greedy, e.g., the conjugate gradient method, the fast sweeping method (Zhao) 2005), matching pursuit (Mallat & Zhang 1993), etc. We hope our work could motivate more researches in combining RL and computational mathematics, and stimulate more exploration on using RL as a tool to tackle the bottleneck problems in computational mathematies.
Our paper is organized as follows. In section 2 we briefly review I-dimensional conservation laws and the WENO schemes. In section 3 discuss how to formulate the process of numerically solving conservation laws into a Markov Decision Process. Then, we present details on how to train a policy network to mimic human expert in choosing discrete schemes a ceatial-temporayy adaptive manner by learning upon WENO. In section 4 we conduct numerical experiments on 1-D conservation laws to demonstrate the performance our trained policy network. Our experimental results show that the trained policy network indeed learned to adaptively choose good discrete schemes that offer better results than the state-of-the-art WENO scheme which is 5th order accurate in space evidence that the proposed RL framework has
and 4th order accurate in time. This serves the potential to design high perforerform numerical schemes for conservation laws in a data-driven fashion. Furthermore, the learned policy network generalizes well other situations such different initial conditions, mesh sizes, temporal discrete schemes, etc. The paper ends with a conclusion in section 5 where possible future research directions are also discussed.
3

---

Under review as a conference paper at ICLR 2020
2 PRELIMINARIES
2.1 NOTATIONS
In this paper, we consider solving the following 1-D conservation laws:
(x, + f=(u(z, 0, a b, t E [0, u(x, un (2)..
For example, f = n² is the famous Burger's Equation. We discretize the (z, t)-plane by choosing mesh with spatial size A and temporal step size At, and define the discrete mesh points (rz, fn) by
2
b
T
3j - a jAr, in ndt with 0,1, ...,,
n
0,
1, ...y N
Ar
AI
We denote 5+ = I, + Ar/2 = a (j A). The finite difference methods will produce approximations Uj to the solution u(z,,tn, the given discrete mesh points. We denote point- wise values of the true solution to be un u(zz,t,,) and the true point- wise flux values to be f3 f(u(a,,t.)) (.)).
2.2 WENO WEIGHTED ESSENTIALLY NON-OSCILLATORY SCHEMES
WENO (Weighted Essentially Non-Oscillatory) (Liu et al.. 1994) is a family of high order accurate finite difference schemes for solving hyperbolic conservation laws, and has been successful for many practical problems. The key idea of WENO is nonlinear adaptive procedure that automatically chooses the smoothest local stencil to reconstruct the numerical flux. Generally, a finite difference method solves Eq by using a conservative approximation to the spatial derivative of the flux:
(ty(t) = - Ar dt
-11--11 1
-f
(2)
where (t) is the numerical approximation to the point value w(x,,t) and f++3 is the numerical flux generated by numerical flux policy
f (ug-r. Uj+a),
which is manually designed. Note that the term "numerical flux policy" is new terminology that we introduce in this paper, which is exactly the policy shall learn using RL. In WENO, n' works as follows. Using the physical flux values {fz-2, f--1,1 fz we could obtain a 3th order accurate polynomial interpolation fJtz. where the indices {j 2, 1.j] is called 'stencil'. We could also use the stencil fit 344 and /+ The key idea of WENO is average (with properly designed weights) all these interpolants to obtain the final reconstruction:
(i.)+1.++2} fj+1,j-2.j+3] to obtain another three interpolants
-+ Wr Et= W,. =
The weight wy depends on the smoothness of the stencil. general principal is: the smoother the stencil, the more accurate is the interpolant and hence the larger is the weight. To ensure convergence, we need the numerical scheme to be consistent and stable Veque 1992). It is known that WENO schemes described above are consistent. For stability, upwinding required in constructing the flux. The most easy way is to use the sign of the Roe speed āz++ = (f++ + //=+)/("/++ -/-
to determine the upwind direction: if ( 2 O, we only average among the three interpolants f
244 and i < O, we use f f and 7+4
Some further thoughts. WENO achieves optimal order of accuracy (up to 5) at the smooth region of the solutions (Shu 1998), while lower order of accuracy at singularities. The key of the WENO method lies in how to compute the weight vector (Wi, W2, W3, W44,, which primarily depends on the smoothness of the solution at local stencils. In WENO, such smoothness is characterized by handcrafted formula, and proven be successful in many practical problems when coupled with high-order temporal discretization. However, it remains unknown whether there are better ways to combine the stencils that optimal order of accuracy in smooth regions can be reserved while, the
4

---

Under review as a conference paper at ICLR 2020
same time, higher accuracy can be achieved near singularities. Furthermore, estimating the upwind directions is another key component of WENO, which can get quite complicated in high-dimensional situations and requires lots of logical judgments (i.e. "if/else"). Can ease the (some time painful) coding and improve the estimation at the aid of machine learning?
METHODS
In this section we present how to employ reinforcement learning to solve the conservation laws given by Eq [ To better illustrate idea, we first show in general how to formulate the process of numerically solving a conservation law into an MDP. We then discuss how to incorporate a policy network with the WENO scheme. Our policy network targets the following
key aspects of WENO: (1) Can we learn to choose better weights to combine the constructed fluxes? (2) Can learn to automatically judge the upwind direction, without complicated logical judgments?
3.1 MDP FORMULATION
Algorithm 1: A Conservation Law Solving Procedure
Input: initial values ug: ui, u... flux f()), Az, Át, evolve time N, left shift r and right shift Output: {U] O, O.... J, N) U 4. j O.....
for to N do
for = to
Compute the numerical flux 1 = -1 Uy UfFo-1) and l a (UT=
--r+1 UTF'' ), e.g., using the WENO scheme
Compute du, (1)
s+ ff-;)
Compute U = duylth
dt ), e.g., using the Euler scheme U3 = UU- + A dey
Return (U31) 0, ..., J, 1,... N}
As shown in Algorithm|] ! the procedure of numerically solving a conservation law is naturally a sequential decision making problem. The key of the procedure is the numerical flux policy πf and the temporal scheme as shown in line 6 and in Algorithm 1 Both policies could be learned using RL. However, in this paper, mainly focus using RL to learn the numerical flux policy πí, while leaving the temporal scheme x' with traditional numerical schemes such as the Euler scheme or the Runge-Kutta methods. A quick review of RL is given in the appendix.
Now, we show how to formulate the above procedure as an MDP and the construction of the state S, action A, reward and transition dynamics P. Algorithm 2 shows in general how RI is incorporated into the procedure. In Algorithm 2 we use a single RL agent. Specifically, when computing UT:
The state for the RL agent is 8i = gs 10
U.. Up-)), where gs is the state function.
In general, the action of the agent used to determine how the numerical fluxes fo and f is computed. In the next subsection, we detail how we incorporate az to be the linear weights of the fluxes computed using different stencils in the WENO scheme.
The reward should encourage the agent to generate a scheme that minimizes the error between its approximated value and the true value. Therefore, we define the reward function as ry g. (U, wf- Un+ - -uj+e). e.g., a simplest choice gr = -Il 1|2-
The transition dynamies P is fully deterministic, and depends on the choice of the temporal scheme at line 10 in Algorithm 2 Note that the next state can only be constructed when we have obtained all the point values in the next time step, i.e., s7 n+1
U does not only depends on action a", but also on actions d 0'+ (action a can only determine the value U.. This subtlety can be resolved by viewing the process under the framework of multi-agent RL, in which at each mesh point we use a distinct agent AftL, and the next state sp11 (UU ... U)+,) depends these agents' joint action af (a"---1--.,, a.
gs (U
5

---

Under review as a conference paper at ICLR 2020
However, it is impractical to train J different agents as J is usually very large, therefore enforce the agents at different mesh point j to share the same weight, which reduces to case of using just a single agent. The single agent can be viewed the choice of a local scheme based on the current state in traditional numerical methods.
counterpart of human designer who decides on
Algorithm 2: General RL Running Procedure
Input: initial values why n², flux f(u), Ax, At, evolve time N, left shift right shift and RL policy n R.. Output: {4, j O, J, N)
U u j= JJ... for Many iterations do Construct initial states of - gr 9U( Uf+s) )) for = O.... for n 1 to N do
for 2 0 Jo do
Compute the action a. = = RLi (8}) that determines how fj44 and fJ- is computed
Compute dey(()
3n (f,7_ f7-1)
Computc U (U7-) du du,(0)
c.g., the Euler scheme UT Up-1 At du,(0)
10
Compute the reward rj (U -U u - U Wjtts
Construct the next states sj n+1 go (2 ..., v++) for j = O,...,J J
12
Use any RI. algorithm to train the RI. policy n nt with the transitions {(57, ag 13,2)))0000
13
Return the well-trained RL policy π RT.
14
3.2 RL EMPOWERED WENO
We now present how to transfer the actions of the RL policy the weights of WENO fluxes. Instead of directly using RL to generate the numerical flux, we use it to produce the weights of numerical fluxes computed using different stencils WENO. Since the weights are part of the configurations of the WENO scheme, our design of action essentially makes the RL policy meta-leamee, and enables more stable learning and better generalization power than directly generating the fluxes.
Specifically, at point 2j (here we drop the time superscript n for simplicity), to compute the numerical flux 4 and J first construct four fluxes { =)= and (5i+ + 1 using four different
stencils just in WENO, and then the RL policy RL to generate the weights of these fluxes:
π (s,))
(w w د -ー -
w J
w w
w/ )
3+
5+
= The numerical flux is then constructed by averaging these fluxes: f--==E-- Ei- wo - and
f+t = ! " Z 3+3'
Note that the determination of upwind direction is automatically embedded in the RL policy since it generates four weights at once. For instance, when the roe speed āj+! 2 O, we expect the 4th
weight } Rs 0 and when } < O, we expect w / = 0. Note that the upwind direction can be very complicated in system of equations or in the high-dimensional situations, and using the policy network to automatically embed such a process could save lots of efforts in algorithm design and implementation. Our numerical experiments show that RRL can indeed automatically determine upwind directions for ID scalar cases. Although this does not mean that it works for systems and/or in high- dimensions, shows the potential of the proposed framework and value for further studies.
4 EXPERIMENTS
In this section, we describe training and testing of the proposed RL conservation law solver and compare it with WENO. More comparisons and discussions can be found in the appendix.

---

Under review as a conference paper at ICLR 2020
4.1 SETUP
In this subsection, we explain the general training setup. We train the RL policy network on the the left-shift
Burger's equation, whose flux is computed as f(u) fu². In all the experiments, will generate two r = 2 and the right shift & = 3. The state function (8,) Is (U Us+s)
vectors: s² := (f-r-1, .... fj+ 4 and s" (fj-r: .... fn-s: ( for computing fz- and fe+ respectively. 5s and 8, will be passed into the same policy neural network Th to produce the desired actions, described in section 3.2 The reward function 9, simply computes the infinity norm, i.e., Dr (Uz-r-1 Mj-r-1,---- Ujts U5++) --))===-(())) -I((, -1-r- Ujts "i++>
The policy network T. RL is a forward Multi-layer Perceptron with 6 hidden layers, each has 64 neurons and use Relu (Goodfellow al. 2 as the activation function. We use the Deep Deterministic Policy Gradient Algorithm (Lillicrap et al. 2015) to train the RL policy.
To guarantee the generalization power of the trained RL agent, randomly sampled 20 initial conditions in the form (z) = a + b func() where |a] + |b| S 3.5, func E {sin, cos) and E (2, 4, 6). The goal of generating such kind of initial conditions is ensure they have similar degree of smoothness and thus similar level of difficulty in learning. The computation domain is -1 1 and 0 S I 0.8 with Az 0.02, At 0.004, and evolve steps N 200 (which ensures the appearance of shocks). When training the RL agent, we use the Euler scheme for temporal discretization. The true solution needed for reward computing generated using WENO on the same computation domain with Ar = 0.001. At = 0.0002 and the 4th order Runge-Kutee Kutta (RK4).
In the following, we denote the policy network that generates the weights the WENO fluxes (as described in section 3.2 as RL-WENO. We randomly generated another different 10 initial conditions in the same form as traminng for testing.
A
0.02
0.04
0.05
RL-WENO
WENO
RL-WENO
WENO
RL-WENO
WENO
0.002
5,66 (1,59)
5,89 (1.74)
8.76 (2.50)
9.09 (2.62)
9.71(2.42)
10.24 (2.84)
0,003
5,64 (1.54)
5,86 (1.67)
8.73 (2.46)
9.06 (2.58)
9.75 (2.41)
10.28 (2,81)
0.004
5.63 (1.55)
5.81 (1.66)
8.72 (2.46)
9.05 (2.55)
9.61 (2.42)
10.13 (2.84)
0,005
5,08 (1.46)
5.19 (1.58)
8.29 (2.34)
8.58 (2.47)
9.30 (2.06)
9.78 (2.69)
0.006
8.71 (2.49)
9.02 (2.61)
9.72 (2.38)
10.24 (2.80)
0.007
(2.49)
8.84 (2.62)
9.59 (2.41)
(2.80)
0.008
8.68 (2.55)
8.93 (2.66)
9.57 (2,49)
10.06 (2.92)
Table 1: Comparison of relative errors (x10-3) of RL-WENO and WENO with standard deviations of the errors among 10 trials in the parenthesis. Temporal discretization: RK4; flux function: 1 u
Ax
0.02
0.04
0.05
At
WL-WENO
WENO
RI-WENO
WENO
RL-WENO
WENO
0.002
4.85 (1,15)
5.17 (1.2()
(..7(1.95)
8.05 (2.02)
(1.9))
8.56 (2.19)
0.003
7.79 (1.96)
8.06 (2.03)
8.18 (1.92)
8.59 (2.18)
0.004
(..9(1.93)
7.98 (2.01)
8.15 (1.95)
8.54 (2.20)
0.005
8.18 (1.94)
8.55 (2.15)
Table Comparison of relative errors (x10-2) of RL-WENO and WENO with standard deviations of the errors among 10 trials in the parenthesis. Temporal discretization: RK4; flux function: 16 u².
4.2 RESULTS
We compare the performance of RL-WENO and WENO. We also test whether the trained RL policy generalize different temporal discretization schemes, mesh sizes and flux functions that not included in training. Table I and Table 2 Treseep present the comparison results, where the number shows the relative error (computed as IUlulla with the 2-norm taking over all 2)) between the approximated solution U and the true solution u, averaged over 250 evolving steps (T = 1.0) and 10 random initial values. Numbers in the bracket shows the standard deviation over the 10 initial conditions. Several entries in the table marked '_' because the corresponding CFL number is not small enough
7

---

Under review as a conference paper at ICLR 2020
to guarantee convergence. Recall that training of the RL-WENO was conducted with Euler time discretization, (Az, At) = (0.02, 0.004), T = 0.8 and f(u) = u²²
Our experimental results show that, compared with the high order accurate WENO (5th order accurate in space and 4th order accurate in time), the linear weights learned by RL not only achieves smaller errors, but also generalizes well to: 1) longer evolving time (7 0.8 for training and T 1.0 for (esting): 2) new time discretization schemes (trained on Euler, tested on RK4); 3) new mesh sizes (see Table and Table 2 for results of varied Ar and At); and 4) new flux function (trained on f(u)- u shown in Table 1
tested on TŌ u² Table 2
Figure I shows some examples of the solutions. As one can see, the solutions generated by RI.-WENO not only achieve the same accuracy as WENO at smooth regions, but also have clear advantage over WENO near singularities which is particularly challenging for numerical PDE solvers and important in applications. Figure 2 shows that the learned numerical flux policy can indeed correctly determine upwind directions and generate local numerical schemes in an adaptive fashion. More interestingly, Figure 2 further shows that comparing to WENO, RL-WENO seems to be able to select stencils in a different way from it, and eventually leads to a more accurate solution. This shows that the proposed RL framework has the potential to surpass human experts in designing numerical schemes for conservation laws.
SACO MIH TED
A- TACE
B- LOG TAR
(a) f(u) = 1 2
(b) (wu) = 2
(c) [ 16 1
(d) f()) 16 4
(e) fu) = 1 2
(f) fu) = 2 u
(g) f(u) = 16 u
(h) f(u) = 16 u
Figure 1: First row: solutions of RL-WENT (red), WENO (blue) and exact solutions (green). Second row: zoom-in views corresponding to the first row.
(2 Si bou latt D1 Mu-
asceall
-100
(a) Weights of f
(b) Weights of sz+
Figure 2: This figure compares the weights generated by the learned numerical flux policy π. RT. and those of WENO. The weights shown in (a) are {w" fuj-ghr- 1/--2: 2: while those in (b) are [ }/--2- In each of the two plots, the 4 numbers in the upper bracket of each location are the weights of RL- WENO and those in the lower bracket are the weights of WENO. The relative errors of RL-WENO and WENO are 8.0 x 10-3 and 2.5 x 10-2 respectivell.
8

---

Under review as a conference paper at ICLR 2020
5 CONCLUSION
In this paper, we proposed a general framework to learn how to solve 1-dimensional conservation laws via deep reinforcement learning. We first discussed how the procedure numerically solving conservation laws can be naturally cast in the form of Markov Decision Process. We then elaborated how relate notions numerical schemes of PDEs with those of reinforeement learning. In particular, we introduced a numerical flux policy which was able to decide on how numerical flux should be designed locally based on the current state of the solution. of our RL policy to make it meta-learner. Our numerical experiments showed that the proposed RI based solver was able outperform high order WENO and was well generalized in various cases.
carefully design the action
As part of the future works, we would like to consider using the numerical flux policy to inference more complicated numerical fluxes with guaranteed consistency and stability. Furthermore, we can use the proposed framework to learn policy that can generate adaptive grids and the associated numerical schemes. Lastly, we would like consider system of conservation laws in 2nd and 3rd dimensional space.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Marcin Andrychowiez, Misha Denil, Sergio Gomez, Matthew W Hoffman, David Pfau, Tom Schaul, Brendan Shillingford, and Nando De Freitas. Learning to learn by gradient descent by gradient descent. In Advances in Neural Information Processing Systems, pp. 001399883999 2016.
Christian Beck, E Weinan, and Arnuff Jentzen. Machine learning approximation algorithms for high- dimensional fully nonlinear partial differential equations and second-order backward stochastic differential equations. Journal of Nonlinear Science, pp. 1-57, 2017.
Samy Bengio, Yoshua Bengio, Jocelyn Cloutier, and Jan Gecsei. On the optimization of a synaptic learning rule. In Preprints Conf. Optimality in Artificial and Biological Neural Networks, pp. 6-8. Univ. of Texas, 1992.
Gert-Jan Both, Subham Choudhury, Pierre Sens, and Remy Kusters, Deepmod: Deep learning for model discovery in noisy data. arXiv preprint arXiv: 1904,09406, 2019.
Bo Chang, Lili Meng, Eldad Haber, Frederick Tung, and David Begert. Multi-level residual networks from dynamical systems view. arXiv preprint arXiv: 1.v::1710 1710. 10348, 2017.
Tian Qi Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differen- tial equations. In Advances in Neural Information Processing Systems, pp. 657 2571-6583, 2018.
Niccolo' Discacciati, Jan S Hesthaven, and Deep Ray. Controlling oscillations in high-order discon- tinuous galerkin schemes using artificial viscosity tuned by neural networks. Technical report, 2019.
Qingnan Fan, Dongdong Chen, Lu Yuan, Gang Hua, Nenghai Yu, and Baoquan Chen. Decouple learning for parameterized image operators. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 442-458, 2018a.
Yuwei Fan, Lin Lin, Lexing Ying, and Leonardo Néneda-Nónez. A multiscale neural network based on hierarchical matrices. arXiv preprint arXiv; 11XXivv1807.01880 80888883. 2018b.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Proceedings the 34th International Conference on Machine Learning- Volume 70, pp. 126 135. JMLR. org. 2017.
Ian Goodfellow, Yoshua Bengio, and Aaron Courville. Deep learning. MIT press, 2016.
Jiequn Han, Aruuf Jentzen, and E Weinan. Solving high-dimensional partial differential equations using deep learning. Proceedings the National Academy of Sciences, 115(34):8055-5500 2018.
Silvia Jerez Galiano and Miguel Uh Zapata. A new tvd flux-limiter method for solving nonlinear hyperbolic equations. Journal of Computational and Applied Mathematics, 234(5):1395-1403 2010.
Meiguang Jin, Stefan Roth, and Paolo Favaro. Noise-blind image deblurring, In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3510-3518, 2017.
Yuehaw Khoo and Lexing Ying. Switchnet: a neural network model for forward and inverse scattering problems. arXiv preprint arXiv:1810.09665. 2018
Yuehaw Khoo, Jianfeng Lu, and Lexing Ying. Solving parametric pde problems with artificial neural networks. arXiv preprint arXiv: 170700331,,, 2017.
Randall J Veque. Numerical methods for conservation laws, volume 132. Springer, 1992.
Randall Le Veque. Finite volume methods for hyperbolic problems, volume 31. Cambridge univer- sity press, 2002.
Ke Li and Jitendra Malik. Learning optimize. arXiv preprint X 1606.0186...8 2016.
10

---

Under review as a conference paper at ICLR 2020
Yingzhou Li, Jianfeng Lu, and Anqi Mao. Variational training of neural network approximations of solution maps for physical models. arXiv preprint uuXi:1905.027789 2019.
Timothy P Lillicrap, Jonathan J Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint
2015.
Xu-Dong Liu, Stanley Osher, and Tony Chan. Weighted essentially non-oscillatory schemes. Journal of computational physics, 115(1):00-212, 1994.
Zichao Long, Yiping Lu, and Bin Dong. Pde-net 2.0: Learning pdes from data with a numeric- symbolic hybrid deep network. arXiv preprint arXiv:1812.04422. 2018a.
Zichao Long, Yiping Lu, Xianzhong Ma, and Bin Dong. Pde-net: Learning pdes from data. In ICML, 2018b.
Yiping Lu, Aoxiao Zhong, Quanzheng Li, and Bin Dong. Beyond finite layer neural networks:
Bridging deep architectures and numerical differential equations. In ICML, 2018.
Jim Magiera, Deep Ray, Jan S Hesthaven, and Christian Rohde. Constraint-aware neural networks for riemann problems. arXiv preprint arXiv: 1ett..904...9 2019.
Stéphane G Mallat and Zhifeng Zhang. Matching pursuits with time-frequency dictionaries. IEEE Transactions signal processing, 41(1)):397-3441 1993.
Craig Michoski, Milos Milosavljevic, Todd Oliver, and David Hatch. Solving irregular and data- enriched diffcrential equations using deep neural networks. arXiv preprint urXiv:1905.04p3: 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Mare G Bellemare. Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. Nature, 518(7540):529 533, 2015.
Tong Qin, Kailiang Wu, and Dongbin Xiu. Data driven governing equations approximation using deep neural networks. arXiv preprint arXiv: 181 181 11111...55 1.05537, 2018.
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part i): Data-driven solutions of nonlinear partial differential equations. arXiv preprint 1711:1711..1551
2017a
Maziar Raissi, Paris Perdikaris, and George Em Karniadakis. Physics informed deep learning (part ii): Data-driven discovery of nonlinear partial differential equations. arXiv preprint arXiv: 1711:1111. 2017b.
Deep Ray and Jan Hesthaven. An artificial neural network as a troubled-cell indicator. Journal of Computational Physies, 367:166- 191.1666-199110 2018.
Jurgen Schmidhuber. Evolutionary principles in self-referential learning. On learning how learn: The meta-meta-.. hook.) Diploma thesis, Institut f. Informatik, Tech. Univ. Munich, 1987.
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. Trust region policy optimization. In Proceedings of the 32nd International Conference on Machine Learning (ICML-15), pp. 1889-1897, 2015.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 110717704777771 2017.
Chi-Wang Wang Shu. Essentially non-oseillatory and weighted essentially non-oscillatory schemes for hyperbolic conservation laws. In Advanced numerical approximation of nonlinear hyperbolic equations, pp. 432. Springer, 1998.
David Silver, Aja Huang, Chris J Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittwieser, loannis Antonoglou, Veda Panneershelvam, Mare Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529(7587):484 489, 2016.
11

---

Under review as a conference paper at ICLR 2020
Joaquin Vanschoren. Meta-learning: A survey, arXiv preprint 181.00354,. 2018.
Shiyin Wei, Xiaowei Jin, and Hui Li. General solutions for nonlinear differential equations: rule-based self-learning approach using deep reinforcement learning. Computational Mechanics, pp. 1-14, 2019.
Weinan. proposal on machine learning via dynamical systems. Communications in Mathematics and Statistics, 5(1):1-11, 2017.
Jason Hickey Michael P. Brenner Yohai Bar-Sinai, Stephan Hoyer. Data-driven discretization: method for systematic coarse graining of partial diffcrential equations. arXiv preprint trXiv:1808.049330 2018.
Xiaoshuai Zhang, Yiping Lu, Jiaying Liu, and Bin Dong. Dynamically unfolding recurrent restorer: A moving endpoint control method for image restoration. ICLR, 2019.
Hongkai Zhao. A fast sweeping method for eikonal equations. Mathematics of computation, 74 (250))603-627, 2005.
12

---

Under review as a conference paper at ICLR 2020
A COMPLEMENTARY EXPERIMENTS
A.1 COMPARISON WITH SUPERVISED LEARNING (SL) BASED METHODS
We first note that most of the neural network based numerical PDE solvers cited in the introduction requires retraining when the initialization, terminal time, or the form the PDE is changed; while the proposed RL solver is much less restricted shown in our numerical experiments. This makes proper comparisons between existing NN-based solvers and our proposed solver very difficult. Therefore, to demonstrate the advantage of our proposed RL PDE solver, would like to propose a new SL method that does not require retraining when the test setting (e.g. initialization, flux function, etc.) is different from the training.
However, as far as we are concerned, it is challenging to design such SL methods without formulating the problem into an MDP. One may think that we can use WENO generate the weights for the stencil at particular grid point on a dense grid, and use the weights of WENO generated from the dense grid as the label train a neural network in the coarse grid. But such setting has a fatal flaw in that the stencils computed in the dense grids are very different from those in the coarse grids, especially near singularities. Therefore, good weights on dense grids might perform very poorly on coarse grids. In other words, simple imitation of WENO on dense grids is not a good idea. One might also argue that instead of learning the weights of the stencils, we could instead generate the discrete operators, such as the spatial discretization of dug Us. or the temporal discretization of dug the numerical fluxes / (u), f (u), etc., on a dense grid, and then them as labels to train neural network in the supervised fashion on coarse grid. However, the major problem with such design is that there is no guarantee that the learned discrete operators obey the conservation property of the equations, and thus they may also generalize very poorly.
After formulating the problem into a MDP, there is indeed one way that we can use tuck-propagation (BP) instead of RI. algorithms to optimize the policy network. Because all the computations on using the stencils to calculate the next-step approximations are differentiable, we can indeed use SL to train the weights. One possible way is to minimize the error (c.g. 2 norm) between the approximated and a fine
the true values, where the true value is pre-computed using more accurate discretization mesh. The framework to train the SL network is described in Algorithm 3 Note that the framework to train the SL network is essentially the same that of the proposed RL-WENG (Algorithm ( The only difference is that we train the SL network using BP and the RL network using DDPG.
Algorithm 3: Using BP instead of RL algorithm to train the policy
Input: initial values up .., u9, flux fu), Az, At, evolve time N, left shift r, right shift and a neural network T
Output: = 0, n N)...NN U. ug, O....,
for Many iterations do Construct initial states 5 = 9. (. U9th) jor = O....
for to N do
for 1 do
Compute the weights (w=:
1- wn: 1 5-
w 0.0
w" n1
:": n--2
w": m.-1
w
3+ 3. wn.1
x x0()))
f-
5-
s+
s+
Compute the fluxes f 1 Σ1 Di=
54 Di= w + s+
pn where prug are
the fluxes computed by WENO
10
Compute de
do;(t) Dr
1 (f 1
f7
}
Compute UT = (UT-
duy dt
e.g., the Euler scheme UT = U - At dey(0)
11
12
Compute the loss for (0) |1(0) uj -,
Un Perform a gradient descent on w.rt (8)
us+s) s) -U, (U7-r-1 - uz - Uf+ u3
13
Construct the next states of g, (u r- -- I.... u}+.) for j O....
15
Return the BP optimized policy "
13

---

Under review as a conference paper at ICLR 2020
However, argue that the main drawback of using SL (BP) optimize the stencils in such way is that it cannot enforce long-ter term accuracy and thus cannot outperform the proposed RL-WENO. To support such claims, we have added experiments using SL to train the weights of the stencils, and the results are shown in table and 4 The SL. policy is trained till achieves very low loss (i.e., converges) in the training setting. However, shown in the table, the SI-trainedd policy does not perform well overall. To improve longer time stability, one may argue that we could design the loss of SL to be the accumulated loss over multiple prediction steps, but in practice as the dynamics of our problem (computations for obtaining multiple step approximations) is highly non-linear, thus the gradient flow through multiple steps can be highly numerically unstable, making it difficult obtain decent result.
0.02
0.04
0.05
A1
RE-WENO
SL.
WEND
BL-WEND
SE
WENO
KLWENO
SL.
WEND
(1.29)
7 (3)
(1.94
12.48 (0.78)
9.09
9.75 (271
0.63
(1.51)
7.37 (1.26)
(1.67)
(2.4
12.44 (0.78)
9.06 (2.58)
12.13 (0.41)
10.28 (2.81)
(1.55)
(1.24
(1.00)
(2.46)
12:44
(1.42)
(0.45)
(2.844
(1.46)
(1.3
(1.58
8.20 (2.34)
12.06 (0.86)
8.58 (2.47)
(2.26
(0.38)
9.78
0106
8.71 (2.49)
12.38 (0.73)
9.02 (2.61)
9.72 (2.38)
12:14 (0.43)
(2.800
(0.83)
(2.41)
0.088
8.68 (2.55)
12.22 (0.70)
8.93 (2.66)
9.57 (2.49)
12.08 (0.40)
10.06 (2.92)
Table Comparison of relative errors (x10-3) of RL-WENO, WENO, and SL-trained policy with standard deviations of the enrors among 10 trials in the parenthesis. Temporal discretization: RK4; flux function: 2 u² RL-weno consistently outperforms WENO and SL-trained policy in all test cases.
A
0,02
0.44
WE-WENO (1.3)
ST.
WLED
SI.
RLWWENO
ST.
0.022
(8.70.99
2617(1261
(.7)
(L12
K05 025(200)
(16
(1.00000
8,56
(2.612
7.70
(2.18)
0104
(12)
(.5
(201
(L.5
8.50 C2.30
0.065
(..((.
(1.0(
8.55
Table 4: Comparison of relative errors (x10 3) of RE-WENO, WENO, and SL-trained policy with
standard deviations of the errors among 10 trials in the parenthesis. Temporal discretization: RK4; flux function: ! " RL-weno consistently outperforms WENO and SL-trained policy in all test cases.
A.2 RL-WENO'S PERFORMANCE ON SMOOTH AND SINGULAR REGIONS
As mentioned in section 2.2 WENO itself already achieves an optimal order of accuracy in the smooth regions. Since RL-WENO can further improve upon WENO, it must have obtained higher accuracy especially near singularities. Here we provide additional demonstrations on how RI--WENO performs in the smooth/singular regions. We run RL-WENO and WENO on set of initial conditions, and record the approximation errors at every locations and then separate the errors the smooth and singular regions for every time step. We then compute the distribution of the errors on the entire spatial-temporal grids with multiple initial conditions. The results are shown in figure 3 In figure 3 the r-axis is the logarithmic (base 10) value of the error and the y-axis is the number of grid points whose error is less than the corresponding value on the r-axis, i.e., the accumulated distribution of the errors. The results show that RI. RI-WENO indeed performs better than WENO near singularities. RL-WENO even achieves better accuracy than WENO the smooth region when the flux function is 16 4
A.3 INFERENCE TIME RL-WENO AND WENO
In this subsection we report the inference time of RL-WEEN and WENO. Although the computation complexity the trained RL policy (a MIP) higher than that of WENO, we could parallel and accelerate the computations using GPU.
Our test is conducted in the following way: for each grid size Ax, we fix the initial condition as uo(x)) 1 + cos(66æ)) the evolving time T 0.8 and the flux function f =². We then use RL- WENO and WENO to solve the problem 20 times, and report the average running time. For completeness, also report the relative error of RL-WENO and WENO in each of these grid sizes in table Note that the relative error computed on average of several initial functions, and our RL--WENO policy is only trained grid (Ar, At) (0.02, 0.004).
14

---

Under review as a conference paper at ICLR 2020
Smooth Reg ons
Near Singulontes
WEND WENO
WENG
4000
3000
1000
3
2
1
(a) Smooth regions, f(u) 2
(b) Near singularities, f(u) = u2²
Smooth Bag ons
Near Singuiontes
WEND AL-WEND
RL-WENC
2500
1900
1000
500
-5
2
-1
-0
(c) Smooth regions, f(u) u
(d) Near singularities, f(u)
1 u
Figure 3: These figures show the total number of grids whose error is under a specific value (i.e. the accumulated distribution function). The æ--axi is the error in logarithmic (base 10) scale. (a) and (c) show the distribution in smooth regions, (b) and (d) are near singularities.
For RL-WENO, test it both CPU and GPU; For WENO, test it purely on CPU, with well-optimized version (e.g., good numpy vectorization in python), and a poor-implemented version (e.g., vectorization, lots of loops). The CPU used for the tests is custom Intel CORE i7, and the GPU is a custom NVIDIA GTX 1080. The results are shown in 5
(Ax, At)
RL- RL-WENO(CPU)
RL-WENO(GPU)
WENO-optimized
WENO-poor
(0.02, 0.004)
2,490
1.650
0.148
2.739
(0.01, 0.002)
7.720
1.700
0.349
10.778
(0.005, 0.001)
26.70
1.628
0.921
44.23
(0.002, 0.0004)
110.92
1.611
1.961
277.88
Table 5: Average inference time (in seconds) for RL-WENG and WENO. Bold numbers the smallest ones.
(Ar, AC)
RL-WENO error
WENO error
(0.02, 0.004)
3.73(0.40)
4.08(0.23)
(0.01, 0.002)
1.86(0.17)
1.99(0.12)
(0.005, 0.001) (0.002, 0.0004)
1.00(0.05)
0.93(0.01)
0.48(0.03)
0.39(0.02)
Table 6: Relative error of RL-WENO and WENO (x10-3, on grid sizes tested table Note RL-WENO is only trained on grid (Ar, At) (0.02, 0.004)
From the table for the RL-WENO (GPU), requires significant more time to finish the computation. The reason that the time cost of the GPU-version of RI--WENO does grow is that GPU, compute
tell that Az decreases, i.e., the grid becomes denser, all methods, except
15

---

Under review as a conference paper at ICLR 2020
all approximations in the next step (i.e., to compute (U)+1 "U."!....UU"" ....Uy!!) given (U). Uf:.... Ff....,U), UT). which dominates the computation cost of the algorithm) together in parallel. Thus, the increase of grids does not affect much of the computation time. Therefore, for coarse grid, well-optimized WENO indeed has clear speed advantage over RI-WENO (even on GPU), but on a much denser grid, RI.-WENO (GPU) be faster than well-optimized WENO by leveraging the paralleling nature of the algorithm.
B REVIEW OF REINFORCEMENT LEARNING
B. REINFORCEMENT LEARNING
Reinforcement Learning (RL) a general framework for solving sequential decision making problems. Recently, combined with deep neural networks, RL has achieved great success in various tasks such as playing video games from raw screen inputs (Mnih et al. 2015), playing Go (Silver et al. 2016), and robotics control (Schulman et al. 2017). The sequential decision making problem RL tackles is usually formulated as a Markov Decision Process (MDP), which comprises five elements: the state space S, the action space A, the reward S x A + R, the transition probability of the environment P: x A x S [0, 1]. and the discounting factor ạ. The interactions between an RL agent and the environment forms trajectory (so, dg, To: ST, sum of all its future rewards:
ST,aT; ar, r...... The return of 7 is the discounted
06 G(T)
Σ YTT
1=0
Similarly, the return a state-action pair (st, ar) is:
Gl a,)
policy æ in RL is probability distribution on the action A given state S: T: : A We say a trajectory J is generated under policy ı if all the actions along the trajectory is chosen following π, i.e., means at TT([s;) and st+1 ~ P(-|st, oz). Given a policy x, the value of state defined as the expected return of all the trajectories when the agent starts at and then follows m:
V (s) = E, (so) = =,T T
Similarly, the value a state-action pair is defined as the expected return of all trajectories when agent starts at s, takes action o, and then follows n.
Q" (s) F., (G(T)) = 8, ((a) a.
As aforementioned in introduction, most RL algorithms the policy æ is optimized with regards to the values Q (s) thus naturally guarantees the long-term accumulated rewards (in setting, the long-term accuracy of the learned schemes). Bellman Equation, one of the most important equations RL, connects the value state and the value of its successor state:
(s, r(s,a) + + P (*())) (a-a(())) ( +TV=(s')
The goal of RL is to find policy ग to maximize the expected discounted sum of rewards starting from the initial state Sp, ((x) = Es-~p[V**ss) where p the initial state distribution. parameterize ı using 0, then we can optimize it using the famous policy gradient theorem:
dJ((o) d0
Exrup" aroto (al)g)(()>s)(=*0
where nT is the state distribution deduced by the policy Ty. In this paper we focus on the case where the action space A is continuous, and lot of mature algorithms has been proposed for such a case, e.g., the Deep Deterministic Policy Gradient (DDPG) (Lillicrap et al. 2015), the Trust Region Policy Optimization algorithm (Schulman al. 2015), and etc.
16


Reviews:
[{"review_id": "rJekPAOaFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to use reinforcement learning for constructing discretziation stencils of numerical schemes. More specifically, the method focuses on the widely used WENO schemes, which are an established class of finite difference schemes. Within this context, the method aims for training models to infer the weighting for a specific stencil with eight flux terms.\n\nFor RL this task requires a continuous action space, and the DDPG algorithm is used for training the policy. The network itself is an MLP with 6 layers, and ca. 20000 weights in total. This is a significant number, given the focus on 1D problems.\n\nThe tests are quite thorough and interesting, while at the same time being limited in scope. The paper targets 1D cases, which make the problem very low-dimensional. Despite the simplicity, only a single data set (Burgers) is used, and a single modified target function with a u^4 term. Targeting 1D casesl, I would have expected a broader range of tests and model equations.\n\nDespite the limited scope of the models, table 1 and 2 assess a nice range of different timestep and discretization parameters. I found it very interesting to see that the method consistently outperforms the regular WENO scheme. The gains are relatively small, with 4-5%, but WENO already represents a quite accurate scheme, so it's surely not easy to outperform it.\n\nWhile reading the paper, I was wondering about the bigger picture, i.e. using RL in the context of discretization stencils. We have model equations, and discretized versions of all operators involved in training. Why employ a \"brute force\" approach like RL here? Wouldn't it be better in terms of efficiency and potentially also accuracy to train the stencils in a supervised manner, e.g., with a more accurate discretization as reference? One could argue that it would be expensive to pre-compute such data, but I think RL scales even worse to higher dimensional problems.\n\nWhat's also missing in the current version is a more thorough discussion of inference and training performance. I guess that despite the small model problems, the training takes a substantial amount of time. And due to the large size of the trained model, which has to be evaluated for every single node in the 1D mesh, it's probably also quite slow. I think this is worth a discussion in the text. One could even estimate the number of operations necessary to evaluate the model, and run a higher-order WENO scheme for a \"fair\" comparison.\n\nMinor, but in equation (1), I guess the t subscript should indicate a material derivative, and just just a time derivative, right? This could be clarified in the text (or written out).\n\nI am somewhat on the edge with this paper - the 1D case for the two equations is carefully evaluated in the submission, and it's great to see the trained model can improve the accuracy across a fairly wide range of settings. As such, it's definitely a good and interesting first step. On the other hand, there are a range of open questions, as outlined above, and it's not clear whether the approach could be easily translated to higher dimensions. I hope the authors can clarify some of these points in the rebuttal, right now I'm leaning towards the positive side.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "r1lYBi9cKH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the author maps the problem of time series PDE into a naive reinforcement learning problem. Under the MDP assumption, the author sets the initial state of the particles as the current state, the flux at all spaces as the possible actions, and map the state-action pair deterministically to the next state of the particle diffusion. The reward is defined as the two norms between the prediction and the Burger’s equation. The naiveness comes from the fact that the typical reinforcement learning problem, the agent needs to decide how to choose an action. In this paper, it is formulated as an intrinsic proper that follows Burger’s equation instead. \n\nWhile the motivation is interesting, the author argues this work is novel due to it does not fall under supervised learning, but rather reinforcement learning. This perspective is not completely correct. The correct category for this work would be more similar to imitation learning using WANO’s algorithm as the expert label. This is a field of supervised reinforcement learning.\n\nThe author’s work has brought the possibility of using neural network architecture in the field of particle diffusion. The benefit is the improved estimation of how particles diffuse in long-horizon conditions. The author has shown in their paper their simple fully connected network has already performed better prediction than the current state of the art non-neural network model: WENO.\n\nWhile the framing of the problem is perhaps novel in the space of PDE, algorithmically there needs to have a breakthrough or new invention. The lack of comparison with other neural-network-based models also hurts the credibility of the model. Therefore, I reject this paper under the ICLR conference. I would suggest that this paper would be better suited as a paper submission under the perspective science field conference instead.\n\nSome suggestions to further improve this paper: The author could add CNN and RNN structure to the prediction model. These structures would further expand other possibilities in the solution space. CNN would help turn the limited 1D problem to a higher-dimensional, a more real-world like problem space. RNN is known for its’ ability to model long horizon problems, perhaps even better breakthrough would happen with these architectures.\n\nAs a whole, the paper is written very well such that even nonexpert can grab onto the logic flow of this paper. The weaknesses of the paper are the lack of diversity in comparison with other models and the paper needs some level of novel breakthrough in an algorithmic sense.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "SygDufkFKB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n##### Rebuttal Response:\nThe other reviewers seem to have understood more than me. Their opinion and the rebuttal did not convince me to update my score. In my opinion the writing must be adapted to be interesting to the ICLR community and the bigger picture should be highlighted more, as the bigger picture is remains quite unclear at the current state.\n\n\n##### Review:\nSummary: \n[...]\n\n\nConclusion:\nI have read the paper multiple times and I still have a problem summarizing the paper with my own words. The contributions summarize the most fundamental works of RL but do not really relate these methods to the proposed approach. Therefore, I am still uncertain about the general motivation and intention of the work as well as the evaluation. Currently I vote for borderline reject as I am familiar with RL & PDE'S but do not understand the motivation and intention. I am leaning towards rejection as the paper is a resubmission from Neurips and has not been substantially improved. However, I am not certain about my evaluation. I am happy to adapt my vote based on the other reviewers and a clarified and better structured paper, which can be submitted during the rebuttal.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes using RL to solve PDEs, with application to solving conservation laws. It is quite borderline, with one reviewer weakly recommending acceptance, one finding the paper interesting but the application not sufficiently novel, and one confessing they have not understood the paper.

I concur with R2 this is a difficult subject matter, but the other reviewers seem satisfied with the clarity of the presentation. R3 seems to believe the paper sufficiently proves the concept to warrant publication. I confess I do not understand R1's argument for lack of novelty, despite my pushing for further detail. I see this as a novel application of RL methods, and R1 admits this will be seen as novel for a PDE conference. I am in favour of a certain degree of interdisciplinarity at ICLR, and believe this paper could bring a bit of subject matter diversity to the programme. However, due to the number of high quality submissions in my area, I'm afraid this one must be rejected due to limited space. The authors are encouraged to submit to another ML conference after addressing (or having addressed) some of the action items from the more sympathetic reviewers.
Decision: Reject