Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper proposes to use reinforcement learning for constructing discretziation stencils of numerical schemes. More specifically, the method focuses on the widely used WENO schemes, which are an established class of finite difference schemes. Within this context, the method aims for training models to infer the weighting for a specific stencil with eight flux terms.

For RL this task requires a continuous action space, and the DDPG algorithm is used for training the policy. The network itself is an MLP with 6 layers, and ca. 20000 weights in total. This is a significant number, given the focus on 1D problems.

The tests are quite thorough and interesting, while at the same time being limited in scope. The paper targets 1D cases, which make the problem very low-dimensional. Despite the simplicity, only a single data set (Burgers) is used, and a single modified target function with a u^4 term. Targeting 1D casesl, I would have expected a broader range of tests and model equations.

Despite the limited scope of the models, table 1 and 2 assess a nice range of different timestep and discretization parameters. I found it very interesting to see that the method consistently outperforms the regular WENO scheme. The gains are relatively small, with 4-5%, but WENO already represents a quite accurate scheme, so it's surely not easy to outperform it.

While reading the paper, I was wondering about the bigger picture, i.e. using RL in the context of discretization stencils. We have model equations, and discretized versions of all operators involved in training. Why employ a "brute force" approach like RL here? Wouldn't it be better in terms of efficiency and potentially also accuracy to train the stencils in a supervised manner, e.g., with a more accurate discretization as reference? One could argue that it would be expensive to pre-compute such data, but I think RL scales even worse to higher dimensional problems.

What's also missing in the current version is a more thorough discussion of inference and training performance. I guess that despite the small model problems, the training takes a substantial amount of time. And due to the large size of the trained model, which has to be evaluated for every single node in the 1D mesh, it's probably also quite slow. I think this is worth a discussion in the text. One could even estimate the number of operations necessary to evaluate the model, and run a higher-order WENO scheme for a "fair" comparison.

Minor, but in equation (1), I guess the t subscript should indicate a material derivative, and just just a time derivative, right? This could be clarified in the text (or written out).

I am somewhat on the edge with this paper - the 1D case for the two equations is carefully evaluated in the submission, and it's great to see the trained model can improve the accuracy across a fairly wide range of settings. As such, it's definitely a good and interesting first step. On the other hand, there are a range of open questions, as outlined above, and it's not clear whether the approach could be easily translated to higher dimensions. I hope the authors can clarify some of these points in the rebuttal, right now I'm leaning towards the positive side.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: In this paper, the author maps the problem of time series PDE into a naive reinforcement learning problem. Under the MDP assumption, the author sets the initial state of the particles as the current state, the flux at all spaces as the possible actions, and map the state-action pair deterministically to the next state of the particle diffusion. The reward is defined as the two norms between the prediction and the Burger’s equation. The naiveness comes from the fact that the typical reinforcement learning problem, the agent needs to decide how to choose an action. In this paper, it is formulated as an intrinsic proper that follows Burger’s equation instead. 

While the motivation is interesting, the author argues this work is novel due to it does not fall under supervised learning, but rather reinforcement learning. This perspective is not completely correct. The correct category for this work would be more similar to imitation learning using WANO’s algorithm as the expert label. This is a field of supervised reinforcement learning.

The author’s work has brought the possibility of using neural network architecture in the field of particle diffusion. The benefit is the improved estimation of how particles diffuse in long-horizon conditions. The author has shown in their paper their simple fully connected network has already performed better prediction than the current state of the art non-neural network model: WENO.

While the framing of the problem is perhaps novel in the space of PDE, algorithmically there needs to have a breakthrough or new invention. The lack of comparison with other neural-network-based models also hurts the credibility of the model. Therefore, I reject this paper under the ICLR conference. I would suggest that this paper would be better suited as a paper submission under the perspective science field conference instead.

Some suggestions to further improve this paper: The author could add CNN and RNN structure to the prediction model. These structures would further expand other possibilities in the solution space. CNN would help turn the limited 1D problem to a higher-dimensional, a more real-world like problem space. RNN is known for its’ ability to model long horizon problems, perhaps even better breakthrough would happen with these architectures.

As a whole, the paper is written very well such that even nonexpert can grab onto the logic flow of this paper. The weaknesses of the paper are the lack of diversity in comparison with other models and the paper needs some level of novel breakthrough in an algorithmic sense.

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: 
##### Rebuttal Response:
The other reviewers seem to have understood more than me. Their opinion and the rebuttal did not convince me to update my score. In my opinion the writing must be adapted to be interesting to the ICLR community and the bigger picture should be highlighted more, as the bigger picture is remains quite unclear at the current state.


##### Review:
Summary: 
[...]


Conclusion:
I have read the paper multiple times and I still have a problem summarizing the paper with my own words. The contributions summarize the most fundamental works of RL but do not really relate these methods to the proposed approach. Therefore, I am still uncertain about the general motivation and intention of the work as well as the evaluation. Currently I vote for borderline reject as I am familiar with RL & PDE'S but do not understand the motivation and intention. I am leaning towards rejection as the paper is a resubmission from Neurips and has not been substantially improved. However, I am not certain about my evaluation. I am happy to adapt my vote based on the other reviewers and a clarified and better structured paper, which can be submitted during the rebuttal.
review_title: Official Blind Review #2

Meta-review: This paper proposes using RL to solve PDEs, with application to solving conservation laws. It is quite borderline, with one reviewer weakly recommending acceptance, one finding the paper interesting but the application not sufficiently novel, and one confessing they have not understood the paper.

I concur with R2 this is a difficult subject matter, but the other reviewers seem satisfied with the clarity of the presentation. R3 seems to believe the paper sufficiently proves the concept to warrant publication. I confess I do not understand R1's argument for lack of novelty, despite my pushing for further detail. I see this as a novel application of RL methods, and R1 admits this will be seen as novel for a PDE conference. I am in favour of a certain degree of interdisciplinarity at ICLR, and believe this paper could bring a bit of subject matter diversity to the programme. However, due to the number of high quality submissions in my area, I'm afraid this one must be rejected due to limited space. The authors are encouraged to submit to another ML conference after addressing (or having addressed) some of the action items from the more sympathetic reviewers.
Decision: Reject