Title: Recursive Regression with Neural Networks: Approximating the HJI PDE Solution
Authors: Vicenç Rubies Royo
Claire Tomlin
Venue: ICLR 2017
Keywords: Supervised Learning
Games
Theory

Abstract: Most machine learning applications using neural networks seek to approximate some function g(x) by minimizing some cost criterion. In the simplest case, if one has access to pairs of the form (x, y) where y = g(x), the problem can be framed as a regression problem. Beyond this family of problems, we find many cases where the unavailability of data pairs makes this approach unfeasible. However, similar to what we find in the reinforcement learning literature, if we have some known properties of the function we are seeking to approximate, there is still hope to frame the problem as a regression problem. In this context, we present an algorithm that approximates the solution to a partial differential equation known as the Hamilton-Jacobi-Isaacs PDE and compare it to current state of the art tools. This PDE, which is found in the fields of control theory and robotics, is of particular importance in safety critical systems where guarantees of performance are a must.

TL;DR: A neural network that learns an approximation to a function by generating its own regression points

Review discussion:
I have no familiarity with the HJI PDE (I've only dealt with parabolic PDE's such as diffusion in the past).
So the details of transforming this problem into a supervised loss escape me.
Therefore, as indicated below, my review should be taken as an "educated guess". I imagine that many readers of ICLR will face a similar problem as me, and so, if this paper is accepted, at the least the authors should prepare an appendix that provides an introduction to the HJI PDE. At a high level, my comments are:
1. It seems that another disadvantage of this approach is that a new network must be trained for each new domain (including domain size), system function f(x) or boundary condition. If that is correct, I wonder if it's worth the trouble when existing tools already solve these PDE's. Can the authors shed light on a more "unifying approach" that would require minimal changes to generalize across PDE's?
2. How sensitive is the network's result to domains of different sizes? It seems only a single size 51 x 51 was tested. Do errors increase with domain size?
3. How general is this approach to PDE's of other types e.g. diffusion?
Thank you for the review. Below I will address the points mentioned by the reviewer.
This method indeed requires retraining if the domain bounds, the dynamics f(x)/f(x,a)/f(x,a,b) or the boundary condition change.
However, to the best of our knowledge, this is also the case for all existing gridding/finite element techniques. This, I am afraid, seems unavoidable for most (if not all) PDE numerical methods.
This approach is meant to offer a different way to compute the approximation to the HJI PDE with some of the advantages mentioned in the "Advantages and Disadvantages" section, including curse of dimensionality alleviation for memory, differentiability of the approximation (finite/gridding methods yield the gradient approximation using the grid, which is itself an approximation) and approximation over the entire domain (not just grid points). Some very recent experiments with the algorithm also hint that the method may scale to state dimensions higher than 4, which is something promising given that gridding methods become intractable beyond state dimension of 4 (again, due to the curse of dimensionality).
In terms of a "unifying approach", one of the core ideas was to discretize the partial derivative for time in such a way that values would "flow" away from the boundary; in that context, we believe this could also work for other PDEs where the left hand-side of the PDE is the only place where the partial derivative with respect to time appears (this includes the diffusion PDE). That would essentially require changing lines 11 and 12 of the algorithm.
Last but not least, in comparison to other neural network approaches to approximating PDE solutions, ours does not require the right-hand side to be differentiable.
The set S \in [-5,5]^2 was used for all experiments. This work has a lot of different parts that can be tweaked to push the algorithm to its limits, and changing the size of S is one of them; given the time and space constraints we have not yet been able to explore all of these limits, but we will certainly be exploring them in the near future.
Approximating solutions to PDEs with NN approximators is very hard.
In particular the HJB and HJI eqs have in general discontinuous and non-differentiable solutions making them particularly tricky (unless the underlying process is a diffusion in which case the Ito term makes everything smooth, but this paper doesn't do that). What's worse, there is no direct correlation between a small PDE residual and a well performing-policy [tsitsiklis? beard? todorov?, I forget]. There's been lots of work on this which is not properly cited.
The 2D toy examples are inadequate. What reason is there to think this will scale to do anything useful?
There are a bunch of typos ("Range-Kutta"?) .
More than anything, this paper is submitted to the wrong venue. There are no learned representations here. You're just using a NN. That's not what ICLR is about. Resubmit to ACC, ADPRL or CDC.
Sorry for terseness.
Despite rough review, I absolutely love this direction of research. More than anything, you have to solve harder control problems for people to take notice...
Thank you for taking the time to review our work.
1.) We are aware that the PDE residual does not ultimately reflect how good an approximation is. One can have an approximation where the function has a "discontinuous jump" in an arbitrarily small time interval and the residual might remain small, even though the approximation is clearly not correct. This is in part why we did not use a single error metric, but used two error metrics instead.
Also, since the PDE residual is still being used in some papers to get a "feeling" for the learning progression, we decided to use it as well.
2.) We used a set of 2D examples to check whether the algorithm would work in the first place. If it does not work in 2D it probably won't work in 3D, 4D etc. This being said, we know that interesting problems are those in high dimensions, and that is what we are currently working towards. To that end, we appended an extra experiment in 3D at the end of the paper.
3.) Finally, we believe that this work is in fact suited for this conference just like other submissions, particularly those in RL, which make use of NNs and solely focus on algorithmic performance and make no mention of learned representations/features.
Moreover, the topic of safety is gaining more importance in the AI community, and thus it seems appropriate to present our work here.
This paper presents an algorithm for approximating the solution of certain time-evolution PDEs. The paper presents an interesting learning-based approach to solve such PDEs. The idea is to alternate between:
1. sampling points in space-time
2. generating solution to PDE at "those" sampled points
3. regressing a space-time function to satisfy the latter solutions at the sampled points (and hopefully generalize beyond those points).
I actually find the proposed algorithm interesting, and potentially useful in practice. The classic grid-based simulation of PDEs is often too expensive to be practical, due to the curse of dimensionality.
Hence, learning the solution of PDEs makes a lot of sense for practical settings.
On the other hand, as the authors point out, simply running gradient descent on the regression loss function does not work, because of the non-differentiablity of the "min" that shows up in the studied PDEs.
Therefore, I think the proposed idea is actually very interesting approach to learning the PDE solution in presence of non-differentability, which is indeed a "challenging" setup for numerically solving PDEs.
The paper motivates the problem (time-evolution PDE with "min" operator applied to the spatial derivatives) by applications in control thery, but I think there is more direct interest in such problems for the machine learning community, and even deep learning community.
For example http://link.springer.com/chapter/10.1007/978-3-319-14612-6_4 studies approximate solution to PDEs with very similar properties (evolution+"min") to develop new optimization algorithms. The latter is indeed used to training deep networks: https://arxiv.org/abs/1601.04114
I think this work would catch even more attention if the authors could show some experiments with higher-dimensional problems (where grid-based methods are absolutely inapplicable).
Thank you very much for taking the time to review the paper and for the attached URLs, we look forward to reading their contents.
Also, we added an extra experiment in 3D at the end.
We appended a 3D experiment in the paper.

Meta-review:
role: PC
data: comment: The basic approach of this paper is to use a neural net to sequentially generate points that can be used as the basis points in a PDE solver. The idea is definitely an interesting one, and all three reviewers are in agreement that the approach does seem to have a lot of potential.
 
 The main drawback of the paper, simply, is that it's unclear whether this result would be of sufficient interest for the ICLR audience. Ultimately, it seems as though the authors are simply training a neural network to generate this points, and the interesting contribution here comes from the application to PDE solving, not really from any advance in the NN/ML side itself. As such, it seems like the paper would be better appreciated (as a full conference paper or journal paper), within the control community, rather than ICLR. However, I do think that as an application, many at ICLR would be interested in seeing this work, even if its likely to have relatively low impact on the community. Thus, I think the best avenue for this paper is probably as a workshop post at ICLR, hopefully with further submission and exposure in the controls community.
 
 Pros:
 + Nice application of ML to a fun problem, generating sample points for PDE solutions
 + Overall well-written and clearly presented
 
 Cons:
 - Unclear contribution to the actual ML side of things
 - Probably better suited to controls conferences

Decision: Invite to Workshop Track