Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper proposes to learn the step size for a Runge-Kutta numerical integrator for solving ordinary differential equations initial value problems. The authors frame the stepsize control problem as a learning problem, based on different performance measures, on ODE dependent inputs and on a LSTM for predicting the next step coefficient. Experiments are performed on 3 ODEs by training and testing in different contexts.
The problem of designing adaptive controllers for ODE numerical schemes is interesting and is probably a new application for ML. The paper makes an effort to introduce the necessary background and for reviewing some classical adaptive controller techniques. The description of the method is relatively clear, but could however be largely improved in order to make it more accessible to the audience.  Many of the arguments and proposed ideas come without justification, some definitions should be made more precise. The construction of the training and test sets should be better explained. The experiments show that the proposed approach leads to fewer evaluations but larger mean errors.  The graphics also show that the local error is smaller for the proposed method than for the baselines which is in contradiction with the global error behavior. This should be clarified – the relations between the two error types should be made clear.  The baseline is not defined in the text so that it is difficult to judge the performance. Why not comparing to several adaptive baselines?

In conclusion, this is an interesting topic, the paper proposes new ideas. A more careful writing and especially a better comparison with sota baselines would greatly improve the paper. 


------ post rebuttal -----
Thanks for the answers. I still think that the ideas are interesting but that the experiments  do not demonstrate enough of the proposed method. I will keep my score.

review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: Updated review: Thanks to the authors for their response to my comments. I believe the strong point of this paper is the novel idea, however, I find the justification for that idea incomplete as author's seems to suggest that the proposed method is probably computationally more expensive (which is opposite to the original motivation of the paper).

---------------------------------------------------------------------------------------------------------------------------------------------------------------------

Summary: This paper casts the problem of step-size tuning in the Runge-Kutta method as a meta learning problem. The paper gives a review of the existing approaches to step size control in RK method. Deriving knowledge from these approaches the paper reasons about appropriate features and loss functions to use in the meta learning update. The paper shows that the proposed approach is able to generalize sufficiently enough to obtain better performance than a baseline. 

I think this paper, in general, is clear and well-written. I believe the idea of the paper is interesting too. 

The paper argues that the main challenge of solving the step size control problem for the RK method is balancing the computation vs accuracy trade-off. Existing methods tackle this problem in different ways and this paper proposes to solve it via meta-learning. However, the paper does not mention how and why meta-learning is expected to tackle this challenge?
So a couple of comments on what set of problems do we expect meta-learning to better tackle this trade-off than the existing methods would have been useful. I am wondering if it is even possible to say something about this in principle? 

The paper argues that the idea behind using meta-learning is to learn behaviour from a given class of problems and then generalize to new unseen problems (from the same or different classes). 
How do we know that these problems are even from same distributions? 
Won't the proposed approach fail spectacularly when the problems are not from the same distribution? It would have been nice if the paper made this distinction even if empirically. 

In the experiments section, I could not find/understand what exactly is the baseline the paper is comparing to. 

I was more interested in a study that compared the performance of MLRK as the number of instances of the training problems are varied. 
This again makes me come back to the original point of computational cost vs accuracy. What is the computational cost of collecting data on 30000 instances of problems? Should we not worry about this cost?
Also, what is the computational cost of the proposed approach and why are we not comparing it to existing approaches/baseline?

minor comments: 
what is tol? it tol the same tolerance as lambda.
 

review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: ## Summary ##

The authors present a method for learning a step-size adaptation strategy for Runge-Kutta methods. The principal contributions of the paper are:

1. They define a loss function that better captures global performance of the controller, rather than just local behavior.
2. They propose a set of input features to a step size controller. It includes the intermediate Runge-Kutta evaluation values, which allow the controller to approximate the derivatives of the Jacobian function $g$.
3. They describe a recurrent architecture for adapting step size over time.

Runge-Kutta methods are a workhorse of ordinary differential equations and choosing step size is one of the central challenges involved in their application. Better methods for step size selection would definitely be of broad interest. As the authors point out, existing methods often consist of hand-tuned heuristics---a feature that often suggests machine learning could provide significant improvements.

While the premise of the paper is very promising, I don't think it is ready to be accepted to ICLR at this time. Most significantly, the experimental results are not particularly compelling. I believe the authors should refine their method, aim for better experimental results and resubmit. I have included more specific comments below.

## Specific Comments ##

1. I think the paper would benefit from a clearer description of the RK step-size selection problem. For instance, for a p^th order RK solver, at each time step,

   * Inputs: t, y(t), g  # Also possibly intermediate values from previous time steps.
   * Select a step size h(t)
   * Evaluate g at p different points based on h(t).
   * Use these evaluations to compute a value of y_(t + h_t)

   For those that aren't familiar with these methods (at ICLR there will be many!) I think this would help explain where the authors' method (and the other methods your describe) fits into the larger algorithm.

2. I think a short explanation of error estimation would be helpful in addition to the reference to Butcher. This estimation is critical to step size adaptation. In particular I think this would be clearer if the authors expanded the paragraph at the bottom of page 4, where they describe error in a polynomial in (t_n, h) whose coefficients are derivatives of g.

3.  The authors' proposed loss function (Eq. 5) includes the true value of y at t_n, y(t_n). They acknowledge that this may make it computationally prohibitive, but I think this point warrants further discussion. Does this mean that their loss function can only be used on problems for which we have a closed form solution (such as harmonic oscillators)? I noticed that in their van der Pol experiments, the authors switch to a more standard loss (Eq. 6). Is this because Eq. 5 is intractable in this example? Is there a reasonable approximation to Eq. 5 that could be used when a closed form solution is not known?

4. There are a number of issues with the experiments that I think could use clarification or improvement:

  (a) The authors compare to a 'baseline' but I don't believe this baseline is defined anywhere. Is it one of the adaptation methods described in Section 2?

  (b) In Table 1, the baseline method achieves lower error, while MLRK uses fewer steps. It is difficult to assess if this is an improvement since this this cost-accuracy tradeoff is at the heart of the Lagrangian formulation. Ideally, shouldn't we be able to adjust 'tol' to trace out some Pareto frontier for the cost-accuracy tradeoff? In this case, wouldn't we hope for MLRK to be able to achieve better accuracy given the same computational budget?

  (c) In the van der Pol experiments, the authors switch to local L1 loss (Eq. 6). Was the intention to experiment with local L1 loss and van der Pol provided an interesting class of examples? Or was the reasoning that Eq. 5 is intractable for van der Pol so they had to use local L1 loss? If Eq. 5 can still be evaluated on these experiments, this might be a more convincing comparison.

  (d) The number of steps required was provided for the harmonic oscillator experiments, but not the van der Pol ones. This would be helpful for comparing the methods.

  (e) What is the computational overhead of running an RNN alongside your solver? Although it doesn't tell the whole story, it would be informative to report wall clock time along with the number of steps required for each method.
review_title: Official Blind Review #2

Meta-review: Summary: This paper casts the problem of step-size tuning in the Runge-Kutta method as a meta learning problem. The paper gives a review of the existing approaches to step size control in RK method. Deriving knowledge from these approaches the paper reasons about appropriate features and loss functions to use in the meta learning update. The paper shows that the proposed approach is able to generalize sufficiently enough to obtain better performance than a baseline. 


The paper was lacking in advocates for its merits, and needs better comparisons with other baselines before it is ready to be published.
Decision: Reject