Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors apply the idea of hypersolvers to numerical optimal control. The idea is to approximate the dynamics with a low-order ODE solver and learn the truncation residual dynamics using a neural net. They also propose a multi-stage hypersolvers method to both learn the system residual dynamics (due to erroneous system model) and truncation residual dynamics (due to ODE solver). The performance of the proposed approach is shown in simulation experiments with a pendulum, a cart-pole, and a quadcopter with comparisons to ODE solvers of different orders.
main_review: The paper proposes to apply the idea of hypersolvers to numerical optimal control. Through learning the truncation residual dynamics, the model is able to approximate the actual dynamics with a combination of low-order ODE solver with learned truncation residual dynamics. Further, if the system model is not perfectly accurate, a multi-stage hypersolver can be used to separately learn the system residual dynamics and truncation residual dynamics. In experiments, the method is shown to perform on par with the higher-order ODE solvers. Also, experiments show that certain activation functions (SIREN and Snake) perform better than the standard ones (Tanh and ReLU).

Here are some of my detailed comments:
1. In (3), \epsilon appeared before it is introduced in (4), hence making it undefined and unclear.
2. In section 3.2, maybe more explanation can be given on how (2) is solved?
3. In the pendulum experiment, the hypersolver approach achieves similar performance in terms of final positions when compared with Midpoint and RK4, but using similar number or 50% less of FLOPS. For the other two experiments, I don’t seem to find how does the method compares in terms of FLOPS or runtime. As in the pendulum case, the proposed hypersolver approach does not outperform a lot in terms of runtime or FLOPS.
4. In the pre-training of the hypersolvers, it seems to require a very large number of epochs, maybe more information can be provided on how many trajectories are used for training and how long the training time is.
5.  In section 5.2, how does the tolerance of the ODE solver affect the experiment results, maybe some ablation studies and explanations can be included to help understand.
6. For the multi-stage hypersolvers, there are some literature on learning the system residual dynamics for control, such as [1][2][3]. Also some ablation studies can be done by comparing the multi-stage hypersolver approach with just learning system residual dyanamics.

[1] Saveriano, Matteo, Yuchao Yin, Pietro Falco, and Dongheui Lee. "Data-efficient control policy search using residual dynamics learning." In 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4709-4715. IEEE, 2017

[2] J. F. Fisac, A. K. Akametalu, M. N. Zeilinger, S. Kaynama, J. Gillula, and C. J. Tomlin. A general safety framework forlearning-based control in uncertain robotic systems.IEEE Transactions on Automatic Control, 64(7):2737–2752,2018.

[3] A. Taylor, A. Singletary, Y. Yue, and A. Ames. Learning for safety-critical control with control barrier functions. InLearning for Dynamics and Control, pages 708–717. PMLR, 2020.

summary_of_the_review: Overall, the paper proposes to use hypersolvers to speed up ODE solving for numerical control. The idea is promising but it needs more experiment results comparisons, especially in terms of runtime, to justify the efficacy of using hypersolvers for optimal control. Also, the paper could consider expanding the explanation on how the optimal control problem (2) is solved in their experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper extends the hypersolvers framework (Poli et al., 2020a) to the setting of optimal control. The hypersolvers framework combines a cheap numerical dynamical system solver with a neural network trained to approximate the truncation error of this solver, which together yield a cheap and relatively accurate solver. This paper shows how to utilize the hypersolvers framework within a controlled dynamical system by exploring different choices of loss functions, training methods, and network architectures for hypersolver pre-training. The authors further provide an augmentation to this framework, called multi-stage hypersolvers, to further account for misspecifications of the dynamics. Via experiments, the authors then demonstrate the performance of optimal control strategies (direct optimal control and MPC) utilizing hypersolvers in different settings (notably the pendulum, cart-pole, and quadcopter settings). They show e.g. that even in cases where their method receives misspecified dynamics, they are able to achieve control performance that is comparable to that obtained in cases where the underlying solver had knowledge of the correct dynamics.
main_review: Strengths: I thought the paper was a crisp and clear read, and that all technical components (both novel and tutorial) and all experiments were both sound and very well-explained.

Weaknesses: The systems that are tested on are relatively small scale/low-dimensional. Given that hypersolver frameworks are presumably most necessary in larger scale systems that are indeed expensive to solve, this is in my view a large limitation in terms of understanding the value proposition of the present work. It would greatly strengthen the present work if demonstrations on such large-scale/high-dimensional systems could be shown.

Minor points (not affecting my score):
* $\psi_\epsilon$ in Equation (3) could be more explicitly defined, as it is of course a critical variable for the rest of the paper.
* The differentiation of the present work from the prior hypersolvers work could be made more clear, and in general the writing could better signpost what aspects of the present work are "new" contributions. If I understand correctly, the major distinction from the prior work is that the prior work considered uncontrollable systems, whereas the present work considers controllable systems, which yields some differences both in terms of how the hypersolver is trained and in how it is used. However, this should be spelled out more explicitly.
summary_of_the_review: I thought this paper was a sound, enjoyable, and clear extension of the previous work on hypersolvers to the setting of controlled dynamical systems. However, providing experiments on larger-dimensional systems would in my view significantly strengthen the paper by more fully demonstrating the potential benefits of the proposed method. That said, I am marginally inclined to accept the paper nonetheless rather than rejecting it solely on that basis.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper descibes the application of hypersolves, which warm-start ODE solvers with neural nets that have distilled many precomputed solutions, to optimal control ODEs. 
This warm-starting improves downstream accuracy when the computational budget is fixed. 
This method is applied for trajectory optimziation for pendulum swing-up and MPC for cartpole and quadcopter systems.
main_review: I thought this paper on the whole was nicely written and easy to understand. 

My main issue with the paper is the limited application and its relevance to the ICLR community. 
The idea of using NNs to warm start ODE solvers is relevant, but this idea is presented in the paper from Poli et al from Neurips 2020 (althought the paper refence of this work is missing the venue).
However the specific application to optimal control ODEs is somewhat limited and would probably be better placed at a control conference like CDC, ACC or L4DC. 
This is reinforced by the experimental settings being quite simple and lwo dimensional. 
While ODE-based optimal control is an important setting, the ML community typically use other methods for continuous control. 

I also had some minor issues with mathematical clarity. For example the losses in Eq 6 and 6 dont have inputs. In the (unnumbered) equation in 4.2 and Eq. 8 , it is not clear where w enters the objective.  
summary_of_the_review: The application of hypersolves to optimal control is somewhat incremental and not of great relevance to the ICLR community. I woudl be willing to change my score if the reviewers and AC disagree.
review_title: null

Meta-review: The authors propose a novel hypersolver framework for solving numerical optimal control problems, learning a low order ODE and a neural network based residual dynamics. They compare their framework with traditional optimal control solvers on a number of control tasks and demonstrate superior performance.

The reviewers are in consensus that the paper makes significant contributions that are validated by the experimental results. The only concern was that the experiments are largely on low dimensional systems, but the reviewers agreed that the results are still worthy of acceptance.
Decision: Accept (Poster)