Venue: NeurIPS 2022 Track Datasets and Benchmarks

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: summary and contributions: The authors present Myriad, a testbed written in JAX which enables machine learning researchers to benchmark imitation learning and reinforcement learning algorithms against trajectory optimization-based methods in challenging real-world environments. Myriad contains 18 optimal control problems presented in continuous time and ranging from biology to medicine to engineering. As such, Myriad strives to serve as a stepping stone toward the application of modern machine learning techniques for impactful real-world tasks.

All environments, optimizers and tools are available in the software package at https://github.com/nikihowe/myriad.
strengths: 1. All tasks are inspired by real-world problems, with applications in medicine, ecology, and epidemiology.
2. Myriad is, as the authors mentioned, the first repository that enables deep learning methods to be combined seamlessly with traditional trajectory optimization techniques. 
3. The system dynamics in Myriad are continuous in time and space, offering several advantages over discretized environments.
4. The authors present a novel control-oriented imitation learning algorithm that combines optimal control with deep learning.
weaknesses: 1. As a real-world testbed paper, I think it will be more helpful to have tools like Jupiter notebooks to directly demonstrate some examples mentioned in the paper. I went through the GitHub page mentioned in the paper. The code is clean and nice but it will be better to have some demonstrations.
2. The paper is well organized but it seems the authors are trying to put too much content in relatively short pages, which make it harder to captcha more details in each section, in this case, the different scenarios. 
correctness: I agree the claims made in the submission are correct and approporiate.
clarity: The paper is well written. But it will be better to have more details in each section, rather than mentioning section 4567 together in the paper. I also recommend that, since you have a github page, you should make it for better use. For example, you can put each section in the github and with a small demo for each scenario.
relation to prior work: Yes
additional feedback: Overall I recommend accepting this paper if we have enough room. The Github page can be better with more demo and details for each section is necessary.
review_title: Myriad: a real-world testbed to bridge trajectory optimization and deep learning

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: summary and contributions: This paper proposed Myriad, which is a real-world testbed that offers many real-world relevant, continuous space and time dynamical system environments for optimal control. Myriad is written in JAX, and both environments and trajectory optimization routines are fully differentiable.
strengths: This paper offered trajectory optimization tools to the machine learning community that is compatible to deep learning workflow. It encorages the development of machine learning algorithms with the goal of addressing real-world problems.
weaknesses: The tasks that Myriad provides is simple and thoroughly studied in optimal control, while ML for trajectory optimization usually aims to solve more complex tasks.
correctness: Yes.
clarity: Yes.
relation to prior work: Yes.
additional feedback: No.
review_title: Good work to fill the blank of hands-on tool for the area

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: Myriad is a testbed consisting of 18 continuous-time, optimal control problems (called systems or environments) from several real-world domains (medicine, ecology, and epidemiology). This work aims to enable comparisons between deep learning methods (like RL and IL) and optimal control methods, for which benchmark reference scores are provided. Written in JAX, Myriad also enables new algorithms like implicit planning over neural ODEs, a novel control-oriented imitation learning technique that is presented in detail.
strengths: Deep learning-based approaches to decision making are not often compared to relevant benchmark results from more classical, continuous-time trajectory optimization, even when such comparisons are possible in principle. Myriad helps bridge this gap by means of a new repository and results presented in the paper. The challenges and the solutions taken by Myriad are explained with care, particularly in the key case where system dynamics need to be learned. The main example of such a learned model is a differentiable Neural ODE, outlined in Algo 2, for which experimental results are provided.

The possibility of combining deep learning and trajectory optimization methods is intriguing.

Several of Myriad’s current limitations (like its inability to handle pixel-based images) are clearly highlighted.
weaknesses: It’s not clear to a deep RL practitioner like me how to use Myriad’s results as benchmarks against which to meaningfully compare RL or IL techniques. 

Environments called cart-pole and pendulum (wrapped in the standard gym API) do exist. But I see no reason to conclude that their underlying equations of motion are at all the same as the systems of the same name covered by Myriad. If they are the same, then the paper should document this correspondence, and compare Myriad’s results in those environments to appropriate RL or IL results, even if only from prior work. 

But if the underlying dynamics of Myriad’s systems are not the same as those of existing gym environments, then Myriad should bridge the gap (between continuous time and discrete time) by implementing gym environments which do share those dynamics. This would provide the missing part of the bridge between deep learning and optimal control that Myriad is targeting.
correctness: I’m not able to assess the correctness of the paper’s formal descriptions of Myriad’s approach to continuous-time trajectory optimization, but I see no problem with the results presented.
clarity: The presentation and writing are of very high quality throughout.
relation to prior work: Short but solid discussion of prior work.
additional feedback: A description of the amount of compute required to produce the paper’s experimental results would be very useful.
review_title: A partial bridge between trajectory optimization and deep learning.

final_score: rating: 5: Marginally below acceptance threshold
review_content: summary and contributions: This paper presents a suite of tasks inspired from real-world problems. It also provides implementations of 5 trajectory optimaiztion algorithms. The environments and the trajectory optimization algorithms are made differentiable. This paper also demonstrates system identification and dynamcis learning via neural ODE on the presented tasks. It also proposes a new imitation learning algorithm that embeds planning into the policy structure by leveraging the differentiating through the non-linear program solver. Finally, references scores of the trajectory optimization with true dynamics, sysID-ed dynamics, learned dynamics, and the proposed imitation learning algorithms are reported. 
strengths: - Benchmarking of trajectory optimization algorithms, and making them differentiable is relevant yet under-explored to the ML community, and this paper provides a good starting point towards filling the gap.
- The experiments span an interesting range of benchmarking trajectory optimization algorithms with true dynamics, dynamics learning, and trajectory optimization with learned dynamics. 
- The repository seems well structured and documented

weaknesses: - Details of the tasks are missing. There is only a single-sentence description of the tasks in the appendix, which makes it difficult to tell if these tasks are really useful and proper for benchmarking trajectory optimization algorithms. For example, the paper claims the tasks to be challenging, but I am not sure if tasks such as pendulum, mountain car, and Cart-Pole Swing-Up can be considered challenging, which are more like standard easy-to-solve benchmarking tasks. I am less familiar with other tasks in medicine, ecology, and epidemiology, and thus not sure if those tasks are really challenging and can be used for benchmarking trajectory optmization methods. Please provide more details of the tasks and the reasons for choosing them. 
- The paper is not very well structured and miss some important details:
Section 4, I don't think such a detailed explanation and comparison between the direct single shooting and direct multiple shooting is necessary (including figure 2), since they are just existing trajectory optimization methods and should not be the focus of the work. I feel this section, could be changed to a discussion on how different trajectory optimization methods work on the **proposed tasks**, in stead of how they work in a toy example as in current figure 1.
Section 5, it seems that the proposed method to do constraint optimization is just to do gradient-descent-ascent on a GPU. I am not sure why this would just be significantly faster than traditional methods well optimized on a CPU. is there any specific optimization used for running it on the GPU? Or at least some quantitative comparison of the speed can be provided to understand if the proposed method is much faster and can be viewed as a contribution.
Besides, how is the trajectory optimizer such as direct single shooting made differentiable? Is it, e.g., just by unrolling the computation graph under a automatic differentiable framework (JAX), or does it leverage some other techniques like implicit function theorem? This is unclear from the current paper. 
- Some statements are not accurate: the abstract states that the paper "enables machine learning researchers to benchmark imitation learning and reinforcement learning algorithms against trajectory optimization-based methods in challenging real-world environments". But later in the limitation section the paper states "it is at present unlikely to be useful for benchmarking RL tasks ...". Please be consistent on the statements. Also, from Table 4 in the appendix, most of the tasks can be defined with just a few (<10) parameters, which seems to indicate these tasks are not really "challenging". 

correctness: Please see details in the weakness section.
clarity: Please see details in the weakness section.
relation to prior work: Yes.
additional feedback: Please see details in the weakness section.
review_title: Review of Myriad: a real-world testbed to bridge trajectory optimization and deep learning 

final_score: rating: 7: Good paper, accept
review_content: summary and contributions: This paper introduces a differentiable trajectory optimization library for use in optimal control and end-to-end learning of trajectory optimization modules. The library uses JAX to solve constrained continuous-time, continuous-control optimal control problems, particularly via a scalable form of gradient descent-ascent on an associated Lagrangian. To support system identification or learnable dynamics, the library uses the neural ODE to support continuous-time evaluation while being efficient to train. The neural ODE may be trained via system identification (i.e., supervised learning on observed trajectories) or in an end-to-end fashion by backpropagating through the trajectory optimization procedure to perform imitation learning. The authors demonstrate their library using all of the aforementioned features on some proposed benchmark problems, including problems that are not usually considered by the optimal control/RL community (e.g., cancer treatment and population dynamics).

Post-rebuttal\
===========\
Raised my score from 5 to 7 after discussion.
strengths: The authors give a variety of benchmark problems, including those that are not usually considered by the optimal control/RL community. The included trajectory optimization algorithms should provide a good counterpart or baselines to compare against RL algorithms.
weaknesses: I’m confused what the major contribution of the paper is: is it the trajectory optimization library or is it the provided problems? If it’s trajectory optimization, there are other libraries that also provide differentiable trajectory optimization (see “Relation to Prior Work”), so this work needs to differentiate itself from the others. If it’s the included problems, they are underemphasized in the paper and need to be discussed more. Since the community is already very familiar with robotics and video game benchmarks, a discussion on these new problems (what they are, why they're interesting, aspects they have that current canonical problems don't have) can make this library a compelling choice.
correctness: The experiments in the paper seemed proper. I’m unable to run the code (see “Documentation”), so I cannot verify any results myself.
clarity: The paper does a pretty good job of showing how the optimization algorithm works, including the pragmatic approach taken for the end-to-end training. That being said, I think Section 4 is a bit too detailed. The discussion on multiple shooting seems excessive and confused me at first when I read the paper. I think that part can be moved to the Appendix for readers that are interested in what multiple shooting is, but I think it’s okay to just go over single shooting and mention that multiple shooting additionally breaks the problem up into subproblems. This should also provide extra space to talk about the included benchmark problems.
relation to prior work: The comparison to prior benchmark libraries (e.g., Gym) looks good. These prior libraries focus on robotics and video games. On the other hand, Myriad includes problems from other domains like biology and ecology, which should be of interest to the community.

I’m unsure whether the authors have properly compared Myriad to related work on differentiable trajectory optimizers. In particular, [1] uses a differentiable form of iLQR, and [2] uses a differentiable form of a Pontryagin solver. While these works are mentioned and cited, it’s only in passing and without a comparison to their work. What are the shortcomings of these two works? What does your library provide compared to these two works?

[1] Amos, B., Jimenez, I., Sacks, J., Boots, B., & Kolter, J. Z. (2018). Differentiable MPC for end-to-end planning and control. Advances in neural information processing systems, 31.

[2] Jin, W., Wang, Z., Yang, Z., & Mou, S. (2020). Pontryagin differentiable programming: An end-to-end learning and control framework. Advances in Neural Information Processing Systems, 33, 7979-7992.
additional feedback: - The paper appears to have extra spacing, which may be due to Overleaf or compiling the document on Windows. I would suggest compiling the document on a Linux program.

- Any reason you used JAX instead of PyTorch?

While the rating is a bit low, I'm happy to increase it as long as the authors address:
- Clarifying the main message of the paper (differentiable trajectory optimization vs. the included problems) and addressing my corresponding concerns
- Helping me overcome the issues running their code out of the box.
review_title: Differentiable trajectory optimization library with optimal control benchmarks

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: summary and contributions: This paper develops a library that contains a suite of real-world continuous control problems and a collection of trajectory optimization implementations, which can serve as an excellent testbed for RL practitioners for challenging control problems. 
strengths: Most RL works in the current literature focus on complex dynamics, such as games, which has discrete timesteps, or robotics, which follows end-effect control to simplify RL learning. Althoug these RL applications are great, there remains a large space of important problems of relatively simple continuous dynamics. Trajectory optimizations have been developed for decades for these problems and it remains particularly important to answer the question whether RL algorithms can be used to further improve the state-of-the-art to these problems. There are only limited works on this domain with very limited testbeds available. I personally appreciate the efforts from the authors to make such a clear library for the RL community. It will be a great testbed for potentially huge RL advances in the continous control problems. 
weaknesses: The paper is generally great. If possible, including some simple RL implementation and results would make the paper even stronger. 
correctness: The paper is technically correct. 
clarity: The paper is clearly written. 
relation to prior work: The discussion is sufficient. 
additional feedback: See my comments before.
review_title: A great library for neural continuous control

Meta-review: metareview: This paper provides a test-bed called Myriad for trajectory optimization and system ID in jax, with the hope of engaging RL practitioners to benchmark against the test bed. The proposed test-bed provides examples ranging from different domains (such as medicine, and biology) deviating from traditional domains that are usually the focus on RL benchmarks. Further, the focus is on continuous time settings. Limitations are clearly specified. Authors have done a good job of describing the testbed and comparing existing methods.

One challenge that remains to be addressed is how useful this testbed really is for RL practitioners. This concern has been raised because it seems like the dynamics of the proposed problems fall on the simpler end of the spectrum. Nonetheless, I am currently of the belief that having a JAX based test-bed for such domains is still valuable and I am hoping will contribute more to reproducible RL results in these domains. Currently it appears that the designer makes a lot of choices around the design and set up of the RL framework. It may be a real concern if the general RL community will not adopt the testbed for this precise reason. More importantly, it did strike me as odd that the contribution and abstract claim testbed for imitation learning and RL but do not provide a simple example of how this could be done. This was the main concern of tBG5. I also agree with CQCv's assessment that requiring dynamics equations to be explicitly provided by the user will significantly cost adoption of the test-bed. Reviewer vypE scored the paper very well but failed to justify the score for me to rely significantly on it. 

My expectation is that the authors will genuinely deliver on all the asks. Further, continue to improve the library to make it more amenable to testing of RL algorithms with a more friendly API. Overall I want to note that significant effort seems to be required to put together the test-bed but also believe all above concerns are valid and authors are highly recommended to incorporate as many changes as possible. The lack of explicit examples of simple baseline rl testing on the test-bed is concerning, irrespective of the API and potential simplicity of the domains (which I believe is not a huge concern if it warrants RL testing in novel domains). I strongly encourage the authors to incorporate feedback and I believe it will make for a much stronger testbed in practice. Hoping authors deliver on this, to the extent possible by camera-ready deadline and after, I am recommending an accept since the testbed has some utility even in its current form (though I am less optimistic about widespread adoption in its current form).
confidence: 3: The area chair is somewhat confident
Decision: Accept