Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a method for model-based RL, particular when time is continous and the problem is partially observable. They test there method on two (toy) problems and show increased regularization and/or data efficency.
strength_and_weaknesses: Strengths:
- The topic of the paper is in a relevant field of research: most works assume discrete time and a fully observable MDP
- The writing appears mathematically rigorous

Weaknesses:

- The paper is not written in a way that it is easy to understand. As someone who is very familiar
with model-based RL, it was very hard to get an understanding of the method. For instance the introduction 
of the method in 3.1 is very complicated. 
-  Experiments are restricted to artificial toy problems. While the problems may be complex I cannot access the applicability to real-world problems
- The evaluation is limited: It seems the only quality parameter the authors check for is the training resolution which would check for data efficency and robustness


Minor:
In Section 5 you write: "For a fair comparison, the hyperparameters of each model are adjusted such that the models
all have an approximately equal number of trainable parameters"

It seems you are mainly testing for data efficiency & robustness (e.g. Table 2) and on toy problems.
I don't see how using the same number of trainable parameters would be reasonable here --
because you are not testing for scalability (speed, memory footprint etc.). Perhaps some models like LSTM require more over-parameterization to work correctly.

clarity,_quality,_novelty_and_reproducibility: Stochastic differential equations and continous time control is not my field of researc so my review of this paper can unfortunately only be limited.

The paper is not written in a way that it is easy to understand. As someone who is familiar
with model-based RL, it was very hard to get an understanding of the method. For instance the introduction 
of the method in 3.1 is very complicated.  Perhaps the authors could start by providing a high-level intuition of how the method
works to make it more approachable.


summary_of_the_review: Overall, I cannot provide a strong opinion on acceptance or rejection. I cannot check the correctness of the math in Section 3. Having said that, I found the empirical part in Section 5 to be both limited (restriction to toy problems) and coarse (the only KPI being performance degradation w.r.t. training resolution).  

I will assume the derivations to be correct, but because of the weaknesses in the empricial evaluation I have doubts on empirical significance.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper discusses a non-Markovian stochastic optimal control problem, with path-dependent dynamics and costs. It discusses the modeling of such problems and presents a numerical method based on a policy optimization procedure. For unknown dynamics, a model-based RL setup is considered, where the algorithm simultaneously learns the dynamics and optimizes the policy. The method is evaluated on two synthetic non-Markovian stochastic control problems.
strength_and_weaknesses: The setup of including the whole path of the stochastic process into the stochastic optimal control problem is new to me. This setup is probably interesting for many applications. 
The theoretical contribution is solid as far as I can tell.
However, I think the developed algorithm is rather simple and pretty naive. 
For example, I think that directly differentiating through the loss is usually not a good idea, as it does not scale very well and something like an adjoint method should be used, see, e.g.,  [1]. The model-based RL setup is also very crude. The proposed inference setup is similar to a method of moments. However, I think maximum likelihood estimation or a full Bayesian setup should be preferred for model estimation, see, e.g. dual control [2] or Bayesian RL [3]. It is not discussed how exploration and exploitation are balanced, though it is a very important component for model-based RL.
The presented synthetic control problems are interesting, though, I would have hoped for a nonlinear problem and a bit more motivated real-world example. Overall I think that the contribution is a bit too marginal.

Typos:
- below equation (1) the definition of $\mu$ and $\sigma$ is not correct (wrong output space for $\mu$)
- Sec 3.2 spaces are not correctly defined (wrong dimension parameters)

[1] Kidger, Patrick, et al. "Efficient and accurate gradients for neural sdes." Advances in Neural Information Processing Systems 34 (2021): 18747-18761.
[2] Stengel, Robert F. Optimal control and estimation. Courier Corporation, 1994.
[3] Ghavamzadeh, Mohammad, et al. "Bayesian reinforcement learning: A survey." Foundations and Trends® in Machine Learning 8.5-6 (2015): 359-483.
clarity,_quality,_novelty_and_reproducibility: - The paper is written pretty well.
- The results are stated clearly
- The setup is novel and interesting, though, the algorithms are pretty naive and not very novel.
- The algorithms should be pretty easy to implement and therefore the results should be reproducible
summary_of_the_review: The paper presents an interesting setup. The results are solid, however, in my opinion, the presented algorithms are pretty naive and are missing many modern aspects of optimal control and reinforcement learning.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper considers the class of non-Markovian stochastic control problems in continuous time. In particular, two cases have been investigated. In the first case, the drift and diffusion coefficients are path-dependent. In the second, a fractional Brownian motion captures the randomness. The paper proposes a numerical approach to finding the optimal policy using neural rough differential equations (Neural RDEs). In the next step, the same problems have been considered under the situation that the system's true dynamics are not known. Although the "REINFORCEMENT LEARNING" term has been used to describe this problem, as the estimation and optimization steps are not taken simultaneously (see the last paragraph of page 6), it is important to note that there is no exploration vs exploitation trade-off present. Finally, according to the numerical examples, the proposed scheme in the paper outperforms the previous approaches.
strength_and_weaknesses: S1- The paper considers an important problem that might be a matter of concern in different applied settings.

S2- Although there are a few typos and long sentences, the paper is well-written and the storyline is clear.

S3- The numerical experiments are discussed clearly.


W1- On the theoretical side, the paper is weak. The informal version of the main (or only) theoretical result is not informative.

W2- In the numerical experiments, only a finite memory is considered, which does not capture the general non-Markovianity, and is reducible to a Markovian problem.

W3- Several technical terms are not defined, including but not limited to the frequently used (N) RDEs.

W4- Restriction of the control policy to a parameterized family with such structures need much intuitive and formal motivations that none is provided.

W5- The incentive for following such a computationally heavy approach where much faster approximation procedures that are capable of fine tuning are not discussed, is a missing point.
clarity,_quality,_novelty_and_reproducibility: The paper is clear and well-written, and demonstrates an acceptable presentation, except the missing technical explanations.
summary_of_the_review: In general, the paper provides an idea to numerically solve non-Markovian stochastic control problems in continuous time, but further required elaborations are missing. I also found the literature review a little outdated/incomplete. I will try to finalize my decision after the rebuttal period to one of the accept or reject options.

_________________________________
post rebuttal: I did not find the rebuttal convincing enough, and edited my review with further informations that are required but missing in the paper.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: 
In this paper, the authors proposed neural rough differential equations model as a model-based reinforcement learning for continuous-time and non-Markovian stochastic control problem. Particularly, the model parameters are learned by minimizing MMD, and the policy is parametrized as a recurrent function. The optimal policy is then obtained by first-order method. 

There are several major issues raised by reviewers:

1, the method is relatively straightforward and the derivation is lacking of justification. Specifically, 

- the justification of the parametrization of policy has not been provided;
- the gradient-based optimal policy seeking may lead to suboptimal policy due to the highly nonlinearity in the objective and dynamcis;
- there is no exploration vs. exploitation mechanism in the proposed method.

2, the empirical experiment is weak. The method is only evaluated in synthetic problem, and no standard benchmark, e.g., MuJoCo, has been tested. There is no state-of-the-art RL algorithms, e.g., [1], have been compared with. It is hard to evaluate the significance of the proposed method.

[1] Wang, Tingwu, Xuchan Bao, Ignasi Clavera, Jerrick Hoang, Yeming Wen, Eric Langlois, Shunshi Zhang, Guodong Zhang, Pieter Abbeel, and Jimmy Ba. "Benchmarking model-based reinforcement learning." arXiv preprint arXiv:1907.02057 (2019).
justification_for_why_not_higher_score: 
The method is straightforward, and the design choice (e.g., policy parametrization and gradient-based planning) is unjustified. 

The empirical comparison is weak on synthetic problem and lacking of competitors.
Decision: Reject