Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: I do not have the background to give a reasonable assessment. So, please ignore this review. 
strengths_and_weaknesses: I do not have the background to give a reasonable assessment. So, please ignore this review. 
questions: No
limitations: I can't see any limitations and potential negative societal impact from their work.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a Thompson-Sampling-based method to learn how to make decisions in a class of continuous-time linear-quadratic (LQ) problems with unknown coefficients in the linear dynamics.

strengths_and_weaknesses: Pros: Reinforcement Learning (RL) for LQ problems is a tropical topic in recent years since it is the building to understand how machine learning methods perform for general decision-making problems. Thompson Sampling, on the other hand, is a practically-popular and theoretically-plausible algorithm for bandits and reinforcement learning method for Markov Decision Process (MDP) due to its efficiency in exploration. Therefore, it is interesting to see what additional benefit Thompson Sampling could bring to the LQ problem compared to the existing methods.

Cons: The contribution seems to be marginal compared to two existing papers [1] and [2]. [1] showed that the continuous-time least-squares algorithm leads to logarithmic regret and hence greedy algorithm (with no exploration) is sufficient for finite-horizon LQ problem. [2] proved an $O(\sqrt{T})$ regret for infinite-horizon ergodic LQ problem in discrete time. 

It is not clear, from both theoretical and empirical perspectives, what is the benefit of using Thompson Sampling (or Guassian exploration on the estimated model parameters). This is because it has been shown in the literature that:

(1) LG problems have self-exploration properties (due to the Gaussian noise in the linear dynamics) and the least-square estimate (exploration-free method) leads to log regret bound [1]

(2) Using linear regression to estimate parameters in the dynamics is sample efficient [3]

[1] Basei, Matteo, et al. "Logarithmic regret for episodic continuous-time linear-quadratic reinforcement learning over a finite-time horizon." arXiv preprint arXiv:2006.15316 (2020).

[2] Abeille, Marc, and Alessandro Lazaric. "Improved regret bounds for thompson sampling in linear quadratic control problems." International Conference on Machine Learning. PMLR, 2018.

[3] Dean, Sarah, et al. "On the sample complexity of the linear quadratic regulator." Foundations of Computational Mathematics 20.4 (2020): 633-679.

==========
I have read the reviews from other reviewers and the responses from the authors. The authors have addressed my concerns and I raised my rating to 6.
questions: Please address the comments listed above.
limitations: Yes.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper studies policies for systems that evolve according to Ito stochastic differential equation (1) with unknown drift parameters. The cost function is quadratic. The goals are: first, to minimize the regret; and second, to accurately estimate the drift parameters. The authors first propose Alg. 1 and show that with high probability it stabilize the process. The they propose Alg 2 using Thompson Sampling. Regret bound and estimation rates are given in Theorem 2. Experiments are conducted comparing the proposed method to reference [2].
strengths_and_weaknesses: Strengths:
There is a gap in the literature on applying TS to controlling the continuous-time diffusion process. The paper first provide algorithms and analysis for such method. Detailed proofs are given in the appendices.

Weaknesses:
The problem setting is very similar to reference [2]. The experimental results in section 6 and appendix D do not  reflect a significant improvement. In fact, the normalized estimation error using randomized estimation seems to be better than the proposed method on average.
questions: The first part of the paper focuses on stabilization. But Definition 1 is given before proper introduction. Moreover, the definition directly refers to the eigenvalues of the closed-loop matrix, which uses a linear feedback. Perhaps it would be better to give some intuitions about stability of the system and what the failure event implies. Also, does Algorithm 2 have similar stability result?

The numerical results in Figure 1 suggest that the randomized estimation performs better in terms of the normalized estimation error. How would you interpret the results?

The problem setting is very similar to reference [2]. It would be better to elaborate the introduction about [2], especially because it is also used as the baseline.
limitations: not applicable
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper discusses Thompson sampling for linear-quadratic continuous-time stochastic optimal control problems. 

The analysis and numerical algorithms are developed using a conjugate Gaussian prior distribution. 
The paper discusses thoroughly theoretical implications, including (i) a bound on the probability of not stabilizing the system under a linear exploration strategy, (ii) a bound on the squared estimation error, and (iii) a finite-time regret bound.

The developed algorithm is numerically investigated using some synthetic examples of stochastic optimal control problems.
strengths_and_weaknesses: It was very surprising to me that nobody until now did a theoretical analysis for this linear-quadratic setting. In my opinion, this paper solidly develops theoretical guarantees for the popular Thompson sampling algorithm. Since model-based RL algorithms in continuous-time are still lacking within machine learning, I very much appreciate this work.
It is nicely written, and I enjoyed reading it. I appreciate that the authors discuss the intuitive implications of the bandit analysis, which is normally filled with hard-to-understand mathematical intricacies.

A minor point I have to critique is, in my opinion, the claim in line 70:  "Additionally, through extensive simulations we illustrate that TS enjoys smaller average regret and substantially lower worst-case regret than the existing RL policies, thanks to its informed exploration.", which is a bit far stretched, as this work only compares one other algorithm to this Thompson sampling strategy.

This brings me to my main critique point, which is that the topics adaptive control and dual control (Bayesian reinforcement learning within machine learning) are not sufficiently mentioned in the paper, see, e.g., [1]. Theory discussing simultaneous optimal estimation and control is unsurprisingly very old. This goes back to the more than 50-year-old works of Feldbaum [2] and is since then been discussed within the control community. This topic, which in principle optimally solves the problem has to be discussed within the related work.
This would also give a nice numerical example. For example, there are very simple dual control problems, see, e.g., [3], where one could try to find the solution to the HJB equation numerically using the finite difference method. This would result in a control strategy that Bayes-optimally selects the actions and hence, balances exploration and exploitation optimally.

However, all critique aside, I am of the opinion that this paper is a solid contribution to the community.

- [1] Stengel, Robert F. Optimal control and estimation. Courier Corporation, 1994.
- [2] Feldbaum, Aleksandr Aronovich. "Dual control theory. I." Avtomatika i Telemekhanika 21.9 (1960): 1240-1249.
- [3] Florentin, J. J. "Optimal, probing, adaptive control of a simple Bayesian system." International Journal of Electronics 13.2 (1962): 165-177.

questions: Something I did not understand was in the paragraph in line 131:
"We assume that the process (1) with the drift parameter [...] is stabilizable. Therefore, [...] exists, is unique, and can be computed using continuous-time Riccati differential equations similar to (5), except that the zero matrix on the right-hand side will be replaced by the derivative of [...] [32–35]."
What is meant by this? I know of the Riccati equation for finite-time optimal control problems, which is an ODE. Though, here an average reward criterion is discussed. Maybe the authors can elaborate a bit on this.

In line 192: "Nonetheless, if there is no such prior, we simply let [...] and [...]." This formulation confuses me. I think it shadows a bit that the hyper-parameters are set in this way because in the proceeding bounds the hyper-parameters $\hat{\mu}_0$ and  $\hat{\Sigma}_0$ are not appearing anymore. Some explanation for this would be appreciated.

The last question I have is: Would it be possible to also derive these bounds under different posterior distributions than a Gaussian? For example when using a sparsity-prior.
limitations: Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This paper proposes and analyzes a Thompson-Sampling based method to learn to control continuous-time linear systems when the costs are quadratic. The authors first propose an algorithm that guarantees stabilization of the diffusion process and then give a second, Thompson-Sampling-based method with regret bounds and estimation rates for the parameters of the linear system. 

The reviews for this paper were generally positive and found this work to positively contribute to our understanding of linear control--- though several reviews noted the similarities with with reference [2] and a general lack of contextualization of the work in the general adaptive and Bayesian control literatures. Nevertheless the results were sound---and extended our understanding of learning and control in the LQ setting, and the paper was well written and easy to follow. 
confidence: Certain
Decision: Accept