Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This data identifies a class of (discretized) optimal control problems for which a quadratic (with respect to actions) approximation to the problem's Q-function is accurate and gives rise to near-optimal policies.  The proposed quadratic parameterization of the Q-function is similar to, but slightly more general than, that proposed by the original Normalized Advantage Functions paper, and this additional generality is necessary.  The authors test several implementations of this approximation on control problems and find it to be effective.
strengths_and_weaknesses: This paper provides a useful theoretical contribution by characterizing a set of RL tasks for which NAF-style Q-function approximations are suitable.  The explanations and derivations are fairly clear and the experiments are easy to follow.

My principal concern with the paper is that its main contribution, while useful, is rather limited in scope.  It is unclear to me whether NAF-style approaches are competitive with the state of the art in continuous action-space RL problems.  The authors do compare against DDPG, which is nice, but not against stronger baselines like SAC.  The relevance of this paper's theoretical contribution to the community is ultimately somewhat dependent on the performance of NAF-style algorithms relative to alternatives.

I think the paper has another major shortcoming independent of the above concern, which is that there is no analysis of problems that *do not* lend themselves to a quadratic Q-function approximation.  Obviously the space of all MDPs is too large to categorize in a single paper, but the authors could help clarify the applicability of NAF by analyzing which aspects of the problem formulation in (6) are essential to the paper's theoretical and empirical results.  For instance, how does the suitability of the NAF approximation, and its actual performance, scale as the dependence of cost on u(t) becomes increasingly non-quadratic?  

Finally, two comments on presentation: (1) I think the motivation for the equation (19), and the three implementations in section 5, could beimproved.  As I understand it, the deviations from the original NAF algorithm are made to account for the possibility that the optimal action in a given state may not belong to the set of actions to the agent.  This point could be made clearer in the text.  (2) The reward-based BNAF and gradient-based BNAF algorithms are introduced rather abruptly, without much motivation.  Under what conditions are we likely to know r(t, x) or g(t, x)?  When should we intuitively expect (without seeing the empirical results) these algorithms to outperform vanilla BNAF?
questions: (1) How does the performance of BNAF (+ variants) compare to more state-of-the-art RL algorithms (e.g. SAC) for continuous action spaces, and to what extent is BNAF compatible with various implementation details that improve performance in these baselines?

(2) How gracefully does BNAF handle violations of the assumptions in equation (6)?
limitations: The authors are clear about the limitations of BNAF and related algorithms, unambiguously defining a set of problems to which these approaches are applicable.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Q-learning is a popular approach to reinforcement learning that is difficult to extend to continuous action spaces. Normalized advantage functions (NAFs) address this issue by assuming that the Q function approximation is quadratic with respect to the controls. This enables us to determine the policy by efficiently computing the argmax with respect to controls of our approximated Q function. While it seems to work in practice, which systems this approximation is suitable for is an open question. To address this concern, the paper analyzes the case where the dynamics are control-affine, the cost is quadratic in the controls, and the problem is formulated in continuous time. First, the authors map the continuous-time optimal control problem to a discrete-time reinforcement learning problem. Next, they propose a slight variation of the NAF approximation family and show how to set the terms given knowledge of the known optimal control problem to bound the Q-function approximation error. They then present how this translates into a bound on the performance. Additionally, they show that these bounds do not hold if we use the original NAF approximation via a counterexample. Finally, the authors evaluate a couple variants of their new approximation family on a set of benchmarks and compare their performance to both the baseline NAF implementation and a policy gradient method.
strengths_and_weaknesses: The question of which control problems the NAF approximation is acceptable to is a relevant and important question. Moreover, the approximation and performance bounds presented in this paper are applicable to a useful class of problems that apply to a wide variety of real-world robotics tasks. The authors also present a number of variants of their Q-function approximations which can incorporate knowledge about the reward and dynamics if known. The experimental results illustrate the benefits of this extended approximation family and the variants in terms of the overall reward achieved on a number of benchmarks. Additionally, the paper is fairly well organized and explains most of the relevant background material.

However, the addition of control constraints to the problems considered seems to be changing the question the paper originally asked. Specifically, there's the question of which problems the NAF approximation is suitable for, and then there's the question of how to adapt NAF in the case of control constraints. Both are useful questions to ask, but they need to be disentangled somehow. For instance, it is unclear if the proposed extended approximation family would be important in the case of unconstrained controls. The main difficulty in applying Theorem 1 and 2 to the original NAF approximation appears to be that we cannot set the policy directly to the optimal as in (21) due to the presence of these constraints. This is what motivates the introduction of the new family in (18, 19). However, in the case where the controls are unconstrained, it seems like the original NAF approximation should be sufficient. While the theorems presented in this paper are interesting and useful regardless, the authors should elaborate on this point.

Moreover, even if the controls are constrained, it is hard to tell from the experiments if the benefits observed really come from this new family or just from incorporating additional information into the Q function. For instance, in the case of box constraints on the controls, one could easily set $\mu$ in the original NAF family such that the neural network output is also clipped or passed through a scaled sigmoid. Similarly, for GB-BNAF, one could also set $\mu$ to be $\frac{1}{2} r^{-1}(t,x) g^T(t, x) \nabla_x V(t, x | \theta^V)$ but clipped or passed through a scaled sigmoid. It would greatly strengthen the paper to disambiguate the source of benefits by performing some ablations. 

Finally, there are a number of grammatical errors throughout the paper that need to be addressed. A sentence or two describing other approaches besides NAF to tackle continuous actions in Q learning would add to the paper. It would also be nice for the authors to include a conclusion section instead of just ending on the related work.
questions: *  If the controls are unconstrained, would Theorems 1 and 2 hold for the original NAF approximation or is there something more that would prevent this?
* Are the observed benefits of BNAF and its variants really due to this new approximation family? Would I observe similar benefits if I used the modifications to NAF I discussed above?
limitations: The authors do discuss the limitations of their work in the introduction. Specifically, they mention that the analysis in this paper only holds for a specific class of problems. Moreover, the Q function approximation they propose is not the true form, even in the more limited class of problems they consider. However, they are able to bound the approximation error and performance, which significantly adds to the paper.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The paper provides analysis of a method for solving optimal control problems based on approximating the value function by means of deep neural networks that results in a quadratic parametrization of the value function in the control effort, thus allowing for an efficient solution of the optimization step needed for deep Q learning. Several theorems are proved to identify and analyze the class of optimal control problems this method is applicable to. Several variants of the algorithms are investigated empirically on four test problems, in comparison with one well known method for continuous Q learning, DDPG. 
strengths_and_weaknesses: The main contribution of the paper is probably the empirical investigation of the proposed method that illustrates that two variants of the algorithm, RB-BNAF and GB-BNAF, have significantly higher performance than DDPG on all test problems, and as such, could be good candidates for solution methods for optimal control problems of a suitable class. Analysis demonstrates that this class included non-linear control-affine dynamical systems with cost functions quadratic in the control effort. These methods could be a good addition to the toolbox of practitioners. Both methods require at least partial knowledge of the system being controller (the reward function and the control-affine term in the dynamics), so these methods cannot be considered proper reinforcement learning methods, but as optimal control methods, they appear to be valid and quite competitive.

However, because the basic NAF algorithm is already known in the literature, this paper's contribution is relatively minor. The class of optimal control problems the methods are applicable to is important in practice, but not that universal. The authors are arguing that by adding a quadratic term to the cost function, many control problems can be made to be of the kind they are considering, but this is a somewhat contrived way of claiming the generality of the method.  


questions: Is DDPG really the current state of the art to be used as a baseline? Shouldn't PPO and/or SAC also be included in empirical comparisons?

On page 2, line47, by "quality index", do you mean the cost (respectively, reward) function?

A minor typo: 
P.8 L.199: "greedy politics" -> "greedy policies"
limitations: I think the authors have adequately addressed the limitations of the proposed methods - in fact, a major objective of the paper is to analyze and understand which classes of optimal control problems these methods are applicable (and possibly, limited) to. I do not see any negative societal impact of this work.
review_title: null

Meta-review: metareview: I am happy to recommend accepting this paper.

I would argue that the stated contribution of this paper - an analysis of when NAF is a good approximation - is somewhat minor.  But the analysis in the paper has a nice additional benefit in that it becomes possible to include domain-specific knowledge (e.g., on the reward function, which I agree is often known) in a straightforward and effective way into the algorithm.  Furthermore, the paper is fairly well executed.

I would encourage the authors to have another careful look at all the comments by all the reviewers, which could make the paper better.  I would particularly agree that an analysis (empirical or theoretical or both) of what happens when the assumptions are violated would make the paper better.
confidence: Less certain
Decision: Accept