Venue: CoRL 2023 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: This paper introduces a novel model that addresses sampling-based stability issues by employing Lyapunov stability certification. Additionally, the model combines this approach with the actor-critic algorithm to achieve optimality and stability in various simulated robotics tasks, outperforming the baselines.
review: Pros:

- This paper presents solid mathematical formulations supported by numerous math proofs in the appendix.

- The authors have invested significant effort in presenting math equations, tables, and figures with a professional and graceful appearance.

Cons:

- Unfortunately, I found it challenging to follow the paper's storyline. In my opinion, the main manuscript lacks self-containment, with insufficient descriptions and explanations. To comprehend the paper, readers are expected to possess the same knowledge and research experience as the authors.

- The main manuscript lacks a connection to related works. While I understand that an 8-page limitation is restrictive for a theory-heavy paper, the absence of a proper discussion of related works makes it difficult to establish a knowledge framework and differentiate this work from previous research. Although the authors include related works in the Appendix, it cannot be assumed that reviewers will read it.

- The paper lacks a clear motivation for this research and its connection to standard Reinforcement Learning (RL). For instance, it would be helpful to explain the difference between using cost and reward in the MDP description and how Cartpole-cost differs from the standard Cartpole task in RL. Additionally, a more detailed description of the costs used in the experiments would be valuable.

- Some of the math definitions and theorems appear to be directly taken from other papers without proper citation. For instance, "Mean Cost Stability" in Definition 2.1 can also be found in Definition 1 of [9], yet the authors state in line 68 that they introduce it as the stability condition in this paper.

- Many of the mathematical formulations in sections 3 and 4 lack proper explanations, making the paper challenging to understand, especially for readers with limited knowledge in the field. For example, from line 86 to line 100 (theorem 3.1), it is unclear what this theorem represents, why it is necessary, and its connection to the formulation in line 98. More reader-friendly explanations are needed to provide insight and intuition, even though the math proofs seem solid. 

- Please provide additional descriptions of the experiments, explaining the task settings and the rationale behind their selection in this paper.

- Despite these drawbacks, I believe this paper is founded on solid mathematical principles. However, the missing part blocks conveying these concepts to the readers. Therefore, I suggest the authors consider whether CORL is the appropriate venue for this work, given the heavy emphasis on theory and math proofs in your work. A journal with a main manuscript length of 20+ pages might be a more suitable choice, allowing for more extensive explanations to ensure a technically smooth and coherent narrative.



Critical issue:

- CORL explicitly requires a limitation section in the main manuscript, which this paper failed to include, putting it at risk of desk rejection (https://www.corl2023.org/instructions-for-authors/). I recommend that the authors carefully review the submission guidelines in the future.


$\textbf{Updated review after rebuttal:}$

Most of my concern has been addressed by the authors. I am in favor of the paper now.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I cannot recommend this paper as I faced difficulties in understanding the main math definitions and theorems presented in this work. The lack of comprehensive motivation and explanations hinders my ability to grasp the work's contribution and judge its significance.

Nevertheless, it is evident that this paper builds upon a very solid mathematical foundation and includes well-structured proofs. If other reviewers with expertise in control theory and stability analysis hold differing opinions, I would suggest considering their recommendations more seriously than mine.

$\textbf{Updated review after rebuttal:}$

After reading the revised version of the paper, I recommend this paper now. The authors spent a lot of the effort to improve the readability of the paper and allow the researchers in a broad area to understand this work. Combining these updates with the solid math fundamentals, I recommend this work should to be accepted.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: The paper presents a Lyapunov-based approach for policy optimization. The state-action value function (critic) is replaced by a Lyapunov candidate, the expected discounted sum of costs, under certain MDP stability conditions. The critic can then be trained by optimizing the standard TD-learning objective. The policy update is formulated to achieve cost optimality such that the cost incurred at a step tends to zero while satisfying stability conditions added with suitable Lagrange multipliers. The multipliers are updated to increase the strength of the constraints gradually and reach the equilibrium state. A constrained critic network is used such that the Lyapunov value of state action is always positive, zero if the state is an equilibrium point. The authors compare the proposed setup with prior lyapunov-based methods on ten simulated environments. Ablations are shown to study the importance of the strength of the stability constraint.
review: The paper presents a generalized approach to sThe paper presents a generalized approach to incorporating convergence stability in policy optimization. The lyapunov critic follows standard Q-function formulation but with cost (instead of reward) and the goal is to minimize the state-action value. The Lagrangian constraint-based policy update rule is intuitive and theoretically justified.

The algorithm is compared with relevant baselines using Lyapunov based formulations. The analysis on stability constraint violations is not explained. Is the same metric used for all the algorithms? What is the metric?

The paper mentions optimal time convergence, how is the convergence time optimal? Does that mean ALAC is sample efficient as compared to the baselines? Such a conclusion cannot be drawn from Fig 10. Can you clarify?

For generalization experiments, how do you calculate the errors like what is a desired goal and an achieved goal? What is a reference signal? Can you please clarify further, maybe, with an example from an environment?
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: I believe that the paper introduces an interesting direction in using lyapunov-based formulations for policy optimization in RL. It introduces new stability conditions and establishes maximum deviation bounds w.r.t. practical implementation. The comparison results are also extensive. However, the results are not convincing from a robotics perspective, as for both Minitaur and Humanoid, the method fails to surpass prior works.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary_of_paper: In this paper, the authors propose a novel model-free reinforcement learning (RL) algorithm. This algorithm (ALAC) borrows techniques from Lyapunov stability analysis to achieve optimal behavior under some stability constraints. The proposed approach is integrated within the Actor-Critic framework. The method is motivated by a theoretical analysis, the algorithm implementation is derived from the theoretical analysis. Finally, the algorithm is successfully implemented on 10 RL tasks related to robotics problems. Comparisons with existing RL algorithms baselines are provided.  
review: The paper provides strong and detailed theoretical results on optimization of accumulated cost under stability constraints. 
The paper demonstrates the superiority of the proposed algorithm compared to existing baselines on a series of RL robotics tasks in simulated environments. 

It would be extremely valuable if the author could clearly state what are the approximations that are made when transitioning from the theory to the actual algorithm. What are the choices dictated by the theoretical analysis and which choices are arbitrary, or heuristics that tend to work well in practice. 

For instance, why is the Lagrange multiplier lambda clipped between 0 and 1? Enforcing positive multiplier is expected for inequality constraints, however it is unclear why the algorithm requires clipping multiplier values larger than 1. The choice seems somewhat arbitrary. It would be great if the authors could motivate this choice with additional details.

Overall, it is hard for the reader to distinguish between the algorithm design choices that are directly stemming from the theoretical analysis and the ones that are only loosely related to the theory section or simple heuristics. 

Minor issues:
cultivated costs  <- accumulated costs
satisfy focal stability <- local stability
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: This paper propose a new algorithm to solve RL problems under stability constraints. The proposed approach is backed by theoretical results and is benchmarked under a number of baselines on a set of robotics tasks in simulation.
review_title: null

Meta-review: The paper presents a rigorous mathematical framework for policy optimization with sampling-based Lyapunov stability. The initial concerns of the reviewers regarding the clarity of the presentation was well addressed by the rebuttal. All reviewers now agree that the paper should be accepted  due to its rigorous mathematical derivation and the convincing results with comparison to relevant baselines. I agree that the paper contains interesting contributions and is of high quality. I therefore recommend acceptance.
Decision: Accept (Poster)