Venue: CoRL 2022 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper investigates sample-efficient fine-tuning of controllers by adding an approximate control Lyapunov function to the conventional cost employed for policy learning. This addition enables the usage of smaller discount factors during adaptation. The approach is demonstrated for sim2real transfer on a pendulum swing-up task, as well as reducing tracking error of a model-based controller on a quadrupedal robot.
strengths_and_weaknesses: **Strengths:**

The paper focuses on an important topic as training policies for robotic hardware (from scratch) is generally expensive, and being able to efficiently fine-tune a given controller to the specific hardware setup can be crucial. Overall the paper is well-written and formatted, clearly states its assumptions and does not overstate its contribution. The authors provide a diverse set of experiments including fine-tuning with a learned policy/value function as well as adapting based on a model-based controller. The hardware demonstrations further underline real-world applicability.

**Weaknesses:**

The paper would profit from elaborating on failure modes and limitations. For example, it would be helpful to discuss the assumption of positive definite cost matrices in the context of general RL problems. It would furthermore be interesting to ablate on the quality of the learned value function (pendulum) to see how well-suited the value function is at different stages of SAC training (or even alternative algorithms). The results in Figure 3 show that the velocity tracking error has been reduced, in fact it now leads the target instead of lagging (elaborate on why?). The resulting controller now appears more shaky than before, which may be due to visualization. It would have been interesting to see a video of the resulting gait to better assess qualitative performance, but a discussion of these phenomena would be good. There are also several instances where the reader is referred to prior work (e.g. construction of CLF), while inclusion in the appendix would make the paper more self-contained.

**Minor:**
- Line 55: hand-desired --> hand-designed
- Figure 2: increasing legend and axis label size for better visibility
- Figure 4: could be helpful to provide dotted red lines to highlight x & y intercepts of black dots;
- Figure 4: floating line above right panel
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: The paper aims to address the important topic of sample-efficient fine-tuning particularly in light of hardware transfer, which is valuable for the robotics community. The exposition is generally well-written, while it would benefit from some expansion of the limitations section (see above). Overall, the paper illustrates the effectiveness of the proposed formulation sufficiently well in the context of the experiments, although additional experimental evaluation would help strengthen the paper. 

------- Post rebuttal -------

The authors have addressed the majority of my comments justifying an increase to weak accept.
issues: - Expand discussion of assumptions and limitation
- Details on value function quality for the pendulum SAC experiment, how is down-stream performance affected over different training snapshots of the first stage
- Expand discussion of quadruped tracking
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The authors address the problem of efficiently learning and fine-tuning stabilising controllers on hardware by introducing a novel cost-shaping method based on Control Lyapunov Functions (CLF). CLFs provide some guarantees about the stability of controllers and an objective that can more greedily be optimised. The latter is useful for data efficiency, as it allows to learn with a smaller discount. The authors provide some bounds on the range of stable discounts given the sub-optimality of a policy. Experiments include fine-tuning a cartpole controller on hardware in seconds, a velocity-tracking controller on a quadruped in minutes, and an inverted pendulum in simulation, the latter for which shows the flexibility and robustness of the approach to approximate CLFs.
strengths_and_weaknesses: Strengths:
- Data efficiency in fine-tuning is a critical topic for robot learning research
- Mixing in ideas from control theory comes with convenient benefits, such as theoretical guarantees and bounds that can help tune parameters (e.g. discount)
- Hardware experiments on both a cartpole and a quadruped
- Flexibility and robustness to choice of CLF, incl. using only a locally-valid approximation and a value function trained purely in sim
- The latter especially may make this approach fairly easily widely applicable
- Paper reads well and provides sufficient theoretical backing

Weaknesses:
- Quite strong assumptions about the nature of the control problem to be solved (stabilising, deterministic, fully-observed, specific shape of cost), though this is pretty common in more theoretically-based work
- The fine-tuning results for the hardware experiments should be included in the main paper
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Additional details required
summary_of_recommendation: Overall I'm leaning towards accepting this submission. It provides a cleanly-presented, theoretically-founded approach for improving data efficiency during fine-tuning, and the results seem to prove the effectiveness. I'm reluctant to make it a strong accept due to the underlying assumptions of the approach and how well those will hold for modern robotics problems, e.g. learning to manipulate from pixels.
issues: - I would recommend to include the hardware fine-tuning baseline in the main paper
- There are "Remark", "Assumption", "Proposition", "Definition", "Theorem" and "Lemma" counters, perhaps these can be omitted to improve readability?
- Minor typos:
  - l120: R is the input [cost]
  - l131: of the from -> of the form
  - l140: the inequality needs to be flipped?
  - l176: + C -> - C
  - l204: missing \gamma
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper considers the problem of the poor sample complexity of RL. To address the problem, the paper introduces a cost-shaping method, which adds a control Lyapunov function (CLF) to the cost formulation. The paper theoretically proved that the proposed cost formulation can lead to a stable controller when small discount factors are used, and therefore reduces the sample complexity. The proposed method is demonstrated on a cartpole and an A1 quadruped. 
strengths_and_weaknesses: 
The problem that the paper considers is important. The proposed method is technically sound, and the experiments demonstrate that the proposed method can indeed stablize the system with higher sample efficiency in real-world environments. The writing of the paper is good.

However, the most important weakness of this paper is that adding a CLF term in the cost function of RL is not a novel idea. Several previous works [1,2] use the same idea to address the stability issues in RL. Moreover, [2] does not need a given CLF approximation, but learns the CLF with the RL process directly. Therefore, the authors may want to discuss their contributions more clearly compared with [1] and [2]. Other weaknesses include the unclear problem statement, the missing comparisons with baselines, the missing ablation studies, and the missing details of the experiments.

[1] Minghao Han, Lixian Zhang, Jun Wang, and Wei Pan. Actor-critic reinforcement learning for control with stability guarantee. IEEE Robotics and Automation Letters, 5(4), pages 6217-6224, 2020.

[2] Ya-Chien Chang and Sicun Gao. Stabilizing neural control using self-learned almost lyapunov critics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1803–1809. IEEE, 2021.
robotics_focus: Sufficient demonstration on hardware
quality_of_the_limitations_section: Limitations are addressed clearly
summary_of_recommendation: Although the paper considers an important problem, the conparison with the previous work is not enough. Two aforementioned papers seem to solve the same problem under milder assumptions. In addition, there are issues with the problem statement, and the experimental results cannot fully support the claims that the paper makes. Therefore, I give this paper a weak reject.

-----------Post discussion Note-----------
I would like to thank the authors for providing additional explanations, experiments, and the revision. The additional comparison with the two works I brought up has addressed my concerns about the novelty of the work. I agree that this work can be complementary to the two works I mentioned. In addition, I found that the additional explanations and details of the experiments are clear and can address my concerns. It could be better if the authors provide more details about how to choose the best discount factors in practice and how the performance of the proposed algorithm changes with different discount factors. I would like to change my score to a weak accept.
issues: 1. The comparsion of the proposed method and the aforementioned two works is missing. Could you state the new contribution of the paper and the key difference of the reward design technique? Could you compare the proposed work with the aforementioned two works in experiments if the problem settings are the same?

2. The problem statement is not very clear to me. Do you want to maximize the cummulative reward (standard RL setting), or just to make the learned controller stable? In the former case, will the added term in the value function make the cummulative reward higher? This is not supported by the experimental results. In the latter case, what's the function of the reward function if you have a CLF approximation?

3. One of the advantanges of the proposed algorithm is the increased sample efficiency. However, this advantage is only proved by a simple inverted pendulum system. For the carpole and the Unitree A1 robot, the paper only provides the conparsion of the fine-tuned policy and the nominal policy, which cannot support the authors' claim about the sample efficiency. In addition, the conparsion of the fine-tuned policy and the nominal policy seems a bit trivial since the fine-tuned policy is supposed to be better. What if we only fine-tune the policy using the original reward function in the real world?

4. In Figure 4, the control limits, the discount factor, and the existance of the CLF vary simultaneously. This is not sufficient to show the efficacy of the proposed method as we cannot observe which specific variable controls the change of the performance. The authors may want to provide more ablation studies to settle this issue.

5. The ablation of the approximation of the CLF is not provided. Will the quality of the approximation affect the learned controller a lot?

6. Could you provide the cummulative reward of the trajectories w.r.t. the training epochs? Since RL is proposed to maximize the cummulative reward, can the proposed method do better? 

7. Can you provide some inituitions behind the assumptions propsed in the paper?
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This  paper proposed a novel way of utilizing Lyapunov functions as the reward function to accelerate the RL process with small discount parameter. Both theorical analysis and real hardware experiment were provided.
strengths_and_weaknesses: Strengths
1. The toptic is very interesting
2. The theorical analysis could support the proposed method

Weaknesses
The main concern of the reviewer is about the experimental results. At the begining of the paper, the experiment looks very cool wicth includes both pendulum and quadruped. On the other hand, the  actual expriment design is not that attractive:

The pendulum task is relative easy for SAC. In this paper, the value function of SAC learned in simulation was selected as the CLF, one part of the reward function to help the policy based RL to quickly learn the real task with zero discount parameter. The policy is based on the SAC neural network which has learned relatively good feature to this task; The reward is also based on the value function which is for modeling the long-term reward with the previous discount parameter. The quadruped experiment is over simplified, especially when the CLF is already given. 

Overall,  It is difficult to judge the practical value of CLF without addressing how hard the current tasks are for existing RL approach.
robotics_focus: Relevant but unlikely to deploy to hardware in near future
quality_of_the_limitations_section: Limitations are not well addressed
summary_of_recommendation: Weak rejection for the limition in the current experiment.
issues: Please address my comments above.
review_title: null

Meta-review: metareview: The paper introduces a theoretically-founded approach to an important problem of efficient fine-tuning. It is well-written, clear, and contains hardware experiments including a quadruped. However, the experimental evaluation is rather simple and lacks comparisons to baselines. 

Main pros:
- The work is theoretically founded and analyzed
- The paper is well-written

Main cons:
- The limitations should be discussed in more detail
- The experimental evaluation is limited and does not fully demonstrate the contributions
- Lack of comparison with other papers that also use CLF term in the cost function of RL

Post-rebuttal update: The revised version of the paper largely addresses the main cons, and especially offered stronger experimental evaluation, further explanations and discussions. 
confidence: 4: The area chair is confident but not absolutely certain
Decision: Accept (Poster)