Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning
Authors: Tyler Westenbroek
Fernando Castaneda
Ayush Agrawal
Shankar Sastry
Koushil Sreenath
Venue: CoRL 2022 Poster (2022)
Keywords: Reinforcement Learning
Control Lyapunov Functions
Stability
Robustness

Abstract: Recent advances in the reinforcement learning (RL) literature have enabled roboticists to automatically train complex policies in simulated environments. However, due to the poor sample complexity of these methods, solving RL problems using real-world data remains a challenging problem. This paper introduces a novel cost-shaping method which aims to reduce the number of samples needed to learn a stabilizing controller. The method adds a term involving a Control Lyapunov Function (CLF) -- an `energy-like' function from the model-based control literature -- to typical cost formulations. Theoretical results demonstrate the new costs lead to stabilizing controllers when smaller discount factors are used, which is well-known to reduce sample complexity. Moreover, the addition of the CLF term `robustifies' the search for a stabilizing controller by ensuring that even highly sub-optimal polices will stabilize the system. We demonstrate our approach with two hardware examples where we learn stabilizing controllers for a cartpole and an A1 quadruped with only seconds and a few minutes of fine-tuning data, respectively. Furthermore, simulation benchmark studies show that obtaining stabilizing policies by optimizing our proposed costs requires orders of magnitude less data compared to standard cost designs.

L Lyapunov Design for Robust and Efficient Robotic Reinforcement Learning
Tyler Westenbroek westenbroeektOerrkelleyccco
Fernando Castañeda Fcastaneda@berkeeeyyooo edu
Ayush Agrawal2:* syush..ggrawal@eerkel.... edu
Shankar Sastry? sastry&coe wtw.eryy@@oeb.bbeeeyy........ edu
Koushil Sreenath" soushils@berdee .dd
DDpartmeen of Electrical Engineering and Computer Sciences, UC Berkeley *Department of Mechanical Engineering, UC Berkeley Equal Contribution
Recent advances the reinforcement learning (RL) literature have enabled roboticists automatically train complex policies in simulated environ- ments. However, due to the poor sample complexity of these methods, solving RL problems using real-world data remains challenging problem. This paper intro- duces a novel cost-shaping method which aims to reduce the number of samples needed learn stabilizing controller. The method adds a term involving a Con- trol Lyapunov Function (CLF) an 'energy-like function from the model-based control literature to typical cost formulations. Theoretical results demonstrate the new costs lead to stabilizing controllers when smaller discount factors are used, which well-known to reduce sample complexity. Moreover, the addition of the CLF term 'robustifies' the search for stabilizing controller by ensuring that even highly sub-optimal polices will stabilize the system. We demonstrate our approach with two hardware examples where we learn stabilizing controllers for cartpole and an Al quadruped with only seconds and a few minutes of fine- tuning data, respectively. Furthermore, simulation benchmark studies show that obtaining stabilizing policies by optimizing our proposed costs requires orders magnitude less data compared to standard cost designs.
Abstract:
Figure I: We leam precise stabilizing policies on hardware for the Quanser cartpole [1] (top) and the Unitree A1 quadruped [2] (bottom) using only seconds and few minutes of real-world data, respectively. A video of experiments can be found here tttpps://yuute be/17kBfitEtnn
'This work supported in part NSF Grants RMMI-1944722 and CMMI-1931852 LOGiCS (Learning- -rrvvin Oracle-Guided Compositional Symbiotic Design of Cyber-Physiaal Systems), and Defense Advanced Research Projects Agency award number FAS755020---0115 The work of Fernando Castañeda was partially supported through fellowship from Fundación Rafacl del Pino, Spain.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

1 Introduction
key challenge in robotics is reasoning about the long-horizon behavior induced by a control policy. This is because important system properties such as stability are inherently long-horizon phenom- ena. In reinforcement learning (RL), the discount factor implicitly controls how far into the future policy optimization algorithms plan when optimizing the objective specified by the user. Standard approaches to designing objective functions for robotic RL, such as penalizing the distance to a ref- erence trajectory, inherently require large discount factor learn control policies which stabilize the system [3, 4]. Unfortunately, problems with large discount factors can be extremely difficult to solve, often requiring vast data sets and careful tuning of apper-parameters [5]. As a number of re- cent success stories have demonstrated [6. 7, 8, 9. 10, 11]. ever-increasing computational resources can be used solve these problems in simulation and deploy the resulting controllers directly on the real-world system. However, because it is impractical to model every detail of complex hardware platforms, achieving the best performance will require learning from real-world data.
This paper introduces cost-shaping framework which enables users to reliably learn stabilizing control policies with small amounts of real-world data by solving problems with small discount fac- tors. Our approach uses Control Lyapunov Functions (CLFs), a standard design tool from the control theory literature [12. 13, 14, 15]. CLFs are 'energy-like" functions for the system which reduce the search for a stabilizing controller to a myopic one-step criterion. In partieular, any controller which decreases the energy of the CLF at each instance of time will stabilize the system. Thus, CLFs reduce the long- horizhooizonn objective of stabilizing the system to a simple one-step condition. When a CLFis is available and the dynamics are known, constructive techniques from the control literature can be used to synthesize a stabilizing controller. However, when there is uncertainty in the dynamics, it is diffieult to guarantee that controller will always decrease the value of the CLF, or that we have even designed a true CLF for the system.
Our approach is to 1) design an approximate CLF for the real-world system using an approximate dynamics model and 2) modify the 'standard' choice of cost functions mentioned above by adding a term which incentivizes controllers which decrease the approximate CLF over time. This technique effectively uses the approximate CLF as supervision for reinforcement learning. enabling the user to embed known system structures into the learning process while retaining the flexibility of RL to overcome unknown dynamics. Indeed, as our analysis demonstrates, when our approach is used reinforcement learning algorithms implicitly learn to 'correct' the approximate CLF provided by the user. When the candidate CLF is close to being a true CLF for the system (in sense we make precise below), a stabilizing controller can be efficiently learned by solving a problem with a small discount factor. Moreover, the addition of the approximate CLF 'robustifies' the search for stabilizing controller by ensuring that even highly suboptimal policies will stabilize the system. Finally, in situations where it is too difficult to design a nominal CLF by hand, we demonstrate how one can be learned using simulation model and the standard style of RL objective discussed above. Specifically, we use the value function learned by the RL. algorithm as an approximate CLF for the real-world system. Altogether, beyond accelerating and robustifying RL, our approach also expands the applicability of CLF--bsed design techniques.
We apply this technique to develop data-efficient fine-tuning strategies, wherein a nominal con- troller developed using a simulation model is refined with small amounts of real-world data. For the A1 experiment, the nominal controller is a model-base control architecture [16], and we hand- design CLF using a highly simplified linearized reduced-orde order model for the system. Even though this model is very crude, we are nonetheless able to learn a precise tracking controller for this 18 DOF system with only 5 minutes of real-world data. For the cartpole swing-up task we used the value function from simulation-based RL problem as the candidate CLF for the real-world system, using the learning process described above. Our fine-tuning approach then learned a robust swing-up up controller after observing only one 10 second trajectory from the real-worke system.
1.1 Related Work
We outline how our approach departs from related work; Appendix A contains further discussion. Discount Factors, Sample Complexity and Reward Shaping: It is well well-understood that the dis- count factor has a significant effect on the size of the data set that RL algorithms need to achieve a desired level of performance. Specifically, it has been shown in numerous contexts [17, 18, 19, 20] that smaller discount factors lead to problems which can be solved more efficiently. This has led to a number of works which explicitly treat the discount factor as a parameter which can be used to control the complexity of the problem alongside reward shaping techniques [21, 22, 5, 23, 24, 25].

---

Compared to these works, our primary contribution is to demonstrate how CLFs can be combined with model-free algorithms to rapidly learn stabilizing controllers for robotic systems.
Fine-tuning with Real World Data: Recently, there has been much interest in using RL to fine- tune policies which have been pre-trate pre-trained in simulation [26, 27, 28, 29]. These methods typically optimize the same cost function with a large discount factor in both simulation and on the real robot. In contrast, using our cost reshaping techniques, we solve different problem with a smaller discount factor on hardware which can be solved more efficiently. In Appendix D, we show that our method outperforms typical fine-tuning approaches under moderate perturbations to the dynamics model. Learning with Control Lyapumov Functions: number of recent works have also tried to over- come the reality gap using data-driven methods to improve CLF-based controllers [30, 31, 32, 33, 34, 35]. While these methods work well when a true CLF for the real-world system is available, our method is more general as we can still efficiently learn stabilizing controllers when only an approximate CLF is available by modulating the discount factor used to optimize our cost.
2 Background and Problem Setting
Throughout the paper we will consider deterministic discrete-time systems of the form:
Fk++1 = F( ux),
(1)
where Ia E C IR" the state time k, Ws time, and F: X' xuI - R" is the transition function for the system. This general nonlinear model is broad enough to cover many important continuous control tasks for robotics. We will let IT denote the space of all control polices T: X + U for the system. To ease exposition, for our theoretical analysis we will focus on the case where the goal is to stabilize the system to a single point, namely the origin. Through our examples we will demonstrate how our cost-shaping technique can be leveraged to achieve more complicated tasks, and in Section we outline a path for extending our theoretical results to these settings in future work.
C X the input applied to the system that
2.1 Control Lyapunov Functions
Control Lyapunov Functions [12, 13, 14, 15] are 'energy-like' functions for the dynamics (I):
Definition 1. We say that a positive definite function W: IR" + R is a Control Lyapunov Function for (1) if the following condition holds for each x E X(00
min W(F(x, u)) W( < 0. weld
The condition (2) ensures that for each E X there exists choice of input which decreases the 'energy' (x)) Any policy which satisfies the one-step condition (F (x))) W(x) < can be guaranteed to asymptotically stabilize the system [36] (see Appendix B for background on stability theory). Given a CLF for the system, model-based methods constructively synthesize a controller which satisfies this property using either closed-form equations [13] or by solving an online (convex) optimization problem [37, 15] to satisfy (2). However, when the dynamics are unknown it is difficult to ensure that we have synthesized 'true' CLF for the system.
Remark 1. (Designing Control Lyapunov Functions) While there is no general procedure for de- signing CL.Fs by hand for general nonlinear systems, there do exist constructive procedures for designing CLFs for many important classes of robotic systems, such as manipulator arms [ [14] and robotic walkers [15] using structural properties of the system, Moreover, in our examples we will investigate how a CLF can be learned from a simulation model and how very coarse CLF candidates can be used to accelerate learning stabilizing controller.
2.2 Stability of Dynamic Programming and Reinforcement Learning
Here we investigate how a common class of cost functions found in the literature can be used to learn stabilizing controllers. In particular, we consider a running cost &: X' x 24 + IR of the form ((x, u) Q(=)= + R(u), where Q: X - RR. is the state cost and R: 21 + IR is the input cost. Both Q and R are assumed to be positive definite (in practice, both are usually quadratic). Given a policy π II, discount factor 7 e [0, 1], and initial condition xo E X, the associated long-run cost is:
Vy
xat ((I,,T(NN))
(3)
K=U
s.t. xb41
(xx)))
3

---

where V X + -UU foo} is the value function associated to л. Small discount factors incentivize policies which greedily optimize a small number of time-steps into the future, while larger discount factors promote policies which reduce the cost in the long-run. We say that a policy m E II is optimal if it achieves the smallest cost from each & E X::
V2()) = -
inf (1)), Va E EX; Tell
where Vi7 x + R U {oo} is the optimal value function. Together Vi and ग. capture the 'ideal' behavior induced by the cost function (3). is well-known [17] that the optimal value function will satisfy the Bellman equation:
(4) and an optimal policy म. will satisfy (x) E arg min [FV; u)) + l(x, ui]. Vx E X. Un- fortunately, it impractical to directly search over 1I to find policy which meets these conditions. This necessitates the use of function approximation schemes (e.g. feed-forward neural networks) to instead represent subset of policies Ii C II to search over. Indeed, modern RL approaches for robotics randomly sample the space of trajectories to optimize problems of the form:
(1()) inf [VV, u)) ((F,u)), Ve weL
inf Emg- No [V7 (zo)]:
(5)
xen
where Xy is a distribution over initial conditions. While this approach enables these methods to optimize high- dimennionn policies, they are data-hungry, can display high- variance and thus fre- quently return highly sub-optimal policies when data is limited. To better understand the effect that this has on the stability of learned policies, for each π E 11 and y E [o, 1] define the optimality gap:
ex( V25 (z).)
The temporal difference equation [17] dictates that for each x E X' the policy satisfies:
VT(x) = V, (1 (())) ((x, (x)).
(6)
From these equations we can obtain:
1 ((z,)(x)) + (1-
VF(((z, V7 (z)
+ 1 ( -( + +(1 (1-))(())))
(8)
1
- - Q(x) + (1 (1-)))())) (z)).
(9)
where we have first rearranged (6), then used (x) = Vx) + e (x), and finally we have used f(x; xxx) 2 Qxx). Inequalities of this sort are the building block for proving the stability of suboptimal polices in the dynamic programming literature [4, 3].
Remark 2. (Value Functions CLFFs) By inspecting the cost (3) we see that V is positive definite (since Q is positive definite). Thus, if the right-hand side of (9) is negative for each X \ {0}, this inequality shows that V is a CLF for (1), and that π is an asymptotically stabilizing control policy. In other words, V is a CLF which is implicitly learned during the training process. Indeed, many RL algorithms directly learn an estimate of the value function, a fact which we later exploit to learn a CLF for the cartpole swing up-task in Section 4 using the nominal simulation environment.
Note that the right hand side of (9) will only be negative if (x) + +17(1) 1z)(x) Since from (3) we know that (z) > Q(x) for each x E X, even the optimal policy (which has no optimality gap) will only be stabilizing if y is large enough. On the other hand, for fixed 7 E (0, 1]. this inequality also quantifies how sub-optimal a policy can be while maintaining stability. To make these observations more quantitative we make the following assumption:
Assumption 1. For each y E [0, 1] there exists C, 2 1 such that Vi (z) s CzQ(2) for each x E X.
Growth conditions of this form are standard in the literature on the stability of approximate dynamic programming [38, 3, 4, 39]. Note that, because the running cost is non -eg-aivae,e,, we have C2' < C.,, rf < ~" In particular, the constant C1 upper-bounds the ratio between the one-step cost and the optimal undiscounted value function. When C₁ is smaller, the optimal undiscounted policy is more 'contractive" and approximate dynamic programming methods converge more rapidly to an optimal solution [38]. Thus, intuitively the constants C, 2 will be smaller when the system is easier to stabilize. The following result is essentially a specialization of the main result from [39];

---

Proposition 1.
Let Assumption I hold and let y E [0, 1] and ग II be fixed. Further assume that
there exists 8 > such that for each x E X we have i) (ç(x) I (2(z) and ii) Cy + 8 T- Then, x asymptotically stabilizes (1).
Proof. Combining conditions i) and ii) with equation (9) yields:
V (F (x))) - ( (z) 2 (-1+ (1 (1-))
Thus the RHS of the preceding equation will be negative-defiitt if C, + 8 < i- which demon- strates the desired result.
Remark 3. (Stability Properties of the Cost Function) In the following section we will derive an analogous result Proposition I for the novel reshaped cost function we propose below. When comparing these results we will primarily focus on the effect of the constants C- 2 1 (and the equivalent constants for the new setting). The C2 constants can be used to bound how large of discount factor is need to stabilize the system. In particular, Proposition 1 implies that the optimal policy will stabilize the system for each ? which satisfies & > 1 C. The Gr constants also characterizes how 'robust' the cost function is to suboptimal policies. In particular, for a fixed discount factor, the policy will stabilize the system if T Cy. Thus smaller values of the City constants permit more suboptimal policies.
3 Lyapunov Design for Infinite Horizon Reinforcement Learning
Our method uses a positive definite candidate Control Lyapunov Function W; R² - I for the
nonlinear dynamics (1), and reshapes (3) to our proposed new long horizon cost V75 : X + Rufoo):
(x0)
(F (xxx)) - ((x)) + E(3r, ((=)))(())))
(10)
k=0
s.t. Fk41 = F(xx,
As we shall see below, our method works best when W is in fact a CLF for the system, but still provides benefits when it is only an "approximate" CLF for the system (in a sense we will make precise later). For each 7 E [0, 1] the new optimal value function is given by:
(x)
= inf HEll
(x).
(II)
The new cost (10) includes the amount that W changes at each time step, and thus encourages choices of inputs which decrease W over time. In this case, the Bellman equation [17] dictates:
inf [>V; (F(x, u)) (W u) + (xx, w)], Var E X, V wEll
(12)
where AW (W u) == ((F(x, W (z). To gain some intuition for the approach let us consider the two extremes where 7 and ? that 1/= = V W solves the Bellman equation. Plugging in this solution demonstrates that any optimal policy मं must satisfy (z) E arg minueuml (F((z, u)) + l(z, u)). This is precisely the optimality condition for the original cost (3) when 7 1, and thus the set of optimal policies for the two problems coincide. Thus, in this case, by embedding the CLF in the cost we are effectively using W as a warm-start initial guess for the optimal value function. In the other extreme where 7 0, from (12) we see that an optimal policy must satisfy (z) E arg min (A. u) + {xx u) Thus, when y the optimal policy attempts to greedily decrease the value of the candidate CLF and the one-step cost on the input. As we shall see below, when intermediate discount factors are used, optimal policies may instead decrease the value W over the course of several steps.
In the case where 7 1, by inspection we see
Using the new cost function (10), each policy must satisfy the new difference equation:
(x) V (F(z, (())) + W (F(z, (())) - W(x) + ((x, (()).
(13)
In our stability analysis, we will use the following composite function as a candidate CLF for (1):
1 (1)))) = ((x) + (x).
(14)
5

---

We provide an interpretation of this curious candidate CLF in Remark 4 below, but first perform an initial analysis similar to the one presented in the previous section. Defining for each π E II, y E [0, 1] and x E X the new optimality gap: (x) = Vi(()) (x),
(15)
and following steps analogous to those taken in (7)-19), we can obtain the following:
(F((,, - - -(z, T (xx) + (1 (1---700 (=)
(16)
- [1x (xx) (1->) (1->) + (5)
(17)
S -Q((x) + (1->)
(18)
Similar to the analysis in the previous section, we will aim to understand when the right-hand side of (18) is negative, as this will characterize when æ stabilizes the system. One key difference between the inequalities (9) and (18) is that, while the original value function Vi* is necessarily positive definite, Vi can actually take on negative values since the addition of the CLF term allows the new running cost in (10) to be negative. As we shall see, this forms the basis for the stability and robustness properties our cost formulation enjoys when W is designed properly.
Remark 4. (Learning Corrections to W) When the right hand side of (18) is negative for each x E X {0}, inequality (18) demonstrates that V is in fact a CLF for (1) and that a stabilizes the system (see Theorem 1). We can think of W as an 'initial guess' for a CLF for the system, while 7 V is a 'correction' to W that is implicitly made by learned policy Roughly speaking, the larger the discount factor, the larger this correction. Thus, the user can trade-off how much the learned policy is able to correct the candidate CLF W against the additional complexity of solving a problem with a higher discount factor, depending on how 'good' they believe the CLF candidate to be.
We first state a general stability result for suboptimal policies associated to the new cost, and then discuss how the choice of W affects the stability of suboptimal control policies:
Assumption 2. For each E [0, I] there exists C.2 ER such that VG (z) S Cz,(2)) for for x E X.
Because the reshaped one-step cost W (F(,,, u)) (xx) + f(ar, u) can take on negative values, so can the C₂ constants. Moreover, in this case it is possibe to have C,¹ 2 Cipv when / "-""" This is because when larger discount factors are used, the optimal policy can benefit from decreasing W further into the future. The following stability result is analogous to Proposition 1:
Theorem 1. Let Assumption 2 hold and let E [0. 1] and π E ίî be fixed. Further assume that there exists 8= > O such that for each x E X we have i) E (1) < 6Q(x) and ii) C2 + 3 < T-y Then, π asymptotically stabilizes (I).
The proof is conceptually similar to the proof of Proposition 1; we delegate the proof to Ap- pendix C for brevity. Indeed, note that the conditions for stability under the new cost are essentially identical to those for the previous cost in Proposition 1.
As alluded to in Remark 3, we will primarily focus comparing how large the constants C.yy and Č. E R are for the two problems, as they control the discount factor required to learn stabiliz- ing policy and also the 'robustness' of the cost to suboptimal controllers. We provide two character- izations which ensure that C₂ < C.,, The first condition is taken from the model-predicte predictive control literature [40, 41], where CLFs are used as terminal costs for finite-horizon prediction problems. Proof of the following result can be found in Appendix C:
Lemma 1. Suppose that for each X the following condition holds: inf ((F(z, u)) W(x) + f(x, u) < O. WEU
(19)
Then Assumption 2 is satisfied with constant Č, < O.
The hypothesis of Lemma implies that i) W is true CLF for the system and ii) W dominates the running cost E, in the sense that W can be decreased more rapidly than accumulates. Effec- tively, this condition implies that is advantageous for polices to myopically decrease W each time step. Consequently, when this condition holds optimal polcies associated to the reshaped costs (10) will stabilize the system for any choice of discount factor.
The following definition generalizes this condition to cases where W may not be a true CLF for the system but can be decreased over several time-steps:

---

forward velocity
0.35
Dur ganmoo0.5 SAC SuGetin, gamena-022
forward velocity [m/s]
0.25
0.20
0.15
0.10
learned policy nominal policy
0.00
desired velocity
2.5
5.0
7.5
10.0
12.5 time [s]
17.5
20.0
number trainin (x10000)
Figure 2: (Left) Plot illustrating improved velocity tracking the learned policy (in dark green) compared to the nominal locomotion controller (in pink) to track desired velocity profie (in dashed black line) using our proposed method on the Unitree A1 robot hardware. (Right) Plot from the simulated benchmark study illus- trating cumulative velocity tracking emorr (lower is better) over 10s rollouts different stages of the training. orange. we show the results of tine-tuning using SAC with a standard RL cost. In blue, we fine-tune using SAC with our reward reshaping method, with a candidate CLF designed on a nominal lincarized model of the robot. In both cases, we plot the results using the discount factor that achieved the best performance.
Definition 2. We say that the candidate CLF W 7-dominates the running cost if for each discount factor 7 S 7 S 1 and x E X we have (x) S (x).
The condition in (2) effectively provides way of characterizing how 'close' W is to being a true CLF for the real-world system. In particular, the larger the further into the future RL algorithms must look to see the benefits decreasing W. Our previous discussion, which showed that V/- V W, demonstrates that every candidate CLF 1-dominates the cost. Moreover, clearly W can only O-dominate the original cost if it is CLF for the system. While this condition is more difficult verify for intermediate values of 7, provides qualitative insight into how even approximate CLFs for the system can still make it easier to obtain stabilizing controllers.
Remark 5. (Robustness of reshaped cost) When the condition of Lemma is satisfied we will have Cr S 0 C.,, implying the new enjoys the desirable robustness properties discussed above. When W satisfies the "approximate CLF condition in Definition (2), it will only enjoy these benefits when the discount factor is large enough. We leave it matter for future work to provide quan- ritative estimates for the Ct constants in these regimes, and to provide sufficient conditions which ensure W d-domiatses the running cost.
Examples and Practical Implementations
We summarize the main results for each of our examples, but leave most details and plots to Ap- pendix D. In every experiment we report, the soft actor-critic algorithm (SAC) [42] is used as the leamnigg algorithm to optimize the various reward structures we investigate.
We apply our approach to train a neural network controller which augments and improves nominal model-based controller [16] for quadruped robot using real-world data. As illustrated by the pink curve in Fig. 2 (left), the nominal controller fails to accurately track desired velocities specified by the user. We design a CLFF around the desired gait using a linearized reduced-order model for the system. We then collect rollouts of 10s on the robot hardware with randomly chosen desired velocity profiles, and solve an RL problem using our cost and a discount factor = O. Our approach is able to learn a policy which significantly improves the tracking performance of the nominal controller within 5 minutes (30 episodes) of hardware data, as shown in Fig. 2 (left). A video of these results can be found in https: /ttpss//youuu. be/ 17kBfi iE5 and more details are provided in Appendix D. Furthermore, in Fig. 2 (right) we benchmark our approach in simulation against an RL agent trained with a 'standard' cost which penalizes the squared error with respect to the desired velocity. As this figure demonstrates, our method is able to rapidly decrease the average tracking error in only around 2 thousand steps from the environment. In contrast, the benchmark approach only able to reach this level of performance for the first time after around 24 thousand steps.
Velocity Tracking for A1 Quadruped:
7

---

A1 Quadruped Walking with an Unknown Load: We attach an un-modeled load to the Al quadruped, that equivalent to one-third the mass of the robot. Fine-tuning on hardware the same base controller from the previous set-up where the CLF is designed to stabilize to the target gait, our approach is able to significantly decrease the tracking error to about one-third its nominal value with only one minute of data collected on the robot hardware as illustrated in Fig. 3 in Appendix D. Additionally,, in Appendix D. we run a simulated benchmark comparison and verify that our method clearly out-performs the 'standard' cost baseline for this task.
We fine-tune swing-up controller for the Quanser cartpole system [1] using real-world data and an initial policy which was pre-trained in simulation but that does not translate well to the real system. Due to the underactuated nature of the system, synthesizing a CLF by hand is challenging. Thus, as alluded to previously, we use a 'typical' cost function of the form (3) and a discount factor of ? 0.999 to learn a stabilizing neural network policy T for a simulation model of the system. Given the discussion in Remark 2, we use the value function Vo associated with the simulation- based policy as the candidate CLF (W = Vo) for our reward reshaping formulation (10). When improving the simulation- based policy Te with real-world data, we keep the parameters of this network fixed and learn an additional smaller policy Ty (so that the overall control action is produced by л + σΦ) using our proposed CI.F-based cost formulation. We solve the reshaped problem with a discount factor y = and collect rollouts of 10s on hardware. Our CI.F-based fine-tuning approach is able to successfully complete the swing-up task after collecting data from just one rollout. After collecting data from an additional rollout, the controller is reliable and robust enough to recover from several pushes. A video of these experiments can be found in https: : youuuu.uuuuu wtt.//tuuut.tttteeeeeee be/17kBfitE5nd and more details and plots of the results are provided in Appendix D. Furthermore, in Appendix D we provide a simulation study comparing a standard fine-tuming approach to our method, showing that our approach is able to more rapidly leam reliable swing-up policy than the baseline and also achieves higher reward. Fine-tuning a Bipedal Walking Controller in Simulation: We also apply our design methodology to fine-tune a model-based walking controller [15] for a bipedal robot with large amounts of dynam- ies uncertainty. Model uncertainty is introduced by doubling the mass of each link of the robot. The nominal controller fails to stabilize the gait and falls within few steps. To apply our method, we design CLF around the target gait as in [15] to be used in our reward formulation. As benchmark comparison, we also train policies with a reward which penalizes the distance to the target motion (no CLF term), as most commonly done RL approaches for bipedal locomotion which use tar- get gaits in the reward [10]. Our approach is able to significantly reduce the average tracking error per episode after only 40000 steps of the environment (corresponding to 40 seconds of data), while the baseline does not reach similar level of performance even after 1.2 million steps, as illustrated in Fig. 7 of Appendix D. Inverted Pendulum with Input Constraints: Our final example demonstrates the utility of our method even when Win is crude guess for CLF for the system, through the use of moderate discount factors. We illustrate this for a simple inverted pendulum simulator by varying the magnitude of the input constraints for the system. We use the procedure from [15] to design candidate CLF for the system. Like many CLF design techniques, this approach assumes there are no input constraints and encourages the pendulum to swing directly up. As the input constraints are tightened, W becomes a poorer candidate CLF, as there is not enough actuation authority to decrease W at each time step. Even in this case, in line with the discussion of Remark 5, if a proper discount factor is used, the addition of the candidate CLF in the reward enables our method to rapidly learn a stabilizing controller for each setting of the input bound. These results are presented in Appendix D.
Fine-tuning a Learned Policy for Cartpole Swing-Up:
5 Discussion and Limitations
As we have mentioned previously, our approach has several limitations. The cost-shaping tech- nique we introduce in Section 3 only provides benefits when W in-fact a reasonable guess for a CLF for the true system. This requires that the user has a dynamics model which captures the primary features of the environments which affect the structure of CLFs for the system. While the cart-pole simulations we provide in the Appendix D provide some intuition for when this will be the case, further research is needed to better understand in what scenarios we can see significant benefits from our method. Nonetheless, our two hardware experiments provide encouraging initial results which indicate that our method can rapidly learn stabilizing controllers using CLF's which are constructed using a nominal dynamics model. More broadly, there are many exciting avenues for further incorporating Lyapunov design techniques with RL, especially offline learning [43].
8

---

References
[1] Quanser. Linear servo base unit with inverted pendulum, Apr 2021. URL https: htwps:///www //ww.
quanser.com/pron. waua:ausstttttaraeerreaaaa--aa--
[2] U. Robotics. A1. URL https:///ww: http:wwww.it..eee.e.ett......t.. unitree. httreeeecccoppoouuussaaaa/////
[3] R. Postoyan, L. Buşomiu, D. Nešić, and J. Daafouz. Stability analysis of discrete-time infinite- horizon optimal control with discounted cost. IEEE Transactions on Automatic Control, 62(6): 7736-2749) 2016.
[4] V. Gaitsgory, L. Grüne, and N. Thatcher. Stabilization with discounted optimal control. Sys- tems & Control Letters, 82:91-98, 2015.
[5] V. François-Lavet, Fonteneau, and D. Ernst. How to discount deep reinforcement learning: Towards new dynamic strategies. arXiv preprint urXiv:1512.02011 2015.
[6] Lee, Hwangbo, I.. Wellhausen, Koltun, and Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47), 2020.
[7] A. Kumar, Fu, D. Pathak, and Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.0400: 2021.
[8] X. B. Peng, E. Coumans, Zhang, T.-W. Lee, Tan, and S. Levine. Learning agile robotic locomotion skills by imitating animals. arXiv preprint arXi::2004.00784 2020.
[9] X. B. Peng, M. Andrychowicz, W. Zaremba, and Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. In 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810. IEEE, 2018.
[10] Z. Li, X. Cheng, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforce- ment learning for robust parameterized locomotion control of bipedal robots. arXiv preprint arXiv:2103. 14295, 2021.
[11] S. Belkhale, Li, G. Kahn, R. McAllister. R. Calandra, and Levine. Model-based meta- reinforcement learning for flight with suspended payloads. IEEE Robotics and Automation Letters, 6(2):1471-1478, 2021.
[12] Z. Artstein. Stabilization with relaxed controls. Nonlinear Analysis: Theory, Methods & Applications, 7(11):1163-113, 1983.
[13] E. D. Sontag. A 'universal' construction of artstein's 's theorem on nonlinear stabilization. Sys- tems and Control Letters, 13(2):117 123, 1989.
[14] A. D. Ames and M. Powell. Towards the unification of locomotion and manipulation through control Iyapunov functions and quadratic programs. Lecture Notes in Control and Information Sciences, 149:219-240, 2013.
[15] A. D. Ames, K. Galloway, K. Sreenath, and J. W. Grizzle. Rapidly exponentially stabiliz- ing control Yyаpunov functions and hybrid zero dynamics. IEEE Transactions on Automatic Control, 59(4):876-88) 891, 2014.
[16] X. Da. Z. Xie, D. Hoeller, B. Boots, A. Anandkumar, Y. Zhu, B. Babich, and A. Garg. Learning contact-adappive controller for robust, efficient legged locomotion. In Conference Robot Learning, pages 883-894. PMLR, 2021.
[17] D. P. Bertsekas and N. Tsitsiklis. Neuro-dynamic programming. Athena Scientific, 1996.
[18] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 170.:06770.06477 2017.
[19] R. Munos and C. Szepesvári. Finite-time bounds for fitted value iteration. Journal of Machine Learning Research, 9(5), 2008.

---

[20] R. Postoyan, M. Granzotto, L. Buşoniu, B. Scherrer, D. Nešsić, and J. Daafouz. Stability guarantees for nonlinear discrete-time systems controlled by approximate value iteration. In 2019 IEEF 58th Conference on Decision and Control (CDC), pages 487-492 IEFF, 2019.
[21] N. Jiang, A. Kulesza, S. Singh, and R. Lewis. The dependence of effective planning horizon on model accuracy. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, pages 1 1188-1189 Citeseer, 2015.
[22] M. Petrik and B. Scherrer. Biasing approximate dynamic programming with a lower discount factor, Advances in neural information processing systems, 21:1265- 2008.
[23] C. Tessler and S. Mannor. Reward tweaking: Maximizing the total reward while planning for short horizons. arXiv preprint arXiv:2002.03337, 2020.
[24] C.-A. Cheng, A. Kolobov, and A. Swaminathan.
Heuristic-guided reinforcement learning. Advances in Neural Information Processing Systems, 111133550-355653 2021.
[25] A. Y. Ng, D. Harada, and S. Russell. Policy invariance under reward transformations: Theory and application to reward shaping. International conference on machine learning, vol- ume 99, pages 278 287, 1999.
[26] L. Smith, J. C. Kew, X. B. Peng. S. Ha, J. Tan, and S. Levine. Legged robots that keep on learning: Fine-tuning locomotion policies in the real world. arXiv preprint arXiv:2110.05457, 2021.
[27] R. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman. Never stop learning: The effectiveness of fine-tuning robotic reinforcement learning. arXiv preprint 20Xiv:200..01190, 2020.
[28] R. Julian, B. Swanson, G. S. Sukhatme, S. Levine, C. Finn, and K. Hausman. Efficient adap- tation for end-to-end vision-based robotic manipulation. 2020.
[29] Z. Mandi, P. Abbeel, and S. James. On the effectiveness of fine-tuning versus meta- reinforcement learning. arXiv preprint arXiv:2206.03277 2022.
[30] A. J. Taylor, V. D. Dorobantu, H. M. Le, Y. Yue, and A. D. Ames. Episodic learning with control Iyapunov functions for uncertain robotic systems. In 2019 IEEE/RS.J International Conference on Intelligent Robots and Systems (IROS), pages 1878-6884, IEEE, 2019.
[31] A. J. Taylor, V. D. Dorobantu, M. Krishnamoorthy, H. M. Le, Y. Yue, and A. D. Ames. A control lyapunov perspective on episodic learning via projection to state stability. In 2019 IEEE 58th Conference on Decision and Control (CDC), pages 1448-1455. IEEE, 2019.
[32] T. Westenbroek, F. Castañeda, A. Agrawal, S. S. Sastry, and K. Sreenath. Learning min-norm stabilizing control laws for systems with unknown dynamics. In 2020 59th IEEE Conference on Decision and Control (CDC), pages 737-744. IEEE, 2020.
[33] T. Westenbroek, A. Agrawal, F. Castañeda, S. S. Sastry, and K. Sreenath. Combining model- based design and model-free policy optimization learn safe, stabilizing controllers. IFAC- PnpersOnI.ine, 54(5):19-22 2021.
[34] F. Castañeda, J. J. Choi, B. Zhang, C. J. Tomlin, and K. Sreenath. Gaussian process- based min-norm stabilizing controller for control-affine systems with uncertain input effects and dy- namics. In 202/ American Control Conference (ACC), pages 3683-3690, 2021.
[35] J. Choi, F. Castañeda, C. Tomlin, and K. Sreenath. Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions. In Robotics: Science and Systems, Corvalis, OR, July 2020.
[36] C. M. Kellett and A. R. Teel. Results on discrete-time control-lyapunov functions. 42nd IEEE International Conference on Decision and Control (IEEE Cat. No. 03CH37475), vol- ume 6, pages 5961-5966. IEEE, 2003.
[37] R. Freeman and P. V. Kokotovic. Robust nonlinear control design: state-space and Lyapunov techniques. Springer Science and Business Media, 2008
10

---

[38] B. Lincoln and A. Rantzer. Relaxing dynamic programming. IEEE Transactions on Automatic Control, 1248):1249-1228 1260, 2006.
[39] V. Gaitsgory, L. Grüne, C. M. Kellett, and S. R. Weller. Stabilization with discounted optimal control: the discrete time case. 2016.
[40] A. Jadbabaie, Yu, and J. Hauser. Receding horizon control of the caltech ducted fan: A con- trol lyapunov function approach. In Proceedings of the 1999 IEEE International Conference on Control Applications (Cat. No. 99CH36328), volume 1, pages 51-56. IEEE, 1999.
[41] G. Grimm, M. J. Messina, S. Tuna, and A. R. Teel. Model predictive control: for want of a local control Iyapunov function, all is not lost. IEEE Transactions on Automatic Control, 50 (5):446-558, 2005.
[42] Haarnoja, Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. CoRR, 0tt//1801.019900 2018.
[43] S. Levine, A. Kumar, G. Tucker, and Fu. Offline reinforcement learning: Tutorial, review and perspectives on open problems. arXiv preprint arXiiv:0005.01643, 2020.
[44] A. Jadbabaie and J. Hauser. On the stability of receding horizon control with a general terminal cost. IEEE Transactions Automatic Control, 5065)6674667,, 2005.
[45] A. Jadbabaie, J. Yu, and J. Hauser. Unconstrained receding-horizon control of nonlinear sys- tems. IEEE Transactions on Automatic Control, 46(5):776-783, 2001.
[46] G. Grimm, M. J. Messina, S. E. Tuna, and A. R. Teel. Examples when nonlinear model predictive control is nonrobust. Automatica, 40(10):1729-1738, 2004.
[47] Sastry. Nonlinear systems: analysis, stability, and control, volume 10. Springer Science & Business Media, 1999.
[48] S. Gu, E. Holly, T. Lillicrap. and S. Levine. Deep reinforeement learning for robotic manipula- tion with asynchronous off-policy updates. In 20/7 IEEE international conference on robotics and automation (ICRA), pages 3389-3396. IEEE, 2017.
[49] C. Chevallereau, G. Abba, Plestan, E. Westervelt, C. C. de Wit, Grizzle, et al. Rabbit: A testbed for advanced control theory. IEEE Control Systems Magazine, 23(5):57-79, 2003.
[50] E. Coumans and Y. Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. http://pybulle org, 2016-2019.
[51] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning to walk via deep 2018.
reinforcement learning. arXiv preprint


Reviews:
[{"review_id": "YFF2BInsA46", "reviewer": "Reviewer_ACEu", "summary": "The authors address the problem of efficiently learning and fine-tuning stabilising controllers on hardware by introducing a novel cost-shaping method based on Control Lyapunov Functions (CLF). CLFs provide some guarantees about the stability of controllers and an objective that can more greedily be optimised. The latter is useful for data efficiency, as it allows to learn with a smaller discount. The authors provide some bounds on the range of stable discounts given the sub-optimality of a policy. Experiments include fine-tuning a cartpole controller on hardware in seconds, a velocity-tracking controller on a quadruped in minutes, and an inverted pendulum in simulation, the latter for which shows the flexibility and robustness of the approach to approximate CLFs.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- Data efficiency in fine-tuning is a critical topic for robot learning research\n- Mixing in ideas from control theory comes with convenient benefits, such as theoretical guarantees and bounds that can help tune parameters (e.g. discount)\n- Hardware experiments on both a cartpole and a quadruped\n- Flexibility and robustness to choice of CLF, incl. using only a locally-valid approximation and a value function trained purely in sim\n- The latter especially may make this approach fairly easily widely applicable\n- Paper reads well and provides sufficient theoretical backing\n\nWeaknesses:\n- Quite strong assumptions about the nature of the control problem to be solved (stabilising, deterministic, fully-observed, specific shape of cost), though this is pretty common in more theoretically-based work\n- The fine-tuning results for the hardware experiments should be included in the main paper", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall I'm leaning towards accepting this submission. It provides a cleanly-presented, theoretically-founded approach for improving data efficiency during fine-tuning, and the results seem to prove the effectiveness. I'm reluctant to make it a strong accept due to the underlying assumptions of the approach and how well those will hold for modern robotics problems, e.g. learning to manipulate from pixels.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- I would recommend to include the hardware fine-tuning baseline in the main paper\n- There are \"Remark\", \"Assumption\", \"Proposition\", \"Definition\", \"Theorem\" and \"Lemma\" counters, perhaps these can be omitted to improve readability?\n- Minor typos:\n  - l120: R is the input [cost]\n  - l131: of the from -> of the form\n  - l140: the inequality needs to be flipped?\n  - l176: + C -> - C\n  - l204: missing \\gamma"}}, {"review_id": "WmHyEf4qLT", "reviewer": "Reviewer_XRvu", "summary": "This paper investigates sample-efficient fine-tuning of controllers by adding an approximate control Lyapunov function to the conventional cost employed for policy learning. This addition enables the usage of smaller discount factors during adaptation. The approach is demonstrated for sim2real transfer on a pendulum swing-up task, as well as reducing tracking error of a model-based controller on a quadrupedal robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\nThe paper focuses on an important topic as training policies for robotic hardware (from scratch) is generally expensive, and being able to efficiently fine-tune a given controller to the specific hardware setup can be crucial. Overall the paper is well-written and formatted, clearly states its assumptions and does not overstate its contribution. The authors provide a diverse set of experiments including fine-tuning with a learned policy/value function as well as adapting based on a model-based controller. The hardware demonstrations further underline real-world applicability.\n\n**Weaknesses:**\n\nThe paper would profit from elaborating on failure modes and limitations. For example, it would be helpful to discuss the assumption of positive definite cost matrices in the context of general RL problems. It would furthermore be interesting to ablate on the quality of the learned value function (pendulum) to see how well-suited the value function is at different stages of SAC training (or even alternative algorithms). The results in Figure 3 show that the velocity tracking error has been reduced, in fact it now leads the target instead of lagging (elaborate on why?). The resulting controller now appears more shaky than before, which may be due to visualization. It would have been interesting to see a video of the resulting gait to better assess qualitative performance, but a discussion of these phenomena would be good. There are also several instances where the reader is referred to prior work (e.g. construction of CLF), while inclusion in the appendix would make the paper more self-contained.\n\n**Minor:**\n- Line 55: hand-desired --> hand-designed\n- Figure 2: increasing legend and axis label size for better visibility\n- Figure 4: could be helpful to provide dotted red lines to highlight x & y intercepts of black dots;\n- Figure 4: floating line above right panel", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper aims to address the important topic of sample-efficient fine-tuning particularly in light of hardware transfer, which is valuable for the robotics community. The exposition is generally well-written, while it would benefit from some expansion of the limitations section (see above). Overall, the paper illustrates the effectiveness of the proposed formulation sufficiently well in the context of the experiments, although additional experimental evaluation would help strengthen the paper. \n\n------- Post rebuttal -------\n\nThe authors have addressed the majority of my comments justifying an increase to weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Expand discussion of assumptions and limitation\n- Details on value function quality for the pendulum SAC experiment, how is down-stream performance affected over different training snapshots of the first stage\n- Expand discussion of quadruped tracking"}}, {"review_id": "SDCs7wS-lL8", "reviewer": "Reviewer_9gjg", "summary": "This  paper proposed a novel way of utilizing Lyapunov functions as the reward function to accelerate the RL process with small discount parameter. Both theorical analysis and real hardware experiment were provided.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1. The toptic is very interesting\n2. The theorical analysis could support the proposed method\n\nWeaknesses\nThe main concern of the reviewer is about the experimental results. At the begining of the paper, the experiment looks very cool wicth includes both pendulum and quadruped. On the other hand, the  actual expriment design is not that attractive:\n\nThe pendulum task is relative easy for SAC. In this paper, the value function of SAC learned in simulation was selected as the CLF, one part of the reward function to help the policy based RL to quickly learn the real task with zero discount parameter. The policy is based on the SAC neural network which has learned relatively good feature to this task; The reward is also based on the value function which is for modeling the long-term reward with the previous discount parameter. The quadruped experiment is over simplified, especially when the CLF is already given. \n\nOverall,  It is difficult to judge the practical value of CLF without addressing how hard the current tasks are for existing RL approach.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Weak rejection for the limition in the current experiment.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "issues": "Please address my comments above.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "KBl6Db1YM_0", "reviewer": "Reviewer_eFRG", "summary": "The paper considers the problem of the poor sample complexity of RL. To address the problem, the paper introduces a cost-shaping method, which adds a control Lyapunov function (CLF) to the cost formulation. The paper theoretically proved that the proposed cost formulation can lead to a stable controller when small discount factors are used, and therefore reduces the sample complexity. The proposed method is demonstrated on a cartpole and an A1 quadruped. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nThe problem that the paper considers is important. The proposed method is technically sound, and the experiments demonstrate that the proposed method can indeed stablize the system with higher sample efficiency in real-world environments. The writing of the paper is good.\n\nHowever, the most important weakness of this paper is that adding a CLF term in the cost function of RL is not a novel idea. Several previous works [1,2] use the same idea to address the stability issues in RL. Moreover, [2] does not need a given CLF approximation, but learns the CLF with the RL process directly. Therefore, the authors may want to discuss their contributions more clearly compared with [1] and [2]. Other weaknesses include the unclear problem statement, the missing comparisons with baselines, the missing ablation studies, and the missing details of the experiments.\n\n[1] Minghao Han, Lixian Zhang, Jun Wang, and Wei Pan. Actor-critic reinforcement learning for control with stability guarantee. IEEE Robotics and Automation Letters, 5(4), pages 6217-6224, 2020.\n\n[2] Ya-Chien Chang and Sicun Gao. Stabilizing neural control using self-learned almost lyapunov critics. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 1803–1809. IEEE, 2021.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Although the paper considers an important problem, the conparison with the previous work is not enough. Two aforementioned papers seem to solve the same problem under milder assumptions. In addition, there are issues with the problem statement, and the experimental results cannot fully support the claims that the paper makes. Therefore, I give this paper a weak reject.\n\n-----------Post discussion Note-----------\nI would like to thank the authors for providing additional explanations, experiments, and the revision. The additional comparison with the two works I brought up has addressed my concerns about the novelty of the work. I agree that this work can be complementary to the two works I mentioned. In addition, I found that the additional explanations and details of the experiments are clear and can address my concerns. It could be better if the authors provide more details about how to choose the best discount factors in practice and how the performance of the proposed algorithm changes with different discount factors. I would like to change my score to a weak accept.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. The comparsion of the proposed method and the aforementioned two works is missing. Could you state the new contribution of the paper and the key difference of the reward design technique? Could you compare the proposed work with the aforementioned two works in experiments if the problem settings are the same?\n\n2. The problem statement is not very clear to me. Do you want to maximize the cummulative reward (standard RL setting), or just to make the learned controller stable? In the former case, will the added term in the value function make the cummulative reward higher? This is not supported by the experimental results. In the latter case, what's the function of the reward function if you have a CLF approximation?\n\n3. One of the advantanges of the proposed algorithm is the increased sample efficiency. However, this advantage is only proved by a simple inverted pendulum system. For the carpole and the Unitree A1 robot, the paper only provides the conparsion of the fine-tuned policy and the nominal policy, which cannot support the authors' claim about the sample efficiency. In addition, the conparsion of the fine-tuned policy and the nominal policy seems a bit trivial since the fine-tuned policy is supposed to be better. What if we only fine-tune the policy using the original reward function in the real world?\n\n4. In Figure 4, the control limits, the discount factor, and the existance of the CLF vary simultaneously. This is not sufficient to show the efficacy of the proposed method as we cannot observe which specific variable controls the change of the performance. The authors may want to provide more ablation studies to settle this issue.\n\n5. The ablation of the approximation of the CLF is not provided. Will the quality of the approximation affect the learned controller a lot?\n\n6. Could you provide the cummulative reward of the trajectories w.r.t. the training epochs? Since RL is proposed to maximize the cummulative reward, can the proposed method do better? \n\n7. Can you provide some inituitions behind the assumptions propsed in the paper?\n"}}]

Author rebuttal:

Decision: Accept (poster)