Robot Reinforcement Learning on the Constraint Manifold
Authors: Puze Liu
Davide Tateo
Haitham Bou Ammar
Jan Peters
Venue: CoRL2021 Oral (2021)
Keywords: Robot Learning
Reinforcement Learning
Constrained Markov Decision Process
Safe Exploration

Abstract: Reinforcement learning in robotics is extremely challenging due to many practical issues, including safety, mechanical constraints, and wear and tear. Typically, these issues are not considered in the machine learning literature. One crucial problem in applying reinforcement learning in the real world is Safe Exploration, which requires physical and safety constraints satisfaction throughout the learning process.  To explore in such a safety-critical environment, leveraging known information such as robot models and constraints is beneficial to provide more robust safety guarantees. Exploiting this knowledge, we propose a novel method to learn robotics tasks in simulation efficiently while satisfying the constraints during the learning process.

Robot Reinforcement Learning on the Constraint Manifold
Puze Liu', Davide Tateo', Haitham Bou-Ammar' and Jan Peters'
Department of Computer Science, Technische Universität Darmstadt, Germany Huawei R&D London, United Kingdom
(puze, davide @roido)@robot-eee drooeerreaoottttaeeee de.
haitham. awmar@huwwei......mmmmmmm. com, j jan. jaw..a...eettttaaaraatttaaaoaaea peters@tu-darmstatt.co
Abstract: Reinforcement learning in robotics is extremely challenging due many practical issues, including safety, mechanical constraints, and wear and tear. Typically, these issues not considered in the machine learning literature. One crucial problem in applying reinforcement learning in the real world Safe Ex- ploration, which requires physical and safety constraints satisfaction throughout the learning process. To explore in such a safety-critical environment, leveraging known information such as robot models and constraints is beneficial provide more robust safety guarantees. Exploiting this knowledge, we propose a novel method to learn robotics tasks in simulation efficiently while satisfying the con- straints during the learning process.
Keywords: Robot Learning, Constrained Reinforcement Learning, Safe Explo- ration
Introduction
Despite the notable success of Deep Reinforcement Learn- ing (RL) in solving complex tasks in the discrete world, video games, as well continuous control problems in simulation [1, 2, 3, 4]. applying RL in the real world re- mains challenging task. One important factor that cannot be neglected in real-world applications is the necessity of satisfying constraints. Many practical considerations can be formulated in the form of constraints, such as safety and mechanical viability. For example, in the robot manipula- tion task, the robot should not take actions that damage the environment and can not take actions that exceed its fea- sible range. However, typical RL algorithms, which maxi- mize the cumulative reward by continuous trial and error, do not take into account the satisfaction of constraints during the exploration process. Exploring the environment while meeting the constraints is challenging problem.
Na
To
Me c(q)
Figure 1: Acting on the Tangent Space of the Constraint Manifold. The constraint e(q) 0 differentiable manifold M, embedded in the original state space. We use a set of basis vectors N: to rep- resent the span of tangent space T.. The tangent space velocity'acceleration determined by coordinate based on bases, and the control action is deter- mined based on the tangent space veloc- ity/acceleration, the resulting trajectory is maintained on the constraint manifold.
Safe exploration is an significant field of RL. which requires to comply with the constraints during the whole learning process [5]. There are several safe exploration frameworks in the literature: possible direction is proposed in [6. 7] that relies on prior knowledge (policies, value functions) to initialize the system in safe region and gradually in- crease the area of exploration using new information ob- tained from the environment. Other approaches rely on the definition of safe policy [8, 9], which tries to pull the agent back safe state. However, these choices require excessive work in defining such policy, and safe policies could conflict against each other when multiple constraints are violated. Finally, other meth- ods incorporate model information of constraints with model-free RI. algorithms and do not require
be
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

the definition of a manual policy [10, 11, 12]. In these approaches, the agent tries to find the feasible action using constrained optimization techniques at each time step.
In this paper, we propose a novel method, Acting on the TAngent Space of the Constraint Manifold (ATACOM), which the agent explores in the tangent space of the constraint manifold, shown in Figure 1. The proposed method convert the constrained RL problem to a typical unconstrained RL problem. This method allows us utilize any model-free RL algorithms while maintaining the constraints below the tolerance. Furthermore, ATACOM can handle both equality and inequality constraints. For example, in the task of robot wiping a table, the end-effector should move on the surface of the table (equality constraints) while the joint positions and velocities are within its joint limits (inequality constraints). In addition, for tasks with equality constraints, our method explores the lower-dinee live--dienssonal manifold embedded in the original action space. To test our method, we demonstrate three different tasks, CircleMoving, PlanarAirHockey, and liwaAirHockey(13) with different combinations of equality and inequality constraints. We test five ofte-offthheart model- free RI. algorithms (PPO, TRPO, DDPG, TD3, SAC) in each environment. The result shows that all algorithms can learn the policy efficiently while maintaining the constraints below the tolerance.
The advantage of ATACOM can be summarized follows: (i) can deal both with equality and inequality constraints. All of the constraints at each time step are maintained below the tolerance during the whole learning process. (ii) does not require an initial feasible policy, the agent can learn from scratch. (iii) requires no manual safe backup policy to move the system back into the safe region. (iv) can be applied to any model-free RL algorithm, using both deterministic and stochastic policies. (v) can focus the exploration on the pewer-dimensions
manifold instead of exploring in the original action space for equality constrained problem. (vi) have better learning performance as the inequality constraints restrict to a smaller feasible state-action space. As a downside, our method requires: (i) differentiable constraint functions. (ii) a sufficient accurate
invertible dynamics model of the robot or well-performed tracking controller. Videos and code can be found: https:/ //stts////eeeeeeeeeeeseeeeseeeee google.
Related Work. In the last decades, Constrained Markov Decision Processes (CMDP) [14] has attracted a lot of interest from RL researchers, to solve constrained control problems. Under this framework, several different forms of constraints have been studied. One important form of con- straint is the expected cost below a threshold. Many works maximize the expected return while maintaining the expected cost below a threshold [11, 15, 16, 17, 18, 19, 20, 21J. Different types of constrained optimization techniques are applied in the policy update process. Achiam et al. pro- posed a trust-region method Constrained Policy Optimization (CPO) inspired from Trust Region Policy Optimization (TRPO) [19]. Liu et al. proposed the interior point method for policy optimiza- tion [16]. Another type of approach is to adapt the Lagrangian relaxation method for the constrained RL setting [14, 17, 15, 18]. Lastly, Chow et al. proposed a method to generate the Lyapunov func- tion that guarantees constraints satisfaction [11, 22]. These approaches focus on the constraint of the cumulative cost and require initial fcasible policy. However, this cumulative cost criterion cannot ensure safety for tasks where avoiding catastrophic failures is crucial, c.g., car crashing.
Other approaches focus on the state dependant constraints, which should be fulfilled at every time step. To meet this requirement, safe exploration methods can be employed. Garcia, et al., proposed a method based on risk function and a baseline agent, where the control action is sampled based on the evaluation of the risk [6]. The shielding [8] and backup policy [9] frameworks interfere with
the control action to pull the system back to the safe states. These approaches require a manual defined safe policy. Berkenkamp, et al. [7], Wachi, et al. [23], Koller, et al. [24), and Hewing, et al.[ proposed model-based approaches to ensure the safety. These approaches start from an initial feasible policy and progressively increase the safe region based on the learned dynamics model. Recent methods also try to incorporate the model and the constraint information with the model- free RL algorithms. Dalal, et al., added a safe layer which analytically finds the closest action w.r.t the policy derived one [10]. Cheng, et al., proposed a barrier function method to guarantee safety during the exploration [12]. Finally, other approaches has also address the safety issue from different perspectives, such as the policy composition [26, 27] and reachability- based approach [28, 29, 30].
Our approach considers the second group of constraints. However, different from other comparable methods, ATACOM does not require an initial policy, it can learn from scratch. In addition, our method does not require backup policy either, as the constraint violations are forecasted and cor- rected at each step. Furthermore, our method is not specifically restricted to any learning algorithm.
2

---

2 Learning on the Constraint Manifold
In this section, we discuss ATACOM in detail. We first introduce the mathematical notation used in this paper. Then, to demonstrate the core concept, we start with a simple scenario that the constraint on the subset of the state variable q and the action can be formulated a function of the state velocity (q). Next, considering the continuity of velocity (sampling over the velocity does not ensure the continuity), we convert the original state constraint to viability constraint that incorporates the velocity of the constraint. The action is chosen as a function of the acceleration a = Qq). From robotics point of view, this a can the torque applied to each joint, and A the inverse dynamics model. Then, to cope with the velocity limit, we add the viability condition to the acceleration. Lastly, we discuss some important practical issues of ATACOM, such as the error correction, the langent space convention that determines the null space bases.
Definitions We consider the CMDP with continuous state-action space. A CMDP is a tuple (S, A, P, Y,C, where S is state space, A is an action space, P S x A S + [0, 1lis transition kernel, y discount factor, and C {a : - Rli E I,... k) is a set of immediate state-constraint functions.
Assumption In this paper, decompose the state variable S into the directly controllable state q E Q and uncontrollable state X', i.e., s (q x=]" We assume that the constraints c(g) S O are known and depend purely the controllable state. In addition, we assume that the action can be determined based on the i-th order time derivative of the controllable state, i.e., {1, 2.. For example, we determine the joint torque using an inverse dynamics model send the desired positions/velociinn
Agg ()), ) i e
obtained via integration a tracking controller (e.g.. .. PID controller). The general form of the constrained reinforcement learning problem can be formulated as
Σ Ex(r(on.ai))
max Ba,. 8
s.t. c(q:) S 0.
2.1 State Constraints
The state constraints defined as
f(g) g 0.
(1)
where f : R + constraints, and F Q. We add the slack variables μ E R in inequality constraints to convert the original constraints (1) into equality constraints
RRR..g RD + R are two C2 mappings for F' equality and G inequality
c(q, p) = [f(g] g + + = 0.
The constraint set (2) is a (F + G) dimensional manifold embedded in (Q + G) dimensional space. We calculate the time derivative of (2)
(v(9) diag(u) 7 = u) 9
èlg. G. jo(9)
(3)
with the Jacobians J, RFQ and Jg RGXQ of f(g) and ( respectively. Both Jacobians are combined into the Jacobian Matrix Je(q. E ((F-G)x(Q-G) of the complete constraint set.
We can find the null space matrix N. Null]Je(g,HH ((Q+))((Q-F) via SVD [31] or = 0. Each column of the orthogonal matrix Na(q. μ1) represents a basis vector of the null space of J. p)). These null space bases can also be viewed as the tangent space bases of the constraint manifold as illustrated in Figure 1. We can construct tangent space velocity of the constraint manifold by coordinate as
QR [32] decomposition, such that N
Cr Ar
Ne(g.
(4)
Substituting l ri]T of (3) by lår "T] of (4) we have the constraint velocity
ö(g, p. q, ( = Je(g, N: ()) 0.
(5)
Equation (5) implies that the constraints not change regardless of the choice of a. Based on this concept, the ATACOM method can be summarized as follows: Starting from a feasible point
3

---

Accelgration
á(q)
Tmax
g(g)
Eyrin
Velocity
Amin
Figure 2: Viability Con- straints
Figure 3: Feasible Ac-
Figure 4: Tangent space bases
celeration Region
(q(0), (()) ((g.u) p)]c(q, = -}. we choose the tangent space velocity ( (t)]" No(g(t), and the corresponding action a(t) (ÿr Thus, the constrained RL problem is converted into an unconstrained RL problem. The resulting trajectory q(t) satisfies the constraints c(q(t), p(t) 0.
2.2 Viability Constraints
For physical system, it is often required continuous velocity command. However, directly sam- pling velocities i does not ensure this continuity. simple solution to sample accelerations, apply force to the system or determine the velocity via integration. Furthermore, when considering inequality constraints, it is also desirable that (gq. q) s 0 when g(q) = 0 to avoid overshooting. We convert the original state constraints (1) to viability constraints inspired by the linear viability condition [33]
f(g) + K,f(K,faga. -) K (J g(g) K Kg (g. 4) =9(g) ( S O,
(6) with diagonal matrices K, E PXF K2 R having all positive entries. The matrices Ky and K, determine the maximum velocities of the constraints and 9 w.r.t to the value of the constraints. The viability constraint of the inequality constraint is illustrated in Figure 2. When g(4) < 0, the upper bound of the constraint velocity Ímax > 0 which means that is still possible to get close to the constraint boundary. However, if g(g) > 0, the upper bound of constraint velocity 9max should be smaller than zero to pull the violations back.
Analogous the derivations from equation (2) and (3), we have f(q) K (
c(g. d. 14) g(g) + (g)4 du²
O.
(7)
and
é(q.4. 4.
K2Jr(g)
J (4)
diag(H))
q ii
+
J (q)q K (g
Jo(4)4 + K, (4.0g 4)
0,
(8)
Jolgu))
v(a.4)
where by(g. = - H by(g, = HTH and H, E FxQxQ Hg GGQQ are Hessians of f(g), (Q),, respectively. We can construct the joint acceleration as
q -) u)ab(a ④) + N)lg.u)a
with the pseudo-inverse (q. (4) and the null space matrix Ne(q. (1) of the Jacobian Je 14), re- spectively. The first term in equation (9) is the necessary acceleration that maintains the curvature of the constraints manifold (7) and the second term is the tangent space acceleration of the constraints. When starting from the point (0), (0), (0) e {(q q, μ) c(qq q, μ)) = 0} and sampling over a, the joint acceleration q and the corresponding action satisfy the constraints.
2.3 Viability Acceleration Bound
In robotics well as other mechanical systems, it is important to consider the velocity constraints of the actuator. Also, the acceleration should be bounded properly avoid overshooting, We again use the concept of viability to determine the upper and lower bound of the acceleration
max (min (Gmax, K. (9 Vmax)) )) amin).
a min (max (amin: K.(q (q Vmin)) )) amax)

---

with the minimum and the maximum joint velocity limits Vmin max and the acceleration limits max- K, 0 constant. The feasible acceleration region is illustrated in Figure 3. Analo- gous to the viability constraints, the feasible region of the acceleration is modified depending on the state of joint velocities. This technique effectively prevents overshooting.
2.4 Error Correction and Control Action Selection
For continnuni systems, the state is obtained certain sampling rate and the action is applied for a certain period. This time discretization results in constraint violations at each time step. Therefore, add error correction term. We construct P-controller with a diagonal matrix Ke for the constraints
q
J Koclq, p.).
(10)
Combining (9) with (10), get the joint acceleration applied the system
-- (K q, ()) (1q. N:(g, p)o. (1I)
The first term the RHS is the necessary accelera- tions/velocities to maintain the constraints and the second term on the RHS is the tangent acceleration that be explored freely. Figure 5 illustrates the vector field of еπто correction term and null space term of the circle constraint. The gray curves show the sampled trajectories converging to the con- straint manifold due the error correction.
Constrairt
Null Spare
Samples
Error Correction
Figure 5: Vector Field of Circle Constraint. The constraint ? + o2
arrows is the blue circle. The cyan N. with a The red arrow demonstrate the error correction term K.c The gray lines show 100 trajectories from dif- ferent initial points. trajectories converge to the constraint manifold.
The control action can be determined by a= aq)) at different levels. For example, to calculate the joint torque when the robot is controlled via torque command. We also apply the integration method determine the desired positions//elocities,, then use a sufficient accurate tracking controller (e.g.- PID controller + Feedforward Term) to track the desired trajectory. However, the tracking errors could potentially cause hazardous constraint violations. In this paper, we control the joint torque calculated by perfect dynamic model in simulation to simplify the analysis and to exclude the constraint violations caused by the tracking error of the controller. We present the block diagram of the controlling framework in Appendix A.
the inverse dynamics model
2.5
Null Space Convention
The orthogonal null space matrix N can he determined through SVD or QR decomposition. How- ever, the representation of the null space bases is unique. It difficult preserve the consistency
Algorithm 1: ATACOM
Input: Constraint: f.g. 9. J Jy. by. by Scale parameter: K., K,, Kg. Time step AT',
1 for each episode do
Initial feasible state slack variable uan.
for each time step do
Sample policy action ae T(-(st).
5
Observe the q Q: from 85.
Compute Jr,b Je(qt, (.x), t/4 = tb(qu, )x C& = c(qu, qv, (u))
7
Compute the RCEF of tangent space basis of N
Compute the tangent space acceleration (qu juu]" + -t [K.cu + vh) + N
8
Clip the joint acceleration q: + clip(qa, at a..) Integrate the slack variable px++1 = pa + f4x AT Apply the control action = A (qu) to the environment.
10
Observe the next state 8 and reward from the environment. Provide the transition tuple (#k, (kk, # 7k) to the RL algorithm
5

---

2 + y²
1/4
(1,0)
0.5
(a) CircularMotion
(b) PlanarAirHockey
(c) liwaAirflockey
Figure 6: Experiment Environments
of the null space bases computed by the numerical decomposition method [34, 35J. To solve this issue, propose convention ensure the uniqueness of the null space bases.
Each column the null space matrix N. a unit vector indicating a direction of (q pi]". However, this unit vector could sometimes contribute majorly the part of the slack variable and the entries for the joint accelerations could be very small. As a result, the joint acceleration obtained from [Ommii maaz] can only cover small region of the acceleration. As illustrated in Figure 4, the red arrow is a unit basis vector the tangent space, and the reachable joint acceleration by a universal scaling factor could only cover part of the feasible joint acceleration, the red area shown in Figure 4. To alleviate the previously mentioned issue, we compute the Reduced Column Echlon Form (RCEF) of the null space matrix N RCEF(N.). Given that the RCEF of matrix is unique, we obtain unique bases of the null space. In addition, for RCEF, each row containing a leading 1 has zeros in all its other entries. Generally speaking, there exist N independent joints whose acceleration can be solely determined by a. where N is the dimensions of the null space. Also, we can define the feasible range of a as a, [gr,inn max] Through this convention, the joint acceleration is able to cover the full feasible range. Null space bases and feasible region are shown as the blue vector and the blue area in Figure
3 Experiments and Evaluation
To illustrate the properties of our approach, we demonstrate three different experiments in this sec- tion. We first demonstrate toy task, CireularMotion, shown in Figure 6a. this task, we consider state equality, inequality. and velocity constraints. Secondly, we show robotic environment with only inequality constraints, Planar AirHokkyy shown in Figure 6b. A DoF planar robot playing the hitting task in the air hockey scenario while keeping the end-effector inside the table boundary and the robot's joint positions and velocities within its limits. Finally, we demonstrate, JiwaAirHockey in Figure 6c, 7-DoF KUKA IIWA robot learning the hitting task in the simulator. In addition to the constraints of the 3-dimensional task, we add an equality constraint to ensure that the robot end-effector stays the table surface. More details can be found in the Appendix B and D.
MortularMMoton. In this task, shown in Figure 6a, the red point tries move along unit circle in 2D space while keeping the velocity of each direction below the velocity limits and maintaining the position above a certain height. The objective is reach the target point (green square) located in (1, The control action is the acceleration a = [ =
We compare ATACOM with two other approaches for the task. (i) TerminatedCircularMoor where the episode terminates when the maximum constraint violations up to a threshold. (ii) ErrorCorree- tionCircularMotion. where the error correction term in (10) is added before the action is applied to the environment. We test five model-free RL algorithms (SAC, DDPG, TD3, TRPO, and PPO implemented in Mushroom-RL [36]) for each approach.
Figure shows the learning curve and constraint violations of all test RL algorithms for ATACOM. Every algorithm is able to improve the learning performance and SAC outperforms the others meth- ods, which matches our expectations. Figure 7b and 7c show the maximum constraint function and maximum joint velocity constraints at each time step. It can be shown that the maximum constraint violations during the whole learning process remain small. The velocity limit violations are zero after 30 epochs which means the learned policies try to fully exploit the constraints.

---

Discounted Return
Cyerr
4max
DDPG
PP0
0.000
00002
TD3
TRAO
-0,000
0.04
-0.006
45
PPQ
-0 00.000
PPO
SAC
0.03
SAC
40
-0.010
00.02
TAPO
10
20
30
40
50
10
20
30
40
50
10
20
30
40
50
(a) Discounted Reward
(b) Max Constraint Violations
(c) Max Velocity Limits Violations
Figure 7: ATACOM for the CircularMotion. 7a shows the discounted return at each epochs. 7b and
7c shows the maximum constraint violations and maximum joint velocity limits violations.
Total Return
Qmar
0.5
ATACOM
0.4
ErronCorrection
300
0.4
Terminated
0.3
200
ATACOM
ErronCorrectoon
100
0.2
Terminated
0.2
ATACOM
0.1
EnnrCoreectio
Terminated
-100
0.0
10
10
30
(a) Total Return
(b) Max Constraint Violations
(c) Max Velocity Limits Violations
Figure 8: Comparison between ATACOM, Terminated-, and EmorCorrettonnCiiculsMoioooo
In Figure we compare ATACOM, TerminateCCircullrVooooo. and ErrorCorectionCiiclllatttttto We select the best learning algorithms for each approach (SAC for all cases). Compared to the baselines, our method focus on lower-dimensional exploration space that avoid the constraint violations while the others do not. We can conclude from the Figures that ATACOM has not only lower constraint violations but also better learning performance among the three approaches.
In this experiment, we apply 3-joints planar robot for the air-hockey hitting task, as illustrated in Figure 6b. The end-effector of the robot is kept on the table surface and the objective is to hit the puck the opponent's goal. In this environment, constraints, i.e., the robot end-effector should stay inside the table's region, and the joint positions and velocities should not exceed its limits. The control action the joint torque obtained by the inverse dynamics model. In this experiment, we assume the dynamics model is perfectly known to eliminate the constraint violation due the tracking error or the model mismatch.
only consider inequality
In this task, we compared ATACOM with the SafeLayer method proposed by Dalal et al.,[10] and the Unconstrained air- hir-hockey environments. Since the SafeLayer method at the beginning requires free exploration process to learn the constraint function, we only compare the learning performance and the constraint violations after this process. For the unconstrained environment, the robot is com- pletely free to explore, and the episodes only terminate when the maximum episode step is reached. In this experiment, able implementation of SafeLayer only supports DDPG. Additional experiment of PlanarDefend can be found in Appendix C.2
only compare the best DDPG result after the parameter sweep, as the avail-
The result is shown in Figure 9. We can see that ATACOM have the best learning performance and the minimum constraint violations among the three methods. SafeLayer did not learn the constraint function of joint velocities properly. Furthermore, the learned constraints appear to be too restrictive to learn good policy. Compared the method of Unconstrained approach, although ATACOM has the same dimension as the Unconstrained, ATACOM explores only in the feasible region while the Unconstrained approach explores the whole state-action space. This consideration explains why ATACOM outperforms the baselines terms of learning performances.
7

---

Discounted Return
Ceeer
4max
1.0
ATACOM
ATACOM
100
SafeLayer
0.8
SafeLayer
Unconstraied
Unconstrained
80
0.6
0.4
SafeLayer
Unconstrained
0.2
20
0.0
20
40
60
80
100
20
40
60
80
100
20
60
80
100
(a) Discounted Return
(b) Max Constraint Violations
(c) Max Velocity Limits Violations
Figure 9: Comparison between ATACOM, SafeLayer [10], Unconstrained AirHockey in DDPG.
liwaAirHockey. In the third experiment, we demonstrate the same air-hockey hitting task with KUKA LBR IIWA14 Robot in the Pybullet simula- tor. In this task, add an equality constraint sure the end-effector stays on the table surface. We also add inequality constraints avoid collision of the end-effector and joint limit constraints as men- tion in the PlanarAirHockey task. In addition, we also add inequality constraints to avoid the collision between the 4/6-th link and the table. We enforce the joint velocity limits in the simulation as the real- world's KUKA controller does. We compare the im- pact of different simulation step sizes, The step size refers to the sampling frequency in the real world, At each simulation step, the torque is computed by the previous agent's action until the new control action is received. The error correction term is added at each time step. We choose the simulation step size 0.02s, 0.004s, 0.002s, and 0.001s and keep agent control frequency to be 50Hz.
120
10P
constraint
Reward
10-2
20-
Discounted Return
Canex
Cony
0.001
0.002
0.004
Figure 10: Box Plot of liwaAirHockey with dif- ferent choice of time step size.
Figure 10 demonstrates the discounted return the final epoch, the maximum constraint violation Cmaz, and the average constraint violations Caeg throughout the learning process. For a sufficiently small step size, such as 0.004s, 0.002s, 0.001s, the learning agent is able learn the hitting policy. When the step size is too big, e.g., 0.02s, the сπто correction term dominates the control action and the agent has poor learning performance. From the result in this experiment, we demonstrate that ATACOM is able to solve high dimensional tasks. The simulation result also provides us the guidance for the real-world application: The higher frequency of sampling and error correction, the smaller constraint violations will occur. In addition, we compare ATACOM with Riemannian Motion Policies [37] in Appendix C.4
Conclusion
In this article, we present ATACOM, safe exploration method for Constrained RL based on the knowledge of the model and the mathematical formulations of constraints. ATACOM explores the tangent space of the constraint manifold. This exploration technique allows to utilize any type of model-free RL method while maintaining the constraint violations below small threshold. From the experiments, we have shown that ATACOM only has small constraints violations but also better learning performance w.r.t. the other baselines. These performance gains occur because ATACOM only focuses the safe region (from inequality constraint) and subspace (from equality constraint) of the whole state-action space.
However, method still has some limitations. Our method requires sufficiently accurate model or a good tracking controller. This assumption does not hold in most real-world applications since model errors, disturbances, and sensor noise could potentially cause unexpected constraint viola- tions. To deploy this method in real-world robots, we will focus on the model mismatch problem and may require a backup policy to avoid too stringent constraint design. Furthermore, our current approach only focuses on the constraint with only controllable state c(q) = O, even preliminary results (A (Appendix E) suggests extension into constraints with the uncontrollable state.

---

Acknowledgments
This project was supported by the CSTT fund from Huawei Tech R&D (UK). We acknowledge also the support provided by China Scholarship Council (No. 201908080039).
References
[1] D. Silver et al. Mastering the Game of Go with Deep Neural Networks and Tree Search. Nature, 529(7587):484-48, 2016.
[2] O. Vinyals, I. Babuschkin, W. M. Czarnecki, M. Mathieu, A. Dudzik, J. Chung, D. H. Choi, R. Powell, T. Ewalds, P. Georgiev, al. Grandmaster Level in StarCraft II Using Multi-Agent Reinforcement Learning. Nature, 575(7782)3300-334 2019.
[3] Y. Duan, X. Chen, X. B. Edu, J. Schulman, P. Abbeel, and P. B. Edu. Benchmarking Deep Reinforcement Learning for Continuous Control. In International Conference on Machine Learning (ICML), 2016.
[4] T. P. Lillicrap. J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wier- stra. Continuous Control with Deep Reinforcement Learning. In International Conference on Learning Representations (ICLR), 2016.
[5] J. García and F. Fernández. A Comprehensive Survey on Safe Reinforcement Learning. Jour- nal of Machine Learning Research, 16: 1337-1484,, 2015.
[6] J. Garcia and F. Fernandez. Safe Exploration of State and Action Spaces in Reinforcement Learning. Journal of Artificial Intelligncee Research, 45:515-566 564,, 2012. ISSN 10769757.
[7] F. Berkenkamp, M. Turchetta, A. P. Schoellig, and A. Krause. Safe Model-based Reinforce- ment Learning with Stability Guarantees. In Conference on Neural Information Processing Systems (NIPS), 2017.
[8] M. Alshickh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, and U. Topcu. Safe Reinforce- ment Learning via Shielding. In AAAI Conference Artificial Intelligence (AAAI), 2018.
[9] A. Hans, D. SchneegaB, A. M. Schäfer, and S. Udluft. Safe Exploration for Reinforcement Learning. In ESANN, 2008. ISBN 2930307080.
[10] G. Dalal, Dvijotham, M. Vecerik, Hester, C. Paduraru, and Y. Tassa. Safe Exploration in Continuous Action Spaces. arXiv preprint arXiv:1801.os;: 2018.
[11] Y. Chow, O. Nachum, E. Duenez-Guzman, and M. Ghavamzadeh. A Lyapunov-based Ap- proach to Safe Reinforcement Learning. In Conference Neural Information Processing Systems (NIPS), 2018.
[12] R. Cheng, G. Orosz, R. M. Murray, and J. W. Burdick. End-to-End Safe Reinforcement Learn- ing through Barrier Functions for Safety- Criticall Continuous Control Tasks. In AAAI Confer- ence Artificial Intelligence, pages 3387-3395. AAAI Press, 2019.
[13] Liu, D. Tateo, Bou-Ammar, and Peters. Efficient and Reactive Planning for High Speed Robot Air Hockey. In 2021 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), 2021.
[14] E. Altman. Constrained Markov Decision Processes with Total Cost Criteria: Lagrangian Approach and Dual Linear Program. Mathematical methods of operations research, 48(3): 387-417, 1998.
[15] C. Tessler, D. Mankowitz, and Mannor. Reward Constrained Policy Optimization. In International Conference Learning Representations (ICLR), 2019.
[16] Y. Liu, Ding, and Liu, IPO: Interior-point Policy Optimization under Constraints. In AAAI Conference on Artificial Intelligence, 2020.

---

[17] A. Stooke, J. Achiam, and P. Abbeel. Responsive Safety in Reinforcement Learning by PID Lagrangian Methods. International Conference on Machine Learning (ICML), 2020.
[18] D. Ding, X. Wei, Z. Yang, Z. Wang, and M. R. Jovanovic. Provably Efficient Safe Exploration via Primal-Dual Policy Optimization. In International Conference on Artificial Intelligence and Statistics (AISTATS), volume 130, 2021.
[19] Achiam, D. Held, A. Tamar, and P. Abbeel. Constrained Policy Optimization. Interna- rional Conference on Machine Learning (ICML), 2017.
[20] H. B. Ammar, R. Tutunov, and E. Eaton. Safe policy search for lifelong reinforcement learning with sublinear regret. In International Conference on Machine Learning PMLR, 2015.
[21] A. I. Cowen-Rivers, D. Palenicek, V. Moens. M. Abdullah, A. Sootla, J. Wang, and H. Ammar. SAMBA: Safe Model-Based & Active Reinforcement Learning, 2020.
[22] Y. Chow, O. Nachum, A. Faust, E. Duenez-Guzman, and M. Ghavamzadeh. Lyapunov-based Safe Policy Optimization for Continuous Control. In Reinforcement Learning for Real Life (RLARealLife) Workshop in the 36 th International Conference on Machine Learning, 2019.
[23] A. Wachi, Y. Sui, Y. Yue, and M. Ono. Safe Exploration and Optimization of Constrained MDPs Using Gaussian Processes. In AAAI Conference on Artificial Intelligence (AAAI), 2018.
[24] T. Koller, F. Berkenkamp, M. Turchetta, and Krause. Learning- Model Predictive Control for Safe Exploration. In IEEE Conference on Decision and Control, 2018.
[25] L. Hewing, K. P. Wabersich, M. Menner, and M. N. Zeilinger. Learning- Based Model Pre- dictive Control: Toward Safe Learning in Control. Annual Review of Control, Robotics, and Autonomous Systems, 3(1):269-296, 2020. ISSN 2573-5144.
[26] N. D. Ratliff, J. Issac, D. Kappler. S. Birchfield, and D. Fox. Riemannian motion policies. arXiv preprint 1rXiv:1801.02855 2018
[27] J. Urain, A. Li, P. Liu, C. D'Eramo, and J. Peters. Composable energy policies for reactive motion generation and reinforcement learning. In Robotics: Science and Systems, 2021.
[28] A. K. Akametalu, J. F. Fisac, J. H. Gillula, S. Kaynama, M. N. Zeilinger, and C. J. Tomlin. Reachability-ban safe learning with gaussian processes. In 53rd IEEE Conference on Deci- sion and Control, pages 1424-1431. IEEE, 2014.
[29] Y. S. Shao, C. Chen, S. Kousik, and R. Vasudevan. Reachability trajectory safeguard (rts): A safe and fast reinforcement learning safety layer for continuous control. IEÈE Robotics and Automation Letters, 6(2):3663 -6700 2021.
[30] H. Krasowski, X. Wang, and M. Althoff. Safe reinforcement learning for autonomous lane changing using set-based prediction. In 2020 IEEE 23rd International Conference on Intelli- gent Transportation Systems (ITSC), pages 1-7. IEEF, 2020.
[31] R. P. Singh and P. W. Likins. Singular Value Decomposition for Constrained Dynamical Sys- tems. Journal of Applied Mechanics, Transactions ASME, 52(4):943-948, 1985.
[32] S. S. Kim and M. J. Vanderploeg. QR Decomposition for State Space Representation of Con- strained Mechanical Dynamic Systems. Journal of Mechanisms, Transmissions, and Automa- tion in Design, 108: 183-188, 1986.
[33] M. Faroni, M. Beschi, N. Pedrocchi, and A. Visioli. Viability and Feasibility of Constrained Kinematic Control of Manipulators. Robotics, 7(3), jul 2018. ISSN 22186581.
[34] L. W. Tu. An introduction to manifolds. Springer., 2011.
[35] L. N. Trefethen and D. Bau III. Numerical linear algebra, volume 50. Siam, 1997.
[36] C. D'Eramo, D. Tateo, A. Bonarini, M. Restelli, and J. Peters. Mushroomrl: Simplifying reinforcement learning research. Journal of Machine Learning Research, 22(131):1-5, 2021.
[37] C.-A. Cheng, M. Mukadam, J. Issac, S. Birchfield, D. Fox, B. Boots, and N. Ratifff Rmpflow: computational graph for automatic motion policy generation. In International Workshop on the Algorithuiic Foundations of Robotics, pages 441-4 441-457. Springer, 2018
10


Reviews:
[{"review_id": "_0xtq6_L_Ki", "reviewer": "Reviewer_XePn", "summary": "This paper proposes to perform constrained RL by letting an RL agent choose robot velocities/accelerations directly on the constraint co/tangent manifold. These RL actions are mapped to robot join accelerations through standard inverse dynamics. To enforce inequality constraints, the proposed method introduces slack variables with associated dynamics that evolve according to the RL agent's actions. The method is demonstrated on a toy example and two arm air hockey tasks that require avoiding exceeding joint limits and an end-effector position constraint.\n\nThe authors have addressed all of my concerns and recommendations, and I think the paper is much stronger now. I already felt it was \"accept\" but feel more confident now!\n\n(updated for final review)", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "STRENGTHS\n- The proposed idea is very clean and easy to understand, and the paper is mostly well-written\n- The method outperforms an existing safe RL method\n- The study of multiple RL agents and sampling times (simulation step sizes) is helpful for the reader to understand which heuristics matter for tuning the performance of this method\n- I really like the idea of the slack variables evolving; it's clever to pull the slack variables \"out\" of an optimization loop and use them to represent constraint dynamics. This might be a standard technique in the optimization literature, but it's neat to see it applied to RL.\n\nWEAKNESSES\n- The method requires twice-differentiable constraints and a known robot model (which the paper admits)\n- The method only considers learning instantaneous actions, as opposed to learning trajectories; so, it's unclear why the RL agent wouldn't drive a robot to an unrecoverable state from which constraint violation is inevitable\n- In the experiments, there are no collision avoidance of the arm body with obstacles or self-collisions; but these are the really interesting and difficult constraints to handle, and it's unclear how to extend the method to handle them\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I would like the authors to:\n1. explain why the method wouldn't accelerate a robot to states from which recovery is impossible,\n2. explain how the method might extend to uncertainty, underactuation, and non-smoothness in the dynamics and constraints,\n3. compare against non-RL based techniques for constrained control and/or trajectory planning\n\nIdeally the authors could also demonstrate the method on, e.g., a walking humanoid task or something really challenging to show how this method significantly outperforms any existing safe RL approaches (I don't know of any approaches that can learn to walk from scratch without falling/crashing).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Note I have put my own references as [R#], listed in the REFERENCES section below; I denote references within the paper as [#]\n\nAbstract:\n- the first sentence could be more specific; WHICH practical issues are not normally considered in the ML literature?\n- avoid contractions (e.g., it's)\n- is the model of the agent uncertain? would be good to clarify in the abstract\n\n1. Introduction\n- The requirement that the constraint manifold is differentiable is quite strong, and would require smooth approximations or else subderivatives/Jacobians of non-smooth constraints (e.g., signed distance from a manipulator to a polytopic obstacle)\n- it is good that the intro clearly states that the dynamics are perfectly known\n- the idea of \"no safe backup policy\" is really odd since there are often feasible states from which a robot can never recover due to underactuation (e.g., a walking robot about to fall over)\n- what prevents the proposed method from, e.g., accelerating a fully-actuated robot safely to a high speed such that the robot cannot avoid collision with an obstacle at the next time step?\n- the authors should consider a couple additional methods in this space that are based on reachability analysis [R1,R2,R3]; each of these have advantages and limitations with respect to the proposed method\n- a key issue with the Dalal et al. method in [10] is that it uses a linear approximation of the robot dynamics at the current state; it would be good to point out the proposed method's advantage of directly considering the nonlinear dynamics\n\n2. Learning on the Constraint Manifold\n- the transpose sign in (2) is $T$ instead of $\\top$\n- the discussion in section 2.5 seems unnecessarily detailed; I think you can just move the sentences on lines 187-188 and 199-200 to be under (11)\n- I would imagine that, as the slack variables tend to zero, the RL agent tends to move more slowly, so that it doesn't violate the inequality constraints. But, I don't see a barrier function (e.g., $-log(\\mu^2))$ to \"encourage\" this behavior, similar perhaps to [R4]. Maybe (6) creates exponential growth in the constraint-avoiding direction because $f(q) = -K_f\\dot{f}(q,\\dot{q})$?\n\n3. Experiments and Evaluation\n- it seems like an inverted pendulum formulation would be more widely-known than the CircularMotion task; I guess the point is to show how the proposed method stays on the equality constraint?\n- the IiwaAirHockey task has equality constraints for the joint positions -- but not for the entire arm to not impact the table? this is frustrating because the really interesting constraint is the collision-avoidance constraint\n- I think the two arm examples should really compare against the variety of methods in  controls and trajectory planning on manifolds, such as [R4,R5]\n\nREFERENCES\n[R1] Akametalu, A.K., Fisac, J.F., Gillula, J.H., Kaynama, S., Zeilinger, M.N. and Tomlin, C.J., 2014, December. Reachability-based safe learning with Gaussian processes. In 53rd IEEE Conference on Decision and Control (pp. 1424-1431). IEEE.\n\n[R2] Shao, Y.S., Chen, C., Kousik, S. and Vasudevan, R., 2021. Reachability-based trajectory safeguard (rts): A safe and fast reinforcement learning safety layer for continuous control. IEEE Robotics and Automation Letters, 6(2), pp.3663-3670.\n\n[R3] Krasowski, H., Wang, X. and Althoff, M., 2020, September. Safe reinforcement learning for autonomous lane changing using set-based prediction. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC) (pp. 1-7). IEEE.\n\n[R4] Bylard, A., Bonalli, R. and Pavone, M., 2021. Composable Geometric Motion Policies using Multi-Task Pullback Bundle Dynamical Systems. arXiv preprint arXiv:2101.01297.\n\n[R5] Ratliff, N., Zucker, M., Bagnell, J.A. and Srinivasa, S., 2009, May. CHOMP: Gradient optimization techniques for efficient motion planning. In 2009 IEEE International Conference on Robotics and Automation (pp. 489-494). IEEE."}}, {"review_id": "B5zJMAeHgp", "reviewer": "Reviewer_n6jm", "summary": "This paper proposes a novel approach to efficiently learn robotics tasks in simulation while satisfying safety constraints by leveraging the knowledge of the robot’s model and mathematical definition of the constraints. The proposed method (ATACOM) explores in the tangent space of the constraint manifold by including the model and constraints information into the computation of the action. This allows to convert the constrained RL problem into an unconstrained RL problem, which allows the application of any type of model-free RL algorithm to solve the task. In addition, the proposed method does not require an initial feasible policy or a safe backup policy as other safe exploration methods found in the literature. The effectiveness of the method is shown in three different experiments with different levels of difficulty and both equality and inequality constraints. The results show ATACOM has comparable or better performance as other safe exploration methods while keeping the constraints below an acceptable tolerance.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- Although the idea of using model-based information to improve the learning is not new and the outcome of improving the efficiency of the method is expected, using the mathematical definition of the constraints to formulate the problem such that the exploration is done in the tangent space of the constrained manifold is a novel and interesting idea. \n- The paper is clear and well written, and the authors provided a detailed explanation of their method with extensive evaluation and discussion of the results. I liked the authors included detailed information about the environments used for the experiments and the details about the implementation of their method in each of these environments e.g. actual rewards terms, equality and inequality constraints, parameters of the learning algorithm, structure of the NN, etc. \n- The paper provides extensive testing and comparison with different methods and RL algorithms.\n\nWeaknesses\n- The downside of the method is its scalability to high-order complex systems since it requires knowing the dynamic model of the system. In addition, the mismatch between the simulation model and the real world model can present significant challenges for the implementation of the method in real world applications. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "- My only recommendations for the authors would be to specify in detail the integration of the action space of the learning algorithm with the control action of the system. For example, what is the output of the NN and how it is integrated into the action/control input used for commanding the system. \n- I think the comparison in Figure 9 does not really provide any insightful result. It is expected that lower sampling frequency has a negative impact in the performance of the method.\n- It would be interesting to see a discussion about the use of different selections for the action space. For example, could the action space of the policy be the desired joint position and apply low-level PD controllers to track such desired positions?\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "In the abstracts the authors mention that although many safe exploration and constrained reinforcement learning algorithms exist in the ML literature, these are not applicable to real robotics tasks, which is a misleading statement. There are several works about Safe Exploration in RL that have been applied in simulated robotic tasks e.g. Berkenkamp et. all - “Safe Model-based Reinforcement Learning with Stability Guarantees” and even real robots e.g. Taylor et all - “Learning for Safety-Critical Control with Control Barrier Functions”.\n\nMinor observations in the grammar:\n“A variety of different practical considerations” is redundant. \n“The ATACOM method can be summarized as followings:”\n“When the starting from the point”\n"}}, {"review_id": "7FhzUJCD4-", "reviewer": "Reviewer_c13Z", "summary": "**Update:**\n\nI appreciate the thorough response from the authors. The explanation and additional experiment address my main concern, about applicability to problems with constraints on uncontrollable parts of the state, so I have raised my score.\n\n-----------\n\nThis paper proposes an approach, called ATACOM, that turns a constrained policy optimization problem into an unconstrained problem. The main idea is that rather than directly picking an action, the policy instead selects which direction to move in the tangent space of the constraint manifold (i.e., the states where the constraints are satisfied). This direction specifies the desired velocity, acceleration, or other higher-order derivative of the controllable state, and the action that achieves this is then executed. A key assumption of this approach is that the dynamics model is known.\n\nThis paper shows that the constraint values do not change, regardless of which tangent space direction is chosen by the policy—so if the constraints are satisfied at the initial controllable state q_0, then they will remain satisfied for the entire trajectory if ATACOM is used.\n\nThe empirical evaluation combines ATACOM with several state-of-the-art deep RL algorithms, and shows that it leads to minimal constraint violations, and outperforms existing approaches: early termination, error correction, and adding a safety layer.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n- The approach is theoretically grounded.\n- This approach can handle several realistic constraints for robot learning: providing a continuous velocity command, bounding the acceleration, and accounting for time discretization.\n- The approach is flexible, and can be combined with any deep RL algorithm for learning the policy. The experiments support this.\n\n**Weaknesses:**\n\n- As is mentioned in the paper, the main limitation is that this approach requires that the dynamics model is perfectly known. This is typically an unrealistic assumption for real-world robotic settings.\n- Another limitation is that the constraints can only apply to parts of the state that the robot can directly control, i.e. joint angles / velocities / accelerations. But there are many examples in which we may want to enforce constraints on parts of the state that the robot indirectly controls, for instance not bumping into dynamic obstacles while moving around, or not spilling water while carrying a cup. It is unclear how these could be expressed in terms of constraints on only controllable parts of the state.\n- The description of the method needs more intuitive explanations, to make it accessible to those who are not already familiar with manifolds and tangent spaces. For instance, a more intuitive description of $\\mathcal{A}$ is that $\\mathcal{A}(\\dot{q})$ produces the action $a$ that achieves velocity $\\dot{q}$, based on my understanding.\n- There are quite a few grammatical errors and typos.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This is a very interesting and promising idea, in the important research area of safe RL for robotics. It is both theoretically grounded and supported by empirical experiments. However, I have a couple of main concerns. I'm not convinced that setting constraints only on the controllable state (i.e., robot joint angles / velocities / accelerations) is enough to capture all/most of the kinds of safe robot RL problems. I don't think the paper addresses this limitation. I also found parts of the description of the method unclear and difficult to parse, so it's difficult to evaluate exactly what the method is doing.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- What happens if the trajectory does not start at a feasible point? Will the agent be able to reduce the constraint violation, or is it unable to because the policy's choice of $\\alpha$ doesn't change the constraint values?\n- Why does the max violation never reach zero in the CircularMotion experiments?\n- I would appreciate a more detailed explanation of why the null space basis can be viewed as the tangent space basis for the constraint manifold (lines 138-139). In general, the paper would be strengthened by adding more intuitive explanations throughout the method section.\n- I recommend that the paper is edited to fix grammatical errors and typos."}}]

Author rebuttal:

Meta-review: Summary: This paper presents a method for safe learning called ATACOM. The method transforms a constrained RL problem into an unconstrained problem. It is general for any model-free RL algorithm, accommodates both equality and inequality constraints, and is shown to be more effective than other baseline methods in simulation experiments. 

Clarity: In general, the paper is clear in its motivation, overarching goal behind the proposed solution, and the analysis of its experimental results. That being said, the authors are encouraged to consider reviewer c13Z’s feedback in adding more information about the intuition behind constraint manifolds. 

Quality: Reviewers appreciated the comparison of ATACOM with a breadth of other state-of-the-art methods and seem mostly convinced that the proposed method outperforms. 

Originality: Reviewers were generally positive about the originality of the theoretical aspects of the proposed approach. 

Significance: On the practical side, reviewers had mixed feedback about the usability of the proposed approach. There is a concern that assumptions are too restrictive and renders the algorithm not practically usable for real-world robots. (See “opportunities for improvement”). 

Pros: 
The paper presents a novel approach to safe learning, an area of strong interest to both robotics and ML communities. 

Main opportunities for improvement: 
I would suggest clearly characterizing and discussing the usability of the proposed approach to be a focus areas for the rebuttal and revision. All reviewers expressed concern that different assumptions in the algorithm (e.g. dynamics model must be known, constraints must be in terms of controllable parts of the state, smooth constraints, etc.) prevent actual application to real robots as claimed. Please see each reviewer’s individual review for specific details. 

Aside from this, reviewers made more minor suggestions for improving the readability of the paper that should be addressed. 

Thank you for considering our feedback, and we look forward to seeing the updated paper. 

========== Final Decision

All reviewers recommend accept, with 2/3 recommending strong accept. The paper also received strong scores in all sub-categories from all reviewers. Most notably, 2/3 reviewers both gave it a 4 in impact and very good in originality. As a result, I think it would be appropriate to highlight this paper in the oral session. 
Decision: Accept (Oral)