SCAPE: Learning Stiffness Control from Augmented Position Control Experiences
Authors: Mincheol Kim
Scott Niekum
Ashish Deshpande
Venue: CoRL2021 Poster (2021)
Keywords: Manipulation
stiffness control
reinforcement learning

Abstract: We introduce a sample-efficient method for learning state-dependent stiffness control policies for dexterous manipulation. The ability to control stiffness facilitates safe and reliable manipulation by providing compliance and robustness to uncertainties. Most current reinforcement learning approaches to achieve robotic manipulation have exclusively focused on position control, often due to the difficulty of learning high-dimensional stiffness control policies. This difficulty can be partially mitigated via policy guidance such as imitation learning. However, expert stiffness control demonstrations are often expensive or infeasible to record. Therefore, we present an approach to learn Stiffness Control from Augmented Position control Experiences (SCAPE) that bypasses this difficulty by transforming position control demonstrations into approximate, suboptimal stiffness control demonstrations. Then, the suboptimality of the augmented demonstrations is addressed by using complementary techniques that help the agent safely learn from both the demonstrations and reinforcement learning. By using simulation tools and experiments on a robotic testbed, we show that the proposed approach efficiently learns safe manipulation policies and outperforms learned position control policies and several other baseline learning algorithms.

SCAPE: Learning Stiffness Control from Augmented Position Control Experiences
Mincheol Kim', Scott Niekum Ashish D. Deshpande'
Department of Mechanical Engineering
*Department of Computer Science
The University of Texas at Austin, USA
mincheol@utexas edu, sni aniekumecs. utexas. edu, ashish0austin, utexas edu
We introduce sample-efficient method for learning state-dependent stiffiness control policies for dexterous manipulation. The ability to control stiff- ness facilitates safe and reliable manipulation by providing compliance and ro- bustness to uncertainties, Most current reinforcement learning approaches to achieve robotic manipulation have exclusively focused on position control, of- ten due to the difficulty of learning high-dimensional stiffness control policies. This difficulty can be partially mitigated via policy guidance such imitation learning. However, expert stiffness control demonstrations are often expensive or infeasible to record. Therefore, we present an approach to learn Stiffness Control from Augmented Position control Experiences (SCAPE) that bypasses this difficulty by transforming position control demonstrations into approximate, suboptimal stiffiness control demonstrations. Then, the suboptimality of the aug- mented demonstrations is addressed by using complementary techniques that help the agent safely learn from both the demonstrations and reinforcement learning, By using simulation tools and experiments on a robotic testbed, we show that the proposed approach efficiently learns safe manipulation policies and outperforms learned position control policies and several other baseline learning algorithms.
Abstract:
Keywords: Manipulation, Stiffness control, Reinforcement learning
1 Introduction
In recent years, deep reinforcement learning has been successfully used improve object manipula- tion with robotic hands [1, 2]. However, one of the primary limitations of these works is that in most robotic hands, the robot joint poses are explicitly controlled through position control and forces are implicitly decided. Lack of explicit control over the forces leads to limited safety and inability to handle uncertainties [3]. These might be critical factors when a robot is operating in unstructured environments and during the exploratory phase of the learning process [4, 5].
Modulation stiffiness in concert with position control has been shown to address robustness, safety, and performance under uncertainties [6, 7, 8]. and has gained much attention in the learning com- munity as well [9]. However, stiffness control imposes additional action dimensions, which affects sample-efficiene of policy learning. This hindrance can be partially mitigated through guidance imitation learning [10]. Expert demonstrations have been successfully collected and used in pol- icy learning for position -ontrol-based robotic hands [1. 11, 12]. However, in the case of stiffness control, such expert demonstrations are expensive and difficult to acquire. Typical demonstrations directly be recorded via various sensors, but stiffness is not measurable quantity, but rather relationship.
In prior literature, admittance control has been used to capture equilibrium position trajectories [13, 14]. Subtle and quick impact perturbations are used measure the compensatory forces and torques employed by the demonstrator while performing the trajectory, which are then used to implicitly calculate the stiffness at certain positions. These stiffness estimates are then used to further estimate and model the stiffness profiles, thereby compounding potential errors. This estimation process is more ambiguous for object manipulation due to the required precision and accuracy, and therefore poses a major challenge to learning stiffness control from demonstrations.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

In this paper, we present a novel learning strategy stratey--Stiffness Control from Augmented Posi- tion control Experiences (SCAPE) learning state-dependent stiffness control policies in high- dimensional problems such as dexterous manipulation. Imitation learning is used in conjunction with reinforcement learning to provide policy guidance, and we propose a way to bypass the need for stiffness demonstrations through an augmentation process. This process leverages the knowl- edge of the robot model such that we do not require expert stiffness control demonstrations. Instead, position control demonstrations are augmented to infer approximate, suboptimal stiffness control demonstrations. To address this suboptimality, we use a Q-filter [11] to prevent the agent from mimicking dangerous choices that may appear in the inferred stiffness demonstrations. We also in- troduce the concept of an imitation regulator that controls the mode of imitation depending on the assessment of the current policy. Ablation studies show that these techniques play meaningful roles in both safety and stability of learning. Through simulation and experiments, we show that SCAPE produces successful policy that is robust different types of realistic uncertainties, and safe in terms of force interaction.
2 Background and Related Works
Some of the notable past works in learning stiffness control rely on a reference trajectory, which we refer to trajectory-dependent approaches. However, the lack of robustness to uncertainties and variability in the task dynamics renders these approaches inapplicable to dexterous manipulation. Other recent research aims to learn stiffness controller that dynamically adapts to the environment, which we refer to as state state- deteedepnndent approaches. In this section, we briefly explain some of the notable attempts to learn stiffness control, and discuss possible shortcomings.
2.1 crajectoor-deppnder Stiffness Controllers
One possible approach is to learn time-indexed gain scheduling through PI [15], which is stochas- tic optimization method that results in time-indexed reference trajectory that can be tracked by the robot without modeling the inverse kinematics or dynamics. This approach adds additional parame- ters to control compliance that the resulting controller takes environment dynamics into account and modulates the gains accordingly [16, 17, 18). However, a solution from P1 can only opti- mize about pre-defined cost function and cannot be used for object-centered manipulation, which requires highly divergent position and stiffness trajectories depending on the goal and observations.
Another approach uses an Incremental Gaussian Mixture Model (IGMM) and Gaussian Mixture Re- gression (GMR) to predict the interaction force for the next time-step, and feed-forward appropriate control effort [19, 20J. The goal of this approach is to learn feed-forward model such that the feed- back stiffness control effort can be minimized. However, this approach makes a critical assumption that expert demonstrations with foree trajectories available, which renders it inapplicable without such demonstrations.
Trajectory planners combined with reinforcement learning can also be used. Once the trajectory is defined, a residual control policy can be learned to adjust the gains according to the current obser- vation. This method is mostly used in simple tasks where trajectory planner is readily available, such as in peg-in oeg-in-hole assembly tasks [21, 22, 23]. While this is a suitable approach for closed envi- ronments, it is less effective for dexterous manipulation where desired trajectories can change based on dynamic observations such as dropping the object or unexpected interaction forces. In addition, due to the lack of policy guidance, the learning process requires complex reward function well as an extensive amount of training time even with a trajectory planner aiding the policy search.
2.2 State-dependent Stiffness Controllers
Due to the specificity of the solution, relying fixed reference trajectory or scheduled gain is bound to result in catastrophic failure in dynamically changing environments. To account for large degree of variability in the environment, state- dede-deeenddnt stiffness control policies have recently been proposed. In this paper, we compare our work with these state-dependent methods, as the existing trajectory-deeendeett approaches are inapplicable in dynamic settings.
In one related approach, during hopping and wiping motions [24], the robots successfully produce stiffness control policies that outperform direct torque control and position control policies. How-

---

Imitation Leeming SR <SRu??
Imitaton regulator
SR
Policy Update
Policy
ungmentad ented Demtoistation
ts.
Experience Buffer
Output
m[s. g
SR SR,
Position Control Demonstrations
Robot Model
Self-Imitation Leaming
Destred stiffness
(%,
O
Destred robot pose
pairs
Inherent (Kussne)
Cobo pose
pase
Emor
Contmller
A ugmentad Demorstrations
Kyusiis
Interaction
E ent
Robot
Commarded robotpose
(s, mts, gl. g)
Experience Buffer
(s, a') pains
(a)
(b)
Figure I: (a) Proposed learning scheme for SCAPE. (b) Proposed augmentation process of position control demonstrations.
ever, this approach is strictly limited to simple and repetitive low- low-dimensional movements. For in- stance, the robot is allowed move only in one direction the hopping task, along pre-defined circular path in the wiping task.
A similar work demonstrates the performance difference based the form of the control policy [25], where variable impedance control outperforms all other controller types in simple tasks. Despite the effort to simplify the problem into a dimer-dimensional manifold by keeping the gripper closed in the door opening task, the variable impedance control (VIC) fails to outperform the fixed impedance control. possible culprit can be the high dimensionality induced by VIC. A similar work [26] observes stability-guarannn
learning for VIC during peg-in-hole tasks.
Other work depicts task-space impedance controller's performance in quadruped locomotion [27]. Variable impedance control policies outperform the direct torque control policy in terms of cumu- lative rewards and robustness disturbances. Interestingly, impedance controllers resulted in more energy-efficient policies although the reward function did not consider energy. that extensively tuned gait planner provided the agent learn successful policy.
worth noting
While these state-dependent approaches show promising results in terms of robustness to uncertain- ties, these existing approaches focus simple single-stage, repetitive tasks and require extensive reward shaping due the lack of policy guidance. In next section, we explain how SCAPE addresses this issue and produces successful state-dependent stiffness control policies, which can be used multi-stage tasks such as grasping and manipulation.
3 Methods
In this section, we present our approach, SCAPE, to learn state-dependent stiffness control without requiring stiffness control demonstrations. Use of imitation learning within reinforcement learning improves policy guidance, thereby enabling nigh-dimensional dexterous manipulation. For SCAPE and all the baselines, we employ Deep Deterministic Policy Gradient with Hindsight Experience Replay (DDPG + HER, [11]). The overall learning scheme is depicted in Fig. 1a, where the stiff- ness control policy takes observed states s and the goal g. and produces an action TTs, g) which contains the desired stiffnness in addition to the desired pose. The controller block is the high-level controller, which employs task-space stiffness control. SR refers to the overall success rate of the current policy (i.c., how often does the object reach the goal states while staying intact?).
commonly used action representation in dexterous manipulation deseribes only the desired posi- tion of the actuator in the inner control loop, hence the name position control. On the other hand, stiffness control policy outputs actions that describe desired position of actuators as well as the desired stiffness in the corresponding joint, as shown in Fig. la. However, as explained earlier, is difficult obtain expert stiffness control demonstrations for imitation learning. Therefore, we augmented position control demonstrations shown in Fig. lb.

---

These demonstrations are typically generated from telcoperation or kinesthetic teaching, and contain desired position trajectories. In our study, we use 25 demonstrations that accomplish task-related kinematic goals without consideration for the object fragility (e.g.. commanded to fully close the grippers). We leverage the fact that the stiffness of the robot known either from the simulation model or the hardware specifications, and that position control works by moving to the desired po- sition with the inherent stiffness or position gain of the actuator. In this paper, we refer to this inherent stiffness Kpassive. Therefore, state-action pairs, (s, a), in common position control demonstrations can be augmented to that of stiffness control demonstrations, (s',a"), where the desired stiffness is Kyassive. Consequently, these augmented demonstrations become suboptimal stiffness control demonstrations since the commanded stiffness of the robot is fixed to Kpossive- We can then infer the reward function of the task from the augmented demonstrations without manual reward shaping, and at the same time use it to learn improved stiffness control. Note that kpassive is n-dimensional stiffness, from which stiffness. In this paper, we modulate the stiffness in the grasping dimension (i.e., Kpassive E R).
choose any dimension of interest and modulate the
3.1 Outperforming the Demonstration
Simple imitation learning in the form of behavioral cloning does not allow the agent to improve be- yond the performance of the demonstrations due to the cloning loss [11]. The weight of the cloning loss can be decreased iteratively, assuming that the agent is able to learn policy equivalent to the demonstrator early in the iteration [1]. But it is unclear how to determine the amount of dependency on the demonstrations with respect to the iteration. Also, simply reducing the dependency does not prevent the agent from cloning the undesirable behaviors seen in the suboptimal demonstrations we use. Therefore, we adopt additional techniques to encourage the policy to outperform the augmented demonstrations and address its suboptimality.
3.1.1 Q-Filter
We use a Q-filter [11] to choose which replay transitions to clone from. The fundamental moti- vation behind learning from demonstrations is the assumption that the demonstrations provide near-optimal action. However, this is not true for the augmented demonstrations. A Q-filter allows the agent compare the values produced by the transitions from demonstrations, (si, ai, g). and the current policy, (8i, ma(si, 9),9) g). ) By comparing their values, the agent does not clone the behav- ior if its current policy provides better action for the given demonstration state. More formally, the cloning loss Loce can be defined as:
(I) However, it is often difficult to infer the subtle difference in the qualities of the policies solely from examining the resulting Q estimates due to the overestimation issue of Q values [28]. Although the usage of the Q-fiter improves the safety of learning, tends produce oscillatory gradients that prevent convergence of the policy due to its Boolean property [11].
Lice = ||a, (s [:)[[[ll11 ((..) (aa,)))((s,,,,,,aa
3.1.2 Imitation Regulator
To improve convergence of our method, we introduce an imitation regulator that observes the overall success rates of the current policy and determines the appropriate source of imitation from: 1) the augmented demonstrations and 2) the agent's 's own past experience. The latter is sometimes referred to as self-imitation learning [29]. The regulator controls the replay buffer DiR used for sampling
transition batches to imitate follows:
DIa [Ddemo: SR < SRref
(DSIL, otherwise
where DSIL and D Deemo refer to the buffers that store the actual experience replay and the aug- mented demonstrations, respectively. SR E [0, is the overall success rate of the current policy. is considered an overall success if the object reaches the goal states and also stays intact throughout the episode. SRref the reference success rate that is empirically found. Put simply, the regulator actively switches the source of demonstrations from Daemo to DSIL once the success rate reaches SRref. This brings three primary benefits. First, the agent no longer references the suboptimal demonstrations which contain undesirable behaviors. Second, the policy converges faster from the minimized oscillation of gradients. Third, by cloning the previously generated actions, the agent leverages exploration, thereby improving upon its current policy.

---

(a)
(b)
(c)
(d)
Figure 2: Initial and final scenes of the (a, b) Block, (c, d) Chip, and (e, f) NuFingers environments.
4
Environments
We use three robotic manipulation environments shown Fig. 2 for simulation and experiments. Details for each environment can be found in Appendix A. For simulation, we robots provided by OpenAI Gym [30]. all environments, we do not use the ground-truth force measurements during training. Instead,
quasi-static force measurements from the series elasticity of the robot, which do not require force sensors (i.e., z passiive [Xdesired Xeurrent)) )). Note that x is defined in the same coordinate frame as Kpassive- For safety evaluation however, unless stated otherwise, we use ground-truth force measurements. A successful policy must meet both task-related (e.g.. did the object reach the goal states?) and safety-related (e.g., is the object intact?) goals. For instance, even if the object reaches the goal states, the task is considered failure if the applied force exceeds the breaking force. The task-related kinematic reward is Ptask (s) + {-1, 0}, and the safety- related reward is (s)
+ (-o0, 0). Combining these reward functions naturally leads the agent to accomplish the kinematic goal while minimizing the estimated interaction force. For all environments, the immediate reward R for the observation is defined as:
R(s) = Ttask (s) + Teafety
Ptask (s) 0,
if kinematic goal is met
-1, otherwise
Teafety (s) a FF| - 3ll
where is the estimated force, & is the joint velocity, and a and B are normalization coefficients that depend on the environment.
Furthermore, we implement a low-pass filter with time constant of 0.05s for all force measure- ments in the simulation. To validate the robustness under realistic conditions, include three types of uncertainties during training: 1) random perturbation to the object within grasp, 2) measurement noise in the object's 's position, and 3) random control failure, that repeats the previous action.
4.1 Quasi-static Pick-and-place Environment (Block)
The Block environment entails a pick-and-place task and used to verify whether SCAPE is capable of learning safe and robust manipulation policy under quasi-static assumptions. In this environment we assume that the ground-truth force matches the estimated force, which is reasonable to assume when the object is in grasp and the involved masses small enough. The observation includes relative positions between the object, gripper, and the goal, as well as the gripper configurations, estimated force, and stiffness. The action includes Cartesian movement of the gripper, change in the gripper configurations, the changes stiffness and its limit.
4.2 Dynamic Pick-and-place Environment (Chip)
Chip environment is a dynamic version of the that SCAPE produces successful manipulation policy even dynamic situations where the agent does not have access to ground-truth force measurements. The robot is required to slide the object up the wall using friction. Thus, the ground-trutte force comes from not only the finger, but also from friction, which depends on velocity and normal force. The observation and action spaces are similar to the Block environment, which is part of the kinematic goal in this case. If only position is considered for the kinematic goal, we have found that the agent constantly moves the object around the goal location. We postulate that this phenomenon arises from kinetic friction being smaller than static friction.
environment, designed to demonstrate
the observation also includes the fingertip velocity Cartesian space,
5

---

4.3 In-hand Manipulation Environment (NuFingers)
To demonstrate the applicability for in-hand manipulation, we use the NuFingers testbed [7]. We first train the agent with domain randomization [2] in a representative environment Gym, and directly transfer the policy to the robot without any fine-tuning to validate its transferabilityy and robustness under uncertainties. The ground-truth force measurements are only used for validation. The observation includes polar coordinates and relative orientation from the object of each finger, forces, joint velocities, and stiffness. The action includes define radial and tangential movements of the fingers. Stiffness modulation is applied the radial direction, which dominates the grasping force. The task-related kinematic goal is rotate the object the desired orientation.
Results
In this section, demonstrate the performance of SCAPE in various environments. Without the proposed augmentation process, baseline algorithms must learn a stiffiess control policy from scratch [24, 25, 27], since expert demonstrations are not available. We compare SCAPE with these approaches for the ablation study in Sec. 5.2, as learning from scratch fails catastrophically. For the main experiments in Sec. 5.1, compare our results with position control (existing approach), so that the difference only lies in the policy parametrization. This comparison demonstrates the importance of using state-dependent stiffness controllers when force-sensitive tasks are involved, as opposed to existing position counterparts that are widely used in dexterous manipulation [1, 2].
5.1 Experimental Results
Figure depicts the success rates over epochs for the Block, Chip, and NuFingers environments. For in-depth assessment, we break down the plots to also show success rates for the safety and kinematic goals. Safety-related success rates refer to the fraction of the time when the experienced force is smaller than the breaking threshold. Note that exceeding it at any time results in an overall failure in the corresponding episode.
Kinematic pick-and-place tasks without uncertainties have been easily solved by the position control policies previous works [31]. However, Fig. shows that for position control policies, the heavy force penalty from large forces discourages exploration and prevents the agent from learning to even grasp the object. SCAPE on the other hand, uses stiffness control policies to successfully minimize the interaction force and reaches 100% overall success rate.
Overall
Overlll
0.5
Safnty
Safety
Safety
3
Suooess
0.5
0.5-
0.5
Existing
SCAPE
Task
Task
0.5
0
50 Pooch
100
100 Epoch
200
20
60
Epoch
(a) Block environment
(b) Chip environment
(c) NuFingers environment
Figure 3: Resulting success rates of SCAPE (solid) compared to position control (dashed). Success
rates for task-related goals (e.g., did the object reach the target states?) and safety-related goals (c.g.. how often did the object stay intact?) are separately plotted. Overall goals entail both goals.

---

(after training)
(evaluation)
(after training)
(evallaaiio)
(atter training)
levaluation)
-10
-10
-20
-20
-40
400
300
200
10
20)
200
200
100
-100
0.4
W//l
0.4
50
0.4
200
0.2
23
0.2
100
10:
50 Epoch
100
15
0
40 60
Time
Epoch
Time
Epoch
(a) Block environment
(b) Chip environment
(c) NuFingers environment
Figure 4: For each of the environments, force experienced by the objects and kinematic error of the resulting policies are shown on the left. The mean Q values, forces, and stiffnesses evaluated after each epoch are shown on the right.
For the Chip environment where the action space is smaller, the position-conttroleed agent learns to complete the kinematic task entailed in the demonstrations some degree shown in Fig. 3b. However, it fails to address the safety issues, thereby breaking the object in almost all of the evalua- tion episodes. Note that unlike in the Block environment, the ground-truth force measurements are used for evaluation. It is notable that SCAPE still produced a successful policy by referencing only the quasi-static force measurements from the series elasticity, which do not include friction.
For the NuFingers environment in Fig. 3c, we find a similar trend as in the Block environment. The position control approach fails learn policy that completes the kinematic task. SCAPE however, reaps the benefits of stiffness control and finds a successful policy. Most importantly, in spite of the model discrepancies between the simulation and the actual robot, the resulting policy proves to be successful after the sim-to-real transfer without additional training. To summarize, it is evident that state-dependent stiffness control policy outperforms the position control policy in terms of safety and robustness under uncertainties and that the SCAPE is capable of producing successful policies,
Figure 4 depicts various data during and after training. It is evident that the proposed approach quickly learns the necessary stiffness for the completion of the kinematic task. The stiffness and interaction force of the system strongly related to another trends the two curves. The existing approach which uses position control on the other hand, does not adjust the stiffness and therefore fails reduce the interaction force.
be seen from the similar
Also, notice that the objects in the Chip and NuFingers environments experience much more force than what the robot exert. This is because for evaluation, we use the ground-truth forces which come from various sources, such as the friction that depends on the normal force and the velocity of the object. The proposed approach successfully minimizes the ground-truth force without the actual measurement. While actual measurements will improve the stability, we expect the improvement to be marginal in the tested environments since large masses explosive movements are not involved.
5.2 Ablation Study
To examine the effects of each technique used in this paper, we examine the Block environment under five different conditions:
Condition 1: Reinforcement learning from scratch [24, 25, 27], without a Q-filter, without an imitation regulator
Condition 2: Reinforcement learning + imitation learning from augmented demonstrations, without Q-filter, without an imitation regulator
Condition 3: Reinforcement learning + imitation learning from augmented demonstrations, without Q-fittrr with an imitation regulator
7

---

Condition
0.8
Condition
0.95
Sucooss Rates
Condition
0.6
Condition
Suposs Retos
0.9
Condition
Condition
Candition
0.4
0.85
Condition
0.2
Cundition
0.8
Condition
0
0.75
0
20
40
60
80
100
40
60
80
100
Epoch
Epoch
(a)
(b)
Figure 5: (a) Overall success rates during evaluation and (b) Safery-related success rates during exploration for different conditions in the Block environment.
Condition 4: Reinforcement learning + imitation learning from augmented demonstrations, with a Q-filter, without an imitation regulator
Condition 5: Reinforcement learning + imitation learning from augmented demonstrations, with Q-filter, with an imitation regulator (SCAPE)
Condition represents the approach taken by the researchers the past [24, 25, 27], where imitation learning not used due to the absence of demonstration data. Conditions 2-5 use the augmented demonstrations introduced in this paper, with different combinations of complementary techniques. Note that condition 5 is used to produce the results in Fig. 3.
Overall success rates generated under the different conditions are shown Fig. 5a. From these results, we confirm that existing approaches [24, 25, 27] do not produce any meaningful results for multi-stage tasks (e.g., approaching an object, grabbing the object, and relocating the object). For such tasks, the augmented demonstrations play crucial role in providing guidance to the policy through imitation learning.
Moreover, in conditions 2 and 3, the agent is unable to filter out undesirable behaviors, thereby consistently breaking the object during exploration as shown in Fig. 5b. Therefore, we confirm that the Q-filter allows the agent to make safe decisions. Also, applying the imitation regulator without the Q-filter fails from satisfying safety goals. Lastly, without the imitation regulator, the agent keeps referencing the suboptimal demonstrations, which causes oscillations and delays convergence. Switching to self-imitation learning using the imitation regulator helps reinforce some of the past good behaviors preventing the policy from diverging, which is shown by the higher mean and smaller variance the condition 5 compared to those condition 4.
Conclusions
We conclude that our approach, SCAPE, is capable of producing successful state-dependent stiff- ness control policy, which plays crucial role in ensuring safety and performance in dexterous manipulation. SCAPE produces competent manipulation skills by improving sample complexity with augmented position control experiences. The suboptimality of the augmented demonstrations alleviated by combination of the Q-filter and the imitation regulator, which results in faster and more stable convergence to a successful policy. These techniques prevent the agent from blindly imitating the suboptimal demonstrations, and help focus on the past desirable experience. Through various manipulation experiments, we validate that SCAPE outperforms the existing position con- trol and stiffness control approaches. Therefore, SCAPE provides both safety and performance such that robust dexterous manipulation can be conveniently learned without stiffness control demonstra- tions. Future work may include extension of our work to seek the feasibility of passive stiffness modulation, which is capability of human hands. Passive stiffness not only determines the safety under robot malfunction but also the dynamic behavior of the robot under sudden impacts.

---

Acknowledgments
This work has taken place in the ReNeu Robotics Lab and Personal Autonomous Robotics Lab (PeARL) The University of Texas at Austin. Effort in the ReNeu Robotics Lab supported, in part, by NSF (1941260, 2019704), Facebook and Dept of VA. PeARL research is supported in part by the NSF (IIS-1724157, IIS- IIS-1638 107, IIS-1749204, IIS-1925082), ONR (N00014-18-2243), AFOSR (FA9550-20-1-0077), and ARO (78372-CS).
References
[1] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzani, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions, arXiv preprint: 1709. 10087, 2017.
[2] M. Andrychowicz, B. Baker, M. Chociej, R. Józefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, J. Schneider, S. Sidor, J. Tobin, P. Welinder, L. Weng, and W. Zaremba. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, pages 3-20, 2020.
[3] M. Li, H. Yin, K. Tahara, and A. Billard. Learning object-level impedance control for bust grasping and dexterous manipulation. IEEE International Conference on Robotics and Automation (ICRA), pages 6784-6791, 2014.
[4] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning walk via deep reinforcement learning. arXiv preprint: 1812. 102.//103, 103, 2018.
[5] B. Samuel, M. Taylor, and P. Stone. Transfer learning for reinforcement learning on a phys- ical robot. Ninth International Conference on Autonomous Agents and Multiagent Systems- Adaptive Learning Agents Workshop (AAMAS-ALA), 2010.
[6] T. D. Niehues, P. Rao, and A. D. Deshpande. Compliance in parallel to actuators for improving stability of robotic hands during grasping and manipulation. The International Journal of Robotics Research, pages 256-269, 2015.
[7] M. Kim and A. Deshpande. Balancing Stability and Stiffness Through the Optimization of Par- allel Compliance: Using Coupled Tendon Routing. IEEE Robotics & Automation Magazine, 2021.
[8] R. Mir, A. Marjaninejad, and F. J. Valero-Cuevas. The utility of tactile force to autonomous learning of in-hand manipulation is task-dependent. arXiv preprint: 2002.02418, 2020.
[9] F. J. Abu-Dakka and M. Saveriano. Variable impedance control and learning review. Fron- tiers in Robotics and AI, 7, 2020.
[10] S. Schaal. Is imitation learning the route to humanoid robots? Trends in cognitive sciences, pages 233-242, 1999.
[11] A. Nair, B. McGrew, M. Andrychowicz, W. Zaremba. and P. Abbeel. Overcoming exploration in reinforcement learning with demonstrations. IEEE International Conference on Robotics and Automation, pages 6292-6299, 2018.
[12] H. Van Hoof, T. Hermans, G. Neumann, and J. Peters. Learning robot in-hand manipulation with tactile features. JEEE-RAS 15th International Conference on Humanoid Robots, pages 121-127. 2015.
[13] Flash. The control of hand equilibrium trajectories in multi-joint arm movements. Biological cybernetics, 57(4)2257-774 1987.
[14] E. Burdet, R. Osu, D. Franklin, T. Yoshioka, T. Milner, and M. Kawato. A method for mea- suring endpoint stiffiness during multi-joint arm movements. Journal of biomechanics, 33(12): 1705-1709, 2000.
[15] E. Theodorou, J. Buchli, and S. Schaal. A generalized path integral control approach to rein- forcement learning. The Journal of Machine Learning Research, pages 3137-3181, 2010.

---

[16] J. Buchli, E. Theodorou, F. Stulp, and S. Schaal. Variable impedance control reinforcement learning approach. Robotics: Science and Systems VI, pages 153-160, 2011.
[17] E. Rombokas, M. Malhotra, E. Theodorou, E. Todorov, and Y. Matsuoka. Tendon-driven variable impedance control using reinforcement learning. Robotics; Science and Systems VIII, pages 069-376,22 2013.
[18] J. Rey, K. Kronander, F. Farshidian, J. Buchli, and A. Billard. Learning motions from demon- strations and rewards with lime-ivariaat dynamical systems based policies. Autonomous Robots, 42(1):45-64, 2018.
[19] M. Mathew, S. Sidhik, M. Sridharan, M. Azad, A. Hayashi, and J. Wyatt. Online Learning of Feed-Forward Models for Task-Space Variable Impedance Control. JEEE-RAS 19th Interna- tional Conference on Humanoid Robots, pages 222-229, 2019.
[20] S. Sidhik, M. Sridharan, and D. Ruiken. Learning hybrid models for variable impedance control of changing-contact manipulation tasks. The Eighth Annual Conference on Advances in Cognitive Systems, 2020.
[21] C. Hertaaen-eraandez, D. Petit, I. Ramirez-Alpizar, T. Nishi, S. Kikuchi, T. Matsubara, and K. Harada. Learning Force Control for Contact-rich Manipulation Tasks with Rigid Position- controlled Robots. IEEE Roboties and Automation Letters, pages 5709-5716, 2020.
[22] C. Heltadan- D. Petit, I. Ramirez-Alpizar, and K. Harada. Variable Compliance Con- trol for Robotic Peg-In-Hole Assembly: A Deep- Daep--Riiforrcmmen--Lemmmig Approach. Applied Sciences, page 6923, 2020.
[23] X. Zhang, Sun, Kuang, and M. Tomizuka. Learning variable impedance control via inverse reinforcement learning for force-related tasks. IEEE Robotics and Automation Letters, 6(2):2225-2232 2021.
[24] M. Bogdanovic, M. Khadiv, and L. Righetti. Learning variable impedance control for contact sensitive tasks. IEEE Robotics and Automation Letters, pages 6129 6136, 2020.
[25] R. Martin-Martin, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks. arXiv preprint: 1906. 088800 2019.
[26] S. A. Khader, H. Yin, P. Falco, and Kragic. Mtabilityguuaranll reinforcement learning for contact-rich manipulation. IEEE Robotics and Automation Letters, 6(1):1-8, 2020.
[27] K. Zhang, J. Lee, 7. Hou, C. de Silva, C. Fu, and N. Hogan. How does the structure embedded in learning policy affect learning quadruped locomotion? arXiv preprint: 2008. 12970, 2020.
[28] H. Van Hasselt, A. Guez, and D. Silver. Deep reinforcement learning with double q q-learning 2015.
arXiv preprint:
[29] J. Oh, Y. Guo, S. Singh, and H. Lee. Self-imitation learning. arXiv preprint: 1806.05635, 2018.
[30] G. Brockman, V. Cheung, L. Pettersson, J. Schneider, J. Schulman, J. Tang, and W. Zaremba. OpenAl Gym. arXiv preprint: 1606. 01 540, 2016.
[31] M. Andrychowicz, Wolski, A. Ray, J. Schneider, Fong, P. Welinder, McGrew, J. Tobin,
O. Abbeel, and W. Zaremba. Hindsight experience replay. Advances in neural information processing systems, pages
2017.
[32] Dhariwal, C. Hesse, O. Klimov, A. Nichol, M. Plappert, A. Radford, J. Schulman, S. Sidor, Y. Wu, and P. Zhokhov. Openai basclines. https://githe: com/openai/baselnnes. 2017.
10


Reviews:
[{"review_id": "m2rt-S6YnPI", "reviewer": "Reviewer_jxpB", "summary": "As previous works have discovered, variable impedance control is an effective action space for RL in robotic manipulation. However, for many tasks, learning a good policy without an initialization trained on expert demonstrations is difficult; furthermore, collecting demonstrations in variable impedance space is considerably more complicated than simpler position control demonstrations.\n\nTo solve this issue, the authors propose SCAPE, with the following key contributions:\n1) A method for converting expert position control demonstrations to suboptimal variable impedance demonstrations, by inferring stiffness from the position controller’s feedback gains.\n2) A novel “Imitation Regulator,” which allows for stable switching from demonstration to the robot’s prior experience in the replay buffer.\n3) A series of comparative and ablative experiments demonstrating the method’s superiority over a) position control (fixed impedance) with demonstrations and b) variable impedance control without demonstrations, in which the proposed method outperformed the baselines.\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n1) The proposed method displayed significant performance gains over properly selected ablations and comparisons. The increase in data efficiency and safety provides a compelling argument for using variable impedance control with demonstrations when they are available. It was particularly great to see real-world experiments to this end.\n2) The explanation of the method was clear and concise.\n\nAddressed Weaknesses\n1) A central motivational point of the paper is that expert demonstrations where impedance is being controlled explicitly are “expensive and difficult to acquire” (line 38). It’s not very clear to the reader where this difficulty comes from, as it is only briefly mentioned in the intro.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The method clearly gives significant gain in data efficiency, performance, and safety to the presented tasks. However, I don’t see much future work being spurred by SCAPE.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "Addressed major comment:\nAuthors are recommended to bolster the motivation of the project in the paper, particularly by elaborating on why converting position control demonstrations is better than collecting demonstrations in the proposed action space.\n\nAddressed Nitpicks:\n1) The authors occasionally make some conclusive statements without sufficient evidence. For instance, that another method’s poor performance “is due to the high dimensionality induced by the variable impedance that 99 hinders policy search” (lines 98-99). It is also not clear to the reader how the spikes in force on the real-world nu-fingers can be attributed to “poor force tracking in MuJoCo” (line 263).\n2) The difference in scale between the forces generated by the baseline & SCAPE in the NuFingers demo make the plot (Fig. 4c) difficult to read.\n3) It is not clear to the reader why the authors refer to their method as “stiffness control,” while all of the cited related methods ([29, 30, 31]) take the convention of calling the action space “impedance control.” I suggest either elucidating the difference between the two, or using the more standard name if there is none. \n"}}, {"review_id": "SLNEu25NfZk", "reviewer": "Reviewer_Lq7F", "summary": "This paper introduces an RL algorithm guided by demonstrations that learns stiffness control policies for dexterous manipulation tasks involving multi-stage actions such as approach->grasp->transport->place object or make contact->move/rotate object->release contact, from position-only demonstrations. As rightfully mentioned, existing RL approaches for such dexterous manipulation tasks have focused solely on position control policies, disregarding the effect of the contact forces on the object that can damage or break it in the real-world evaluation. This can be alleviated by learning state-dependent variable impedance/stiffness control policies for object manipulation. Although there exists RL approaches for learning variable impedance control policies for contact-rich manipulation tasks, such as wiping, shoveling, peg-in-hole, etc. most of these approaches focus on a specific control policy tailored to the nominal action, not including the grasping or releasing of a tool. In this work, this is achieved by adopting the HER (Hindsight Experience Replay) + Q-filter approach [19] and augmenting it with a self-imitation learning strategy [35] which regulates the source of imitation, either from demonstrations or from past experiences. The success of the method lies in the augmentation of position-only demonstration trajectories with estimated stiffness, that although suboptimal, are capable of guiding the learning of a stiffness control policy that can minimize interaction forces while reaching task success. The algorithm was evaluated in 3 contact-rich simulation environments of varying complexity and 1 real-world experiment showcasing the transfer of a learned stiffness control policy in simulation to the real-world test bed with no extra fine tuning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The RL+demos algorithm presented in this paper has two major strengths above state-of-the-art that contribute to improving safety and scale: \n(i) learning stiffness control policies using solely position trajectories while using no force measurements. This is useful as oftentimes manipulators, especially grippers or robot fingers, are not equipped with reliably force/torque sensing yet they are tasked to be used in contact-rich scenarios. Further, due to the demonstrations the robot does not learn the state-dependent stiffness function from scratch but it finds an optimal policy from the suboptimal demonstrations. \n(ii) learning state-dependent stiffness control policies for multi-stage object manipulation tasks. As mentioned above, previous approaches for variable impedance control RL have failed to learn such multi-stage tasks where the manipulators make or break contacts. \nThe evaluation of the proposed algorithm is thorough and convincing, the paper would benefit from clarifying many important details related to the strengths listed above. \n\nFirst of all, while the overall claim of not needing force measurements is compelling and useful, it gets confusing when we get to the reward function (3) and see that it needs the applied force to shape the reward. Only until reading the appendix does one discover that this force is in fact the estimated force from series elasticity of the gripper or the fingers. This estimation procedure must be highlighted and explained in the main text. Is it simply estimated by F_{est}=k_{est}\\delta x? What would the displacement be? If it is computed by some displacement is it of the fingertips/joints? Furthermore, is this force estimated from the k_{passive} stiffness used for the suboptimal demonstrations? Is there a relation there? I would assume k_{est}=k_{passive} however this is not clear from the text.\n\nSecond, although the appendix properly describes the action/state spaces of each environment, the description in the main text of the learned stiffness control policies and the low-level controller is lacking. One would assume that the “controller” box in the diagram from Fig. 1a is a joint position controller that takes q_{des} and k_{des}, however is the controller a simple P-controller? PD-controller? Is this the same for all environments? Further, the structure of the policy is not discussed or the off-policy method used, is it DDPG as in [19] and the original HER approach?\n\nThird, it would be important to discuss the type of demonstrations that this approach would work for. When using kinesthetic demonstrations, the stiffness of the robot when demonstrating the task, k_{passive}, is not the desired stiffness during task execution as this will be very low or even 0 (depending on the type of kinesthetic teaching being used either gravity compensation, impedance or admittance control). In the experiments presented in this paper, it seems that the demonstrations are given through teleoperation. However, this is not really discussed anywhere and is what I inferred from reading the related papers. It would be great if the authors clarified this and possibly discussed how this approach can be extended to other types of demonstration interfaces.  \n\nFourth, it would be interesting to see how this approach fairs with real force/torque measurements in the demonstrations rather than the estimated ones. Would the learning converge faster? Would there be more or less oscillations when optimizing with the suboptimal demonstrations? This is important to understand the applicability of this approach to other platforms/demonstrations where force/torque measurements are available. \n\nFifth, the related work section is missing some relevant papers that the authors might find interesting:\n\n1* https://arxiv.org/pdf/2004.10886.pdf\n\n2* https://link.springer.com/article/10.1007/s10514-017-9636-y\n\n3* https://arxiv.org/pdf/2103.15406.pdf \n\n1* and 2* are RL approaches that learn variable impedance control policies. 3* not so relevant to this paper but might find interesting as it explores the effect of stiffness in a simulation environment on the learning parameters.\n\nFinally, for the block environment, it would be interesting to see how much of an improvement the proposed method is as opposed to the position-control only policy approach offered in [19]. This approach is similar to Condition 4 in the ablation study, but not quite because it’s considering the safety constraints in the reward function. However, the approach in [19] is equivalent to RL+IL with non-augmented demos + Q-filter with only task-success as reward. It would seem (from comparing the results in [19]) that SCAPE is much faster than [19]-- while at the same time keeping the block safe. Does this mean that using stiffness control policies yields more efficient learning than position-control only? This would be an interesting question to answer and analyze. \n\n+ After reading this paper, I finally understood the benefits of the HER approach. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This paper offers an interesting solution to a challenging problem in RL for dexterous manipulation. Hence, I believe it can be impactful but the paper needs some pruning and clarifications on methods and design choices before it is publishable. \n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Clarify issues, questions and suggestions I mention in the “Strengths and Weaknesses” section. Also, if running a new test simulation for the block environment with the pure approach from [19] is not feasible, mentioning the convergence of that paper + analysis of when the gripper exceeds breaking force on the object (as compared to SCAPE) would be sufficient."}}, {"review_id": "QFFoCLdpw8u", "reviewer": "Reviewer_JwaB", "summary": "This paper presents SCAPE, an approach for learning stiffness control from standard position control demonstrations by augmenting them with the inherent passive stiffness of the robot. This demonstration augmented with the inherent stiffness, which can be obtained from the model or the physical specs, can serve as a sub-optimal demonstration of a stiffness control trajectory. SCAPE learns a position control policy by imitation learning from these demonstrations; in order to avoid learning from highly sub-optimal actions an additional Q-filter is used. This Q-filter switches learning between the demonstration and the robot’s actions at a given state depending on their relative Q values. To further reduce oscillatory behaviour when learning with such a switching loss and to encourage exploration, an additional imitation regulator is used. This regulator switches to learning completely from self-experience once the policy starts achieving significant success thereby allowing the policy to bootstrap from its own experience towards the end of learning. The approach is tested on three robot manipulation tasks in simulation and the real world and shows good performance compared to baseline methods, both in terms of final performance and obeying safety constraints. An ablation also evaluates the different parts of the proposed approach.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper proposes an approach to leverage position control demonstrations to learn stiffness control policies by demonstration augmentation. The approach is evaluated on three manipulation tasks and shows good results compared to a baseline method. The ablation study also provides useful insights on the usage of the different newly proposed components in this context. A few comments:\n1. The proposed approach needs position control demonstrations to bootstrap policy learning. How many demonstrations are needed for stable learning? How many demos are used for the results presented in the paper? It would be useful if these numbers are provided and an additional ablation is run w.r.t the number of demonstrations used for training.\n2. While it is mentioned that baseline methods cannot use imitation learning since they do not have access to the augmented demonstrations, it would be interesting to see if the imitation learning loss is used just to optimise the action dimensions while the online RL is used to optimise the stiffness actions. This would be a hybrid approach where an initial policy would be trained with imitation learning (this would only be in the standard action space) and an additional policy could be trained on top of this policy using RL which sets the stiffness actions. This would be a better and fairer comparison compared to the baseline methods as it does have access to the demonstrations but not the augmented stiffness component. \n3. It looks like the presented baseline approach does pure position control and does not actually do any stiffness control. Is it possible to compare against a baseline that learns stiffness control from scratch on the proposed tasks? This would also give a clearer picture of the “near-optimal” stiffness control policy as by learning from the scratch the policy would not be biased by the sub-optimal demonstrations. It would also shed light on how well SCAPE does in terms of performance compared to a near-optimal policy.\n4. Can the approach be extend to multi-joint stiffness control? What would be the main limitations in this regard? A discussion on this would be useful.\n5. The approach uses the inherent stiffness of the manipulator to augment the position control demonstrations. What would happen if there is noise in this stiffness parameter, i.e., there is mismatch between the used stiffness and the actual stiffness of the system? Were any experiments done with noise added to this stiffness parameters? Is SCAPE able to recover a good policy even in this setting? ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the proposed approach is well motivated and shows good results across the tasks it is evaluated on. That being said the baselines seem weak and more analysis and comparison is needed to clearly establish the strengths of the method.\n\nAfter rebuttal: I've increased my score to a \"Weak Accept\" as the rebuttal addressed several of my concerns.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Further experiments are needed as discussed above. More details on the experimental setup and ablations can also add strength to the paper.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "5U-NLBk9dji", "reviewer": "Reviewer_V3q7", "summary": "The paper proposes an insight for reinforcement learning of variable impedance control policies with only kinematic demonstrations. \nThe main idea is to train a policy for impedance control with behavior cloning using kinematic trajectories and the default stiffness from the model of the robot's controllers. \nThis provides a suboptimal impedance policy for the agent to imitate and improve upon.\nNoting the shortcomings of an existing approach to RL with demonstrations (the Q-filter), the paper also proposes an \"imitation regulator\" that helps to trade off policy exploration with remaining faithful to demonstrations. \n", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper presents an insight for learning from demonstration with variable impedance control - if only kinematic demonstrations are available, as opposed to the full force profile, the hand/arm's default impedance is informative as to what impedance value the agent's policy ought to be initialized with. \n- The paper also proposes an improvement to the Q-filter algorithm using success rate rather than Q-value thresholds, which they observe to have an improvement on stability. \n- Hardware sim2real results are successful. \n\nWeaknesses\n- The assumptions could be stated more explicitly. In particular, the method assumes that the demonstration was provided as a kinematic path and executed via the robot's \"standard\" position controller with stiffness k_passive. This could be the case - e.g. if the arm were naively executing commands from a  keyboard/mouse or motion planner - but there are other possibilities as well, e.g. kinesthetic teaching, or a demonstration recorded with a task-space controller rather than a joint space controller, etc. I think there are additional cases here that at least warrant mention or a clarification of the problem setting. \n\n- I was surprised the position-controlled baseline failed the blocks task (wrt performance) given that it succeeds in the original version of the problem. \n\n- The \"series elasticity\" force measurement procedure needs to be elaborated upon. \n- The performance boost from the proposed imitation regulator is relatively marginal. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "To my knowledge, the paper is one of the first to present results on learning from demonstration in the context of variable impedance control, but lacks sufficient novelty, clarity, and depth. ", "technical_quality": "Fair: Significant additional evidence required to support the claims.", "clarity_of_presentation": "Fair: Major revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "- Damping is entirely omitted. Is everything implicitly critically damped?\n\nMinor Issues \n- Line 24-25 \"in an unstructured environments\" \n- I had some difficulty reading the plots because I'm colorblind - please consider using a more readily readable palette. \n- It might be useful to point readers to the recent review \"Variable Impedance Control and Learning - A Review\" from Abu-Dakka and Saveriano. "}}]

Author rebuttal:

Meta-review: Strengths: 

** The paper focuses on an important and interesting problem, learning for stiffness/force control. The experimental focus here highlights the distinction between what is possible w/ position control only versus stiffness control. This can be hard to do and I think the paper does it well.

Weaknesses:

** Unclear and weak baselines: it is not completely clear what is the baseline reported in section 5.1. Sect 5 says “we compare our results with position control (existing approach) so that the difference only lies in the policy parameterization”. I’m assuming this means that the baseline algorithm cannot change the stiffness from the passive value. Would it be possible to compare to a baseline that is identical to SCAPE except for the addition of the methods proposed in sect 3 (i.e. the same action space) or is that what you’re doing already? Also, a reviewer asks for a comparison to a method that uses IL to learn a position policy and RL to learn a stiffness policy. Generally, it seems that stronger baselines would be appropriate here.

** Important details are missing in the main text of the paper. For example, what exactly is the RL algorithm that is augmented with the ideas in sect 3 and what is the baseline algorithm? I believe it is DDPG + HER, but this should be explicit. Loss functions would help here. How were demonstrations obtained and how many were used? 

** Some of the reviewers were unclear about the high level motivations of the paper. It would also help to frame the problem more formally, i.e. clarify the assumptions about the absence of stiffness demonstrations and why stiffness demonstrations are hare to obtain. I think the contributions could be stated more explicitly also: the basic methods proposed in sect 3 come from the literature (maybe IR is the key contribution?). It seems the main contribution here is in the application. So, why do we refer to this as a new method, i.e. “SCAPE”?

Post-rebuttal:

The authors have made significant improvements to the motivation and explanations of key points identified by the reviewers. The authors have also added additional experimental comparisons that help understand the method better.



Decision: Accept (Poster)