Venue: CoRL 2021 Conference

Reviews:

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper presents SCAPE, an approach for learning stiffness control from standard position control demonstrations by augmenting them with the inherent passive stiffness of the robot. This demonstration augmented with the inherent stiffness, which can be obtained from the model or the physical specs, can serve as a sub-optimal demonstration of a stiffness control trajectory. SCAPE learns a position control policy by imitation learning from these demonstrations; in order to avoid learning from highly sub-optimal actions an additional Q-filter is used. This Q-filter switches learning between the demonstration and the robot’s actions at a given state depending on their relative Q values. To further reduce oscillatory behaviour when learning with such a switching loss and to encourage exploration, an additional imitation regulator is used. This regulator switches to learning completely from self-experience once the policy starts achieving significant success thereby allowing the policy to bootstrap from its own experience towards the end of learning. The approach is tested on three robot manipulation tasks in simulation and the real world and shows good performance compared to baseline methods, both in terms of final performance and obeying safety constraints. An ablation also evaluates the different parts of the proposed approach.
strengths_and_weaknesses: This paper proposes an approach to leverage position control demonstrations to learn stiffness control policies by demonstration augmentation. The approach is evaluated on three manipulation tasks and shows good results compared to a baseline method. The ablation study also provides useful insights on the usage of the different newly proposed components in this context. A few comments:
1. The proposed approach needs position control demonstrations to bootstrap policy learning. How many demonstrations are needed for stable learning? How many demos are used for the results presented in the paper? It would be useful if these numbers are provided and an additional ablation is run w.r.t the number of demonstrations used for training.
2. While it is mentioned that baseline methods cannot use imitation learning since they do not have access to the augmented demonstrations, it would be interesting to see if the imitation learning loss is used just to optimise the action dimensions while the online RL is used to optimise the stiffness actions. This would be a hybrid approach where an initial policy would be trained with imitation learning (this would only be in the standard action space) and an additional policy could be trained on top of this policy using RL which sets the stiffness actions. This would be a better and fairer comparison compared to the baseline methods as it does have access to the demonstrations but not the augmented stiffness component. 
3. It looks like the presented baseline approach does pure position control and does not actually do any stiffness control. Is it possible to compare against a baseline that learns stiffness control from scratch on the proposed tasks? This would also give a clearer picture of the “near-optimal” stiffness control policy as by learning from the scratch the policy would not be biased by the sub-optimal demonstrations. It would also shed light on how well SCAPE does in terms of performance compared to a near-optimal policy.
4. Can the approach be extend to multi-joint stiffness control? What would be the main limitations in this regard? A discussion on this would be useful.
5. The approach uses the inherent stiffness of the manipulator to augment the position control demonstrations. What would happen if there is noise in this stiffness parameter, i.e., there is mismatch between the used stiffness and the actual stiffness of the system? Were any experiments done with noise added to this stiffness parameters? Is SCAPE able to recover a good policy even in this setting? 
summary_of_recommendation: Overall, the proposed approach is well motivated and shows good results across the tasks it is evaluated on. That being said the baselines seem weak and more analysis and comparison is needed to clearly establish the strengths of the method.

After rebuttal: I've increased my score to a "Weak Accept" as the rebuttal addressed several of my concerns.
issues: Further experiments are needed as discussed above. More details on the experimental setup and ablations can also add strength to the paper.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: As previous works have discovered, variable impedance control is an effective action space for RL in robotic manipulation. However, for many tasks, learning a good policy without an initialization trained on expert demonstrations is difficult; furthermore, collecting demonstrations in variable impedance space is considerably more complicated than simpler position control demonstrations.

To solve this issue, the authors propose SCAPE, with the following key contributions:
1) A method for converting expert position control demonstrations to suboptimal variable impedance demonstrations, by inferring stiffness from the position controller’s feedback gains.
2) A novel “Imitation Regulator,” which allows for stable switching from demonstration to the robot’s prior experience in the replay buffer.
3) A series of comparative and ablative experiments demonstrating the method’s superiority over a) position control (fixed impedance) with demonstrations and b) variable impedance control without demonstrations, in which the proposed method outperformed the baselines.


strengths_and_weaknesses: Strengths
1) The proposed method displayed significant performance gains over properly selected ablations and comparisons. The increase in data efficiency and safety provides a compelling argument for using variable impedance control with demonstrations when they are available. It was particularly great to see real-world experiments to this end.
2) The explanation of the method was clear and concise.

Addressed Weaknesses
1) A central motivational point of the paper is that expert demonstrations where impedance is being controlled explicitly are “expensive and difficult to acquire” (line 38). It’s not very clear to the reader where this difficulty comes from, as it is only briefly mentioned in the intro.

summary_of_recommendation: The method clearly gives significant gain in data efficiency, performance, and safety to the presented tasks. However, I don’t see much future work being spurred by SCAPE.
issues: Addressed major comment:
Authors are recommended to bolster the motivation of the project in the paper, particularly by elaborating on why converting position control demonstrations is better than collecting demonstrations in the proposed action space.

Addressed Nitpicks:
1) The authors occasionally make some conclusive statements without sufficient evidence. For instance, that another method’s poor performance “is due to the high dimensionality induced by the variable impedance that 99 hinders policy search” (lines 98-99). It is also not clear to the reader how the spikes in force on the real-world nu-fingers can be attributed to “poor force tracking in MuJoCo” (line 263).
2) The difference in scale between the forces generated by the baseline & SCAPE in the NuFingers demo make the plot (Fig. 4c) difficult to read.
3) It is not clear to the reader why the authors refer to their method as “stiffness control,” while all of the cited related methods ([29, 30, 31]) take the convention of calling the action space “impedance control.” I suggest either elucidating the difference between the two, or using the more standard name if there is none.
review_title: null

final_score: rating: Weak Reject: I recommend rejecting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: The paper proposes an insight for reinforcement learning of variable impedance control policies with only kinematic demonstrations. 
The main idea is to train a policy for impedance control with behavior cloning using kinematic trajectories and the default stiffness from the model of the robot's controllers. 
This provides a suboptimal impedance policy for the agent to imitate and improve upon.
Noting the shortcomings of an existing approach to RL with demonstrations (the Q-filter), the paper also proposes an "imitation regulator" that helps to trade off policy exploration with remaining faithful to demonstrations. 

strengths_and_weaknesses: Strengths
- The paper presents an insight for learning from demonstration with variable impedance control - if only kinematic demonstrations are available, as opposed to the full force profile, the hand/arm's default impedance is informative as to what impedance value the agent's policy ought to be initialized with. 
- The paper also proposes an improvement to the Q-filter algorithm using success rate rather than Q-value thresholds, which they observe to have an improvement on stability. 
- Hardware sim2real results are successful. 

Weaknesses
- The assumptions could be stated more explicitly. In particular, the method assumes that the demonstration was provided as a kinematic path and executed via the robot's "standard" position controller with stiffness k_passive. This could be the case - e.g. if the arm were naively executing commands from a  keyboard/mouse or motion planner - but there are other possibilities as well, e.g. kinesthetic teaching, or a demonstration recorded with a task-space controller rather than a joint space controller, etc. I think there are additional cases here that at least warrant mention or a clarification of the problem setting. 

- I was surprised the position-controlled baseline failed the blocks task (wrt performance) given that it succeeds in the original version of the problem. 

- The "series elasticity" force measurement procedure needs to be elaborated upon. 
- The performance boost from the proposed imitation regulator is relatively marginal. 



summary_of_recommendation: To my knowledge, the paper is one of the first to present results on learning from demonstration in the context of variable impedance control, but lacks sufficient novelty, clarity, and depth. 
issues: - Damping is entirely omitted. Is everything implicitly critically damped?

Minor Issues 
- Line 24-25 "in an unstructured environments" 
- I had some difficulty reading the plots because I'm colorblind - please consider using a more readily readable palette. 
- It might be useful to point readers to the recent review "Variable Impedance Control and Learning - A Review" from Abu-Dakka and Saveriano.
review_title: null

final_score: rating: Weak Accept: I recommend accepting the paper, but will not argue for my recommendation if the majority of other reviewers have a different opinion.
review_content: summary: This paper introduces an RL algorithm guided by demonstrations that learns stiffness control policies for dexterous manipulation tasks involving multi-stage actions such as approach->grasp->transport->place object or make contact->move/rotate object->release contact, from position-only demonstrations. As rightfully mentioned, existing RL approaches for such dexterous manipulation tasks have focused solely on position control policies, disregarding the effect of the contact forces on the object that can damage or break it in the real-world evaluation. This can be alleviated by learning state-dependent variable impedance/stiffness control policies for object manipulation. Although there exists RL approaches for learning variable impedance control policies for contact-rich manipulation tasks, such as wiping, shoveling, peg-in-hole, etc. most of these approaches focus on a specific control policy tailored to the nominal action, not including the grasping or releasing of a tool. In this work, this is achieved by adopting the HER (Hindsight Experience Replay) + Q-filter approach [19] and augmenting it with a self-imitation learning strategy [35] which regulates the source of imitation, either from demonstrations or from past experiences. The success of the method lies in the augmentation of position-only demonstration trajectories with estimated stiffness, that although suboptimal, are capable of guiding the learning of a stiffness control policy that can minimize interaction forces while reaching task success. The algorithm was evaluated in 3 contact-rich simulation environments of varying complexity and 1 real-world experiment showcasing the transfer of a learned stiffness control policy in simulation to the real-world test bed with no extra fine tuning.
strengths_and_weaknesses: The RL+demos algorithm presented in this paper has two major strengths above state-of-the-art that contribute to improving safety and scale: 
(i) learning stiffness control policies using solely position trajectories while using no force measurements. This is useful as oftentimes manipulators, especially grippers or robot fingers, are not equipped with reliably force/torque sensing yet they are tasked to be used in contact-rich scenarios. Further, due to the demonstrations the robot does not learn the state-dependent stiffness function from scratch but it finds an optimal policy from the suboptimal demonstrations. 
(ii) learning state-dependent stiffness control policies for multi-stage object manipulation tasks. As mentioned above, previous approaches for variable impedance control RL have failed to learn such multi-stage tasks where the manipulators make or break contacts. 
The evaluation of the proposed algorithm is thorough and convincing, the paper would benefit from clarifying many important details related to the strengths listed above. 

First of all, while the overall claim of not needing force measurements is compelling and useful, it gets confusing when we get to the reward function (3) and see that it needs the applied force to shape the reward. Only until reading the appendix does one discover that this force is in fact the estimated force from series elasticity of the gripper or the fingers. This estimation procedure must be highlighted and explained in the main text. Is it simply estimated by F_{est}=k_{est}\delta x? What would the displacement be? If it is computed by some displacement is it of the fingertips/joints? Furthermore, is this force estimated from the k_{passive} stiffness used for the suboptimal demonstrations? Is there a relation there? I would assume k_{est}=k_{passive} however this is not clear from the text.

Second, although the appendix properly describes the action/state spaces of each environment, the description in the main text of the learned stiffness control policies and the low-level controller is lacking. One would assume that the “controller” box in the diagram from Fig. 1a is a joint position controller that takes q_{des} and k_{des}, however is the controller a simple P-controller? PD-controller? Is this the same for all environments? Further, the structure of the policy is not discussed or the off-policy method used, is it DDPG as in [19] and the original HER approach?

Third, it would be important to discuss the type of demonstrations that this approach would work for. When using kinesthetic demonstrations, the stiffness of the robot when demonstrating the task, k_{passive}, is not the desired stiffness during task execution as this will be very low or even 0 (depending on the type of kinesthetic teaching being used either gravity compensation, impedance or admittance control). In the experiments presented in this paper, it seems that the demonstrations are given through teleoperation. However, this is not really discussed anywhere and is what I inferred from reading the related papers. It would be great if the authors clarified this and possibly discussed how this approach can be extended to other types of demonstration interfaces.  

Fourth, it would be interesting to see how this approach fairs with real force/torque measurements in the demonstrations rather than the estimated ones. Would the learning converge faster? Would there be more or less oscillations when optimizing with the suboptimal demonstrations? This is important to understand the applicability of this approach to other platforms/demonstrations where force/torque measurements are available. 

Fifth, the related work section is missing some relevant papers that the authors might find interesting:

1* https://arxiv.org/pdf/2004.10886.pdf

2* https://link.springer.com/article/10.1007/s10514-017-9636-y

3* https://arxiv.org/pdf/2103.15406.pdf 

1* and 2* are RL approaches that learn variable impedance control policies. 3* not so relevant to this paper but might find interesting as it explores the effect of stiffness in a simulation environment on the learning parameters.

Finally, for the block environment, it would be interesting to see how much of an improvement the proposed method is as opposed to the position-control only policy approach offered in [19]. This approach is similar to Condition 4 in the ablation study, but not quite because it’s considering the safety constraints in the reward function. However, the approach in [19] is equivalent to RL+IL with non-augmented demos + Q-filter with only task-success as reward. It would seem (from comparing the results in [19]) that SCAPE is much faster than [19]-- while at the same time keeping the block safe. Does this mean that using stiffness control policies yields more efficient learning than position-control only? This would be an interesting question to answer and analyze. 

+ After reading this paper, I finally understood the benefits of the HER approach. 

summary_of_recommendation: This paper offers an interesting solution to a challenging problem in RL for dexterous manipulation. Hence, I believe it can be impactful but the paper needs some pruning and clarifications on methods and design choices before it is publishable. 

issues: Clarify issues, questions and suggestions I mention in the “Strengths and Weaknesses” section. Also, if running a new test simulation for the block environment with the pure approach from [19] is not feasible, mentioning the convergence of that paper + analysis of when the gripper exceeds breaking force on the object (as compared to SCAPE) would be sufficient.
review_title: null

Meta-review: Strengths: 

** The paper focuses on an important and interesting problem, learning for stiffness/force control. The experimental focus here highlights the distinction between what is possible w/ position control only versus stiffness control. This can be hard to do and I think the paper does it well.

Weaknesses:

** Unclear and weak baselines: it is not completely clear what is the baseline reported in section 5.1. Sect 5 says “we compare our results with position control (existing approach) so that the difference only lies in the policy parameterization”. I’m assuming this means that the baseline algorithm cannot change the stiffness from the passive value. Would it be possible to compare to a baseline that is identical to SCAPE except for the addition of the methods proposed in sect 3 (i.e. the same action space) or is that what you’re doing already? Also, a reviewer asks for a comparison to a method that uses IL to learn a position policy and RL to learn a stiffness policy. Generally, it seems that stronger baselines would be appropriate here.

** Important details are missing in the main text of the paper. For example, what exactly is the RL algorithm that is augmented with the ideas in sect 3 and what is the baseline algorithm? I believe it is DDPG + HER, but this should be explicit. Loss functions would help here. How were demonstrations obtained and how many were used? 

** Some of the reviewers were unclear about the high level motivations of the paper. It would also help to frame the problem more formally, i.e. clarify the assumptions about the absence of stiffness demonstrations and why stiffness demonstrations are hare to obtain. I think the contributions could be stated more explicitly also: the basic methods proposed in sect 3 come from the literature (maybe IR is the key contribution?). It seems the main contribution here is in the application. So, why do we refer to this as a new method, i.e. “SCAPE”?

Post-rebuttal:

The authors have made significant improvements to the motivation and explanations of key points identified by the reviewers. The authors have also added additional experimental comparisons that help understand the method better.



Decision: Accept (Poster)