Redundancy Resolution as Action Bias in Policy Search for Robotic Manipulation
Authors: Firas Al-Hafez
Jochen J. Steil
Venue: CoRL2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Evolution Strategies
Redundancy Resolution
Action Bias
Velocity Control

Abstract: We propose a novel approach that biases actions during policy search by lifting the concept of redundancy resolution from multi-DoF robot kinematics to the level of the reward in deep reinforcement learning and evolution strategies. The key idea is to bias the distribution of executed actions in the sense that the immediate reward remains unchanged. The resulting biased actions favor secondary objectives yielding policies that are safer to apply on the real robot. We demonstrate the feasibility of our method, considered as policy search with redundant action bias (PSRAB), in a reaching and a pick-and-lift task with a 7-DoF Franka robot arm trained in RLBench - a recently introduced benchmark for robotic manipulation - using state-of-the-art TD3 deep reinforcement learning and OpenAI's evolutionary strategy. We show that it is a flexible approach without the need of significant fine-tuning and interference with the main objective even across different policy search methods and tasks of different complexity. We evaluate our approach in simulation and on the real robot. Our project website with videos and further results can be found at: https://sites.google.com/view/redundant-action-bias


Reviews:
[{"review_id": "zONA1JLbxJp", "reviewer": "Reviewer_9a3E", "summary": "The paper presents a redundancy-resolution approach for achieving the first and secondary objectives simultaneously. The key idea is to generate action biases based on kinematic redundancy to achieve the secondary objective, while those biases do not affect the first task objective. The biases are computed using null-space projection that is toward the direction of the negative gradient of the secondary objective. This concept is tested with both the deep RL algorithm and evolution strategy. The authors have evaluated the proposed method on two tasks, reaching and pick-and-lift task, of 7-DoF Frank Robot. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper proposes a structured approach for achieving the task with two objectives.\n+ The concept of redundancy resolution can be used with different learning algorithms.\n- The method is a bit specific to two-objective scenarios.\n- The results presented in Figure 2 seem not impressive: rather, it seems like the baseline without redundancy resolution achieves a higher success rate.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I *was* very positive about the paper because the paper discusses a decent mathematical approach for achieving the first and secondary tasks. The idea of null-space projection space would not be new (in fact, I would suggest a couple of papers [1, 2] that exploit null-space projection in different ways) but the combination with deep RL seems to be interesting. Therefore, I believe the paper proposes an interesting problem and a reasonable solution to it.\n[1] De Lasa, M. and Hertzmann, A., 2009, October. Prioritized optimization for task-space control. In 2009 IEEE/RSJ International Conference on Intelligent Robots and Systems (pp. 5755-5762). IEEE.\n[2] Gielniak, M.J., Liu, C.K. and Thomaz, A.L., 2011, May. Task-aware variations in robot motion. In 2011 IEEE International Conference on Robotics and Automation (pp. 3921-3927). IEEE.\n\nOn the other hand, Figure 2 seems to be surprising. Among four scenarios, the proposed redundancy resolution approach outperforms the baseline for only one case (OpenAI-ES for the reaching task). For two pick-and-lift scenarios, the baseline seems to dominate the proposed approach with a large margin. The authors quickly describe some other properties, such as variances of learning curves, but I am not sure that is very meaningful when the main criteria (success rate) shows a big difference. Therefore, I believe this must be clearly resolved at the rebuttal phase.\n\nFigure 1 is also a bit confusing. From the figure, I assumed that the authors will introduce the concept of the novel task redundancy (rather than using the simple kinematic redundancy) and proposed the method on top of it. If the task redundancy is not used in the paper, I would suggest removing it.\n\nSection 3 seems too long: I don’t feel like it is necessary to review all the methods, such as actor-critic methods, DDPG, TD3, and OpenAI-ES. The proposed technique should be generic to the choice of the policy optimization algorithm. I would suggest greatly shortening it and adding more experiments and discussions.\n", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "The authors must resolve the performance issue that is presented in Figure 2. If it is not improved, the point of the paper will be weakened.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "qinpCppzxsg", "reviewer": "Reviewer_pAEv", "summary": "The paper presents an approach to treat secondary objectives, such as robot safety constraints or collision avoidance, in reinforcement learning and evolutionary strategies as part of redundancy resolution in redundant systems.  The key idea is to add a bias to the actions that does not influence the immediate reward where the main objective is formulated (here, reaching and picking-and-lift task). The authors choose the bias actions (joint velocity commands) to generate motion in the null-space of the Jacobian. This formulation allows secondary objectives to be incorporated, e.g., desired reference joint configurations to avoid skewed configurations and might come from expert demonstrations or the distance of the links to external objects. The authors tested their new approach policy search with redundant action bias (PSRAB) on a simulated Franka robot and transferred the trained policy onto the real system. The authors empirically show that PSRAB does not greatly influence the main objective by performing an ablation study (RAB on or off). ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper addresses the interesting research direction of reducing the burden to design an effective reward function for policy search methods by handing over secondary objectives to existing conventional methods. The general direction of adding bias that does not change the immediate reward by adding actions that act in the null-space is very clever. Also, the paper is clearly written, and the colorful and precise figures help the understanding greatly. \n\nHowever, the paper can be also be criticized:\n1. The authors claim that PSRAB requires little hyperparameter tuning in the introduction and conclusion but do not address or compare this point in the main part of the paper. It would be good to clarify this point more clearly. Do the authors refer to the fact that the number of hyperparameters in the reward function decreases?\n2. The authors refer to the problem of applying RL and ES to real-world scenarios in the introduction (indicating that the presented method solves this issue) but perform training exclusively in simulation. Adding some training on a real robot would be helpful (even on a simplified task). \n3. The authors claim that with their method, the policy is safer in keeping a safe distance to objects. It would be interesting to empirically show how much more safety constraints are violated if such constraints are formulated in the reward function instead of within the redundancy resolution part. Alternatively, the authors could compare to methods from the safe RL community. \n4. To what extent is PSRAB capable of handling secondary objectives? Can all possible other objectives be handled? What is the relationship to multi-objective RL?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, the paper is theoretically sound and describes a clever idea that is worth to be investigated further in the future. Also, the paper appears to be polished and almost camera-ready. The only downside is that the feasibility has not been shown on a real system. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "To improve the paper, the authors should address the points of critique (1. to 4.).  ", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "K-3P4qQ_vA", "reviewer": "Reviewer_S6P5", "summary": "In this work, the authors present an approach for learning safer and more natural-looking manipulation policies using techniques based on redundancy resolution. Specifically, the authors argue that the actions can be biased based on secondary objectives in the space of redundant states defined by the equivalence of rewards. The major claims in the paper include:\n- The new approach does not interfere with the main objective;\n- Biased policies are safer to execute on a real robot and look more naturally;\n- The approach does not require fine-tuning;\n- The approach can help to escape local minima during the policy training;\n\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The idea of utilizing redundancy for biasing actions looks interesting;\n- The approach has a promise of not introducing strong interference with the main objective;\n- The proposed approach does look relatively simple to add to policy learning methods;\n- The authors demonstrate the effectiveness of the approach with real robotics experiments, including several tasks;\n\nWeaknesses:\n- For redundancy resolution knowledge of jacobians is required limiting its applicability;\n- Since the method introduces changes in action space directly, it is not straightforward to apply it with the most popular RL on-policy frameworks, such as PPO;\n- It does seem that equation (11) requires differentiability of the loss with respect to the state, which in some cases may not be possible;\n- Writing requires some additional work (see Issues);\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The presented approach proposes a novel view on secondary objective optimization in policy learning settings with a promise of less interference with the primary objective in the scope of manipulation tasks. Experiments with the real robotic arm look interesting and highlight some of the advantages of the approach, but a more detailed ablation study is required to make a strong conclusion. On top of that, the approach does have certain limitations narrowing its scope. Given all these points I suggest “weak accept” for this work.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {"issues": "- The general claim that the approach helps escaping local minima requires better support.\n- The claim regarding “absence of interference with the main objective” also requires better support. Redundancy resolution is a bias that keeps only the local reward indifferent. It is unclear that the cumulative reward would not change either due to the overall shift in trajectories distribution. \n- More clear explanation of the method is needed. For example, it is not straightforward how equation (11) contributes to safer behavior? Is only L_{2} loss utilized to bias the actions? \n- Explanation of how the method is adapted for specific tasks is also required. \n- The authors note that the method does not require hyperparameter tuning, yet, section 5.2 states that the choice of \\alpha does influence the final performance. In general, I would expect some sensitivity ablation experiments for this claim.\n- I advise using different letters for Jacobian and the performance measure.\n- To the best of my understanding, all experiments include only a single seed of optimization and do not propose extensive ablation studies. \n"}}, {"review_id": "EYVmuAeE80J", "reviewer": "Reviewer_g7Ri", "summary": "This paper proposes a method Policy Search with Redundant Action Bias (PSRAB) to constrain the action space for robotic manipulation by exploiting the kinematic redundancy. It finds a perturbation to the action that satisfies a secondary objective without changing the environmental reward. This method is shown to work across policy search methods. \nThey demonstrated in simulation and real-world reaching and pick-and-lift experiments that the proposed method reduces the kinematic redundancy. Also this method is shown to work across two different types of policy search methods: reinforcement learning and evolutionary strategy.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n1. The idea of leveraging the null space of the Jacobian and auxiliary reward functions to constrain the action space without affecting the end effector location is interesting\n\n2. The method seems to provide a general solution to incorporate secondary objectives in robotic manipulation applications \n\n3. The method can be combined with different types of policy search methods: reinforcement learning and evolutionary strategy.\n\nWeaknesses/Suggestions\n\n1, I really like the idea as stated in the strengths but I feel that the experiments make it difficult to assess the efficacy of the method:\n\n  a. The learning curves are shown in Figure 2 without key information about the number of trials/random seeds. Also, is it showing the mean of multiple runs or just a smoothed curve of one run?\n\n  b. In 3 out of the 4 subplots in Figure 2, the proposed RAB method doesn’t show any advantage. That brings concerns about the efficacy of this method.\n\n  c. I guess that the following is due to sudden context switching but it’s very confusing. “Figure 3 presents …. TD3 was also able to solve the pick-and-lift task, as shown in the lower-left plot”. I guess that this “TD3” and ”lower-left plot” refer to Figure 2. This context switching makes it difficult to read. Also, where’s the discussion on Figure 3 then?\n\n  d. “Further, it can be seen that the agent without RAB has a significantly higher variance in performance” What is the variance referring to here? I guess that it goes back to the missing information in point (a) but is it referring to variance across time steps? Or of multiple runs? \n\n  e. I feel that it’s missing a baseline method, that is, TD3 and OpenAI-ES where the secondary loss is added to the reward function, and the evaluation can use the original environment reward function.\n\n2. There are some inaccuracies in the descriptions throughout the paper:\n\n  a. I feel that what’s being said on Line 163-165 is a necessary condition to defining “task redundancy” based on identical rewards (Line 157). Rearranging the description will improve the clarity here. Without the assumption that the reward is a function of the end-effector position, there are plenty of cases where two actions can have identical rewards but vastly different returns. It is not a “redundancy” in such cases.\n \n  b. Line 302: “We provide more material …., especially on the real robot, on our project website” The project website shows a placeholder page only.\n\n  c. Line 211 reads “the code can be found on our project websites” but there’s no code there. \n\n\n3. Line 188 mentions L without defining it. The paper discusses two secondary loss functions and it’s hence not clear if L depends on the application or is a linear combination of both functions.\n\n4. The method requires knowing the location of the object/obstacle. In real-world scenarios, this information is generally not available. That brings a concern for the practicality of the proposed collision avoidance loss function.\n\n5. Typoes:\n\n    - “we want to minimize the distance between robot links and obstacles” : should be “maximize”\n\n    - The pseudoinverse formula on line 177 misses an inverse symbol.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I base my recommendation mainly on the novelty of the idea and the experiments. I find the idea to be novel enough but the issues in the experiments raise some concerns. I’m recommending weak rejection but am willing to up the score if some concerns can be addressed during the rebuttal.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Please refer to the items discussed in the weaknesses/suggestions section.\n\n**Edit**: addressed in the revision", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes an approach for learning safe manipulation policies through policy search. The main idea is to bias actions to increase value of secondary (safety) objectives while preventing a reduction of immediate reward. That is, the goal is to perform the task according to the original objective while increasing safe and more natural looking behavior.

Pros:
- Interesting idea of maintaining performance w.r.t. the primary objective while increasing safety during robotic manipulation
- Experiments in simulation and on a real robot
- Well written paper
- Safety is important in robot learning applications

Cons:
- Experimental evaluation should be complemented with more seeds and ablation studies
- Performance issue mentioned by reviewer 9a3E in Figure 2 should be explained. Other problems in experimental evaluation mentioned by reviewer g7Ri should be discussed.

Update: The authors were able to clarify issues such as the purpose of the secondary objective further. Most issues have been resolved and reviewers agree on acceptance.

Decision: Accept (Poster)