Learning to Grasp the Ungraspable with Emergent Extrinsic Dexterity
Authors: Wenxuan Zhou
David Held
Venue: CoRL 2022 Oral (2022)
Keywords: Manipulation
Reinforcement Learning
Extrinsic Dexterity

Abstract: A simple gripper can solve more complex manipulation tasks if it can utilize the external environment such as pushing the object against the table or a vertical wall, known as "Extrinsic Dexterity." Previous work in extrinsic dexterity usually has careful assumptions about contacts which impose restrictions on robot design, robot motions, and the variations of the physical parameters. In this work, we develop a system based on reinforcement learning (RL) to address these limitations. We study the task of "Occluded Grasping" which aims to grasp the object in configurations that are initially occluded; the robot needs to move the object into a configuration from which these grasps can be achieved. We present a system with model-free RL that successfully achieves this task using a simple gripper with extrinsic dexterity. The policy learns emergent behaviors of pushing the object against the wall to rotate and then grasp it without additional reward terms on extrinsic dexterity. We discuss important components of the system including the design of the RL problem, multi-grasp training and selection, and policy generalization with automatic curriculum. Most importantly, the policy trained in simulation is zero-shot transferred to a physical robot. It demonstrates dynamic and contact-rich motions with a simple gripper that generalizes across objects with various size, density, surface friction, and shape with a 78% success rate. 

Learning to Grasp the Ungraspable with Emergent Extrinsic Dexterity
Wenxuan Zhou Roboties Institute
David Held Robotics Institute
Carnegie Mellon University
Carnegie Mellon University dheld@andrew. cmu, edu
wenxuanz@andrex .om.
cmu. edu
simple gripper can solve more complex manipulation tasks if it utilize the external environment such as pushing the object against the table or a vertical wall, known as "Extrinsic Dexterity." Previous work extrinsic dexterity usually has careful assumptions about contacts which impose restrictions on robot design, robot motions, and the variations of the physical parameters. In this work, we develop system based on reinforcement learning (RL) to address these limi- tations. We study the task of "Occluded Grasping" which aims grasp the object in configurations that are initially occluded; the robot needs to move the object into a configuration from which these grasps can be achieved. We present a sys- tem with model-free RI that successfully achieves this task using simple gripper with extrinsic dexterity. The policy learns emergent behaviors pushing the ob- ject against the wall to rotate and then grasp it without additional reward terms on extrinsic dexterity. We discuss important components of the system includ- ing the design of the RL problem, multi-grasp training and selection, and policy generalization with automatic curriculum. Most importantly, the policy trained simulation is zero-shot transferred to a physical robot. It demonstrates dynamic and contact-rich motions with simple gripper that generalizes across objects with various size, density, surface friction, and shape with a 78% success rate. Videos can be found at https: / /sstts/t../////////////////eeee/ .owgle wwwwwciw//ccccecc. har///rappuuue ungraspable.
Abstract:
Keywords: Manipulation, Reinforcement Learning, Extrinsic Dexterity
Introduction
Humans have dexterous multi-fingered hands; however, similarly dexterous robot hands are ex- pensive and fragile. Instead, simple hand can achieve dexterous manipulation by leveraging the environment, known as "Extrinsic Dexterity" [1]. For example, a simple gripper can rotate an object in-hand by pushing it against the table [2], or lifting an object by sliding it along vertical sur- face [3]. With the exploitation of external resources such as contact surfaces or gravity, even simple grippers can perform skillful maneuvers that are typically studied with a multi-fingered dexterous hand. Different from a common practice of considering the robot and an object of interest in isola- tion, extrinsic dexterity focuses on a holistic view of considering the interactions among the robot, the object, and the external environment.
Previous work in extrinsic dexterity has demonstrated variety of tasks such in in-hand reorientation with a simple gripper, prehensile pushing, or shared grasping [1, 2, 3]. However, the underlying approaches come with several limitations such relying on hand-designed motions or primitives with limited capability of generalizing across different objects, making assumptions about contact locations and contact modes, or requiring specific gripper design [1, 2, 3, 4, 6, 7]. Instead, we build system with reinforcement learning (RL) to remove these limitations. With RL, the agent can learn closed-loop policy of how the robot should interact with the object and the environment to solve the task, taking into account both planning and control. In addition, when trained with domain randomization, the policy can learn to be robust to different variations of physics. These properties of RL can enable extrinsic dexterity in more general setting.
In this work, we study "Occluded Grasping"
example of task that requires extrinsic dexterity. The goal of this task is to grasp an object in poses that are initially occluded. Consider, for example, robot that needs to grasp cereal box lying on its side table; the desired grasp is not reachable
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

We present system for the "Occluded Grasping" task with extrinsic dexterity. The goal of this task is to reach occluded grasp configuration (indicated by transparent gripper). The figure shows an example of the emergent behavior the policy and successful sim2real transfer.
Figure I:
because it is partially occluded by the table (Figure 1). To achieve this grasp with parallel gripper, the robot might rotate the object by pushing against vertical wall expose the desired grasp and then reach it. This task is in contrast with common grasping tasks which focus on reaching an unoceluded grasp in free space with a static or near-static scene [8, 9, 10].
The goal of this work is to build a system for the "Oecluded Grasping" task as an example of the physical robot. We investigate design
combination of RL and extrinsic dexterity that works choices of such a system and emphasize the simplicity of the method. With model-free RI., we design reward function that optimizes pre-grasp and grasping motion without the separation of stages as previous work in pre-grasp [11, 12, 131. By placing the object in the bin and using a com- pliant low-level controller, the agent shows emergent extrinsic dexterity behavior without additional reward terms. We also incorporate a set of desired grasps with a training curriculum and grasp selection procedure during evaluation. We improve the policy with Automatic Domain Random- ization [14] over physical parameters which robustify the contact-rich behaviors across noise and environment variations. In the experiments, we provide comprehensive evaluation of the system in simulation to analyze the importance of each component. The policy is zero-shot transferred the physical robot and successfully executes similar behaviors to complete the task. The policy achieves a success rate of 78% and shows generalization across various out-o--distribution objects. Our main contribution the real robot experiment results. Existing work with simple hand has not shown such behaviors on the real robot with similar level of complexity in contact events and generalization across objects at the same time.
2 Related Work
2.1 Extrinsic dexterity
"Extrinsic dexterity" is type of manipulation skill that enhances the intrinsic capability of a robot hand using external resources including external contacts, gravity, dynamic motions of the arm [1]. Previous work in extrinsic dexterity has demonstrated complex tasks with a simple grip- per including in-hand reorientation [1, 4], prehensile pushing [2, 5]. shared grasping [3], and more. Their methods are based on hand-crafted trajectories [1]. task-specific motion primitives [3, 4, 15), or planning over contact modes [2, 6. 7] simplify the problem. They relies on careful assump- tions on contacts such as assuming a fixed number of sticking contacts between the fingertips and the object. In this work, we take alternative approach to RL to learn closed-loop policy that considers both planning and control without limitations on contact events. The resulting policy shows more versatile contact switches beyond prior work and can be transferred to a physical robot.
2.2 Grasping
Grasping is an important task in robot manipulation and has been studied from various aspects.
Grasp generation: One area of study grasping is to generate stable grasps using analytical ap- proaches [16. 17] or learning approaches [8, 9, 18, 19, 20]. In paper, assume that the desired grasp configurations are given which may come from any grasp generation method.
To execute grasp following grasp generation, motion planner is usually used to generate a collision-free path towards the desired grasp configuration [10, 21, 22, 23). All of these works aim at achieving the unoccluded grasp configurations in static near-static scenes. Instead,
Grasp execution:

---

our work focuses on complementary direction of achieving occluded grasp locations by interacting with the object of interest. Another line of work in grasping uses an end-to-end pipeline without the separation of grasp generation and grasp execution [24, 25]. However, they do not demonstrate performing the occluded grasps studied in this work.
Pre-Grasp manipulation: To deal with occluded grasps, prior work has studied pre-grasps as a preparatory stage of the grasping task. Typical motions for pre-grasps include rotation through planar pushing [12], sliding the object to the edge of the table [13, 26], or rotate the object against the wall [11]. Sun et al. [11] is the most related to our work, but they use specially designed end- effector to perform the pre-grasp motion and then use second gripper to grasp. We demonstrate that the full grasping task can be solved with a single gripper without special requirements on the end-effectors. These previous work typically separates pre-grasp motion and grasp execution into two stages and impose restrictions on the transitions of the stages. Instead, we co-optimize pre-grasp and grasp execution throughout the episode without explicit separation of the stages.
2.3 Reinforcement Learning for Manipulation
Previous work in RL for manipulation usually treats the object and the robot in isolation from the environment without considering extrinsic dexterity. RL has been applied to dexterous manipulation with multi--inggeed hand and shows contact-rich behaviors [14, 27. 28]. In contrast, with parallel gripper, prior work focuses on tasks with limited contacts and object motions without utilizing the environment [29, 30]. This is the first work that demonstrates extrinsic dexterity with a simple parallel gripper using RL.
3 Task Definition: Occluded Grasping
The goal of common grasp execution task is to move the end-effector E' close to a given de- sired grasp pose g. The desired grasp might come from any grasp generation method [8, 9, 20] as the input to the grasp execution task. As shown in Figure 2, we define an "Occluded Grasping" task to be a subset of the grasp ex- ecution tasks where the input desired grasp is initially occluded. To clarify, the term "oc- cluded" in this work is more than visual occlu- sion. It means the desired grasp intersects with the table and moving the gripper in free space cannot solve this task. The robot has interact with the object to make the grasp pose reachable. moves with the object. Formally, the grasp execution is defined to be successful if the position dif- ference (g E) and the orientation difference (g E') are less than the pre-defined thresholds er and eg respectively at the end of an episode. After successfully reaching a desired grasp pose, the gripper will be closed to complete the grasp. In addition, when the input to the system is a set of grasps G = instead of single grasp, the agent may select any of the grasp to approach to.
Figure 2: Notations: WE denotes the pose of the end-effector in the world frame W the target grasp in the object frame ي Six marker locations m, in green on the target grasp are used to calculate the occlusion penalty.
denotes
The grasp Og is defined in the object frame O and
4 Learning dexterous grasping with Reinforcement Learning
We build system that learns a closed-loop policy for the occluded grasping task defined above with model-free RL. In this section, we will discuss important design choices of the system including the design of the RL problem, the extrinsic environment, and the choice of low-level controller. Then we will discuss how to deal with a set of grasps by training with a grasp curriculum and selecting the best grasp during evaluation. We also include Automatic Domain Randomization [14] to improving the generalization of the policy across environment variations.
4.1 Preliminaries: Goal-conditioned Reinforcement Learning
We define a Markov Decision Process (MDP) with states 8t S, actions a; A, reward function r: S x AA- > R, and discount factor 7, The state space, action space and the reward function for our task will be discussed in detail in the next section. The goal is to find policy r(ar(st) that maximizes the return R₃- E * at). A Q-function is defined to be the expected return
3

---

End-etfector Delta Movement &e
Desired Hndd -effector Pose Eg
Target grasp
Operational Space Controller
Joint torque command (100R))
Observations:
Check Joint
End-effictor pase 'E
Policy (MLP)
(2Ho)
Objoct pose "O
End effecteo pose the object frume 'F
Figure 3: Outline of policy execution: Given the observations, the policy outputs end-effector delta movement (Section 4.2) to the low-level controller (Section 4.4).
of the policy (=) En [ ar]. In goal-conditioned RL, we define set of goals E G correspond to the reward function (st, ae) [31]. To train a policy with a set of goals, both the policy and the Q-function will now take the goal 17 as input, given by [aulss, (sc, 77) and QF (st. c Qe; 17)- In the occluded grasping task, we use the desired grasps as goals.
4.2 RL Problem Design
Observation and Action Space: The observation that is input to the policy includes target grasp configuration in the object frame 9g. the pose of the end-effector in the world frame W E and the grasp Og input but
object pose in the world frame WO. Note that the policy only takes will discuss how to deal with a set of grasps in Section 4.5. For real robot experiments, we use Iterative Closest Point (ICP) for pose estimation of the object which matches template point cloud of the object to the current point cloud [32]. The action space of the policy is the delta pose of the end-effector AF in its local frame which passed into low-level controller (Section 4.4). An outline of the policy execution shown in Figure 3. More details can be found in Appendix B.
Reward: We design the reward function to optimize the pre-grasp motion and grasp execution without separating them into two stages as in previous work [11, 13, 12, 26]:
r= aD,g P((.))
(la)
D(g, E) a1 AT a2 E)
(Ib)
where an, 02 and / are the weights for the reward terms. The first term of Equation 1a, D(g, FF), is the pose difference between the target grasp and the current end-effector pose, which is to optimize for reaching the grasp. This term is expanded in Equation lb to include the translational and rotational distance, as described in Section 3. The second term of Equation la is the target grasp occlusion penalty which is to penalize the agent if the target grasp configuration is in collision with the table. This corresponds to pre-grasp objective. To measure how much the target grasp occluded by the table, we set six marker points on the target gripper (Figure 2) denoted as m; and compare the height of the markers with the table top. If a marker is below the table top, the height difference will be used as the penalty. Including this occlusion penalty can effectively reduce the local optima where the gripper will reach close to the target grasp (which is occluded) without trying to move the object. Note that we did not impose any reward terms that are explicitly related to extrinsic dexterity. In our system, the use of extrinsic dexterity is an emergent behavior of policy optimization given our objective and environmental setup.
4.3 Extrinsic Environment
To exploit the benefits of extrinsic dexterity from object-scene interaction in this task, we construct the scene as having an object in a bin, instead of leaving the object on the table as shown in Figure 2. We also make the workspace of the robot large enough such that the robot can move the object to make contacts with the walls (during which the robot itself may also make contact with the wall). In the experiments, we will show that the policy will learn to utilize the wall to rotate the object. Without the wall, is not able to find a strategy that can perform the task with the parallel gripper.
4.4 Choice of Low-level Controller
The choice of low-level controller is important for this task due to the fact that we expect the agent to use extrinsic dexterity which involves rich contacts among the gripper, the object and the bin. We choose Operation Space Control (OSC) as the lower-level controller to execute the policy output which operates at a higher frequency (100Hz) than the RL policy (2Hz) [33] (Figure 3). Given a desired pose of the end-effector, OSC first calculates the corresponding force and torque at the end- effector to minimize the pose error according to a PD controller with gain Kp and Kg, Then, the desired force and torque of the end-effector will be converted into desired joint torques according to

---

the model of the robot. We choose relatively low gains so that the controller becomes compliant in the end-effector space. There are two benefits of compliant OSC in such a contact-rich manipula- tion task with extrinsic dexterity. First, being compliant in end-effector space allows safe execution of the motions without smashing the gripper on the objects or the bin. Limiting the delta pose and selecting proper gains Kp. Ka will limit the final force and torque output of the end-effector. If we use a controller that is compliant in the joint configuration space instead, we will not have direct control over the maximum force the end-effector might have on the object and the bin. Second, as shown in Martin-Martin et al. [34], using OSC as the low-level controller might speed up RL training and improve sim2real transfer for contact-rich manipulation.
4.5 Multi-grasp Training and Grasp Selection
In this section, we consider the scenario in which a set of desired grasp configurations are given instead of just a single one. During training, given a set of grasps Gtrain- we aim at covering as many grasp configurations as possible. As we will show in the experiments, reaching different grasps might require a significantly different behavior. Learning directly over diverse set of goals might create difficulties for policy learning [35, 36]. We use automatic curriculum following OpenAI et al. [14] to gradually expand the set of grasps to be trained with. We start the training with just single fixed grasp; after the policy has achieved success rate larger than a threshold, it will be trained on a slightly larger set with grasps close to the initial grasp location.
During testing, if a set of grasps Gtest is provided, we can select the best grasp within the set to improve the performance of the grasping task, following previous work in integrated grasp and motion planning [10, 21, 22]. With value-based model-free RL algorithms such as Soft Actor Critic [37]. the policy is trained together with a Q-function (defined in Section 4.1). We propose to select the grasp that maximizes the learned Q-function for the given observation and action: g" = arg mäxg~GGrere Q(s:, 4t, 9) The selection can be performed for each timestep L, or at the be- ginning of an episode when / = tu- We include both implementations in the experiments. This can select the grasp that is most easily reached which depends both on the environmental configuration as well as how well the policy has learned to achieve different grasp configurations.
4.6 Improving Policy Generalization
To improve generalization across environment variations, we train the policy with Automatic Do- main Randomization (ADR) [14]. Similar to the multi-grasp curriculum, the policy first trained on single environment with single object, and gradually expands the range of randomization au- tomatically according to its performance. This significantly reduces the effort of tuning the range of domain randomization. We randomize over different variations of the environment properties such as object size, density, and friction coefficient. We also randomize the parameters of the controller to improve sim2real transfer. More descriptions the ADR procedure can be found in Appendix C.
Experiments
We build the simulation environment for this task using Robosuite [38] and the MuJoCo simula- tor [39] as shown in Figures I and 2. The environment contains a Franka Emika Panda robot with a parallel gripper and an object in the bin in front of it. We focus on grasping large flat objects from the side since they cannot be grasped with a top-down motion. The policy is trained in simulation with Soft Actor Critic [37]. In this section, we include the results simulation to discuss each com- ponent of the proposed system. We then evaluate zero-shot sim2real transfer on a physical Panda robot across different objects. Implementation details can be found in Appendix B.
5.1 Training Curves and Ablations
Durs Sperse Reward
w/o Penalty
OSC (Ours)
Joint Position High-gain OSC
We first evaluate our method by train- ing the policies with a single de- sired grasp in the default environ- ment without ADR. Figure 4a shows the training curve of the proposed method and the ablations. The policy trained with the complete system can reach success rate of 100% before 4000 episodes which corresponds to 160000 environment steps. To evalu-
w/o Walls
Jeint Torque
1.00
1.00
Success
1000 2000 2000 4000 Episndes
5000
2000 3000 4000 5000
Figure 4: Training curves and ablations: (a) ablations on the reward function and the wall. (b) ablations on the controller.
5

---

(a) Local optima: An example of local optima where the gripper uses the bottom finger lift the object (instead of top) and then fails to move the object between its two fingers to prepare for the grasp.
(b) Standing object: One of the successful strategies is flip the object until it stands the side and then reach the grasp.
(c) MultiGrasp-Front When the desired grasp is at the corner the policy flips the object by pushing on the side and then move close to the grasp.
(d) MultiGrasp-Side: When the grasp side, the policy can use another side of the wall to rotate the object and reach the desired grasp.
Figure Visualizations of the policies in different scenarios.
ate the importance of extrinsic dexterity, we remove the walls of the bin. The resulting policies have 0% success rate and push the object outside of the table. For ablations on the reward function, remove the occlusion penalty (the second term of Equation la) and also try a sparse reward. Without local optima (an example shown in
the occlusion penalty, the policy is more likely to get stuck Figure 5a) and thus the success rate becomes lower. With the alternative of {-1,0) sparse reward, the policy learns much slower. We also experiment with different low-level controllers. Both joint torque and joint position control lead to worse performance which indicates the importance of using end-effector coordinates. With less compliant controller by increasing the gains of OSC, the suc- cess rate becomes lower which demonstrates the importance of compliance for contact-rich tasks in addition to the safety considerations.
5.2 Emergent Behaviors
Figure shows typical strategy of successful policy which involves multiple stages of contact switches. The gripper first moves close to the object and makes contact on the side of the object with the top finger. It then pushes the object against the wall rotate it. During this stage, the gripper usually maintains fixed or rolling contact with the object, but sliding also occurs. The object might have sliding sticking contacts with the wall and the ground. After the gripper has rotated bit further and the bottom fingertip is below the object, the gripper will let the object drop on the bottom finger. After that, the gripper will try match the desired pose more precisely. At this point, the policy has executed the grasp successfully and it is ready to close the gripper. This type of learned contact-rich behaviors with simple gripper has not been shown previous work. In Section 5.5, we will further demonstrate that it can be transferred to a physical robot.
One of the key decisions in this strategy to use the top finger to rotate the object instead of the bottom finger. One might suppose an alternative approach which is to use the bottom finger to scoop the object against the wall and then directly roll the finger underneath the object to reach the grasp. However, this strategy is not physically feasible on the parallel gripper due to the limited degree of freedom of the finger. We observe that the policies that follow this strategy during exploration usually get stuck at local optima without successfully reaching the grasp (Figure 5a). Another type of successful strategy is to flip the object to stand on side and then move to the grasp (Figure 5b). This strategy relies on the fact that the object remains stable after the rotation. We will show in the real-robot experiments that for a non-box object, the object may lie on the wall to maintain stability.
5.3 Multi-grasp Experiments
Multi-grasp Training: Going beyond a single desired grasp, we generate the grasp configurations around the side of the object and parameterized the grasps into a continuous grasp ID in the range of [0, 4) (Figure 6). We train two types of multi-grasp policies with curriculum: MultiGrasp -rontt which starts from grasp ID=1.5 and MultiGrasp-Side which starts from grasp ID=2.5. As bascline,

---

Crasp-Side
MustiGrasp
MultiGrasp
Front
Side
1,00
Argmax
1.00 = 0.00
1.00 = 0.00
ArgmaxQ-to
1.00 - 0.00
1.00 1 0.00
PoseDiff
1.00 -: 0.00
0.96 1 0.08
PoseDiff-to
1.00 0.00
0.50 + 0.43
0.5
Uniform
0.54 0.16
0.90 + 0.06
0.00
Table Comparison of grasp selection
Grasp conigurge on
methods: Side grasp policies achieve better performance when using the Q-
Figure 6: Left: Grasp configurations. Right: Multi- Grasp Training results with and without curriculum.
function to select the grasp.
we train a policy by uniformly sampling from the entire set of grasps without curriculum (No cur- riculum). Figure shows the performance of these policies evaluated across all grasp IDs. Without curriculum, the agent has difficulties in reaching any of the grasps. With the automatic curriculum, both MulliGrasp-Fromm and MudiGrrap-SSdde expand from a single grasp most of the grasps on one side of the object. Figures 13a and 13b include qualitative examples of the behaviors which shows that it may require completely different behavior for different grasps.
Grasp Selection: We compare grasp selection methods with MultiGrasp- Front and MultiGrasp- Side. We sample 50 grasps from the training range of the policy the beginning of each episode. The grasp selection methods will choose a grasp from this set as the input to the policy. We evaluate the following grasp selection options: ArgmaxQ selects the grasp that corresponds to the highest Q-value. PoseDiff selects the grasp according to the closest distance to the current gripper pose according to Equation lb (with the same weights the reward function). Both ArgmaxQ and ArgmaxQ select grasp for each timestep. Alternatively, ArgmaxQ-lo and PoseDiff-lo only selects a grasp during the first timestep of the episode. Uniform samples grasp from the set uniformly. The results are summarized in Table For MultiGrasp-Side using the Q-function for grasp selection better than the other approaches. Since the policy has a more complicated maneuver reach the side (Figure 13b), the Q-function can capture the difficulty of the goal better than pose difference.
5,4 Policy Generalization
In this section, we evaluate the generalization the policy across environment variations: open loop trajectories (Open Loop), policies trained over fixed environment (Fixed Env) and policies trained with ADR (With ADR). The open loop trajectories are obtained by rolling out the Fixed Env policies in the default environment. We sam- ple 100 environments from the range covered by the ADR policies (Appendix C) and plot the per- centage of environments that are above certain performance metric (Figure 7). The closed-loop policies are much better than open-looy trajectories. even further. Sensitivity analysis on single physical parameters can be found in Appendix A.
Env
Open Loop
1.0
Hinltewarr Mean
-2.
0.6
-7.5
-10.0
0.2
-12.5
0.0 0.00
0.75
0.00
0.5
0.75
Percentage of Environments
Percentage Environments
Figure 7: Evaluation the generalization of the policies by sampling 100 environments.
With ADR, the generalization can be improved
5.5 Real-robot experiments
We execute the single grasp policies on the real robot with zero-shot sim2real transfer over 10 test cases with different dimensions, densities, surface frictions, and sizes shown in Figure 8. For non-box objects, the poses are defined with respect to the bounding boxes. The bounding boxes are obtained by running Principle Component Analysis (PCA) on the scanned object point cloud. More details of the real robot experiments can be found in Appendix D. Note that most of the objects are dutttf-disttibution. We evaluate 10 episodes for each test case and summarize the results in Figure 8. The success is measured by being able to close the gripper and lift the object at the end of the episode. We first compare the policies with and without Automatic Domain Randomization, denoted as w ADR and w/o ADR respectively. Quantitatively, the policy with ADR achieves success rate of 78% while the policy without ADR achieves 33%. Interestingly, the policy with ADR achieves 24/30 successes over the bottle, the Cool Whip container, and the container with reversed
7

---

Success w/o ADR
Success
Success
Object-ID
w/ ADR
Init-pose
Box-O (128g)
9/10
9/10
7/10
Box-0 (237g)
6/10
10/10
8/10
Box-O (345g)
3/10
4/10
4/10
Box-1
5/10
8/10
8/10
Box-0
Box-1
Box- 2
Box-3 3
Box-2
2/10
9/10
9/10
Box-3
0/10
7/10
9/10
Toy Bag
7/10
7/10
9/10
Bottle
W/10
1/10
Contuiner
0/10
10/10
1/10
Containcr-rev
0/10
6/10
0/10
Average
33%
78%
56%
Toy bag
Bottle
Container
Contaiine-reverse
Figure 8: We evaluate the policy on the real robot with various test objects. The policy trained in
simulation box-shape objects generalize to the real robot and other shapes. With ADR, the policy achieves 45% better success rate.
initial pose. This demonstrates that although the policy is only trained with boxes in simulation, also generalize other shapes box. However, when the object with an out-of-distribution shape has a very different transition dynamics, the policy could fail. Qualitatively, both policies being evaluated exhibit similar strategies as discussed in Section 5.2. In fact, a single policy network may execute either the dropping strategy (Figure the standing strategy (Figure 5b) depending on the current state. We also include additional results when the initial object location is not close to the wall, denoted as Init-pose in Figure 8. We finetune the w/ ADR policy expand further over the range of initial object locations. The success rate remains similar for most objects, but this setting becomes more challenging for non- box shapes. Videos of the full real robot evaluation including failure cases and recovery behaviors can be found on the website These real robot results are valuable to the field of manipulation because is beyond what has been shown with a simple hand considering the combined complexity of contact events, object motion and object generalization.
extent when represent the object with its bounding
6 Limitations
One limitation of this work is that the policy is trained with box-shape objects. Although it may generalize to other shapes some extent as shown in the experiments, the policy might be improved by including other shapes during training. In addition, the pose of the object alone may not be sufficient generalize to novel objects; using better representation of the shape such as a point cloud or key-points could improve generalization across shapes. However, these changes would also increase the training complexity. Another limitation is that we assume a reasonably accurate robot and gripper model, in terms of geometries, kinematic and dynamic parameters. It would be interesting to explore how to extend the method to transfer across robots and grippers.
7 Conclusion and Takeaways
In this work, we study the "Occluded Grasping" task where the robot with a parallel gripper aims to reach a grasp configuration using extrinsic dexterity. We present system that learns a closed-loop policy for this task with reinforcement learning. In the experiments, we demonstrate the importance of each component of the system. We also show that the policies and generalize to various objects. One potential extension of our work is to train the policy with a wide variety of object shapes which may require image-based or point cloud-based policies. Also, the pipeline can potentially be extended to other extrinsic dexterity tasks.
be executed the real robot
Despite the simplicity of the proposed method, we would like to emphasize the following takeaways from this work: First, we provide a concrete example that a simple gripper can do much more than pick-and-place while being cheaper and easier to maintain than a dexterous hand, following previous work in extrinsic dexterity. We envision more future work this direction in manipulation. Second, RL can be good option to generate policies with emergent extrinsic dexterity, and sim2real transfer works reasonably well with our proposed system. Our work takes a step towards deploying contact- rich policies with simple gripper in the real world.
'nttps:/ssites. http::///ttaaooooooooc////////ap
8

---

Acknowledgments
This material is based upon work supported by the National Science Foundation under Grant No. IIS-1849154 and LG Electronics. We thank Daniel Seita, Thomas Weng, Tao Chen, Homanga Bharadhwaj, and Chris Paxton for the valuable feedback.
References
[1] N. C. Dafle, A. Rodriguez, R. Paolini, B. Tang, S. S. Srinivasa, M. Erdmann, M. T. Mason, Lundberg, H. Staab, and T. Fuhlbrigge. Extrinsic dexterity: In-hand manipulation with external forces. In 2014 IEEE International Conference on Robotics and Automation (ICRA), pages 1578-1585. IEEE, 2014.
[2] N. Chavan-Dafle and A. Rodriguez. Sampling-based planning of in-hand manipulation with external pushes, 2017.
[3] Y. Hou, Z. Jia, and M. Mason. Manipulation with shared grasping. In Robotics: Science and Systems, 2020.
[4] Y. Hou, Z. Jía, and M. T. Mason. Fast planning for 3d any-pose-reorienting using pivoting. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 1631-1638, 2018.
[5] N. Chavan-Dafle, R. Holladay, and A. Rodriguez. In-hand manipulation via motion cones, 2019.
[6] X. Cheng, E. Huang. Y. Hou, and M. T. Mason. Contact mode guided sampling- based plan- ning for quasistatic dexterous manipulation in 2d. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 6520-6526. IEEE, 2021.
[7] X. Cheng, E. Huang, Hou, and M. T. Mason. Contact mode guided motion planning for quasidynamic dexterous manipulation in 3d. arXiv preprint arXiv:2105.1444: 2021.
[8] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In International Conference on Computer Vision (ICCV), 2019,
[9] A. Murali, A. Mousavian, C. Eppner, C. Paxton, and D. Fox. 6-dof grasping for target-driven object manipulation in clutter, 2020.
[10] L. Wang, Y. Xiang, and D. Fox. Manipulation trajectory optimization with online grasp syn- thesis and selection. In Robotics: Science and Systems (RSS), 2020.
[11] Z. Sun, K. Yuan, W. Hu, C. Yang, and Z. Li. Learning pregrasp manipulation of objects from ungraspable poses, 2020.
[12] L. Chang, S. S. Srinivasa, and N. Pollard. Planning pre-grasp manipulation for transport tasks. In 2010 IEEE International Conference on Robotics and Automation, pages 2697-2704, IEEE, 2010.
[13] K. Hang. A. S. Morgan, and A. M. Dollar. Pre-grasp sliding manipulation of thin objects using soft, compliant, underactuated hands. IEEE Robotics and Automation Letters, 4(2): 662-669, 2019. 20..110099AAAA211..9922222.
[14] OpenAI, I. Akkaya, M. Andrychowicz, M. Chociej. M. Litwin, B. McGrew, A. Petron, A. Paino, M. Plappert, G. Powell, Ribas, Schneider, N. Tezak, J. Tworek, P. Welinder, L. Weng, Q. Yuan, W. Zaremba, and L. Zhang. Solving rubik's s cube with robot hand, 2019.
[15] C. Eppner and O. Brock. Visual detection of opportunities exploit contact in grasping using contextual multi-armed bandits. In 20/7 IEEE/RS International Conference on Intelligent Robots and Systems (IROS), pages 273-278, 2017. doi:10.1 L09/TROS.001..8000
[16] K. B. Shimoga. Robot grasp synthesis algorithms: survey. The International Journal of Robotics Research, 15(3):33--266, 1996.

---

[17] V.-D. Nguyen. Constructing force-closure grasps. The International Journal of Robotics Re- search, 733)3-16, 1988.
[18] L. Pinto and A. Gupta. Supersizing self-supervision: Learning to grasp from 50k tries and 700 robot hours. In 2016 IEEE international conference robotics and automation (ICRA), pages 3406-3413. IEFE, 2016.
[19] J. Bohg, A. Morales, T. Asfour, and D. Kragic, Data-driven grasp synthesis-a survey, IEEE Transactions on Robotics, 3022)289-309, 2013.
[20] A. Murali, W. Liu, K. Marino, S. Chernova, and A. Gupta. Same object, different grasps: Data and semantic knowledge for task-oriented grasping, 2020.
[21] N. Vahrenkamp, M. Do, T. Asfour, and R. Dillmann. Integrated grasp and motion planning- In 2010 IEEE International Conference on Robotics and Automation, pages 2883-2888, 2010. doi:10.1 1111110000000000000000000
[22] Fontanals, B.-A. Dang- Vu, O. Porges, Rosell, and M. A. Roa. Integrated grasp and motion planning using independent contact regions. In 20/4 IEEE-RAS International Conference on Humanoid Robots, pages 887-893, 2014. doi:10. 109/HUMANOIDS
10014.7044444
[23] L. Wang. Y. Xiang, W. Yang. A. Mousavian, and D. Fox. Boal-auxiliary actor-critic for 6d robotic grasping with point clouds, 2021.
[24] D. Kalashnikov, A. Irpan, P. Pastor, J. Ibarz, A. Herzog, E. Jang, D. Quillen, E. Holly, M. Kalakrishnan, V. Vanhoucke, and S. Levine. Qt-opt: Scalable deep reinforcement learning for vision-based robotic manipulation. 2018.
[25] S. Song, A. Zeng. J. Lee, and T. Funkhouser. Grasping in the wild: Learning 6dof closed-loop grasping from low-cost demonstrations. Robotics and Automation Letters, 2020.
[26] J. King, M. Klingensmith, C. Dellin, M. Dogar, P. Velagapudi, N. Pollard, and S. Srinivasa. Pregrasp manipulation as trajectory optimization. In Proceedings of Robotics: Science and Systems, Berlin, Germany, June 2013. doi: 150.0./555........ 15607/RSS.2013....0155.. 0005.
[27] Chen, J. Xu, and Agrawal. A system for general in-hand object re-orientation. Conference on Robot Learning. 2021.
[28] A. Nagabandi, K. Konolige, S. Levine, and V. Kumar. Deep dynamics models for learning dexterous manipulation. In Conference on Robot Learning, pages 1101-1112. PMLR, 2020.
[29] A. X. Lee, C. M. Devin, Y. Zhou, T. Lampe, K. Bousmalis, J. T. Springenberg. A. Byravan, A. Abdolmaleki, N. Gileadi, D. Khosid, et al. Beyond alllaee-- Tackling robotic stack- ing of diverse shapes. In 5th Annual Conference on Robot Learning, 2021.
[30] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P. Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world, 2017.
[31] T. Schaul, D. Horgan, K. Gregor, and D. Silver. Universal value function approximators. In International conference on machine learning, pages 1312-1320. PMLR, 2015.
[32] S. Rusinkiewicz and M. Levoy. Efficient variants of the icp algorithm. In Proceedings third international conference on 3-D digital imaging and modeling, pages 145-152. IEEE, 2001.
[33] O. Khatib. A unified approach for motion and force control of robot manipulators: The opera- tional space formulation. IEEE Journal on Robotics and Automation, 3(1):43-53, 1987.
[34] R. Martín-Martin, M. A. Lee, R. Gardner, S. Savarese, J. Bohg, and A. Garg. Variable impedance control in end-effector space: An action space for reinforcement learning in contact-rich tasks. In 2019 IEEE/RSJ International Conference on Intelligent Robots and Sys- tems (IROS), pages 1010-1017. IEEE, 2019.
[35] D. Ghosh, A. Singh, A. Rajeswaran, Kumar, and S. Levine. Divide-and-conquer reinforce- ment learning, arXiv preprint (tf::1711.097777 2017.
10

---

[36] T. Yu, S. Kumar, A. Gupta, S. Levine, K. Hausman, and C. Finn. Gradient surgery for multi- task learning. arXiv preprint arXiv:2001.06782. 2020.
[37] Haarnoja, Zhou, P. Abbeel, and S. Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. In International conference machine learning, pages 1861-1870. PMLR, 2018.
[38] Y. Zhu, J. Wong, A. Mandlekar, and R. Martin-Martín. robosuite: A modular simulation framework and benchmark for robot learning. arXiv preprint (oXiv:2009.12293, 2020.
[39] E. Todorov, T. Erez, and Tassa. Mujoco: A physies engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 5026- 5033. IEEE, 2012.
[40] M. Andrychowicz, Wolski, A. Ray, J. Schneider, R. Fong, P. Welinder, McGrew, Tobin,
P. Abbeel, and W. Zaremba. Hindsight experience replay. arXiv preprint arXiv: 1707.01999 2017.
[41] K. Zhang, M. Sharma, J. Liang, and O. Kroemer. A modular robotic arm control stack for research:
and frankapy. arXiv preprint arXiv:2011.02398,, 2020.


Reviews:
[{"review_id": "9qScqRYj-Nv", "reviewer": "Reviewer_58SX", "summary": "The paper proposed an RL-based training framework to train a parallel jaw gripper to grasp objects in configurations that necessitate extrinsic dexterity.  Specifically, objects placed on a flat bin are considered. The reward function is designed such that extrinsic manipulations emerges from training without any dedicated reward terms. Detailed experiments demonstrate that a policy trained in simulation with domain randomization can generalize across different objects parameters, and transfers to a physical robot platform without the need for any additional training.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths  (in no particular order)\n\n- I thoroughly enjoyed reading this paper. It is expertly written and clearly organized. I consistently found myself asking a question and finding the answer in the very next paragraph. \n- The framework leverages a simple but powerful idea: A dense yet simple reward measuring progress when combined with good curriculum design and domain randomization will enable the robot to figure out that the external environment can be used to its benefit.\n- Design choices across the framework are justified and appropriately motivated with references to backup claims.\n- The trained policy enjoys zero-shot generalization to the real-world, thanks to Automatic Domain Randomization. Further, the objects used in the physical robot experiment were out of distribution.\n- The evaluation is thorough and the results conclusively demonstrate the capabilities of the proposed methods and provide insights into the need for different design elements. \n- The supplementary website and the videos are very helpful to understand the approach and the learned strategies. I appreciated the discussion of failure modes.\n\n### Questions and suggestions (in no particular order)\n\n- What would happen if the initial location of the object was not in contact with any wall? Will the robot learn to push the object in some direction until contact is made to establish support, or would it simply attempt to lift the object from where it is and fail? I would assume this depends on the training curriculum. But it might be worthwhile to discuss this.\n- The paper cites the requirement of a specific gripper design as a limitation of some existing methods. However, the proposed method is trained and tested on the same gripper. How would the proposed approach generalize across grippers or robots? Given that the paper leverage operational space control, it might not be far fetched to investigate if the policy trained on one platform will work on another with some tuning off the controller gains.\n- The experiments, while comprehensive and well done as stated above, left me to wonder why the proposed approach was not compared to some of the existing methods (at least the ones that rely on learning such as Refs. [27] and [28] and do not require considerable manual design).\n- How sensitive were the choice of weights $\\alpha$ and $\\beta$ in the reward function, and how much reward engineering was necessary?\n- Given that OSC requires a model of the robot, is perfect knowledge of the robot model assumed? This assumption is somewhat buried in the paper and would be better to make it more explicit.\n- The grasp selection method involves maximization of the learned Q-function. But it is not clear at what point in time this optimization is carried out. Is it at the beginning before the robot starts its approach? \n- Will the source code be made publicly available?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a simple and elegant learning framework that expands the capabilities of parallel jaw grippers by training policies that implicitly learn to extrinsic manipulation skills. The paper is well writing and the experiments are detailed and provided convincing evidence to support the claims.", "technical_quality": "Excellent: Very convincing and thorough.", "clarity_of_presentation": "Excellent: Enjoyable to read.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "issues": "Please respond to my questions under the \"Questions and Suggestions\" section above and feel free to push back on any of my comments if they exhibit misunderstanding on my part.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}, {"review_id": "3VGlqQ8oOCaU", "reviewer": "Reviewer_Deoh", "summary": "This paper proposes a sim-to-real reinforcement learning approach for grasping objects with external dexterity. The specific focus of this paper is to grasp objects lying at the bottom of shallow flat container, from a grasp pose which would cause the gripper body to collide with the container bottom. This is called an occluded grasp. The solution is to push and tip the object against one of the walls of the container, thereby removing the occlusion from the grasp. Experiments are performed in both simulation and on a real robot.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- Extrinsic dexterity is under-explored in robot learning. This paper presents a good reference point for environment and reward design and low level robot control.\n- The paper is well written and easily understandable.\n- Real robot experiments test the validity of simulation results.\n- The ablation studies showing the importance of domain randomization and robot control mode (operational space vs. joint space) are useful for readers wanting to draw conclusions about robot RL.\n\n## Weaknesses\n- The method depends on live accurate object tracking, because it is a policy observation. However, the paper lacks an analysis of the behaviour of the trained policy with noisy object pose estimates. Noise in different DOFs of the 6-DOF object pose might affect the policy performance differently. Such an analysis would help readers choose object tracking algorithms if they want to replicate this system for their task.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I am recommending to accept this paper because of the clear research question and well-executed experiments to answer it. The paper can be further strengthened by addressing the weakness pointed out above.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "Not necessary for acceptance, but the paper can be strengthened by including experiments with noisy object tracking.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (oral)