DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation
Authors: Yuzhe Qin
Binghao Huang
Zhao-Heng Yin
Hao Su
Xiaolong Wang
Venue: CoRL 2022 Poster (2022)
Keywords: Dexterous Manipulation
Policy Learning
Point Clouds
Sim-to-Real

Abstract: We propose a sim-to-real framework for dexterous manipulation which can generalize to new objects of the same category in the real world. The key of our framework is to train the manipulation policy with point cloud inputs and dexterous hands. We propose two new techniques to enable joint learning on multiple objects and sim-to-real generalization: (i) using imagined hand point clouds as augmented inputs; and (ii) designing novel contact-based rewards. We empirically evaluate our method using an Allegro Hand to grasp novel objects in both simulation and real world. To the best of our knowledge, this is the first policy learning-based framework that achieves such generalization results with dexterous hands. Our project page is available at https://yzqin.github.io/dexpoint.

DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation
Yuzhe Qin*!
Binghao Huang
Zhao-Heng Heng Yin
Hao Sul
Xiaolong Wang'
UC San Diego
2KUST
We propose sim-to--ral framework for dexterous manipulation which can generalize new objects of the category in the real world. The key of our framework is to train the manipulation policy with point cloud inputs and dexterous hands. We propose two techniques enable joint learning on multiple objects and sim-to-real generalization: (i) using imagined hand point clouds augmented inputs: and (ii) designing novel contact-based rewards. We empirically evaluate our method using an Allegro Hand to grasp novel objects in both simulation and real world. To the best of our knowledge, this is the first policy learning- based framework that achieves such generalization results with dexterous hands. Our project page is available at https://yzqin. .rrhinn..ttti github io/dexpoint.
Abstract:
Keywords: Dexterous Manipulation, Point Clouds, Sim-to-Real
Grasp Known Object
Train
Door Opening
Sim2Real Transfer
Grasp
Novel
Object
Test
Door
Opening
Figure 1: We introduce reinforcement learning method which takes the point cloud as input for two manip- ulation tasks; grasping and door opening. By introducing several techniques in the policy learning process, point cloud based policy trained purely in simulation can successfully generalize novel objects and transfer to real-world without any real-worls world data.
* indicates equal contribution.
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

Introduction
Dexterous manipulation has remained to be one of the most challenging problems in robotics [1]- While multi-finger hands create ample opportunities for robots to flexibly manipulate objects in our daily life, the nature of the high degree of freedom and high-dimensional action space creates significant optimization challenges for both search-based planning algorithms and policy learning algorithms. Recent efforts using model-free Reinforcement Learning have achieved encouraging results on complex manipulation tasks [1. 2]. However, it still faces many challenges in generalizing to diverse objects and being deployed on finger hand in the real-world.
For example, the dexterous manipulation framework proposed by OpenAl et al. [1] can solve in-hand manipulation of Rubik's Cube with RL and transfer to the real robot hand. However, the policy is only trained with one particular object and it is not able to generalize to diverse objects. To achieve cross-object generalization, recent efforts proposed to learn robust 3D point cloud representations [3, 4, 5, 6] with diverse objects using RL in simulation. While point cloud input has also been shown casier for Sim2Real transfer [7] given its focus on geometry instead of texture, the assumption on the access of complete object point clouds and ground truth states limit the transferability of above methods to the real robot deployment. Among these works, Chen et al. [3] showed that cross-object generalization is achievable in simulation without knowing the shape of the object to grasp, but its requirement of real-time access to the object states is itself a very challenging robot perception problem, especially under large occlusions during hand object interaction.
In this paper, we provide a sim-to-real reinforcement learning framework for generalizable dexterous manipulation, using two tasks with the Allegro Hand[8]: (i) object grasping where the test objects has not been seen during training: (ii) door opening where the test doors have levers of novel shape that has not been used in training. The tasks are visualized in Figure 1.
the robot needs first rotate the lever to unlock the door latch and then pull the lever in a circular motion to open the door.
We perform studies by training point cloud based reinforcement learning policy in the grasping and door opening task. With this approach, we list the three key discoveries of our framework for learning generalizable point cloud policy below:
(i) We justified that it is possible to achieve direct sim-to-real transfer for a dexterous manipulation policy with category-level generalizability when we use point cloud as the data representation.
(ii) Raw point clouds captured by sensors often come with heavy occlusions and noise: only a very small portion of the points from the observation are representing the robot fingers. We propose to imagine the complete robot finger point clouds according to the robot kinematic model and use them to augment the occluded real point cloud observations. We find that explicitly augmenting the input by imagined points can help achieve better robustness and sample efficiency for reinforcement learning.
(iii) Different from existing works that add contact information to the input of RI., design novel reward using contact pair information without adding contact to the observation. This practice remarkably improves sample efficiency as well as learning stability and avoids the dependency on contact sensor that is often unavailable for real robot models.
Related Work
Dexterous Manipulation. Dexterous manipulation aims enable robotic hands to achieve human- level dexterity for grasping and manipulating objects. Analytical methods have been proposed to adopt planning solve this problem [9, 10, 11, 12, 13, 14, 15J. However, they rely on detailed object models, which are not accessible when testing on unseen objects. To mitigate this issue, researchers propose a learning- based method for dexterous manipulation [16]. Some methods [17, 1, 1, 3, 5] take a reinforcement learning (RL) approach, while another line of work [18, 19, 20, 21] also proposes to learn from demonstration using imitation learning (IL.) to acquire the control policy. Recently, the combination of RL and IL [22, 23, 24, 25, 26, 4] has also shown encouraging dexterous manipulation results in simulation. However, most methods are either learning policy for one single object or assume access to object states produced by perfect detector which increases challenges to Sim2Real
2

---

transfer. In this paper, we surpass these limitations by introducing training on multiple objects and applying point cloud imputs for control.
Point Cloud in Robotic Manipulation. Point cloud representation has been widely applied in the robotics community. Researchers have studied matching the observed point clouds to object in grasping dataset [27] and executing the corresponding grasping action [28, 14]. For Saaming-based approaches, one line of research has focused on first estimating grasp proposals or affordance given the point cloud input and then planning accordingly for manipulation [29, 30, 31, 32, 33, 34, 35, 36]. While these methods are designed for parallel-jaw grippers, recent advancements have also been made for grasping with dexterous hands with similar approaches [37, 38, 39, 40J. However, this line of methods requires feedback from motion planning to estimate whether a grasp is plausible. Oftentimes, a stable grasp pose is proposed but it is not achievable by planning. To achieve efficient and flexible manipulation, Reinforcement Learning policy with point cloud inputs proposed [3, 5, 4], which allows the robot hand to flexibly adjust its pose while interacting with the object. However, these approaches still face challenges transferring to the real robot, given the noisy, occluded point clouds in the real world, and training RL policy with noisy point clouds increases the optimization difficulty. In this paper, we propose to use imagined point clouds, contact information, and multi-object training to combat these problems and achieve Sim2Real generalization.
Using Contact Information for Manipulation. Humans can manipulate objects purely from tactile sensing without seeing them. This biological fact inspires researchers to integrate contact and tactile information into the learning pipeline [41, 42, 43. 44, 45, 46, 47]. For example, both tactile and visual information inputs are combined together in [46] for decision making. The tactile informa- tion also utilized as inputs for RL-baaeed manipulation policies [48, 49, 50, 23, 51]. For example, visual-tactile sensor is used with an Allegro hand in simulator for playing piano [51]. Instead of using contacts as inputs, some methods also use contact and tactile information as reward to encour- age exploration and boost policy learning [49, 23]. Motivated by these works, we provide novel design of reward based on each contact link of the robot hand, which encourages more reasonable grasping behavior. Our design does not require real tactile sensor when training in simulation or deploying in the real robot.
3 Approach
Our objective is to train generalizable point cloud policy on a dexterous robot hand-arm system that is able grasp wide range of ob- jects or open an closed door with RL. We aims at Sim-to-Real transfer without any real-world training or data. During testing, the robot can only access the single-viewed point cloud and the robot proprioception data. As is discussed before, training such policy comes with nu- merous technical challenges, including reward design and imperfect point cloud information. In this work, we propose novel reward design technique based on contact and imagined point cloud model to deal with these challenges.
Trugitad Claad
Rahel Prephherpias
Figure 2: Real-experiment Setup: we use an Alle- gro Hand attached an XArmó and a RealSense D435 camera facing forward the robot.
Preliminaries: We model the dexterous manipulation problem as a Partially Observable Markov Decision Process (POMDP) M = S, A, R, T,U). Here, O is the observation space, S is the underlying state space, A is the action space. R is the reward function, T is the transition dynamics, and 14 generates agent's ' observation. At timestep t, the environment is at the state E S. The agent observes o, ~ (-(ss) E O. The agent takes action a; and receives reward r, R(s,, a.). The environment state at timestep + 1 then transit to St+1 ~ T(st, az). The objective of the agent is to maximize the return E'oo're where ? is discount factor.
System Sctup: Many previous works on learning- dexterous manipulation attach the hand to a fixed platform to simplify the experiment environment in the real world. In this work, we create a more flexible and powerful dexterous manipulation system which includes both the robotic hand and the arm (Figure 2). Concretely, we attach the Allegro Hand to an X.rmm robot. Allegro Hand
3

---

is a 16-DoF anthropomorphic hand with four fingers and XArıō is a 6-DoF robot arm. We place RealSense D435 camera at the right front of the robot to capture the point cloud. This setup brings additional challenges to RI. exploration and Sim2Real deployment. We use SAPIEN [52] platform with uses a full-physics simulator to build the environment of the whole system. The simulation time step is 0.005s to ensure stable contact simulation. Each control step lasts for 0.05s.
Tasks and Objects: In this paper, we expect our robot to perform the grasping task over diverse set of objects and to open a locked door by rotating the lever. In the grasping task, we first select a random object from an object dataset and place it on the table. The robot is then required to move it to a target pose. Moreover, the robot should be able to generalize to different initial states, so we randomize both the initial pose and the goal pose for each trial. In simulation experiments, we use bottles and cans from both ShapeNet [53] dataset and YCB [54] dataset. In real-world experiments, we use novel unseen objects to test the policy. In the door opening task, the robot hand is required first to rotate the lever to unlock the door latch and then pull the lever in circular motion. We use three doors to test the policies both in the simulation and the real world, only one used for training and the other two are unseen doors. We also randomize the initial pose for each trial.
The observation contains both visual and proprioceptive information with four modalities: (1) Observed point cloud provided by the camera; (2) Proprioception signals of the robot including joint positions and end-effector position; (3) Imagined hand point cloud proposed in Sec. 3.2: (4) Object goal position provided in each trial. All the information is accessible on the real robot. The dimension of each observation modality is shown Figure 3.
Observation Space:
Action Space: The action is responsible for controlling both the 6-DOF robot arm and the 16-DOF hand. It has 6 + 16 22 dimension in total. The robot arm is parametrized by the 6D translation and rotation of the end-effector relative to a reference pose. We use the damped least square inverse kinematics solver with a damping constant À 0.05 compute the joint motion. Each finger joint of the Allegro hand is controlled by position controller. Both robot arm and hand are controlled by PD controllers.
Network Architecture: The network architecture is visualized in Figure 3,
3.1 Reward Design with Oracle Contact
Since we aim to solve the dexterous manipulation problem with pure RL, the reward design is central the method. We need good reward function ensure proper interaction between the robotic hand and the object. The whole interaction process consists of two phases. The first phase is simply reach the object. The second phase is to grasp the object and move it to the target, which is more challenging. For the first phase, we encourage reaching with the following reaching reward:
Σ
Freach
finger Cr + d(Xinger, Xobj)
Here, Xinger and Xobj are the Cartesian position of each fingertip and the target object. Note that Xobj is available when we perform training in simulation. However, using this reward alone cannot ensure proper contact for grasping. For example, the robot can touch the object with the back of the hand rather than the palm and then get stuck in this local minimum. Therefore, we introduce a novel contact reward to guarantee meaningful contact behavior:
Tcontact: IsContact object) AND
(fnnger (bbjet), object) 2 2
(2)
finger
This contact reward function outputs a boolean value in {0, 1}. It outputs 1 only if the thumb is in contact with the object and there are more than one finger in contact with the object. Intuitively, it encourages the robot to cage the object within fingers. In this case, the robot can quickly find out stable grasping and lift the object the target location. The lifting behavior is encouraged by
Fiftt = Toontact (ift Xtargot).
The Lift function is basically in the form of Equation I and the main difference is that it will return a large reward value upon task completion. The overall reward function a weighted combination of the terms above plus a control penalty:
R- Wresch I reach WWonntactFoontee + Wiit Tife Wpenalty penaltyy

---

Concatenate, dim
Concatenate, dim
Observed Point Cloud
One-Hot Indicator
PointNet
Actor MLP
Action
Imagined Point Gloud
Indicator
MLP
Critic MLP
Value
Observed Point Cloud & Imagined Point Cloud
Robot Proprioception
Figure 3: Architecture: our feature extractor takes the observed point cloud, imagined point cloud, robot proprinception, and goal pose as input output feature emhedding. Both actor and critic take the feature predict action and value. The red point represented imaged point cloud robot hand. Note that our network does not require RGB information.
3.2 Imagined Hand Point Cloud
The usage of point cloud comes with two challenges. The first challenge is occlusion, which may occur both the object under manipulation and the hand itself. When the robot hand is interacting with an object, the fingers may be occluded by the object. Since we do not assume tactile sensors in this work, this occlusion problem can be serious. The second challenge the low point cloud resolution during RL training, where we can only use limited number of points due to the memory limit. In this case, the number of points from hand finger may not be adequate to precisely capture the spatial relationship between the robot and the object. We propose simple yet effective method to handle both issues in a unified manner. Our idea is to use an imagined hand point cloud in the observation to help the robot to the interaction.
We provide one example Figure 3. Black points indicate the point cloud captured by the camera, in which some important details of fingers are missing. These missing details provide crucial infor- mation of the interaction. Though such interaction information can also be inferred by combining the information from both the proprioception and visual input, we find that the best way is to syn- thesize these missing details. Concretely, we can compute the pose for each finger link forward kinematics given the joint position from the robot joint encoder and the robot kinematics model. Then, we synthesize the imagined point cloud (blue points Figure 3) by sampling the points from the mesh of each finger link. This process is possible in both simulation and the real world.
3.3 Training
We adopt Proximal Policy Optimization (PPO) [55] to train the agent in simulation, and then de- ployed real without real-world fine-tuning. The network architecture is illustrated in Figure 3. Both value and policy networks share the same visual feature extraction backbone. We concatenate the observed point cloud with the imagined hand point cloud together the input to the feature extractor. We also attach one-hot encoding to each point which indicates whether it is observed or an imagined point.
4 Experiments
4.1 Experimental Setup
Point Cloud Pre-processing: enable smooth transfer from simulation to real-world, we apply the same data preprocessing procedure to the point cloud captured by the camera. It involves four steps: (i) Crop the point cloud to the work region with manually-definee bounding-box; (ii) Down- sample the point cloud uniformly to 512 points; (iii) Add distance-dependees Gaussian noise to the simulated point cloud to improve the sim2real robustness; (iv) Transform point cloud from the camera frame to the robot base frame using camera pose. In simulation, we use the ground-truth
5

---

Girgie
egg Con
Condact
Contlit
34
Figure 4: Training Curves. The left two plots show the single-object and multi-object training curve of (a) bottle category and (b) can category. The right three plots show the ablation results on the (c) grasping bottle (d) grasping can and (e) door opening. The x-axis is the training iterations and y-axis is the normalized episodic return. The shaded area indicates standard error and the performance is evaluated on five random seeds.
Bottle
Can
Settings
Known Obj.
Novel Obj.
Known Obj.
Novel Obj-
Single Obj. Training
0.81 + 0.09
0.60 + 0.06
0.96 + 0.04
0.63 + 0.18
Multi Obj. Training
0.83 + 0.16
0.81 + 0.15
0.93 + 0.07
0.68 + 0.09
Table 1: Experiment on Multi-ohject Training. We evaluate the policy trained with single and multiple objects on bottle (upper two rows) and can (bottom two rows) categories with point cloud input. We test them on both known or novel objects. The success rate are reported on 5 seeds.
camera pose with multiplicative noise for frame transformation. In the real-world, we perform hand- eye calibration get the camera extrinsic parameters.
Evaluation Criterion: We evaluate the performance of a policy by its success rate. For grasping tasks, task is considered success if dot < 0.05m in simulation, where d is the distance between object position and goal position. In real world, the task considered as success if the XY position of the object is within 5cm from the target position and the height of the object is at least 15em from table top. For the door opening, the task is considered as success the door is opened to at least around 45 degrees.
GrgenGrasp Baseline: We choose the EigenGrasp [56] as the grasp representation. Given an object mesh model, we use the Grasplt [57] to search valid grasp for Allegro Hand. Then, we use the RRTConnect [58] motion planner implemented in OMPL [59] to plan a joint trajectory to the pre- grasp pose and then plan screw motion from pre-grasp pose to the grasp pose. Finally, we close all fingers based on searched grasp pose and lift the object to the target. Note that different from our approach, the baseline method requires complete object model to search for grasps and ground-truth object pose to align the grasp pose in the robot frame. To evaluate the performance of baseline on novel objects, we first build a grasp database on ShapeNet bottle and can categories using GraspIt. Given the sensory data of new object, we scarch for the most similar objects in the dataset and use the query grasps for the novel object. Here we compare the performance of our method with baselines in the real-world.
We train RL in two settings for grasping: (i) training on a single object (ii) training on multiple objects jointly. For the single object grasping. we perform experiments on both "tomato soup can" and "mustard bottle" from YCB. For the multi-object training, we choose 10 objects from the can or bottle categories of ShapetNet. We randomly choose one object to train for each episode in the multi-object training. Here, we use can and bottle as experimental subjects since they represent two different basic grasping patterns [60] for anthropomorphic hand: precision grasp and power grasp. For door opening, we only train the policy on the door with fixed lever geometry.
Training:
4.2 Comparison of Single-object and Multi-object Training
We plot the training curve of RL in Figure (a) and (b). In general, our method can learn to grasp and move the object to the goal pose within 600 iterations, where each iteration contains 20K environment steps. Then we evaluate the policy trained on both known and novel objects, and the results are shown in Table 1. We run 100 trials to compute the average success rate. On grasping known objects, we find that agent trained on single-object outperforms the agent trained on multi- object by small margin, for both learning efficiency and final success rate. However, agent trained on multi-objects does much better at grasping novel objects. Our results suggest that using multiple object during training is important, and is of great importance for novel object generalization.

---

Figure Real-experiment: We evaluate our point cloud policy on various unseen objects.
Bottle
Can
Door
Settings
Known Ohj.
Novel Obj.
Known Ohj.
Novel Obj.
Known Obj.
Novel Obj
w/o Imagined PC.
0.60 + 0.46
0.56 + 0.51
0.91 + 0.17
0.63 = 0.07
0.14 + 0.28
0.11 = 0.27
w/o Contact Rew.
0.00 0.00
0.00 + 0.00
0.06 + 0.08
0.03 + 0.06
0.21 + 0.26
0.20 + 0.25
w/o Both Ours
0.00 0.00
0.00 + 0.00
0.00 + 0.00
0.00 + 0.00
0.00 : 0.00
0.00 0.00
0.83 0.16
0.81 + 0.15
0.93 + 0.07
0.68 - 0.09
0.92 + 0.06
0.79 0.11
Table 2: Ablation Study: we investigate the influence of contact-based reward design and imaged point cloud. We evaluate the success rate both known and novel objects under four settings: (i) without imaged point cloud; (ii) without contact reward: (iii) without both; (iv) with both.
4.3 Ablation Results in Simulation
We ablate two key innovations of the work: the reward design with oracle contact and the imaged hand point cloud. We perform experiments on four different variants: (i) without imagined point cloud; (ii) without contact-based reward design; (iii) without both imagined point cloud and contact based reward design; (iv) standard approach with both techniques. Note that the variant (iii) is an approximation of [3] in our environments and tasks. We compare both the learning curve and the evaluation success rate of these four variants. For grasping task, Figure 4 (c) and (d) show the results on bottle and can categories, and Table 2 shows the success rate. Our findings can be summarized follows.
First, we find that contact reward information is of vital importance for training the point cloud RI. policy on the multi-fing fulti-finggr robot hand. Without using contact reward, the agent can hardly learn anything (red and green curve in the figure) and get nearly zero success rate during evaluation for both bottle and can categories. By encouraging the contact between fingers and object, the RL agent can avoid getting stuck in local minimums and learn meaningful manipulation behavior.
Second, the imagined point cloud can also improve the training and test performance for both cat- egories, though is not so important as the contact reward. is shown in the learning curves of Figure 4 (c) and (d), the policy utilizing the imagined point cloud as input can learn faster in the early stage of the training and show much smaller variances. An interesting fact that imagined point cloud is more beneficial for the bottle category than the can category. One possible reason is that grasping a bottle requires multi- finger coordination to perform a power grasp. Such coordina- the detailed finger information provided by the imagined point cloud.
tion is highly dependent The imagined point cloud can help the agent to better see the fingers even if they are occluded by the object, e.g., fingers behind the object.
The experiments on the door opening task also support these findings. As shown in Figure 4 (e), the policy trained with contact based reward and imaged hand point cloud outperform other abla- tion methods, which also demonstrate the effectiveness of our two key design. Compared with the grasping experiments, we can observe larger variations during policy training. One possible reason that door opening suffers from heavier occlusion than grasping, when the door lever grasped by the robot hand, which influence the temporal consistence of PointNet feature.
7

---

4.4 Real- World Evaluation
Method
Bottle
Can
Mixed Cutegory
Bottle
EigenCrasp Oracle
0.66
0.45
N/A
EigenGrassp
0.50
0.41
N/A
Single Obj. Train
0.75
0.06
0.75 - 0.08
N/A
Multi Obj. Train
0.87
0.03
0.83 0.13
0.73 + 0.12
Table 3: Real-World Grasping Experiment: This experiment
consist 3 categories, which incorporate 26 objects: bottles, Can
Mixed Objects
cans, and 10 other objects in multiple mixed categories.
Scttings
Original Door Novel Door
Novel Door
Dos
Single Door Train 0.72 +
0.60 0.03
0.67= = 0.01
Table 4: Real-Ve Real-World World Door Opening Experiment: This experi-
ment consist of 3 different doors, the first one on the left is used for training and the other are for testing.
Doors
We perform sim2real experiments to evaluate the performance of our method in the real world. As shown in Figure 2, we attached an Allegro hand onto a XArm-6 robot arm to grasp the object on the front table. We apply the same data-preprocessing steps for both simulated environment and real-world as mentioned in Sec.4.1.
The task execution sequence is visualized on the bottom row of Figure 1. Both Single Obj. Training and Multi Obj. Training will be evaluated in the corresponding category, while policies training with multiple objects even evaluated in the Mixed category with unseen objects. We run 10 independent trials seeds for each object-policy pair.
The real-world evaluation results are shown in Table 3 and Table 4. The policy trained in the simu- lator with point cloud input directly transfer to the real-world without fine-tuning. For both two tasks, our policy can even deploy on objects that have never been seen during the training. Moreover, We find that for grasping, training on multiple objects object training. Compared with the EigenGrasp baseline shown Table 3, our policies trained on both single-object and multi-object performs better, even if the EigenGrasp baseline use the object model information. Since EigenGrasp + motion planning is a open-loop manipulation policy, small error in object and scene modeling, e.g. initial object pose or object geometry, can lead fail- ure in final grasp results. In contrast, our methods works in a closed-loop fashion with point cloud observation, which does not require privilege knowledge about the object.
ensure better performance than single-
Conclusion and Limitation
Limitations In our experiments, we only train and test our method on two tasks, which limits the scope of the proposed method. More dexterous manipulation tasks will our future research di- rection. Another potential improvement of our method is to use Recurrent Neural Network and temporal information for policy networks. will enable us to do long-horizon tasks.
Conclusion To the best of our knowledge, our approach is the first work to train a dexterous ma- nipulation reinforcement learning policy with point cloud inputs that can transfer the real world. We justified that direct sim-to-real transfer is possible for two manipulation tasks with point cloud representation.
Acknowledgments
This work was supported, in part, by grants from NSF CCF-21 12665 (TILOS), NSF 1730158 CI- New: Cognitive Hardware and Software Ecosystem Community Infrastructure (CHASE-CI), NSF ACI-1541349 CC*DNI Pacific Research Platform, the Industrial Technology Innovation Program (200181 12, Development of autonomous manipulation and gripping technology using imitation learning based on visualtactile sensing) funded by the Ministry of Trade, Industry and Energy of the Republic of Korea, and gifts from Meta, Google, Qualcomm. We also thanks Fanbo Xiang for suggestion shader setting in SAPIEN, Ruihan Yang for helpful discussion on RL training.

---

References
[1] O. M. Andrychowicz, Baker, M. Chociej, Jozefowicz, B. McGrew, J. Pachocki, A. Petron, M. Plappert, G. Powell, A. Ray, et al. Learning dexterous in-hand manipulation. The International Journal of Robotics Research, 39(1):3-20, 2020.
[2] H. Zhu, A. Gupta, A. Rajeswaran, S. Levine, and V. Kumar. Dexterous manipulation with deep reinforcement learning: Efficient, general, and low-cost. In 2019 International Conference on Robotics and Automation (ICRA), pages 3631-3657. IEEE, 2019.
[3] Chen, J. Xu, and P. Agrawal. A system for general in-hand object re-orientation. In Confer- ence on Robot Learning, pages 297-307. PMLR, 2022.
[4] Y.HH. Wu, J. Wang, and X. Wang. Learning generalizable dexterous manipulation from human grasp affordance. arXiv preprint 220.i:2200.00320, 2022.
[5] W. Huang, I. Mordatch, P. Abbeel, and D. Pathak. Generalization in dexterous manipulation via geometry-aware multi-tasi talti-task learning. arXiv preprint arXiv:2111.0362. 2021.
[6] Mu, Z. Ling, Xiang, D. Yang, X. Li, S. Tao, Z. Huang, Z. Jia, and H. Su. Maniskill: Generalizable manipulation skill benchmark with large-scale demonstrations. arXiv preprint arXiv:2107.14483 14483, 2021.
[7] X. Zhang. R. Chen, Xiang. Qin, J. Gu, Z. Ling, M. Liu, Zeng. S. Han, Huang, et al. Close the visual domain gap by thysics-grounder active stereovision depth sensor simulation. arXiv preprint arXiv:2201. 1924, 2022.
[8] W. Roboties.
Allegro
robot
hand.
htttp:://www.vonkkkroootttt
research-robot-hand
[9] J. K. Salisbury and J. Craig. Articulated hands: Force control and kinematic issues. The International journal of Robotics research, 1(1):4-17, 1982.
[10] I.. Han and J. C. Trinkle. Dexterous manipulation by rolling and finger gaiting. In IEEE International Conference on Robotics and Automation, volume 1, pages 730-735. IEEE, 1998.
[11] D. Rus. In-hand dexterous manipulation of piecewise-smooth 3-d objects. The International Journal of Robotics Research, 1999.
[12] D. Prattichizzo and C. Trinkle. Grasping. In Springer handbook of robotics, pages 955-988. Springer, 2016.
[13] A. Bicchi and V. Kumar. Robotic grasping and contact: A review. In IEEE International Conference on Robotics and Automation, volume 1, pages 348-353, IEEE, 2000.
[14] J. Bohg, A. Morales, T. Asfour, and D. Kragic. Data-driven grasp synthesis survey, IEEE Transactions robotics, 30(2):289-309, 2013.
[15] M. R. Dogar and S. S. Srinivasa. Push-grasping with dexterous hands: Mechanics and method. 2010.
[16] O. Kroemer, S. Niekum, and G. D. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. Journal of machine learning research, 22(30), 2021.
[17] S. Levine, N. Wagener, and P. Abbeel. Learning contact-rich manipulation skills with guided policy search. In IEEE International Conference on Robotics and Automation, ICRA, pages 156-163. IEEE, 2015.
[18] A. Gupta, C. Eppner, S. Levine, and P. Abbeel. Learning dexterous manipulation for soft robotic hand from human demonstrations, In IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 3786-3793 IEEE, 2016.
[19] T. Zhang, Z. McCarthy, O. Jow, D. Lee, X. Chen, K. Goldberg, and P. Abbeel. Deep imita- tion learning for complex manipulation tasks from virtual reality teleoperation. In 2018 IEEE International Conference on Robotics and Automation (ICRA), pages 5628-5635, 2018

---

[20] I. Radosavovic, X. Wang. L. Pinto, and J. Malik. State-only imitation learning for dexterous manipulation. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 7865-7871. IEEE, 2020.
[21] E. Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In IEFE International Conference Robotics and Automation (ICRA), pages 4613-4619. IEEE, 2021.
[22] A. Rajeswaran, V. Kumar, A. Gupta, G. Vezzami, J. Schulman, E. Todorov, and S. Levine. Learning complex dexterous manipulation with deep reinforcement learning and demonstra- tions. In Robotics: Science and Systems, 2018.
[23] S. Christen, S. Stevšić, and O. Hilliges. Guided deep reinforcement learning of control policies for dexterous human-robot interaction. In 2019 International Conference on Robotics and Automation (ICRA), pages 2161-2167. IEFE, 2019.
[24) Y. Qin, H. Su, and X. Wang. From one hand to multiple hands: Imitation learning for dexterous manipulation from single-camera teleoperation. IEEE Robotics and Automation Letters, 7(4): 10873-10881, 2022.
[25] R. Jeong, Springenberg. Kay, D. Zheng, A. Galashov, N. Heess, and F. Nori. Learning dexterous manipulation from suboptimal experts. In J. Kober, F. Ramos, and C. J. Tomlin, editors, Conference on Robot Learning, volume 155 of Proceedings of Machine Learning Re- search, pages 915-934. PMLR, 2020.
[26] Y. Qin, Y.-H. Wu, S. Liu, H. Jiang. R. Yang. Y. Fu, and Wang. Dexmv: Imitation learning for dexterous manipulation from human videos. arXiv preprint 2xXiv:2108.0500 2021.
[27] Pomerleau, Colas, and R. Siegwart. A review of point cloud registration algorithms for mobile robotics. Foundations and Trends in Robotics, 4(1):1-104, 2015.
[28] Mahler, J. Liang, Niyaz, M. Laskey, Doan, X. Liu, A. Ojea, and K. Goldberg. Dex- net 2.0: Deep learning to plan robust grasps with synthetic point clouds and analytic grasp metries. Robotics: Science and System, 2017.
[29] A. ten Pas, M. Gualtieri, K. Saenko, and R. Platt. Grasp pose detection in point clouds. The International Journal of Robotics Research, 36(13 (6(1--14)14555144777 2017.
[30] H. Liang, X. Ma, S. Li, M. Görner, S. Tang, B. Fang. F. Sun, and J. Zhang. Pointnetgpd: Detecting grasp configurations from point sets. In 2019 International Conference on Robotics and Automation (ICRA), pages 3629-3635, 2019. doi: 10. SAT.CCAA...1..8944...
[31] A. Mousavian, C. Eppner, and D. Fox. 6-dof graspnet: Variational grasp generation for object manipulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2901-2910, 2019.
[32] Y. Qin, R. Chen, H. Zhu, M. Song, J. Xu, and H. Su. S4g: Amodal single- single-shot se(3) grasp detection in cluttered scenes. Proceedings of the Conference on Robot Learning, volume 100 of Proceedings of Machine Learning Research, pages 53-65. PMLR, 2020.
[33] Ni, W. Zhang, X. Zhu, and Q. Cao. Pointnet++ grasping: Learning an end-to-end spatial grasp generation algorithm from sparse point clouds. In 2020 IEEE International Conference on Robotics and Automation (ICRA), pages 3619-3625, 2020. doi: 109//CRA44945.20000 9196740.
[34] C. Wu, Chen, Q. Cao, Zhang, Y. Tai, Sun, and K. Jia. Grasp proposal networks: end-to-end solution for visual learning of robotic grasps. Advances in Neural Information Processing Systems, 1374317--33,84, 2020.
[35] W. Wei, Y. Luo, F. Li, G. Xu, J. Zhong, L. and P. Wang. Gpr: Grasp pose refinement net- work for cluttered scenes. 202/ IEEE International Conference on Robotics and Automation (ICRA), pages -292-4302. IEEE, 2021.
10

---

[36] W. Wei, D. Li, P. Wang, Y. Li. W. Li, Y. Luo. and J. Zhong. Dvgg: Deep variational grasp generation for dextrous manipulation. IEEE Robotics and Automation Letters, 2022.
[37] S. Andrews and P. G. Kry. Goal directed multi-finger manipulation: Control policies and analysis. Computers & Graphics, 2013.
[38] J. Varley, J. Weisz, J. Weiss, and P. Allen. Generating multi-fingered robotic grasps via deep learning. In 2015 IEEE/RSJ international conference intelligent robots and systems (IROS), pages 4415-4420 IEEE, 2015.
[39] S. Brahmbhatt, A. Handa, J. Hays, and D. Fox. Contactgrasp: Functional multi-finger grasp synthesis from contact. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 2386-22862 2393. IEEE, 2019.
[40] Q. Lu, K. Chenna, B. Sundaralingam, and T. Hermans. Planning multi-fingered grasps prob- abilistic inference a learned deep network. In Robotics Research, pages 2020.
Springer,
[41] Y. Chebotar, O. Kroemer, and J. Peters. Learning robot tactile sensing for object manipulation. In 2014 IEEE/RSJ International Conference Intelligent Robots and Systems, pages 3368- 3375, 2014. 1oi:10.1 10401
2014.99331...
[42] W. Yuan, S. Dong, and E. H. Adelson. Gelsight: High-resolution robot tactile sensors for estimating geometry and force. Sensors, 17(12):7762, 2017.
[43] R. Calandra, A. Owens, D. Jayaraman, Lin, W. Yuan, J. Malik, E. H. Adelson, and S. Levine. More than a feeling: Learning to grasp and regrasp using vision and touch. IEEE Robotics and Automation Letters, 3(4):3300-3300 2018.
[44] A. Murali, Y. Li, D. Gandhi, and A. Gupta. Learning to grasp without seeing. In International Symposium on Experimental Robotics, pages 375-386. Springer, 2018.
[45] M. Lambeta, P.-W. Chou, S. Tian, B. Yang, B. Maloon, V. R. Most, D. Stroud, R. Santos, A. Byagowi, G. Kammerer, al. Digit: A novel design for low-cost compact high-resolution tactile sensor with application to in-hand manipulation. IEEE Robotics and Automation Letters, (0D3:3838-38555 2020.
[46] M. A. Lee, Y. Zhu, P. Zachares, M. Tan, K. Srinivasan, S. Savarese, L. Fei-Fei, A. Garg, and J. Bohg. Making sense of vision and touch: Learning multimodal representations for contact- rich tasks. IEEE Transactions on Robotics, 36(3):582-596, 2020.
[47] R. Bhirangi, T. Hellebrekers, C. Majidi, and A. Gupta. Reskin: versatile, replaceable, lasting tactile skins. arXiv preprint arXiv:2/11 1.00071, 2021.
[48] H. Van Hoof, T. Hermans, G. Neumann, and J. Peters. Learning robot in-hand manipulation with tactile features. In 2015 JEEE-RAS 15th International Conference Humanoid Robots (Humanoids), pages 121-127. IEEE, 2015.
[49] N. Vulin, S. Christen, S. Stevšić, and O. Hilliges. Improved learning of robot manipulation tasks via tactile intrinsic motivation. IEEE Robotics and Automation Letters, 602):2194-2211, 2021.
[50] S. H. Huang. M. Zambelli, Kay, M. Martins, Tassa, P. M. Pilarski, and R. Hadsell. Learning gentle object manipulation with curiosity-driven deep reinforcement learning. arXiv preprint arXiv: 08440, 2019.
[51] H. Xu, Y. Luo, S. Wang, T. Darrell, and R. Calandra. Towards learning to play piano with dexterous hands and touch. arXiv preprint arXiv:21066 106.02040, 2021.
[52] Xiang, Y. Qin, K. Mo, Xia, H. Zhu, Liu, M. Liu, H. Jiang, Y. Yuan, H. Wang, L. Yi, A. X. Chang, L. J. Guibas, and H. Su. SAPIEN: A simulated past-based interactive environ- ment. In The IEFE Conference on Computer Vision and Puttern Recognition (CVPR), June 2020.

---

[53] A. X. Chang, T. Funkhouser. L. Guibas, P. Hanrahan, Q. Huang, Z. Li. S. Savarese, M. Savva, Song, H. Su, al. Shapenet: An information-rich 3d model repository. arXiv preprint WrXiv:/5/2.03011 2015.
[54] B. Calli, A. Singh, A. Walsman, S. Srinivasa, P. Abbeel, and A. M. Dollar. The ycb object and model set: Towards common benchmarks for manipulation research. 2015 international conference advanced robotics (ICAR), pages 510-517. IEEE, 2015.
[55] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv: 1707.06347 2017.
[56] M. Ciocarlie, C. Goldfeder, and P. Allen. Dexterous grasping via eigengrasps: A low- dimensional approach high-complexity problem. In Robotics: Science and systems ma- nipulation workshop-sensing and adapting to the real world, 2007.
[57] A. T. Miller and P. K. Allen. Graspit! versatile simulator for robotic grasping. IEEE Robotics Automation Magazine, 144:: 10-122, 2004.
[58] J. Kuffner and M. LaValle. Rit-connect: An efficient approach to single-query path planning. In Proceedings 2000 ICRA. Millennium Conference. IEEE International Conference on Robotics and Automation. Symposia Proceedings (Cat. No. 00CH37065), volume 2, pages 995-1001. IEEE, 2000.
[59] I. A. Sucan, M. Moll, and L. E. Kavraki. The open motion planning library. IEEE Robotics & Automation Magazine, 19(4):72-82, 2012.
[60] J. R. Napier. The prehensile movements of the human hand. The Journal of bone and joint surgery. British volume, 38(4):902-913 1956.
12


Reviews:
[{"review_id": "V6WjJvLPOv", "reviewer": "Reviewer_Cjyv", "summary": "The authors introduce a sim2real policy transfer pipeline that learns grasping of rigid objects via an Allegro hand attach to an xArm robot in simulation and transfers the policy learned via reinforcement learning or behavioral cloning to the real robot.\nTo accomplish the sim2real transfer, the observed point clouds are augmented by an \"imagined\" point cloud that synthesizes the occluded parts of the robot hand based on proprioceptive state measurements. Furthermore, a contact-based reward function is introduced during training in simulation that encourages contact between the thumb and at least two fingers of the robot hand and the object being grasped. Experiments are conducted on a subset of the YCB objects, as well as as on bottles unseen during training.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths:\nThe proposed method is technically sound and well motivated. The introduced contact reward function is shown to have a significant impact on the training performance.\n\nThe authors provide a useful discussion of limitations of the current work.\n\n# Weaknesses\nThe paper is focused on grasping, not \"dexterous manipulation\". The objects are very similar to each other where the robot motion to grasp it does not require any meaningful variation to grasp even the unseen objects, since they all have roughly cylindrical shape. The robot hand essentially just needs to move to the object and then close itself to grasp it. As such, the work does not sufficiently demonstrate its applicability to truly novel objects.\n\nThe experiments are lacking metrics for the stability of the grasp that results from these motions. Judging by the supplied video recordings, the hand often does not firmly hold the object in place, so some quantification of robustness to external disturbances is needed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The contributions to include imagined point clouds in the observations and to add contact information to the reward function are promising, but more evaluation is necessary to verify the robustness of the sim2real transfer. Given the simplistic experiments where the objects have very similar shapes and the lack of comparison against other dexterous grasping works, the sim2real performance is difficult to assess.\n\nEdit: Updated my review to weak accept in response to the authors (see explanation below).", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Relevant but unlikely to deploy to hardware in near future", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "The success metrics between sim and real differ - in simulation a goal position must be reached, but on the real world experiments it suffices to lift up the object a certain height. I do not understand why it is \"inconvenient\" (l. 190) to measure distance to a target location. This makes the real-world experiments much easier to accomplish.\n\nWhy is the observation augmented by a one-hot encoding to indicate whether the point is real or imagined (line 176)? Has this made any difference in the training process?\n\nThe caption of Table 1 needs to be updated, the objects types bottle and can do not appear in the upper and bottom two rows.\nThe caption of Table 2 also does not fit to the table but talks about two figures that are not there."}}, {"review_id": "Pi6EBaqXSX4", "reviewer": "Reviewer_6SGW", "summary": "The paper proposes a method for robotic grasping with a dexterous robot hand in the real world. The method is point-cloud based and fully trained in simulation. The authors propose a shaped reward, including contact information between robotic hand and the object to grasp. Furthermore they propose to construct the point-cloud of the robotic hand from proprioceptive information and use this imagined point cloud as an additional input to their method. \nThe authors ablate their modelling choices in simulation and  demonstrate that their trained method can be transferred to the real world without fine-tuning.  ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The authors demonstrate that their method can be transferred to the real world without fine-tuning. \n- The formulation of the reward seems to be useful and lead to good results.\n- The idea of adding the imagined gripper point cloud seems improve the training.\n\nWeaknesses:\n- The evaluation the authors perform is rather limited: They only train and evaluate on cans and bottles in simulation. In the real world the evaluation is even more limited, as the authors only evaluate on 4 different bottles. \n- There is no comparison to other methods like e.g. Unigrasp [L. Shao et al., \"UniGrasp: Learning a Unified Model to Grasp With Multifingered Robotic Hands,\" in IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 2286-2293, April 2020, doi: 10.1109/LRA.2020.2969946].", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The ideas seem nice and promising, but the evaluation is not very thorough. There is no comparison to any baseline (e.g. Unigrasp). Also the evaluation is performed only on very view objects of limited variability. In particular the real world experiments are performed only four objects of type bottle. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Formulation of loss:\n- line 140:  how is $x_{obj}$ is defined? Is it the center of the object?\n- eq.2: How is inContact defined? How is it determined if contact was established?\n\nExperiments: \n- There is no comparison to a baseline.\n- The evaluation is performed only on a very limited set of objects (bottles and cans) and only on four objects of type bottle. \n- Table 2: The caption seems to be wrong. There is no learning curve visible here. \n"}}, {"review_id": "CF5MMF8zdh", "reviewer": "Reviewer_BaTP", "summary": "This work proposes a policy learning framework for utilizing multi-fingered hands, such as the Allegro platform, to grasp rigid objects. The input corresponds to a point cloud, which is augmented with proprioceptive and object pose information to construct the observation space. The action space consists of the arm’s 6 joint angles and the end-effector's 16 DoFs. Relatively to prior efforts, this work introduces a contact-based reward function (which is where most of the argued benefits arise from) and an “imagination”- based point cloud augmentation technique (for mitigating occlusions and sensor noise). The final outcome is category-based generalization to novel object instances and zero-shot sim2real transfer for grasping. An ablation study has been conducted (in simulation) to compare different training strategies and how training over multiple objects affects the approach. The performance in the real world was tested with four objects. ", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The problem that the paper attempts to solve involves a high-dimensional control challenge, due to the presence of a multi-fingered hand. For such end-effectors, which bring the promise of dexterous manipulation, grasping singulated objects (which is the focus of this work) can still be challenging especially as object variability increases. \n- The ideas on the “oracle contact” and the imagined hand point cloud seem reasonable for the proposed objective. \n- The authors have tested their approach in real-world robot experiments and seemed to have achieved zero-shot Sim2Real transfer, which is highly desirable.\n- The paper is mostly clearly written, with proper formalism and an accompanying ablation in simulation. The visualizations are helpful.\n- The related work section is rather comprehensive and covers the important subareas related to this work.\n\nWeaknesses:\n- The expression \"Dexterous Manipulation\" used in the title is rather misleading. It should be replaced by the term ‘Multi-fingered Grasping’, which is what this work is focusing on.\n- The fact that the method is both trained and tested only on 2 distinct object categories + the nature of the contact-based reward may imply that the approach learns a shape prior and that this is the main reason this method succeeds. What happens in object categories significantly outside this shape prior still remains open for interpretation.\n-The reward function seems to have multiple components that are aggregated together with a rather naive weighting scheme. The reward structure does not seem to reflect the structure of the task (e.g., first approach the object and then lift it) as the weighting scheme seems to be constant over the duration of the trajectory instead of time-dependent. It would be helpful to further explore how the different components can be integrated into an effective reward function and the role of the weights in this process.  Some of this discussion is left for the supplementary material but it would be better if the penalty term in the overall reward, as well as the multi-task weights’ scheduling scheme, were introduced in the main paper.\n- In contrast to some related efforts (e.g., [3]), access to the object geometries seems to be a requirement for this work during training. \n- The “Limitations” section is more of ‘Future Work’ one. It would be better if the authors acknowledged in a more comprehensive manner the limitations (e.g., there is still a gap until the grasping problem targeted is solved).\n- The experiments in the ablation study between single and multiple objects do not communicate a clear message. In particular, the training curves for the two object categories provided do not give a clear indication which method is better and why. Since this is referring to simulation-based experiments, adding more categories may be helpful to evaluate a pattern. \n- There is a mismatch between what is described for the Multi-object case between the main paper and the appendix. In the second one, only the ShapeNet objects are mentioned, while in the first one the authors mention that they randomly sample from both datasets. \n- According to the ablation study, the approximation of the previous, related approach ([3]) achieves 0% success rate. How does this match the results presented in that work, which focused on in-hand object reorientation and must have included contacts. \n- The provided videos do not visualize the goal pose at each test case (like in the paper’s diagram), which makes it harder for the reviewer to evaluate the task’s success/failure. Furthermore, there is no video that clearly showcases the effect of adding solely the contact-based reward term, the strongest point of the paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "This a rather well-written paper, with solid fundamentals, aiming to solve a high-dimensional grasping challenge. Both contributions presented are reasonable and presented in sufficient detail. The paper provides real-world experiments and demonstrates Sim2Real transfer. The results are intuitive, even though there are some points that would need further clarification per the comments above. The quantitative results of the method are well-organized and indicate an improvement in terms of grasping success even though there are still failure cases.. Some additional discussion of the qualitative results would be helpful. ", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are not well addressed", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "Change the title of the paper to replace “Dexterous Manipulation” with “Multi-fingered Grasping”.\n\nThis algorithm is tested on a quite standard setup (single tabletop object, uncluttered, a commonly-used hand) and has virtually no inference comparison point. So, papers like the ‘Multi-fingered Active Grasp Learning’ by Lu et al., ‘Learning Continuous Grasping Function with a Dexterous Hand from Human Demonstrations’ by Ye et al. (admittedly it has been published after the submission deadline)), ‘Multifingered Grasping Based on Multimodal Reinforcement Learning’ by Liang et al.  and other similar works could be included in the related work section. One of them could potentially be used as a comparison point to this algorithm’s performance.\n\nIt would be appreciated if the authors provided failure examples and look into which part of their method is causing them and in which scenarios.\n\nHow many training iterations would be required if somebody would want to train for a new object category?\n\nPlease provide some justification for the following choices: a) The Object/Goal Pose Rotational components are not included in the reward term. b) Only two fingers are needed for the contact reward term to be active.\n\nThe authors should add legends to their plot diagrams (including measurement units).\n"}}, {"review_id": "0GUAwldsIgF", "reviewer": "Reviewer_aTKU", "summary": "The paper proposes two techniques to learn dexterous manipulation policies with point clouds as observation. The authors propose using Reinforcement Learning to learn the policy and evaluate the performance of the learned policy when transferred from simulation to reality.\n\nThe paper's main contributions are:\n1. Show that it is possible to achieve sim-to-real transfer for dexterous grasping policies when using point cloud as observations.\n2. Two techniques to guarantee proper learning: \n         (a) a Pointcloud augmentation to deal with possible camera occlusions\n         (b) a hand-object contact reward function.\n", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "In the reviewer's view, the strengths of the paper are:\n1. The demonstration of the utility of point cloud observation for sim-to-real transfer.\n\nIn the reviewer's view, the weaknesses of the paper are:\n1. The paper is very task specific. The authors present the method they have used to train a reinforcement learning agent to grasp objects given point clouds. To do so, they introduce a reward function and the observation augmentation methods but it is not clear if these methods would also benefit in other possible dexterous manipulation tasks as it has been only evaluated for grasping. Evaluating the method in additional manipulation tasks would be beneficial to show the benefit of the proposed method.\n2. The solution is validated only with PPO. It would be interesting to observe the performance of the proposed method with other reinforcement learning algorithms.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper presents a problem and provides a solution to it. The authors show that the method works for different objects and show that the method transfer to real robots. The paper itself seems very specific to the problem of learning through reinforcement learning and a grasping policy through point clouds. But, it is properly executed and the authors show the performance of the proposed method and validate the benefit of the proposed techniques. \n\nTherefore, I suggest a weak acceptance.\n\n", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Additional details required", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "4: The reviewer is confident but not absolutely certain that the evaluation is correct"}, "extra_text": {"issues": "1. Adding a new experiment on a different task. It could be related to grasping, such as opening a drawer.\n2. Evaluating the benefits of the proposed techniques with different RL algorithms. The method should work, but the paper would be more complete if an experiment with SAC and TD3 are added."}}]

Author rebuttal:

Decision: Accept (poster)