## 3 Results
### Object Manipulation Performance by T-DRL  
The average success rates of T-DRL and NPG for grasping and relocation tasks of the six objects are shown in Table 1. The success rate achieved by our T-DRL was 90.1%, outperforming the 65.3% achieved by NPG. For the objects, such as light bulb, bottle, and camera, the baseline policy could not learn a proper grasping strategy, resulting in a low success rate. However, for objects such as the bottle and camera (those with values under 50%), we noticed the policy achieved to increase the reward by hitting the object and lifting it in the air. This is not a desired behavior that we expected the DRL policy to learn without proper guidance. The results for each object reported in Table 1 are the averaged success rate for policies trained by T-DRL and NPG, which were trained with three random initializations per object. In total, we trained 36 different DRL policies (18 T-DRL and 18 NPG) for grasping and relocation in a workstation with two RTX-2080 GPU for more than 100 h.  
In the left column of Figure 4, the average cumulative reward for all objects is shown, where the solid line represents the mean and the shadow area the standard deviation of the rewards. Overall, the cumulative reward from T-DRL achieves a higher success rate under 1000 iterations with 70% of the maximum return. This increment in speed is significant compared to NPG. The right column of Figure 4 illustrates the time-series frames of the DRL policies during grasping and relocation. Without shaped rewards, the resulting actions were unnatural and failed to grasp the object. This is shown in the row frames of Figure 4g, where the light bulb was hit and thrown into the air with force. In Figure 4i, the robot hand merely pushed the bottle off the table and did not learn to lift it. In Figure 4k, a similar behavior is shown with the light bulb. This phenomenon is all part of the learning process as DRL continues to explore the environment. By T-DRL, we were able to minimize these errors and prevent unnatural behaviors, manipulating the objects in a manner that resembled human-like actions.  
The NPG policy grasps the hammerhead and reaches the target location with a similar success ratio compared to T-DRL, as shown in Figure 4c,d. While it achieves a high success rate, this hand pose does not comply with the object affordance that is inferred when using hand pose priors in the training loop. T-DRL reduces the expectation that the policy learns these behaviors. More importantly, T-DRL generated a pre-shape of the hand by extending the fingers, as seen in the row frames of Figure 4h,j, and then proceeded to close the hand grasping the object.  
\begin{table}
\begin{tabular}{c c c} \hline \hline \multirow{2}{*}{**Objects**} & **NPG** & **T-DRL** \\  & **(Mean \(\pm\) std)** & **(Mean \(\pm\) std)** \\ \hline Apple & 91.58 (\(\pm\)3.03) & 94.68 (\(\pm\)1.42) \\ Banana & 88.36 (\(\pm\)8.12) & 92.63 (\(\pm\)3.20) \\ Hammer & 89.03 (\(\pm\)4.85) & 94.37 (\(\pm\)2.98) \\ Light-bulb & 68.53 (\(\pm\)8.45) & 87.61 (\(\pm\)7.14) \\ Bottle & 21.81 (\(\pm\)7.51) & 90.35 (\(\pm\)4.20) \\ Camera & 32.40 (\(\pm\)6.06) & 81.21 (\(\pm\)3.52) \\ \hline \hline \end{tabular}
\end{table}
Table 1: Success rate (%) of natural manipulation tasks using NPG and T-DRL.  
Figure 4: On the left, the average sum of rewards for object manipulation by NPG (blue, top) and T-DRL (purple, bottom). On the right, the evaluation time-series frames from grasping and relocation of each object using the best policies of NPG (blue, top) vs. T-DRL (purple, bottom) for apple (**a**) vs. (**b**), hammer (**c**) vs. (**d**), banana (**e**) vs. (**f**), light bulb (**g**) vs. (**h**), bottle (**i**) vs. (**j**), and camera (**k**) vs. (**l**).