## 2 Methods
### Object Manipulation T-DRL
#### 2.2.2 DRL with Reward Shaping  
The DRL policy illustrated in Figure 1 is trained to control the robot's movements for generating a natural grasping hand pose for object manipulation. The model-free DRL policy \(\pi_{\Theta}(a_{t}s_{t})\) describes the control problem of the robot hand as a Markov Decision Process \((S,A,R)\), where \(s\in S\), \(S\subseteq\mathbb{R}^{n}\) is an observation vector describing the current location of the robot hand and object in Cartesian coordinates. The agent's actions \(a\in A\), \(A\)\(\subseteq\mathbb{R}^{m}\) control the robot hand's movements to interact with the object. A reward function \(r_{t}=R(s,a,s^{\prime})\) evaluates the completeness of the manipulation task.  
To optimize the parameters of the DRL policy we follow the implementation of natural policy gradient (NPG) in [38], which computes the policy gradient as:  
\[\
abla_{\theta}J(\theta)=\frac{1}{NT}\sum_{i=1}^{N}\sum_{t=1}^{T}\
abla_{ \theta}log\pi_{\theta}\Big{(}a_{t}^{i}\Big{|}s_{t}^{i}\Big{)}\,A^{\pi}\Big{(} s_{t}^{i},a_{t}^{i}\Big{)} \tag{3}\]  
Then, it pre-conditions the gradient with the Fisher Information Matrix, \(F_{\theta}\) and makes the following normalized gradient ascent update:  
\[\theta_{k+1}=\theta_{k}+\sqrt{\frac{\delta}{\
abla_{\theta}J(\theta)^{\text{ T}}\cdot F_{\theta_{k}}^{-1}\cdot\
abla_{\theta}J(\theta)}}\cdot F_{\theta_{k}}^{-1} \cdot\
abla_{\theta}J(\theta) \tag{4}\]where \(\delta\) is the step size. The advantage function \(A^{\pi}\) is the difference between the value for the given state-action pair and the value function of the state. The NPG in [38] uses the general advantage estimator (GAE), which is defined as:  
\[A^{GAE}=\sum_{l=0}^{T}(\gamma\lambda)^{l}\delta_{l+l}^{V} \tag{5}\]  
where, \(\delta_{t}^{V}=r_{t}+\gamma V(s_{t+1})-V(s_{t})\) defines the temporal difference residual between consecutive predictions, assuming a value function \(V\) that approximates the true value function, \(V^{\pi}\). Both the policy and value networks share the same architecture and were defined by a three-layer neural network with 64 units in the hidden layers.  
The reward function maximizes the sum of the expected rewards at the end of each training episode. This function measures the similarity of the robot hand's pose to the optimal hand pose estimated by the transformer network. It also evaluates how close the object is to the target location at the end of an episode. Defining the reward function to solve the manipulation tasks is expressed as follows:  
\[r_{g}=\begin{cases}\lambda_{1}\times r_{joints}&i<t_{150}\\ \lambda_{2}\times r_{tro}+\lambda_{3}\times r_{o:t}&o.w.\end{cases} \tag{6}\]  
where \(r_{joints}\) is the mean squared error between the hand pose of 21-DoF, derived from the transformer network, and the robot hand's pose after following the policy actions. Empirically, we found that by optimizing for \(r_{joints}\) during the first 150 timesteps of each iteration, the policy could learn how to shape the robot hand before grasping an object. The second term of the reward function minimizes the distance between the hand and the object \(r_{R:o}\) and the distance of the object to the target location \(r_{o:t}\). The values of \(\lambda_{1},\lambda_{2}\) and \(\lambda_{3}\) are constant weights to balance the total reward, \(r_{g}\), which measures the completeness of the task. The value of \(r_{g}\) gives a larger reward when at the end of the episode the object finishes near the target location.