## 4 Implementation
### Test Setup
#### 4.3.1 Test 1  
Here, the objective was to learn a controller that guides the robot to a place near the table so that the arm could plan a trajectory up to the object. The robot and object initial poses were variable, which are defined in Equations (14) and (15). The robot's initial \(y_{r}^{w}\) coordinate was constant so that the robot always began at the same distance from the table. The state codification is defined in Equation (16) and it is composed of the following 10 variables:  
Robot's position in world coordinate system: \(x_{r}^{w},x_{r}^{w}\)  
Robot's rotation on \(z\) axis in world coordinate system: \(yaw_{r}^{w}\)  
Robot's linear velocities on \(x\) and \(y\) axes: \(v_{x},v_{y}\)  
Robot's angular velocity in \(z\) axis: \(\omega_{z}\)  
Object's position in world coordinate system: \(x_{obj}^{w},y_{obj}^{w}\)  
Distance between the robot and the object: \(d(p_{r},p_{obj})\)  
Remaining time steps to end the episode: \(t\)  
\[x_{r}^{w}=[0.0,3.0]\ \ \ \ \ y_{r}^{w}=-1.5\ \ \ \ yaw_{r}^{w}=[-\pi,\pi] \tag{14}\]  
\[x_{obj}^{w}=[0.5,2.5]\ \ \ \ \ y_{obj}^{w}=[0.5,0.75]\ \ \ \ z_{obj}^{w}=1.2 \tag{15}\]  
\[s=[x_{r}^{w}\ \ y_{r}^{w}\ \ yaw_{r}^{w}\ \ v_{x}\ \ v_{y}\ \ \omega_{z}\ \ x_{obj}^{w}\ \ y_{obj}^{w}\ \ d(p_{r},p_{obj})\ \ \ t]\in\mathbb{R}^{10} \tag{16}\]  
The linear and the angular velocities included in the state were the velocities sent in the previous time-step to the robot (previous action). The reward function used is defined in Equation (17). A nonlinear function was used to give higher rewards when the robot was close to the object and high velocities and accelerations were penalised to encourage smooth driving. Here, \(\Delta v\) means the velocity difference (acceleration) between current and previous action. Instead, \(v\_{high}\) penalised each linear or angular velocity being higher than a threshold. As mentioned before, the maximum linear and angular velocities were 1 m/s and 1 rad/s, respectively. In addition, in the last time-step of the episode, an additional reward was given if the arm could plan a trajectory up to the object. The number of remaining time-steps was included in the state for robot to be aware when this last step was coming. The _collision_ variable was equal to 1 when a collision occurred, and 0 otherwise.  
\[r(s_{t},a_{t})=\frac{1}{d(p_{r},p_{obj})}\cdot(1-collision)-0.5\cdot\Delta v-0.5 \cdot v\_{high}(a_{t})+100\cdot success \tag{17}\]  
\begin{table}
\begin{tabular}{l l} \hline
**PPO Setup Hyper-Parameters** & \\ \hline Actor/Critic learning rate & \(f(step)=step\cdot 3\times 10^{-4}\) \\ Clip-range & 0.2 \\ Discount factor \(\gamma\) & 0.99 \\ Batch size & 512 \\ Mini-batch size & 64 \\ Updates &
\begin{tabular}{l} _training time\_steps_ \\ _batch\_size_ \\ 10 \\ \end{tabular} \\ \hline
**DDPG Setup Hyper-Parameters** & \\ \hline Actor’s learning rate & \(1\times 10^{-4}\) \\ Critic’s learning rate & \(1\times 10^{-3}\) \\ Batch size & 128 \\ Discount factor \(\gamma\) & 0.99 \\ Critic l2 regularisation & \(1\times 10^{-2}\) \\ Running epochs & 500 \\ Cycles per epoch & 10 \\ Rollouts per cycle & 512 \\ Updates & _epochs\(\cdot\) cycles_ \\ Training iterations per update & 100 \\ \hline \end{tabular}
\end{table}
Table 2: Hyper-parameters.  
\[v\_high(a_{t})=\sum_{i=1}^{3}a_{t_{i}},\text{ if }a_{t_{i}}>k \tag{18}\]