# Occupancy Reward-Driven Exploration with Deep Reinforcement Learning for Mobile Robot System
## 5 Reward Estimation and Training Options
### The Reward Estimation  
In Section 4.4, we proposed the reward function \(r_{t}\) in Equation (9) for the mobile-robot exploration. The value limit or range for the reward function, which is important to set when using the deep-RL technique, was also discussed. The point is that the RL agent defines the good and bad actions according to the reward function. If the value fluctuates in a chaotic way, then the actor-critic network cannot determine the positive and negative decisions. This is the main reason the value limits for the reward function are defined. When estimating the value, know the upper and lower limits for the parameters in the reward function must be known, which are the linear velocity, the angular velocity, the sensing range distance, and the explored segment.  
In Table 2, the value limits for the reward parameters are presented. It can be seen that the upper limit for the observation is 4.095, which is the maximum sensor range in the simulation. The lower limit is zero. The continuous actions are linear \(v\) and angular \(w\) velocities, with 0.4 as upper limits and 0 and \(-0.4\) as lower limits, respectively. The maximum explored segment \(E\) is 239, which the sensor of the robot can occupy in the probability occupancy map at time \(t\) visiting completely new and free areas. The parameters are normalized in the upper and lower thresholds of the reward function, \(-0.2\) and 0.8, respectively.  
To adjust the parameters, the coefficients \(k\), \(q\), \(d\), \(f\) are introduced in Equation (9). However, the priorities of the four parameters in the reward function are included in the coefficient as well, which affects the robot motion in the exploration process. Thus, the priorities can be described as follows: 30% for linear velocity, \(-20\%\) for angular velocity, 20% for sensing ranges, and 30% for the explored segment. Converting the percentage to numbers, the coefficients are as follows: \(k=0.75\), \(q=-1.25\), \(d=0.07\), and \(f=0.0013\). In real-world applications, the proposed coefficients can be used without changes when the robot is Turtlebot2 and the laser sensor is Hokuyo (model no. urg-04k-ug01). For other cases, the values of the coefficients should be calculated individually according to robot kinematics and sensor specifications.  
Figure 8 illustrates the above discussion on parameters affecting the reward function to a greater and lesser extent and the adjustment in their values in one common range. The lower and upper limits are \(-0.2\) and \(0.8\), respectively.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline  & \(\mathbf{O}\) & \(\mathbf{v}\) & \(\mathbf{w}\) & \(\mathbf{E}\) \\ \hline Upper limit & \([4.095\dots 4.095]^{\prime}\) & 0.4 & 0.4 & 239 \\ Lower limit & \([0\dots 0]^{\prime}\) & 0 & \(-0.4\) & 0 \\ \hline \hline \end{tabular}
\end{table}
Table 2: The value limits for the reward parameters.  
Figure 7: Actor network with 50-D input and 2-D output.  
It is appropriate to clarify the reason why angular velocity (\(w\)) has a negative value in the reward function. When the robot drives straight in an obstacle-free area, \(w\) equals 0. Driving straight is the most optimal motion according to the mapping and the power energy cost. This is why any other values of the angular velocity, \(w<0\) for turning to the right side and \(w>0\) to the left side, are negative for the reward function.