# Autonomous Driving Control Based on the Perception of a Lidar Sensor and Odometer
## 3 DRL Methods  
In this section, DRL methods are introduced. Specifically, the study considers autonomous control of a UGV in order to achieve the goal of intelligent path planning and obstacle avoidance. All existing DRL concepts that have been proposed in the literature are frameworks only, and therefore design work and experiments are still required in order to realize a specific application. This is the main contribution of this study. In Figure 1, the block diagram detailing the relationship between Gazebo and the methods used is presented. This is the practical topology used in the ROS environment and experiments, where the orange oval-shaped nodes are main modules that interact with each other repeatedly via the topics (shown in green rectangles) that provide some specific information. First, the communication interface of the Gazebo simulator is connected to node \(/\)_gazebo_. Then, \(/\)_gazebo_ provides node \(/DRL\) with messages through the following three topics: \(/\)_gazebo\(/\)_model\(_\)states_, \(/\)_odom_, and \(/\)_scan_. More specifically, \(/\)_gazebo\(/\)model\(_\)states_ provides the coordinates of the goal, the current position of the UGV, and other information about the simulation environment, \(/\)_odom_ provides the pose and the current position accumulated by the odometer, and \(/\)_scan_ provides the information about the simulation environment obtained by the lidar sensor mounted on the UGV. Note that the lidar sensor is a 2D laser scanner capable of sensing 360 degrees that collects a set of data around the UGV, and then the lidar and odometer data are used to obtain the distance and heading information for the training and testing procedures of the DRL algorithms. Next, according to above-mentioned simulation environment information, \(/DRL\) sends the action instructions based on the control strategy of the corresponding DRL method as feedback to \(/\)_gazebo_ via the topic \(/\)_cmd\(\_\)vel_. Finally, \(/\)_gazebo_ operates the UGV and updates all the states of the UGV and the simulation environment. Again, based on the ROS architecture, for the applications of different DRL approaches (DDPG and DQN), it only needs to replace the source code in \(/\)_DRL_ with another example. The design of the reward mechanism is divided into the following three parts:  
1. Non-collision;
2. Collision;
3. Reaching the goal. Obviously, for the states of collision and reaching the goal, the current procedure will be terminated. The state of non-collision is formulated as shown below:  
\[reward=(C*\frac{1}{\textit{distance}})-\textit{heading}, \tag{3}\]  
Figure 1: The practical topology used in the experiments. where \(C\) is a factor chosen empirically according to the simulation settings, _distance_ is the distance between the goal and the UGV, defined as  
\[distance=\sqrt{(x_{goal}-x_{UGV})^{2}+(y_{goal}-y_{UGV})^{2}}, \tag{4}\]  
and _heading_ is the angle difference between the goal and the UGV, given by  
\[heading=|goal_angle-yaw_{UGV}|, \tag{5}\]  
in which \(yaw_{UGV}\) is the movement around the yaw axis of the UGV and \(goal_angle\) can be calculated as:  
\[goal_angle=tan^{-1}(\frac{y_{goal}-y_{UGV}}{x_{goal}-x_{UGV}}). \tag{6}\]  
Note that the main concepts of the reward architecture in Equation (3) are:  
* _distance_: Putting _distance_ in the denominator means that getting closer to the goal will result in a higher reward;
* _heading_: Subtracting _heading_ means that facing the goal will result in a higher reward;
* _variable_\(C\): A factor that should be determined empirically. For different settings, different values of \(C\) may induce different performances. In the simulation, \(C=5\). Since the action space of DQN is discrete, the action mechanism devised for the UGV in the experiments includes fewer action types, e.g., the rotation mode is simulated with fixed states. On the other hand, the action space of DDPG is continuous, and the action mechanism devised for the UGV in the experiments includes more/continuous/infinite action types in the simulation.