# Rough-Terrain Path Planning Based on Deep Reinforcement Learning
## 3 Path-Planning Algorithm Design
### Design of A-DQN Algorithm Based on Heuristic Exploration
#### 3.3.2 Action Spaces and Reward Functions  
An action space is the set of all possible actions selected by an agent on a map. Action space is divided into discrete action space and continuous action space. This paper pays more attention to the global-path-planning effect of large-scale elevation raster maps, and the DQN works in discrete action space, so we use discrete action space in this paper. The action space of the agent includes eight actions, namely, front, back, left, right, left front, left back, right front, and right back, which are denoted by the numbers 0-7 (i.e., \(A\in\{0,1,2,\ldots,7\}\)).  
In reinforcement learning, a reward function is used to represent how good or bad the selected action is in a given state, helping the agent learn how to optimize its behavior through trial and error to maximize long-term gains. The reward function directly affects the learning process of an agent by guiding it to choose a better behavioral strategy through reward or punishment. A well-designed reward function is the basis for achieving effective learning and optimizing the behavior of an agent.  
In this paper, we aim to balance the path length and energy consumption, so we take information including the distance, number of steps, and slope to design the combination reward. In order to ensure that the agent moves closer and closer to the target point, the reward (r\({}_{1}\)) is set at about the distance from the current point (\(P_{c}\)) to the target point (\(P_{d}\)), and the larger the distance, the larger the penalty:  
\[r_{1}=-D_{2}(P_{c},P_{d})/50 \tag{17}\]  
In order to make the agent travel route smoother and for the tradeoff between the path and energy consumption, a slope penalty is used to evaluate the slope:  
\[r_{2}=-E(P_{c},P_{c+1})\cdot k_{1} \tag{18}\]  
The terrain penalty is defined according to the terrain-type label (\(t_{\textit{terrain}}\)) as follows:  
\[r_{3}=-k_{2}\cdot\ t_{\textit{terrain}} \tag{19}\]  
Upon reaching the goal, an additional reward is provided: r\({}_{4}\) = 50 \(\times\) k\({}_{1}\). The total reward _return_ is calculated as follows:  
\[return=r_{1}+r_{2}+r_{3}+r_{4}\]  
paper is a four-channel graph (\(4\times 50\times 50\)). The four channels represent the elevation map, unmanned vehicle position, end point position, and terrain level. The first convolutional layer contains eight convolution kernels of a \(3\times 3\) size, which are convolved to obtain eight \(50\times 50\) feature maps, and the convolution step is 1 and the padding is 1 in the convolutional operation. The second convolution layer contains 16 convolution kernels of a \(3\times 3\) size, which are convolved to obtain eight \(50\times 50\) feature maps, and the convolution step is 1 and the padding is 1. The second convolutional layer contains 16 convolutional kernels of a \(3\times 3\) size, which are convolved to obtain 16 \(50\times 50\) feature maps; the first fully connected layer has 40,000 nodes, and the 16 matrices output from the second convolutional layer are transformed and spliced into a one-dimensional vector of a \(1\times 40\),000 size; the second fully connected layer contains 256 nodes, and the output layer has 8 nodes representing the Q-values of eight actions.