## 2 APF-D3 QNPER Algorithm Design
### Network Structure of Improved DQN Algorithm  
The network structure of the improved DQN algorithm is illustrated in Figure 2. The input to the network consists of three components: visual image information, state information, and APF action output. The state information incorporates lidar point cloud data and vehicle state information \(\theta_{t},v_{t},w_{t}\). The visual image information is processed using a convolutional neural network to extract relevant feature information. The lidar point  
Figure 1: Algorithm flow.  
cloud information within the state information is passed through a LSTM network, which effectively captures dependencies across a long sequence of segments and strengthens the correlation between data. The action pairs output by the APF algorithm, serving as prior knowledge, undergo feature extraction through several fully connected layers. The network architecture incorporates the Dueling DQN algorithm, with fully connected layers producing the action dominance function and the value function. By combining these outputs, the algorithm generates the Q-value matrix for the action set A corresponding to the given state.  
For visual image information, the improved DQN algorithm employs three convolutional layers for feature extraction. The dimensions of these layers are \(8\times 8\times 4\), \(4\times 4\times 32\), and \(3\times 3\times 64\), respectively. The number of output channels for the convolutional layers is 32, 64, and 64, respectively. The stride sizes for these layers are 4, 2, and 1, respectively. The convolutional layers utilize the \"SAME\" padding scheme to handle the image edges. Regarding the state information, the algorithm utilizes an LSTM network with a cell size of 256. The LSTM network is responsible for capturing the dependencies in the sequence of lidar point cloud data and reinforcing the correlation between the data segments. In an unknown environment, the LSTM can capture state information at different time steps to build a dynamic model of the environment. By passing current state, action, and reward information to the LSTM, it can learn the time-varying characteristics of the environment, enabling the intelligence to better understand the changes in the environment. The long-term memory properties of the LSTM allow it to remember previous states and action sequences so that long-term effects can be taken into account in path planning. This is useful for making decisions in complex environments where path planning often requires considering possible effects after multiple steps. LSTMs can consider both temporal and spatial information, thus improving the spatio-temporal sensitivity of the algorithm. In path planning, intelligence can predict possible future states and paths as well as select the best action based on the current state and historical trajectories. For the APF value information, two fully connected layers are employed for feature extraction. These layers consist of 200 and 100 neurons, respectively. The rectified linear unit (ReLU) function serves as the activation function for all layers. After processing the input data from the visual image, state information, and APF value, the Concat function is used to flatten the data into a one-dimensional vector. This vector is then split into two parts after passing through a fully connected layer. The output size for the action part is \(1\times 9\), and for the value part, it is \(1\times 1\). In visual image information processing, the relu activation function is mainly used; in LSTM networks, both activation functions sigmoid and tanh are used.  
The ReLU activation function was born in 2011 and has been widely used in deep learning since its birth, as shown in Figure 3. The expression of the ReLU activation function is shown in Equation (15), which speeds up the training speed of the network, and the gradient is 1 when \(x>0\), which can solve the problem of gradient vanishing. But, when  
Figure 2: Network structure of improved DQN algorithm.