## 2 APF-D3 QNPER Algorithm Design
### APF-Based Action Output  
In simple environments, the DQN-based path planning algorithm can effectively utilize lidar and deep vision information as inputs to the network model. Through continuous interaction with the environment, the network parameters can be trained until convergence is achieved. However, in complex environments, the optimization of a large number of network parameters becomes more challenging, leading to slower or even difficult convergence. In complex and unknown environment path planning scenarios, where dynamic obstacles with unknown locations and unpredictable paths exist, DRL algorithms may encounter states that were not encountered during the training process. As a result, the planning results generated by these algorithms may yield unsatisfactory return values. To solve this problem, an APF-based action output method is proposed. In this paper, according to the combined force F of the artificial potential field algorithm, the required outputs \(\left(v_{x},v_{y}\right)\) of the unmanned vehicle are output from the \(x\)-axis and y-axis directions, respectively; to ensure the dynamics constraints of the unmanned vehicle, the combined force is firstly standardized, as shown in Equations (8)-(10). \[F_{norm}=\sqrt{F_{x}^{2}+F_{y}^{2}} \tag{8}\]  
\[F_{x}=F_{x}/F_{norm} \tag{9}\]  
\[F_{y}=F_{y}/F_{norm} \tag{10}\]  
where \(F_{x}\) and \(F_{y}\) denote the combined force in the \(x\)-axis and y-axis directions of the unmanned vehicle, which is less than one for both after normalization. According to the combined force, the velocity pair \(\left(v_{x},v_{y}\right)\) is the output, as shown in Equations (11) and (12). \[v_{x}=k_{x}\times F_{x}\times v_{lim}+\left(1-k_{x}\right)*v_{x0} \tag{11}\]  
\[v_{y}=k_{y}\times F_{y}\times v_{lim}+\left(1-k_{y}\right)*v_{y0} \tag{12}\]  
where \(k_{x}\) and \(k_{y}\) denote the weight parameters of the force field, and \(\left(v_{x0},v_{y0}\right)\) denotes the current velocity pair of the unmanned vehicle. According to the difference between the angle of the velocity pair \(\left(v_{x},v_{y}\right)\) and the yaw angle of the unmanned vehicle itself, the required steering angular velocity \(w_{apf}\) is calculated, and the steering angular velocity is limited to the interval of \(\left(-w_{lim},w_{lim}\right)\); and the value of the linear velocity v in the direction of the combined force of the velocity pair \(\left(v_{x},v_{y}\right)\) is calculated, and the linear velocity is limited to the interval of \(\left(-v_{lim},v_{lim}\right)\). The linear velocity \(\left(v_{apf},w_{apf}\right)\) is limited to the \(\left(-v_{lim},v_{lim}\right)\) interval as the final output of the APF algorithm. First, we utilize the lidar information, along with the starting point and target point location information, the autonomous vehicle's positional information, and the vehicle kinematic model as inputs to the APF algorithm. The APF algorithm outputs \(\left[v_{apf},w_{apf}\right]\) information, representing the desired linear and angular velocities of the autonomous vehicle in that state. This output information from the APF algorithm is then employed as prior knowledge in the DQN algorithm. It guides the direction of the optimization process for the state action value function. When the model training is finished, the DQN algorithm usually performs action selection based on the maximum output of the state action value function, as shown in Equation (13). \[a_{max}=max_{a}Q(s,a;\theta) \tag{13}\]where \(\theta\) is the neural network model parameters; s is the state s, in which the intelligent body is in; \(a\) is the action taken by the intelligent body in that state; \(max_{a}Q(s_{t},a;\theta)\) is under the trained network model; and the DQN algorithm selects the action \(a\) that makes the \(Q\) function maximum in state s, when \(a=a_{max}\).