# An Enhanced Deep Q Network Algorithm for Localized Obstacle Avoidance in Indoor Robot Path Planning
## 3 Implementation Based on PER-D2 MQN Algorithm
### Per-D2 mqn  
The bootstrap mechanism in RL based on time difference learning usually uses the estimated value of the current state to replace the true value of the next state. M-DQN introduces an additional guidance signal \"log-policy\". If the optimal deterministic policy \(\pi_{*}\) is known, after the policy is \"log\" operated, the probability of selecting the optimal action is 0, and the probability of selecting the non-optimal action is \(-\infty\). Such a guidance signal is more advantageous in policy selection because it can effectively suppress the selection of non-optimal policies. However, the optimal policy \(\pi_{*}\) is usually unknown, so \(\pi\) is used instead of \(\pi_{*}\) and the random policy is assumed to satisfy numerical stability. The loss function of the DQN can be represented by \(\hat{\mathbb{E}}_{B}\bigg{[}\Big{(}q_{\theta}(s_{t},a_{t})-\hat{q}_{dqn}(r_{t },s_{t+1})\Big{)}^{2}\bigg{]}\), where the regression target of DQN \(\hat{q}_{dqn}\) can be expressed as Equation (7):  
\[\hat{q}_{dqn}(r_{t},s_{t+1})=r_{t}+\gamma\sum\limits_{a^{\prime}\in A}\pi_{ \overline{g}}(a^{\prime}|s_{t+1})q_{\overline{g}}(s_{t+1},a^{\prime})\ with\ \pi_{\overline{g}}\in\mathcal{G} \big{(}q_{\overline{g}}\big{)} \tag{7}\]  
DQN can derive M-DQN by modifying the regression objective, but the M-DQN adopts a stochastic policy (\(softmax\) policy, satisfying \(\pi=sm(q)\Leftrightarrow\pi(a|s)=\frac{exp(s,a)}{\sum\limits_{a^{\prime}\in \mathcal{G}\backslash\overline{g}}(a^{\prime})}\)), whereas DQN computes a deterministic policy (\(greedy\) policy, satisfying \(\pi(a|s)=1\) for \(a\in argmax_{a^{\prime}}q(s,a^{\prime})\) for \(\pi\in\mathcal{G}(q)\)). A simple solution is to maximize not only the return but also the entropy of the resulting strategy, i.e., to adopt the maximum entropy RL viewpoint by introducing the Soft Actor-Critic (SAC) [14] algorithm into the DQN to form the Soft-DQN (S-DQN), whose regression objective is represented by Equation (8):  
\[ \begin{array}{ll}\hat{q}_{s-dqn}(r_{t},s_{t+1})=&r_{t}+\gamma\sum\limits_{a^ {\prime}\in A}\pi_{\overline{g}}(a^{\prime}|s_{t+1})\\ &\big{(}q_{\overline{g}}(s_{t+1},a^{\prime})-\tau\ln\pi_{\overline{g}}\big{(} a^{\prime}|s_{t+1}\big{)}\big{)}\ \text{with}\ \pi_{\overline{g}}=sm\big{(}q_{\overline{g}}/\tau\big{)}\end{array}  \tag{8}\]  
where \(\tau\) is the temperature coefficient scaling the entropy, \(r_{t}\) is the instant reward, \(\gamma\) is the discount factor, \(a^{\prime}\) is the action at moment \(t+1\), \(s_{t+1}\) is the state at moment \(t+1\), and \(A\) denotes the action space. When _limit_\(\tau\to 0\), S-DQN is denoted as DQN. The regression target of M-DQN is to add scaled log-policy to the instant reward of S-DQN, i.e., \(\alpha\tau ln\pi_{\overline{g}}(a_{t}|s_{t})\). The specific expression is shown in Equation (9):  
\[ \begin{array}{ll}\hat{q}_{M-dqn}(r_{t},s_{t+1})=&r_{t}+\alpha\tau\ln\pi_{ \overline{g}}(a_{t}|s_{t})+\gamma\sum\limits_{a^{\prime}\in A}\pi_{\overline{ g}}\big{(}a^{\prime}|s_{t+1}\big{)}\\ &\big{(}q_{\overline{g}}\big{(}s_{t+1},a^{\prime}\big{)}-\tau\ln\pi_{\overline {g}}(a^{\prime}|s_{t+1})\big{)}\ \text{with}\ \pi_{\overline{g}}=sm\big{(}q_{\overline{g}}/\tau\big{)}\end{array}  \tag{9}\]  
Setting \(\alpha\in[0,1]\) as the scaling factor and when \(\alpha=0\), \(\hat{q}_{M-dqn}=\hat{q}_{s-dqn}\). Then, the loss function of M-DQN can be represented as \(\hat{\mathbb{E}}_{B}\bigg{[}\Big{(}q_{\theta}(s_{t},a_{t})-\hat{q}_{M-dqn}(r_ {t},s_{t+1})\Big{)}^{2}\bigg{]}\). Experiments in the reference [15] demonstrate that M-DQN outperforms any of the distributed RL algorithms on Atari games and show that M-DQN is suitable for any TD-based RL. PER-D2 MQN firstly adopts the PER mechanism to assign priorities to each experience, samples are weighted according to the priorities, and the TD errors of the experiences are used to update the priorities; then, the network structure of Duel DQN is utilized to improve the computational efficiency of Q-values.