# An Enhanced Deep Q Network Algorithm for Localized Obstacle Avoidance in Indoor Robot Path Planning
## 3 Implementation Based on PER-D2 MQN Algorithm
### Theoretical Foundation  
Markov decision process (MDP) is a mathematical framework for RL problems. It provides a structured method to describe the environment in RL and defines the rules for the agent to make decisions in the environment, as shown in Figure 2. Although MDP is a widely used model for decision-making problems, in the real world, we usually cannot fully observe the entire state of the environment. For example, a robot's 2D LIDAR sensor can only provide limited distance information and cannot fully describe its environment. In this case, using POMDP [33; 37] to define the problem is more in line with the actual situation. Specifically, POMDP can be described as a tuple \(<S.A.T.R.\gamma.\Omega.O>\). The elements of the tuple are interpreted as follows:  
\(S\): All states of the environment; \(\forall s\in S\)  
\(A\): All actions the agent can take; \(\forall a\in A\)  
\(T\): \(S\times A\rightarrow\pi(S)\) is the state transition function, which gives the probability that the agent takes action \(a\) in a certain state and ends up in state \(s\); \(T(a,s,s^{\prime})=\pi(s^{\prime}|s,a),\forall s,s^{\prime}\in S\) and \(\forall a\in A\)  
\(R\): \(S\times A\rightarrow\mathbb{R}\) is the instant reward;  
\(\gamma\in[0,1]\) is the discount factor;  
\(\Omega\) is all the information that the agent may observe;  
\(O\): \(S\times A\rightarrow\pi(\Omega)\) is an observation function that gives the probability of observing o if the agent takes action \(a\) and ends up in state \(s^{\prime}\); \(O(s^{\prime},a,o)=\pi(o|s^{\prime},a),\forall s,s^{\prime}\in S\) and \(\forall a\in A\) and \(\forall o\in\Omega\).  
As a typical RL algorithm, the Q Learning algorithm can iteratively estimate the optimal action value function \(q_{*}(\text{s},\text{a})\) by updating Equation (1):  
\[q(s_{t}a_{t})\gets q(s_{t},a_{t})+\alpha[r_{t}+\gamma\max_{a^{\prime}}q_{* }(s_{t+1},a^{\prime})-q(s_{t},a_{t})] \tag{1}\]  
where \(q(s_{t},a_{t})\) denotes the current estimated value of taking action \(a_{t}\) in state \(s_{t}\); \(s_{t}\) denotes the state of the intelligent body at time step \(t\); \(a_{t}\) denotes the action taken by the intelligent body at time step \(t\); \(r_{t}\) denotes the immediate reward obtained by the agent at time step \(t\); \(s_{t+1}\) denotes the new agent at time step \(t+1\) state; \(\alpha\) is the learning rate, which controls the learning speed; \(\gamma\) is the discount factor, which weighs the importance of immediate and future rewards. In practice, the optimal value \(q\) function \(q_{*}\) is usually unknown, and we can only use the value function \(q(s_{t},a_{t})\) of the current strategy in place of the value function \(q_{*}\) of the optimal strategy. This process is often called bootstrapping. In short, bootstrapping means updating the current estimate \(q(s_{t},a_{t})\) to a new estimate \(q(s_{t+1},a^{\prime})\), i.e., improving one's own estimate by using the current estimate.  
Q Learning algorithms are inefficient and difficult to scale when faced with complex and multi-state real-life problems. DRL provides an effective way to solve this problem  
Figure 2: Markov-based RL process.  
under the POMDP framework: DQN, which approximates the Q value function by training an Artificial Neural Network (ANN). The trained ANN model serves as a function approximator for the Q function, capable of managing complex state spaces. The agent chooses actions based on observation vectors from the environment, utilizing a model with a Deep Neural Network (DNN) architecture.