# Obstacle Avoidance Drone by Deep Reinforcement Learning and Its Racing with Human Pilot
## 3 Deep Reinforcement Learning and U-Net Segmentation Model
### Learning in Discrete Action Space  
In discrete action space, an agent chooses to act according to a policy \(\pi\), which usually has a form of greedy learning. Here, \(\pi\) is equal to a state-value function \(V\), since \(\pi\) can be seen to learn by greedily choosing the best action given a state. One of the popular forms for estimating the value is \(Q\) function. The connection between \(Q\) and \(V\) can be expressed as  
\[ \begin{array}{l}Q^{\pi}(s,a)=E[R|s_{t}=s,a_{t}=a,\pi]\\ V^{\pi}(s)=E_{a\sim\pi(s)}[Q^{\pi}(s,a)]\end{array} , \tag{3}\]  
where \(a\) is a chosen action given a state \(s\). Then, by defining the optimal \(Q\) function as  
\[Q^{*}(s_{t},a_{t})=max_{\pi}Q^{\pi}(s_{t},a_{t}), \tag{4}\]  
\[a=argmax_{a_{t+1}\in A}Q^{*}(s_{t+1},a_{t+1}), \tag{5}\]  
and  
\[V^{*}(s_{t})=max_{a}Q^{*}(s_{t},a_{t}), \tag{6}\]  
Optimal \(Q\) function satisfies the Bellman equation:  
\[Q^{*}(s_{t},a_{t})=E_{s_{t+1}}[r+\gamma max_{a_{t+1}}Q^{*}(s_{t+1},a_{t+1})[s _{t},a_{t}]. \tag{7}\]  
For high dimensional data, i.e., image, the Deep Q Network (DQN) can effectively learn to solve a given task [6]. Loss function for optimizing the DQN:  
\[Loss=||y_{t}^{DQN}-Q(s_{t},a_{t};\theta)||, \tag{8}\]  
\[\gamma_{t}^{DQN}\equiv r_{t+1}+\gamma maxQ(S_{t+1},a_{t+1},\theta), \tag{9}\]  
where \(\theta\) is the parameters of a network for a current optimization step. However, it had been shown that using the on-line parameter \(\theta\) for learning could cause unstable learning in vanilla DQN. To solve this issue, Double DQN [6] has been proposed. In Double DQN, a target network is copied for everypre-defined time step and is used to estimate the target value in the optimization step. Therefore, the target value \(Y\) in time step \(t\) in Double DQN is as follows:  
\[Y_{t}^{DoubleDQN}\equiv r_{t+1}+\gamma maxQ(S_{t+1},a_{t+1},\theta^{-}), \tag{10}\]  
where \(\theta^{-}\) stands for the fixed parameters of the target network,  
\[e_{t}=(e_{t},a_{t},r_{t},s_{t+1}), \tag{11}\]  
\[D=e_{1},e_{2},...,e_{t}, \tag{12}\]  
To delete the correlation of every experience accumulated in the buffer, which causes significant deterioration of the stability of the network, DQN utilizes a technique called experience replay. Equations (11) and (12) show how experiences are accumulated in the buffer, where \(s_{t}\) is a state, \(a_{t}\) is an action, and \(r_{t}\) is a reward. With experience replay, the network uses randomly sampled mini-batches of experiences from D instead of using the data that is accumulated sequentially. An improvement of DQN, called Double DQN [8] is proposed. To avoid the problem of over-optimistic estimation of \(Q\) value, Double DQN uses different values to select and evaluate an action while DQN uses the same values for selecting and evaluating values:  
\[Y_{t}^{Double}=r+\gamma Q(s_{t+1},argmaxQ(s_{t+1},a_{t+1},;\theta);\theta^{-}). \tag{13}\]  
Instead of using different values to select and evaluate the action, Dueling Deep Q-network (Dueling DQN) [41] separates networks into a value network and an advantage network. The value network is used to estimate the quality of the state while the advantage network is used to estimate the qualities of each action,  
\[a=argmax_{a_{t+1}\in A}Q^{*}(s_{t+1},a_{t+1}), \tag{5}\]0 \[a=argmax_{a_{t+1}\in A}Q^{*}(s_{t+1},a_{t+1}), \tag{5}\]1  
Among several equations to estimate \(Q(s_{t},a_{t};\theta,\alpha,\beta)\) in Dueling DQN, we adopt the Equation (14) as it is known to stabilize the optimization process. Given that there are two ways to improve the performance of DQN in previous sections, we suggest in this paper that there is a better way for an agent to learn by combining Double DQN with Dueling DQN, called it Double Dueling DQN (DD-DQN). In our experiment, we use Equation (14) to calculate \(Q\) value as it is known to be robust for removing correlation.