# Obstacle Avoidance Drone by Deep Reinforcement Learning and Its Racing with Human Pilot
## 4 Method
### Method for Continuous Action Space
#### 4.2.1 Critic-Dependent Segmentation Model
During testing, the segmentation model receives RGB input and then produces a predicted path in terms of reward without further processing indicated as optical flow and feedback from \(\Psi\) as shown in Figure 4. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer** & **Layer Output Dimension** & **Activation** \\ \hline \multicolumn{3}{l}{* Actor network} \\ \multicolumn{3}{l}{Input} & 25 \(\times\) 1 \\ \multicolumn{3}{l}{Fully connected} & 64 \(\times\) 25 \\ \multicolumn{3}{l}{Fully connected} & 3 \\ \hline \multicolumn{3}{l}{* Critic network} \\ \multicolumn{3}{l}{Input} & 25 \(\times\) 1 \\ \multicolumn{3}{l}{Fully connected} & 64 \(\times\) 25 \\ \multicolumn{3}{l}{Fully connected} & 1 \\ \hline \multicolumn{3}{l}{\multirow{2}{*}{Optimizer}} & Adam((betas = 0.9,0.99), \\ \multicolumn{3}{l}{weight decay=0,} \\ \multicolumn{3}{l}{lr = 1e\({}^{-4}\))} \\ \hline \multicolumn{3}{l}{Weight,} \\ \multicolumn{3}{l}{Bias unit} & \multicolumn{3}{l}{Kaiming He,} \\ \multicolumn{3}{l}{Constant(0)} \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Detailed specification of our Actor and Critic networks. During training, all 3 networks such as U-Net, Actor, and Critic networks are required, whereas only the U-Net and Actor network are utilized during testing. Note that all 3 networks, including the U-Net in Figure 5, use identical optimizer, and weight initialization method. Note that * indicates types of networks. Although using policy gradient can be optimal in a certain environment, the problem in using vision input in the continuous action space arises when all the state has high dimensions with a similar appearance. In other words, \(\pi\) and \(\Psi\) cannot effectively learn from succeeding similar frames that require different actions. To overcome such an issue with high dimensional data, we adopt a segmentation model that compresses the information while preserving the useful features as mentioned earlier. Let \(f_{seg}\) and \(\theta_{seg}\) denote the segmentation model and its parameter, then \(s_{t}\) for training our actor \(\pi\) and critic network \(\Psi\) are given as  
\[s_{t}=\{\bar{W}_{i,j},...,\bar{W}_{h,w}\}, \tag{25}\]  
where \(W\) is the set of the values in the zones separating the output of \(f_{seg}\), as Equation (24), and \(\bar{W}\) means an averaged value of \(W\). Our actor and critic networks accept an input that has \(h\times w\) dimension for training and testing. The reward is calculated using the optical flow vector \(V\) and the segmentation model's output. Given \(S_{t}\) and \(S_{t+1}\), \(V\) can be calculated and indicates zones with the direction and speed. Then, the corresponding zones in a segmentation map \(f_{seg}(S_{t}|g_{seg})\) are selected and averaged to become a reward for a given action \(a_{t}\). Apart from U-net generated reward, a reward is given as \(-1\) whenever the drone collides so that the critic can give both \(\pi\) and \(f_{seg}\) negative signals to learn. So that, the reward at time step \(t\) using the segmented output \(f_{seg}(S_{t}|\theta_{seg})\) is computed as follow:  
\[r_{t}= \begin{cases}\frac{1}{m}\sum_{i=1}^{m}\bar{W}_{i}|V_{t}&if\text{ not collides}\\ -1&\text{otherwise}\end{cases} , \tag{26}\]  
where \(\bar{W}_{i}|V_{t}\) stands for an averaged value of a zone \(W_{i}\) indicated by \(t_{th}\) optical flow vector \(V_{t}\). \(m\) is the number of zones where \(V_{t}\) is superposed. Note that the reward value ranges from \(-1\) to \(1\), because of Sigmoid function at the last layer of U-Net as shown in Figure 5. Action \(a_{t}\) produced by the actor network \(\pi\) using \(s_{t}\) in Equation (25) is as follows:  
\[a_{t}=\pi(s_{t}|\theta_{\pi}), \tag{27}\]  
where \(\theta_{\pi}\) stands for the parameter of \(\pi\). With these \(s_{t},a_{t},r_{t},s_{t+1}\) defined above, our actor and critic networks make a typical optimization step in RL by utilizing an experience replay buffer. During  
Figure 5: Specification of our U-Net architecture.