# Obstacle Avoidance Drone by Deep Reinforcement Learning and Its Racing with Human Pilot
## 4 Method
### Method for Continuous Action Space
#### 4.2.1 Critic-Dependent Segmentation Model  
Our actor-critic networks aim to assist the segmentation network by generating a label map. There are two steps for generating the training data in terms of predicted reward. The first is to recognize which direction the agent chooses to go within an image. An optical flow algorithm [49] is adopted in calculating these vectors. The second is to generate a label map for the segmentation model based on the predicted reward using the optical flow vectors. To measure the direction where the RL agent chooses to go, we calculate optical flow vectors using two sequential images from the environment, \(S_{t}\) and \(S_{t-1}\). Here, \(S_{t}\) stands for the raw RGB FPV on the time step \(t\). Then the equation we use to calculate x movement \(V^{{}^{\prime}}_{x}\) and y movement \(V^{{}^{\prime}}_{y}\) is given as follow:  
\[ \begin{bmatrix}V^{{}^{\prime}}_{x}\\ V^{{}^{\prime}}_{y}\end{bmatrix} = \begin{bmatrix}\sum_{i}I_{x}(q_{i})^{2}&\sum_ {i}I_{X}(q_{i})^{2}I_{y}(q_{i})\\ \sum_{i}I_{y}(q_{i})I_{x}(q_{i})&\sum_{i}I_{y}(q_{i})^{2}\end{bmatrix} ^{-1} \begin{bmatrix}-\sum_{i}I_{x}(q_{i})I_{t}(q_{i})\\ -\sum_{i}I_{y}(q_{i})I_{t}(q_{i})\end{bmatrix} , \tag{22}\]  
where \(q_{i}\) denotes a pixel value inside a window, and \(I\) stands for the partial derivative of \(S_{t}\) and \(S_{t-1}\) with respect to the position \((x,y)\). The size of the window for calculating optical flow is set to \(10\) in this study. To get the vectors indicating where the drone moves, instead of the movement itself, we rotate \(V^{{}^{\prime}}\) using a rotate matrix as follow:\[ \begin{bmatrix}V_{x}\\ V_{y}\end{bmatrix} = \begin{bmatrix}cos\theta&sin\theta\\ -sin\theta&cos\theta\end{bmatrix}\begin{bmatrix}V_{x}^{\prime}\\ V_{y}^{\prime}\end{bmatrix} , \tag{23}\]  
where \(\theta\) is set by 180 to obtain aforementioned drone's movement. Using \(V\) calculated above, the second process is to make the label map. Using parallel displacement of vectors, we first move \(V\) to the center of the image, so that the calculation of the direction and speed for generating the label map could be more straightforward. After this displacement, as shown in Figure 3, we separate an array into zones with directions that \(V\) can have. Then, each direction of the vector and scaled speed are superposed on the zones. We use coefficient \(\eta\), which is set to 20 in this study, for the speed value to calculate how many zones are superposed under each vector's direction. For every vector, an array filled with zero is initialized, and zones under a vector are filled with value one. Each of these arrays is flattened into 1-Dimension and fed into the critic network for estimating the direction. Let \(W_{ij}\) denotes a set of values exhibited in a zone at \(i_{th}\) row and \(j_{th}\) column, and \(V_{n}\) denotes \(n_{th}\) optical flow vector, then each sample \(C_{n}\) used as input for a critic network to make a label map are given as  
\[C_{n} =\{\bar{W}_{i,j},...,\bar{W}_{h,w}\}, \tag{24}\]  
where \(\bar{W}_{i,j}\) means an average value of \(W_{i,j}\). \(h\) and \(w\) are the number of zones for height and width, respectively. Then, each \(C_{n}\) is fed into the critic network \(\Psi\) to receive its predicted reward. The label map can be produced by filling a zeros array with the output values of \(\Psi\) corresponding to zones indicated by \(V_{n}\) of every sample \(C_{n}\). Note that filling the windows with the values from \(\Psi\) suggests each window in a label map will have the predicted reward since the critic network learns to predict the reward (Section 3.2). Figure 3 shows the label map generated with this sequential process. Input dimension for actor and critic networks is 25 with \(h\) and \(w\) both set to 5, as shown in Table 1. Given \(S_{t-1}\) as an input and the corresponding label map described above as a target, our segmentation model learns simultaneously with actor-critic networks. As the RL model performs well, the segmentation model improves as well.