# A Novel Searching Method Using Reinforcement Learning Scheme for Multi-UAVs in Unknown Environments
## 2 Materials and Methods
### Environmental Model  
The environment \(E\) is represented as an \(L_{x}\times L_{y}\) sea area, and the information for the search map can be defined as \(P_{mm}(k)=[\begin{array}{cc}p_{mm}(k)&\chi_{mm}(k)\end{array}]\), where \(p_{mm}(k)\in[0,1]\) is the existence probability of the targets on the grid \((m,n)\) with \((m\in\{1,2,\ldots,L_{x}\},n\in\{1,2,\ldots,L_{y}\})\) at instant \(k\). \(\chi_{mm}(k)\in[0,1]\) is the certainty value, where \(\chi_{mm}(k)=1\) represents that the UAVs fully understand the target information, and \(\chi_{mm}(k)=0\) represents that the UAVs have no information on the target.  
_Target probability map_: Before the search task begins, the search map is given a certain initial value, which reflects the prior information of the target (obtained by external intelligence reconnaissance). With the continuous search, the search map information mastered by the UAVs is constantly updated.  
Considering the influence of sensor uncertainty, the updating equation of the target probability was designed as follows:  
\[p_{mn}(k+1)=\left\{\begin{array}{ll}\tau p_{nm}(k)+\Delta p_{nm}(k)&no\;access \\ \frac{pp_{mn}(k)}{p_{F}+(p_{D}-p_{F})p_{mn}(k)}&access\cap b_{k}=1\\ \frac{(1-p_{D})p_{mn}(k)}{1-p_{F}+(p_{F}-p_{D})p_{mn}(k)}&access\cap b_{k}=0 \end{array}\right., \tag{1}\]  
where \(p_{D}\) and \(p_{F}\) are the sensor detection rate and false alarm rate, respectively. \(\tau\in[0,1]\) denotes a discount factor, which represents the forgetting factor of the probability map. Since only a few grids are accessed at the same time, and these grids' probability change affect other grids, the changing probability \(\Delta p_{mn}(k)\) in grid \((m,n)\) is defined as  
\[\Delta p_{mn}(k)=\sum_{(i,j)\in D(k)}[p_{ij}(k)-p_{ij}(k+1)]/(L_{x}\times L_{y} -N_{v}), \tag{2}\]  
where \(D(k)\) is a set of all accessed grids at time \(k\). When UAVs access \((m,n)\), the update of \(p_{nm}(k)\) is related to the detection variable \(b_{k}\) of the platform sensor. \(b_{k}=1\) represents that the airborne sensor detected the target, and \(b_{k}=0\) represents that the sensor did not detect the target.  
_Certainty value map_: As the search task proceeds, UAVs have a constant understanding of the search area. The certainty value map reflects the degree of understanding of the whole map. Due to the probability of sensor detection and false alarm, \(\chi_{mn}(k)\) is used to represent the degree of determination of the information at the grid \((m,n)\) by UAVs at time \(k\). When the grid is not accessed, the certainty of the grid decreases. As the number of times the grid is detected increases, the degree of determination of the information at the grid is increased by the UAVs. The update equation is as follows:  
\[\chi_{mn}(k+1)=\left\{\begin{array}{ll}\tau_{c}\chi_{mn}(k),&no\;access\\ \chi+(1-\chi)\chi_{mn}(k),&access\end{array}\right., \tag{3}\]  
where \(\tau_{c}\) is the information factor of certainty; \(\chi\in[0,1]\) is a constant, and its value is related to sensor performance. \(\chi=1\) represents that the UAV can fully grasp the information when it carries out information testing; that is, \(p_{D}=1\) and \(p_{F}=0\). \(\chi=0\) represents that the information is not available, which is equivalent to the complete failure of the sensor.