# A Novel Searching Method Using Reinforcement Learning Scheme for Multi-UAVs in Unknown Environments
## 1 Introduction  
With the rapid development of sensors, wireless communication, intelligent control, and other technologies, the functions and the application fields of unmanned group systems are increasing day by day. Because of their expansibility, strong cooperation, and low loss, the cooperative theory and applied research on unmanned group systems received increasing attention in the fields of academia, industry, and national defense [1]. Multi-unmanned aerial vehicle (UAV) cooperative search systems can effectively improve the search efficiency, especially for the search of dynamic targets under complex sea conditions such as uncertainty, strong interference, and so on. Therefore, multi-UAV cooperative sea area searching is one of the important research directions of unmanned group systems [2]. The multi-UAV area target search problem was widely studied by research groups around the world. Generally, the target search problem is divided into two categories: static target search and moving target search. For static targets, the traditional search method is a covering search (e.g., echo search, traversing search, etc.) [3; 4; 5; 6]. This search method generally maximizes the coverage of the task area to find as many targets as possible. In References [6; 7], a search map model was established according to the existence probability of target, and distributed model predictive control was used to solve the problem, which effectively reduced the solution scale of the search decision problem. In Reference [8], considering the limitations of sensing and communication capabilities, coverage and topology control algorithms were designed for the path planning of mobile agents. For dynamic targets, a Bayesian method was used to calculate the average detection time and average detection probability [9; 10], but it was only suitable for searching a single target. In Reference [11], a target motion prediction model based on Markov chain was proposed, and a greedy iterative decision methodbased on distributed model predictive control was designed to solve the problem. For a mobile target in a closed and bounded region, a receding-horizon cooperative search algorithm was presented [12]. With the goal of minimizing search time, a strategy of \"the reward of discount time\" was proposed, and a cross-entropy optimization algorithm was used to find the location where the target has the highest probability as soon as possible [13]. In Reference [14], the problem of scheduling multiple UAVs to search for missing tourists was addressed, and a method for estimating tourist location probabilities which change with topographic features, weather conditions, and time was proposed. In addition, some intelligent algorithms were also applied to such problems. In Reference [15], a novel objective function that naturally and coherently integrates the conflicting objectives of target detection, target tracking, and vehicle survivability into a single scalar index was designed, and a modified particle swarm optimization algorithm was used to determine which trajectory is the best on average at detecting and tracking targets. Combined with the traditional Maximum-Q-learning (MAX-Q) algorithm, a multi-UAV cooperative search strategy was proposed, which effectively completes the search tasks of independent targets and clustering targets but is limited to static targets [16; 17]. In References [18; 19], a reinforcement learning (RL) algorithm was used for path planning without colliding with obstacles in unknown environments. In order to solve the above problems, this paper proposes a multi-UAV cooperative search method based on RL, which fully considers the characteristics of an unknown sea area. It synthesizes a target probability map and certainty value map to establish a multi-UAV sea area search map, and the concept of the territory awareness information map is proposed to coordinate the tracks between multiple UAVs. The search efficiency function is obtained according to the extended search map.