# A Novel Searching Method Using Reinforcement Learning Scheme for Multi-UAVs in Unknown Environments
## 4 Simulation Results
### Independent Random Distribution of Targets  
The nine warship targets were randomly distributed in an unknown sea area with a range of 10 nm \(\times\) 10 nm, which excludes the already controlled sea area and shallow sea area. Each warship carried out its mission independently, and four UAVs searched the warships. The initial positions of UAVs were located at the four corners of the sea area. In the initial stage (\(k=1\) to 120) of RL, due to the lack of prior target information, the whole sea area was searched. As shown in Figure 2, the four curves with different colors are the trajectories of the four UAVs. Because the UAVs did not know the specific location, shape, or size of the shallow area, they would slip into the shallow area for a search. However, the targets were unable to enter these areas. With continuous learning, the information masteered by the UAVs increased, and the shallow area (represented by \(\times\)), the sea area controlled by the red side (circular sea area), and the position of warships (pentagonal stars and diamonds in the Figure 2 ) were gradually identified. Figure 3 shows the search trajectories of UAVs over a period of time with the increase in learning time. UAVs detected shallow water with a size of 1.00 nm \(\times\) 2.50 nm, and the horizontal and longitudinal coordinate ranges were respectively 46 to 51 and 25 to 39. Meanwhile, the circular sea area controlled by the red side with a center coordinate of (10, 35) and a radius of 0.83 nm was also detected. It can be seen that, after learning and understanding the sea area information, UAVs no longer searched the sea area where it was impossible for the target ship to appear. Figure 4 shows the sea area information mastered by the UAVs after the search mission, including the sea area covered by the shallow area, and the nine warships cruising along a straight line. Figure 3: Sea area information learning search. Figure 2: Reinforcement learning (RL) search initial stage. _Case 1--Dynamic targets_: Under this condition, the search effect based on the Q-learning algorithm was compared with random search and traversing search, and the Monte Carlo method was used for 500 experiments. Random search is a non-cooperative search method. When the UAV makes a decision, it randomly selects a direction to search. The traversing search is a fixed search mode. Full coverage of the area can be achieved as long as the duration of flight is allowed. The simulation diagrams of random search and traversing search are shown in Figures 5 and 6. The comparison results are shown in Table 1. The numbers of targets found by the three search methods were compared, and it is clear that the efficiency of the RL search was the highest; at each statistical moment, is the value was about one more than that of the random search. Over time, traversing searches may also find all targets, but the efficiency is extremely low because the targets are moving. This result is due to the fact that the random search and traversing search do not update the search map and, thus, search for invalid areas. At this point, the RL search shows obvious advantages. Figure 4: Unmanned aerial vehicle (UAV) search result under proposed method. Figure 5: Random search result. _Case 2--Static targets:_ An experiment with the initial probability of targets according to a uniform distribution was designed to verify the search efficiency of the algorithm for different numbers of targets. In total, 500 simulations were carried out to obtain the average number of targets found in various cases, and the running time of each simulation was 2000 s. The search results of the three methods for different numbers of targets are compared in Table 2.  
In this scene, as long as the search track of the UAVs can completely cover the whole mission area \(E\), most of the targets can be found in the case of reliable detection by airborne sensors.