Multiple Cooperative Searchers
for Aerial Surveillance
Satoshi Hoshino and Koya Kikuchi
Utsunomiya University, 7-1-2 Yoto, Utsunomiya, Tochigi 321-8585, JAPANhosino@cc.utsunomiya-u.ac.jp

Abstract. In this paper, we focus on an aerial surveillance system composed of single sentinel and multiple searchers. In the system, unidentiﬁed ﬂight vehicles, UFVs, are surveillance targets. The searchers are
required to approach the UFVs for identiﬁcation. In this regard, however, faster UFVs move while avoiding the searchers. In order for the
searchers to approach and detect such sophisticated UFVs, cooperative
actions play an important role. Therefore, we apply reinforcement learning to the searchers. In the initial learning phase, however, searchers
based on random actions seldom detect UFVs. Thus the searchers cannot acquire eﬀective actions to approach the target UFV. Therefore, we
further apply transfer learning to this problem. Through surveillance
simulations, we show the eﬀectiveness of the cooperative actions by themultiple searchers for the UFVs in the aerial surveillance system.
Keywords: multi-agent systems, reinforcement learning, cooperative actions, aerial surveillance

1

Introduction

In recent decades, unmanned aerial vehicles, UAVs, have been employed in several application domains, such as wildﬁre tracking, monitoring, search and rescue, and surveillance and reconnaissance [1][2]. In this paper, we focus on anaerial surveillance system composed of multiple UAVs. Basilico and Carpin havepresented an aerial surveillance system composed of two types of UAVs [3], as
illustrated in Fig. 1.
As the aerial agents, single sentinel and multiple searchers operate in thesystem. The sentinel operates at a place higher than the searchers. A sensing
range of the sentinel is drawn by a pink circle. If an unidentiﬁed event occurs inthe surveillance area, the sentinel detects the position. However, the sentinel is
not capable of identifying the event precisely. On the other hand, searchers areable to identify the event in the sensing range drawn by red circles. Basilico and
Carpin have succeeded in identifying events by dispatching the searchers.
For aerial surveillance with multiple UAVs, the area is divided and assigned
to each of the UAVs [4]. In order to cover the assigned area, the UAV followed
a path based on a deterministic zigzag pattern. On the other hand, cooperative2

S. Hoshino and K. Kikuchi
Sentinel

SearcherSearcherSearcherSensing range of searcher= detection area
Sensing range of sentinel = surveillance area

Fig. 1. Aerial surveillance system composed of sentinel and searchers

path planning for multiple UAVs in the same area has been proposed [5]. Furthermore, multi-UAV path planning in consideration of the connectivity among
the UAVs has been proposed [6]. However, static targets have been assumed tooccur randomly at any place, as described in [7].
In this paper, we assume unidentiﬁed ﬂight vehicles, UFVs, as surveillancetargets. As well as the system illustrated in Fig. 1, one sentinel and multiple
searchers are used. Compared to the static targets, searchers are required toapproach UFVs in consideration of the movement.
In our previous work [8], the movement of the UFVs was stochastically predicted. According to the movement, optimal approaching strategies have been
proposed with the use of the value iteration method. However, the UFVs wereassumed to move linearly ignoring searchers. Hence, it is diﬃcult to predict the
movement if the UFVs change the actions due to the searchers. Therefore, weapply reinforcement learning to the searchers in consideration of the interaction.
In the ﬁeld of multi-agent reinforcement learning, this issue has been treated asa pursuit game [9]. However, faster pursuers than evaders have been assumed.
To our knowledge, a superior evader was assumed to move at the same speed as
pursuers [10].
In the aerial surveillance system, on the other hand, we assume the UFVs
faster than the searchers. Furthermore, the UFVs are allowed to move while
avoiding the searchers through interaction. Therefore, it might be diﬃcult forthe searchers to approach and detect such sophisticated UFVs as long as thesearchers take actions independently from each other. For this challenge, cooperative actions among the multiple searchers play an important role.
In the reinforcement learning for cooperative surveillance, the searcher stateis deﬁned on the basis of a target UFV and other searchers. As for a series ofactions, action weights (or policies) in the states are calculated depending on a
reward based on the surveillance result. A searcher in a state selects a next action
based on the action weight. In this regard, however, the state space is too large

Cooperative Searchers for Aerial Surveillance3

to explore. For multi-agent reinforcement learning, a deep Q-network, DQN, hasbeen used for Q-value approximation [11]. In stead of state exploration, therefore,a relationship between the state and action weight is approximately represented
by a deep neural network. Through the state-action mapping, action weights
are derived as the outputs even for the ﬁrst input states. In the surveillancesystem, the searchers operate by repeating the state observation and policybased decision making.
In the initial learning phase, however, searchers based on random actions
seldom detect UFVs. Thus the searchers cannot acquire eﬀective actions to approach the target UFV. Therefore, we further apply transfer learning [12] to thisproblem. In the initial learning phase, smaller environments with limited state
space are provided so that the searchers easily approach and detect UFVs. The
learning results are then transferred to the original surveillance environment.
Through surveillance simulations, we show the eﬀectiveness of the cooperative
actions by the multiple searchers for the UFVs in the aerial surveillance system.

2

Multi-Agent Aerial Surveillance System

Within the surveillance area as illustrated in Fig. 1, the sentinel constantly tracks
the positions of searchers and UFVs. The searchers are allowed to communicate
with the sentinel. Thus a searcher has information about the target UFV andother searchers via the sentinel. Fig. 2 illustrates the top view of the surveillancesystem with three searchers for an UFV as the target. The pink circle indicatesthe surveillance area. The UFV moves below the searchers.

SearcherSearcherSearcher UFV(a) Searchers vs. in- (b)Approaching (c) Detection (succoming UFV
searcherscess)(d) Failure

Fig. 2. Top view of aerial surveillance system with multiple (three) searchers

In Fig. 2(a), the incoming UFV is detected by the sentinel. The informationis shared with the three searchers. The UFV moves toward a destination locatedat the edge of the surveillance area. In Fig. 2(b), the searchers approach the UFV
by taking actions in each state repeatedly. In Fig. 2(c), eventually, a searcher
succeeds in detecting the UFV within the red sensing range. On the other hand,in Fig. 2(d), the UFV reaches the destination without being detected by the

4

S. Hoshino and K. Kikuchi

searchers. In this case, the searchers fail in detecting the UFV. As well as thesearchers, UFVs in the surveillance area are also allowed to have information
about the searchers by radar. Thus the UFVs are able to move toward thedestinations while avoiding the searchers.
For reinforcement learning, the surveillance process from the detection of
the incoming UFV by the sentinel to the success or failure of detection by thesearchers is called episode. At the end of each episode, a reward is given to
the searchers depending on the surveillance result, success or failure. With each
episode, the searchers gradually learn eﬀective actions to approach the UFVs.

3

Reinforcement Learning for Multiple Searchers

3.1

State Definition

In the aerial surveillance system, faster UFVs move so as to avoid the searchers.
In order for the searchers to approach and detect such sophisticated UFVs,
cooperative actions among the multiple searchers play an important role. For
cooperative surveillance, the searcher state is deﬁned on the basis of the relative
position and direction to the target UFV and other searchers as illustrated inFig. 3. The polar coordinate system centered on the host searcher reduces the
combination of parameters used for state deﬁnition.

Searcher j

φs
θs

ds

µsj

dsj

UFV

Searcher(a) State based on UFV

Searcher(b) State based
searcher(s)

on

other

Fig. 3. Deﬁnition of searcher state in polar coordinate system

The positions and moving directions of the searcher and UFV are represented
by x, y, and θ in the 2D plane. As described in section 2, the information is known
to both the searchers and UFV. In Fig. 3(a), ds and θs indicate the distance anddirection angle of the searcher to the UFV, respectively. A relative direction
between the searcher and UFV, φs , is further included in the state. In Fig. 3(b),Cooperative Searchers for Aerial Surveillance5

dsj and θsj indicate the distance and direction angle of the searcher to the othersearcher, j.
As described above, the searcher state at time t in an episode, st , is deﬁned
by the combination of dsj∈J and θsj∈J in Fig. 3(b) in addition to ds , θs , andφs in Fig. 3(a). As for the dsj∈J and θsj∈J , J indicates a set of other searchers
j excepting the host searcher i, i = j. Hence, it is expected that the searchers
can learn the approaching actions taking the moving direction of the UFV andcooperative relations with other searchers into account.
3.2

Discretized actions

In a state, st , searchers take actions so as to approach the UFV. The searcher
action, deﬁned by the directional change Δθ and moving distance d, is represented as at = {Δθta , dat }. Given continuous values of Δθ and d, the action spacebecomes too large to explore.
In the system, the searcher action is discretized by dividing into N directions.
In doing so, a set of actions in state st is represented by At = {a1t , a2t , · · · , aNt }.
A searcher selects an action, at , from a set of the discretized actions At . For1
instance, the searcher that selected action a1t rotates θta and moves a distance1of dat .
3.3

Rewards based on surveillance results

At the end of each episode, positive or negative rewards are given to the searchers
depending on the surveillance result, success (see Fig. 2(c)) or failure (see Fig.
2(d)). If the negative rewards are equally given to the searchers in case of failure,
two series of actions of approaching and leaving searchers are both evaluated as
ineﬀective for surveillance. In this case, therefore, the negative rewards dependingon the ﬁnal distance to the UFV are given to the searchers as follows:r(success) orR=(1)d− fDin r (failure),
where r is a reward constant, df in is the distance to the UFV at the end of theepisode, and D is a diameter of the surveillance area.
From Eq. (1), if any of the searchers detects the UFV, the positive rewards
are equally given to all the searchers. On the other hand, the negative reward
increases as the ﬁnal distance to the UFV increases in case of failure.

4

Surveillance Policy

4.1

Profit sharing with expected value

In order for a searcher in state st at time t to take action at , action weight
W (st , at ) is used as an indicator. As the action weight is increased through6

S. Hoshino and K. Kikuchi

reinforcement learning, the searcher tends to select action at in state st . The
action weight in each state is calculated from rewards. As described in section
3.3, the rewards are given to the searchers at the end of the episode dependingon the surveillance result.
After the episode, action weights corresponding to a series of actions arecalculated. For this purpose, we present proﬁt sharing [13] with expected value,
PSw/EV. In addition to the given reward, the action weight is calculated inconsideration of the expected value of the reward in each state. After that, theaction weight is updated as follows:
W (st , at ) ← W (st , at ) + γ (τ −t) {R − E(st )},(2)

where γ is a discount rate, τ is an end time of the episode, and E(st ) denotes the
expected value of the reward in state st , E(st ) = Σe R(st )/N (st ), where Σe R(st )indicates the sum of rewards given in episodes e in which the searcher observed
state st and Ne (st ) is the number of episodes e.
During each episode, a series of actions, a1 , a2 , · · ·, in states, s1 , s2 , · · ·, isstored. After the episode, the action weight at time t, W (st , at ), based on state
st and action at is updated according to the diﬀerence between the given reward
R and expected value of the reward in state st , E(st ). From Eq. (2), W is
increased if R is higher than E(st ), and vice versa, through the PSw/EV.
4.2

RBFnet for state-action mapping

The searcher state is deﬁned by continuous values as described in section 3.1.
Thus in section 4.1, it is impossible to explore the whole state space and calculate
W (S, A) for all the state-action pairs, s ∈ S and a ∈ A. In this paper, therefore,a relationship between the state s and action weight W (s) is formulated. For
state-action mapping, a radial basis function network [14], RBFnet, is used.
The RBFnet is based on a deep neural network composed of three, input,
hidden, and output layers. In the RBFnet, radial basis functions are applied tothe units in the hidden layer. As a result, the relationship between the input
state s and output action weight W is represented by the following equation:
W (s; ds , θs , φs , dsj∈J , θsj∈J ) =Kwk ϕk (s),

(3)

k=1

where K is the number of units in the hidden layer. wk denotes connection
weights between k-th unit in the hidden layer and output units. Since theaction is discretized
into N directions, the connection weights are wk =searcher

a1
a2
aN
wk , w k , · · · , wk
for the N output units. ϕk () is the radial basis function
based on the following Gaussian kernel:||s − μk ||ϕk (s) = exp −
,
(4)
σk2

Cooperative Searchers for Aerial Surveillance

7

where μk and σk are a central value and variance of the Gaussian kernel in thek-th unit.
Through state-action mapping in Eq. (3), the RBFnet accepts not only anobserved state, s, but also the other similar states even if the states are notobserved by the searcher. For these input states, the actions weights of Wa1 (s)to WaN (s) are derived from the N output units. Fig. 4 illustrates an example
structure of the RBFnet with three searchers. The input layer is composed of
3 + 2J units, i.e., three units for the UFV and 2 × J units for other searchersj ∈ J. Given the three searchers, J = 2, the input layer is composed seven units
in total. The hidden and output layers are composed of K Gaussian units andN units, respectively.
Hidden (Gaussian units)
Input

'1
1

w1a

ds
µs
Normalize

Ás

s'

ds2

s

W a1 (s)W a2 (s)W a3 (s)

1

µs2

wKa

ds3
µs3

Output

W aN(s)'K

Fig. 4. Structure of RBFnet with three searchers

The input state, s, is ﬁrst oﬀ normalized to s . Hence, each parameter of ds ,θs , φs , ds2 , θs2 , ds3 , and θs3 has a value of [0, 1]. These values are then fed as
the inputs to the RBFnet. After that, ϕ1 to ϕK are calculated by Eq. (4) inthe K Gaussian units. Through the connection weights of w, the action weights,
Wa1 (s) to WaN (s), are calculated by Eq. (3) and derived as the outputs.
After each episode, action weights are updated by Eq. (2). For the outputs(st ) derived from the RBFnet, only the output unit, Wat (st ),of Wa1t (st ) to WaNtcorresponding to action at taken in state st is used as the target. In other words,
the connection weights to the output unit in Eq. (3) are updated so that Wat (st )equals to W (st , at ) as follows: wk ← wk + ηδϕk (st ), where η is a learning rateand δ is a update value, calculated from δ = γ (τ −t) {R − E(st )}.
4.3

Boltzmann action selection based on action weights

In state st at time t, the searcher probabilistically selects an action depending on(st ). For this purpose, Boltzmann action
the action weights of Wa1t (st ) to WaNtselection [15] is used. The searchers repeat the decision making until the end

8

S. Hoshino and K. Kikuchiof the episode. For Boltzmann action selection, each action weight Wat (st ) isconverted to the following probability, p:
Wat (st )
exp
T

,
p(at |st ) =(5)
Wan (st )Ntan expTtwhere T is the temperature, which is a time-varying parameter.
From Eq. (5), the probability, p, increases as the action weight, Wat (st ),is increased. Therefore, an action with higher weight tends to be selected. In
general, parameter T decreases to 0 with time. Eventually, the action weight
has a greater impact on Eq. (5). Thus the Boltzmann action selection throughstate-action mapping in section 4.2 is the surveillance policy for the searchers.

5

Transfer Learning

In the initial learning phase, action weights W are uniformly given to all thestates s ∈ S for all the actions a ∈ A. The surveillance policy allows the searcher
to select each action with equal probability as expressed in Eq. (5). In otherwords, the searchers gradually learn eﬀective actions from the random actions
by detecting the target UFV fortuitously.
However, the searchers based on the random actions seldom detect sophisticated UFVs. As long as the searchers fail to detect the UFV, the negative
rewards continue to be given to the searchers. Even if a searcher partially takes
eﬀective actions, the action weights are not increased in case of failure. As aresult, the searchers cannot acquire the eﬀective actions to approach the UFV
through reinforcement learning.
This learning problem is due to the large state space. Therefore, we further
apply transfer learning to this problem. In transfer learning, the ﬁnal action
weights WA (SM −1 ) in smaller environment DM −1 are used as the initial action
weights WA (SM ) in larger environment DM . The state space of DM −1 and DM
is represented as SM −1 ⊆ SM . Fig. 5 illustrates the process of transfer learning.

Searcher 3

S1

S2
WA(S1) S1

WA(S2)

UFV

SM-1
SM-2

WA(SM-1)

SM
SM-1

・・・

Searcher 2
Searcher 1

D1

D2

DM-1

Fig. 5. State space expansion for transfer learningDM

Cooperative Searchers for Aerial Surveillance

9

In Environment D1 , the state space S1 is limited to around the UFV. In thestate space, the searchers are already close to the UFV and arranged so as to surround the UFV. In these states, even the searchers based on the random actions
can detect the UFV. Moreover, the searchers are allowed to observe the state
space and update the action weights suﬃciently. After that, the action weights,
WA (S1 ), in environment D1 are transferred to the expanded environment, D2 ,and used as the initial action weights. The searchers mainly observe the additional state space, S2 − S1 , and update the action weights WA (S2 ). This process
is repeated until the size of the environment reaches the surveillance area DM .
Through transfer learning, random actions of the searchers in the initial
learning phase are conﬁned to the limited state space and additional state space.
In the initial limited state space, the searchers detect the UFVs with random
actions. Therefore, more positive rewards are given to the searchers. As a result,the learning eﬃciency is increased, and the searchers in the surveillance area areallowed to learn the eﬀective actions to approach the UFV with episodes.

6

Surveillance Simulation

6.1

Simulation settings

In this simulation experiment, single sentinel and three searchers are used forsurveillance. In each episode, a target UFV comes into the surveillance area.
Fig. 6 illustrates the environment of aerial surveillance system. Please note thatﬁxed-wing UAVs diﬀerent from drones are assumed in the system.

UFV

Searcher 3

030

Searcher 1

Searcher 2
Destination(a) Three searchers vs. target UFV

50

]
m
[k

]
m
[k

(b) Sensing range

Fig. 6. Surveillance environment

The initial positions of three searchers are as indicated in Fig. 6(a). As for thetarget UFV, both the incoming position and destination are randomly selectedfrom the edge of the surveillance area. In Fig. 6(b), the sensing ranges of thesentinel and searchers are provided. The radius of the surveillance area is 300

10

S. Hoshino and K. Kikuchi[km]. Within the radius of 50 [km], the searchers detect the UFV, and the detail
information is identiﬁed.
The ﬁxed-wing searchers move at a velocity of 300 [km/h]. Unlike drones, thesearchers cannot control the moving direction rapidly and precisely. Hence, theaction is discretized as A = {a1 , a2 , a3 , a4 , a5 }, by dividing into N = 5 directions.
Thus the searcher based on action a = {Δθa , da } rotates the direction −90 [deg]by a1 , −45 [deg] by a2 , 0 [deg] by a3 , 45 [deg] by a4 , and 90 [deg] by a5 , andmoves a distance of da = 30 [km], in each step corresponding to 0.1 [h].
The UFV moves at a velocity of 450 [km/h]. In consideration of the interaction with the searchers, the moving direction of the UFV is determined by
the resultant force of Fa and Fr . Fa , the attractive force is generated from thedestination as follows:
a1 dd + b1 (dd < dth1 ) orFa (dd ) =(6)(dd ≥ dth1 ),b1where dd is a distance to the destination, a1 and b1 are coeﬃcients, and dth1 is a
threshold for the distance. Fr , the repulsive force is generated from each of thesearchers as follows:
a2 di + b2 (di < dth2 ) orFr =Fr (di ) =(7)0(di ≥ dth2 ),iwhere di is a distance to the i-th searcher, i = {1, 2, 3}. a2 and b2 are coeﬃcientsand dth2 is a threshold for the distance.
From Eq. (6) and Eq. (7), within a certain distance of dth1 and dth2 , theattractive force and repulsive force increase as the distance reduces, and aﬀect
the UFV. These coeﬃcients and thresholds are given as a1 = −6, b1 = 210,dth1 = 30, a2 = −0.3, b2 = 60, and dth2 = 200. As well as a superior evader in[10], the movement model of the UFVs is unknown to the searchers.
6.2

Learning conditions

Under the following three learning conditions, the searchers acquire the eﬀective
actions to approach the UFV depending on the combination of the input statesand presence or absence of transfer learning:
1. state based on the UFV through non-transfer learning;
2. state based on the UFV and other searchers through non-transfer learning;
3. state based on the UFV and other searchers through transfer learning.
In 1, the searcher state is deﬁned by ds , θs , and φs . In 2, the searcher state isdeﬁned by ds2 , θs2 , ds3 , and θs3 in addition to ds , θs , and φs . These normalized
parameters are fed as the inputs to the RBFnet in Fig. 4. For these input states,
action weights are derived from Eq. (3). In 3, the searchers ﬁrst operate in asmaller surveillance area with a radius of 200 [km] for transfer learning. After
1000 episodes, the action weights are transferred to the original surveillance area.

Cooperative Searchers for Aerial Surveillance

11

In Eq. (1), the reward constant is given as r = 1. As for the PSw/EV, thediscount rate in Eq. (2) is given as γ = 0.9. In Eq. (3), the number of Gaussianunits in the RBFnet is K = 107 . The variance of the Gaussian kernel in Eq. (4)is σ = 0.142. For Boltzmann action selection, a constant value is given to thetemperature in Eq. (5) as T = 90 to reduce hyperparameters.
6.3

Simulation results

Success (detection) rate

Surveillance actions acquired by the searchers through reinforcement learning
are evaluated on the basis of the success rate in each 1000 episodes. Fig. 7
shows the results under the three conditions for 20000 episodes in total.
40

1

2

3
3

30

2

20

10
1
0
0

5000

10000

15000

20000

# of episodes

Fig. 7. Transition of success rate: # of detections / 1000 (episodes)

As indicated by  plots, the success rate of surveillance under 1 was notincreased with episodes. The ﬁnal success rate remained at around 5 [%]. On theother hand, the success rate of surveillance under 2 increased to 22.6 [%], i.e.,
more than quadruple compared to the result of 1, as indicated by  plots. The
surveillance actions of the searchers under the conditions, 1 and 2, are comparedin an episode. Fig. 8 shows the approaching trajectories of the searchers for thesame target UFV after 20000 episodes. Arrows denote the moving directions of
the searchers and UFV at the time.
In Fig. 8(a), the searchers directly approached the UFV independently fromeach other. As a result of the actions, the searchers failed in detecting the UFV.
In Fig. 8(b), on the other hand, the searchers cooperatively approached so as
to surround the UFV. As a result of the cooperative actions, the searchers succeeded in detecting the UFV. These results show that the searchers learned thecooperative actions eﬀective for surveillance based on the state deﬁned by the
UFV and other searchers.
In Fig. 7, while the ﬁnal success rate under 2 was 22.6 [%], the initial successrate for the ﬁrst 1000 episodes was 1.5 [%]. This result was as low as the initial

12

S. Hoshino and K. Kikuchi

UFV

Destination(a) 1: UFV escaped from searchers(b) 2: detected UFV

Fig. 8. Comparison of trajectories — a series of actions of three searchers

success rate under 1. In contrast, the initial success rate under 3 was increased
to 8.2 [%] as indicated by ◦ plots. This result shows that the searchers eﬃciently
learned the eﬀective actions. Because of the increased learning eﬃciency, theﬁnal success rate was also increased to 32.2 [%]. Compared to the result of 2,
the surveillance performance was increased more than 10 [%].
As described above, the ﬁnal success rates were 5.4, 22.6, and 32.2 [%] depending on the learning conditions. For your information, the searchers based on
an optimal approaching strategy proposed in [8] resulted in the success rate of
13.4 [%]. As for the remaining failure rates, the UFVs reached the destinations
at a rate of 70.1(/94.6), 43.1(/77.4), and 37.4(/67.8) [%], respectively. From theresults, the eﬀectiveness of the cooperative actions by the multiple searchers for
the UFVs in the aerial surveillance system was shown.

7

Conclusions

In this paper, we proposed an aerial surveillance system composed of single sentinel and multiple searchers. For sophisticated UFVs, the searchers successfullyacquired eﬀective actions through reinforcement learning. In the surveillance
simulations, the searchers approached the target UFV in cooperation with eachother. As a result, the success rate of surveillance was increased to 22.6 [%] from
5.4 [%]. Furthermore, transfer learning increased the initial success rate from
1.5 [%] to 8.2 [%]. Because of the increased learning eﬃciency, the ﬁnal successrate was also increased to 32.2 [%], and the system yielded the best surveillance
performance. From the results, the eﬀectiveness of the cooperative actions by
the multiple searchers for the UFVs in the aerial surveillance system was shown.
In this experiment, we used a computer with Intel Core i7-6950X (CPU),NVIDIA TITAN RTX (GPU), and 64G bytes of memory. As for the computational cost, condition 1 (three parameters for state deﬁnition) spent aboutCooperative Searchers for Aerial Surveillance13

24 hours and conditions 2 and 3 (seven parameters for state deﬁnition) spent
about four days in the simulations. In order for the surveillance system to further
increase the success rates, more searchers are required. However, the computational cost increases with the number of searchers. For the curse of dimensionality, transfer learning would be a possible solution. In future works, therefore,
we will expand the state space little by little in applying transfer learning.

References
1. Cabreira, T. M., Brisolara, L. B., and Ferreira Jr., P. R.: Survey on Coverage Path
Planning with Unmanned Aerial Vehicles, Drones, Vol. 3, No. 4, 2019.
2. Zhou, X., Yi, Z., Liu, Y., Huang, K., and Huang. H.: Survey on Path and View
Planning for UAVs, Virtual Reality & Intelligent Hardware, Vol. 2, No. 1, pp. 56–69,
2020.
3. Basilico, N. and Carpin, S.: Deploying Teams of Heterogeneous UAVs in Cooperative Two-Level Surveillance Missions, IEEE/RSJ International Conference onIntelligent Robots and Systems, pp.610–615, 2015.
4. Maza, I. and Ollero, A.: Multiple UAV Cooperative Searching Operation Using
Polygon Area Decomposition and Eﬃcient Coverage Algorithms, Distributed Autonomous Robotic Systems, pp. 221–230, 2007.
5. Flint, M., Polycarpou, M., and Fernandez-Gaucherand, E.: Cooperative Control for
Multiple Autonomous UAV’s Searching for Targets, IEEE Conference on Decision
and Control, pp. 2823–2828, 2002.
6. Hayat, S., Yanmaz, E., Bettstetter, C., and Brown, T. X.: Multi-Objective Drone
Path Planning for Search and Rescue with Quality-of-Service Requirements, Autonomous Robots, Vol. 44, pp. 1183–1198, 2020.
7. Basilico, N., Chung, S., and Carpin, S.: Distributed Online Patrolling with MultiAgent Teams of Sentinels and Searchers, Distributed Autonomous Robotic Systems,
pp. 3–16, 2016.
8. Hoshino, S., Katsumoto, T., Saitoh, S., Itabashi, N., Kabayama, T., and Kono,
M.: Aerial Surveillance System with Multiple Agents for Moving Targets, SICE
Annual Conference, pp. 427–432, 2020.
9. Arai, S. and Sycara, K.: Eﬀective Learning Approach for Planning and Scheduling in Multi-Agent Domain, International Conference on Simulation of Adaptive
Behavior, pp. 507–516, 2000.
10. Wang, Y., Dong, L., and Sun, C.: Cooperative Control for Multi-Player PursuitEvasion Games with Reinforcement Learning, Neurocomputing, Vol. 412, 101–114,
2020.
11. Omidshaﬁei, S., Pazis, J., Amato, N., How, J. P., and Vian, J.: Deep Decentralized Multi-Task Multi-Agent Reinforcement Learning under Partial Observability,
International Conference on Machine Learning, pp. 2681–2690, 2017.
12. Taylor, M. E. and Stone, P.: Transfer Learning for Reinforcement Learning Domains: A Survey, Journal of Machine Learning Research, Vol. 10, No. 5, pp. 1633–
1685, 2009.
13. Grefenstette, J. J.: Credit Assignment in Rule Discovery Systems Based on Genetic
Algorithms, Machine Learning, Vol. 3, pp. 225–245, 1988.
14. Poggio, T. and Girosi, F.: Networks for Approximation and Learning, Proceedingsof the IEEE, Vol. 78, No. 9, pp. 1481–1497, 1990.
15. Sutton, R. S. and Barto, A. G.: Reinforcement learning, MIT Press, pp. 32–33,
1998.